A remote sensing image classification algorithm and method based on parallel 3D-2D-1D CNN
By combining the parallel 3D-2D-1D CNN structure with space-spectral features and spectral features, the problems of low classification accuracy and high calculation cost of hyperspectral remote sensing images are solved, and the combination of high accuracy and low cost is achieved, which significantly reduces the calculation amount And it shows excellent classification effects in practical applications.
Patent Information
- Application Number
- CN202111493066.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-12-08
AI Technical Summary
The existing hyperspectral remote sensing image classification method has problems with low classification accuracy and high computational cost. Multi-scale 3D CNN sacrifices computing efficiency, split 3D CNN fails to effectively fuse space-spectral information, and 3D-2D CNN based on attention mechanism Spectral characteristic advantages are not fully considered.
The parallel 3D-2D-1D CNN structure is used to combine the null-spectral features, spatial features and spectral features of remote sensing images, and features are extracted through parallel 3D CNN, 2D CNN and 1D CNN layers, and over-similarity is alleviated through feature stitching and Dropout functions. Together, use Linear classifier to implement classification.
Improve classification accuracy, and significantly reduce computing costs, achieve coexistence of high accuracy and low cost, and reduce the calculation volume of 3D CNN on multiple scales to 48%, and show excellent classification results on the actual data set and Low calculation cost.
Smart Images

Figure CN114220021B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of remote sensing image processing, and more specifically, to a remote sensing image classification method and algorithm based on parallel 3D-2D-1D CNN. Background Art
[0002] Hyperspectral remote sensing technology uses hyperspectral sensors to simultaneously image the target area with continuous bands to obtain hyperspectral images with combined spatial and spectral information. Hyperspectral images contain rich spectral information and spatial information of objects and are widely used in agricultural remote sensing, environmental protection, earthquake prevention and disaster relief, and other fields.
[0003] In the early stages of hyperspectral image classification, the main methods used were neighbor functions, decision trees, and linear functions. However, the main problem with these classic classifiers is that they are prone to the Hughes phenomenon (referring to the phenomenon that in the process of hyperspectral analysis, as the number of bands involved in the operation increases, the classification accuracy "increases first and then decreases"). In order to solve this phenomenon, Camps-Valls et al. proposed the support vector machine (SVM) method, which can effectively suppress the occurrence of the Hughes phenomenon and has become the mainstream classification method for a long time. The disadvantage is that the SVM algorithm has low classification accuracy and cannot be applied in practice, but this problem has been effectively solved in deep learning, among which the convolutional neural network has the best effect.
[0004] Convolutional neural network (CNN) has attracted great attention because of its layer-by-layer structure, which can extract more implicit and deeper features in layers. Common hyperspectral classification methods based on CNN models: 2D CNN extracts image spatial information, and 3DCNN can take into account the extraction of spectral information and spatial information. In the application of convolutional neural networks, some scholars first used 2D CNN to extract the spatial features of hyperspectral images. The application of 2D CNN has higher classification accuracy than the SVM model, but ignores the spectral information unique to spectral images. Before this, some scholars also used the AutoEncoder model to extract hyperspectral features, but this method only considered the spectral information of spectral images and ignored the spatial information of the image. Makantasis et al. proposed a new deep learning-based method that combines 2D CNN with a multi-layer perceptron, uses 2D CNN to extract image spatial information, and uses the multi-layer perceptron for classification; Li Yang et al. proposed a three-dimensional convolutional neural network (3D CNN) framework for remote sensing image classification that does not rely on any preprocessing or postprocessing. This method effectively fuses the spectral information and spatial information of hyperspectral remote sensing images through a three-dimensional convolution kernel. Although this method effectively improves the classification accuracy, it also increases the computational cost.
[0005] In response to the above problems of low classification accuracy and high computational cost, scholars have proposed different methods: some scholars have proposed a multi-scale three-dimensional deep convolutional neural network (M3D DCNN) model, which can extract spectral features and spatial features from remote sensing images in an end-to-end manner, and improve classification accuracy by effectively combining the two. Others have proposed to reduce the computational cost by splitting 3DCNN into one two-dimensional convolution and two one-dimensional convolutions. Another proposed a 3D-2D CNN algorithm based on the attention mechanism, which first uses the attention mechanism to reduce the dimensionality of the spectral image, then uses 3D CNN to extract the joint spatial-spectral information and uses 2D CNN to learn more abstract spatial information in the image. Compared with 3D CNN, this method reduces the computational cost while maintaining the classification accuracy. Although the above methods have effectively improved the problems of low classification accuracy and high computational cost, there are still problems: multi-scale 3D CNN improves classification accuracy at the expense of computational efficiency; split 3D CNN fails to effectively fuse spatial-spectral information; 3D-2D CNN based on the attention mechanism does not fully consider the advantages of spectral features. Summary of the invention
[0006] In view of this, in order to simultaneously solve the problems of low classification accuracy and high computational cost in the existing remote sensing image classification and distribution, the present invention provides a remote sensing image classification method and algorithm based on parallel 3D-2D-1D CNN, which achieves improved classification accuracy by fusing the spatial-spectral features, spatial features and spectral features of remote sensing images through a parallel 3D-2D-1D CNN structure; the formed algorithm can effectively reduce the computational cost.
[0007] In the first aspect, the present application provides a remote sensing image classification method based on parallel 3D-2D-1D CNN:
[0008] This method combines the spatial-spectral features, spatial features and spectral features of remote sensing images, utilizes the feature information of remote sensing images, and adopts the method of parallel 3D CNN, 2D CNN and 1D CNN, which effectively improves the computational efficiency while maintaining the computational cost at a low level.
[0009] In the second aspect, the present application provides a remote sensing image classification algorithm based on parallel 3D-2D-1D CNN, comprising the following steps:
[0010] 1) Input data: split the original image into The small blocks are input into the system, and the image features are initially extracted through 3D CNN, while the single-channel input map is converted into a 16-channel feature map;
[0011] 2) Feature extraction: Use parallel 3D-2D-1D CNN layers to extract the spatial-spectral features, spatial features, and spectral features of remote sensing images respectively.
[0012] The calculation formulas of 3D CNN, 2D CNN and 1D CNN are shown in formula (1), formula (2) and formula (3):
[0013]
[0014]
[0015]
[0016] In the above formula, Represents the number of convolutional layers, For the Tier The pixel deviation in the feature vector is Indicates Convolution kernel The convolution kernel at position, and Indicates the size of the convolution kernel, which respectively indicates the length and width of the convolution kernel. Represents the size of the three-dimensional convolution kernel, which is the value in the spectral dimension of the hyperspectral image. represents the activation function, Representative Layer The feature map is at position The value at .
[0017] 3) Learning high-level and abstract features of images: concatenating the extracted feature maps, and then using a set of parallel 3D-2D-1D CNN layers to further extract high-level and abstract features of remote sensing images;
[0018] 4) Alleviate overfitting: Add the Dropout function to randomly remove a certain proportion (the specific proportion to be removed is determined based on the most effective results) of the selected neuron nodes to prevent overfitting;
[0019] 5) Realize the classification of remote sensing images: Use Linear classifier to classify test samples.
[0020] In combination with the second aspect, the use of parallel 3D-2D-1D CNN layers in step 2 to respectively extract the spatial-spectral features, spatial features and spectral features of the remote sensing image is specifically stated as follows: assuming the number of channels is n, n 3D CNNs are used to extract the image spatial-spectral features, n 2D CNNs are used to further extract the image spatial features, and n 1D CNNs are used to further extract the image spectral features; then the extracted feature images are spliced and transmitted to the next layer.
[0021] In combination with the second aspect, the step 3 further uses a set of parallel 3D-2D-1D CNN layers to further extract high-level features and abstract features of the remote sensing image. Specifically, n 3D CNNs, n 2D CNNs and n 1D CNNs are used to extract features of the spliced image, so as to further integrate the spatial-spectral features, spatial features and spectral features, and further extract the high-level features and abstract features of the remote sensing image.
[0022] The technical means of the present invention brings the following beneficial effects:
[0023] 1) The present invention utilizes convolutional neural networks of different dimensions to extract the characteristics of features of remote sensing images in different dimensions, utilizes 3D CNN, 2D CNN and 1D CNN to extract the spatial-spectral features, spatial features and spectral features of the image respectively, and fuses the obtained features of different dimensions through the same parallel 3D-2D-1D CNN layer; the three-dimensional convolutional neural network can fuse the spatial features and spectral features of the objects, the two-dimensional convolutional neural network helps to identify the objects with unclear spectral features, and the one-dimensional convolutional neural network can extract deeper spectral information. By fusing the features of the three dimensions, the advantages of the features of each dimension can be well utilized, thereby effectively improving the classification accuracy, that is, improving the classification precision.
[0024] 2) The present invention does not use all 3D CNNs, but replaces part of 3D CNNs with 2D CNNs and 1D CNNs, thereby improving classification accuracy while reducing computational costs.
[0025] It is worth pointing out that the above-mentioned improvement in classification accuracy and reduction in computing cost are achieved simultaneously. Normally, these two technical advantages cannot be satisfied at the same time, that is, the cost of an algorithm with high classification accuracy is usually high, while the classification accuracy of an algorithm with low cost is usually very low; and the technical means of the present application enables these two advantages to "coexist" at the same time, which has very important guiding significance.
[0026] Other features and advantages of the present invention will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present invention.
[0027] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are specifically listed below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is the overall network structure diagram of the present invention.
[0029] Figure 2 This is a schematic diagram comparing the classification results of different algorithms for the Indian Pines dataset.
[0030] Including a, b, c, d, e, f. DETAILED DESCRIPTION
[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0032] The currently mature remote sensing image classification algorithms cannot take into account both the classification accuracy and computational cost. Generally, the computational cost of algorithms with high classification accuracy is often high, while the cost of algorithms with low classification accuracy is often reduced accordingly; but in actual needs, we often pursue high classification accuracy and low computational cost. For this purpose, this application proposes the following technical solution. To facilitate understanding of this embodiment, the remote sensing image classification algorithm based on parallel 3D-2D-1D CNN disclosed in the embodiment of the present invention is first introduced in detail.
[0033] See also Figure 1 What is shown is a flowchart of the remote sensing image classification algorithm, which includes the following steps:
[0034] The first step is to input data: split the original image into The small blocks of the image are input into the system, and the image features are initially extracted through 16 3D CNN convolution kernels, while the single-channel input image is converted into a 16-channel feature map;
[0035] The second step is feature extraction: 3D CNN is used in parallel to extract the spatial-spectral features of remote sensing images, 2D CNN is used to extract spatial features, and 1D CNN is used to extract spectral features.
[0036] The calculation formulas of 3D CNN, 2D CNN and 1D CNN are shown in formula (1), formula (2) and formula (3):
[0037]
[0038]
[0039]
[0040] In the above formula, Represents the number of convolutional layers, For the Tier The pixel deviation in the feature vector is Indicates Convolution kernel The convolution kernel at position, and Indicates the size of the convolution kernel, which respectively indicates the length and width of the convolution kernel. Represents the size of the three-dimensional convolution kernel, which is the value in the spectral dimension of the hyperspectral image. represents the activation function, Representative Layer The feature map is at position The value at .
[0041] The third step is to learn the high-level and abstract features of the image: concatenate the extracted feature maps, and then use a set of parallel 3D-2D-1D CNN layers (using 3D, 2D, and 1D CNNs to extract features from the input image, respectively) to further extract the high-level and abstract features of the remote sensing image.
[0042] The fourth step is to alleviate the overfitting phenomenon: add the Dropout function to randomly remove neuron nodes at a ratio of 0-80% (the specific ratio to be removed is determined based on the final result) to prevent the occurrence of overfitting;
[0043] The fifth step is to achieve the classification of remote sensing images: use the Linear classifier to classify the test samples.
[0044] Through the above steps, the goal of high accuracy and low cost can be achieved;
[0045] The above algorithm is used to reduce the calculation cost, and the specific performance is as follows:
[0046] For example, the input hyperspectral data block shape is , the number of convolution kernels in a multi-scale 3D CNN layer is , the size of a single convolution kernel is (Here we assume that the size of each convolution kernel of the multi-scale 3D CNN is equal), the step size of the convolution operation is 1, and the padding strategy is used to make the input and output data sizes of a convolution kernel in the parallel structure equal. Then, the computational complexity of a convolution operation for a multi-scale 3D CNN structure is As shown in formula (4):
[0047]
[0048] If the model achieves the best effect when there is an L-layer parallel structure, then the optimal model calculation amount is As shown in formula (5):
[0049]
[0050] Assume that each layer of the parallel 3D-2D-1D CNN model proposed in the present invention contains 3D convolution kernels, 2D convolution kernels and One-dimensional convolution kernel (where ), then the computational effort of this model As shown in formula (6):
[0051]
[0052] Among them, in each layer of parallel structure:
[0053] The computational cost of 3D convolution operation
[0054] The computational cost of two-dimensional convolution operation
[0055] The computational cost of one-dimensional convolution operation ;
[0056] The ratio of the computational effort of the parallel 3D-2D-1D CNN model to the multi-scale 3D CNN model is As shown in formula (7):
[0057]
[0058] Among them, it is advisable to take , ,but . It can be seen that the algorithm model proposed in the present invention has greatly reduced computational complexity compared to the multi-scale 3D CNN model (Q3 represents the computational complexity of the multi-scale 3D CNN, and Q2 represents the computational complexity of the technical solution of the present application. The ratio of the two is far less than 1, only 0.48, that is, the computational complexity of the technical solution of the present application is 48% of the computational complexity of the multi-scale 3D CNN).
[0059] 4) Alleviate overfitting: Add the Dropout function to randomly remove 50% of the neuron nodes to prevent overfitting.
[0060] 5) Realize the classification of remote sensing images: Use Linear classifier to classify test samples. Example
[0061] This embodiment uses three open source hyperspectral datasets, Indian Pines, Pavia Center and Pavia University, as classification objects and performs image classification according to the present invention. Figure 2 As shown, the classification results are compared with those of SVM, 2DCNN, 3D CNN and multi-scale 3D CNN methods, where Figure a is the real group and f is the technical group of this application. It can be seen that this application is the closest to the real group and has the highest graphic completeness.
[0062] The classification accuracy of the data set is shown in Tables 1 and 2. In the objective evaluation index comparison table shown in Table 1, the present invention achieves the optimal classification results in terms of the comparison of 0A, AA and KAPPA coefficients. In terms of the time index comparison in Table 2, the computational cost of the present invention is reduced by half relative to that of the multi-scale 3D CNN. This verifies the effectiveness and feasibility of the algorithm described in the present invention.
[0063] Table 1 Classification accuracy of Indian Pines, Pavia Center and Pavia University datasets / %
[0064]
[0065] Table 2 Training time and test time of each dataset in different algorithms / s
[0066]
[0067] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims
1. A remote sensing image classification algorithm method based on parallel 3D-2D-1D CNN, characterized by: The following steps are involved: 1) Input data: The original image is divided into small blocks of h×ww×C and input into the system. The image features are initially extracted through the 3D CNN convolution kernel, and the single-channel input image is converted into a multi-channel feature map corresponding to the number of convolution kernels. 2) Feature extraction: Use parallel 3D-2D-1D CNN layers to extract the spatial-spectral features, spatial features, and spectral features of remote sensing images respectively; The calculation formula for parallel 3D CNN, 2D CNN and 1D CNN is: In the formula, l represents the number of convolutional layers, b lj is the pixel deviation in the jth eigenvector of the lth layer, represents the convolution kernel at the position pqr on the mth convolution kernel, P1 and Q1 represent the size of the convolution kernel, which respectively represent the length and width of the convolution kernel, R1 represents the size of the three-dimensional convolution kernel, which is the value on the spectral dimension of the hyperspectral image, and f represents the activation function. Represents the value of the mth feature map in layer 1-1 at position (x+p)(y+q)(z+r); 3) Learning high-level and abstract features of images: concatenating the extracted feature maps, and then using a set of parallel 3D-2D-1DCNN layers to further extract high-level and abstract features of remote sensing images; 4) Alleviate overfitting: Add the Dropout function to randomly remove 0-80% of the neuron nodes to prevent overfitting; 5) Realize the classification of remote sensing images: Use Linear classifier to classify test samples.
2. The remote sensing image classification algorithm method based on parallel 3D-2D-1D CNN according to claim 1 is characterized in that: In the step 3), a set of parallel 3D-2D-1D CNN layers is used to further extract high-level features and abstract features of the remote sensing image. Specifically, n 3D CNNs, n 2D CNNs and n 1D CNNs are used to extract features of the spliced image, so as to further integrate the spatial-spectral features, spatial features and spectral features, and further extract the high-level features and abstract features of the remote sensing image.
