A hyperspectral classification method based on spatial-spectral dimension conversion
Through the hyperspectral classification method based on the airspace-spectral dimension conversion, the problems of information leakage and low pixel utilization in hyperspectral image classification in the prior art are solved, and the effect of full pixel classification and efficient utilization of hyperspectral data is achieved.
Patent Information
- Application Number
- CN202210007108.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-05
- Publication Date
- 2025-04-29
- Estimated Expiration
- 2042-01-05
AI Technical Summary
The existing hyperspectral image classification methods have problems such as information leakage and low pixel utilization during the dataset division process, and cannot fully utilize spectral information.
The hyperspectral classification method based on airspace-spectral dimension conversion is adopted, and the training set, verification set and test set are divided by equally spaced and non-overlapping, and the airspace-spectral dimension conversion is carried out to construct the hyperspectral classification network DSSFN, and the features are extracted and fusion classification is performed using SFCI-branch and SAS-branch branches.
The classification of all pixels is realized, the utilization rate and classification efficiency of scarce hyperspectral data are improved, and the network branch structure is simple, with good classification performance and generalization capabilities.
Smart Images
Figure CN114330591B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to a hyperspectral classification method based on spatial-spectral dimension conversion. Background Art
[0002] At present, traditional image classification methods rely on manually designed computational models to obtain the spectral features, spatial features, and spatial spectrum features of images. For example, spectral features are extracted using methods such as linear discriminant analysis, independent component analysis, and principal component analysis; spatial features are extracted using methods such as low-rank representation, morphological attributes, and gray-level co-occurrence matrix; and spatial spectrum features are obtained by designing methods such as three-dimensional wavelet transform, three-dimensional Gabor transform, and three-dimensional scattering wavelet transform. However, these shallow or mid-level features have limited expression capabilities and can no longer meet the high-precision requirements of modern classification tasks.
[0003] With the advent of the first aerial spectral imaging system AIS-1, hyperspectral remote sensing images have attracted widespread attention from scientific researchers and defense departments for their unique airspace information and spectral information and their high-altitude imaging advantages. On the one hand, modern military reconnaissance methods based on aerospace remote sensing technology have the characteristics of wide reconnaissance range, no geographical restrictions, fast target detection, and strong anti-concealment performance, which can provide combat troops with military intelligence that is difficult to obtain through other means. On the other hand, with the successive launch of civilian hyperspectral remote sensing satellites such as China's Gaofen-5 and Italy's PRISMA, hyperspectral images have been widely used in natural resource exploration, environmental pollution monitoring and natural disaster assessment, and hyperspectral image processing technology has gradually demonstrated its important application value in all walks of life.
[0004] Hyperspectral image classification technology is an important part of hyperspectral image processing technology. It aims to automatically and accurately identify the category of ground objects by using the unique spatial and spectral information in hyperspectral images. With the development of deep learning technology, researchers have proposed a series of high-performance hyperspectral image classification methods based on different types of advanced neural network architectures, such as: ResNet-based methods, Inception-based methods, DenseNet-based methods, and SENet-based methods. Many research results have shown that one-dimensional, two-dimensional, and three-dimensional convolutions can automatically and effectively extract the spectral, spatial, and spatial-spectral features of hyperspectral images. At the same time, combined with a multi-branch feature extraction structure, the classification accuracy of existing classification methods on public data sets is close to the "saturation" state.
[0005] Most of the existing deep learning-based classification methods use pixels as the center to perform image block segmentation during the data set division process, and then divide the segmented image blocks into training sets and test sets. This data division mode leads to a large number of overlapping areas in the training set and the test set, resulting in "information leakage", which directly affects the reliability of the performance evaluation of the classification method. If the training pixels are sent to the network for testing again, the classification performance will be too optimistic. In addition, the classification network based on the center pixel can only predict the category information of the center pixel, and the pixel utilization rate is low, wasting the already small number of labeled samples. In the study of hyperspectral image classification, most methods are based on the joint spatial-spectral method to extract features. These methods can only extract information on adjacent spectra, and cannot simultaneously extract information on non-adjacent spectra that are far away, and cannot fully utilize spectral information. Summary of the invention
[0006] The purpose of the present invention is to provide a hyperspectral classification method based on spatial-spectral dimension conversion, which can realize classification prediction of each pixel in an input image block and make full use of spectral information to improve the utilization rate and classification efficiency of scarce hyperspectral data.
[0007] To achieve the above object, the present invention adopts the following technical solution: a hyperspectral classification method based on spatial-spectral dimension conversion, the method comprising the following steps in order:
[0008] (1) The original hyperspectral dataset and the corresponding label vector dataset are divided into equally spaced and non-overlapping sets to obtain the training set X1, validation set X2, test set X3, as well as the training set label Y1, validation set label Y2, and test set label Y3;
[0009] (2) Perform spatial-spectral dimension conversion on the divided training set X1, validation set X2, and test set X3 to obtain the converted new training set X1_1, new validation set X2_1, and new test set X3_1; at the same time, perform dimension conversion on the training set label Y1, validation set label Y2, and test set label Y3 to obtain the new training set label Y1_1, new validation set label Y2_1, and new test set label Y3_1;
[0010] (3) Construct a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion;
[0011] (4) Input the training set X1, the validation set X2, and the new training set label Y1_1 and the new validation set label Y2_1 into the SAS-branch of the hyperspectral classification network DSSFN; input the new training set X1_1, the new validation set X2_1, and the new training set label Y1_1 and the new validation set label Y2_1 into the SFCI-branch of the hyperspectral classification network DSSFN, and use the Adam algorithm for iterative optimization to obtain the optimal network DSSFN with the best classification performance;
[0012] (5) Input the test set X3, the new test set X3_1 and the corresponding new test set label Y3_1 into the optimal network DSSFN to predict the classification results.
[0013] The step (1) specifically includes the following steps:
[0014] (1a) First, the entire original hyperspectral dataset is divided into W*H*C hyperspectral image blocks, where W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands;
[0015] (1b) Discard the hyperspectral image patches without any annotation information and use the hyperspectral image patches with only one pixel annotation as the test set, recorded as test set-1;
[0016] (1c) Sort the hyperspectral image blocks with more than one type of pixel label one by one according to the order of division and divide them into T groups. Randomly select one of the groups as the training set X1, and randomly select another group from the remaining data as the validation set X2. The remaining T-2 groups are used as the test set-2. Finally, the test set-1 and the test set-2 are combined into the test set X3. The labels at the corresponding positions of the training set X1, the validation set X2, and the test set X3 are used as the training set label Y1, the validation set label Y2, and the test set label Y3, respectively.
[0017] The step (2) specifically includes the following steps:
[0018] (2a) Extract the one-dimensional vector C*1 of all pixels in the training set X1, validation set X2, and test set X3 in the spectral dimension, where C represents the number of spectral bands;
[0019] (2b) Transpose the one-dimensional vector C*1 to get 1*C, and multiply the two vectors to get the spectral correlation matrix C*C;
[0020] (2c) The training set X1, validation set X2 and test set X3 data of size W*H*C are converted into data sets X1_1, X2_1 and X3_1 of size C*C*(W*H); W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands; the hyperspectral image blocks are obtained by dividing the entire original hyperspectral dataset;
[0021] (2d) The labels of size W*H*1 are stretched and dimensionally transformed to be converted into new training set labels Y1_1, new validation set labels Y2_1, and new test set labels Y3_1 of size 1*1*(W*H).
[0022] The step (3) specifically includes the following steps:
[0023] (3a) constructing a spectral full-channel interaction network branch SFCI-branch, wherein the spectral full-channel interaction network branch SFCI-branch includes a first three-dimensional convolutional layer, a first BN batch normalization layer, and a first RELU activation layer; a second three-dimensional convolutional layer, a second BN batch normalization layer, and a second RELU activation layer; a third three-dimensional convolutional layer, a third BN batch normalization layer, and a third RELU activation layer; a fourth three-dimensional convolutional layer, a fourth BN batch normalization layer, and a fourth RELU activation layer; a fifth three-dimensional convolutional layer, a fifth BN batch normalization layer, and a fifth RELU activation layer;
[0024] (3b) constructing a spatial domain-continuous spectrum feature extraction network branch SAS-branch, wherein the spatial domain-continuous spectrum feature extraction network branch SAS-branch includes a sixth three-dimensional convolution layer, a sixth BN batch normalization layer, and a sixth RELU activation layer; a seventh three-dimensional convolution layer, a seventh BN batch normalization layer, and a seventh RELU activation layer; an eighth three-dimensional convolution layer, an eighth BN batch normalization layer, and an eighth RELU activation layer; a ninth three-dimensional convolution layer, a ninth BN batch normalization layer, and a ninth RELU activation layer; a tenth three-dimensional convolution layer, a tenth BN batch normalization layer, and a tenth RELU activation layer; an eleventh three-dimensional convolution layer, an eleventh BN batch normalization layer, and an eleventh RELU activation layer; a first reshape layer and a second reshape layer;
[0025] (3c) The spectral full-channel interaction network branch SFCI-branch and the spatial-continuous spectral feature extraction network branch SAS-branch are integrated to construct a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion. The hyperspectral classification network DSSFN based on spatial-spectral dimension conversion includes a first Concate layer, a twelfth three-dimensional convolution layer and a classifier.
[0026] The input of the first three-dimensional convolutional layer is the new training set X1_1 and the new validation set X2_1. The output of the first three-dimensional convolutional layer is connected to the input of the first BN batch normalization layer, and the output of the first BN batch normalization layer is connected to the input of the first RELU activation layer; the output of the first RELU activation layer is used as the input of the second three-dimensional convolutional layer, and the output of the second three-dimensional convolutional layer is connected to the input of the second BN batch normalization layer, and the output of the second BN batch normalization layer is connected to the input of the second RELU activation layer; the output of the second RELU activation layer is used as the input of the third three-dimensional convolutional layer, and the output of the third three-dimensional convolutional layer is connected to the input of the third BN batch normalization layer, and the output of the third BN batch normalization layer is connected to the input of the third BN batch normalization layer. The output of the third RELU activation layer is used as the input of the fourth three-dimensional convolutional layer, the output of the fourth three-dimensional convolutional layer is connected to the input of the fourth BN batch normalization layer, and the output of the fourth BN batch normalization layer is connected to the input of the fourth RELU activation layer; the output of the fourth RELU activation layer is used as the input of the fifth three-dimensional convolutional layer, the output of the fifth three-dimensional convolutional layer is connected to the input of the fifth BN batch normalization layer, the output of the fifth BN batch normalization layer is connected to the input of the fifth RELU activation layer, and the output of the fifth RELU activation layer is used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
[0027] The input of the sixth three-dimensional convolutional layer is the training set X1, the validation set X2, and the test set X3. The output of the sixth three-dimensional convolutional layer is connected to the input of the sixth BN batch normalization layer, and the output of the sixth BN batch normalization layer is connected to the input of the sixth RELU activation layer; the output of the sixth RELU activation layer is used as the input of the seventh three-dimensional convolutional layer, and the output of the seventh three-dimensional convolutional layer is connected to the input of the seventh BN batch normalization layer, and the output of the seventh BN batch normalization layer is connected to the input of the seventh RELU activation layer; the output of the seventh RELU activation layer is used as the input of the eighth three-dimensional convolutional layer, and the output of the eighth three-dimensional convolutional layer is connected to the input of the eighth BN batch normalization layer, and the output of the eighth BN batch normalization layer is connected to the input of the eighth RELU activation layer; the output of the eighth RELU activation layer is used as the input of the ninth three-dimensional convolutional layer, The output of the ninth three-dimensional convolution layer is connected to the input of the ninth BN batch normalization layer, and the output of the ninth BN batch normalization layer is connected to the input of the ninth RELU activation layer; the output of the ninth RELU activation layer is used as the input of the tenth three-dimensional convolution layer, and the output of the tenth three-dimensional convolution layer is connected to the input of the tenth BN batch normalization layer, and the output of the tenth BN batch normalization layer is connected to the input of the tenth RELU activation layer; the output of the tenth RELU activation layer is used as the input of the eleventh three-dimensional convolution layer, and the output of the eleventh three-dimensional convolution layer is connected to the input of the eleventh BN batch normalization layer, and the output of the eleventh BN batch normalization layer is connected to the input of the eleventh RELU activation layer; the output of the eleventh RELU activation layer is used as the input of the first reshape layer and the second reshape layer respectively, and the outputs of the first reshape layer and the second reshape layer are both used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
[0028] The output of the first Concate layer is used as the input of the twelfth three-dimensional convolution layer, and the output result of the twelfth three-dimensional convolution layer is classified through a classifier.
[0029] The convolution kernel size of the first three-dimensional convolution layer is 10×10×1, and the number of convolution kernels is 64; the convolution kernel size of the second three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 64; the convolution kernel size of the third three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 128; the convolution kernel size of the fourth three-dimensional convolution layer is 5×5×1, and the number of convolution kernels is 256; the convolution kernel size of the fifth three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 256.
[0030] The convolution kernel size of the sixth three-dimensional convolution layer is 1×1×10, and the number of convolution kernels is 64; the convolution kernel size of the seventh three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the eighth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the ninth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh three-dimensional convolution layer is 1×1×3, and the number of convolution kernels is 128; the first and second reshape layers reshape the output of the eleventh RELU activation layer into a size of 1×1×36, and the number is 128.
[0031] It can be seen from the above technical solution that the beneficial effects of the present invention are as follows: First, the present invention constructs a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion, which consists of two branches: in the spectral full-channel interaction network branch SCFI-branch, the new data after the spatial-spectral dimension conversion of the original data can be used to simultaneously extract non-continuous spectral features with a long distance; in the spatial-continuous spectral feature extraction network branch SAS-branch, three-dimensional convolution is used to simultaneously extract spatial and continuous spectral features, and in addition, the features obtained from the two branches are fused and classified. Second, the present invention realizes full-pixel classification, improves the utilization rate and classification efficiency of scarce hyperspectral data, and the branch structure of the network is relatively simple, with good classification performance and good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of the method of the present invention;
[0033] Figure 2 This is a schematic diagram of the test information leakage problem that occurs based on the classification network;
[0034] Figure 3 This is a diagram of the partitioning method based on equally spaced data;
[0035] Figure 4 Schematic diagram of the spatial-spectral dimension conversion algorithm;
[0036] Figure 5 It is the structure diagram of the spectral full channel interaction network branch SFCI-branch;
[0037] Figure 6 The overall structure diagram of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion. DETAILED DESCRIPTION
[0038] like Figure 1 As shown, a hyperspectral classification method based on spatial-spectral dimension conversion includes the following steps in sequence:
[0039] (1) The original hyperspectral dataset and the corresponding label vector dataset are divided into equally spaced and non-overlapping sets to obtain the training set X1, validation set X2, test set X3, as well as the training set label Y1, validation set label Y2, and test set label Y3;
[0040] (2) Perform spatial-spectral dimension conversion on the divided training set X1, validation set X2, and test set X3 to obtain the converted new training set X1_1, new validation set X2_1, and new test set X3_1; at the same time, perform dimension conversion on the training set label Y1, validation set label Y2, and test set label Y3 to obtain the new training set label Y1_1, new validation set label Y2_1, and new test set label Y3_1;
[0041] (3) Constructing a dual-fully convolutional network for Hyperspectral Image Classification via the Spatial-Spectral dimension conversion (DSSFN).
[0042] (4) Input the training set X1, the validation set X2, the new training set label Y1_1, and the new validation set label Y2_1 into the SAS-branch (Spatial and Adjacent-Spectral feature extraction branch) of the hyperspectral classification network DSSFN; input the new training set X1_1, the new validation set X2_1, the new training set label Y1_1, and the new validation set label Y2_1 into the SFCI-branch (Spectral full-channel interactive feature extraction branch) of the hyperspectral classification network DSSFN, and use the Adam algorithm for iterative optimization to obtain the optimal network DSSFN with the best classification performance;
[0043] (5) Input the test set X3, the new test set X3_1 and the corresponding new test set label Y3_1 into the optimal network DSSFN to predict the classification results.
[0044] The step (1) specifically includes the following steps:
[0045] (1a) First, the entire original hyperspectral dataset is divided into W*H*C hyperspectral image blocks, where W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands; the original hyperspectral dataset uses the Salinas valley dataset;
[0046] (1b) Discard the hyperspectral image patches without any annotation information and use the hyperspectral image patches with only one pixel annotation as the test set, recorded as test set-1;
[0047] (1c) Sort the hyperspectral image blocks with more than one type of pixel label one by one according to the order of division and divide them into T groups. Randomly select one of the groups as the training set X1, and randomly select another group from the remaining data as the validation set X2. The remaining T-2 groups are used as the test set-2. Finally, the test set-1 and the test set-2 are combined into the test set X3. The labels at the corresponding positions of the training set X1, the validation set X2, and the test set X3 are used as the training set label Y1, the validation set label Y2, and the test set label Y3, respectively.
[0048] The proportion of the data set divided in step (1) is shown in Table 1:
[0049] Table 1 Salinasvalley dataset is divided into test set, validation set, and training set ratio
[0050]
[0051] The evaluation indicators of various methods for the classification of the Salinas valley dataset are as follows: Figure 2 As shown:
[0052] Table 2 Evaluation indicators of various methods for classification of the Salinasvalley dataset
[0053]
[0054] The step (2) specifically includes the following steps:
[0055] (2a) Extract the one-dimensional vector C*1 of all pixels in the training set X1, validation set X2, and test set X3 in the spectral dimension, where C represents the number of spectral bands;
[0056] (2b) Transpose the one-dimensional vector C*1 to get 1*C, and multiply the two vectors to get the spectral correlation matrix C*C;
[0057] (2c) The training set X1, validation set X2 and test set X3 data of size W*H*C are converted into data sets X1_1, X2_1 and X3_1 of size C*C*(W*H); W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands; the hyperspectral image blocks are obtained by dividing the entire original hyperspectral dataset;
[0058] (2d) The labels of size W*H*1 are stretched and dimensionally transformed to be converted into new training set labels Y1_1, new validation set labels Y2_1, and new test set labels Y3_1 of size 1*1*(W*H).
[0059] The step (3) specifically includes the following steps:
[0060] (3a) constructing a spectral full-channel interaction network branch SFCI-branch, wherein the spectral full-channel interaction network branch SFCI-branch includes a first three-dimensional convolutional layer, a first BN batch normalization layer, and a first RELU activation layer; a second three-dimensional convolutional layer, a second BN batch normalization layer, and a second RELU activation layer; a third three-dimensional convolutional layer, a third BN batch normalization layer, and a third RELU activation layer; a fourth three-dimensional convolutional layer, a fourth BN batch normalization layer, and a fourth RELU activation layer; a fifth three-dimensional convolutional layer, a fifth BN batch normalization layer, and a fifth RELU activation layer;
[0061] (3b) constructing a spatial domain-continuous spectrum feature extraction network branch SAS-branch, wherein the spatial domain-continuous spectrum feature extraction network branch SAS-branch includes a sixth three-dimensional convolution layer, a sixth BN batch normalization layer, and a sixth RELU activation layer; a seventh three-dimensional convolution layer, a seventh BN batch normalization layer, and a seventh RELU activation layer; an eighth three-dimensional convolution layer, an eighth BN batch normalization layer, and an eighth RELU activation layer; a ninth three-dimensional convolution layer, a ninth BN batch normalization layer, and a ninth RELU activation layer; a tenth three-dimensional convolution layer, a tenth BN batch normalization layer, and a tenth RELU activation layer; an eleventh three-dimensional convolution layer, an eleventh BN batch normalization layer, and an eleventh RELU activation layer; a first reshape layer and a second reshape layer;
[0062] (3c) The spectral full-channel interaction network branch SFCI-branch and the spatial-continuous spectral feature extraction network branch SAS-branch are integrated to construct a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion. The hyperspectral classification network DSSFN based on spatial-spectral dimension conversion includes a first Concate layer, a twelfth three-dimensional convolution layer and a classifier.
[0063] The input of the first three-dimensional convolutional layer is the new training set X1_1 and the new validation set X2_1. The output of the first three-dimensional convolutional layer is connected to the input of the first BN batch normalization layer, and the output of the first BN batch normalization layer is connected to the input of the first RELU activation layer; the output of the first RELU activation layer is used as the input of the second three-dimensional convolutional layer, and the output of the second three-dimensional convolutional layer is connected to the input of the second BN batch normalization layer, and the output of the second BN batch normalization layer is connected to the input of the second RELU activation layer; the output of the second RELU activation layer is used as the input of the third three-dimensional convolutional layer, and the output of the third three-dimensional convolutional layer is connected to the input of the third BN batch normalization layer, and the output of the third BN batch normalization layer is connected to the input of the third BN batch normalization layer. The output of the third RELU activation layer is used as the input of the fourth three-dimensional convolutional layer, the output of the fourth three-dimensional convolutional layer is connected to the input of the fourth BN batch normalization layer, and the output of the fourth BN batch normalization layer is connected to the input of the fourth RELU activation layer; the output of the fourth RELU activation layer is used as the input of the fifth three-dimensional convolutional layer, the output of the fifth three-dimensional convolutional layer is connected to the input of the fifth BN batch normalization layer, the output of the fifth BN batch normalization layer is connected to the input of the fifth RELU activation layer, and the output of the fifth RELU activation layer is used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
[0064] The input of the sixth three-dimensional convolutional layer is the training set X1, the validation set X2, and the test set X3. The output of the sixth three-dimensional convolutional layer is connected to the input of the sixth BN batch normalization layer, and the output of the sixth BN batch normalization layer is connected to the input of the sixth RELU activation layer; the output of the sixth RELU activation layer is used as the input of the seventh three-dimensional convolutional layer, and the output of the seventh three-dimensional convolutional layer is connected to the input of the seventh BN batch normalization layer, and the output of the seventh BN batch normalization layer is connected to the input of the seventh RELU activation layer; the output of the seventh RELU activation layer is used as the input of the eighth three-dimensional convolutional layer, and the output of the eighth three-dimensional convolutional layer is connected to the input of the eighth BN batch normalization layer, and the output of the eighth BN batch normalization layer is connected to the input of the eighth RELU activation layer; the output of the eighth RELU activation layer is used as the input of the ninth three-dimensional convolutional layer, The output of the ninth three-dimensional convolution layer is connected to the input of the ninth BN batch normalization layer, and the output of the ninth BN batch normalization layer is connected to the input of the ninth RELU activation layer; the output of the ninth RELU activation layer is used as the input of the tenth three-dimensional convolution layer, and the output of the tenth three-dimensional convolution layer is connected to the input of the tenth BN batch normalization layer, and the output of the tenth BN batch normalization layer is connected to the input of the tenth RELU activation layer; the output of the tenth RELU activation layer is used as the input of the eleventh three-dimensional convolution layer, and the output of the eleventh three-dimensional convolution layer is connected to the input of the eleventh BN batch normalization layer, and the output of the eleventh BN batch normalization layer is connected to the input of the eleventh RELU activation layer; the output of the eleventh RELU activation layer is used as the input of the first reshape layer and the second reshape layer respectively, and the outputs of the first reshape layer and the second reshape layer are both used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
[0065] The output of the first Concate layer is used as the input of the twelfth three-dimensional convolution layer, and the output result of the twelfth three-dimensional convolution layer is classified through a classifier.
[0066] The convolution kernel size of the first three-dimensional convolution layer is 10×10×1, and the number of convolution kernels is 64; the convolution kernel size of the second three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 64; the convolution kernel size of the third three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 128; the convolution kernel size of the fourth three-dimensional convolution layer is 5×5×1, and the number of convolution kernels is 256; the convolution kernel size of the fifth three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 256.
[0067] The convolution kernel size of the sixth three-dimensional convolution layer is 1×1×10, and the number of convolution kernels is 64; the convolution kernel size of the seventh three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the eighth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the ninth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh three-dimensional convolution layer is 1×1×3, and the number of convolution kernels is 128; the first and second reshape layers reshape the output of the eleventh RELU activation layer into a size of 1×1×36, and the number is 128.
[0068] like Figure 2 As shown in the figure, assuming that Patch1 in the image is selected as the training set, according to the previous division method, Patch2 and Patch3 that overlap with Patch1 may be selected as the test set. The overlap between the two will lead to the leakage of training-test information, and the trained samples will be sent to the network again as the test set.
[0069] like Figure 3 As shown in (b) and (c) in the figure, the entire original hyperspectral dataset is first divided into data blocks of the same size, and the data blocks without label information are removed. The data blocks containing single label information are used as test set 1, as shown in Figure 3 As shown in (d) and (e), the hyperspectral image blocks with more than one type of pixel label are sorted into 9 groups according to the order of division, and one group is randomly selected as the validation set and test set 2, and test set 1 and test set 2 are combined as the test set. Repeat the experiment by selecting different groups as training sets, test sets, and validation sets to ensure the accuracy of the experimental results.
[0070] like Figure 4 As shown in the figure, in order to simultaneously extract information on the spectral dimension that is farther away, each pixel in the patch block is converted from the spatial domain to the spectral dimension. First, the vector X of the pixel in the spectral dimension is extracted, with a size of C*1, and then it is transposed to obtain a vector X' with a size of 1*C. The two vectors are multiplied to obtain the spectral autocorrelation matrix. After the above product processing, the data with a size of W*H*C is converted into new data of C*C*(W*H).
[0071] like Figure 5 As shown in Figure 1, a fully convolutional network is designed to extract information on distant spectral dimensions. The network consists of multiple three-dimensional convolutional layers, BN batch normalization layers, and RELU activation layer modules.
[0072] like Figure 6As shown, the SCFI-branch and the SAS-branch are merged to obtain the overall network DSSFN.
[0073] In summary, the present invention constructs a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion, which consists of two branches: in the spectral full-channel interaction network branch SCFI-branch, the new data after the spatial-spectral dimension conversion of the original data can be used to simultaneously extract non-continuous spectral features with a long distance; in the spatial-continuous spectral feature extraction network branch SAS-branch, three-dimensional convolution is used to simultaneously extract spatial and continuous spectral features, and in addition, the features obtained by the two branches are fused and classified. The present invention realizes full-pixel classification, improves the utilization rate and classification efficiency of scarce hyperspectral data, and the branch structure of the network is relatively simple, with good classification performance and good generalization ability.
[0074] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as falling within the protection scope of the present invention.
Claims
1. A hyperspectral classification method based on spatial-spectral dimension conversion, characterized by: The method comprises the following steps in order: (1) The original hyperspectral dataset and the corresponding label vector dataset are divided into equally spaced and non-overlapping sets to obtain the training set X1, validation set X2, test set X3, as well as the training set label Y1, validation set label Y2, and test set label Y3; (2) Perform spatial-spectral dimension conversion on the divided training set X1, validation set X2, and test set X3 to obtain the converted new training set X1_1, new validation set X2_1, and new test set X3_1; at the same time, perform dimension conversion on the training set label Y1, validation set label Y2, and test set label Y3 to obtain the new training set label Y1_1, new validation set label Y2_1, and new test set label Y3_1; (3) Construct a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion; (4) Input the training set X1, the validation set X2, and the new training set label Y1_1 and the new validation set label Y2_1 into the SAS-branch of the hyperspectral classification network DSSFN; input the new training set X1_1, the new validation set X2_1, and the new training set label Y1_1 and the new validation set label Y2_1 into the SFCI-branch of the hyperspectral classification network DSSFN, and use the Adam algorithm for iterative optimization to obtain the optimal network DSSFN with the best classification performance; (5) Input the test set X3, the new test set X3_1 and the corresponding new test set label Y3_1 into the optimal network DSSFN to predict the classification results; The step (2) specifically includes the following steps: (2a) Extract the one-dimensional vector C*1 of all pixels in the training set X1, validation set X2, and test set X3 in the spectral dimension, where C represents the number of spectral bands; (2b) Transpose the one-dimensional vector C*1 to get 1*C, and multiply the two vectors to get the spectral correlation matrix C*C; (2c) The training set X1, validation set X2 and test set X3 data of size W*H*C are converted into data sets X1_1, X2_1 and X3_1 of size C*C*(W*H); W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands; the hyperspectral image blocks are obtained by dividing the entire original hyperspectral dataset; (2d) The labels of size W*H*1 are stretched and dimensionally transformed to be converted into new training set labels Y1_1, new validation set labels Y2_1, and new test set labels Y3_1 of size 1*1*(W*H).
2. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 1 is characterized in that: The step (1) specifically includes the following steps: (1a) First, the entire original hyperspectral dataset is divided into W*H*C hyperspectral image blocks, where W and H represent the width and height of the divided hyperspectral image blocks, respectively, and C is the number of spectral bands; (1b) Discard the hyperspectral image patches without any annotation information and use the hyperspectral image patches with only one pixel annotation as the test set, recorded as test set-1; (1c) Sort the hyperspectral image blocks with more than one type of pixel label one by one according to the order of division and divide them into T groups. Randomly select one of the groups as the training set X1, and randomly select another group from the remaining data as the validation set X2. The remaining T-2 groups are used as the test set-2. Finally, the test set-1 and the test set-2 are combined into the test set X3. The labels at the corresponding positions of the training set X1, the validation set X2, and the test set X3 are used as the training set label Y1, the validation set label Y2, and the test set label Y3, respectively.
3. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 1 is characterized in that: The step (3) specifically includes the following steps: (3a) constructing a spectral full-channel interaction network branch SFCI-branch, wherein the spectral full-channel interaction network branch SFCI-branch includes a first three-dimensional convolutional layer, a first BN batch normalization layer, and a first RELU activation layer; a second three-dimensional convolutional layer, a second BN batch normalization layer, and a second RELU activation layer; a third three-dimensional convolutional layer, a third BN batch normalization layer, and a third RELU activation layer; a fourth three-dimensional convolutional layer, a fourth BN batch normalization layer, and a fourth RELU activation layer; a fifth three-dimensional convolutional layer, a fifth BN batch normalization layer, and a fifth RELU activation layer; (3b) constructing a spatial domain-continuous spectrum feature extraction network branch SAS-branch, wherein the spatial domain-continuous spectrum feature extraction network branch SAS-branch includes a sixth three-dimensional convolution layer, a sixth BN batch normalization layer, and a sixth RELU activation layer; a seventh three-dimensional convolution layer, a seventh BN batch normalization layer, and a seventh RELU activation layer; an eighth three-dimensional convolution layer, an eighth BN batch normalization layer, and an eighth RELU activation layer; a ninth three-dimensional convolution layer, a ninth BN batch normalization layer, and a ninth RELU activation layer; a tenth three-dimensional convolution layer, a tenth BN batch normalization layer, and a tenth RELU activation layer; an eleventh three-dimensional convolution layer, an eleventh BN batch normalization layer, and an eleventh RELU activation layer; a first reshape layer and a second reshape layer; (3c) The spectral full-channel interaction network branch SFCI-branch and the spatial-continuous spectral feature extraction network branch SAS-branch are integrated to construct a hyperspectral classification network DSSFN based on spatial-spectral dimension conversion. The hyperspectral classification network DSSFN based on spatial-spectral dimension conversion includes a first Concate layer, a twelfth three-dimensional convolution layer and a classifier.
4. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 3 is characterized in that: The input of the first three-dimensional convolutional layer is the new training set X1_1 and the new verification set X2_1, the output of the first three-dimensional convolutional layer is connected to the input of the first BN batch normalization layer, and the output of the first BN batch normalization layer is connected to the input of the first RELU activation layer; The output of the first RELU activation layer is used as the input of the second 3D convolutional layer, the output of the second 3D convolutional layer is connected to the input of the second BN batch normalization layer, and the output of the second BN batch normalization layer is connected to the input of the second RELU activation layer; The output of the second RELU activation layer is used as the input of the third three-dimensional convolutional layer, the output of the third three-dimensional convolutional layer is connected to the input of the third BN batch normalization layer, and the output of the third BN batch normalization layer is connected to the input of the third RELU activation layer; the output of the third RELU activation layer is used as the input of the fourth three-dimensional convolutional layer, the output of the fourth three-dimensional convolutional layer is connected to the input of the fourth BN batch normalization layer, and the output of the fourth BN batch normalization layer is connected to the input of the fourth RELU activation layer; the output of the fourth RELU activation layer is used as the input of the fifth three-dimensional convolutional layer, the output of the fifth three-dimensional convolutional layer is connected to the input of the fifth BN batch normalization layer, the output of the fifth BN batch normalization layer is connected to the input of the fifth RELU activation layer, and the output of the fifth RELU activation layer is used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
5. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 3 is characterized in that: The input of the sixth three-dimensional convolutional layer is the training set X1, the validation set X2, and the test set X3; the output of the sixth three-dimensional convolutional layer is connected to the input of the sixth BN batch normalization layer, and the output of the sixth BN batch normalization layer is connected to the input of the sixth RELU activation layer; the output of the sixth RELU activation layer is used as the input of the seventh three-dimensional convolutional layer, the output of the seventh three-dimensional convolutional layer is connected to the input of the seventh BN batch normalization layer, and the output of the seventh BN batch normalization layer is connected to the input of the seventh RELU activation layer; The output of the seventh RELU activation layer is used as the input of the eighth three-dimensional convolutional layer, the output of the eighth three-dimensional convolutional layer is connected to the input of the eighth BN batch normalization layer, and the output of the eighth BN batch normalization layer is connected to the input of the eighth RELU activation layer; the output of the eighth RELU activation layer is used as the input of the ninth three-dimensional convolutional layer, the output of the ninth three-dimensional convolutional layer is connected to the input of the ninth BN batch normalization layer, and the output of the ninth BN batch normalization layer is connected to the input of the ninth RELU activation layer; The output of the ninth RELU activation layer is used as the input of the tenth 3D convolutional layer, the output of the tenth 3D convolutional layer is connected to the input of the tenth BN batch normalization layer, and the output of the tenth BN batch normalization layer is connected to the input of the tenth RELU activation layer; The output of the tenth RELU activation layer is used as the input of the eleventh three-dimensional convolution layer, the output of the eleventh three-dimensional convolution layer is connected to the input of the eleventh BN batch normalization layer, and the output of the eleventh BN batch normalization layer is connected to the input of the eleventh RELU activation layer; the output of the eleventh RELU activation layer is used as the input of the first reshape layer and the second reshape layer respectively, and the outputs of the first reshape layer and the second reshape layer are both used as the input of the first Concate layer of the hyperspectral classification network DSSFN based on spatial-spectral dimension conversion.
6. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 3 is characterized in that: The output of the first Concate layer is used as the input of the twelfth three-dimensional convolution layer, and the output result of the twelfth three-dimensional convolution layer is classified through a classifier.
7. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 3 or 4, characterized in that: The convolution kernel size of the first three-dimensional convolution layer is 10×10×1, and the number of convolution kernels is 64; the convolution kernel size of the second three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 64; the convolution kernel size of the third three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 128; the convolution kernel size of the fourth three-dimensional convolution layer is 5×5×1, and the number of convolution kernels is 256; the convolution kernel size of the fifth three-dimensional convolution layer is 3×3×1, and the number of convolution kernels is 256.
8. The hyperspectral classification method based on spatial-spectral dimension conversion according to claim 3 or 5, characterized in that: The convolution kernel size of the sixth three-dimensional convolution layer is 1×1×10, and the number of convolution kernels is 64; the convolution kernel size of the seventh three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 64; the convolution kernel size of the eighth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 128; the convolution kernel size of the ninth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the tenth three-dimensional convolution layer is 3×3×3, and the number of convolution kernels is 256; the convolution kernel size of the eleventh three-dimensional convolution layer is 1×1×3, and the number of convolution kernels is 128; the first and second reshape layers reshape the output of the eleventh RELU activation layer into a size of 1×1×36, and the number is 128.