A hyperspectral image classification method based on refined spatial-spectral joint feature extraction

Abstract

The present invention relates to a hyperspectral image classification method based on refined spatial-spectral joint feature extraction, which belongs to the field of remote sensing and image processing, and solves the problems of insufficient samples, imprecise spatial information extraction and low classification accuracy of existing methods. It includes obtaining the original hyperspectral image and preprocessing it to obtain training pixels and test pixels of known categories, and constructing a training sample set based on the training pixels; constructing a hyperspectral image classification neural network with refined spatial-spectral joint feature extraction, and training it using the training sample set to obtain a trained hyperspectral image classification neural network; constructing a test sample set based on the test pixels, inputting the test sample set into the trained hyperspectral image classification neural network, and applying a weighted voting method to determine the classification result of each test pixel in the test sample set; if the accuracy of the classification result of the test pixel and the known category is greater than a preset threshold, the trained neural network is used for classification prediction. Accurate classification of hyperspectral images is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing and image processing, and in particular to a hyperspectral image (HSI, Hyperspectral Image) classification method based on refined spatial-spectral joint feature extraction. Background Art

[0002] Hyperspectral image classification technology emerged in the late twentieth century and developed from traditional multispectral image classification technology. Hyperspectral images are obtained by imaging objects in hundreds or thousands of continuous and narrow wavelength bands through imaging spectrometers. Hyperspectral images organically integrate the two-dimensional spatial information reflecting the geometric and texture characteristics of objects with the one-dimensional spectral information reflecting the reflection, radiation or scattering characteristics of objects, and have the characteristics of "image and spectrum in one". Hyperspectral image classification refers to assigning a category label to each pixel in the image with the help of the rich spectral or spatial information contained in the hyperspectral image. Hyperspectral image classification technology is widely used in environmental monitoring, target recognition and other fields, and has high practical value.

[0003] In the past two decades, domestic and foreign scholars have conducted extensive and in-depth research on hyperspectral image classification technology. After the emergence of deep learning technology, due to its excellent feature extraction ability, scholars have studied a large number of hyperspectral image classification technologies based on deep learning. Among them, hyperspectral image classification technology based on CNN (Convolutional neural network) is one of the most representative methods.

[0004] The existing hyperspectral image classification methods based on CNN face the following problems: insufficient training samples and limited deep networks. Since the annotation of hyperspectral image training samples requires a large cost, the existing hyperspectral image classification methods based on CNN face the problem of insufficient training samples, which makes it impossible to design the network deeper, thus affecting the feature extraction effect; unable to extract spatial information in a refined manner. Hyperspectral images have rich spatial information around their pixels, but the existing hyperspectral image classification based on CNN cannot extract spatial information around specific pixels in a refined manner; lack of classification performance improvement measures in the test phase. In the test phase, the existing hyperspectral classification based on CNN lacks effective measures to improve the accuracy of classification performance. The above three defects limit the classification performance of hyperspectral images. Summary of the invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a hyperspectral image classification method based on refined spatial-spectral joint feature extraction, so as to solve the technical problems of insufficient training samples, imprecise spatial information extraction and low classification accuracy in the test phase of the hyperspectral image classification method of the existing method.

[0006] The present invention provides a hyperspectral image classification method based on refined spatial-spectral joint feature extraction, comprising the following steps:

[0007] Step S1, obtaining an original hyperspectral image and performing preprocessing to obtain training pixels and test pixels of known categories, and constructing a training sample set based on the training pixels;

[0008] Step S2, constructing a hyperspectral image classification neural network with refined spatial-spectral joint feature extraction, and using the training sample set for training to obtain a trained hyperspectral image classification neural network;

[0009] Step S3, constructing a test sample set based on the test pixel, inputting the test sample set into the trained hyperspectral image classification neural network, and applying the weighted voting method to determine the classification result of each test pixel in the test sample set; if the accuracy of the classification result of the test pixel and the known category is greater than a preset threshold, the trained hyperspectral image neural network is used for classification prediction.

[0010] Furthermore, the step S1 comprises:

[0011] Performing normalization processing on the hyperspectral image;

[0012] The normalized hyperspectral image has a total of K categories, and all categories include L pixels in total. The L pixels are divided into M training pixels and N test pixels according to a preset ratio; the training pixels and the test pixels both include K category pixels;

[0013] Taking the training pixel as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set.

[0014] Furthermore, taking the training pixel as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set, including:

[0015] For any two training pixels X among the M training pixels i and X j Composed pixel pair D ij =[x i x j ], where x i 、x j They are primary and auxiliary pixels respectively;

[0016] Construct the primary pixel x i The spatial block with a size of 9×9 is centered The spatial block includes 81 pixels, with the main pixel X i As the center, Divided into a 5×5 upper left corner square Top right square Lower left square Lower right square Main pixel X i Centered on Divide the space into a central space block with a size of 5×5

[0017] The multi-region space blocks are Then the pixel pair-multi-region space block combination is C ij =[D ij P i ];

[0018] C ij Tags ij The value selection rules are as follows:

[0019]

[0020] Among them, k is the training pixel x i and x j Known categories of

[0021] Based on each training sample C ij and the corresponding label y ij , and obtain the training sample set, expressed as Among them, X new The number of categories is K+1, including the added 0th category.

[0022] Further, the hyperspectral image classification neural network includes a spectral branch, a spatial branch, a first fully connected layer and a second fully connected layer;

[0023] The spectral branch is used to adjust the pixel pair D ij Perform refined spectral feature extraction, including input layer, C1, C2, P1, C3, C4, P2, C5, C6, P3 and C7 in sequence, among which C1~C7 are convolution layers and P1~P3 are pooling layers;

[0024] The spatial branch includes six spatial branches for D ij The corresponding 6 spatial blocks are used for refined spatial feature extraction, and each branch includes 3 convolutional layers.

[0025] Furthermore, corresponding The three convolutional layers of the spatial branch and The sizes are 5×5×q, 3×3×128 and 3×3×256 respectively; where q is the spectral band;

[0026] The remaining three convolutional layers of the five spatial branches and The convolutional layer sizes are 3×3×q, 2×2×128, and 2×2×256, respectively; where x∈{b,c,d,e,f}.

[0027] Furthermore, the hyperspectral image classification neural network training is performed using the sample training set as follows:

[0028] Define hyperparameters; set D ij Input the spectral branch to obtain the spectral feature vector, and convert P i middle The six spatial branches of the spatial branch are input in sequence, and are respectively subjected to three layers of convolution in sequence to obtain corresponding vectors and sequentially concatenated to obtain spatial feature vectors;

[0029] Connecting the spectral feature vector with the spatial feature vector to obtain a feature vector, using the feature vector as the input of the first fully connected layer, and outputting a vector with a dimension of 128;

[0030] The vector with a dimension of 128 is input into the second fully connected layer to obtain an output vector with a dimension of K+1, which is the training sample C. ij The prediction category of

[0031] Using the cross entropy function as the loss function for forward propagation, and then using the Adam optimizer for back propagation to optimize all parameters of the hyperspectral image classification neural network, wherein all parameters include the weights and biases of each layer;

[0032] If the number of training iterations reaches the number of training iterations Epoch, the training is terminated and a trained hyperspectral image classification neural network is obtained.

[0033] Further, constructing a test sample set based on the test pixels includes:

[0034] For any test pixel z among the N test pixels p , with z p The spatial dimension is 9×9 space block Q centered at p , where p = 1, 2, ..., N;

[0035] With z p The main pixel is the spatial block Q p In addition to z p The remaining test pixels are auxiliary pixels, and 80 pixel pairs are obtained, and then 80 pixel pair-multi-region spatial block combinations are obtained. Each pixel pair-multi-region spatial block combination is a test sample; the test samples corresponding to all test pixels constitute a test sample set.

[0036] Further, the test sample set is input into the trained hyperspectral image classification neural network, and a K+1 dimensional vector is output; the value of each position of the K+1 dimensional vector is the classification category probability;

[0037] The first element of the K+1-dimensional vector is removed from the 0th category, and the maximum element value of the remaining K-dimensional vector is used to obtain the classification result of the test pixel using the weighted voting method, including:

[0038] After inputting the 80 pixel pairs-multi-region space block combinations in the test sample into the trained hyperspectral image classification neural network, 80 predicted category values ​​are obtained;

[0039] In each pixel pair-multi-region spatial block combination, the weights of pixels that are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the main pixel are set to 1.0, 0.9, 0.8, and 0.7, respectively;

[0040] The weighted voting value v of the kth class k ,as follows:

[0041]

[0042] in, The predicted category values ​​of the kth category are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the main pixel, respectively. The category with the largest weighted voting value is taken as the test pixel z p The classification results.

[0043] Furthermore, cross entropy is used as the loss function for forward propagation as follows:

[0044]

[0045] Among them, y is the true label value, is the predicted label value for the network output, batch_size is the number of training samples for batch training, y jk is the value of the kth position of the true label of the jth sample in the training batch, Predict the value of the label at the kth position for the jth example in the training batch.

[0046] Furthermore, for an original hyperspectral image with h rows, w columns, and q spectral bands Normalization is performed as follows:

[0047]

[0048] Among them, H is the normalized hyperspectral image, and m and n are the maximum and minimum element values ​​of H0, respectively.

[0049] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0050] 1. The problem of insufficient training samples is alleviated by constructing a pixel pair-multi-region spatial block method, which significantly increases the number of training samples and overcomes the problem of insufficient training samples for hyperspectral image classification. This allows the network to be designed deeper and achieve better feature extraction and classification effects.

[0051] 2. Through the refined extraction of spatial-spectral joint features, the spectral and spatial feature information of the image can be more accurately extracted from both the spatial and spectral dimensions, thus improving the classification accuracy of hyperspectral images;

[0052] 3. Further improve the hyperspectral classification performance in the test phase by applying the weighted voting method. By cleverly applying the hyperspectral image property that neighboring pixels have a greater probability of belonging to the same class and pixels closer to the central pixel have a greater probability of belonging to the same class as the central pixel, a weighted voting algorithm is constructed in the test phase to improve the classification performance of the neural network;

[0053] 4. By refining the spatial information around the pixels, the local characteristics of the objects can be better captured.

[0054] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0056] Figure 1 This is a flow chart of a hyperspectral image classification method based on refined spatial-spectral joint feature extraction;

[0057] Figure 2 The architecture diagram of the image classification algorithm for refined spatial-spectral joint feature extraction;

[0058] Figure 3 Schematic diagram of the network for refining spatial-spectral joint feature extraction;

[0059] Figure 4 This is a schematic diagram of the space block in the upper right corner;

[0060] Figure 5 Schematic diagram of weighted voting strategy. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0062] The present invention proposes a hyperspectral image classification method based on refined spatial-spectral joint feature extraction. The method mainly refines the extraction of spatial and spectral features of hyperspectral images by designing a refined spatial-spectral joint feature extraction network, and uses a weighted voting algorithm in the testing phase, ultimately achieving better hyperspectral image classification results.

[0063] A specific embodiment of the present invention discloses a hyperspectral image classification method based on refined spatial-spectral joint feature extraction, such as Figure 1 As shown, the following steps are included:

[0064] Step S1, obtaining an original hyperspectral image and preprocessing it to obtain training pixels and test pixels of known categories, and constructing a training sample set based on the training pixels; wherein the original hyperspectral image refers to a hyperspectral image of a target recognition area collected from a satellite, airborne, etc. by a hyperspectral sensor.

[0065] Step S2, constructing a hyperspectral image classification neural network with refined spatial-spectral joint feature extraction, and using the training sample set for training to obtain a trained hyperspectral image classification neural network;

[0066] Step S3, constructing a test sample set based on the test pixel, inputting the test sample set into the trained hyperspectral image classification neural network, and applying the weighted voting method to determine the classification result of each test pixel in the test sample set; if the accuracy of the classification result of the test pixel and the known category is greater than a preset threshold, the trained hyperspectral image neural network is used for classification prediction.

[0067] The present invention alleviates the problem of insufficient training samples by constructing a pixel pair-multi-region spatial block method, and combines refined spectral and spatial feature extraction and the use of a weighted voting method in the test phase, thereby significantly improving the accuracy and efficiency of hyperspectral image classification and solving the technical problems of insufficient samples, imprecise spatial information extraction, and low classification accuracy.

[0068] In the step S1, the original hyperspectral image is normalized; the normalized hyperspectral image has K categories, all categories include L pixels, and the L pixels are divided into M training pixels and N test pixels according to a preset ratio; the training pixels and the test pixels both include K category pixels; with the training pixels as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set.

[0069] Specifically, step S1 includes steps S11-S12.

[0070] Step S11: Acquire the original hyperspectral image and perform preprocessing to obtain training pixels and test pixels of known categories.

[0071] (1) Obtaining original hyperspectral images. When studying hyperspectral image classification algorithms, the industry’s publicly available hyperspectral image datasets are generally used. Hyperspectral sensors (i.e., hyperspectral imagers) collect hyperspectral images of target identification areas from satellites, airborne, and other sources.

[0072] Obtain hyperspectral images. The spectral resolution is 10 -2λ Spectral images within the order of magnitude range are called hyperspectral images. Hyperspectral images are remote sensing images with extremely high spectral resolution that can capture the reflection or emission characteristics of objects in continuous narrow wavelength bands, which can be used to accurately identify and classify substances. Hyperspectral sensors are used to image the target area of ​​the object to be detected in tens to hundreds of continuous and subdivided spectral bands of the electromagnetic spectrum.

[0073] (2) Normalize the hyperspectral image.

[0074] For an original hyperspectral image with h rows, w columns, and q spectral bands Normalization is performed as shown in formula (1):

[0075]

[0076] Among them, H is the normalized hyperspectral image, and m and n are the maximum and minimum element values ​​of H0, respectively.

[0077] The h row is the height of the hyperspectral image, that is, the number of rows of the image, and the w column is the width of the image, that is, the number of columns of the image.

[0078] m is the maximum observed value of all pixels in the original hyperspectral image in all bands; n is the minimum observed value of all pixels in the original hyperspectral image in all bands.

[0079] If m=n, that is, all element values ​​are the same, the normalized hyperspectral image will be an all-zero image; otherwise, m≠n, the normalized hyperspectral image will be rescaled according to its relative position to the minimum and maximum values.

[0080] The form of H0 is a three-dimensional tensor. Through normalization, all element values ​​in the normalized hyperspectral image are normalized to between 0 and 1.

[0081] The normalization process prevents certain image features from dominating the training process due to their large values.

[0082] (3) Divide the training pixels and the test pixels. The normalized hyperspectral image has K categories, and K categories have L pixels. In each category, tr% of the pixels are randomly selected as training pixels, and the remaining te% of the pixels are used as test pixels. Among them, tr% = 1-te%.

[0083] The total number of training pixels is shown in formula (2):

[0084] M=L×tr% Formula (2)

[0085] The total number of test pixels is as shown in formula (3):

[0086] N=L×te% Formula (3)

[0087] Exemplarily, the number of training pixels is 10% of the total number of pixels, and the number of test pixels is 90% of the total number of pixels. K-type pixels are evenly distributed in the training pixels and the test pixels.

[0088] Step S12: construct a training sample set based on the training pixels.

[0089] like Figure 2 As shown, the pixel pair-multi-region spatial block algorithm is applied to construct a training sample set.

[0090] Taking the training pixel as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set, including:

[0091] For any two training pixels x among the M training pixels i and x j Composed pixel pair D ij =[x i x j ], where x i 、x j They are primary and auxiliary pixels respectively;

[0092] Construct the primary pixel x i The spatial block with a size of 9×9 is centered The spatial block includes 81 pixels, with the main pixel X i As the center, Divided into a 5×5 upper left corner square Top right square Lower left square Lower right square Main pixel X i Centered on Divide the space into a central space block with a size of 5×5

[0093] The multi-region space blocks are Then the pixel pair-multi-region space block combination is C ij =[D ij P i ];

[0094] C ij Label y ij The value selection rule is shown in formula (4):

[0095]

[0096] Among them, k is the training pixel x i and x j Known categories of

[0097] Based on each training sample C ij and the corresponding label y ij , and obtain the training sample set, expressed as Among them, x new The number of categories is K+1, including the added 0th category.

[0098] For the k-th class of pixels, k∈{1, 2, ...K}, assuming the number of pixels used for training is n k , then the entire training pixel set is as shown in formula (5):

[0099]

[0100] Among them, x i (i=1, 2, ..., M) is the i-th training pixel.

[0101] The total number of training pixels M is as shown in formula (6):

[0102]

[0103] In addition, the training pixels X i The label is y i ∈{1, 2, ..., K}.

[0104] Use X i and X j Two pixels can form a pixel pair. Any two pixels can form a pixel pair. A pixel pair consists of pixels X i and X j Composition, [x i x j ] and [x j x i ] are two different pixel pairs, the pixel on the left is the main pixel, and the pixel on the right is the auxiliary pixel. In the present invention, all pixels can be used as main pixels.

[0105] Any two pixels in the training pixel set X and the multi-region spatial block corresponding to the main pixel constitute a pixel pair-multi-region spatial block combination for training. Then all pixel pairs-multi-region spatial block combinations for training together constitute the training set X. new , as shown in formula (7):

[0106]

[0107] Due to the addition of class 0, the training set X new There are K+1 categories, and the number of training samples corresponding to the training pixels of the kth category (k∈{1, 2, ..., K}) is n k (n k -1), n k is the number of training pixels in the kth class. Except for class 0, the number of training samples in each class is expanded to the number of training pixels (n k -1) times. Since there are more training samples in category 0, in order to balance the training sample set, only the same number of training samples in category 0 are randomly selected, and the number of training samples in category 0 is also n. k (n k -1).

[0108] Step S1 divides the original hyperspectral image into training pixels for neural network model training and test pixels for neural network model performance evaluation, and constructs a pixel pair-multi-region spatial block combination with each training pixel as the center to obtain a rich training sample set, thereby alleviating the technical problem of insufficient training samples;

[0109] Step S2 includes steps S21-S22.

[0110] Step S21, constructing a hyperspectral image classification neural network for refined spatial-spectral joint feature extraction.

[0111] The hyperspectral image classification neural network includes a spectral branch, a spatial branch, a first fully connected layer and a second fully connected layer;

[0112] The spectral branch is used to adjust the pixel pair D ij Perform refined spectral feature extraction, including input layer, C1, C2, P1, C3, C4, P2, C5, C6, P3 and C7 in sequence, among which C1~C7 are convolution layers and P1~P3 are pooling layers;

[0113] The spatial branch includes six spatial branches for D ij The corresponding 6 spatial blocks are used for refined spatial feature extraction, and each branch includes 3 convolutional layers.

[0114] The overall network structure is as follows Figure 3As shown in Figure 1, it includes a spectral branch composed of a refined spectral feature extraction CNN, a spatial branch composed of a refined spatial feature extraction CNN, and the first and second fully connected layers. Due to the long spectral branch, it is difficult to use it together with other network structures. Figure 3 Therefore, the spectral branch structure of the CNN for fine-tuning spectral feature extraction is shown in Figure 3 The display is omitted in the figure, and the detailed information of the complete network structure of the spectral branches is shown in Table 1.

[0115] (1) Construct the spectral branch, which is a CNN network for extracting refined spectral features. By applying the pixel pair-multi-region spatial block algorithm, the number of training samples is expanded, so the neural network can be designed to be deeper and less prone to overfitting. The spectral branch consists of 7 convolutional layers C1~C7 and 3 pooling layers P1~P3. The specific network structure parameters are shown in Table 1.

[0116] Let the i-th convolutional layer of the spectral branch be denoted as C i , the i-th pooling layer is denoted as P i .

[0117] For the padding method of the convolutional layer, SAME means the input size is the same as the output size, and VALID means no padding.

[0118] VALID: This padding method does not perform any padding on the input data, and the convolution kernel slides directly on the input data for convolution operation. Since there is no additional padding, the size of the output feature map is usually smaller than the input data. The specific reduction in size depends on the size of the convolution kernel and the stride. Convolution layers using the VALID padding method are usually used to reduce the spatial size of the feature map;

[0119] SAME: This padding method pads zeros at the edges of the input data so that the size of the output feature map after the convolution operation is the same as the size of the input data. This is usually achieved by padding a certain number of zeros around the input data so that the size of the output feature map remains unchanged regardless of how the convolution kernel slides. Convolutional layers using SAME padding are often used to maintain the spatial size of the feature map for subsequent convolution or pooling operations.

[0120] The 7 convolutional layers C1_C7 can extract the deep spectral features of the input pixel pairs, and the 3 pooling layers P1_P3 are used to reduce the redundant information in the spectral dimension. Finally, the output tensor of C7 is converted into a vector form through vectorization.

[0121] Table 1: Specific network structure and parameters of CNN for refined spectral feature extraction

[0122]

[0123] (2) Construct a spatial branch, which is a CNN for extracting refined spatial features. Figure 4 As shown, pixels belonging to the category of the center pixel are represented by shading, and heterogeneous pixels are represented by blanks.

[0124] A central pixel is a specific pixel that is analyzed or focused on in a hyperspectral image. A heterogeneous pixel refers to a pixel in a hyperspectral image that may contain a variety of different surface materials or objects due to the diversity of surface cover types. For example, a pixel may contain spectral information of vegetation, soil, and water at the same time. The spectral response of such a pixel is not single, but a mixture of multiple substances, so it is called a heterogeneous pixel. In hyperspectral image analysis, the presence of heterogeneous pixels may increase the complexity of classification and recognition.

[0125] In the 9×9 spatial block, the shaded area is smaller than the blank area. If only the 9×9 spatial block is used for feature extraction, the classification effect will be greatly affected by heterogeneous pixels. However, if the spatial block in the 5×5 bold frame in the upper right corner is used for feature extraction, the influence of heterogeneous pixels can be better avoided.

[0126] Based on this idea, a refined spatial feature extraction CNN is designed to The input branch network structure parameters are shown in Table 2, with the upper left corner square Top right square Lower left square Lower right square Central space block The input branch network structure parameters are shown in Table 3.

[0127] The multi-region space block P i As the input of the spatial branch, it undergoes three spatial convolutions to refine the spatial features, and finally the vectors of each branch are concatenated into one vector.

[0128] There are six spatial branches in total, spatial blocks and It serves as the input of the six branches in turn.

[0129] correspond The three convolutional layers of the spatial branch and The sizes are 5×5×q, 3×3×128 and 3×3×256 respectively; where q is the spectral band;

[0130] The remaining three convolutional layers of the five spatial branches and The convolutional layer sizes are 3×3×q, 2×2×128, and 2×2×256, respectively; where x∈{b,c,d,e,f}.

[0131] Table 2: Specific network structure and parameters of CNN for refined spatial feature extraction (in is the input space branch)

[0132]

[0133]

[0134] Table 3: CNN network structure parameters for refined spatial feature extraction (in For the input branch)

[0135]

[0136] (3) Construct the first and second fully connected layers. Connect the output vector of the spectral branch with the output vector of the spatial branch, and pass through the first fully connected layer to obtain a feature vector of dimension fn. fn is a vector of dimension 128;

[0137] After the second fully connected layer, an output vector of dimension K+1 is obtained, which represents the input sample C ij Category.

[0138] Step S22: training using the training sample set to obtain a trained hyperspectral image classification neural network.

[0139] The hyperspectral image classification neural network training is performed using the sample training set as follows:

[0140] Define hyperparameters; set D ij Input the spectral branch to obtain the spectral feature vector, and convert P i middle The six spatial branches of the spatial branch are input in sequence, and are respectively subjected to three layers of convolution in sequence to obtain corresponding vectors and sequentially concatenated to obtain spatial feature vectors;

[0141] Connecting the spectral feature vector with the spatial feature vector to obtain a feature vector, using the feature vector as the input of the first fully connected layer, and outputting a vector with a dimension of 128;

[0142] The vector with a dimension of 128 is input into the second fully connected layer to obtain an output vector with a dimension of K+1, which is the training sample C. ij The prediction category of

[0143] Using the cross entropy function as the loss function for forward propagation, and then using the Adam optimizer for back propagation to optimize all parameters of the hyperspectral image classification neural network, wherein all parameters include the weights and biases of each layer;

[0144] If the number of training iterations reaches the number of training iterations Epoch, the training is terminated and a trained hyperspectral image classification neural network is obtained.

[0145] (1) Define hyperparameters. Define the number of training set iterations as Epoch, the learning rate as LR, and the number of training samples for batch training as batch_size.

[0146] (2) Use the loss function for forward propagation. Use cross entropy as the loss function for forward propagation, as shown in formula (8):

[0147]

[0148] Among them, y is the true label value, is the predicted label value for the network output, batch_size is the number of training samples for batch training, y jk is the value of the kth position of the true label of the jth sample in the training batch, Predict the value of the label at the kth position for the jth example in the training batch.

[0149] (3) Back propagation: The Adam (Adaptive Moment Estimation) optimizer commonly used in the industry is used to optimize all parameters of the neural network (weights and biases of each layer) and perform back propagation.

[0150] (4) Judgment of training end condition: If the number of training iterations is equal to Epoch, then the training ends; if the number of training iterations is less than Epoch, then the training continues.

[0151] Step S2 constructs a refined spatial-spectral joint feature extraction network, which is a deep learning network including two branches, namely a spectral branch and a spatial branch;

[0152] The spectral branch is designed as a refined spectral feature extraction CNN, focusing on extracting the spectral features of the image; the spatial branch is designed as a refined spatial feature extraction CNN, focusing on extracting the spatial features of the image. The neural network is forward propagated using the training sample set, and the training loss is calculated using the loss function based on the difference between the network output and the true label. Backward propagation is then performed based on the loss value to adjust the network parameters to minimize the prediction error. After multiple iterations, a trained neural network is obtained.

[0153] Step S3 includes steps S31-S32.

[0154] Step S31: construct a test sample set based on the test pixels.

[0155] Get the test sample set. For any pixel z among the N test pixels p (p = 1, 2, ..., N), with z p The spatial block Q with a spatial dimension of 9×9 can be obtained as the center p .like Figure 5 As shown. p The main pixel is the spatial block Q p The remaining pixels included in are auxiliary pixels, and there are (9×9-1) auxiliary pixels, so 80 pixel pairs-multi-region spatial block combinations can be obtained, and these combinations together constitute a test pixel z p Corresponding test samples. Since there are N test pixels in total, the number of test samples is N.

[0156] Since neighboring pixels in a hyperspectral image belong to the same class with a greater probability, and the closer the pixel is to the central pixel, the greater the probability that it belongs to the same class as the central pixel, a weighted voting method is proposed in the testing phase to achieve higher classification accuracy.

[0157] Step S32: input the test sample set into the trained hyperspectral image classification neural network, and apply a weighted voting method to determine the classification result of each test pixel in the test sample set.

[0158] The testing process based on the weighted voting algorithm is as follows Figure 5 shown.

[0159] Input the test sample set into the trained hyperspectral image classification neural network, and output a K+1 dimensional vector; the value of each position of the K+1 dimensional vector is the classification category probability;

[0160] The first element of the K+1-dimensional vector is removed from the 0th category, and the maximum element value of the remaining K-dimensional vector is used to obtain the classification result of the test pixel using the weighted voting method, including:

[0161] After inputting the 80 pixel pairs-multi-region space block combinations in the test sample into the trained hyperspectral image classification neural network, 80 predicted category values ​​are obtained;

[0162] In each pixel pair-multi-region spatial block combination, the weights of pixels that are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the main pixel are set to 1.0, 0.9, 0.8, and 0.7, respectively;

[0163] The weighted voting value v of the kth class k , as shown in formula (9):

[0164]

[0165] in, The predicted category values ​​of the kth category are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the main pixel, respectively. The category with the largest weighted voting value is taken as the test pixel z p The classification results.

[0166] If the accuracy of the classification result of the test pixel and the known category is greater than the preset threshold, the trained hyperspectral image neural network is used for classification prediction. Otherwise, the trained hyperspectral image neural network will no longer be used in specific applications and needs to be further optimized.

[0167] The test sample is input into the trained hyperspectral image classification neural network to obtain the output vector. For each pixel pair-multi-region spatial block combination input into the neural network, the neural network will output a K+1-dimensional vector, and the value of each position represents the probability of belonging to the corresponding category. The 0th category is meaningless during the test process, so the first element of the K+1-dimensional vector is removed. The category corresponding to the maximum element value of the remaining K-dimensional vector is the predicted category of the neural network for the pixel pair-multi-region spatial block combination.

[0168] After inputting the 80 pixel pairs-multi-region spatial block combinations in the test samples into the neural network, 80 classification prediction values ​​were obtained. Since the pixels closer to the center pixel have a greater probability of belonging to the same category as the center pixel, based on the test experience summary, the weights of pixels 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the center pixel were set to 1.0, 0.9, 0.8, and 0.7, respectively.

[0169] Finally, the category with the largest weighted voting value is used as the test sample Q p and test pixel z p The predicted category of Figure 5 In the test sample Q, the weighted voting value of the third category is the largest, so the test sample Q p and test pixel z p The predicted category is category 3.

[0170] Step S3 applies the pixel pair-multi-region spatial block algorithm to construct a test sample set. The test sample set is constructed with the test pixel as the center. In the test phase, the weighted voting method is applied to classify and predict the test samples using the trained neural network. Considering the correlation between pixels in the hyperspectral image, especially the pixels that are close in space are more likely to belong to the same category; according to the weighted voting results, the final category label of each test pixel is determined to improve the accuracy of classification.

[0171] Exemplarily, the present invention can be applied to business scenarios such as environmental monitoring and target identification. The application of the present invention in business scenarios of geographical survey and forest tree species identification is summarized as follows:

[0172] Tree species identification is one of the important contents of forest resource monitoring. Timely and accurate acquisition of tree species information is helpful for the sustainable management and assessment of forest resources.

[0173] Traditional tree species identification mainly relies on manual field surveys, which is time-consuming and labor-intensive. The present invention can make up for the shortcomings of digital manual exploration of forests. The hyperspectral image of the forest has rich spectral information and spatial information. By applying the method of the present invention, the hyperspectral image of the forest is input into the refined spatial-spectral joint feature extraction network, and the feature information is refined from the spectral dimension and the spatial dimension at the same time, the subtle differences between different tree species are detected, and the enhanced voting algorithm is used to classify the forest tree species types, thereby achieving high-precision tree species identification.

[0174] A hyperspectral image classification method based on refined spatial-spectral joint feature extraction according to an embodiment of the present invention has the following beneficial effects:

[0175] 1. The problem of insufficient training samples is alleviated by constructing a pixel pair-multi-region spatial block method, which significantly increases the number of training samples and overcomes the problem of insufficient training samples for hyperspectral image classification. This allows the network to be designed deeper and achieve better feature extraction and classification effects.

[0176] 2. Through the refined extraction of spatial-spectral joint features, the spectral and spatial feature information of the image can be more accurately extracted from both the spatial and spectral dimensions, thus improving the classification accuracy of hyperspectral images;

[0177] 3. Further improve the hyperspectral classification performance in the test phase by applying the weighted voting method. By cleverly applying the hyperspectral image property that neighboring pixels have a greater probability of belonging to the same class and pixels closer to the central pixel have a greater probability of belonging to the same class as the central pixel, a weighted voting algorithm is constructed in the test phase to improve the classification performance of the neural network;

[0178] 4. By refining the spatial information around the pixels, the local characteristics of the objects can be better captured.

[0179] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A hyperspectral image classification method based on refined spatial-spectral joint feature extraction, characterized in that: The steps include: Step S1, obtaining an original hyperspectral image and performing preprocessing to obtain training pixels and test pixels of known categories, and constructing a training sample set based on the training pixels; Step S2, constructing a hyperspectral image classification neural network with refined spatial-spectral joint feature extraction, and using the training sample set for training to obtain a trained hyperspectral image classification neural network; Step S3, constructing a test sample set based on the test pixel, inputting the test sample set into the trained hyperspectral image classification neural network, and applying a weighted voting method to determine the classification result of each test pixel in the test sample set; if the accuracy of the classification result of the test pixel and the known category is greater than a preset threshold, using the trained hyperspectral image neural network for classification prediction; The step S1 comprises: Performing normalization processing on the hyperspectral image; The normalized hyperspectral image has a total of K categories, and all categories include L pixels in total. The L pixels are divided into M training pixels and N test pixels according to a preset ratio; the training pixels and the test pixels both include K category pixels; Taking the training pixel as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set; Taking the training pixel as the center, pixel pair-multi-region space block combination is performed to obtain a training sample set, including: For any two training pixels x among the M training pixels i and x j Composed pixel pair D ij =[x i x j ], where x i 、x j They are primary and auxiliary pixels respectively; Construct the primary pixel x i The spatial block with a size of 9×9 is centered The spatial block consists of 81 pixels, with the primary pixel x i As the center, Divided into a 5×5 upper left corner square Top right square Lower left square Lower right square Take the main pixel x i Centered on The central space block with a space size of 5×5 is divided into The multi-region space block is Then the pixel pair-multi-region space block combination is C ij =[D ij P i ]; C ij Label y ij The value selection rules are as follows: Among them, k is the training pixel x i and x j Known categories of Based on each training sample C ij and the corresponding label y ij , and obtain the training sample set, expressed as Among them, X new The number of categories is K+1, including the added 0th category.

2. The method according to claim 1, characterized in that: The hyperspectral image classification neural network includes a spectral branch, a spatial branch, a first fully connected layer and a second fully connected layer; The spectral branch is used to adjust the pixel pair D ij Perform refined spectral feature extraction, including input layer, C1, C2, P1, C3, C4, P2, C5, C6, P3 and C7 in sequence, among which C1~C7 are convolution layers and P1~P3 are pooling layers; The spatial branch includes six spatial branches for D ij The corresponding 6 spatial blocks are used for refined spatial feature extraction, and each branch includes 3 convolutional layers.

3. The method according to claim 2, characterized in that: correspond The three convolutional layers of the spatial branch and The sizes are 5×5×q, 3×3×128 and 3×3×256 respectively; where q is the spectral band; The remaining three convolutional layers of the five spatial branches and The convolutional layer sizes are 3×3×q, 2×2×128, and 2×2×256, respectively; where x∈{b,c,d,e,f}.

4. The method according to claim 3, characterized in that: The hyperspectral image classification neural network training is performed using the training sample set as follows: Define hyperparameters; set D ij Input the spectral branch to obtain the spectral feature vector, and convert P i middle The six spatial branches of the spatial branch are input in sequence, and respectively go through three layers of convolution in sequence to obtain corresponding vectors and sequentially concatenate them to obtain spatial feature vectors; Connecting the spectral feature vector with the spatial feature vector to obtain a feature vector, using the feature vector as the input of the first fully connected layer, and outputting a vector with a dimension of 128; The vector with a dimension of 128 is input into the second fully connected layer to obtain an output vector with a dimension of K+1, which is the training sample C. ij The prediction category of Using the cross entropy function as the loss function for forward propagation, and then using the Adam optimizer for back propagation to optimize all parameters of the hyperspectral image classification neural network, wherein all parameters include the weights and biases of each layer; If the number of training iterations reaches the number of training iterations Epoch, the training is terminated and a trained hyperspectral image classification neural network is obtained.

5. The method according to claim 1, characterized in that: Constructing a test sample set based on the test pixels, including: For any test pixel z among the N test pixels p , with z p The spatial dimension is 9×9 space block Q centered at p , where p = 1, 2, ..., N; With z p The main pixel is the spatial block Q p In addition to z p The remaining test pixels are auxiliary pixels, and 80 pixel pairs are obtained, and then 80 pixel pair-multi-region spatial block combinations are obtained. Each pixel pair-multi-region spatial block combination is a test sample; the test samples corresponding to all test pixels constitute a test sample set.

6. The method according to claim 5, characterized in that: Input the test sample set into the trained hyperspectral image classification neural network, and output a K+1 dimensional vector; the value of each position of the K+1 dimensional vector is the classification category probability; The first element of the K+1-dimensional vector is removed from the 0th category, and the maximum element value of the remaining K-dimensional vector is used to obtain the classification result of the test pixel using the weighted voting method, including: After inputting the 80 pixel pairs-multi-region space block combinations in the test sample into the trained hyperspectral image classification neural network, 80 predicted category values ​​are obtained; In each pixel pair-multi-region spatial block combination, the weights of pixels that are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the primary pixel are set to 1.0, 0.9, 0.8, and 0.7, respectively; The weighted voting value v of the kth class k ,as follows: in, The predicted category values ​​of the kth category are 0 pixels, 1 pixel, 2 pixels, and 3 pixels away from the main pixel, respectively. The category with the largest weighted voting value is taken as the test pixel z p The classification results.

7. The method according to claim 4, characterized in that: Use cross entropy as the loss function for forward propagation as follows: Among them, y is the true label value, is the predicted label value for the network output, batch_size is the number of training samples for batch training, y jk is the value of the kth position of the true label of the jth sample in the training batch, Predict the value of the label at the kth position for the jth example in the training batch.

8. The method according to claim 1, characterized in that: For an original hyperspectral image with h rows, w columns, and q spectral bands Normalization is performed as follows: Among them, H is the normalized hyperspectral image, and m and n are the maximum and minimum element values ​​of H0, respectively.

Images