Hyperspectral image open set classification method based on inverse prototype and MLP network

Abstract

The invention discloses a hyperspectral image open set classification method based on an inverse prototype and an MLP network, and relates to the technical field of hyperspectral image classification. Firstly, a hyperspectral classification data set is prepared and divided into a training set, a verification set and a test set; establishing a classification model; the classification model comprises a spatial-spectral feature extractor, an inverse prototype module, an MLP network and a known category classifier; training and verifying the classification model by using the training set and the verification set, and storing the trained classification model; and inputting the test set into the trained classification model to obtain a classification result. According to the method, the inverse prototype is generated for each known category, the distance from a sample of each known category to the corresponding inverse prototype is maximum, the distance from a sample of an unknown category to the inverse prototype is minimum, and the risk of open space is reduced. An MLP network is designed, and unknown classes are automatically recognized by learning a comprehensive distance from a sample to an inverse prototype, so that the model gets rid of dependence on a manual threshold value.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image classification, and in particular to an open set classification method for hyperspectral images based on an inverse prototype and an MLP network. Background Art

[0002] Hyperspectral image (HSI) is an important data type in remote sensing. It is widely used in ecological and agricultural protection, urban environmental monitoring, geological assessment, resource exploration, marine monitoring and military reconnaissance, and has great practical significance and application value. HSI contains hundreds of spectral bands, which provides rich information for accurate classification. In recent years, deep learning methods have been applied to hyperspectral image classification and have achieved excellent performance. However, the currently widely used deep learning algorithms are still developed based on the closed world assumption. This closed world assumption simplifies the problem and promotes the development of deep learning technology to a certain extent, but it limits the generalization ability of the model and reduces the adaptability of the model to new environments. In actual scenarios, deep learning models may encounter completely new situations that have never been seen before, which are beyond the scope of the closed world assumption. This not only affects the performance of the model, but also challenges its applicability and reliability. For example, hyperspectral image classification tasks are performed in two scenarios: in different geographical areas and in the same geographical area but at different times. In both cases, new category samples may be generated due to spatial or temporal changes. Therefore, it is important to design deep learning methods to be able to handle and adapt to unknown data or environments, remain stable and safe when encountering unforeseen situations, and quickly adapt and learn from new situations.

[0003] The Open Set Recognition method requires the model to not only classify samples belonging to known categories, but also to be able to identify unknown samples that do not belong to known categories, which to a certain extent solves the problem that deep learning methods rely on closed set assumptions. Most OSR methods identify unknown classes through thresholds, which need to be set manually and rely on experience and data, and are difficult to set to the optimal level. At the same time, many methods set prototypes for known classes and classify samples based on the distance from the sample to the prototype. However, due to the overlap of the distance from the sample to the prototype, the open set classification effect of hyperspectral images is poor. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention discloses a hyperspectral image open set classification method based on inverse prototype and MLP network to solve the problem of existing OSR method relying on threshold decision and the problem of overlapping distances from samples to prototypes. After training, the method can not only classify samples belonging to known categories, but also identify unknown samples that do not belong to known categories, and can be applied to actual scenarios.

[0005] The hyperspectral image open set classification method based on inverse prototype and MLP network includes the following steps:

[0006] S1: Prepare a hyperspectral classification dataset and divide it into a training set, a validation set, and a test set according to a set ratio;

[0007] The hyperspectral classification data set includes a number of cube blocks of a specific size divided from the hyperspectral image, each cube block is used as a sample, a part of the classification data set is samples of known categories, and the other part is samples of unknown categories, and the category of the samples of known categories is the category of the central pixel; the training set, the validation set and the test set all include samples of known categories and samples of unknown categories;

[0008] S2: Establishing a classification model; the classification model includes a spatial spectrum feature extractor, an inverse prototype module, an MLP network and a known category classifier;

[0009] The spatial-spectral feature extractor is used to extract the spatial-spectral feature F(x) of the input sample, and includes a spatial branch and a spectral branch. The spatial branch and the spectral branch are symmetrical in structure, and both include a number of sequentially connected Conv3D-BN rectified linear units ReLU and a pooling module AvgPooling. The convolution kernel sizes of the two branches are different. The spatial branch is used to extract the spatial features of the input sample, and the spectral branch is used to extract the spectral features of the input sample. Then, the spatial features and the spectral features are added to obtain the spatial-spectral features F(x) as the input of the inverse prototype module;

[0010] The inverse prototype module is used to establish an inverse prototype for each known category, and calculate the comprehensive distance from the empty spectrum feature of the input sample to each inverse prototype as the input of the MLP network;

[0011] The MLP network is used to establish a mapping relationship between the comprehensive distance and the binary label of the input sample, and output the predicted binary label of the input sample according to the input comprehensive distance; the binary label includes 1 for known categories and 0 for unknown categories;

[0012] The known category classifier calculates the probability that the input sample belongs to each known category through a softmax function according to the comprehensive distance from the empty spectrum feature of the input sample whose binary label is predicted to be a known category to the inverse prototype, and takes the known category with the largest probability as the classification result;

[0013] S3: Use the training set and validation set to train and validate the classification model, and save the trained classification model;

[0014] S3.1: Initialize the empty spectrum feature extractor in the classification model, input the samples in the training set into the empty spectrum feature extractor, and obtain the empty spectrum features F(x) of the samples in the training set;

[0015] S3.2: For K known categories, randomly set a learnable vector for each known category and assign it as the inverse prototype of the known category, and calculate the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the K inverse prototypes;

[0016] Specifically, the Euclidean distance from the empty spectrum feature of the sample in the training set to the inverse prototype is calculated according to formula (1), and then the similarity between the empty spectrum feature of the sample in the training set and the inverse prototype is calculated according to formula (2). Finally, the Euclidean distance is subtracted from the similarity according to formula (3) to obtain the comprehensive distance from the empty spectrum feature of the sample in the training set to the inverse prototype.

[0017]

[0018] d d (F(x),p k )=F(x)·p k (2)

[0019] d(F(x),p k ) = d e (F(x),p k )-d d (F(x),p k ) (3)

[0020] Among them, x represents the sample, d e (F(x),p k ) represents the Euclidean distance from the sample’s empty spectrum feature to the inverse prototype of the kth known category, p k represents the inverse prototype of the kth known category, k is the number of the inverse prototype, and m represents the dimension of the empty spectrum feature of the sample; d d (F(x),p k ) represents the similarity between the sample's empty spectrum feature and the inverse prototype of the kth known category; d(F(x),p k ) represents the comprehensive distance from the sample's empty spectrum feature to the inverse prototype of the k-th known category;

[0021] S3.3: Input the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the K inverse prototypes into the MLP network for open set classification. The MLP network maps the comprehensive distance to the hidden layer, passes the output result of the hidden layer to the output layer, obtains the predicted binary label and calculates the open set classification loss L o ;

[0022] The predicted binary label and the true binary label are calculated according to formula (4) to calculate the open set classification loss L o :

[0023]

[0024] Among them, L o represents the open set classification loss, n represents the number of samples, t i represents the binary label value of the i-th sample, p(t i =1) indicates the probability that the binary label value of the i-th sample is predicted by the MLP network to be 1;

[0025] S3.4: Input the comprehensive distance from the empty spectrum feature F(x) of the sample whose binary label is predicted as a known category to the K inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, take the known category with the largest probability as the classification result and calculate the known category classification loss L c (x; F, p);

[0026]

[0027] L c (x;F,p)=-logp(y=k∣x,F,p) (6)

[0028] Where p(y=k|x,F,p) represents the probability that sample x belongs to the kth known category, and i is the number of the known category;

[0029] S3.5: Open set recognition loss L o and known class classification loss L c (x; F, p) are added to calculate the total loss L;

[0030] L=L o +L c (x; F, p) (7)

[0031] S3.6: Input the samples in the validation set into the empty spectrum feature extractor of the current classification model to extract the empty spectrum features of the samples in the validation set;

[0032] S3.7: Keep the positions of the K inverse prototypes unchanged and calculate the comprehensive distance from the empty spectrum features of the samples in the validation set to the K inverse prototypes;

[0033] S3.8: Input the comprehensive distance from the empty spectrum features of the samples in the validation set to the K inverse prototypes into the MLP network to obtain the predicted binary labels, compare the predicted binary labels with the true binary labels, and calculate the recognition accuracy of the unknown categories;

[0034] S3.9: Input the comprehensive distance from the empty spectrum feature of the sample whose binary label is predicted to be a known category to the K inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, and then obtain the classification result of the known category, compare the classification result with the actual known category label, and calculate the classification accuracy of the known category;

[0035] S3.10: Determine whether the accuracy requirement is met based on the recognition accuracy of unknown categories and the classification accuracy of known categories. If yes, end the training, save the trained classification model and execute S4. Otherwise, perform back propagation based on the total loss L, update the classification model parameters and continue the next round of training and verification.

[0036] S4: Input the test set into the trained classification model to obtain the classification result.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] The invention provides a hyperspectral image open set classification method based on inverse prototype and MLP network. The method generates an inverse prototype for each known category to achieve the maximum distance between each known category sample and its corresponding inverse prototype, and the minimum distance between the unknown category sample and the inverse prototype, thereby reducing the risk of open space. The MLP network is designed to automatically identify unknown categories by learning the comprehensive distance between the sample and the inverse prototype, so that the model is free from the dependence on manual thresholds. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of an open set classification method for hyperspectral images based on an inverse prototype and an MLP network provided by an embodiment of the present invention;

[0040] Figure 2 A flowchart of a specific implementation method of S3 provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to solve the problem that the existing OSR method relies on manual thresholds to judge known and unknown categories and the problem that the distances between known and unknown categories and prototypes overlap, resulting in poor classification results, this embodiment provides an open set classification method for hyperspectral images based on inverse prototype and MLP network, such as Figure 1 and Figure 2 As shown, the following steps are included:

[0042] S1: Prepare a hyperspectral classification dataset and divide it into a training set, a validation set, and a test set according to a set ratio;

[0043] The hyperspectral classification data set includes a number of cube blocks of a specific size divided from the hyperspectral image, each cube block is used as a sample, a part of the classification data set is samples of known categories, and the other part is samples of unknown categories, and the category of the samples of known categories is the category of the central pixel; the training set, the validation set and the test set all include samples of known categories and samples of unknown categories;

[0044] In this implementation, the Pavia University hyperspectral dataset is used. The hyperspectral image size is 610×340, with 103 spectral bands, and is labeled as 9 categories. Classes 1-7 are set as known classes, and classes 8-9 are set as unknown classes. The hyperspectral image is divided into cubes I∈R of size 7×7. 7×7×c , where 7×7 is the spatial size of the cube and c is the number of bands. All cubes are used as hyperspectral classification datasets. The hyperspectral classification dataset is divided into training set, validation set and test set in a ratio of 6:2:2. Each set contains both known category samples and unknown category samples.

[0045] S2: Establishing a classification model; the classification model includes a spatial spectrum feature extractor, an inverse prototype module, an MLP network and a known category classifier;

[0046] The spatial-spectral feature extractor is used to extract the spatial-spectral feature F(x) of the input sample, and includes a spatial branch and a spectral branch. The spatial branch and the spectral branch are symmetrical in structure, and both include a number of sequentially connected Conv3D-BN rectified linear units ReLU and a pooling module AvgPooling. The convolution kernel sizes of the two branches are different. The spatial branch is used to extract the spatial features of the input sample, and the spectral branch is used to extract the spectral features of the input sample. Then, the spatial features and the spectral features are added to obtain the spatial-spectral features F(x) as the input of the inverse prototype module;

[0047] The inverse prototype module is used to establish an inverse prototype for each known category, and calculate the comprehensive distance from the empty spectrum feature of the input sample to each inverse prototype as the input of the MLP network;

[0048] The MLP network is used to establish a mapping relationship between the comprehensive distance and the binary label of the input sample, and output the predicted binary label of the input sample according to the input comprehensive distance; the binary label includes 1 for known categories and 0 for unknown categories;

[0049] Further, the MLP network comprises an input layer, a first hidden layer, a second hidden layer and an output layer connected in sequence, wherein the number of input units of the input layer is the number of inverse prototypes, the first hidden layer has 8 hidden units, the second hidden layer has 4 hidden units, and the output layer has 2 output units;

[0050] The known category classifier calculates the probability that the input sample belongs to each known category through a softmax function according to the comprehensive distance from the empty spectrum feature of the input sample whose binary label is predicted to be a known category to the inverse prototype, and takes the known category with the largest probability as the classification result;

[0051] S3: Use the training set and validation set to train and validate the classification model, and save the trained classification model; the purpose of training is to make each known class sample away from its corresponding inverse prototype, classify the known class samples according to the distance from the known class samples to the inverse prototype, and gather the unknown class samples into a specific space for automatic identification through the MLP network;

[0052] S3.1: Initialize the empty spectrum feature extractor in the classification model, and convert the samples I∈R in the training set 7×7×c Input the empty spectrum feature extractor to obtain the empty spectrum features F(x) of the samples in the training set;

[0053] The initialization specifically includes: initializing the parameters of the empty spectrum feature extractor including model weight, bias, initial learning rate and training times epochs;

[0054] In this implementation, the model initial parameters and bias are set, the initial learning rate is set to 0.01, and the number of training epochs is set to 1000 rounds. 7×7×c The feature extractor is input, and the feature extractor extracts 64-dimensional spectral features and 64-dimensional spatial features. The spectral features and the spatial features are added to obtain 64-dimensional spatial spectrum features.

[0055] S3.2: For each of the seven known categories, randomly set a learnable vector and assign it as the inverse prototype of the known category. The size of each vector is 64 dimensions. Calculate the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the seven inverse prototypes.

[0056] Specifically, first, the Euclidean distance from the empty spectrum feature of the sample in the training set to the inverse prototype is calculated according to formula (1), and the size is (n, 7), where n represents the number of samples in the training set. Then, the similarity between the empty spectrum feature of the sample in the training set and the inverse prototype is calculated according to formula (2), and the size is also (n, 7). Finally, according to formula (3), the Euclidean distance is subtracted from the similarity to obtain the comprehensive distance from the empty spectrum feature of the sample in the training set to the inverse prototype, and the size is (n, 7).

[0057]

[0058] d d (F(x),p k )=F(x)·p k (2)

[0059] d(F(x),p k ) = d e (F(x),p k )-d d (F(x),p k ) (3)

[0060] Among them, x represents the sample, d e (F(x),p k ) represents the Euclidean distance from the sample’s empty spectrum feature to the inverse prototype of the kth known category, p k represents the inverse prototype of the kth known category, k is the number of the inverse prototype, and m represents the dimension of the empty spectrum feature of the sample; d d (F(x),p k ) represents the similarity between the sample's empty spectrum feature and the inverse prototype of the kth known category; d(F(x),p k ) represents the comprehensive distance from the sample's empty spectrum feature to the inverse prototype of the k-th known category;

[0061] S3.3: Input the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the 7 inverse prototypes into the MLP network for open set classification. The MLP network maps the comprehensive distance to the hidden layer, passes the output result of the hidden layer to the output layer, obtains the predicted binary label and calculates the open set classification loss L o ;

[0062] There are also two true binary labels, 1 for known categories and 0 for unknown categories. The predicted binary labels and true binary labels are compared according to formula (4) to calculate the open set classification loss L o :

[0063]

[0064] Among them, L o represents the open set classification loss, n represents the number of samples, t i represents the binary label value of the i-th sample, p(t i =1) indicates the probability that the binary label value of the i-th sample is predicted by the MLP network to be 1;

[0065] S3.4: Input the comprehensive distance from the empty spectrum feature F(x) of the sample whose binary label is predicted as a known category to the K inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, take the known category with the largest probability as the classification result and calculate the known category classification loss L c (x; F, p);

[0066]

[0067] L c (x;F,p)=-logp(y=k∣x,F,p) (6)

[0068] Where p(y=k|x,F,p) represents the probability that sample x belongs to the kth known category, and i is the number of the known category;

[0069] S3.5: Open set recognition loss L o and known class classification loss L c (x; F, p) are added to calculate the total loss L;

[0070] L=L o +L c (x; f, p) (7)

[0071] S3.6: Input the samples in the validation set into the empty spectrum feature extractor of the current classification model to extract the empty spectrum features of the samples in the validation set;

[0072] S3.7: Keep the positions of the 7 inverse prototypes unchanged and calculate the comprehensive distance from the empty spectrum features of the samples in the validation set to the 7 inverse prototypes;

[0073] S3.8: Input the comprehensive distance from the empty spectrum features of the samples in the validation set to the 7 inverse prototypes into the MLP network to obtain the predicted binary labels, compare the predicted binary labels with the true binary labels, and calculate the recognition accuracy of the unknown categories;

[0074] S3.9: Input the comprehensive distance from the empty spectrum feature of the sample whose binary label is predicted to be a known category to the 7 inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, and then obtain the classification result of the known category, compare the classification result with the actual known category label, and calculate the classification accuracy of the known category;

[0075] S3.10: Determine whether the accuracy requirement is met based on the recognition accuracy of unknown categories and the classification accuracy of known categories (the accuracy remains stable for multiple rounds or the accuracy continues to decrease for multiple rounds). If yes, end the training, save the trained classification model and execute S4. Otherwise, perform back propagation based on the total loss L, update the classification model parameters and continue the next round of training and verification.

[0076] Among them, the inverse prototype is updated so that the comprehensive distance from each known category sample to its corresponding inverse prototype is maximized, and the comprehensive distance from the unknown category sample to the inverse prototype is minimized;

[0077] S4: Input the test set into the trained classification model to obtain the classification result.

Claims

1. A hyperspectral image open set classification method based on inverse prototype and MLP network, characterized in that: The following steps are involved: S1: Prepare a hyperspectral classification dataset and divide it into a training set, a validation set, and a test set according to a set ratio; S2: Establishing a classification model; the classification model includes a spatial spectrum feature extractor, an inverse prototype module, an MLP network and a known category classifier; S3: Use the training set and validation set to train and validate the classification model, and save the trained classification model; S4: Input the test set into the trained classification model to obtain the classification result.

2. The hyperspectral image open set classification method based on inverse prototype and MLP network according to claim 1 is characterized in that: The hyperspectral classification dataset described in S1 includes a number of cubic blocks of a specific size divided from a hyperspectral image, each cubic block serves as a sample, a part of the classification dataset is samples of known categories, and the other part is samples of unknown categories, and the category of the samples of known categories is the category of the central pixel; the training set, validation set and test set all include samples of known categories and samples of unknown categories.

3. The hyperspectral image open set classification method based on inverse prototype and MLP network according to claim 1 is characterized in that: The spatial-spectral feature extractor in S2 is used to extract the spatial-spectral feature F(x) of the input sample, including a spatial branch and a spectral branch. The spatial branch and the spectral branch are symmetrical in structure, both including a number of sequentially connected Conv3D-BN rectified linear units ReLU and a pooling module AvgPooling. The convolution kernel sizes of the two branches are different. The spatial branch is used to extract the spatial features of the input sample, and the spectral branch is used to extract the spectral features of the input sample. Then, the spatial features and the spectral features are added to obtain the spatial-spectral features F(x) as the input of the inverse prototype module; The inverse prototype module is used to establish an inverse prototype for each known category, and calculate the comprehensive distance from the empty spectrum feature of the input sample to each inverse prototype as the input of the MLP network; The MLP network is used to establish a mapping relationship between the comprehensive distance and the binary label of the input sample, and output the predicted binary label of the input sample according to the input comprehensive distance; the binary label includes 1 for known categories and 0 for unknown categories; The known category classifier calculates the probability that the input sample belongs to each known category through a softmax function based on the comprehensive distance from the empty spectrum feature of the input sample whose binary label is predicted to be a known category to the inverse prototype, and takes the known category with the largest probability as the classification result.

4. The hyperspectral image open set classification method based on inverse prototype and MLP network according to claim 3 is characterized in that: S3 specifically includes: S3.1: Initialize the empty spectrum feature extractor in the classification model, input the samples in the training set into the empty spectrum feature extractor, and obtain the empty spectrum features F(x) of the samples in the training set; S3.2: For K known categories, randomly set a learnable vector for each known category and assign it as the inverse prototype of the known category, and calculate the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the K inverse prototypes; Specifically, the Euclidean distance from the empty spectrum feature of the sample in the training set to the inverse prototype is calculated according to formula (1), and then the similarity between the empty spectrum feature of the sample in the training set and the inverse prototype is calculated according to formula (2). Finally, the Euclidean distance is subtracted from the similarity according to formula (3) to obtain the comprehensive distance from the empty spectrum feature of the sample in the training set to the inverse prototype. d d (F(x),p k )=F(x)·p k (2) d(F(x),p k )=d e (F(x),p k )-d d (F(x),p k ) (3) Among them, x represents the sample, d e (F(x),p k ) represents the Euclidean distance from the sample’s empty spectrum feature to the inverse prototype of the kth known category, p k represents the inverse prototype of the kth known category, k is the number of the inverse prototype, and m represents the dimension of the empty spectrum feature of the sample; d d (F(x),p k ) represents the similarity between the sample's empty spectrum feature and the inverse prototype of the kth known category; d(F(x),p k ) represents the comprehensive distance from the sample's empty spectrum feature to the inverse prototype of the k-th known category; S3.3: Input the comprehensive distance from the empty spectrum feature F(x) of the sample in the training set to the K inverse prototypes into the MLP network for open set classification. The MLP network maps the comprehensive distance to the hidden layer, passes the output result of the hidden layer to the output layer, obtains the predicted binary label and calculates the open set classification loss L o ; S3.4: Input the comprehensive distance from the empty spectrum feature F(x) of the sample whose binary label is predicted as a known category to the K inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, take the known category with the largest probability as the classification result and calculate the known category classification loss L c (x; F, p); S3.5: Open set recognition loss L o and known class classification loss L c (X; F, p) are added to calculate the total loss L; L=L o +L c (x;F,p) (4) S3.6: Input the samples in the validation set into the empty spectrum feature extractor of the current classification model to extract the empty spectrum features of the samples in the validation set; S3.7: Keep the positions of the K inverse prototypes unchanged and calculate the comprehensive distance from the empty spectrum features of the samples in the validation set to the K inverse prototypes; S3.8: Input the comprehensive distance from the empty spectrum features of the samples in the validation set to the K inverse prototypes into the MLP network to obtain the predicted binary labels, compare the predicted binary labels with the true binary labels, and calculate the recognition accuracy of the unknown categories; S3.9: Input the comprehensive distance from the empty spectrum feature of the sample whose binary label is predicted to be a known category to the K inverse prototypes into the softmax function to calculate the probability that the sample belongs to each known category, and then obtain the classification result of the known category, compare the classification result with the actual known category label, and calculate the classification accuracy of the known category; S3.10: Determine whether the accuracy requirements are met based on the recognition accuracy of unknown categories and the classification accuracy of known categories. If so, end the training, save the trained classification model and execute S4. Otherwise, perform backpropagation based on the total loss L, update the classification model parameters and continue the next round of training and verification.

5. The hyperspectral image open set classification method based on inverse prototype and MLP network according to claim 4 is characterized in that: The open set classification loss described in S3.3 is: Among them, L o represents the open set classification loss, n represents the number of samples, t i represents the binary label value of the i-th sample, p(t i =1) indicates the probability that the MLP network predicts the binary label value of the i-th sample to be 1.

6. The hyperspectral image open set classification method based on inverse prototype and MLP network according to claim 5, characterized in that: The known category classification loss L described in S3.4 c (x; F, p) is: L c (x;F,p)=-logp(y=k∣x,F,p) (7) Among them, p(y=k|x,F,p) represents the probability that sample x belongs to the kth known category, and i is the number of the known category.

Images