Hyperspectral open set classification method combining feature reconstruction and prototype constraint

Abstract

The invention relates to the technical field of image classification, in particular to a hyperspectral open set classification method combining feature reconstruction and prototype constraint, and the method comprises the steps: carrying out the blocking and labeling of each piece of original sample data in an original sample data set; inputting the sample data blocks into a classification model composed of an encoder, a decoder, a prototype classification module and an unknown category detection module; constructing and comparing prototype loss, reconstruction error loss and cross entropy loss by using an encoder, a decoder and a prototype classification module; constructing a total loss function on the basis of comparison prototype loss, reconstruction error loss and cross entropy loss, and performing iterative training on the classification model until convergence to obtain a trained classification model; and performing open set identification based on the sample feature distribution and the reconstruction error loss by using an unknown category detection module, and evaluating the possibility score of the unknown category in combination with the prototype distance and the reconstruction loss so as to determine the unknown category. According to the method, high-precision hyperspectral open set classification is well realized.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image classification, and in particular to a hyperspectral open set classification method combining feature reconstruction and prototype constraints. Background Art

[0002] Hyperspectral images are widely used in many fields such as environment, agriculture, ocean, and military due to their rich spatial and spectral information. Hyperspectral images usually contain hundreds or even thousands of narrow spectral bands, providing rich spectral information and capable of presenting the intrinsic distinguishing properties of scene targets in a refined manner. By classifying ground objects pixel by pixel, each pixel in the hyperspectral image can be assigned an independent ground object label, such as grass, road, forest, building, soil, river, etc. Therefore, hyperspectral image classification has become a powerful means for humans to achieve scientific detection of large areas and difficult-to-access targets.

[0003] In recent years, classification systems based on convolutional neural networks have greatly improved the accuracy of hyperspectral image classification. However, in practical applications, classification systems are often deployed to work in an unstable and open environment. Not all categories belong to the label set in a closed system, and there will be test samples of unknown categories. The classification system not only needs to correctly classify samples belonging to the closed set, but also needs to reject unknown test samples. It is difficult to accurately model unknown categories with complex distributions by relying on a single basis, and generative methods rely on the quality of known categories.

[0004] In order to solve this technical problem, researchers at home and abroad have proposed some hyperspectral open set classification methods based on deep learning. For data from known categories, the system classifies it according to the data features of the known categories it has mastered. For data from unknown categories, the system can detect that it is not from a known category, and use the discriminant model or the generative model to label it as "unknown category", and then use other learning methods to identify this type of data in the future.

[0005] However, the inventors of the present application have found that the currently proposed hyperspectral open set classification methods have the following problems.

[0006] First, due to the spatial-spectral joint characteristics of hyperspectral images, convolutional neural networks are limited by the receptive field in extracting hyperspectral image classification, making it difficult to focus on channel information and unable to ensure effective extraction of feature maps, which in turn affects the open set classification effect.

[0007] Second, the currently proposed hyperspectral open set classification methods rely on the distribution of training data for modeling. However, in complex imaging scenarios, the distribution of training data and test data may be inconsistent. Identifying unknown categories through outliers is prone to recognition errors and classification errors.

[0008] Third, in the process of hyperspectral open set classification, due to the task bias of neural networks and the imbalance of training data distribution, the classifier is prone to over-compress intra-class features in pursuit of accuracy, which in turn causes the classification model to ignore important information that helps distinguish unknown categories. Summary of the invention

[0009] In view of this, an embodiment of the present application proposes a hyperspectral open set classification method that combines feature reconstruction and prototype constraints, which can effectively identify unknown categories while maintaining the accuracy of hyperspectral closed set classification, thereby achieving high-precision and fast hyperspectral open set classification, which has a great positive effect on promoting the widespread application of hyperspectral images.

[0010] In a first aspect, an embodiment of the present application proposes a hyperspectral open set classification method combining feature reconstruction and prototype constraints, the method comprising: dividing each original sample data in the original sample data set into blocks to obtain a training set consisting of a plurality of sample data blocks, and taking the label of the center pixel point at the top layer of the sample data block as the label of the entire sample data block; wherein the original sample data is a hyperspectral image data cube; inputting the sample data block into a classification model composed of an encoder, a decoder, a prototype classification module and an unknown category detection module, using the encoder to extract features from the sample data block to obtain a corresponding feature vector, and at the same time calculating the mean of each known category based on the feature vector, constructing a prototype library, and constructing a comparative prototype loss based on the prototype library; using the decoder to reconstruct features based on the feature vector to obtain a reconstructed sample data block, and based on the sample data block and the reconstructed sample The reconstruction error loss is constructed based on the data block; the prototype classification module is used to perform closed set classification based on the feature vector and the prototype library to obtain the closed set classification result of the sample data block, and the cross entropy loss is constructed based on the closed set classification result and the label of the sample data block; the total loss function is constructed based on the comparison of prototype loss, reconstruction error loss and cross entropy loss, and the classification model is iteratively trained until convergence based on the total loss function to obtain the trained classification model; the unknown category detection module of the trained classification model is used to perform open set recognition based on the sample feature distribution of the training set and the reconstruction error loss, and the possibility score of the unknown category is evaluated by combining the prototype distance and the reconstruction loss, the possibility scores of all sample data blocks in the training set are analyzed, the tail is modeled using the Weibull distribution, the generalized Pareto distribution is used to approximate its cumulative distribution function, and the first 50% of the tail of the entire cumulative distribution function is determined to be the unknown category.

[0011] Optionally, the shape of the original sample data in the original sample dataset is , is the number of rows of pixels in the original sample data, is the number of columns of pixels in the original sample data, is the number of bands of the original sample data. Each pixel of the original sample data is annotated with a label representing the category. According to the preset division standard, each original sample data in the original sample data set is divided into blocks, and several sample data blocks are divided from each original sample data. A training set is formed based on all sample data blocks, and the label of the top-level center pixel of the sample data block is used as the label of the entire sample data block; wherein the shape of each sample data block is , , .

[0012] Optionally, the encoder consists of multiple convolutional layers and global average pooling layers; The encoder is used to extract features from the sample data block to obtain the corresponding feature vector, including: First, the sample data blocks are concatenated together through two parallel convolutions to obtain the first intermediate feature , the first intermediate feature It is expressed by the formula: ; in, represents a sample data block, and represents two parallel convolutions, Indicates splicing; Then for the first intermediate feature After adding the residual module of the channel attention mechanism, the second intermediate feature is obtained , the second intermediate feature It is expressed by the formula: ; in, and Both represent convolution, Represents the channel attention mechanism; Then the second intermediate feature After the joint weighting of channel attention and spatial attention, the third intermediate feature is obtained , the third intermediate feature It is expressed by the formula: ; in, represents the sigmoid activation function, and represents a multi-layer perceptron, represents the ReLU activation function, Represents the second intermediate feature Medium pixel The value of represents convolution, Indicates the total number of channels, Represents the first channels, Represents a vector product operation; Finally, the third intermediate feature Perform average pooling to obtain the corresponding feature vector .

[0013] Optionally, the mean of each known category is calculated based on the feature vector to construct a prototype library, which is achieved by the following formula: ; ; in, represents the total number of known categories, Indicates Prototypes corresponding to known categories, A prototype library representing the construction, Indicates that it belongs to The total number of feature vectors of known categories, Indicates Belongs to feature vectors of known categories; Select the feature vectors with confidence less than the preset confidence threshold to calculate the mean as the placeholder of the unknown category. The placeholder of the unknown category is expressed by the formula: ; in, Indicates the total number of feature vectors whose confidence is less than the preset confidence threshold, Indicates feature vectors whose confidence is less than the preset confidence threshold, A placeholder for an unknown class; The comparison prototype loss is constructed based on the prototype library and is implemented by the following formula: ; ;

[0014] ; in, express and The Euclidean distance between Indicates the maximum value of the distance between prototypes in the prototype library. Indicates of known categories The distance between the feature vector and each prototype in the prototype library, Represents the constructed comparison prototype loss.

[0015] Optionally, the decoder adopts a deconvolution structure symmetrical to that of the encoder; The reconstruction error loss is constructed based on the sample data block and the reconstructed sample data block, which is implemented by the following formula: ; in, Indicates the number of sample data blocks input for each iteration, Indicates the first A shallow semantic representation, The decoder reconstructs A shallow semantic representation, Represents the reconstruction error loss obtained by construction.

[0016] Optionally, a prototype classification module is used to perform closed set classification based on the feature vector and the prototype library to obtain a closed set classification result for the sample data block, including: Calculate the Euclidean distance between the feature vector and each prototype in the prototype library; Select the category corresponding to the prototype with the smallest Euclidean distance as the closed set classification result of the sample data block; The cross entropy loss is constructed based on the closed set classification results and the labels of the sample data blocks, which is implemented by the following formula: ; in, represents the closed set classification result, Indicates the label, Represents the constructed cross entropy loss.

[0017] Optionally, a total loss function is constructed based on the contrastive prototype loss, the reconstruction error loss, and the cross entropy loss, which is implemented by the following formula: ; in, , and is the preset weight parameter, is the total loss function constructed.

[0018] Optionally, the likelihood score for the unknown class is constructed as follows: ; in, is the preset assessment adjustment factor, Represents the probability that the sample data block belongs to each known category, The probability that a sample data block belongs to an unknown category; Analyze the likelihood scores of all sample data blocks in the training set, use Weibull distribution to model the tail, and use generalized Pareto distribution to approximate its cumulative distribution function, which is achieved through the following formula: ; in, represents the Weibull distribution, represents the generalized Pareto distribution, and are all learnable parameters, and The value of is determined through model training; The first 50% of the tail of the entire cumulative distribution function is determined as the unknown category, which is achieved by the following formula: ; in, Adjust the parameters for the preset mapping.

[0019] The present application proposes a hyperspectral open set classification method combining feature reconstruction and prototype constraints, which chooses to reconstruct shallow semantic representation instead of high-dimensional original image data, significantly reducing the computational complexity of the model, while making feature extraction more focused on key information for class distinction. Shallow semantic representation carries high-recognition features of known categories, while retaining sufficient distinguishing ability for the complexity of unknown categories, thereby effectively improving the classification efficiency and accuracy of the classification model in an open set environment. Based on the manifold representation generated by the encoder, efficient distinction of known categories is achieved by modeling each known category separately. In addition, by using the encoder reconstruction error as a measure of class attribution, the sample data blocks can be effectively classified, ensuring that the classification model performs well in both the rejection of unknown categories and the recognition of known categories, effectively improving the robustness of the classification. By introducing placeholders for unknown categories in contrastive prototype learning, the risk of overlap between known and unknown categories in the feature space is significantly reduced. At the same time, unlike contrastive learning based on reciprocity points, the present application allows known categories to be arranged in layers with placeholders as the center of the circle, optimizing the utilization of the feature space. It can still maintain fast convergence characteristics when there are a large number of categories, effectively improving the convergence efficiency and stability of the overall algorithm.

[0020] In a second aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a hyperspectral open set classification method combining feature reconstruction and prototype constraints as described in the first aspect above.

[0021] In a third aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement a hyperspectral open set classification method combining feature reconstruction and prototype constraints as described in the first aspect above.

[0022] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 is a flow chart of a hyperspectral open set classification method combining feature reconstruction and prototype constraints provided in one embodiment of the present application; Figure 2 is a schematic diagram of a data preprocessing process provided in one embodiment of the present application; Figure 3 is a structural diagram of a classification model provided in one embodiment of the present application; Figure 4 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application; Figure 5 is a schematic diagram of the structure of a decoder provided in one embodiment of the present application; Figure 6 It is a structural schematic diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. In the various embodiments of the present application, many technical details are proposed in order to make the reader better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is only for the convenience of description, and the specific implementation mode of the present application should not constitute any limitation. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0026] In recent years, researchers at home and abroad have proposed some hyperspectral open set classification methods based on deep learning, which are mainly divided into three types: methods using confidence, methods using prototype distance, and methods using reconstruction loss.

[0027] Liu et al. proposed a hyperspectral open set classification method based on a deep network. The algorithm assumes that there are relatively consistent activation vectors for most categories. The average activation vector of each known class sample is obtained according to the theory of the nearest class average. During testing, the output vector of the penultimate layer of the network is updated according to the distance between the sample activation vector and the average activation vector of each known class to adapt to the open set classification environment. However, this method fails to effectively consider the second-order and higher-order statistical characteristics, and is not suitable for data that conforms to the long-tail distribution, which will lead to inaccurate modeling of the Weibull distribution model.

[0028] Xie et al. proposed a prototype network based on feature consistency for hyperspectral open set classification. This method designed a three-layer convolutional network to extract discriminative features, introduced a contrastive clustering module to enhance discriminability, and constructed a prototype set of extracted features. Finally, a prototype-guided open set module was designed to identify known and unknown samples. This method constructed prototype sets of all categories from the extracted features. However, the intra-class features were compressed to a limited point during the prototype learning process, which may cause the model to filter out some necessary information that helps to distinguish unknown categories.

[0029] Zhou et al. designed a spectral spatial latent reconstruction method that enhances the learned feature representation by reconstructing the spectral and spatial features of the hyperspectral spectrum, thereby retaining the spectral spatial information that helps reject unknown categories and distinguish known categories, and using the feature-level reconstruction errors of spectral and spatial features to distinguish unknown categories. However, because the reconstruction method is only based on known sample modeling, it cannot fully recognize the complex and diverse information of unknown categories, and is very dependent on the quality of the training set when dealing with open set problems.

[0030] The inventors of the present application have discovered that the currently proposed hyperspectral open set classification methods have the following problems.

[0031] First, the spatial-spectral joint characteristics of hyperspectral images determine that they need to pay attention to the characteristics of both spatial and channel dimensions in classification tasks. However, traditional convolutional neural networks are limited by fixed receptive fields when extracting features and cannot effectively capture significant features across dimensions. At the same time, the feature extraction modules commonly used in traditional convolutional neural networks cannot fully distinguish between relevant and irrelevant features, resulting in the model generating a large number of redundant feature maps, which weakens the classification performance.

[0032] Second, in practical applications, the acquisition scenarios of hyperspectral images are complex and diverse, resulting in inconsistent distributions of training data and test data. For example, the spectral characteristics of ground objects vary due to changes in imaging conditions or acquisition by different sensors. This distribution inconsistency makes it difficult for existing open set recognition methods that rely on the distribution of training data to accurately model unknown class samples. In particular, when using outliers for unknown class recognition, the model is prone to misclassifying abnormal samples of known classes as unknown classes or ignoring some truly unknown class samples. In addition, traditional outlier detection methods are usually based on the feature distribution of known categories in the training data, ignoring the potential complexity of hyperspectral images and the diversity of unknown class samples, resulting in the model showing high uncertainty when facing unseen data.

[0033] Third, in the process of hyperspectral classification, the task bias of neural networks and the imbalance of training data distribution often lead to the classifier over-compressing intra-class features while pursuing high accuracy. This phenomenon will cause the model to ignore some key information that can distinguish unknown categories, especially under the complex interaction of spectral and spatial features in hyperspectral images. This problem is more prominent. In addition, the imbalanced data distribution will cause the network to tend to the high-frequency features of known categories during training, thereby weakening the model's sensitivity to unknown categories and ultimately affecting the robustness of classification.

[0034] In order to solve the above technical problems, an embodiment of the present application proposes a hyperspectral open set classification method combining feature reconstruction and prototype constraints, which is applied to an electronic device, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is explained by taking the server as an example. The implementation details of the hyperspectral open set classification method combining feature reconstruction and prototype constraints proposed in this embodiment are specifically described below. The following content is only the implementation details provided for the convenience of understanding, and is not necessary for the implementation of this solution.

[0035] The specific process of the hyperspectral open set classification method combining feature reconstruction and prototype constraint proposed in this embodiment can be as follows: Figure 1 As shown, including: Step 101 , divide each original sample data in the original sample data set into blocks to obtain a training set consisting of a plurality of sample data blocks, and use the label of the top-most center pixel of the sample data block as the label of the entire sample data block.

[0036] In the specific implementation, before the server performs model training, it is necessary to divide the original sample data in the original sample data set into blocks, thereby obtaining a training set consisting of several sample data blocks. It should be noted that the original sample data in the original sample data set is specifically a hyperspectral image data cube. The data block operation is designed based on the characteristics of the spatial-spectral union of hyperspectral data. The separated sample data blocks exist in the form of three-dimensional data cubes. After completing the data block division, the server also needs to use the label of the top-most center pixel of the sample data block as the label of the entire sample data block. This label is used to characterize the category to which the sample data block belongs.

[0037] In an example, the shape of the original sample data is Figure 2 As shown, the shape of the original sample data in the original sample data set is , is the number of rows of pixels in the original sample data, is the number of columns of pixels in the original sample data, is the number of bands of the original sample data. Each pixel of the original sample data is labeled with a label representing a category. The server divides the data into groups according to the preset classification standard (i.e. and ) Divide each original sample data in the original sample data set into blocks, divide several sample data blocks from each original sample data, form a training set based on all sample data blocks, and use the label of the top-level center pixel of the sample data block as the label of the entire sample data block to characterize the category to which the sample data block belongs. It should be noted that the shape of each sample data block is the same; the shape of each sample data block is , , .

[0038] In one example, when constructing a training set, the server also needs to construct a test set for performance testing of the trained classification model.

[0039] Step 102, input the sample data block into the classification model composed of an encoder, a decoder, a prototype classification module and an unknown category detection module, use the encoder to extract features of the sample data block, obtain the corresponding feature vector, and calculate the mean of each known category based on the feature vector, construct a prototype library, and build a comparative prototype loss based on the prototype library.

[0040] In the specific implementation, the server needs to build a classification model consisting of an encoder, a decoder, a prototype classification module, and an unknown category detection module in advance. After obtaining the training set, the sample data block can be input into the training model. The sample data block first enters the encoder of the classification model. The encoder extracts features from the sample data block to obtain the corresponding feature vector. At the same time, the mean of each known category is calculated based on the feature vector, a prototype library is constructed, and a comparative prototype loss for model training is constructed based on the prototype library.

[0041] In an example, the specific structure of the classification model can be as follows Figure 3 shown.

[0042] In one example, the specific structure of the encoder can be as follows Figure 4 As shown in the figure, it consists of multiple convolutional layers and global average pooling layers, which is used to extract spatial and spectral features in hyperspectral images.

[0043] First, the encoder concatenates the sample data blocks together through two parallel convolutions to obtain the first intermediate feature , the first intermediate feature It is expressed by the formula: ; in, represents a sample data block, and represents two parallel convolutions, Indicates splicing.

[0044] Then, the encoder processes the first intermediate feature After adding the residual module of the channel attention mechanism, the second intermediate feature is obtained , the second intermediate feature It is expressed by the formula: ; in, and Both represent convolution, Represents the channel attention mechanism.

[0045] After that, the encoder converts the second intermediate feature After the joint weighting of channel attention and spatial attention, the third intermediate feature is obtained , the third intermediate feature It is expressed by the formula: ; in, represents the sigmoid activation function, and represents a multi-layer perceptron, represents the ReLU activation function, Represents the second intermediate feature Medium pixel The value of represents convolution, Indicates the total number of channels, Represents the first channels, Represents a vector product operation.

[0046] Finally, the encoder processes the third intermediate feature Perform average pooling to get the corresponding feature vector .

[0047] In one example, in addition to feature extraction, the encoder is also responsible for constructing a prototype library. The encoder calculates the mean of each known category based on the feature vector and constructs the prototype library, which can be achieved by the following formula: ; ; ; in, represents the total number of known categories, Indicates Prototypes corresponding to known categories, A prototype library representing the construction, Indicates that it belongs to The total number of feature vectors of known categories, Indicates Belongs to feature vectors of known categories, Represents an encoder.

[0048] After the classification is completed, the encoder will select the feature vectors with confidence less than the preset confidence threshold in the classification results and calculate the mean as the placeholder for the unknown category. The placeholder for the unknown category is expressed by the formula: ; in, Indicates the total number of feature vectors whose confidence is less than the preset confidence threshold, Indicates feature vectors whose confidence is less than the preset confidence threshold, A placeholder for an unknown category.

[0049] Every The update weight is set to , .

[0050] After that, the encoder constructs the contrastive prototype loss based on the prototype library for model training. The purpose of the contrastive prototype loss is to encourage the classification model to learn semantic information related to known categories and minimize the overlap of features between classes. The constructed contrastive prototype loss can be expressed by the formula: The comparison prototype loss is constructed based on the prototype library and is implemented by the following formula: ; ;

[0051] ; in, express and The Euclidean distance between Indicates the maximum value of the distance between prototypes in the prototype library. Indicates of known categories The distance between the feature vector and each prototype in the prototype library, Represents the constructed comparison prototype loss.

[0052] By imposing a normalization constraint on the feature vectors, the encoder is better able to extract compact and representative feature representations. These features are then passed to the decoder for reconstruction verification and to the prototype classification module for classification and subsequent open set recognition.

[0053] Step 103: Use the decoder to perform feature reconstruction based on the feature vector to obtain a reconstructed sample data block, and construct a reconstruction error loss based on the sample data block and the reconstructed sample data block.

[0054] In a specific implementation, the encoder inputs the feature vector into the decoder, and the decoder performs feature reconstruction based on the feature vector to obtain a reconstructed sample data block, and constructs a reconstruction error loss based on the sample data block and the reconstructed sample data block.

[0055] In one example, the specific structure of the encoder can be as follows Figure 5As shown in the figure, the decoder adopts a deconvolution structure symmetrical to the encoder. Its main purpose is to verify the quality of the features extracted by the encoder by reconstructing the input data, while improving the classification model's ability to recognize unknown categories. The decoder gradually restores the high-dimensional feature vector to data consistent with the shape of the input hyperspectral cube. By comparing the shallow features of the reconstructed samples output by the decoder with the shallow features of the input samples, the reconstruction error can be calculated. The decoder constructs the reconstruction error loss based on the sample data block and the reconstructed sample data block, which can be achieved by the following formula: ; in, Indicates the number of sample data blocks input for each iteration, Indicates the first A shallow semantic representation, The decoder reconstructs A shallow semantic representation, Represents the reconstruction error loss obtained by construction.

[0056] In this way, back propagation adjusts the weights of the encoder and decoder to ensure that the reconstructed samples are as close to the input samples as possible. Since the decoder is only trained on known categories, when the input belongs to samples of unknown categories, the model cannot effectively reconstruct, resulting in a large reconstruction error. Therefore, the reconstruction error can be used as one of the bases for detecting unknown categories.

[0057] Step 104, using the prototype classification module to perform closed-set classification based on the feature vector and the prototype library, obtain a closed-set classification result for the sample data block, and construct a cross-entropy loss based on the closed-set classification result and the label of the sample data block.

[0058] In the specific implementation, the prototype classification module will perform closed-set classification based on the feature vector and the prototype library to obtain the closed-set classification result of the sample data block, and construct the cross-entropy loss based on the closed-set classification result and the label of the sample data block.

[0059] In one example, the prototype classification module needs to calculate the Euclidean distance between the feature vector and each prototype in the prototype library, and then select the category corresponding to the prototype with the smallest Euclidean distance as the closed set classification result of the sample data block.

[0060] In an example, the prototype classification module constructs a cross entropy loss based on the closed set classification results and the labels of the sample data blocks, which can be implemented by the following formula: ; in, represents the closed set classification result, Indicates the label, Represents the constructed cross entropy loss.

[0061] Step 105, constructing a total loss function based on the comparison prototype loss, the reconstruction error loss and the cross entropy loss, and iteratively training the classification model based on the total loss function until convergence, to obtain a trained classification model.

[0062] In the specific implementation, after obtaining the contrast prototype loss, reconstruction error loss and cross entropy loss, the server can construct a total loss function based on the contrast prototype loss, reconstruction error loss and cross entropy loss, and iteratively train the classification model based on the total loss function until convergence to obtain a trained classification model.

[0063] In one example, the server constructs a total loss function based on contrastive prototype loss, reconstruction error loss, and cross entropy loss, which is implemented by the following formula: ; in, , and is the preset weight parameter, is the total loss function constructed.

[0064] Step 106, using the unknown category detection module of the trained classification model, based on the sample feature distribution and reconstruction error loss of the training set, perform open set recognition, combine the prototype distance and reconstruction loss to evaluate the possibility score of the unknown category, analyze the possibility scores of all sample data blocks in the training set, use Weibull distribution to model the tail, use generalized Pareto distribution to approximate its cumulative distribution function, and determine that the first 50% of the tail of the entire cumulative distribution function is the unknown category.

[0065] In the specific implementation, the classification model uses the prototype distance to realize the classification of known categories, and combines the prototype distance and reconstruction loss to jointly identify unknown categories. The server uses the unknown category detection module of the trained classification model to perform open set recognition based on the sample feature distribution and reconstruction error loss of the training set, and combines the prototype distance and reconstruction loss to evaluate the possibility score of the unknown category, analyze the possibility scores of all sample data blocks in the training set, use Weibull distribution to model the tail, use generalized Pareto distribution to approximate its cumulative distribution function, and determine that the first 50% of the tail of the entire cumulative distribution function is unknown category.

[0066] In one example, the likelihood score for an unknown class is constructed as follows: ; in, is the preset assessment adjustment factor, Represents the probability that the sample data block belongs to each known category, The probability that a sample data block belongs to the unknown class.

[0067] In an example, analyzing the likelihood scores of all sample data blocks in the training set, modeling the tail using the Weibull distribution, and approximating its cumulative distribution function using the generalized Pareto distribution can be achieved using the following formula: ; in, represents the Weibull distribution, represents the generalized Pareto distribution, and are all learnable parameters, and The value of is determined through model training.

[0068] In an example, determining the first 50% of the entire cumulative distribution function tail as unknown categories can be achieved by the following formula: ; in, Adjust the parameters for the preset mapping.

[0069] A hyperspectral open set classification method combining feature reconstruction and prototype constraints proposed in this embodiment chooses to reconstruct shallow semantic representation instead of high-dimensional original image data, which significantly reduces the computational complexity of the model and makes feature extraction more focused on key information for class distinction. Shallow semantic representation carries high-recognition features of known categories, while retaining sufficient distinguishing ability for the complexity of unknown categories, thereby effectively improving the classification efficiency and accuracy of the classification model in an open set environment. Based on the manifold representation generated by the encoder, efficient distinction of known categories is achieved by modeling each known category separately. In addition, by using the encoder reconstruction error as a measure of class attribution, the sample data blocks can be effectively classified, ensuring that the classification model can perform well in both the rejection of unknown categories and the recognition of known categories, effectively improving the robustness of the classification. By introducing placeholders for unknown categories in contrastive prototype learning, the risk of overlap between known and unknown categories in the feature space is significantly reduced. At the same time, unlike contrastive learning based on reciprocity points, the present application allows known categories to be arranged in layers with placeholders as the center of the circle, optimizing the utilization of the feature space. It can still maintain fast convergence characteristics when there are a large number of categories, effectively improving the convergence efficiency and stability of the overall algorithm.

[0070] The step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0071] Another embodiment of the present application provides an electronic device, whose specific structure is as follows: Figure 6 As shown, it includes: at least one processor 201; and a memory 202 that is communicatively connected to the at least one processor 201; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 so that the at least one processor 201 can execute a hyperspectral open set classification method combining feature reconstruction and prototype constraints as described in the above-mentioned method embodiments.

[0072] Among them, the memory and the processor can be connected in a bus manner, and the bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and will not be further described in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be one component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0073] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0074] Another embodiment of the present application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement a hyperspectral open set classification method combining feature reconstruction and prototype constraints as described in the above method embodiments.

[0075] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (such as a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0076] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A hyperspectral open set classification method combining feature reconstruction and prototype constraints, characterized in that: include: Each original sample data in the original sample data set is divided into blocks to obtain a training set consisting of several sample data blocks, and the label of the central pixel point at the top layer of the sample data block is used as the label of the entire sample data block; wherein the original sample data is a hyperspectral image data cube; The sample data block is input into the classification model composed of an encoder, a decoder, a prototype classification module and an unknown category detection module. The encoder is used to extract features from the sample data block to obtain the corresponding feature vector. At the same time, the mean of each known category is calculated based on the feature vector, a prototype library is constructed, and a comparative prototype loss is constructed based on the prototype library. Using a decoder to perform feature reconstruction based on the feature vector to obtain a reconstructed sample data block, and constructing a reconstruction error loss based on the sample data block and the reconstructed sample data block; The prototype classification module is used to perform closed-set classification based on the feature vector and the prototype library to obtain the closed-set classification result of the sample data block, and the cross entropy loss is constructed based on the closed-set classification result and the label of the sample data block; A total loss function is constructed based on the contrast prototype loss, reconstruction error loss, and cross entropy loss, and the classification model is iteratively trained until convergence based on the total loss function to obtain a trained classification model; The unknown category detection module of the trained classification model is used to perform open set recognition based on the sample feature distribution and reconstruction error loss of the training set. The possibility score of the unknown category is evaluated by combining the prototype distance and reconstruction loss. The possibility scores of all sample data blocks in the training set are analyzed, the tail is modeled using the Weibull distribution, and its cumulative distribution function is approximated by the generalized Pareto distribution. The first 50% of the tail of the entire cumulative distribution function is determined to be the unknown category.

2. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 1, characterized in that: The shape of the original sample data in the original sample data set is , is the number of rows of pixels in the original sample data, is the number of columns of pixels in the original sample data, is the number of bands of the original sample data. Each pixel of the original sample data is annotated with a label representing the category. According to the preset division standard, each original sample data in the original sample data set is divided into blocks, and several sample data blocks are divided from each original sample data. A training set is formed based on all sample data blocks, and the label of the top-level center pixel of the sample data block is used as the label of the entire sample data block; wherein the shape of each sample data block is , , .

3. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 1, characterized in that: The encoder consists of multiple convolutional layers and global average pooling layers; The encoder is used to extract features from the sample data block to obtain the corresponding feature vector, including: First, the sample data blocks are concatenated together through two parallel convolutions to obtain the first intermediate feature , the first intermediate feature It is expressed by the formula: ； in, represents a sample data block, and represents two parallel convolutions, Indicates splicing; Then for the first intermediate feature After adding the residual module of the channel attention mechanism, the second intermediate feature is obtained , the second intermediate feature It is expressed by the formula: ； in, and Both represent convolution, Represents the channel attention mechanism; Then the second intermediate feature After the joint weighting of channel attention and spatial attention, the third intermediate feature is obtained , the third intermediate feature It is expressed by the formula: ； in, represents the sigmoid activation function, and represents a multi-layer perceptron, represents the ReLU activation function, Represents the second intermediate feature Medium pixel The value of represents convolution, Indicates the total number of channels, Represents the first channels, Represents a vector product operation; Finally, the third intermediate feature Perform average pooling to obtain the corresponding feature vector .

4. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 3, characterized in that: Based on the feature vector, the mean of each known category is calculated and the prototype library is constructed, which is achieved through the following formula: ； ； in, represents the total number of known categories, Indicates Prototypes corresponding to known categories, A prototype library representing the construction, Indicates that it belongs to The total number of feature vectors of known categories, Indicates Belongs to feature vectors of known categories; Select the feature vectors with confidence less than the preset confidence threshold to calculate the mean as the placeholder of the unknown category. The placeholder of the unknown category is expressed by the formula: ； in, Indicates the total number of feature vectors whose confidence is less than the preset confidence threshold, Indicates feature vectors whose confidence is less than the preset confidence threshold, A placeholder for an unknown class; The comparison prototype loss is constructed based on the prototype library and is implemented by the following formula: ； ； ； in, express and The Euclidean distance between Indicates the maximum value of the distance between prototypes in the prototype library. Indicates of the known categories The distance between the feature vector and each prototype in the prototype library, Represents the constructed comparison prototype loss.

5. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 4, characterized in that: The decoder adopts a deconvolution structure symmetrical to the encoder; The reconstruction error loss is constructed based on the sample data block and the reconstructed sample data block, which is implemented by the following formula: ； in, Indicates the number of sample data blocks input for each iteration, Indicates the first A shallow semantic representation, The decoder reconstructs A shallow semantic representation, Represents the reconstruction error loss obtained by construction.

6. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 5, characterized in that: The prototype classification module is used to perform closed-set classification based on the feature vector and the prototype library to obtain the closed-set classification results of the sample data block, including: Calculate the Euclidean distance between the feature vector and each prototype in the prototype library; Select the category corresponding to the prototype with the smallest Euclidean distance as the closed set classification result of the sample data block; The cross entropy loss is constructed based on the closed set classification results and the labels of the sample data blocks, which is implemented by the following formula: ； in, represents the closed set classification result, Indicates the label, Represents the constructed cross entropy loss.

7. The hyperspectral open set classification method combining feature reconstruction and prototype constraint according to claim 6, characterized in that: The total loss function is constructed based on the contrast prototype loss, reconstruction error loss and cross entropy loss, which is implemented by the following formula: ； in, , and is the preset weight parameter, is the total loss function constructed.

8. A hyperspectral open set classification method combining feature reconstruction and prototype constraint according to any one of claims 4 to 7, characterized in that: The likelihood score for the unknown class is constructed as follows: ； in, is the preset assessment adjustment factor, Represents the probability that the sample data block belongs to each known category, The probability that a sample data block belongs to an unknown category; Analyze the likelihood scores of all sample data blocks in the training set, use Weibull distribution to model the tail, and use generalized Pareto distribution to approximate its cumulative distribution function, which is achieved through the following formula: ； in, represents the Weibull distribution, represents the generalized Pareto distribution, and are all learnable parameters, and The value of is determined through model training; The first 50% of the tail of the entire cumulative distribution function is determined as the unknown category, which is achieved by the following formula: ； in, Adjust the parameters for the preset mapping.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a hyperspectral open set classification method combining feature reconstruction and prototype constraints as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is possible to implement a hyperspectral open set classification method combining feature reconstruction and prototype constraints as claimed in any one of claims 1 to 8.

Images