Hyperspectral image open set classification method and device based on fractional domain information enhancement and hypersphere prototype learning strategy

Abstract

The invention discloses a hyperspectral image open set classification method and device based on fractional domain information enhancement and a hypersphere prototype learning strategy, and belongs to the technical field of hyperspectral image open set classification. In order to solve the problem that a high misclassification risk exists between a known category and an unknown category in an existing hyperspectral image classification method, the method comprises the following steps: firstly, obtaining fractional domain information of hyperspectral data based on weighted fractional Fourier transform, and then fusing the fractional domain information with spatial spectral domain information; deep feature extraction is carried out on the hyperspectral image through a double-branch network, a hypersphere prototype learning strategy is adopted, utilization of a measurement space is optimized, and features of known categories are restrained to be evenly distributed on a hypersphere; and carrying out identification based on a closed set classifier of a known category prototype, and meanwhile, realizing open set identification by utilizing a hypersphere prototype radius so as to obtain a final open set classification result.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image open-set classification, and specifically relates to a hyperspectral image open-set classification method and device. Background Art

[0002] Hyperspectral images contain rich spectral information and spatial information, and can accurately reveal the unique fingerprint effects of different surface coverings, and have important application values in many fields such as coastal wetland monitoring, crop planting structure analysis, and forestry resource management. Hyperspectral image classification technology analyzes the spectral information of each pixel in the image in detail and assigns an accurate class label to each pixel. In recent years, the research in the field of hyperspectral image classification has gradually focused on the open-set problem. Open-set classification not only needs to accurately classify known classes, but also needs to identify unknown samples not included in the training data during the test phase, as Figure 1 shown in (b). However, due to the insufficient utilization of the metric space in existing open-set methods, misclassification often occurs between known classes and unknown classes; moreover, in a complex open environment, relying on a single threshold strategy to identify unknown classes often has poor effects. Summary of the Invention

[0003] The present invention aims to solve the problem of high misclassification risk between known classes and unknown classes in existing hyperspectral image classification methods.

[0004] A hyperspectral image open-set classification method based on fractional-domain information enhancement and hypersphere prototype learning strategy, comprising:

[0005] S100. Applying a weighted fractional Fourier transform to the hyperspectral image X in the spectral dimension to obtain fractional-domain information f α ; at the same time, performing convolution processing on the hyperspectral image X to obtain shallow spatio-spectral features; splicing the hyperspectral image X, its fractional-domain information f α and shallow spatio-spectral features together along the channel dimension, denoted as hyperspectral data f A ;

[0006] S200. Transmitting f A to a spectral feature extraction network E spe and a spatial feature extraction network E spa respectively; the spectral feature extraction network E spe uses 2D convolutional residual blocks and channel attention modules to extract features to obtain enhanced spectral features; the spatial feature extraction network E spa uses 3D convolution and multi-scale spatial attention modules to extract features to obtain spatial features; splicing the spectral features extracted by the spectral feature extraction network and the features extracted by the spatial feature extraction network together, and performing average pooling to obtain the final output feature f d, where d is the feature dimension;

[0007] Feature f d After being processed by the closed-set classifier, the predicted probability under the closed-set condition is obtained;

[0008] S300. Based on the hypersphere prototype set P and the hypersphere prototype radius set R, perform open-set classification on the hyperspectral image;

[0009] The process of obtaining the hypersphere prototype set P and the hypersphere prototype radius set R includes:

[0010] S301. Initialize N equidistantly distributed points on the hypersphere as hypersphere prototypes. The problem of uniformly distributing on the unit hypersphere is simplified to dividing the unit circle into N equal-angle slices, and then the hypersphere prototype matrix W and N uniformly distributed points are obtained; Match the feature f d extracted by the network with these points on the sphere, and based on network training to optimize the matching process, determine the matching relationship between the class label and the hypersphere prototype;

[0011] S302. Feature f d After being processed by the closed-set classifier, the predicted probability under the closed-set condition is obtained; further, the mean of all feature vectors in the same class is determined as the hypersphere prototype of this class, and then a set P = {p1,...p N} of N hypersphere prototypes corresponding to the number of classes N is obtained; Calculate the Euclidean distance mean of all sample features within each class to their corresponding class hypersphere prototypes, and define it as the hypersphere prototype radius of this class, thus obtaining the radius set R = {r1,...,r N}; After the network model is trained, the hypersphere prototype set P and the hypersphere prototype radius set R are obtained;

[0012] The process of performing open-set classification on the hyperspectral image includes:

[0013] First, use the trained closed-set classifier to perform closed-set prediction on the test data, and perform preliminary classification on the test data. Calculate the difference Δ(·) between the maximum value and the second-largest value in the closed-set prediction probability. For a test sample t i , when Δ(t i ) is less than or equal to the threshold τ, it indicates that the sample is an uncertain class; Assign the class label predicted by the closed-set classification to the feature vector of the sample of the uncertain class, that is, the class with the highest predicted probability Calculate the Euclidean distance between the sample feature vector and the corresponding class hypersphere prototype ; If the value exceeds the hypersphere prototype radius of the class corresponding to the class label predicted by the classification prediction identify the pixel as an unknown class.

[0014] Furthermore, the problem of uniform distribution on the unit hypersphere is reduced to dividing the unit circle into N equally angled slices, and the process of obtaining the hypersphere prototype matrix W and N uniformly distributed points includes:

[0015] Introduce the Gaussian potential kernel function G t (u, v) estimates the uniform distribution of hypersphere prototypes on the high-dimensional hypersphere and is defined as where denotes "defined as", denotes the set of positive real numbers, i.e., the output of the Gaussian potential kernel function; u, v are input vectors and are located on S d ;

[0016] The uniform distribution of points on the hypersphere is achieved by optimizing the uniformity loss function, and the uniformity loss function is defined as:

[0017]

[0018] where w i 、w j are the hypersphere prototype vectors in the hypersphere prototype matrix and satisfy the independent and identically distributed condition. W is randomly initialized in the first round of training; P w represents the hypersphere prototype distribution on the hypersphere, and i.i.d. means independent and identically distributed;

[0019] Based on minimizing the uniformity loss function, N uniformly distributed points are obtained.

[0020] Furthermore, match the features f d extracted by the network with these points on the sphere, and optimize the matching process based on network training. During the process of optimizing the matching process based on network training, the total loss function L of network training = L ce +L sp , L ce is the cross-entropy loss of the closed-set classifier; is the hypersphere prototype loss, where represents the bi-directional mapping relationship from the class label to the hypersphere prototype; i represents the i-th sample, and n s represents the number of samples; this mapping supervises the within-class compactness by reducing the similarity between the sample feature f i ∈f d and its matching hypersphere prototype W; the mapping relationship between the feature f d and its corresponding hypersphere prototype is learned and determined through the hypersphere prototype loss function.

[0021] Furthermore, when applying the weighted fractional Fourier transform to the hyperspectral image X in the spectral dimension, the hyperspectral image pixel X iThe weighted fractional Fourier transform is expressed as follows:

[0022]

[0023] where Ψ is the kernel function of the WFrFT of order α, n is the channel index and must be an integer, and f i α is the frequency information of the pixel after the WFrFT of order α.

[0024] Furthermore, the kernel function is as follows:

[0025]

[0026] where u1 represents the fractional domain transformation plane, and ω l (α) represents the weighting coefficient of the WFrFT.

[0027] Furthermore, the weighting coefficient of the WFrFT is as follows:

[0028]

[0029] where l = 0, 1, 2, 3,

[0030] Furthermore, in the process of obtaining the shallow spatio-spectral features by convolving the hyperspectral image X, two layers of convolution are used for processing. The first layer uses a 1×1 convolution kernel to extract the shallow features of the spectral dimension of the data; the second layer uses a 3×3 convolution kernel to extract the shallow features of the spatial dimension of the data; and after each layer of convolution, there is a batch normalization BN and a ReLU activation function immediately following.

[0031] Furthermore, the spectral feature extraction network E spe extracts features using the residual block and the channel attention module as follows:

[0032] The input data first passes through the sequentially connected SRBs to capture the spectral features of different depths. SRBs are spectral residual blocks, and each SRB contains multiple layers of Conv2D, and there is a BN layer and a Relu activation function after each layer of Conv2D;

[0033] The outputs of the two SRBs are respectively passed through the CAM to obtain the attention weights of the features of different depths, obtaining the attention weights of the features of different depths; the attention weights of the features of different depths are summed, and multiplied by the output of the second SRB, and finally the enhanced spectral features are obtained;

[0034] Furthermore, the spatial feature extraction network E spa extracts features using 3D convolution and the multi-scale spatial attention module as follows:

[0035] First, it is processed through two layers of Conv3D, each followed by a BN layer and a Relu activation function. Then, SAM is introduced. SAM uses different-sized convolutional kernels through three parallel branches, with the convolution being Conv2D, and each Conv2D being followed by a BN layer and a Relu activation function. During the processing of the three parallel branches, first, the outputs of two parallel branches are multiplied, then passed through a softmax layer and multiplied with the output of the third parallel branch as the final output of SAM. Finally, the outputs of the two layers of Conv3D are fused with the output of SAM to obtain spatial features.

[0036] A hyperspectral image open-set device based on fractional-domain information enhancement and hypersphere prototype learning strategy. The device includes a processor and a memory, and at least one instruction is stored in the memory. The at least one instruction is loaded and executed by the processor to implement the hyperspectral image open-set classification according to any one of claims 1 to 9.

[0037] Beneficial effects:

[0038] The present invention aims to effectively improve the classification performance of the model by optimizing the utilization of the metric space, enhancing the separation between known classes, and minimizing the misclassification risk between known and unknown classes. The hyperspectral image open-set classification algorithm proposed by the present invention designs a frequency-space-spectrum information aggregation module at the data input stage, which uses the weighted fractional Fourier transform to fuse the fractional-domain information and the space-spectrum domain information of the data, significantly enhancing the discrimination ability of spectral features. At the feature extraction stage, a spectral and spatial dual-branch network is designed to effectively extract deep features containing rich class discrimination information. In addition, the hypersphere prototype learning strategy can make the features of known classes evenly distributed on the unit hypersphere, not only optimizing the utilization of the metric space to achieve the maximum separation between classes, but also improving the discrimination ability of the network for known classes. Finally, an open-set recognition module based on the prototypes of known classes and their prototype radii is designed at the inference stage to effectively identify unknown classes. Through experimental analysis, the method proposed by the present invention can obtain OA values of 0.9524, 0.9769, and 0.9503 on three datasets respectively. Description of the drawings

[0039] Figure 1 Schematic diagrams for closed-set classification and open-set classification, where (a) is for closed-set classification and (b) is for open-set classification.

[0040] Figure 2 Flowchart of the hyperspectral image open-set classification method based on fractional-domain information enhancement and hypersphere prototype learning strategy.

[0041] Figure 3 Schematic diagram of the spectral feature extraction network.

[0042] Figure 4 It is a schematic diagram of the spatial feature extraction network.

[0043] Figure 5 They are the false color image, true value image and classification result image of Dataset I, where (a) is the false color image, (b) is the true value image, and (c) is the classification result image.

[0044] Figure 6 They are the false color image, true value image and classification result image of Dataset II, where (a) is the false color image, (b) is the true value image, and (c) is the classification result image.

[0045] Figure 7 They are the false color image, true value image and classification result image of Dataset III, where (a) is the false color image, (b) is the true value image, and (c) is the classification result image. Detailed implementation manners

[0046] The present invention integrates the fractional domain information and the spatial-spectral domain information of hyperspectral data; the present invention also extracts deep features from hyperspectral images by building a dual-branch network, and adopts a hypersphere prototype learning strategy to optimize the utilization of the metric space and constrain the features of known classes to be evenly distributed on the hypersphere. In addition, the present invention also designs an open set recognition module based on the known class prototypes and their prototype radii to obtain the final open set classification result. The present invention will be further described below in conjunction with the specific implementation manners.

[0047] Detailed implementation manner 1: Combining Figure 1 To illustrate this implementation manner,

[0048] First, a frequency-spatial-spectral information aggregation module (FSSIA) is designed to obtain the fractional domain information of the hyperspectral image by using the weighted fractional Fourier transform and integrate it with the spatial-spectral domain information of the hyperspectral image; then, through the dual-branch feature extraction networks E spe and E spa , deep feature extraction is performed on the data integrated by FSSIA; through the hypersphere prototype learning strategy (HSPL), the deep features extracted by the network are matched with the separately trained and fixed uniformly distributed hypersphere prototypes, and the features of known classes are constrained to be evenly distributed on the hypersphere to optimize the utilization of the metric space; finally, an open set recognition module (OSR) based on the known class prototypes and their prototype radii is also designed, where the known class prototypes and prototype radii are saved after model training. By calculating the Euclidean distance between the feature vector of the data to be measured and the predicted known class prototypes and judging the size relationship between it and the corresponding prototype radius, it is determined whether the data to be measured belongs to an unknown class.

[0049] This implementation manner is an open set classification method for hyperspectral images based on fractional domain information enhancement and hypersphere prototype learning strategy, including:

[0050] S1. Integrate the fractional domain information and spatial-spectral domain information of the hyperspectral image:

[0051] Take the patch of the hyperspectral image as the input, and regard the class (ground object class) label value of the central pixel as the label of the entire patch. Let represent the input patch, where p×p and c represent the size of the patch and the number of channels of pixel X i respectively.

[0052] First, use the Frequency-Spatial-Spectral Information Aggregation (FSSIA) module to integrate the fractional domain information and the conventional spatial-spectral domain information of the hyperspectral image.

[0053] To obtain the fractional domain information f α of the hyperspectral image X, this method applies the weighted fractional Fourier transform in the spectral dimension. As an extension of the fractional Fourier transform, the weighted fractional Fourier transform (WFrFT) can capture key frequency information more effectively by weighting different frequency components. Specifically, the weighted fractional Fourier transform of the input hyperspectral image pixel X i is expressed as follows:

[0054]

[0055] where Ψ is the kernel function of the WFrFT of order α, n is the channel index and must be an integer, and f i α is the frequency information of the pixel after the WFrFT of order α.

[0056] The definition of the kernel function is as follows:

[0057]

[0058] where u1 represents the fractional domain transformation plane, and ω l (α) represents the weighting coefficient of the WFrFT, and its definition is as follows:

[0059]

[0060] where l = 0, 1, 2, 3,

[0061] Meanwhile, the hyperspectral image X is processed through two layers of convolution to obtain shallow spatio-spectral features: the first layer uses a 1×1 convolution kernel to extract shallow features in the spectral dimension of the data; the second layer uses a 3×3 convolution kernel to extract shallow features in the spatial dimension of the data. To enhance the non-linear expression ability of the network, batch normalization BN and ReLU activation function are followed after each layer of convolution.

[0062] Finally, the hyperspectral image X, its fractional domain information f α and the shallow spatio-spectral features are concatenated together along the channel dimension, denoted as the hyperspectral data f integrated by FSSIA A , which will be jointly input into the dual-branch feature extraction network subsequently.

[0063] S2. The dual-branch network extracts deep features of the hyperspectral image:

[0064] To extract deep features containing richer class discriminant information, f A is respectively transmitted to the spectral feature extraction network E spe and the spatial feature extraction network E spa .

[0065] The spectral feature extraction network E spe contains two spectral residual blocks and a channel attention CAM module, as Figure 3 shown. The input data first passes through sequentially connected SRBs to capture spectral features at different depths. SRBs are spectral residual blocks, and each SRB contains multiple layers of Conv2D, and each layer of Conv2D is followed by a BN layer and a Relu activation function; SRBs can retain low-level feature information; two layers of Conv2D are adopted in this embodiment. Then, the outputs of the two SRBs are respectively passed through CAM to obtain the attention weights of features at different depths, and the attention weights of features at different depths are obtained. To enhance the multi-level feature representation ability of the network, the attention weights of features at different depths are summed and multiplied by the output of the second SRB, and finally enhanced spectral features are obtained.

[0066] The spatial feature extraction network E spa contains two layers of Conv3D (3D convolution) and a multi-scale spatial attention SAM module, as Figure 4 shown. First, in the input hyperspectral data f AInsert dimension 1 after the channel dimension of to adapt to the input requirements of Conv3D; two layers of Conv3D can capture its local correlations in the spatial and spectral dimensions, and each Conv3D is followed by a BN layer (3D) and a Relu activation function. After the Conv3D processing is completed, the output feature map is reshaped to match the input format of Conv2D. Then, SAM is introduced. SAM uses different-sized convolutional kernels through three parallel branches, and the convolution is Conv2D (2D convolution), and each Conv2D is followed by a BN layer and a Relu activation function to capture features with different receptive fields, enabling the model to focus on the spatial regions important for the classification task. During the processing of the three parallel branches, first multiply the outputs of two parallel branches, and then multiply the result by the output of the third parallel branch after passing through the softmax layer as the final output of SAM. Finally, fuse the outputs of the two layers of Conv3D and the output of SAM, which not only retains the global information of the original features but also incorporates the local details of multi-scale attention, obtaining a more effective feature representation.

[0067] Then, concatenate the spectral features extracted by the spectral feature extraction network with the features extracted by the spatial feature extraction network, and obtain the final output feature f after average pooling. d , where d is the feature dimension.

[0068] Feature f d During the training process and actual classification, it will pass through a closed-set classifier to obtain the predicted probability under closed-set conditions.

[0069] S3. Optimize the metric space using the hyperspherical prototype learning strategy:

[0070] As Figure 2 shown, this method uses the hyperspherical prototype learning (HSPL) strategy to evenly distribute the features of known classes on the hypersphere, which improves the utilization rate of the metric space to a certain extent. By making full use of the geometric properties of the hypersphere, HSPL can strengthen the compactness of intra-class features while maintaining the inter-class separability of known classes. Different from previous prototype learning methods, this method first pre-defines prototypes with uniform distribution and no class information on the hypersphere, and then optimizes the matching between class information and hyperspherical prototypes during the network training process.

[0071] HSPL first initializes N equidistantly distributed points on the hypersphere as hyperspherical prototypes in a way independent of the training data and network architecture. For the problem of data dimension d = 2 and N ground object classes with prototypes evenly distributed on the unit hypersphere, it can be simplified to dividing the unit circle into N equal angles (each angle is Slices of (); however, when d > 3, such an optimal uniform distribution solution cannot be used for processing. Therefore, the Gaussian potential kernel function G is introduced t (u, v) is used to estimate the uniform distribution of prototypes on the high-dimensional hypersphere and is defined as where denotes "defined as", denotes the set of positive real numbers, i.e., the output of the Gaussian potential kernel function.

[0072] The expression of the Gaussian potential kernel function is as follows:

[0073]

[0074] where t is the bandwidth parameter, and S d denotes the unit hypersphere in d dimensions, and u, v are input vectors located on S d ; the Gaussian potential kernel function G t (u, v) can measure the similarity between u and v, and the larger the value, the more similar they are.

[0075] Then, the uniform distribution of points on the hypersphere can be achieved by optimizing the uniformity loss function, which is defined as:

[0076]

[0077] where w i , w j are the prototype vectors in the prototype matrix and satisfy the independent and identically distributed condition. W is randomly initialized in the first round of training; P w represents the prototype distribution on the hypersphere, and i.i.d. means independent and identically distributed. It should be noted that the prototype distribution on the hypersphere does not depend on the specific training data and network architecture, but only on the dimension d of the metric space and the number of classes N.

[0078] Solving based on minimizing formula (5) can obtain N uniformly distributed points.

[0079] The uniformity loss can obtain N uniformly distributed points on the hypersphere (these points have no semantic information of classes, just uniformly distributed). By optimizing function (6) during network training, that is, matching the features (f with class label information d ) extracted by the network with these points on the sphere, and in each training process, the class represented by each point on the sphere may not be the same. Network training is to optimize this matching process to achieve the purpose of uniformly distributing the features of known classes on the hypersphere. After obtaining the solution of the hypersphere prototype, HSPL adopts a dynamic matching strategy based on class labels to hypersphere prototypes to align the information of known classes with the corresponding hypersphere prototypes. The features f output by the networkd The mapping relationship with its corresponding prototype can be learned and determined through the hypersphere prototype loss function; the hypersphere prototype loss is as follows:

[0080]

[0081] Among them, represents the bi-directional mapping relationship from the class label to the prototype; i represents the i-th sample, and n s represents the number of samples. This mapping supervises the within-class compactness by reducing the similarity between the sample feature f i ∈ f d and its matching prototype W.

[0082] Actually, the hypersphere prototypes trained separately have no semantic information and are just uniformly distributed points. Let's assume they are 1, 2, 3, 4, etc. respectively; then the class labels are grass, tree, water, road, etc. In the first training, maybe tree corresponds to 1, grass corresponds to 2, water corresponds to 3, and road corresponds to 4; in the second training, maybe grass corresponds to 1, tree corresponds to 2, road corresponds to 3, and water corresponds to 4. The matching relationship shown in formula (6) is to make such a match.

[0083] In addition, in order to ensure the performance of the closed-set classifier, the cross-entropy loss is also introduced during the training of the entire network as follows:

[0084]

[0085] Among them, y i,j represents the label that the i-th sample truly belongs to the j-th known class, while p i,j is the probability that the classifier predicts that the i-th sample belongs to the j-th class.

[0086] Therefore, the total loss function of the entire network can be expressed as:

[0087] L = L ce + L sp (8)

[0088] Based on the total loss function L, the training of the overall network model is realized. Using fixed hypersphere prototypes to assist the classification loss not only avoids the need to recalculate the class prototypes during each training process, but also can obtain more compact and representative known class prototypes.

[0089] S4. Unknown class recognition based on known class prototypes and their prototype radii:

[0090] Furthermore, the mean of all feature vectors in the same class is determined as the prototype of this class, and then a set P = {p1,... p N}. Meanwhile, calculate the mean Euclidean distance from the features of all samples within each class (category) to their corresponding class prototypes, and define it as the prototype radius of that class, thus obtaining the radius set $R = \{r_1,..., r$ N}. Save the obtained prototype set $P$ and prototype radius set $R$ after training the network model for identifying unknown classes during the inference phase.

[0091] To overcome the limitation of relying solely on a single threshold to identify unknown classes, this method designs a new open-set recognition (OSR) module based on known class prototypes and their prototype radii, as Figure 2 shown. By the process of obtaining the radius set above, an adaptive-sized prototype radius is learned for each known class, which can more finely characterize the feature distribution of each known class, thereby improving the accuracy of the open-set recognition task. The processing procedure of the open-set recognition module is as follows:

[0092] First, use the trained closed-set classifier to perform closed-set prediction on the test data and make a preliminary inference about the identity of the test data. Considering the uncertain characteristics of the logit vector distribution of unknown classes, this method does not directly make a judgment based on the magnitude of the logit value, but quantifies this uncertainty by calculating the difference $\Delta(\cdot)$ between the maximum and the second-largest values in the closed-set prediction probability. For a given test sample $t$ i ,

[0093]

[0094] where $\tau$ is a hyperparameter. To make the identity inference of the test sample strict, this method sets $\tau$ to 0.8.

[0095] If there is a large uncertainty in the prediction probability of the sample, it is classified into the uncertain set. Then, assign the class label of the closed-set classification prediction (the class with the highest predicted probability, i.e., ) to the feature vector of the sample in the uncertain set, and calculate the Euclidean distance between the sample feature vector $f$ i and the corresponding class prototype , denoted as If the value exceeds the prototype radius of the class corresponding to the class label of the classification prediction, it indicates that the sample has a low similarity to the predicted known class, and then the pixel is identified as an unknown class.

[0096] The present invention proposes an open - set classification method for hyperspectral images based on fractional - domain information enhancement and hypersphere prototype learning strategy. By applying the weighted fractional Fourier transform and the hypersphere prototype learning strategy, this method can fully exploit and utilize the distribution characteristics of known classes, not only effectively identifying unknown classes, but also ensuring excellent classification performance for other known classes.

[0097] Generally speaking, the complete algorithm of the present invention is shown in Table 1, where D L represents the training data, and D T represents the test data, and E represents the double - branch network.

[0098] Table 1

[0099]

[0100] Example:

[0101] The effectiveness of the open - set classification method for hyperspectral images based on fractional - domain information enhancement and hypersphere prototype learning strategy proposed by the present invention is illustrated using three commonly used datasets. The detailed information of the three datasets used is listed in Table 2. The experimental results use the overall accuracy (OA), per - class accuracy, average accuracy (AA), and Kappa coefficient as evaluation metrics. The higher the values of all evaluation metrics, the better the classification effect.

[0102] Table 2 Detailed information of the hyperspectral images used

[0103]

[0104] For different data, the optimal parameter settings of the method of the present invention are shown in Table 3. The parameter α is the order of the weighted fractional Fourier transform, and τ is the threshold of the closed - set prediction probability. The algorithm of the present invention is implemented using the PyTorch framework in Python 3.7; all experiments are carried out on the same hardware platform: GTX - 3090 GPU, Intel 4210R CPU, 40GB memory. In the network structure, the patch size is set to 9×9, and the embedding dimension number is 32. In addition, to improve the training efficiency, the batch size is set to 512, and the number of epochs is set to 200. It should be noted that the algorithm of the present invention is optimized through two independent steps: first, fixed and uniformly distributed prototype points are obtained on the hypersphere using formula (5), and then the optimal model parameters are solved by optimizing formula (9) based on these prototype points.

[0105] Table 3 Optimal parameters and evaluation metric values on three groups of experimental data

[0106]

[0107] The classification result graphs of Datasets I, II, and III are asFigures 5 - 7 as shown Embodiment 2 of the specific implementation method:

[0109] This embodiment is a hyperspectral image open-set device based on fractional-domain information enhancement and hypersphere prototype learning strategy. The device includes a processor and a memory. It should be understood that any device including a processor and a memory described in the present invention may also include other units and modules for display, interaction, processing, control, etc. through signals or instructions, as well as other functions;

[0110] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the hyperspectral image open-set classification based on fractional-domain information enhancement and hypersphere prototype learning strategy.

[0111] Those skilled in the art should understand that the stored at least one instruction is a computer program product corresponding to the method or system. It should be understood that the instruction includes any computer program product, software, or computerized method corresponding to the method described in the present invention; the instruction can be used to program a computer system or other electronic devices. The computer storage medium may include a readable medium having instructions stored thereon, which may include, but are not limited to, magnetic storage media and optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM), and flash memory layers, or other types of media suitable for storing electronic instructions.

[0112] Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of methods, systems, and computer program products according to the embodiments of the present application, and can also be used for corresponding devices. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, as well as the combination of processes and / or blocks in the flowchart and / or block diagram. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing in the process Figure 1 one process or multiple processes and / or blocks Figure 1a device for the functions specified in one or more boxes.

[0114] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 a box or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 one or more processes and / or boxes Figure 1 a box or more boxes.

[0116] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0117] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

[0118] The above numerical examples of the present invention are only for explaining in detail the calculation model and calculation process of the present invention, rather than limiting the implementation manner of the present invention. For those of ordinary skill in the art, other different forms of changes or variations can be made based on the above description. It is impossible to enumerate all the implementation manners here. Any obvious changes or variations derived from the technical solutions of the present invention still fall within the protection scope of the present invention.

Claims

1. A hyperspectral image open-set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy, characterized in that Including: S100. Apply the weighted fractional Fourier transform to the hyperspectral image X in the spectral dimension to obtain fractional domain information f α ; At the same time, perform convolution processing on the hyperspectral image X to obtain shallow spatio-spectral features; The hyperspectral image X, its fractional domain information f α and the shallow spatial-spectral features are concatenated along the channel dimension, denoted as the hyperspectral data f A ; S200. Transmit f A to the spectral feature extraction network E spe and the spatial feature extraction network E spa respectively; The spectral feature extraction network E spe extracts features using 2D convolutional residual blocks and channel attention modules to obtain enhanced spectral features; The spatial feature extraction network E spa extracts features using 3D convolution and multi-scale spatial attention modules to obtain spatial features; The spectral features extracted by the spectral feature extraction network are concatenated with the features extracted by the spatial feature extraction network, and the final output feature f d is obtained after average pooling, where d is the feature dimension; Feature f d After being processed by a closed-set classifier, the predicted probability under closed-set conditions is obtained; S300, perform open-set classification of hyperspectral images based on the hypersphere prototype set P and the hypersphere prototype radius set R; The process of obtaining the hypersphere prototype set P and the hypersphere prototype radius set R includes: S301. Initialize N points evenly distributed on the hypersphere as hypersphere prototypes. The problem of evenly distributing on the unit hypersphere is simplified to dividing the unit circle into N equal - angle slices, and then obtaining the hypersphere prototype matrix W and N evenly - distributed points; match the features f d extracted by the network with these points on the sphere, optimize the matching process based on network training, and determine the matching relationship between the class labels and the hypersphere prototypes; S302, Feature f d After being processed by the closed-set classifier, the predicted probability under the closed-set condition is obtained; further, the mean value of all feature vectors in the same category is determined as the hypersphere prototype of this category, and then a set P = {p1,...p N} of N hypersphere prototypes corresponding to the number of categories N is obtained; calculate the mean Euclidean distance from all sample features within each class to its corresponding class hypersphere prototype, and define it as the hypersphere prototype radius of this class, so as to obtain the radius set R = {r1,...,r N}; After training the network model, the hypersphere prototype set P and the hypersphere prototype radius set R are obtained; The process of performing open-set classification of hyperspectral images includes: First, use the trained closed-set classifier to perform closed-set prediction on the test data, and conduct a preliminary classification of the test data. Calculate the difference Δ(·) between the maximum value and the second-largest value in the closed-set prediction probability. For a test sample t i , when Δ(t i ) is less than or equal to the threshold τ, it indicates that the sample is of an uncertain category; assign the class label predicted by the closed-set classification to the feature vector of the sample of the uncertain category, that is, the category with the highest predicted probability Calculate the Euclidean distance between the sample feature vector and the corresponding class hypersphere prototype If the value exceeds the radius of the hypersphere prototype of the category corresponding to the class label predicted by the classification identify the pixel as an unknown category.​ 2. The hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to claim 1, characterized in that, The process of reducing the problem of uniformly distributing points on the unit hypersphere to dividing the unit circle into N equally angled slices, and then obtaining the hypersphere prototype matrix W and N uniformly distributed points includes: Introduce the Gaussian potential kernel function G t (u, v) estimates the uniform distribution of the hypersphere prototypes on the high-dimensional hypersphere and defines it as where denotes "defined as", represents the set of positive real numbers, i.e., the output of the Gaussian potential kernel function; u, v are input vectors and lie on S d ; The uniform distribution of points on the hypersphere is achieved by optimizing the uniformity loss function, and the uniformity loss function is defined as: where, w i and w j are the hypersphere prototype vectors in the hypersphere prototype matrix , and satisfy the independent and identically distributed conditions. W is randomly initialized in the first round of training; P w represents the hypersphere prototype distribution on the hypersphere, and i.i.d. represents independent and identically distributed; Solve for N uniformly distributed points based on minimizing the uniformity loss function.

3. The hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to claim 1, wherein Match the features f extracted from the network d with the points on these spheres, and optimize the matching process based on network training. During the optimization of the matching process based on network training, the total loss function L of network training is L = L ce + L sp , where L ce is the cross-entropy loss of the closed-set classifier; is the hypersphere prototype loss, where θ * M represents the bijective mapping relationship from the class label to the hypersphere prototype; i represents the i-th sample, and n s represents the number of samples; this mapping supervises the within-class compactness by reducing the similarity between the sample feature f i ∈ f d and its matching hypersphere prototype W; the mapping relationship between the feature f d and its corresponding hypersphere prototype is learned and determined through the hypersphere prototype loss function.

4. A hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 1 to 3, characterized in that, When applying the weighted fractional Fourier transform to the hyperspectral image X in the spectral dimension, the hyperspectral image pixel X i has the following expression of the weighted fractional Fourier transform: where Ψ is the kernel function of the α-order WFrFT, n is the channel index and must be an integer, and is the frequency information of the pixel after the α-order WFrFT.

5. A hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 4, characterized in that, The kernel function is as follows: Among them, u1 represents the fractional-domain transformation plane, and ω l (α) represents the weighting coefficient of the WFrFT.

6. The open-set classification method for hyperspectral images based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 5, characterized in that The weighting coefficients of WFrFT are as follows: where l = 0, 1, 2, 3, 7. A hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 1 to 3, characterized in that, In the process of obtaining shallow spatio-spectral features by convolving the hyperspectral image X, two layers of convolution are used for processing. The first layer uses a 1×1 convolution kernel to extract shallow features in the spectral dimension of the data; The second layer uses a 3×3 convolution kernel to extract shallow features in the spatial dimension of the data; and after each layer of convolution, batch normalization BN and ReLU activation function are immediately followed.

8. A hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 1 to 3, characterized in that, Spectral feature extraction network E spe The process of extracting features using residual blocks and channel attention modules is as follows: The input data first passes through sequentially connected SRBs to capture spectral features of different depths. SRBs are spectral residual blocks, and each SRB contains multiple layers of Conv2D, and each layer of Conv2D is followed by a BN layer and a Relu activation function; The outputs of two SRBs are respectively passed through CAM to obtain the attention weights of features of different depths, obtaining the attention weights of features of different depths; perform a summation operation on the attention weights of features of different depths, and multiply the result by the output of the second SRB to finally obtain enhanced spectral features.

9. A hyperspectral image open set classification method based on fractional domain information enhancement and hypersphere prototype learning strategy according to any one of claims 1 to 3, characterized in that, Spatial Feature Extraction Network E spa The process of extracting features using 3D convolution and multi-scale spatial attention modules is as follows: First, perform two layers of Conv3D processing, and each Conv3D is followed by a BN layer and a Relu activation function; then introduce SAM. SAM uses different-sized convolution kernels through three parallel branches, the convolution is Conv2D, and each Conv2D is followed by a BN layer and a Relu activation function; during the processing of the three parallel branches, first multiply the outputs of two parallel branches, then pass through a softmax layer and then multiply by the output of the third parallel branch as the final output of SAM; finally, fuse the outputs of two layers of Conv3D and the output of SAM to obtain spatial features.

10. A hyperspectral image open-set device based on fractional domain information enhancement and hypersphere prototype learning strategy, characterized in that, The device includes a processor and a memory. At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement an open-set classification of hyperspectral images based on fractional-domain information enhancement and hypersphere prototype learning strategy according to any one of claims 1 to 9.

Images