Hyperspectral image open set classification method based on multi-scale convolution and feature fusion

Abstract

The invention provides a hyperspectral image open set classification method based on multi-scale convolution and feature fusion, and aims to improve the classification precision of a known category and an unknown category in a hyperspectral image and the robustness of a classification model. According to the method, spectral and spatial features are fused through a deep learning model, and the problems of unknown category identification and feature optimization in hyperspectral image classification are solved. Firstly, spectral features are extracted through multi-scale convolution, and in combination with a channel attention mechanism, capture of most discriminative spectral information is enhanced. And secondly, through a multi-branch structure and a convolution enhancement technology, the expression of spectral features is optimized, and the perception capability of complex spatial information is improved. And the spatial features are enhanced through a high-frequency enhancement technology, so that the sensitivity to detail and edge information is improved. Through the innovative feature extraction and fusion strategy, the classification precision of the hyperspectral image open set and the robustness of the classification model are improved.

Description

Technical Field

[0001] The invention discloses a hyperspectral image open set classification method based on multi-scale convolution and feature fusion, which belongs to the field of image processing technology, and specifically belongs to the field of hyperspectral open set image classification technology. Background Art

[0002] The present invention proposes an open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, aiming to effectively improve the classification performance of hyperspectral images in complex scenes, especially for the problem of distinguishing known categories from unknown categories in open set classification. In order to capture the rich spectral and spatial features in hyperspectral images, the present invention introduces a multi-scale convolution structure to extract features from information of different scales. By applying deep convolution operations in the spectral dimension, key spectral features are extracted from multiple bands, thereby improving the recognition ability of complex objects. In addition, in order to further optimize the representation of spectral features, this patent adopts a multi-branch convolution structure, and convolution kernels of different sizes can extract features from different scales and fuse them, so that the model can perceive the complexity of hyperspectral images more comprehensively.

[0003] In recent years, the application of hyperspectral images in the field of remote sensing has attracted widespread attention and has gradually become one of the core research directions. This advanced imaging technology provides rich spectral information by capturing hundreds of continuous and narrow spectral bands. These spectral data can record the reflection or emission characteristics of objects in various spectral bands in detail, thereby revealing their complex physical, chemical and other unique properties. This detailed information dimension makes hyperspectral images show great potential in surface feature recognition, classification and potential attribute mining, injecting new vitality into remote sensing science. Due to the complexity and accuracy of its high-dimensional data, hyperspectral images have greatly improved the analysis depth and research level in the field of remote sensing. By analyzing the multidimensional spectral characteristics of hyperspectral images, scientists can not only accurately identify surface materials, but also reveal their potential physical and chemical characteristics, which is of great significance in environmental monitoring, agricultural resource assessment and geological exploration.

[0004] With the rise of deep learning, hyperspectral image classification technology has also developed rapidly. In recent years, a variety of deep learning methods such as generative adversarial networks, convolutional neural networks, and graph neural networks have been applied to the classification task of hyperspectral images. These methods have significantly improved the classification accuracy by deeply mining spectral and spatial features. However, most studies are based on the closed set assumption, that is, the training data and test data share the same category label space and completely cover all categories. Although this assumption is more common in laboratory environments, it is often not true in practical applications. Real test data may contain unknown categories that have never appeared in the training set, and methods based on the closed set assumption often misclassify these unknown categories as known categories, resulting in inaccurate results.

[0005] To solve this problem, researchers have gradually focused on open set classification methods, which can identify unknown categories and improve the adaptability of classification models in real scenarios. This method is particularly important for the classification task of hyperspectral remote sensing images, because it can not only improve the classification performance of known categories, but also effectively reduce the misclassification of unknown categories, thereby improving the robustness of the model in complex environments. By exploring the characteristic laws of unknown categories and combining the characteristics of deep learning, open set classification provides new possibilities and development directions for the widespread application of hyperspectral images.

[0006] This patent adopts a hyperspectral image open set classification method based on multi-scale convolution and feature fusion (Multi-Scale Convolution with Feature Fusion for Open-Set Classification, MSCF-OSC), which divides the hyperspectral data into n-1 known categories and 1 unknown category, where the known categories are divided into training sets and test sets according to a certain ratio, and the unknown categories are only used in the test phase. First, the training samples will pass through the spectral feature extraction module and the spatial feature extraction module to extract and optimize the spectral features and spatial features respectively. In the spectral feature extraction module, three-dimensional convolution, Squeeze-Efficiency Module (SE), spectral residual extraction module, Multi-Branch Block (MBB) and Convolution Enhancement Block (CEB) are used to gradually extract and enhance spectral features. These features are then sent to the spatial feature extraction module, which optimizes the spatial features through three-dimensional convolution, efficient channel attention module (ECA) and high-frequency enhancement module (HFE). At the same time, the spatial residual extraction module is used to further improve the expressiveness of features. The optimized spectral and spatial features are spliced ​​together and then passed to the fully connected layer for classification. In the classification process, contrastive learning and metric learning methods are introduced to improve the classification performance of the model. By optimizing the contrast loss, the ability to distinguish between categories is enhanced. At the same time, the strategy based on the distance threshold helps to identify unknown categories. Finally, the model outputs the classification results and significantly improves the overall performance. Summary of the invention

[0007] In view of the shortcomings of the prior art, the present invention proposes an open set classification method for hyperspectral images, which realizes the accurate distinction between known categories and unknown categories by combining feature extraction and optimization. According to the spectral and spatial characteristics of hyperspectral images, a multi-scale feature extraction framework is constructed to jointly model the complex spectral information and spatial structure. The extraction of spectral features focuses on the effective fusion of multi-band information, which enhances the recognition and expression ability of features, while the spatial features capture key edge and structural information through specific enhancement strategies to optimize the perception ability of the model. Through the fusion of multi-level features and the classification optimization mechanism, the model effectively improves the recognition ability of known categories, and enhances the adaptability and robustness to unknown categories, thereby significantly improving the overall performance of classification.

[0008] The hyperspectral image open set classification algorithm based on multi-scale convolution and feature fusion described in the present invention includes:

[0009] The data is divided into n-1 known categories and 1 unknown category. The known categories are divided into training sets and test sets in proportion, and the unknown categories are only used for testing. The training samples are sequentially input into the spectral feature extraction module and the spatial feature extraction module for feature extraction. The spectral feature extraction module gradually extracts and enhances the spectral features by combining three-dimensional convolution, Squeeze-Efficiency Module (SE), spectral residual extraction module, Multi-Branch Block (MBB) and Convolution Enhancement Block (CEB); then the features are sent to the spatial feature extraction module to extract and optimize the spatial features by combining three-dimensional convolution, Efficient Channel Attention (ECA) and High-Frequency Enhancement (HFE), and then sent to the spatial residual extraction module for further processing to output the final spatial features. Subsequently, the extracted spectral and spatial features are spliced ​​and sent to the fully connected layer for classification, and the classification performance is optimized by combining contrastive learning and metric learning. Specifically, the contrastive loss is used to optimize the category distinction ability, and the unknown category is identified based on the distance threshold. Finally, the classification results are output to optimize the model performance.

[0010] Input the original hyperspectral data set to be classified. Divide the n categories in the data into n-1 known categories and 1 unknown category. Among them, the data of known categories are divided into training sets and test sets according to a certain ratio, while the data of unknown categories do not participate in training and are directly merged with the test set of known categories as the final test set.

[0011] The training samples are input into the spectral shallow feature extraction module for processing. This module uses a 7×7×7 three-dimensional convolution kernel and a 16-fold channel compression ratio SE attention mechanism to efficiently extract spectral features and key band information in hyperspectral images. The SE attention module extracts channel-level features through global average pooling, and then re-weights the importance of different bands through two layers of fully connected networks.

[0012] First, spectral features of different scales are extracted through a 3D convolutional layer. Through the operation of deep convolution kernels, the model can effectively fuse and extract spectral information of different bands. Then, SE is introduced to enhance the feature selection ability between channels, thereby optimizing the perception of spectral features. Through this stage, the network can focus on the most discriminative spectral information in the hyperspectral image, providing high-quality spectral features for the subsequent spatial feature extraction module.

[0013] In this step, the input hyperspectral image data is preprocessed and passed to the spectral shallow feature extraction module. This module mainly extracts spectral features through 3D convolution and inputs them into the SE module to enhance the spectral feature selection ability between channels. The spectral feature extraction process can be expressed by the following steps and formulas:

[0014] 3D convolution operation: The input hyperspectral data is X, where B is the batch size, C is the number of channels, and H and W are the height and width of the image, respectively. Next, a 3D convolution operation is performed for feature extraction, and the output features are, where is the number of output channels, and is the image size after convolution.

[0015] The convolution operation formula is:

[0016]

[0017] After the convolution operation, the SE module is used to optimize the perception of spectral features:

[0018]

[0019] Where W1 and W2 are the weights of the fully connected layer, δ is the ReLU activation function, and σ is the Sigmoid activation function.

[0020] The features extracted above are input into a multi-branch block consisting of 1×1 convolution and 3×3 convolution. This module extracts features at multiple spatial scales through convolution kernels of different sizes, further enhancing the model's perception of multi-scale information. 1×1 convolution focuses on extracting local fine-grained information, while 3×3 convolution captures spatial features in a wider range. Combining the characteristics of hyperspectral images, the multi-branch block captures fine-grained information between different bands through parallel processing while retaining the overall spatial structural characteristics. Finally, the features extracted by each branch are weighted fused to output features with stronger discrimination and expression capabilities.

[0021] The purpose of the multi-branch block is to capture the multi-scale spatial information of the input features through convolution operations of different scales. In the multi-branch structure, each branch uses a different convolution kernel, which enables the model to learn features at different scales and integrate these features by summing them.

[0022] For the input feature, the multi-branch block divides it into multiple branches, each branch is processed by a different convolution operation, and then they are summed.

[0023] Branch 1:

[0024]

[0025] O is the index of the output channel, k d ,k h ,k w is the offset of the convolution kernel in depth, height, and width. The convolution kernel size is 3×3×3.

[0026] Branch 2:

[0027]

[0028] W o,c,0,0,0 is the convolution kernel weight of 1×1×1.

[0029] Each convolution branch extracts features of different scales, and finally sums their outputs to obtain fused features. The multi-branch block extracts multi-scale spatial features through convolution kernels of different sizes, and sums or concatenates them, thereby enhancing the model's perception of spatial information.

[0030]

[0031] The features extracted by the multi-branch block are input into the convolution enhancement block consisting of 7×7 depthwise separable convolution and 1×1 pointwise convolution. The depthwise convolution performs convolution operations independently for each input channel to capture the local high-frequency detail features of the hyperspectral image; the pointwise convolution fuses information in the channel dimension through 1×1 convolution to enhance the feature expression ability between channels.

[0032] The convolution enhancement block performs high-frequency enhancement on features through deep convolution operations. It uses deep separable convolution technology to perform convolution operations on each input channel separately, thereby reducing the number of parameters and increasing the efficiency of the model. And it enhances the channel by using point convolution to improve the network's perception of high-frequency features (such as edges and textures). Let the input be x, and after deep convolution and point convolution, the output is out. The calculation process of the convolution enhancement block is:

[0033] Depthwise convolution: Each input channel is convolved independently, the convolution kernel size is 7×7, and the output is x'.

[0034]

[0035] Y mid (h+i,w+j,d+1,c) is the value of the c-th channel located at (h+i,w+j,d+l) in the input feature map. d (i,j,l,c) is the weight of the depth convolution kernel, corresponding to the cth channel of the input feature. k is the size of the convolution kernel. D(h,w,d,c) is the output of the depth convolution.

[0036] Pointwise convolution 1: The number of channels is increased through 1×1 convolution, and the output is P.

[0037]

[0038] W1(c',c) is the weight of the point-by-point convolution kernel, which converts the input channel c to the output channel c'. C is the number of input channels, and c' is the index of the output channel.

[0039] Apply the GELU activation function to the output of the point-by-point convolution P(h,w,d,c'):

[0040] G(h,w,d,c′)=GELU(P(h,w,d,c′))

[0041] Point-by-point convolution 2: Restore the number of channels to the original input number of channels, and the output is O(h,w,d,c).

[0042]

[0043] W2(c,c') is the weight of point-wise convolution 2.

[0044] The obtained features are then input into the cascade structure of the double-layer spectral residual extraction module for further spectral feature extraction.

[0045] The convolution enhancement block enhances the sensitivity to high-frequency information (such as edges and textures) through a combination of deep convolution and point convolution, improving the model's ability to capture detailed features.

[0046] The extracted spectral features are input into the spatial feature extraction module, which extracts preliminary features through three-dimensional convolution and further enhances the spatial features by combining ECA and high-frequency enhancement modules. Subsequently, multi-level feature extraction is achieved through the joint operation of multiple two-dimensional residual extraction modules, SE modules and ECA modules.

[0047] Extract preliminary spatial-spectral joint features through 3D convolution:

[0048] O 3D =Conv3D(O)

[0049] The obtained features are then input into the cascade structure of the double-layer spectral residual extraction module for further spectral feature extraction.

[0050] Extraction of low-frequency parts: Extract low-frequency features through average pooling operation. Average pooling is an operation that downsamples the input feature map, which can retain the overall information of the image but lose details.

[0051]

[0052] X low is the input feature map, b is the batch index, c is the number of channels, h and w are the height and width, k is the size of the pooling kernel, and s is the step size.

[0053] Afterwards, the low-frequency features are restored to their original size by upsampling. The upsampling operation restores the low-frequency feature map to the same size as the input feature map so that it can be compared with the input. If the size of the feature map after upsampling is inconsistent with the input feature map, the size is adjusted by interpolation.

[0054]

[0055] H, W are the height and width of the original feature map, H', W' are the height and width of the low-frequency feature map after downsampling. α, β are the upsampling scaling factors, i.e., the input-output size ratio.

[0056] The high frequency part is obtained by the difference between the input image and the upsampled low frequency part:

[0057]

[0058] Multi-branch feature enhancement is performed on the high-frequency and low-frequency parts respectively: Finally, the enhanced high-frequency and low-frequency features are fused by addition, and the high-frequency enhanced feature is obtained as Y e .

[0059] The ResSPA module further processes the high-frequency enhanced features. The complete calculation process is as follows:

[0060] Y = LeakyReLU(Y e +BN(Conv2D2(ReLU(Conv2D1(Y e ))))))·W atten )

[0061] Y out =concat(Y+O)

[0062] W atten is the channel weight vector generated by SE and ECA attention mechanisms, which represents the weight between channels. BN is the batch normalization operation. Conv2D1 is the first layer of two-dimensional convolution operation, the convolution kernel size is 3×3, and the number of output channels is the same as the number of input channels. Conv2D2 is the second layer of two-dimensional convolution operation, the convolution kernel size is 3×3. X+… represents the residual connection, which adds the input features to the processed features. LeakyReLU is the final activation function to ensure nonlinear expression.

[0063] The classification of unknown classes and the calculation of loss functions are as follows: First, the anchor point of each class is defined as:

[0064]

[0065] in is the scaling factor, e i is the unit eigenvector of the i-th class.

[0066] Then calculate the Euclidean distance between the sample and the class anchor point, where z is the sample feature vector after feature extraction:

[0067] d(z,c i )=||zc i ||2

[0068] The distances from the sample to all class anchors are combined into a vector: d(z,C) = [d(z,c1),d(z,c2),d(z,c3),...,d(z,c n )].

[0069] Calculate the rejection score:

[0070]

[0071] Among them, in this formula represents element-wise multiplication, d i represents the i-th element in the vector, Indicates d i By negating and then exponentiating, smaller values ​​will get larger weights. It represents the sum of the exponential values ​​of all elements for normalization. If the minimum rejection score γ≤θ, the sample is classified as the corresponding known class; if the minimum rejection score γ>θ, the sample is classified as an unknown class.

[0072] The extracted spatial feature Y and the extracted spectral feature O are concatenated to obtain a joint feature F.

[0073] Compute the Euclidean distance:

[0074]

[0075] Among them, f i is the characteristic vector of the sample, c j is the anchor point of category j, and N is the feature dimension. Euclidean distance d ij It is used to measure the similarity between a sample and the category center. The smaller the distance, the more similar the sample is to category j.

[0076] In order to optimize the classification ability of the model, contrastive loss is used to minimize the distance between samples of the same type and maximize the distance between samples of different types. The formula for contrastive loss is:

[0077]

[0078] Among them, y i Indicates whether the sample pair i belongs to the same category (y i =1 means the same type, y i =0 means different classes), m is a set margin hyperparameter that controls the minimum distance between classes. This loss function brings samples of the same class closer together and pushes samples of different classes farther apart.

[0079] For detection of unknown classes, for each input sample x i , first calculate its relationship with all known categories of anchor points c j The Euclidean distance d between ij , and then compared with the set threshold θ. If the distance between the sample and all known categories is greater than the threshold θ, it is marked as an unknown category. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flow chart of an open set classification algorithm for hyperspectral images based on multi-scale convolution and feature fusion in the method of the present invention;

[0081] Figure 2 It is a model structure diagram of the hyperspectral image open set classification method based on multi-scale convolution and feature fusion in the method of the present invention;

[0082] Figure 3It is a structural diagram of a spectrum feature extraction module in the method of the present invention;

[0083] Figure 4 The PaviaU dataset ground pseudo-color map and true value map used in the method of the present invention;

[0084] Figure 5 It is a pseudo-color image of the classification results of the model used in the present invention and other models. DETAILED DESCRIPTION

[0085] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0086] The hyperspectral image open set classification method based on multi-scale convolution and feature fusion in this specific implementation manner is as follows: Figure 1 As shown, the following steps are included:

[0087] Step 1: Input the original hyperspectral data set to be classified. Divide the n categories in the data into n-1 known categories and 1 unknown category. The data of known categories are divided into training sets and test sets according to a certain ratio, while the data of unknown categories do not participate in training and are directly merged with the test set of known categories as the final test set.

[0088] Step 2: Input the training samples into the spectral feature extraction module for spectral feature extraction. The spectral feature extraction module combines three-dimensional convolution, SE module, three-dimensional residual extraction module, multi-branch block and convolution enhancement module, gradually extracts and enhances spectral features through layered operations, and finally aggregates features through the average pooling layer. Figure 3 shown.

[0089] Step 2.1: Input the training samples into the SEM module. This module efficiently extracts the spatial-spectral joint features and key band characteristics in the hyperspectral image through a 7×7×7 three-dimensional convolution kernel and a SE attention module with a 16-fold channel compression ratio. The 7×7×7 three-dimensional convolution can simultaneously capture the local patterns and global correlations of pixels in the spatial and spectral dimensions, while the SE module focuses on the key bands through global average pooling and a two-layer fully connected network, suppressing redundant interference, and finally generating multi-scale features with high discrimination, providing richer feature expressions for subsequent classification.

[0090] Step 22: Input the shallow features extracted in step 22 into the residual spectral feature extraction module composed of two 3×3×3 convolution kernels and residual connections. This module processes features through two layers of 3×3×3 three-dimensional convolution and residual connection, effectively alleviating the gradient vanishing problem. By cascading two such residual spectral feature extraction modules, the spectral feature extraction capability is further enhanced, providing more stable and efficient feature expression for subsequent feature optimization and classification tasks.

[0091] Step 23: Input the features obtained in step 22 into a multi-branch block composed of 1×1 convolution and 3×3 convolution. This module extracts features at multiple spatial scales through convolution kernels of different sizes, further enhancing the model's perception of multi-scale information. 1×1 convolution focuses on extracting local fine-grained information, while 3×3 convolution captures spatial features in a wider range. Combined with the characteristics of hyperspectral images, the multi-branch block captures fine-grained information between different bands through parallel processing while retaining the overall spatial structure characteristics. Finally, the features extracted by each branch are weighted fused to output features with stronger discrimination and expression capabilities.

[0092] Step 24: Input the features extracted in step 23 into the convolution enhancement block consisting of 7×7 depthwise separable convolution and 1×1 pointwise convolution. The depthwise convolution performs convolution operations independently for each input channel to capture the local high-frequency detail features of the hyperspectral image; the pointwise convolution fuses information in the channel dimension through 1×1 convolution to enhance the feature expression ability between channels.

[0093] Step 3: Use the spectral features obtained in step 24 to enter the spatial feature extraction module to extract spatial features. After extracting preliminary features through three-dimensional convolution, the spatial features are further enhanced by combining ECA and high-frequency enhancement modules. Then, through the joint operation of multiple two-dimensional residual extraction modules, SE modules and ECA modules, multi-level feature extraction is achieved.

[0094] Step 31: Input the spectral features obtained in step 24 into the spatial feature extraction module composed of 49×3×3 three-dimensional convolution and ECA. The 49×3×3 three-dimensional convolution jointly extracts spatial and spectral features to capture edge, texture and shape information; the ECA module effectively models the inter-channel dependency through 3×1 lightweight convolution, focusing on key spatial features and reducing redundant calculations.

[0095] Step 32: Input the spatial features extracted in step 31 into the high-frequency enhancement module. This module first performs 3×3 average pooling to extract low-frequency features, and then restores the original size through 3×3 upsampling. The low-frequency and high-frequency features are processed separately through 3×3 convolution, and finally the two are fused to enhance the detail information and improve the recognition of spatial features.

[0096] Step 3.3: Input the features extracted in step 3.2 into the two-dimensional residual extraction module. This module first extracts spatial features through 3×3 two-dimensional convolution combined with residual connection to enhance the spatial information expression capability, and then further optimizes the feature expression by combining the SE module and the ECA module. The SE module generates channel descriptors through global average pooling, recalibrates channels through two fully connected layers (1×1 convolution) and Sigmoid activation, generates weights and weights each channel to focus on important features. The ECA module captures local dependencies between channels through 1D convolution (1×3 or 1×5), enhances the expression capability of features, and does not need to introduce additional parameters to maintain computational efficiency. Finally, the original feature information is retained through residual connection to improve the stability and performance of the model.

[0097] Step 4: After splicing the spatial features extracted in step 3.3 and the spectral features extracted in step 2.4, they are sent to the fully connected layer for final classification. In this process, the feature similarity of each category is evaluated by calculating the Euclidean distance, and the classification discrimination ability is optimized. The contrast loss is used to optimize the distance between categories, so that samples of the same category are closer and samples of different categories are farther away. In addition, threshold detection is used to identify unknown categories, that is, when the distance between the sample and the anchor points of all known categories exceeds the set threshold, the sample is marked as an unknown category. The final classification result is output through the softmax function, and the classification performance and unknown category recognition ability of the model are optimized in combination with the cross entropy loss or contrast loss.

[0098] In order to verify the effectiveness of the model proposed in this patent for hyperspectral data classification, the Pavia hyperspectral dataset is used to test the classification algorithm. The Pavia University dataset (PaviaU dataset for short) is a classic hyperspectral remote sensing image dataset, which consists of hyperspectral images of the surrounding areas of the University of Pavia, Italy, and is taken using a ground push-broom imaging spectrometer (Reflective Optics System Imaging Spectrometer, ROSIS). This dataset has a high spatial resolution of 1.3 meters per pixel and 115 effective spectral bands after noise band removal. It is widely used in research tasks such as hyperspectral image classification, target detection, and open set classification. It is one of the standard test datasets in the field of hyperspectral images.

[0099] Table 1 Object categories in the PaviaU dataset

[0100] category Class Name Number of samples per category 1 Asphalt 6631 2 Meadows 18649 3 Gravel 2099 4 Trees 3064 5 Painted metal sheels 1345 6 Bare Soil 5029 7 Bitumen 1330 8 Self-Blocking Bricks 3692 9 Shadows 947

[0101] The experiment compares the classification performance of the proposed model (MSCF-OSC), open-set classification based on supervised contrastive learning (OSC-SCL) and multitask deep learning method that simultaneously conducts classification and reconstruction in the open world (MDL4OW) on the PaviaU dataset, as shown in Table 1. MDL4OW is a model for hyperspectral image classification, which is particularly suitable for open set scenarios with unknown categories. It uses a multi-task learning method to combine classification tasks and reconstruction tasks, and identifies unknown categories by comparing reconstructed data with original data, while improving the classification performance of known categories. OSC-SCL improves the ability to distinguish between known and unknown categories by combining spectral contrast learning with anchor optimization strategy. Its characteristic is to use the contrast constraint of spectral features to enhance the distinguishability in spatial-spectral joint features, which is suitable for the open set classification problem of hyperspectral images.

[0102] Table 2 Classification results of PaviaU dataset

[0103]

[0104]

[0105] In the classification task of known categories (1-8 categories), the model proposed in the present invention shows excellent performance, which is very close to the results of OSC-SCL and significantly better than MDL4OW. In particular, in the classification of categories 7 and 8, the model proposed in the present invention achieved an accuracy of 99.5% and 97.3% respectively, with stable performance and high precision. This shows that the multi-scale convolution and feature fusion mechanism of the model proposed in the present invention has extremely strong expression ability in the classification of known categories, and can efficiently extract key features in hyperspectral images.

[0106] In the recognition of unknown categories (Novel), which is of particular concern in open set classification, the model proposed in the present invention significantly outperforms OSC-SCL (68.96%) and MDL4OW (45.61%) with an accuracy rate of 81.9%. This result shows that the design of the model proposed in the present invention is more suitable for handling open set classification problems and can better distinguish between known categories and unknown categories. This is due to the enhancement of high-frequency information and the optimized design of feature fusion in the model, which greatly improves the generalization ability of the model proposed in the present invention.

[0107] In terms of overall classification performance, the model proposed in the present invention achieved an overall accuracy (OA) of 97.1%, an average accuracy (AA) of 96.2%, and an F1 score of 98.1%, which is better than other comparison methods. This shows that the model proposed in the present invention not only performs well in a single category, but is also more balanced and robust in the overall classification task. In particular, the improvement in the F1 score shows that the model can achieve a better balance between precision and recall.

[0108] In order to subjectively evaluate the classification effect, Figure 4 , 5 The true value map of the PaviaU dataset and the pseudo-color map of the classification results of each method are shown respectively. It can be seen that the method in this paper is closer to the actual distribution of objects, and the area of ​​misclassification is greatly reduced. The experimental results of the model proposed in this paper on the PaviaU dataset prove its significant advantages in the open set classification task of hyperspectral images. Its multi-scale convolution and feature fusion strategy greatly improves the recognition ability of new categories while extracting features of known categories, providing a new and effective solution for the field of hyperspectral image classification.

Claims

1. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The data is divided into n-1 known categories and 1 unknown category. The known categories are divided into training sets and test sets in proportion, and the unknown categories are only used for testing. The training samples are sequentially input into the spectral feature extraction module and the spatial feature extraction module for feature extraction. The spectral feature extraction module gradually extracts and enhances the spectral features by combining three-dimensional convolution, Squeeze-Efficiency Module (SE), spectral residual extraction module, Multi-Branch Block (MBB) and Convolution Enhancement Block (CEB); then the features are sent to the spatial feature extraction module to extract and optimize the spatial features by combining three-dimensional convolution, Efficient Channel Attention (ECA) and High-Frequency Enhancement (HFE), and then sent to the spatial residual extraction module for further processing to output the final spatial features. Subsequently, the extracted spectral and spatial features are spliced ​​and sent to the fully connected layer for classification, and the classification performance is optimized by combining contrastive learning and metric learning. Specifically, the contrastive loss is used to optimize the category distinction ability, and the unknown category is identified based on the distance threshold. Finally, the classification results are output to optimize the model performance.

2. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: Input the original hyperspectral data set to be classified. Divide the n categories in the data into n-1 known categories and 1 unknown category. Among them, the data of known categories are divided into training sets and test sets according to a certain ratio, while the data of unknown categories do not participate in training and are directly merged with the test set of known categories as the final test set.

3. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The training samples are input into the spectral shallow feature extraction module. This module efficiently extracts the spatial-spectral joint features and key band characteristics in the hyperspectral image through a 7×7×7 three-dimensional convolution kernel and a SE attention module with a 16-fold channel compression ratio. The 7×7×7 three-dimensional convolution can simultaneously capture the local patterns and global correlations of pixels in the spatial and spectral dimensions, while the SE module focuses on the key bands through global average pooling and a two-layer fully connected network, suppressing redundant interference, and finally generating multi-scale features with high discrimination, providing richer feature expressions for subsequent classification. First, spectral features of different scales are extracted through a 3D convolutional layer. Through the operation of deep convolution kernels, the model can effectively extract spectral information of different bands. The spectral information extracted by the convolutional layer is input into the SE module to enhance the spectral feature selection ability between channels, thereby optimizing the perception of spectral features. Through this stage, the network can focus on the most discriminative spectral information in the hyperspectral image, providing high-quality spectral features for the subsequent spatial feature extraction module. In this step, the input hyperspectral image data is preprocessed and then passed to the spectral shallow feature extraction module. This module mainly extracts spectral features through 3D convolution, and then performs preliminary enhancement of spectral features through the SE module. The spectral feature extraction process can be expressed by the following steps and formulas: The 3D convolution operation formula is: The input hyperspectral data is X, c is the number of channels, h and w are the height and width of the image respectively. Next, a 3D convolution operation is performed to extract features, and the output feature is Y, where o is the number of output channels, h' and w' are the image sizes after convolution. b0 represents the bias term. After the convolution operation, the SE module is used to optimize the perception of spectral features: Where W1 and W2 are the weights of the fully connected layer, δ is the ReLU activation function, and σ is the Sigmoid activation function. Then Y' is input into the spatial residual extraction module to further extract spectral features.

4. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The features obtained in 3 are input into a multi-branch block consisting of 1×1 convolution and 3×3 convolution. This module extracts features at multiple spatial scales through convolution kernels of different sizes, further enhancing the model's ability to perceive multi-scale information. 1×1 convolution focuses on extracting local fine-grained information, while 3×3 convolution captures spatial features in a wider range. Combining the characteristics of hyperspectral images, multi-branch blocks capture fine-grained information between different bands through parallel processing while retaining the overall spatial structure characteristics. Finally, the features extracted by each branch are weighted and fused to output features with stronger distinguishing power and expression ability. The purpose of the multi-branch block is to capture the multi-scale spatial information of the input features through convolution operations of different scales. In the multi-branch structure, each branch uses a different convolution kernel, which enables the model to learn features at different scales and integrate these features by summing them. For the input feature Y', the multi-branch block divides it into multiple branches, each branch is processed by a different convolution operation, and then they are summed. Branch 1: O is the index of the output channel, k d , k h , k w is the offset of the convolution kernel in depth, height and width, and the convolution kernel size is 3×3×3. b1 is the bias term. Branch 2: W o,c,0,0,0 is the convolution kernel weight of 1×1×1. b2 is the bias term. Each convolution branch extracts features of different scales, and finally sums up their outputs to obtain the fused features. The multi-branch block extracts multi-scale spatial features through convolution kernels of different sizes and sums them up, thereby enhancing the model's perception of spatial information.

5. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The features extracted by 4 are input into the convolution enhancement block composed of 7×7 depthwise separable convolution and 1×1 pointwise convolution. The depthwise convolution performs convolution operation independently for each input channel to capture the local high-frequency detail features of the hyperspectral image; the pointwise convolution fuses information in the channel dimension through 1×1 convolution to enhance the feature expression ability between channels. The convolution enhancement block uses the depth-separable convolution technique to perform convolution operations on each input channel separately, thereby reducing the number of parameters and increasing the efficiency of the model. It also enhances the channel by using point convolution to improve the network's perception of high-frequency features (such as edges and textures). The input is Y mid , after depth convolution and point convolution, the output is O(h,w,d,c), and the calculation process of the convolution enhancement block is: Depthwise separable convolution: Each input channel is convolved independently with a kernel size of 7×7. Y mid (h+i,w+j,d+1,c) is the value of the c-th channel located at (h+i,w+j,d+l) in the input feature map. d (i,j,l,c) is the weight of the depth convolution kernel, corresponding to the cth channel of the input feature. k is the size of the convolution kernel. D(h,w,d,c) is the output of the depth convolution. Point convolution 1: The number of channels is enhanced through 1×1 convolution, and the output is P. W1(c',c) is the weight of the point-by-point convolution kernel, which converts the input channel c to the output channel c'. C is the number of input channels, and c' is the index of the output channel. Apply the GELU activation function to the output of the point-by-point convolution P(h,w,d,c'): G(h,w,d,c′)=GELU(P(h,w,d,c′)) Point-by-point convolution 2: Restore the number of channels to the original input number of channels, and the output is O(h,w,d,c). W2(c,c') is the weight of point-wise convolution 2. The obtained features are then input into the cascade structure of the double-layer spectral residual extraction module for further spectral feature extraction. The convolution enhancement block enhances the sensitivity to high-frequency information (such as edges and textures) through a combination of depth convolution and point convolution, thereby improving the model's ability to capture detailed features.

6. Hyperspectral image open set classification method based on multi-scale convolution and feature fusion, characterized by: The spectral features extracted in 5 are input into the spatial feature extraction module, which extracts preliminary features through three-dimensional convolution and further enhances the spatial features by combining ECA and high-frequency enhancement modules. Subsequently, multi-level feature extraction is achieved through the joint operation of multiple two-dimensional residual extraction modules, SE modules and ECA modules. O(h,w,d,c) extracts preliminary spatial-spectral joint features through three-dimensional convolution: THE 3D =Conv3D(O)。 7. Hyperspectral image open set classification method based on multi-scale convolution and feature fusion, characterized by: The extracted spatial features O 3D Input to the high-frequency enhancement module. This module first performs 3×3 average pooling to extract low-frequency features, and then restores the original size through 3×3 upsampling. The low-frequency and high-frequency features are processed separately through 3×3 convolution, and finally the two are fused to enhance the detail information and improve the recognition of spatial features. Extraction of low-frequency parts: Extract low-frequency features through average pooling operation. Average pooling is an operation that downsamples the input feature map, which can retain the overall information of the image but lose details. X low is the input feature map, b is the batch index, c is the number of channels, h and w are the height and width, k is the size of the pooling kernel, and s is the step size. Afterwards, the low-frequency features are restored to their original size by upsampling. The upsampling operation restores the low-frequency feature map to the same size as the input feature map so that it can be compared with the input. If the size of the feature map after upsampling is inconsistent with the input feature map, the size is adjusted by interpolation. H, W are the height and width of the original feature map, H', W' are the height and width of the low-frequency feature map after downsampling. α, β are the upsampling scaling factors, i.e., the input-output size ratio. The high frequency part is obtained by the difference between the input image and the upsampled low frequency part: Multi-branch feature enhancement is performed on the high-frequency and low-frequency parts respectively: Finally, the enhanced high-frequency and low-frequency features are fused by addition, and the high-frequency enhanced feature is obtained as Y e . The high-frequency enhancement module effectively separates and enhances the high-frequency and low-frequency parts of the input features, and combines multi-branch blocks for multi-scale feature extraction, thereby improving the model's ability to understand details (such as edges and textures) and overall structure. This helps improve classification performance and detail perception in hyperspectral images or other visual tasks.

8. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The features extracted in 7 are input into the spatial residual extraction module to further process the high-frequency enhanced features, and then the spatial feature Y and the spectral feature O extracted in 5 are spliced ​​together. The complete calculation process is as follows: Y=LeakyReLU(Y e +BN(Conv2D2(ReLU(Conv2D1(Y e )))))·W atten ) AND out =concat(AND+OR) W atten is the channel weight vector generated by SE and ECA attention mechanisms, which represents the weight between channels. BN is the batch normalization operation. Conv2D1 is the first layer of two-dimensional convolution operation, the convolution kernel size is 3×3, and the number of output channels is the same as the number of input channels. Conv2D2 is the second layer of two-dimensional convolution operation, the convolution kernel size is 3×3. X+… represents the residual connection, which adds the input features to the processed features. LeakyReLU is the final activation function to ensure nonlinear expression.

9. An open set classification method for hyperspectral images based on multi-scale convolution and feature fusion, characterized by: The features extracted in 8 are input into the classification module. The classification method adopted by the present invention is based on metric learning and contrastive learning methods, and the classification performance of the model is optimized by calculating the Euclidean distance between the sample and the class anchor point and using contrastive loss. The classification of unknown classes and the calculation of loss functions are as follows: First, the anchor point of each class is defined as: in is the scaling factor, e i is the unit eigenvector of the i-th class. Then calculate the Euclidean distance between the sample and the class anchor point, where z is the sample feature vector after feature extraction: d(z,c i )=||z-c i ||2 The distances from the sample to all class anchors are combined into a vector: d(z,C) = [d(z,c1),d(z,c2),d(z,c3),...,d(z,c n )]. Calculate the rejection score: Among them, in this formula represents element-wise multiplication, d i represents the i-th element in the vector, Indicates d i By negating and then exponentiating, smaller values ​​will get larger weights. It represents the sum of the exponential values ​​of all elements for normalization. If the minimum rejection score γ≤θ, the sample is classified as the corresponding known class; if the minimum rejection score γ>θ, the sample is classified as an unknown class. The extracted spatial feature Y and the extracted spectral feature O are concatenated to obtain a joint feature F. Compute the Euclidean distance: Among them, f i is the characteristic vector of the sample, c j is the anchor point of category j, and N is the feature dimension. Euclidean distance d ij It is used to measure the similarity between a sample and the category center. The smaller the distance, the more similar the sample is to category j. In order to optimize the classification ability of the model, contrastive loss is used to minimize the distance between samples of the same type and maximize the distance between samples of different types. The formula for contrastive loss is: Among them, y i Indicates whether the sample pair i belongs to the same category (y i =1 means the same type, y i =0 means different classes), m is a set margin hyperparameter that controls the minimum distance between classes. This loss function brings samples of the same class closer together and pushes samples of different classes farther apart. For detection of unknown classes, for each input sample x i , first calculate its relationship with all known categories of anchor points c j The Euclidean distance d between ij , and then compared with the set threshold θ. If the distance between the sample and all known categories is greater than the threshold θ, it is marked as an unknown category.

Images