Semi-supervised hyperspectral image intelligent classification method

Abstract

The invention discloses a semi-supervised hyper-spectral image intelligent classification method. Marked samples and unmarked samples are utilized to carry out accurate ground feature recognition. According to the method, firstly, a multi-scale deformable spectral spatial feature extraction module is constructed, features can be extracted and fused on multiple scales, and the introduction of deformable convolution can extract distribution features more conforming to samples. Then, through a proposed pseudo label generation strategy, Gaussian function weighting is adopted, and the utilization efficiency and the model performance of the pseudo labels are further improved. In the training process, a high number and high quality pseudo labels are kept to effectively utilize unmarked samples. And finally, capturing remote dependence and learning effective feature representation of the hyperspectral remote sensing image through Transform. Experimental results prove that the method has effectiveness and superiority under the condition of limited labeled samples.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and hyperspectral remote sensing image processing, and in particular to a semi-supervised hyperspectral image intelligent classification method. Background Art

[0002] Hyperspectral remote sensing images have hundreds of continuous bands with very high spectral resolution, which can jointly provide rich spectral and spatial information to accurately identify the type of land cover. Therefore, hyperspectral image classification has been widely used in natural environment monitoring, precision agriculture, mineral detection, military defense security, natural disaster assessment and other fields. However, hyperspectral images have rich spectral and spatial information. How to efficiently extract and fully utilize this information is the key to improving classification accuracy, especially when training samples are limited.

[0003] Traditional classification methods directly use hundreds of spectra in hyperspectral images as pixel features for classification. For example, Support Vector Machine (SVM) and Random Forests (RF) have achieved high classification accuracy, but there is a dimensionality curse problem when processing high-dimensional data. In addition, these methods are difficult to fully explore the correlation information between spectral and spatial features.

[0004] In recent years, deep learning technology has made great progress. Its representative achievement, convolutional neural network (CNN), has been widely used in various fields, including hyperspectral image classification, due to its excellent feature extraction performance. Chen et al. first used two-dimensional CNN to extract spatial spectral features from hyperspectral data. Two-dimensional CNN well preserves the original data structure through local connection, parameter sharing and shift invariance, but it can only process two-dimensional features and fails to fully utilize the high-dimensional characteristics of spectral information. Three-dimensional CNN expands the dimension of convolution kernel and can capture spectral and spatial features at the same time. The three-dimensional autoencoder proposed by Mei et al. uses a three-dimensional convolutional network to extract joint spectral and spatial features and shows good performance. However, the high computational cost of the three-dimensional convolution kernel limits its practicality. In order to balance the feature extraction efficiency and computational complexity, some methods use a dual-branch network structure to process spectral and spatial features respectively. For example, a one-dimensional CNN or a fully connected network is used as the spectral branch, and a two-dimensional CNN is used as the spatial branch, and finally the two are fused to generate joint features. This design improves the classification accuracy to a certain extent, but there is still a problem of insufficient capture of long-range dependency information.

[0005] Although the above CNNs have achieved good results in hyperspectral image classification, there are still some limitations. For CNN, it is difficult to capture the long-range information in HSI well. The emergence of Transformer successfully solves the shortcomings of CNN in capturing long-range information and makes up for the receptive field limitation of CNN. Unlike CNN, the Transformer architecture is a self-attention-based structure combined with a multi-layer perceptron (MLP), which can overcome the limitation of the receptive field, model the long-range dependencies of the tags in the sequence, and more easily capture long-range information. Vision Transformer (VIT) and its variants have approached or even surpassed the CNN-based framework, and have state-of-the-art performance on mainstream computer vision tasks. Hong et al. first used VIT for hyperspectral image classification, taking HSI blocks as sequence data from the spectral dimension, and using group spectral embedding and Transformer encoder modules to learn local detailed spectral representations. Sun et al. proposed a spectral spatial feature tokenization Transformer (SSFTT), which effectively extracts local features by combining CNN modules and uses Gaussian weighted tokenizers to extract high-level semantic features. Although Transformer performs well in capturing long-range information, its ability to model short-range relationships is insufficient, resulting in insufficient local feature extraction. In addition, most current Transformer methods fail to fully utilize multi-scale spectral information and there is still room for improvement.

[0006] The above training methods belong to supervised learning methods, which require a sufficient number of training samples to train the model. When the training samples are insufficient, the model parameters cannot be updated well. In addition, the number of labeled samples is quite limited because data annotation is time-consuming and laborious. To solve this problem, semi-supervised learning methods that combine labeled and unlabeled data for training have received widespread attention. For example, Jia et al. used autoencoders to extract features from unlabeled data in an unsupervised manner, and then used limited labeled samples to fine-tune the model. Yao et al. proposed an active learning semi-supervised framework to construct a reliable pseudo-label set by actively selecting samples that produce high label consistency based on SLIC. Wang et al. proposed an adaptive DropBlock enhanced generative adversarial network (ADGAN) to solve the mode collapse problem and training data imbalance problem. He et al. proposed a semi-supervised generative adversarial network combined with VIT to better preserve spectral sequence information and alleviate category imbalance. The pseudo-label generation method is widely used because of its simplicity and effectiveness. The pseudo-label generation method uses some algorithms to generate labels (i.e., "pseudo-labels") for unlabeled samples, and uses these pseudo-labels to further train the network. The pseudo-label generation method is mainly divided into two types: fixed threshold strategy and adaptive threshold strategy.

[0007] The fixed threshold strategy sets a confidence threshold and selects only high-confidence unlabeled samples for loss calculation. This method can effectively ensure the quality of pseudo labels, but it also discards a large number of pseudo labels with low confidence but possibly correct, resulting in low utilization of unlabeled samples.

[0008] The adaptive threshold strategy dynamically adjusts the confidence threshold according to the current model training status, thereby selecting more pseudo labels for training. Compared with the fixed threshold strategy, the adaptive threshold strategy can more fully tap the potential of unlabeled samples and help improve the coverage of training data. However, this strategy inevitably introduces a certain proportion of erroneous pseudo labels, which may mislead the model training and affect the final performance.

[0009] In summary, although hyperspectral image classification technology has made significant progress in supervised and semi-supervised learning methods, there are still many challenges to be solved in terms of how to efficiently extract spectral and spatial information and deal with data scarcity and imbalance. Summary of the invention

[0010] Purpose of the invention: In view of the limitations of the above-mentioned prior art, the present invention proposes a semi-supervised intelligent classification method for hyperspectral images. By designing a multi-scale deformable spectral spatial feature extraction module, the present invention can effectively extract and fuse spectral and spatial features at multiple scales, and introduce deformable convolution to accurately capture key features that conform to the sample distribution. In addition, the present invention proposes a pseudo-label generation strategy based on Gaussian function weighting, which further improves the model performance on the basis of balancing the quantity and quality of pseudo-labels. Combined with the powerful long-range dependency modeling capability of Transformer, the present invention can efficiently learn the multi-scale feature representation of hyperspectral remote sensing images, providing a more accurate and robust solution for hyperspectral image classification.

[0011] Technical solution: To achieve the above-mentioned invention object, the present invention adopts the following technical solution:

[0012] Step 1: First, preprocess the data and use PCA to reduce the dimension of HSI data. While keeping the spatial structure unchanged, extract the most informative band in the spectral dimension. The data after dimension reduction is expressed as The spatial dimension is represented by H×W, and B represents the number of bands in the spectral dimension;

[0013] Step 2: Divide the reduced-dimensional data into multiple blocks, each block consists of the target pixel and its adjacent pixels, expressed as Where M×N is the block size;

[0014] Step 3: Randomly divide the samples into labeled data sets, unlabeled data sets, and test sets according to the fixed number of samples in each category. Perform weak and strong enhancement on the samples in the unlabeled data set at the same time, laying the foundation for subsequent consistency regularization training;

[0015] Step 4: Construct a multi-scale deformable convolution module, which introduces deformable convolution kernels of different scales and extracts feature information of different scales in parallel through a multi-branch structure, effectively enhancing the feature expression capability;

[0016] Step 5: Construct a multi-scale deformable Transformer network, combining the multi-scale deformable convolution module constructed in step 4 with the Transformer structure. The multi-scale deformable convolution module is used to extract local spatial spectral feature information, while the Transformer structure is used for global feature modeling, thereby achieving more efficient feature extraction and classification. The network receives both labeled and unlabeled data for optimization through semi-supervised learning;

[0017] Step 6: For labeled samples, the model performs supervised learning through the cross entropy loss function to learn the correspondence between data and labels; for unlabeled samples, a consistency regularization strategy is used to make the weakly enhanced version and the strongly enhanced version of the input image generate consistent prediction results. During the training process, the proposed pseudo-label weighting strategy is combined to balance the quantity and quality of pseudo-labels to make better use of unlabeled samples;

[0018] Step 7: Combine the multi-scale deformable Transformer network with the pseudo-label weighting strategy and perform iterative training until the neural network model converges to obtain the final training model. Finally, use a fully connected network to post-process the model output results to obtain the classification results of each pixel.

[0019] Preferably, in step 3, for the same unlabeled sample, a weakly enhanced version and a strongly enhanced version are generated at the same time. Weak enhancement is achieved by randomly flipping the input image horizontally (mirror flipping); while strong enhancement includes multiple enhancement methods, specifically: randomly rotating or horizontally flipping the input image, applying a random enhancement module, and adding noise. The random enhancement module includes the following operations: randomly rotating the image by a certain angle, shearing the image horizontally or vertically (tilting and stretching the image in a certain direction), and translating the image content horizontally or vertically (moving the image on the canvas).

[0020] Preferably, in step 4, the multi-scale deformable convolution module is constructed as follows:

[0021] In the multi-scale deformable convolution module, we introduced deformable convolution kernels of different scales, namely 3x3, 5x5 and 7x7. These convolution kernels are calculated in parallel in different branches in the same layer to extract feature information of different scales. The output features of each convolution layer are respectively passed through the batch normalization layer and the ReLU activation layer. Finally, the features output by the convolution layers of different scales are fused through the Concat and 1x1 convolution operations.

[0022] Preferably, the deformable convolution dynamically adjusts the position of the convolution kernel by introducing a learnable offset, which is implemented as follows:

[0023] Given an input feature map, the output of a traditional convolution can be expressed as:

[0024]

[0025] in is the sampling range of the convolution kernel, is the weight of the convolution kernel, p0 represents the pixel position on the feature map, Represents the position index in the sampling range, that is, the offset relative to p0. Indicates that the input feature map is at position The deformable convolution adds a learnable offset to each point

[0026]

[0027] Since the new sampling point with offset is added If is a non-integer, a bilinear interpolation algorithm is used to get the final pixel value:

[0028]

[0029] Where p represents any non-integer sampling point q lists all integer spatial locations in the feature map x, and G(·, ·) is the bilinear interpolation kernel.

[0030] Finally, the complete process can be described by the following formula:

[0031]

[0032] in, represents the sampling range of the i-th convolution kernel, Indicates that the i-th convolution kernel in the l-th layer is at position p n The weight, Z (l) (p0+p n +Δp i ) indicates that the input feature map is at position p0+p n +Δpi The value at represents the bias of the i-th convolution kernel in the l-th layer, Z (l+1) (p0) indicates the output feature map value of the l+1th layer at position p0.

[0033] Preferably, in step 5, the multi-scale deformable Transformer network is constructed as follows:

[0034] (1) Initial feature extraction: The input is passed through a 3D convolution module for feature extraction. The module includes a 3D convolution layer, a batch normalization layer, and a ReLU activation function, which can effectively extract the initial spatial spectral features of the hyperspectral image.

[0035] (2) Feature dimension reorganization: The output of the 3D convolution module is reorganized in the channel and spatial dimensions, flattened into a two-dimensional plane representation, and adapted to the input requirements of the subsequent multi-scale deformable convolution module.

[0036] (3) Multi-scale deformable convolution module: A multi-branch deformable convolution module is constructed, which contains deformable convolution kernels of three scales: 3×3, 5×5, and 7×7. Local feature information of different scales is extracted through parallel operations. Combined with deformable convolution, it can adaptively capture more flexible spatial structure information. The convolution results are spliced ​​in the channel dimension to retain feature information of different scales.

[0037] (4) 2D convolution fusion feature: The concatenated feature tensor is further processed by a 2D convolution module. This module not only integrates the spatial dimension of the feature, but also compresses and fuses the information of the channel dimension to generate a feature map with a unified expression. At the same time, the batch normalization layer and ReLU activation function are combined in the fusion process to enhance the distinguishing ability of the feature.

[0038] (5) Position embedding and classification tag addition: The fused feature sequence is added with position embedding information to preserve the spatial position information of the input data and help the model understand the order of the input sequence. In addition, a classification tag is added to the feature sequence for subsequent output results in classification tasks to predict the category.

[0039] (7) Global feature modeling: The Transformer structure is used to model the feature sequence, and the long-range feature dependencies are captured through the multi-head self-attention mechanism (Multi-head Self-attention) and multi-layer perceptron (MLP), and the feature expression is optimized by combining residual connection and layer normalization operations.

[0040] (8) Feature extraction and classification: The feature representation of the classification mark is extracted from the output of the Transformer, and the final classification prediction is completed through a multi-layer perceptron (MLP) containing layer normalization and fully connected layers to output the category label of each pixel in the hyperspectral image.

[0041] Preferably, in step 6, the pseudo-label weighting strategy is implemented as follows:

[0042] A truncated Gaussian function is used to fit the sample confidence distribution and weight the loss function, assigning lower weights to pseudo labels that may be correct and higher weights to pseudo labels with higher confidence. The size of the weighted value determines the contribution of the sample to the loss function, achieving better generalization performance. The sample weight value λ(p) is defined as follows:

[0043]

[0044] Among them, p is the predicted probability vector of the unlabeled sample, max(p) is the maximum value in p, that is, the highest confidence of the model for a certain unlabeled sample, and λ max is the upper limit of the weighted value, when max(p) is greater than or equal to μ t When , directly take the upper limit of weighted value λ max .μ t is the mean confidence value in the current training stage, σ t is the standard deviation of the confidence in the current training stage, which is obtained by the highest confidence value max(p) of the unlabeled samples, that is:

[0045]

[0046]

[0047] Among them, B U is the batch size of unlabeled samples, max(p i ) is the highest confidence of the i-th unlabeled sample, is the variance of the confidence at the current training stage. We then use an exponential moving average (EMA) with momentum m to aggregate the statistics of the previous batches to obtain a more stable estimate:

[0048] μ t =mμ t-1 +(1-m)μ b

[0049]

[0050] Among them, we initialize μ0 as Initialized to 1.0.

[0051] Adaptively estimating Gaussian parameters based on the confidence distribution improves the generalization ability of the model. During the training process, as the model's learning ability improves and the confidence increases, the tail of the Gaussian function grows exponentially, and erroneous pseudo-labels with low confidence are given lower weights. t The left and right pseudo-labels are utilized more effectively.

[0052] Compared with the prior art, the present invention has the following technical effects:

[0053] 1. Multi-scale feature extraction and enhanced robustness. The multi-scale deformable convolution module proposed in this invention uses three deformable convolution kernels of different scales and expansion rates to effectively learn discriminative features at different scales. Deformable convolution can dynamically adjust the shape and size of the convolution kernel according to the structure and characteristics of the input data, thereby adapting to complex spatial features, improving the model's ability to capture detailed information in hyperspectral images, and significantly enhancing robustness.

[0054] 2. Optimization of pseudo-label generation strategy. This paper designs a pseudo-label weighting strategy based on truncated Gaussian function, which weights the unlabeled data by fitting the confidence distribution. During the training process, the quantity and quality of pseudo-labels can be guaranteed at the same time, unlabeled data can be fully utilized, and the limitation of data scarcity on model performance can be alleviated.

[0055] 3. Efficient spatial-spectral information modeling. The present invention combines the CNN network with the Transformer structure to fully mine the spatial and spectral information in hyperspectral images from the shallow to the deep. This combination can concisely and efficiently express the multi-level semantic features of hyperspectral images, significantly improving the accuracy and efficiency of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a model structure diagram for semi-supervised hyperspectral image classification based on multi-scale deformable Transformer and pseudo-label generation proposed in an example of the present invention.

[0057] Figure 2 Schematic diagram of the multi-scale deformable convolution module proposed in this example of the present invention.

[0058] Figure 3 : Pseudo-color images of the WHLK dataset hyperspectral image used in the embodiments of the present invention and corresponding true value labels.

[0059] Figure 4The present invention provides a simulation diagram of the classification of objects in the WHLK dataset using existing supervised and semi-supervised classification methods, as well as a simulation diagram of the classification of objects using the present invention. Figure (a) is a real pseudo-color image, Figure (b) is a real object label map, Figure (c) is the classification result of the HybridSN model, Figure (d) is the classification result of the HSMSN model, Figure (e) is the classification result of the 3DAES model, Figure (f) is the classification result of the DMSGe model, Figure (g) is the classification result of the S3Net model, Figure (h) is the classification result of the AMGCFN model, and Figure (i) is the classification result of the model of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0061] like Figure 1 As shown, a semi-supervised hyperspectral image intelligent classification method disclosed in an embodiment of the present invention specifically comprises the following steps:

[0062] Step 1: First, preprocess the data and use PCA to reduce the dimension of HSI data. The data after dimension reduction is expressed as The spatial dimension is represented by H×W, and B represents the number of bands in the spectral dimension.

[0063] Step 2: Divide the reduced-dimensional data into multiple blocks, each block consists of the target pixel and its adjacent pixels, expressed as Where M×N is the block size.

[0064] Step 3: Randomly divide the samples into labeled data sets, unlabeled data sets, and test sets according to a fixed number of samples in each category. Weak enhancement and strong enhancement are performed on the samples in the unlabeled data set at the same time to lay the foundation for subsequent consistency regularization training. Weak enhancement is achieved by randomly flipping the input image horizontally (mirror flipping); while strong enhancement includes multiple enhancement methods, specifically: randomly rotating or horizontally flipping the input image, applying a random enhancement module, and adding noise. The random enhancement module includes the following operations: randomly rotating the image by a certain angle, shearing the image horizontally or vertically (tilting and stretching the image in a certain direction), and translating the image content horizontally or vertically (moving the image on the canvas).

[0065] Step 4: Construct a multi-scale deformable convolution module, which introduces deformable convolution kernels of different scales and extracts feature information of different scales in parallel through a multi-branch structure, effectively enhancing the feature expression capability.

[0066] In the multi-scale deformable convolution module, we introduced deformable convolution kernels of different scales, namely 3x3, 5x5 and 7x7. These convolution kernels are calculated in parallel in different branches in the same layer to extract feature information of different scales. The output features of each convolution layer are respectively passed through the batch normalization layer and the ReLU activation layer. Finally, the features output by the convolution layers of different scales are fused through the Concat and 1x1 convolution operations.

[0067] Deformable convolution dynamically adjusts the position of the convolution kernel by introducing a learnable offset, which is implemented as follows:

[0068] Given an input feature map, the output of a traditional convolution can be expressed as:

[0069]

[0070] in is the sampling range of the convolution kernel, is the weight of the convolution kernel, p0 represents the pixel position on the feature map, Represents the position index in the sampling range, that is, the offset relative to p0. Indicates that the input feature map is at position The deformable convolution adds a learnable offset to each point

[0071]

[0072] Since the new sampling point with offset is added If is a non-integer, a bilinear interpolation algorithm is used to get the final pixel value:

[0073]

[0074] Where p represents any non-integer sampling point q lists all integer spatial locations in the feature map x, and G(·, ·) is the bilinear interpolation kernel.

[0075] Finally, the complete process can be described by the following formula:

[0076]

[0077] in, represents the sampling range of the i-th convolution kernel, Indicates that the i-th convolution kernel in the l-th layer is at position p n The weight, Z (l) (p0+p n +Δp i ) indicates that the input feature map is at position p0+p n +Δp iThe value at represents the bias of the i-th convolution kernel in the l-th layer, Z (l+1) (p0) indicates the output feature map value of the l+1th layer at position p0.

[0078] Step 5: Construct a multi-scale deformable Transformer network, combining the multi-scale deformable convolution module constructed in step 4 with the Transformer structure. The multi-scale deformable convolution module is used to extract local spatial spectral feature information, while the Transformer structure is used for global feature modeling, thereby achieving more efficient feature extraction and classification. The network receives both labeled and unlabeled data for optimization through semi-supervised learning.

[0079] The multi-scale deformable Transformer network is constructed as follows:

[0080] (1) Initial feature extraction: The input is passed through a 3D convolution module for feature extraction. The module includes a 3D convolution layer, a batch normalization layer, and a ReLU activation function, which can effectively extract the initial spatial spectral features of the hyperspectral image.

[0081] (2) Feature dimension reorganization: The output of the 3D convolution module is reorganized in the channel and spatial dimensions, flattened into a two-dimensional plane representation, and adapted to the input requirements of the subsequent multi-scale deformable convolution module.

[0082] (3) Multi-scale deformable convolution module: A multi-branch deformable convolution module is constructed, which contains deformable convolution kernels of three scales: 3×3, 5×5, and 7×7. Local feature information of different scales is extracted through parallel operations. Combined with deformable convolution, it can adaptively capture more flexible spatial structure information. The convolution results are spliced ​​in the channel dimension to retain feature information of different scales.

[0083] (4) 2D convolution fusion feature: The concatenated feature tensor is further processed by a 2D convolution module. This module not only integrates the spatial dimension of the feature, but also compresses and fuses the information of the channel dimension to generate a feature map with a unified expression. At the same time, the batch normalization layer and ReLU activation function are combined in the fusion process to enhance the distinguishing ability of the feature.

[0084] (5) Position embedding and classification tag addition: The fused feature sequence is added with position embedding information to preserve the spatial position information of the input data and help the model understand the order of the input sequence. In addition, a classification tag is added to the feature sequence for subsequent output results in classification tasks to predict the category.

[0085] (7) Global feature modeling: The Transformer structure is used to model the feature sequence, and the long-range feature dependencies are captured through the multi-head self-attention mechanism (Multi-head Selfattention) and multi-layer perceptron (MLP), and the feature expression is optimized by combining residual connection and layer normalization operations.

[0086] (8) Feature extraction and classification: The feature representation of the classification mark is extracted from the output of the Transformer, and the final classification prediction is completed through a multi-layer perceptron (MLP) containing layer normalization and fully connected layers to output the category label of each pixel in the hyperspectral image.

[0087] Step 6: For labeled samples, the model performs supervised learning through the cross entropy loss function to learn the correspondence between data and labels; for unlabeled samples, a consistency regularization strategy is used to make the weakly enhanced version and the strongly enhanced version of the input image generate consistent prediction results. During the training process, the proposed pseudo-label weighting strategy is combined to balance the quantity and quality of pseudo-labels to make better use of unlabeled samples. The specific implementation is as follows:

[0088] A truncated Gaussian function is used to fit the sample confidence distribution and weight the loss function, assigning lower weights to pseudo labels that may be correct and higher weights to pseudo labels with higher confidence. The size of the weighted value determines the contribution of the sample to the loss function, achieving better generalization performance. The sample weight value λ(p) is defined as follows:

[0089]

[0090] Among them, p is the predicted probability vector of the unlabeled sample, max(p) is the maximum value in p, that is, the highest confidence of the model for a certain unlabeled sample, and λ max is the upper limit of the weighted value, when max(p) is greater than or equal to μ t When , directly take the upper limit of weighted value λ max .μ t is the mean confidence value in the current training stage, σ t is the standard deviation of the confidence in the current training stage, which is obtained by the highest confidence value max(p) of the unlabeled samples, that is:

[0091]

[0092]

[0093] Among them, B U is the batch size of unlabeled samples, max(p i ) is the highest confidence of the i-th unlabeled sample, is the variance of the confidence at the current training stage. We then use an exponential moving average (EMA) with momentum m to aggregate the statistics of the previous batches to obtain a more stable estimate:

[0094] μ t =mμ t-1 +(1-m) μ b

[0095]

[0096] Among them, we initialize μ0 as Initialized to 1.0.

[0097] Adaptively estimating Gaussian parameters based on the confidence distribution improves the generalization ability of the model. During the training process, as the model's learning ability improves and the confidence increases, the tail of the Gaussian function grows exponentially, and erroneous pseudo-labels with low confidence are given lower weights. t The left and right pseudo-labels are utilized more effectively.

[0098] Step 7: Combine the multi-scale deformable Transformer network with the pseudo-label weighting strategy and perform iterative training until the neural network model converges to obtain the final training model. Finally, use a fully connected network to post-process the model output results to obtain the classification results of each pixel.

[0099] In order to verify the effect of the present invention, based on the above technical solution, this embodiment carried out a simulation experiment, and the specific results and analysis are as follows:

[0100] 1. Simulation conditions:

[0101] In this embodiment, the experimental data set uses the WHU-Hi-LongKou (WHLK) data set collected over Longkou Town, Hubei Province, China. The data was acquired by a DJI Matrice 600Pro drone platform equipped with a Headwall Nano-Hyperspec sensor. The pixel size of the dataset image is 550×441, consisting of 270 bands, with a spectral range of 0.4 to 1.0 μm. The dataset contains 9 types of ground objects and 204,542 samples. Figure 3 Provides a visual representation of the dataset. Figure 3 (a) shows a pseudo-color image of the real hyperspectral image. Figure 3 (b) is the real object label map. Figure 3(c) is the attached true value label image, where different colors represent different ground object categories. In addition, this experiment randomly selects 10 labeled samples from each category as the training set, and uses 10% of the samples as the unlabeled training set for pseudo label generation, and the remaining samples as the test set. Table 1 lists the category names of all surface cover objects, the number of samples contained in each category, and the data division, providing a detailed reference for experimental analysis.

[0102] Table 1 WHLK feature categories and data division

[0103]

[0104] In order to comprehensively evaluate the performance of the present invention, the simulation experiment of the present invention adopts the overall accuracy (OA), average accuracy (AA) and Kappa coefficient of the classification results as quantitative evaluation indicators of the classification performance of the hyperspectral image classification task, and draws the classification results of the surface cover as qualitative evaluation indicators.

[0105] 2. Comparison and analysis of simulation results

[0106] Table 2 Comparison of the accuracy of ground object classification of real hyperspectral images by the existing technology method and the method of the present invention

[0107]

[0108] It can be seen from the quantitative results in Table 2 that the performance of the present invention on the data set is better than other comparative methods. The multi-scale deformable convolution module proposed in the present invention shows a stronger learning ability for irregular-shaped objects through flexible feature extraction and effective feature alignment, thereby significantly improving the classification accuracy of hyperspectral images. In addition, the pseudo-label generation strategy generates high-quality prediction labels for unlabeled data, expands the scale of the training data set, and uses these pseudo-labels to help the model learn the global relationship between samples. This strategy enables the model to learn from richer samples while deeply understanding the complex relationships between samples, thereby improving the generalization ability of the model and ultimately achieving better classification results.

Claims

1. A semi-supervised hyperspectral image intelligent classification method, characterized in that: The steps include: Step 1: First, preprocess the data and use PCA to reduce the dimension of HSI data. While keeping the spatial structure unchanged, extract the most informative band in the spectral dimension. The data after dimension reduction is expressed as The spatial dimension is represented by H×W, and B represents the number of bands in the spectral dimension; Step 2: Divide the reduced-dimensional data into multiple blocks, each block consists of the target pixel and its adjacent pixels, expressed as Where M×N is the block size; Step 3: Randomly divide the samples into labeled data sets, unlabeled data sets, and test sets according to the fixed number of samples in each category. Perform weak and strong enhancement on the samples in the unlabeled data set at the same time, laying the foundation for subsequent consistency regularization training; Step 4: Construct a multi-scale deformable convolution module, which introduces deformable convolution kernels of different scales and extracts feature information of different scales in parallel through a multi-branch structure, effectively enhancing the feature expression capability; Step 5: Construct a multi-scale deformable Transformer network, combining the multi-scale deformable convolution module constructed in step 4 with the Transformer structure. The multi-scale deformable convolution module is used to extract local spatial spectral feature information, while the Transformer structure is used for global feature modeling, thereby achieving more efficient feature extraction and classification. The network receives both labeled and unlabeled data for optimization through semi-supervised learning; Step 6: For labeled samples, the model performs supervised learning through the cross entropy loss function to learn the correspondence between data and labels; For unlabeled samples, a consistency regularization strategy is used to make the weakly enhanced version and the strongly enhanced version of the input image generate consistent prediction results. During the training process, the proposed pseudo-label weighting strategy is combined to balance the quantity and quality of pseudo-labels to make better use of unlabeled samples; Step 7: Combine the multi-scale deformable Transformer network with the pseudo-label weighting strategy and perform iterative training until the neural network model converges to obtain the final training model. Finally, use a fully connected network to post-process the model output results to obtain the classification results of each pixel.

2. A semi-supervised hyperspectral image intelligent classification method according to claim 1, characterized in that: In step 3, for the same unlabeled sample, a weak enhancement version and a strong enhancement version are generated at the same time. Weak enhancement is achieved by randomly flipping the input image horizontally (mirror flipping); while strong enhancement includes multiple enhancement methods, specifically: randomly rotating or horizontally flipping the input image, applying a random enhancement module, and adding noise. The random enhancement module includes the following operations: randomly rotating the image by a certain angle, shearing the image horizontally or vertically (tilting and stretching the image in a certain direction), and translating the image content horizontally or vertically (moving the image on the canvas).

3. The semi-supervised hyperspectral image intelligent classification method according to claim 1, characterized in that: In step 4, the construction of the multi-scale deformable convolution module is as follows: In the multi-scale deformable convolution module, we introduced deformable convolution kernels of different scales, namely 3x3, 5x5 and 7x7. These convolution kernels are calculated in parallel in different branches in the same layer to extract feature information of different scales. The output features of each convolution layer are respectively passed through the batch normalization layer and the ReLU activation layer. Finally, the features output by the convolution layers of different scales are fused through the Concat and 1x1 convolution operations.

4. A semi-supervised hyperspectral image intelligent classification method according to claim 3, characterized in that: Deformable convolution dynamically adjusts the position of the convolution kernel by introducing a learnable offset, which is implemented as follows: Given an input feature map, the output of a traditional convolution can be expressed as: in is the sampling range of the convolution kernel, is the weight of the convolution kernel, p0 represents the pixel position on the feature map, Represents the position index in the sampling range, that is, the offset relative to p0. Indicates that the input feature map is at position The deformable convolution adds a learnable offset to each point Since the new sampling point with offset is added If is a non-integer, a bilinear interpolation algorithm is used to get the final pixel value: Where p represents any non-integer sampling point q lists all integer spatial locations in the feature map x, and G(·, ·) is the bilinear interpolation kernel. Finally, the complete process can be described by the following formula: in, represents the sampling range of the i-th convolution kernel, Indicates that the i-th convolution kernel in the l-th layer is at position p n The weight, Z (l) (p0+p n +Δp i ) indicates that the input feature map is at position p0+p n +Δp i The value at represents the bias of the i-th convolution kernel in the l-th layer, Z (l+1) (p0) indicates the output feature map value of the l+1th layer at position p0.

5. The semi-supervised hyperspectral image intelligent classification method according to claim 1, characterized in that: In step 5, the multi-scale deformable Transformer network is constructed as follows: (1) Initial feature extraction: The input is passed through a 3D convolution module for feature extraction. The module includes a 3D convolution layer, a batch normalization layer, and a ReLU activation function, which can effectively extract the initial spatial spectral features of the hyperspectral image. (2) Feature dimension reorganization: The output of the 3D convolution module is reorganized in the channel and spatial dimensions, flattened into a two-dimensional plane representation, and adapted to the input requirements of the subsequent multi-scale deformable convolution module. (3) Multi-scale deformable convolution module: A multi-branch deformable convolution module is constructed, which contains deformable convolution kernels of three scales: 3×3, 5×5, and 7×7. Local feature information of different scales is extracted through parallel operations. Combined with deformable convolution, it can adaptively capture more flexible spatial structure information. The convolution results are spliced ​​in the channel dimension to retain feature information of different scales. (4) 2D convolution fusion feature: The concatenated feature tensor is further processed by a 2D convolution module. This module not only integrates the spatial dimension of the feature, but also compresses and fuses the information of the channel dimension to generate a feature map with a unified expression. At the same time, the batch normalization layer and ReLU activation function are combined in the fusion process to enhance the distinguishing ability of the feature. (5) Position embedding and classification tag addition: The fused feature sequence is added with position embedding information to preserve the spatial position information of the input data and help the model understand the order of the input sequence. In addition, a classification tag is added to the feature sequence for subsequent output results in classification tasks to predict the category. (7) Global feature modeling: The Transformer structure is used to model the feature sequence, and the long-range feature dependencies are captured through the multi-head self-attention mechanism (Multi-head Self-attention) and multi-layer perceptron (MLP), and the feature expression is optimized by combining residual connection and layer normalization operations. (8) Feature extraction and classification: Extract the feature representation of the classification mark from the output of the Transformer, and complete the final classification prediction through a multi-layer perceptron (MLP) containing layer normalization and fully connected layers to output the category label of each pixel in the hyperspectral image.

6. The semi-supervised hyperspectral image intelligent classification method according to claim 1, characterized in that: In step 6, the pseudo-label weighting strategy is implemented as follows: A truncated Gaussian function is used to fit the sample confidence distribution and weight the loss function, assigning lower weights to pseudo labels that may be correct and higher weights to pseudo labels with higher confidence. The size of the weighted value determines the contribution of the sample to the loss function, achieving better generalization performance. The sample weight value λ(p) is defined as follows: Among them, p is the predicted probability vector of the unlabeled sample, max(p) is the maximum value in p, that is, the highest confidence of the model for a certain unlabeled sample, and λ max is the upper limit of the weighted value, when max(p) is greater than or equal to μ t When , directly take the upper limit of weighted value λ max .μ t is the mean confidence value in the current training stage, σ t is the standard deviation of the confidence in the current training stage, which is obtained by the highest confidence value max(p) of the unlabeled samples, that is: Among them, B U is the batch size of unlabeled samples, max(p i ) is the highest confidence of the i-th unlabeled sample, is the variance of the confidence at the current training stage. We then use an exponential moving average (EMA) with momentum m to aggregate the statistics of the previous batches to obtain a more stable estimate: m t =mm t-1 +(1-m)m b Among them, we initialize μ0 as Initialized to 1.

0. Adaptively estimating Gaussian parameters based on the confidence distribution improves the generalization ability of the model. During the training process, as the model's learning ability improves and the confidence increases, the tail of the Gaussian function grows exponentially, and erroneous pseudo-labels with low confidence are given lower weights. t The left and right pseudo-labels are utilized more effectively.

Images