Semi-supervised hyperspectral image classification methods based on regular and irregular constraints

Abstract

This invention relates to a semi-supervised hyperspectral image classification method based on regular and irregular constraints, belonging to the field of hyperspectral remote sensing; hyperspectral image data preprocessing; multi-scale image input fusion based on neural networks to generate stitched image patches; and calculation of the regular loss value Θ based on regular constraints generated by a deconvolutional network according to the stitched image patches. R Irregularity constraint generation based on superpixel clustering, and calculation of irregularity loss value Θ. I This invention introduces unlabeled samples based on superpixel constraints and employs semi-supervised learning; calculates the loss function Θ using multiple acquired constraints; obtains an optimized neural network based on the loss function Θ; inputs hyperspectral image data into the optimized neural network, which then classifies features according to accumulated experience; this invention improves the classification performance of hyperspectral images, effectively reduces discrete misclassification noise, and introduces a semi-supervised approach, reducing the number of samples required.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral remote sensing and relates to a semi-supervised hyperspectral image classification method based on regular and irregular constraints. Background Technology

[0002] In recent years, hyperspectral remote sensing has gained widespread attention in the field of remote sensing. Hyperspectral remote sensing images record continuous spatial and spectral information of ground objects. Compared with traditional remote sensing imagery, the advantages of hyperspectral imagery are mainly reflected in the following aspects: high spatial and spectral resolution; wide spectral range with hundreds of continuous spectral bands; and the mixing of spatial and spectral information. These advantages greatly broaden the scope and depth of remote sensing applications in practical life and work. However, the characteristics of hyperspectral imaging also place higher demands on our classification techniques and accuracy. First, the spectral information in hyperspectral remote sensing images is highly correlated and difficult to distinguish, easily leading to problems such as objects with the same spectrum but different spectra and objects with different spectra. Dimensional redundancy makes classification methods that perform well in low-dimensional space difficult in hyperspectral classification. How to reduce dimensionality while preserving rich information is a problem that needs to be studied and solved. Second, hyperspectral image classification generally requires a large number of manually labeled samples, but compared to a large number of unlabeled samples, the number of labeled samples is small, and the cost of labeling is also high.

[0003] With the development of artificial intelligence, deep learning-based methods have shown great potential in hyperspectral image classification. Deep neural networks, one of the representatives of deep learning, process complex data and extract deep features through multi-layered neural network structures. Compared with traditional methods, the features they acquire are more abstract, giving them a greater advantage in the field of hyperspectral image classification. Currently, most deep learning-based hyperspectral classification methods use deep networks to extract spatial-spectral information, combining dimensionality reduction methods such as PCA and SAE with classifiers for pixel-level classification. However, hyperspectral images have high spatial dimensions and complex data, often exhibiting "different objects with the same spectrum" or "different spectra with the same object." This leads to the "salt-and-pepper noise" problem in pixel-level classification tasks, where misclassified pixels severely affect classification accuracy. Furthermore, most classification methods rely on a single feature extraction approach, lacking effective constraints to control the learning process, resulting in limited feature extraction capabilities, information loss, or insufficient feature effectiveness. Finally, considering the limited number of labeled samples in hyperspectral images, it is necessary to continue exploring the integration of semi-supervised learning methods to make greater use of unlabeled samples. Summary of the Invention

[0004] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a semi-supervised hyperspectral image classification method based on regular and irregular constraints, which improves the classification effect of hyperspectral images, effectively reduces discrete misclassification noise, and at the same time introduces a semi-supervised approach to reduce the number of samples required.

[0005] The solution of the present invention is:

[0006] Semi-supervised hyperspectral image classification methods based on regular and irregular constraints include:

[0007] Data preprocessing of hyperspectral images;

[0008] Multi-scale image input fusion based on neural networks generates stitched image patches;

[0009] Based on the stitched image patches, rule constraints are generated using a deconvolutional network, and the rule loss value Θ is calculated. R ;

[0010] Irregularity constraint generation based on superpixel clustering, and calculation of irregularity loss value Θ. I ;

[0011] Unlabeled sample introduction and semi-supervised learning based on superpixel constraints;

[0012] The loss function Θ is calculated using the acquired constraints;

[0013] The loss function Θ is fed into the neural network for backpropagation to update and optimize the parameters of the neural network, resulting in an optimized neural network.

[0014] The hyperspectral image data is input into the optimized neural network, which extracts features from the hyperspectral image data and classifies the features based on accumulated experience.

[0015] In the aforementioned semi-supervised hyperspectral image classification method based on rule-based and irregular constraints, the hyperspectral image data preprocessing method is as follows:

[0016] Outliers are eliminated and normalized before the hyperspectral image data is processed by the input network.

[0017] In the aforementioned semi-supervised hyperspectral image classification method based on rule-based and irregular constraints, the specific method for eliminating outliers is as follows:

[0018] Outliers in hyperspectral image data are detected and corrected by interpolation calculations to replace them.

[0019] In the aforementioned semi-supervised hyperspectral image classification methods based on rule-based and irregular constraints, the specific method for normalization processing is as follows:

[0020] The values ​​of the hyperspectral image data are scaled proportionally to uniformly map them to the [0, 1] interval.

[0021] In the aforementioned semi-supervised hyperspectral image classification method based on rule constraints and irregular constraints, the multi-scale image input fusion method based on neural networks is as follows:

[0022] S11. Obtain image patches of different sizes as input to the neural network;

[0023] S12. Input image patches of different sizes into convolutional layers of different sizes respectively, and obtain outputs of the same size;

[0024] S13. The output results are stitched together to obtain the stitched image blocks.

[0025] In the aforementioned semi-supervised hyperspectral image classification method based on rule constraints and irregular constraints, the method for generating rule constraints based on deconvolutional networks is as follows:

[0026] S21. Use the stitched image patches as input to a convolutional neural network to obtain feature vectors.

[0027] S22. Using eigenvectors A feature map X′ with the same scale as the input image patch is obtained through a deconvolution layer. j ;

[0028] S23. Calculate the input image patch using the mean square loss function.

[0029] S24, Calculation With X′ j The loss value between these two values ​​is denoted as the rule loss value Θ. R .

[0030] In the aforementioned semi-supervised hyperspectral image classification method based on rule constraints and irregular constraints, the rule loss value Θ R The calculation method is as follows:

[0031]

[0032] In the aforementioned semi-supervised hyperspectral image classification method based on regular and irregular constraints, irregular constraints are generated based on superpixel clustering, and the irregularity loss value Θ is calculated. I The method is as follows:

[0033] S31. Based on the obtained feature vector Perform mean-shift clustering on the images;

[0034] S32. Calculate the average spectral features within each superpixel using the clustering results.

[0035] S33, through feature vectors and average spectral characteristics Calculate the square of the L2 norm of the difference After dividing the result by the balancing parameter σ, the average value is calculated over all superpixels to obtain the irregular constraint loss value, denoted as the irregular loss value Θ. I :

[0036]

[0037] In the aforementioned semi-supervised hyperspectral image classification methods based on regular and irregular constraints, the introduction of unlabeled samples based on superpixel constraints and the semi-supervised learning specifically involve:

[0038] S41. Calculate the loss value using rule constraints, without the participation of sample labels;

[0039] S42. Irregular constraints are used to introduce full-graph data, including unlabeled samples, during mean-shift clustering.

[0040] In the aforementioned semi-supervised hyperspectral image classification method based on regular and irregular constraints, the method for forming the loss function using the acquired multiple constraints is as follows:

[0041] S51. Calculate the cross-entropy loss value Θ based on supervised classification. CE ;

[0042] S52, Regarding the rule loss value Θ R Irregular loss value Θ I and cross-entropy loss value Θ CE We perform a weighted summation to obtain the loss function Θ; Θ = λ1Θ R +λ2Θ I +λ3Θ CE In the formula, λ1, λ2, and λ3 are the balance parameters of regular constraints, irregular constraints, and cross-entropy loss, respectively.

[0043] The advantages of this invention compared to the prior art are:

[0044] (1) This invention enhances feature extraction capabilities through multi-scale convolutional layer input and additional constraints, while retaining the advantages of graph-based semi-supervised learning in graph convolutional networks by utilizing irregular constraints. Ultimately, it achieves improved accuracy in hyperspectral image classification and effective reduction of discrete misclassified noise.

[0045] (2) This invention supplements relevant information through multi-scale input. By fusing and extracting feature vectors from multi-scale images, information from different field-of-view ranges in the shared space can more realistically represent ground objects. Furthermore, the model integrates self-supervised rule constraints and clustering-based irregular constraints. Rule constraints reconstruct multi-scale images through deconvolution, evaluating the completeness of information and thus preserving more important details during feature extraction. Irregular constraints introduce neighboring pixel relationships based on superpixels, alleviating the salt-and-pepper noise problem caused by pixel-level classification and further improving the network's learning ability.

[0046] (3) This invention enables unlabeled samples to participate in the learning process of the network through continuous iterative optimization of superpixel clustering results, achieving a semi-supervised effect and reducing the amount of labeled samples required. Attached Figure Description

[0047] Figure 1 This is a flowchart of the semi-supervised hyperspectral image classification process of the present invention;

[0048] Figure 2 This is a schematic diagram of the multi-scale pixel block and multi-scale input process of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating the image data classification of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the embodiments.

[0051] This invention provides a semi-supervised hyperspectral image classification method based on regular and irregular constraints, which mainly solves the following problems: reducing misclassification noise in pixel-level image classification; enhancing the learning ability of deep networks through constraints to obtain more effective features; and reducing the use of labeled samples by allowing unlabeled samples to participate in the network's learning through semi-supervised methods.

[0052] Semi-supervised hyperspectral image classification methods based on regular and irregular constraints, such as Figure 1 As shown, the specific steps include the following:

[0053] Data preprocessing of hyperspectral images; The data preprocessing method for hyperspectral images is to eliminate outliers and perform normalization before the hyperspectral image data is input into the network for processing.

[0054] The specific method for eliminating outliers is to detect outliers in the hyperspectral image data and then use interpolation calculations to replace the outliers for correction.

[0055] The specific method of normalization is to scale the values ​​of the hyperspectral image data proportionally so that they are uniformly mapped to the [0, 1] interval.

[0056] A multi-scale image input fusion method based on neural networks is used to generate stitched image patches. The method is as follows:

[0057] S11. Obtain image patches of different sizes as input to the neural network;

[0058] S12. Input image patches of different sizes into convolutional layers of different sizes respectively, and obtain outputs of the same size;

[0059] S13. The output results are stitched together to obtain the stitched image blocks.

[0060] Based on the stitched image patches, rule constraints are generated using a deconvolutional network, and the rule loss value Θ is calculated. R The method for generating rule constraints based on deconvolutional networks is as follows:

[0061] S21. Use the stitched image patches as input to a convolutional neural network to obtain feature vectors.

[0062] S22. Using eigenvectors A feature map X′ with the same scale as the input image patch is obtained through a deconvolution layer. j ;

[0063] S23. Calculate the input image patch using the mean square loss function.

[0064] S24, Calculation With X′ j The loss value between these two values ​​is denoted as the rule loss value Θ. R .

[0065] Rule loss value Θ R The calculation method is as follows:

[0066]

[0067] Irregularity constraint generation based on superpixel clustering, and calculation of irregularity loss value Θ. I Irregularity constraint generation based on superpixel clustering, and calculation of irregularity loss value Θ. I The method is as follows:

[0068] S31. Based on the obtained feature vector Perform mean-shift clustering on the images;

[0069] S32. Calculate the average spectral features within each superpixel using the clustering results.

[0070] S33, through feature vectors and average spectral characteristics Calculate the square of the L2 norm of the difference After dividing the result by the balancing parameter σ, the average value is calculated over all superpixels to obtain the irregular constraint loss value, denoted as the irregular loss value Θ. I :

[0071]

[0072] Unlabeled sample introduction and semi-supervised learning based on superpixel constraints; specifically, unlabeled sample introduction and semi-supervised learning based on superpixel constraints are as follows:

[0073] S41. Calculate the loss value using rule constraints, without the participation of sample labels;

[0074] S42. Irregular constraints are used to introduce full-graph data, including unlabeled samples, during mean-shift clustering.

[0075] The loss function Θ is calculated using the acquired constraints; the method for forming the loss function using the acquired constraints is as follows:

[0076] S51. Calculate the cross-entropy loss value Θ based on supervised classification. CE ;

[0077] S52, Regarding the rule loss value Θ R Irregular loss value Θ I and cross-entropy loss value Θ CE We perform a weighted summation to obtain the loss function Θ; Θ = λ1Θ R +λ2Θ I +λ3Θ CE In the formula, λ1, λ2, and λ3 are the balance parameters of regular constraints, irregular constraints, and cross-entropy loss, respectively.

[0078] The loss function Θ is then fed into the neural network for backpropagation to update and optimize the network's parameters, resulting in an optimized neural network.

[0079] The hyperspectral image data is input into the optimized neural network, which extracts features from the hyperspectral image data and classifies the features based on accumulated experience.

[0080] Example

[0081] This embodiment provides the following detailed technical solution:

[0082] 1. Data preprocessing of hyperspectral images

[0083] Before inputting image products into the network for processing, outliers should be eliminated and the data normalized to scale it proportionally and place it within the specific range of (0,1). Normalization eliminates the influence of dimensions between indicators, improving the comparability of data indicators. After data standardization, the raw data is on the same order of magnitude, making it suitable for comprehensive comparison and evaluation. The most typical example of this is data normalization. Furthermore, data normalization accelerates the gradient descent process to find the optimal solution, thus speeding up the convergence of the training network and improving the accuracy of the network model to some extent. The specific steps are as follows:

[0084] (1) Detect outliers in the image and perform correction by interpolation calculation to replace the outliers;

[0085] (2) Data standardization: scaling the data values ​​proportionally to map them uniformly to the [0,1] interval.

[0086] 2. Multi-scale image input fusion based on neural networks

[0087] Image recognition suffers from the problem of field of view. The larger the input data range, the larger the field of view, containing contextual information at different scales. According to multi-view learning theory, data with multiple views can more realistically represent real objects, and images at different scales can be considered as different views. Therefore, our model uses multi-scale images as input. By using multi-scale input, we obtain views at different scales, compensating for the lack of contextual information between pixels in hyperspectral images, enriching the information in the feature space, and thus improving the model's classification accuracy and generalization ability.

[0088] In the process of deriving multi-scale pixel blocks from hyperspectral data, different convolutional layers are used to process the multi-scale pixel blocks. (i, j) is the center pixel of the multi-scale input image block. For example... Figure 2 As shown, n1, n2, and n3 represent the sizes of pixel blocks with different scales divided from the input hyperspectral data. The specific steps are as follows:

[0089] (1) Based on the center pixel, obtain image patches of different sizes (n1, n2, n3) as input to the neural network;

[0090] (2) Multi-scale image patches are input into convolutional layers of corresponding sizes and outputs of the same size (n'×n'×d') are obtained;

[0091] (3) Concatenate the output results.

[0092] 3. Rule Constraint Generation Based on Deconvolutional Networks

[0093] The concatenated feature maps are used as input to the network and processed through a series of convolutional and fully connected layers to extract feature vectors. To evaluate the quality of the extracted features, the original scale input data is reconstructed using deconvolution, and the mean squared loss error calculation rule is applied as a constraint. This constraint compares the multi-scale input data and the reconstructed data to evaluate the information integrity in the model's learning, aiming to retain more effective details in the shared subspace. The specific steps are as follows:

[0094] (1) Input the spliced ​​feature map into a multi-layer convolutional neural network to obtain the feature vector, and use convolutional layers and fully connected layers to transform the n'×n'×d' feature map into a 1'×1'×f' feature vector;

[0095] (2) Input the feature vectors into the deconvolution layer to reconstruct a feature map with the same scale as the input image patch;

[0096] (3) Calculate the loss value between the input image patch and the feature map generated by deconvolution using the mean squared loss function (MSE). This can be expressed by the formula:

[0097]

[0098] It is the original input data of the basic feature extraction network, X′ j Θ is the reconstructed image output by the deconvolution layer, where m is the scale of different images, and Θ is the reconstructed image. R It represents rules and constraints.

[0099] 4. To further utilize graph-based data, the network model introduces relationships between surrounding pixels, which typically belong to the same category and are called superpixels. Superpixel constraint information can further optimize the network's iterative optimization process, avoiding the salt-and-pepper noise problem caused by pixel-based classification. This constraint will be automatically updated based on the model's optimization and feedback information. This irregular constraint is mainly generated based on the mean-shift clustering algorithm. First, a high-dimensional sphere is created by fitting the mean-shift model using feature vectors. Then, the high-dimensional sphere is shifted according to the objective function of minimizing mean-shift to generate superpixels. Each superpixel obtained by the clustering model can be represented by the average value of its internal spectral feature vectors. Furthermore, the irregular constraint loss established by the average spectral features can be expressed by formula (2).

[0100]

[0101] in, This is the output value of the last layer in the basic feature extraction network, where i represents the i-th superpixel. σ is the average spectral feature of the i-th superpixel, σ is the balance parameter, and t is the total number of superpixels.

[0102] The specific steps for generating irregular constraints based on superpixel clustering are as follows:

[0103] (1) Perform mean-shift clustering on the image based on the obtained feature vectors to obtain superpixels;

[0104] (2) Calculate the average spectral features within each superpixel using the clustering results;

[0105] (3) The result of irregular constraint is calculated by using eigenvectors and average spectral features. The calculation method can be expressed by formula (2).

[0106] 5. Furthermore, training a reliable hyperspectral image classification model in practical applications requires a large number of labeled samples, which incurs significant time and manpower costs. To address this, we utilize the comparison between multi-scale input patches and feature maps generated by deconvolution to generate regular constraints; irregular constraints are constructed based on the superpixel information of the image data. The image data contains both labeled and unlabeled samples, and feedback information is generated through constraint loss. Neither loss calculation involves labeled samples. This constraint method allows the model to be trained with fewer labeled samples and to classify hyperspectral images in a semi-supervised manner. The steps for introducing unlabeled samples are as follows:

[0107] (1) The loss value is calculated using rule constraints, without the participation of sample labels;

[0108] (2) Irregular constraints are used to introduce full-map data, including unlabeled samples, during mean-shift clustering.

[0109] 6. To obtain more discriminative features and generate classification results for each input sample, the feature results output by the network model are input into the softmax layer. This end-to-end classification network can perform supervised classification of hyperspectral images and provide a clear objective for network training. The loss value for supervised classification is obtained through the cross-entropy loss function. The supervised loss value is weighted and added to the previously obtained regular and irregular constraint loss values ​​to obtain the overall loss function of the network model, as shown in formula (3):

[0110]

[0111] Where λ1, λ2, and λ3 are the balancing parameters for regular constraints, irregular constraints, and cross-entropy loss, respectively, and these parameters are responsible for adjusting the weight of each constraint. j is the scale of the input data, and n is the number of predicted samples. The loss function is constructed through three constraints. The framework employs the Adam optimizer and backpropagation learning. Through backpropagation, the weights and biases of each layer are updated. Other adjustable parameters are changed based on the results of each experiment until the model achieves optimal classification accuracy. The specific steps are as follows:

[0112] (1) Calculate the cross-entropy loss value based on supervised classification;

[0113] (2) Perform a weighted summation of the regular loss value, the irregular loss value, and the cross-entropy loss value;

[0114] (3) Calculate the loss value during model learning and iteratively update the network parameters.

[0115] 7. Train the model and use it for image classification:

[0116] Determine the number of image categories, select approximately 50 pixels from each category for labeling, and iteratively train the model. During training, perform tests and scoring at regular intervals, and save the model with the highest accuracy. The specific steps are as follows:

[0117] The loss function Θ is then fed into the neural network for backpropagation to update and optimize the network's parameters, resulting in an optimized neural network.

[0118] The hyperspectral image data is input into an optimized neural network. The neural network extracts features from the hyperspectral image data and classifies the features based on accumulated experience, such as... Figure 3 As shown.

[0119] This invention enhances feature extraction capabilities through multi-scale convolutional layer inputs and additional constraints, while leveraging irregular constraints to preserve the advantages of graph-based semi-supervised learning in graph convolutional networks. Ultimately, this achieves improved accuracy in hyperspectral image classification and a significant reduction in discrete misclassification noise.

[0120] This model supplements relevant information through multi-scale inputs. By fusing multi-scale images and extracting feature vectors, information from different viewpoints in the shared space can more realistically represent ground objects. Furthermore, the model integrates self-supervised rule constraints and clustering-based irregular constraints. Rule constraints reconstruct multi-scale images through deconvolution, evaluating the completeness of information and preserving more important details during feature extraction. Irregular constraints introduce superpixel-based neighbor-pixel relationships, mitigating the salt-and-pepper noise problem caused by pixel-level classification and further improving the network's learning ability. Moreover, the iteratively optimized superpixel clustering results allow unlabeled samples to participate in the network's learning process, achieving a semi-supervised effect and reducing the need for labeled samples.

[0121] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A semi-supervised hyperspectral image classification method based on regular constraints and irregular constraints, characterized in that: include: Data preprocessing of hyperspectral images; Multi-scale image input fusion based on neural networks generates stitched image patches; According to the spliced image block, a rule loss value is calculated based on rule constraint generation of a deconvolution network ; The method for generating rule constraints based on deconvolutional networks is as follows: S21, taking the spliced image block as an input of a convolutional neural network to obtain a feature vector ; S22, utilizing the eigenvector obtaining a feature map with the same size as the input image block through a deconvolution layer ; S23, calculating the input image block by a mean square loss function ; S24, compute between the values of the loss value, denoted as the rule loss value ; Rule loss value The calculation method is: Irregular constraint generation based on superpixel clustering, compute irregular loss value ; Irregular constraint generation based on superpixel clustering, calculate irregular loss value The method is: S31, based on the obtained feature vector Mean shift clustering is performed on the image; S32, calculate the average spectral feature within each superpixel using the clustering result ; S33、by the eigenvector and average spectral features The difference between the L2 norm square , the resulting results divided by the balance parameter After averaging all superpixels, the irregular constraint loss value is obtained, which is recorded as the irregular loss value : Unlabeled sample introduction and semi-supervised learning based on superpixel constraints; Utilizing the acquired regular loss value and an irregular loss value constructing a loss function ; The loss function is brought into the neural network for back propagation, the parameters of the neural network are updated and optimized, and an optimized neural network is obtained. The hyperspectral image data is input into the optimized neural network, which extracts features from the hyperspectral image data and classifies the features based on accumulated experience. 2.The method of claim 1, wherein: The data preprocessing method for the hyperspectral image is as follows: Outliers are eliminated and normalized before the hyperspectral image data is processed by the input network. 3.The method of claim 2, wherein: The specific methods for eliminating outliers are as follows: Outliers in hyperspectral image data are detected and corrected by interpolation calculations to replace them. 4.The method of claim 3, wherein: The specific method for normalization is as follows: The values ​​of the hyperspectral image data are scaled proportionally to uniformly map them to the [0, 1] interval.

5. The method of claim 1, wherein the method is based on rule constraints and irregular constraints. The neural network-based multi-scale image input fusion method is as follows: S11. Obtain image patches of different sizes as input to the neural network; S12. Input image patches of different sizes into convolutional layers of different sizes respectively, and obtain outputs of the same size; S13. The output results are stitched together to obtain the stitched image blocks. 6.The method of claim 1, wherein: The introduction of unlabeled samples based on superpixel constraints and semi-supervised learning are specifically as follows: S41. Calculate the loss value using rule constraints, without the participation of sample labels; S42. Irregular constraints are used to introduce full-graph data, including unlabeled samples, during mean-shift clustering.

7. The method of claim 1, wherein the method is based on rule constraints and irregular constraints. using the obtained regular loss value and an irregular loss value constructing a loss function The method is as follows: S51, calculate a cross-entropy loss value based on supervised classification ; S52, the regular loss value , the irregular loss value , and the cross-entropy loss value are weighted and summed to obtain the loss function ; ; where, , , are balance parameters of the regular constraint, the irregular constraint, and the cross-entropy loss, respectively.

Images