A hyperspectral image classification method based on large visual model

Abstract

The application discloses a hyperspectral image classification method based on a large visual model. A spatial feature consistency constraint module is designed to shorten the spatial distance between existing labels and generated pseudo labels. The mask is updated according to the point prompt update. The confidence of the mask is improved, so that reliable pseudo labels are generated. In addition, in order to integrate spectral information, a spectral voting strategy is proposed to capture segmentation maps from different spectral bands, and an extreme constraint is used to determine the final mask. Thanks to the spectral voting mechanism, SAM suitable for processing natural images is successfully applied to hyperspectral image classification. The finally generated pseudo labels are combined with the existing labels and input into a 2DCNN classification network to complete the hyperspectral image classification task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of remote sensing image processing, and particularly relates to a hyperspectral image classification method based on a large visual model. BACKGROUND

[0002] In the field of earth observation and remote sensing, the importance of hyperspectral images is increasingly prominent. Compared with conventional color images, hyperspectral images can provide more abundant and detailed spectral information of ground objects by collecting a large number of continuous band information. The richness of this information makes hyperspectral images have a wide application prospect in land cover classification, environmental monitoring, agricultural management and other fields.

[0003] However, the hyperspectral image classification task also faces many challenges and opportunities. First, the data dimension of hyperspectral images is usually high, and each pixel point contains hundreds or even thousands of bands, which leads to the complexity and dimension disaster of data, bringing challenges to feature extraction and classification. Second, traditional methods based on spectral feature extraction and hand-crafted features often perform poorly when facing hyperspectral images, because the complexity and diversity of spectral features make the process of hand-crafted features very cumbersome, and are also limited by the limitations of manual feature extraction. Under the deficiency of traditional methods, in recent years, with the development of deep learning technology, especially the rise of convolutional neural networks (CNN), the hyperspectral image classification task has ushered in new opportunities. CNN can automatically learn feature representation, avoiding the defects of hand-crafted features in traditional methods, so it has achieved significant performance improvement in hyperspectral image classification tasks. However, although deep learning methods perform well in hyperspectral image classification, there is still a common problem, that is, a large amount of labeled data is needed to train the model, and the cost of obtaining large-scale labeled data is often very high in practice.

[0004] Under such background, semi-supervised learning is proposed as one of the effective ways to solve the problem of insufficient labeled data. Semi-supervised learning can effectively expand the existing labeled data set by using unlabeled data to improve model performance, thereby improving the generalization ability of the model. In the task of hyperspectral image classification, the application of semi-supervised learning also shows great potential. Initially, self-supervised learning is used to generate pseudo labels, thereby increasing the available labeled data. Although the classification accuracy is improved by generating pseudo labels, and the sample label expansion is realized. But still face the challenge of low reliability of generated pseudo labels.

[0005] Large language models have achieved remarkable success in the field of natural language processing. For example, models such as GPT-3 with billions of parameters have achieved success in the field of zero / few-shot learning. In the field of computer vision, pre-trained large visual-language models CLIP have demonstrated strong zero-shot generalization performance in various downstream visual tasks. In addition, the SegmentAnything Model (SAM) can perform class segmentation given an image and visual cues such as boxes, points, or masks. However, due to the particularity of hyperspectral images, these large visual models cannot achieve satisfactory results when directly applied to hyperspectral classification. SUMMARY

[0006] In order to enable large visual models to be successfully referenced in the classification of hyperspectral images, the present application proposes a hyperspectral image classification method based on a large visual model, a spatially guided iterative prompt selection framework based on the large visual model SAM to optimize the selection of pseudo-labels. Specifically, we input the hyperspectral image into the large visual model SAM in a channel manner, and use the prompt "point prompt" to expand the label in a semi-supervised scenario. Subsequently, we optimize the label estimation based on the spatial feature distance. We simultaneously use a spectral channel voting method to iteratively update the generated pseudo-labels to participate in the subsequent CNN network training.

[0007] The present application proposes a hyperspectral image classification method based on a large visual model, and the specific scheme is as follows:

[0008] Firstly, by converting the hyperspectral image into a three-channel image queue, the classification prediction of the hyperspectral image is processed by using the large visual model SAM, and an initial classification mask image queue is generated;

[0009] Secondly, for the mask image queue output by the large visual model SAM, a spectral voting strategy is proposed, and for each pixel, the information of multiple spectral channels is used for classification, and then the final classification result is determined by voting, so as to realize label class unmixing, generate a unique prediction mask, and obtain the pseudo-label with high confidence in the current round;

[0010] Thirdly, by combining spatial information, the distance between the generated pseudo-label and the real label in the feature space is constrained, and for each pixel, similarity calculation is performed according to the spatial feature and the label space to obtain a confidence score. The updated pseudo-label is input into the large visual model SAM to perform mask prediction in the next round;

[0011] Fourthly, the pseudo-label obtained in the third step is combined with the small amount of labeled labels to input into the 2DCNN classification network for hyperspectral image classification task.

[0012] The present application has the following beneficial technical effects:

[0013] The present application solves the predicament that it is difficult to generate high-confidence pseudo labels in the semi-supervised learning method, and effectively realizes sample expansion. By introducing a spectral voting strategy, the large visual model SAM is successfully embedded into the framework, and the structural prior information of the large model is fully utilized; by spatial information consistency constraint, the spatial distance between the pseudo label and the class label is narrowed, and the reliability of generating the pseudo label is further improved, providing data support for subsequent semi-supervised hyperspectral classification. BRIEF DESCRIPTION OF DRAWINGS

[0014] Fig. 1 is a general flowchart of the present application;

[0015] Fig. 2 is a general framework diagram proposed by the present application;

[0016] Fig. 3 is a schematic diagram of the large visual model SAM processing hyperspectral images to form a mask. DETAILED DESCRIPTION

[0017] In this part, the execution steps of the proposed method and the implementation details of each component will be explained. First, as shown in Figure 1 The hyperspectral image classification method based on the large visual model proposed by the present application includes the following steps:

[0018] First, by converting the hyperspectral image into a three-channel image queue, the large visual model SAM can process the classification prediction of the hyperspectral image, and generate an initial classification mask image queue;

[0019] Second, for the mask image queue output by the large visual model SAM, a spectral voting strategy is proposed. For each pixel, use the information of multiple spectral channels for classification, and then determine the final classification result by voting, so as to realize label class unmixing, generate a unique prediction mask, and obtain high-confidence pseudo labels for the current round;

[0020] Third, combined with spatial information, by constraining the distance between the generated pseudo label and the real label in the feature space, for each pixel, similarity calculation is performed according to the spatial feature and the label space to obtain a confidence score. The updated pseudo label is input into the SAM model to perform the next round of mask prediction;

[0021] Fourth, the pseudo label obtained in the third step is combined with the small amount of labeled labels to input into the 2DCNN classification network to perform the hyperspectral image classification task.

[0022] The general framework diagram is as follows Figure 2As shown, it can be embedded as a plug-in module before a hyperspectral classification network to achieve seamless information transfer. Specifically, the input hyperspectral image is first segmented into multiple three-channel images and then fed into the SAM model. A limited amount of annotation information is used as point cue information stimulus to initialize mask prediction. Mask prediction is based on the multi-channel images. Spectral voting is used to normalize the prediction. A subset of high-confidence labels is selected as cue input, and pixel features generated by SAM are used to strengthen consistency constraints and optimize the estimation of pseudo-labels.

[0023] First, the hyperspectral image is decomposed into a series of three-channel images, grouped according to spectral channels, so that the hyperspectral image can be integrated into the SAM image encoder module for segmentation purposes. The input image is defined as... ,in, These represent the image's width, height, and number of channels, respectively. We also create an image queue, where a three-channel image serves as the basic unit. By treating the three channels as a single batch, the final result was... Batch processing. A queue of spectral images is matched against labels for each class in the training set, using these labels as point cues for SAM initialization. Following a class-based queue processing approach, the current class label is treated as foreground, while the remaining class labels are considered as background for predicting the current class mask. A confidence threshold is set; labels with confidence scores above the threshold are designated as foreground, and this iterative process continues, assigning the remaining labels as background to optimize label generation. (Point-based processing...) For example, the threshold is as follows:

[0024] (1)

[0025] in, It is a category The predicted samples in It is the confidence level of the sample. and This is the confidence threshold. From this equation, we can obtain... Indicates prospects, Indicates the background. When using a hyperspectral image... Extending along the spectral dimension When iterating and predicting masks, class confusion may occur. To better utilize spectral information, a spectral voting strategy is proposed to address this problem. Assuming that the hyperspectral image contains... Each category, SAM prediction will result in a representation. The output tensor of the mask. For each pixel. and each category ,let Indicates in The number of binary classification predictions in the category. Specifically, if the prediction result of the th image in the current category is positive, then increase it by 1; otherwise, keep it as 0. Subsequently, calculate the total number of predictions for this pixel in all categories :

[0026] (2)

[0027] After rounds of processing, select the category with the highest total number of predictions as the final category for this pixel:

[0028] (3)

[0029] Among them, represents the final category of pixel . This process ensures that each pixel is predicted times for each category and is finally assigned to the category with the highest total number of predictions.

[0030] To better optimize the point prompt, a spatial information consistency constraint module is designed. Taking the category label generation process as an example. The feature representation of the point prompt is . The feature representation of sample is . Calculate its similarity with the prompt point feature.

[0031] (4)

[0032] Among them, is the similarity between and , is the feature of the category label. The confidence of sample is the average of its similarities with all labels, expressed as:

[0033] (5)

[0034] Aims to minimize the distance between the generated label and the existing labels in this category, while maximizing the separation from other categories. Therefore, binary classification loss is used to optimize the current pseudo-label generation:

[0035] (6)

[0036] Among them, indicates whether to use the pseudo-label to train the network.

[0037] (7)

[0038] (8)

[0039] wherein, is calculated as uncertainty, is a threshold value of positive sample uncertainty. When the uncertainty falls below the threshold value, it indicates that the pseudo label is reliable, The label can be used. As Figure 3 shown, an example of two classes in a hyperspectral image generating a mask through the SAM architecture iteration. It can be seen that the generated mask is more reliable through continuous iteration, and finally the high-confidence pseudo label in the generated mask is integrated into the classification network. The classification network architecture adopts a 2D CNN structure to complete the whole image segmentation prediction process.

Claims

1. A hyperspectral image classification method based on a large visual model, characterized in that, The method comprises the following steps: The first step is to process the classification prediction of the hyperspectral image by converting the hyperspectral image into a three-channel image queue and using a large visual model SAM, to generate an initial classification mask image queue; The second step is to propose a spectral voting strategy for the classification mask image queue output by the large visual model SAM, to classify each pixel using information of multiple spectral channels, and then determine the final classification result by voting, so as to realize label class unmixing, generate a unique prediction mask, and output a high-confidence pseudo label; The third step is to combine spatial information, constrain the distance between the generated pseudo label and the real label in the feature space, and for each pixel, calculate the similarity between the spatial feature and the label space to obtain a confidence score, and input the updated pseudo label into the large visual model SAM for the next round of mask prediction; The fourth step is to input the pseudo label obtained in the third step and the small amount of labeled labels into a 2DCNN classification network for a hyperspectral image classification task.

2. The hyperspectral image classification method based on large visual model according to claim 1, characterized in that: In the first step, the converted three-channel image queue is input into the large visual model SAM for classification prediction, and the classification label of each pixel is determined according to the probability distribution or threshold output by the model, including: For a hyperspectral image, three-channel images are synthesized according to the spectral sequence every three channels, and the large visual model SAM reads the three-channel images; Each class label is input into the large visual model SAM as a point prompt for the current class for mask prediction; According to the label confidence level, the high-confidence pseudo label of the current round is obtained as a point prompt for the next round.

3. The hyperspectral image classification method based on large visual model according to claim 1, characterized in that: In the second step, the spectral voting strategy is used to classify the pixels using information of multiple bands, and the final classification result is determined by voting, including: For each class, a binary classification mask is generated, and a mask is generated every three channels; All class masks are queued and voted, and the number of class predictions for each pixel is summed up; Finally, a mask image containing complete class predictions is generated, and a high-confidence pseudo label is output.

4. The hyperspectral image classification method based on large visual model according to claim 1, characterized in that: In the third step, the distance between the generated pseudo label and the real label in the feature space is constrained, including: The real label and the pseudo label are mapped to the feature space, and the distance between them is calculated using the cosine similarity distance measurement method; The similarity between the spatial feature and the label space is calculated to obtain the confidence score of each pixel; The spatial similarity of the current class is used as a loss function to learn the point prompt encoder in the large visual model SAM to narrow the class and update the point prompt.

5. The hyperspectral image classification method based on large visual model according to claim 1, characterized in that: In the fourth step, the pseudo label is combined with the small amount of labeled labels to input into the 2DCNN classification network for a hyperspectral image classification task, including: A 2D convolutional neural network 2DCNN is established, which takes a hyperspectral image as input and outputs a prediction result for classifying the image; The merged label dataset, including a small amount of real labels and pseudo labels, is used to train the 2DCNN network, and the network parameters are adjusted by minimizing the loss function; The trained network is evaluated on the performance on the hyperspectral image classification task using a validation or test set.

Images