High-spectral zero-shot classification method and system based on visual language model for large scene

Abstract

The application discloses a large-scene hyperspectral zero-sample classification method and system based on a visual language model, and comprises the following steps: generating an initial pseudo-label image based on a visual language model for a hyperspectral image; calculating the spectral angle between each pixel in the initial pseudo-label image and a spectral fingerprint vector, and filtering the initial pseudo-label image based on the spectral angle to obtain an optimized pseudo-label image; and training a hyperspectral segmentation network based on the hyperspectral image and the optimized pseudo-label image, so that a mapping relationship between the hyperspectral image and the ground object categories thereof is established to realize the ground object category classification of the hyperspectral image, wherein the hyperspectral segmentation network is composed of an encoder, a decoder and a classifier, and the encoder is composed of N-level cascaded multi-core convolution residual modules. The application aims to accurately identify and segment different ground object categories in a complex hyperspectral image without manual annotation.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral remote sensing image classification technology, specifically to a large-scene hyperspectral zero-shot classification method and system based on a visual language model. Background Technology

[0002] Compared with traditional visible light or multispectral images, hyperspectral imaging technology can acquire imaging data of dozens or even hundreds of narrow bands within a continuous electromagnetic spectrum. This extremely high spectral resolution can not only capture the spatial geometry of the target area, but also deeply reflect the extremely subtle physical properties and chemical composition differences of the surface and interior of the ground objects, just like drawing a unique "spectral fingerprint" for various materials. With this advantage, hyperspectral image interpretation technology has been widely and deeply applied to applications such as fine-grained ground object segmentation in complex urban scenes, large-scale dynamic monitoring of ecological environment, exploration and investigation of hidden mineral resources, and assessment of vegetation health status. In order to improve the classification accuracy of large-scene hyperspectral images, a large number of deep learning-based methods have been proposed. These methods have effectively improved the accuracy of scene interpretation. However, existing hyperspectral image classification technology still faces two major challenges: (1) Traditional deep learning methods rely heavily on the fully supervised learning paradigm, and model training requires a lot of manpower to obtain high-precision pixel-level real labels, which is extremely costly; (2) Although the visual language models introduced in recent years have zero-sample generalization ability, they are mainly based on visible light images for pre-training. When applied directly to hyperspectral data, the multi-band physical properties cannot be fully utilized, resulting in a large number of misclassified regions in the generated pseudo-labels. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a large-scene hyperspectral zero-sample classification method and system based on a visual language model, which addresses the above-mentioned problems of the prior art. The present invention aims to accurately identify and segment different land cover categories in complex hyperspectral images without the need for manual annotation.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A large-scene hyperspectral zero-shot classification method based on a visual language model includes the following steps: S101 is the initial pseudo-label map generated for the hyperspectral image based on the visual language model; S102, calculate the spectral median vector of pixels under each land cover category in the hyperspectral image as the spectral fingerprint vector, calculate the spectral angle of each pixel in the initial pseudo-label image with the spectral fingerprint vector of its initial land cover category, and filter the initial pseudo-label image based on the spectral angle to obtain the optimized pseudo-label image. S103, a hyperspectral segmentation network is trained based on hyperspectral images and their median spectral fingerprints to optimize pseudo-label maps, thereby establishing a mapping relationship between hyperspectral images and their land cover categories for land cover category classification of hyperspectral images.

[0005] Optionally, step S101 includes: S201, acquire the hyperspectral image to be identified and extract its corresponding synthetic RGB band image, and at the same time acquire the set of land cover category names of the target scene; S202, input the synthesized RGB band image into the pre-trained visual language model, and extract the image feature vector through the visual encoder in the visual language model; S203, input the set of land cover category names into the text encoder of the visual language model to extract semantic feature vectors for each category; S204 calculates the original confidence score of each pixel belonging to each predefined category by cross-modal alignment of semantic feature vector and image feature vector, and generates the initial pseudo-label map according to the confidence maximization principle.

[0006] Optionally, the step of calculating the original confidence score of each pixel belonging to each predefined category through cross-modal alignment of semantic features and image features, and generating an initial pseudo-label map according to the confidence maximization principle includes: S301, for each category name in the set of land cover category names, obtain the pixel representation of that category name output by the visual language model. Semantic mask probability and global existence score at the location; obtain the instance mask probability of several independent target objects output by the segmentation head in the visual language model, and the corresponding instance score output by the scoring module; S302, multiply the instance mask probability of each independent target object by the corresponding instance score, and perform a pixel-by-pixel maximum value operation on the semantic mask probability mentioned above. Multiply the maximum value result by the global existence score of the category name to calculate the query-level confidence score. : ; In the above formula, To perform the maximum value operation, For the q-th category name in pixels Semantic mask probability at the location, This is a maximum value operation applied to all independent target objects. For the qth category name, the first The instance mask probability of an independent target object For the qth category name, the first The score for each individual target object instance. The global existence score for the q-th category name; S303, based on predefined land cover categories Grouping and mapping category names: mapping to the same category The feature category of each pixel is calculated by taking the maximum value for each pixel from the confidence scores of all query texts. raw confidence level : ; In the above formula, Indicates mapping to land cover categories The collection of all synonym query texts; S304, Obtain pixels that represent image features. In all predefined land cover categories The maximum confidence level is determined; when the maximum confidence level is greater than or equal to a preset probability threshold, the pixel is assigned to the land cover category corresponding to the maximum confidence level; when the maximum confidence level is less than the preset probability threshold, the pixel is marked as the background category, thereby generating an initial pseudo-label map.

[0007] Optionally, step S102 includes: S401, traverse each land cover category in the initial pseudo-label image and calculate the spectral median vector of the pixels under each land cover category as the spectral fingerprint vector. S402, Calculate the spectral angle of each pixel in the initial pseudo-label image and its spectral fingerprint vector under its initial land cover category: ; In the above formula, It is the spectral angle. For the spectral vector of a pixel, This is the spectral fingerprint vector of the land cover category to which the pixel belongs; S403: For each land cover category, calculate the ratio of the number of valid pixels with spectral angles less than the threshold to the initial number of pixels in that land cover category. If the ratio is lower than the preset protection threshold, no filtering operation is performed for that land cover category; otherwise, the filtering operation is performed to remove pixels with spectral angles less than the preset protection threshold for that land cover category, thereby obtaining the optimized pseudo-label image.

[0008] Optionally, step S103 includes: reducing the dimensionality of the hyperspectral image; training a hyperspectral segmentation network based on the dimensionality-reduced hyperspectral image and its median spectral fingerprint to optimize the pseudo-label map, thereby establishing a mapping relationship between the hyperspectral image and its land cover categories for land cover category classification of the hyperspectral image; the hyperspectral segmentation network consists of an encoder, a decoder, and a classifier, wherein the encoder consists of N-level cascaded multi-kernel convolutional residual modules, and the output of each level of convolutional residual module is connected in series with a max pooling module; the decoder consists of N-level cascaded multi-kernel convolutional attention modules, and the output of each level of convolutional attention module is connected in series with an upsampling module; and after the upsampling module of the first N-1 levels of multi-kernel convolutional attention modules, a gated fusion attention module is connected in series, the gated fusion attention module is used to perform gated fusion of the output features of the upsampling module and the output features of the max pooling module of the corresponding level of multi-kernel convolutional residual module, and the classifier is used to classify the feature map obtained by the decoder to obtain the final classification result map.

[0009] Optionally, the processing of input features by the multi-kernel convolutional residual module includes: sequentially passing the input features through 1×1 convolution, batch normalization (BN), ReLU activation, multi-kernel convolution module, 1×1 convolution, batch normalization (BN), and then performing residual addition with the original input features of the multi-kernel convolutional residual module to obtain the output features of the multi-kernel convolutional residual module. The processing of input features by the multi-kernel convolution module includes: processing the input features of the multi-kernel convolution module in parallel using three branches, each of which is activated by convolution, batch normalization (BN), and ReLU activation. The convolution kernel sizes in the three branches are 1×1, 3×3, and 5×5, respectively. The output features obtained from the three branches are then fused element-wise and shuffled through channels to promote information interaction to obtain the output features of the multi-kernel convolution module.

[0010] Optionally, the multi-kernel convolutional attention module consists of a spectral attention module, a spatial attention module, and a multi-kernel convolutional residual module connected in sequence. The spectral attention module processes the input features by performing parallel processing on two branches. One branch performs max pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the first-path features, while the other branch performs average pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the second-path features. The two-path features are then superimposed and processed by the Sigmoid function to obtain the spectral feature weights. The spectral feature weights are then multiplied by the input features of the spectral attention module to obtain the output features of the spectral attention module. The spatial attention module processes the input features by performing max pooling and average pooling on the input features. The features obtained from max pooling and average pooling are then superimposed and processed by 7×7 convolution and the Sigmoid function to obtain the spatial feature weights. The spatial feature weights are then multiplied by the input features of the spatial attention module to obtain the output features of the spatial attention module.

[0011] Optionally, the gating fusion attention module's processing of input features includes: processing the decoded features of the current layer... Corresponding skip connection features from the encoder Parallel processing is performed using two branches, each executing 3×3 convolution and batch normalization (BN). The features obtained from the two branches are then stacked and sequentially activated by ReLU, followed by 1×1 convolution, BN, and a sigmoid function, before being used as a gating mask. Then the gate mask Corresponding skip connection features from the encoder Element-wise multiplication, plus the original skip connection features from the encoder. The output features of the gated fusion attention module are obtained.

[0012] Optionally, when training the hyperspectral segmentation network based on the hyperspectral image and its median spectral fingerprint to optimize the pseudo-label map, the learning rate is dynamically adjusted using a cosine annealing strategy during the iterative update of the hyperspectral segmentation network. ; In the above formula, Let be the learning rate at the t-th iteration. and These are the maximum and minimum values ​​of the learning rate, respectively. This represents the current iteration number. This represents the maximum number of iterations.

[0013] Furthermore, the present invention also provides a large-scene hyperspectral zero-shot classification system based on a visual language model, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the large-scene hyperspectral zero-shot classification method based on the visual language model.

[0014] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The method of the present invention includes using a visual language model and a multi-kernel convolutional network to obtain classification results of large-scene hyperspectral images. The method for classifying large-scene hyperspectral images includes the following steps: generating an initial pseudo-label map based on a visual language model; optimizing the pseudo-label map based on median spectral fingerprints; and training a multi-kernel hyperspectral segmentation network based on refined pseudo-labels to obtain classification results. The present invention utilizes the zero-shot prior knowledge of the visual language model, eliminating the dependence on a large amount of manually labeled data; by introducing the physical characteristics of hyperspectral data, it uses median spectral fingerprints to perform physical consistency filtering on pseudo-labels, effectively solving the semantic drift problem of visual language models in remote sensing scenarios; at the same time, the present invention leverages the advantages of multi-kernel convolutional networks in multi-scale feature extraction, which can better simulate the edges and details of different land features in remote sensing images, thereby obtaining better land feature classification accuracy. The present invention can accurately identify different land feature categories in large-scene hyperspectral images without manual annotation. The present invention combines the powerful zero-shot prior knowledge of the visual language model with the physical characteristics of hyperspectral data, and uses median spectral fingerprints to perform physical consistency filtering on initial pseudo-labels. Meanwhile, this invention leverages the advantages of multi-kernel convolutional networks in multi-scale feature extraction, completing self-supervised iteration of the network based on refined pseudo-labels. This method can accurately identify and segment different land cover categories in complex hyperspectral images without requiring manual annotation. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the network structure of the hyperspectral segmentation network in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the network structure of the multi-core convolutional residual module in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the network structure of the multi-core convolution module in an embodiment of the present invention.

[0019] Figure 5 This is a schematic diagram of the network structure of the multi-kernel convolutional attention module in an embodiment of the present invention.

[0020] Figure 6 This is a schematic diagram of the network structure of the spectral attention module in an embodiment of the present invention.

[0021] Figure 7 This is a schematic diagram of the network structure of the spatial attention module in an embodiment of the present invention.

[0022] Figure 8 This is a schematic diagram of the network structure of the gated fusion attention module in an embodiment of the present invention.

[0023] Figure 9 Examples of hyperspectral images and label images in the PaviaC dataset in this embodiment of the invention are shown, where (a) is a hyperspectral image and (b) is a label image.

[0024] Figure 10 The figures show a comparison of the visualization results of the method of this embodiment and the existing method on the PaviaC dataset, where (a) is the visualization result of S2GCL, (b) is the visualization result of BGPC, (c) is the visualization result of SAPC, (d) is the visualization result of EGFSC, (e) is the visualization result of ESTMC, (f) is the visualization result of CeFMC, (g) is the visualization result of SegEarth, (h) is the visualization result of SegEarth3, (i) is the visualization result of SPECIAL, and (j) is the visualization result of the method of this embodiment.

[0025] Figure 11 Examples of hyperspectral images and label images in the GF5-CS dataset in this embodiment of the invention are shown, where (a) is a hyperspectral image and (b) is a label image.

[0026] Figure 12 The figures show a comparison of the visualization results of the method of this embodiment and the existing method on the GF5-CS dataset, where (a) is the visualization result of S2GCL, (b) is the visualization result of BGPC, (c) is the visualization result of SAPC, (d) is the visualization result of EGFSC, (e) is the visualization result of ESTMC, (f) is the visualization result of CeFMC, (g) is the visualization result of SegEarth, (h) is the visualization result of SegEarth3, (i) is the visualization result of SPECIAL, and (j) is the visualization result of the method of this embodiment. Detailed Implementation

[0027] This invention aims to leverage the zero-shot generalization capability of visual language models and the advantages of multi-kernel networks in multi-scale feature extraction to accurately identify different land cover categories in large-scale hyperspectral images without the need for manual annotation. To enable those skilled in the art to better understand the technical solution of this invention, the following will provide a more detailed description of the technical solution in conjunction with the accompanying drawings of the embodiments of this invention.

[0028] like Figure 1 As shown, the large-scene hyperspectral zero-shot classification method based on a visual language model in this embodiment includes the following steps: S101 is the initial pseudo-label map generated for the hyperspectral image based on the visual language model; S102, calculate the spectral median vector of pixels under each land cover category in the hyperspectral image as the spectral fingerprint vector, calculate the spectral angle of each pixel in the initial pseudo-label image with the spectral fingerprint vector of its initial land cover category, and filter the initial pseudo-label image based on the spectral angle to obtain the optimized pseudo-label image. S103, a hyperspectral segmentation network is trained based on hyperspectral images and their median spectral fingerprints to optimize pseudo-label maps, thereby establishing a mapping relationship between hyperspectral images and their land cover categories for land cover category classification of hyperspectral images.

[0029] The visual language model includes a visual encoder, a text encoder, and a segmentation head. The visual encoder and text encoder are used for visual encoding and text encoding, respectively. The segmentation head performs image segmentation based on the semantic feature vectors and image feature vectors obtained from the visual and text encoding. The visual language model can be chosen according to needs. For example, as an optional implementation, this embodiment uses the SAM3 (SegmentAnythingModel3) model, which is a cue-based segmentation model derived from SAM and SAM2SAM3.

[0030] In this embodiment, step S101 includes: S201, acquire the hyperspectral image to be identified and extract its corresponding synthetic RGB band image, and at the same time acquire the set of land cover category names of the target scene; S202, input the synthesized RGB band image into the pre-trained visual language model, and extract the image feature vector through the visual encoder in the visual language model; S203, input the set of land cover category names into the text encoder of the visual language model to extract semantic feature vectors for each category; S204 calculates the original confidence score of each pixel belonging to each predefined category by cross-modal alignment of semantic feature vector and image feature vector, and generates the initial pseudo-label map according to the confidence maximization principle.

[0031] In this embodiment, the step of calculating the original confidence score of each pixel belonging to each predefined category through cross-modal alignment of semantic features and image features, and generating the initial pseudo-label map according to the confidence maximization principle includes: S301, for each category name in the set of land cover category names, obtain the pixel representation of that category name output by the visual language model. Semantic mask probability and global existence score at the location; obtain the instance mask probability of several independent target objects output by the segmentation head in the visual language model, and the corresponding instance score output by the scoring module; S302, multiply the instance mask probability of each independent target object by the corresponding instance score, and perform a pixel-by-pixel maximum value operation on the semantic mask probability mentioned above. Multiply the maximum value result by the global existence score of the category name to calculate the query-level confidence score. : ; In the above formula, To perform the maximum value operation, For the q-th category name in pixels Semantic mask probability at the location, This is a maximum value operation applied to all independent target objects. For the qth category name, the first The instance mask probability of an independent target object For the qth category name, the first The score for each individual target object instance. The global existence score for the q-th category name; S303, based on predefined land cover categories Grouping and mapping category names: mapping to the same category The feature category of each pixel is calculated by taking the maximum value for each pixel from the confidence scores of all query texts. raw confidence level : ; In the above formula, Indicates mapping to land cover categories The collection of all synonym query texts; S304, Obtain pixels that represent image features. In all predefined land cover categories The maximum confidence level is determined; when the maximum confidence level is greater than or equal to a preset probability threshold, the pixel is assigned to the land cover category corresponding to the maximum confidence level; when the maximum confidence level is less than the preset probability threshold, the pixel is marked as the background category, thereby generating an initial pseudo-label map.

[0032] In this embodiment, step S102 is the spatial median filtering step, which includes: S401, traverse each land cover category in the initial pseudo-label image, calculate the spectral median vector of the pixels under each land cover category as the spectral fingerprint vector, which can be used as the global representative of the land cover category; S402, Calculate the spectral angle of each pixel in the initial pseudo-label image and its spectral fingerprint vector under its initial land cover category: ; In the above formula, It is the spectral angle. For the spectral vector of a pixel, This is the spectral fingerprint vector of the land cover category to which the pixel belongs; S403: For each land cover category, calculate the ratio of the number of valid pixels with spectral angles less than a threshold to the initial number of pixels in that category. If the ratio is lower than a preset protection threshold, no filtering operation is performed for that land cover category; otherwise, pixels with spectral angles less than the preset protection threshold are filtered out for that land cover category, thus obtaining an optimized pseudo-label image. The preset protection threshold can be selected as needed, for example, a value of 25%. If the ratio is lower than the preset protection threshold, no filtering operation is performed, and the initial prediction result is directly retained to maintain the stability of the category baseline; otherwise, pixels with spectral angles less than the preset threshold are filtered out for that category, thus obtaining an optimized pseudo-label image.

[0033] In this embodiment, step S103 includes: reducing the dimensionality of the hyperspectral image; training a hyperspectral segmentation network based on the dimensionality-reduced hyperspectral image and its median spectral fingerprint to establish a mapping relationship between the hyperspectral image and its land cover categories for the purpose of classifying land cover categories in the hyperspectral image.

[0034] like Figure 2 As shown, the hyperspectral segmentation network in this embodiment consists of an encoder, a decoder, and a classifier. The encoder is composed of N cascaded multi-kernel convolutional residual (MKCR) modules, and the output of each convolutional residual module is connected in series with a max pooling module. The decoder is composed of N cascaded multi-kernel convolutional attention modules, and the output of each convolutional attention module is connected in series with an upsampling module. Furthermore, a gated fusion attention module is connected in series after the upsampling module of the first N-1 multi-kernel convolutional attention modules. The gated fusion attention module is used to perform gated fusion of the output features of the upsampling module and the output features of the max pooling module of the corresponding multi-kernel convolutional residual module before outputting the result. The classifier is used to classify the feature map obtained by the decoder to obtain the final classification result map.

[0035] like Figure 3As shown, in this embodiment, the processing of input features by the multi-kernel convolutional residual module includes: sequentially passing the input features through 1×1 convolution, batch normalization (BN), ReLU activation, multi-kernel convolution module, 1×1 convolution, batch normalization (BN), and then adding the residuals to the original input features of the multi-kernel convolutional residual module to obtain the output features of the multi-kernel convolutional residual module, such as... Figure 4 As shown, the processing of input features by the multi-kernel convolution module in this embodiment includes: processing the input features of the multi-kernel convolution module in parallel using three branches, each branch being activated by convolution, batch normalization (BN), and ReLU activation functions, with convolution kernel sizes of 1×1, 3×3, and 5×5 respectively. The output features obtained from the three branches are then fused element-wise and shuffled through channels to promote information interaction, resulting in the output features of the multi-kernel convolution module. The element-wise fusion of the output features obtained from the three branches can be expressed as: ; In the above formula, As a feature of fusion, It is the ReLU activation function. For batch normalization, For the convolution of the k-th branch, The input features are those of the multi-kernel convolutional module. The extracted multi-scale features are fused element-wise, then channel-shuffled to promote information exchange. After dimensionality reduction via 1×1 convolution, the residuals are added to the module input to output the encoded features at each level. For coding features at each level Max pooling operation Spatial size compression is performed to obtain the corresponding downsampled features. Its expression is: ; Downsampling features It serves as input to the next level encoder and is also passed to the decoder as a feature of the skip connections.

[0036] During the decoding process, the downsampled features first pass through a multi-kernel convolutional residual attention (MKCRA) module. For example... Figure 5 As shown, in this embodiment, the multi-kernel convolutional attention module consists of a spectral attention module, a spatial attention module, and a multi-kernel convolutional residual module connected in sequence.

[0037] like Figure 6As shown, the spectral attention module processes the input features in parallel using two branches. One branch performs max pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the first set of features. The other branch performs average pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the second set of features. The two sets of features are then superimposed and processed by the Sigmoid function to obtain the spectral feature weights, which can be expressed as: ; in, For spectral feature weights, For the Sigmoid function, For 1×1 convolution, It is the ReLU activation function. and For average pooling and max pooling, The input features of the spectral attention module are denoted as ; the output features of the spectral attention module are obtained by multiplying the spectral feature weights by the input features, which can be expressed as: ; In the above formula, The input features for the spectral attention module, For spectral feature weights, The features are weighted by spectral attention, i.e., the output features of the spectral attention module.

[0038] like Figure 7 As shown, the spatial attention module processes the input features by performing max pooling and average pooling on the input features respectively. The features obtained from max pooling and average pooling are then stacked, processed by a 7×7 convolution, and then used as spatial feature weights, which can be expressed as: ; in, For spatial feature weights, It is a 7×7 convolution. This represents the channel concatenation operation; then, the spatial feature weights are multiplied by the input features of the spatial attention module to obtain the output features of the spatial attention module, which can be expressed as: ; In the above formula, These are the output features of the spatial attention module.

[0039] like Figure 8 As shown, in this embodiment, the Gated Fusion Attention (GFA) module processes the input features including: decoding the features of the current layer... Corresponding skip connection features from the encoder Parallel processing is performed using two branches, each executing 3×3 convolution and batch normalization (BN). The features obtained from the two branches are then stacked and sequentially activated by ReLU, followed by 1×1 convolution, BN, and a sigmoid function, before being used as a gating mask. Then the gate mask Corresponding skip connection features from the encoder Element-wise multiplication, plus the original skip connection features from the encoder. The output features of the gated fusion attention module can be represented as: ; ; in, This is the output feature of the gated fusion attention module. The multi-kernel convolutional attention and gated fusion process described above is repeated to progressively restore the feature map to the high spatial resolution size of the original input. Finally, the feature map is projected onto the category space through a classification layer, outputting the final land cover classification probability map: ; In the above formula, This is the final probability map for land cover classification. The final fused decoded feature map is aligned to the original image resolution.

[0040] In this embodiment, when training the hyperspectral segmentation network based on the hyperspectral image and its median spectral fingerprint to optimize the pseudo-label map, the learning rate is dynamically adjusted using a cosine annealing strategy during the iterative update of the hyperspectral segmentation network. ; In the above formula, Let be the learning rate at the t-th iteration. and These are the maximum and minimum values ​​of the learning rate, respectively. This represents the current iteration number. This represents the maximum number of iterations.

[0041] In this embodiment, when training the hyperspectral segmentation network based on the optimized pseudo-label map of the hyperspectral image and its median spectral fingerprint, the refined pseudo-label map is used as a supervision signal, and the segmentation model is iteratively updated using a loss function that minimizes the cross-entropy. The loss function... The function expression is: ; In the above formula, This represents the total number of valid pixels within the set. The true category value of the pseudo-label. This represents the predicted probability output by the hyperspectral segmentation network.

[0042] To validate the large-scene hyperspectral zero-shot classification method based on the visual language model in this embodiment, two large-scene hyperspectral datasets, the PaviaC dataset and the GF5-CS dataset, were used for validation experiments. Each dataset contains one hyperspectral image and one label image. Figure 9 Examples of hyperspectral images and label images in the PaviaC dataset in this embodiment of the invention are shown, where (a) is a hyperspectral image and (b) is a label image. Figure 11 Examples of hyperspectral and label images from the GF5-CS dataset in this embodiment of the invention are shown, where (a) is a hyperspectral image and (b) is a label image. Nine representative unsupervised classification methods—S2GCL, BGPC, SAPC, EGFSC, ESTMC, CeFMC, SegEarth, SegEarth3, and SPECIAL—were compared in the experiments. Five evaluation metrics were used: overall accuracy (OA), average accuracy (AA), Kappa coefficient, normalized mutual information (NMI), and adjusted Land coefficient (ARI). Higher values ​​indicate better classification performance. The final results are shown below. Figure 10 and Figure 12 As shown in the table. Based on this, Table 1 presents the objective evaluation metrics of nine representative unsupervised classification methods (S2GCL, BGPC, SAPC, EGFSC, ESTMC, CeFMC, SegEarth, SegEarth3, SPECIAL) and the method proposed in this embodiment on the PaviaC dataset. Table 2 presents the objective evaluation metrics of nine representative unsupervised classification methods (S2GCL, BGPC, SAPC, EGFSC, ESTMC, CeFMC, SegEarth, SegEarth3, SPECIAL) and the method proposed in this embodiment on the GF5-CS dataset.

[0043] Table 1: Objective performance metrics of the method in this embodiment and existing methods on the PaviaC dataset.

[0044] Table 2: Objective performance metrics of the method in this embodiment and existing methods on the GF5-CS dataset

[0045] As shown in Table 1, the large-scene hyperspectral zero-shot classification method based on the visual language model in this embodiment (the method in this embodiment) outperforms nine representative unsupervised classification methods (S2GCL, BGPC, SAPC, EGFSC, ESTMC, CeFMC, SegEarth, SegEarth3, SPECIAL) on the PaviaC dataset in terms of objective evaluation metrics. As shown in Table 2, the large-scene hyperspectral zero-shot classification method based on the visual language model in this embodiment (the method in this embodiment) outperforms nine representative unsupervised classification methods (S2GCL, BGPC, SAPC, EGFSC, ESTMC, CeFMC, SegEarth, SegEarth3, SPECIAL) on the GF5-CS dataset in terms of objective evaluation metrics. Therefore, the large-scene hyperspectral zero-shot classification method based on the visual language model in this embodiment utilizes the visual language model to generate initial pseudo-labels, eliminating the reliance on manually labeled data. Furthermore, this method designs an optimization mechanism based on median spectral fingerprints. This mechanism effectively solves the semantic drift problem of large models in remote sensing scenarios through physical consistency filtering. Meanwhile, this method constructs a multi-kernel convolutional network to extract multi-scale spatial features. Within this network, multi-kernel convolutional attention and gated fusion attention modules are applied to effectively fuse contextual features from different levels. This embodiment's large-scene hyperspectral zero-shot classification method based on a visual language model achieves good performance in large-scene hyperspectral image classification without the need for manual annotation.

[0046] Furthermore, this embodiment also provides a large-scene hyperspectral zero-shot classification system based on a visual language model, including a microprocessor and a memory interconnected, wherein the microprocessor is programmed or configured to execute the large-scene hyperspectral zero-shot classification method based on the visual language model.

[0047] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A large-scene hyperspectral zero-shot classification method based on a visual language model, characterized in that, Includes the following steps: S101 is the initial pseudo-label map generated for the hyperspectral image based on the visual language model; S102, Calculate the spectral median vector of pixels under each land cover category in the hyperspectral image as the spectral fingerprint vector, calculate the spectral angle of each pixel in the initial pseudo-label image with its initial land cover category's spectral fingerprint vector, and filter the initial pseudo-label image based on the spectral angle to obtain an optimized pseudo-label image, including: S401, Traverse each land cover category in the initial pseudo-label image and calculate the spectral median vector of pixels under each land cover category as the spectral fingerprint vector; S402, Calculate the spectral angle of each pixel in the initial pseudo-label image with its initial land cover category's spectral fingerprint vector: ； In the above formula, It is the spectral angle. For the spectral vector of a pixel, S403: For each land cover category, calculate the ratio of the number of valid pixels with spectral angles less than the threshold to the initial number of pixels. If the ratio is lower than the preset protection threshold, no filtering operation is performed for that land cover category. Otherwise, for that land cover category, the filtering operation filters out pixels with spectral angles less than the preset protection threshold, thereby obtaining an optimized pseudo-label image. S103 trains a hyperspectral segmentation network based on hyperspectral images and their optimized pseudo-label maps, enabling it to establish a mapping relationship between hyperspectral images and their land cover categories for land cover category classification of hyperspectral images.

2. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 1, characterized in that, Step S101 includes: S201, acquire the hyperspectral image to be identified and extract its corresponding synthetic RGB band image, and at the same time acquire the set of land cover category names of the target scene; S202, input the synthesized RGB band image into the pre-trained visual language model, and extract the image feature vector through the visual encoder in the visual language model; S203, input the set of land cover category names into the text encoder of the visual language model to extract semantic feature vectors for each category; S204 calculates the original confidence score of each pixel belonging to each predefined category by cross-modal alignment of semantic feature vector and image feature vector, and generates the initial pseudo-label map according to the confidence maximization principle.

3. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 2, characterized in that, The process of calculating the original confidence score of each pixel belonging to each predefined category through cross-modal alignment of semantic features and image features, and generating the initial pseudo-label map according to the confidence maximization principle includes: S301, for each category name in the set of land cover category names, obtain the pixel representation of that category name output by the visual language model. Semantic mask probability and global existence score at the location; obtain the instance mask probability of several independent target objects output by the segmentation head in the visual language model, and the corresponding instance score output by the scoring module; S302, multiply the instance mask probability of each independent target object by the corresponding instance score, and perform a pixel-by-pixel maximum value operation on the semantic mask probability mentioned above. Multiply the maximum value result by the global existence score of the category name to calculate the query-level confidence score. : ； In the above formula, To perform the maximum value operation, For the qth category name in pixels Semantic mask probability at the location, This is a maximum value operation applied to all independent target objects. For the qth category name, the first The instance mask probability of an independent target object For the qth category name, the first The score for each individual target object instance. The global existence score for the q-th category name; S303, based on predefined land cover categories Grouping and mapping category names: mapping to the same category From the confidence scores of all query texts, the maximum value is taken pixel by pixel to calculate the land cover category of each pixel. raw confidence level : ； In the above formula, Indicates mapping to land cover categories The collection of all synonym query texts; S304, Obtain pixels that represent image features. In all predefined land cover categories The maximum confidence level is determined; when the maximum confidence level is greater than or equal to a preset probability threshold, the pixel is assigned to the land cover category corresponding to the maximum confidence level; when the maximum confidence level is less than the preset probability threshold, the pixel is marked as the background category, thereby generating an initial pseudo-label map.

4. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 1, characterized in that, Step S103 includes: reducing the dimensionality of the hyperspectral image; training a hyperspectral segmentation network based on the dimensionality-reduced hyperspectral image and its median spectral fingerprint to optimize the pseudo-label map, thereby establishing a mapping relationship between the hyperspectral image and its land cover categories for land cover category classification of the hyperspectral image; the hyperspectral segmentation network consists of an encoder, a decoder, and a classifier, wherein the encoder consists of N-level cascaded multi-kernel convolutional residual modules, and the output of each level of convolutional residual module is connected in series with a max pooling module; the decoder consists of N-level cascaded multi-kernel convolutional attention modules, and the output of each level of convolutional attention module is connected in series with an upsampling module; and after the upsampling module of the first N-1 levels of multi-kernel convolutional attention modules, a gated fusion attention module is connected in series, the gated fusion attention module is used to perform gated fusion of the output features of the upsampling module and the output features of the max pooling module of the corresponding level of multi-kernel convolutional residual module, and the classifier is used to classify the feature map obtained by the decoder to obtain the final classification result map.

5. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 4, characterized in that, The processing of input features by the multi-kernel convolutional residual module includes: sequentially passing the input features through 1×1 convolution, batch normalization (BN), ReLU activation, multi-kernel convolution module, 1×1 convolution, batch normalization (BN), and then adding the residuals to the original input features of the multi-kernel convolutional residual module to obtain the output features of the multi-kernel convolutional residual module. The processing of input features by the multi-kernel convolution module includes: processing the input features of the multi-kernel convolution module in parallel using three branches, each of which is activated by convolution, batch normalization (BN), and ReLU activation. The convolution kernel sizes in the three branches are 1×1, 3×3, and 5×5, respectively. The output features obtained from the three branches are then fused element-wise and shuffled through channels to promote information interaction to obtain the output features of the multi-kernel convolution module.

6. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 4, characterized in that, The multi-kernel convolutional attention module consists of a spectral attention module, a spatial attention module, and a multi-kernel convolutional residual module connected in sequence. The spectral attention module processes the input features by performing parallel processing on two branches. One branch performs max pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the first-path features; the other branch performs average pooling, 1×1 convolution, ReLU activation, and 1×1 convolution to obtain the second-path features. The two-path features are then superimposed and processed by the Sigmoid function to obtain the spectral feature weights. These spectral feature weights are then multiplied by the input features of the spectral attention module to obtain the output features of the spectral attention module. The spatial attention module processes the input features by performing max pooling and average pooling on the input features. The features obtained from max pooling and average pooling are then superimposed and processed by a 7×7 convolution and the Sigmoid function to obtain the spatial feature weights. These spatial feature weights are then multiplied by the input features of the spatial attention module to obtain the output features of the spatial attention module.

7. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 4, characterized in that, The gating fusion attention module processes input features including: decoding features from the current layer. Corresponding skip connection features from the encoder Parallel processing is performed using two branches, each executing 3×3 convolution and batch normalization (BN). The features obtained from the two branches are then stacked and sequentially activated by ReLU, followed by 1×1 convolution, BN, and a sigmoid function, before being used as a gating mask. Then the gate mask Corresponding skip connection features from the encoder Element-wise multiplication, plus the original skip connection features from the encoder. The output features of the gated fusion attention module are obtained.

8. The large-scene hyperspectral zero-shot classification method based on a visual language model according to claim 1, characterized in that, When training the hyperspectral segmentation network based on the optimized pseudo-label map of the hyperspectral image and its median spectral fingerprint, the learning rate is dynamically adjusted using a cosine annealing strategy during the iterative update of the hyperspectral segmentation network. ； In the above formula, Let be the learning rate at the t-th iteration. and These are the maximum and minimum values ​​of the learning rate, respectively. This represents the current iteration number. This represents the maximum number of iterations.

9. A large-scene hyperspectral zero-shot classification system based on a visual language model, comprising an interconnected microprocessor and a memory, characterized in that, The microprocessor is programmed or configured to execute the large-scene hyperspectral zero-shot classification method based on a visual language model as described in any one of claims 1 to 8.

Images