A hyperspectral image category incremental classification method based on feature drift calibration

Abstract

The application provides a hyperspectral image category incremental classification method based on feature drift calibration, and belongs to the technical field of remote sensing image processing and intelligent classification. First, input current incremental task samples, historical category auxiliary samples and a frozen previous stage classification model to construct a fusion training batch containing new and old categories. Second, an old category output distillation constraint is constructed based on the current model output and the previous stage model output to maintain the historical category discrimination response. Then, a channel-space relationship distillation is established by using intermediate features, and feature drift is inhibited by combining feature anchoring and old category relationship bridging. In addition, a drift projector is fitted after the training is completed to calibrate the historical category prototype and the relationship memory bank. Finally, the next stage classification is supported by using the calibrated memory. Experimental results show that the method has good category retention capability on the Houston and Salinas data sets, and provides a new method for hyperspectral image category incremental classification.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image classification, and more specifically to an incremental classification method for hyperspectral images based on feature drift calibration. Background Technology

[0002] Hyperspectral images possess continuous, fine-grained spectral and spatial texture information, enabling precise identification of targets such as crops, urban features, vegetation cover, mineral distribution, and ecological environments. Therefore, they are widely used in precision agriculture, urban planning, environmental monitoring, disaster assessment, and resource exploration. As remote sensing observation areas, acquisition times, and application tasks constantly change, hyperspectral image classification models often need to incrementally learn newly emerging land cover categories while maintaining the ability to recognize previously learned historical categories.

[0003] In incremental hyperspectral image classification tasks, if the model is trained using only samples from the new category of the current task, the model parameters will rapidly shift towards the current category distribution, leading to a significant decrease in the ability to distinguish historical categories—a phenomenon known as catastrophic forgetting. Traditional sample replay methods mitigate forgetting by saving original samples from historical categories and reusing them in subsequent training phases. However, these methods require long-term storage of historical hyperspectral original image patches, pixel-level spectral spatial tensors, or spatial coordinate information, which may be constrained in practical remote sensing scenarios by factors such as storage costs, data sharing limitations, privacy protection, and data management regulations.

[0004] To avoid preserving historical original samples, existing incremental learning methods without original samples typically utilize the previous stage model, historical category statistics, or a small amount of compact memory to assist in the current stage training. However, relying solely on knowledge distillation from the classification output layer often only constrains the model's final prediction results and struggles to maintain the internal response relationships of hyperspectral images in the spectral channel and spatial structure dimensions. Furthermore, using only historical category auxiliary samples for ordinary supervised training can easily cause these historical category auxiliary samples to drift in the current model's feature space, leading to a mismatch between the historical category prototypes and relational memory and the updated model's feature space.

[0005] The inventors discovered that in incremental classification of hyperspectral images, the ability to retain old classes depends not only on whether historical classes participated in the current training stage, but also on whether the current model can maintain the output response, channel-space internal relationships, and feature space structure of the old classes formed in the previous stage. Furthermore, after the current training stage is completed, the historical class prototypes and relation memory also need to be calibrated as the model's feature space changes; otherwise, the drift of old class features will be amplified in the next stage of incremental classification. Summary of the Invention

[0006] This invention proposes a hyperspectral image category incremental classification method based on feature drift calibration, combining old class output distillation, channel-space relation distillation, feature anchoring, old class relation bridging, and post-training drift projector calibration. The method first inputs the current incremental task samples, historical category auxiliary samples, and a frozen previous-stage hyperspectral image classification model to construct a fused training batch containing the current and historical categories. Then, the frozen previous-stage model provides the current-stage model with old class output responses and intermediate feature relation references, and the current-stage hyperspectral image classification model is trained through various constraints. Subsequently, after training, the drift projector is fitted based on the feature changes of the historical category auxiliary samples before and after training, and the historical category prototype and relation memory are calibrated to ensure that the compact historical memory fits the updated model feature space.

[0007] The objective of this invention is achieved as follows:

[0008] A hyperspectral image category incremental classification method based on feature drift calibration includes the following steps:

[0009] Step a: Input the current incremental task sample, historical category auxiliary samples, and the frozen hyperspectral image classification model from the previous stage;

[0010] Step b: Combine the current incremental task samples and historical category auxiliary samples into a fusion training batch containing the current category and historical categories;

[0011] Step c: Input the fused training batches into the current stage hyperspectral image classification model and the frozen previous stage hyperspectral image classification model respectively to obtain the current model output, the previous stage model output and the corresponding intermediate feature representations;

[0012] Step d: Construct supervised classification constraints based on the current model output, and construct old class output distillation constraints based on the current model output and the previous stage model output;

[0013] Step e: Construct channel-spatial relationship distillation constraints by utilizing the intermediate feature representations of the current stage hyperspectral image classification model and the frozen previous stage hyperspectral image classification model;

[0014] Step f: Construct feature anchoring constraints and old class relationship bridging constraints for historical category auxiliary samples to suppress feature drift of historical category auxiliary samples during the current training process;

[0015] Step g: Combine supervised classification constraints, old class output distillation constraints, channel-space relation distillation constraints, feature anchoring constraints, and old class relation bridging constraints to train the hyperspectral image classification model for the current stage and output the incremental classification results.

[0016] Step h: After training, a drift projector is fitted based on the feature changes of historical category auxiliary samples before and after training. The drift projector is then used to calibrate the historical category prototype and relation memory bank to obtain a compact historical memory for the next stage of category incremental classification task.

[0017] Beneficial effects:

[0018] This invention proposes a hyperspectral image category incremental classification method based on feature drift calibration. This method reduces the risk of forgetting old classes by using historical category auxiliary samples in the current training stage without preserving historical original hyperspectral image samples. It preserves the historical category discrimination response of the previous stage model through old class output distillation, maintains the spectral channel relationships and spatial structure relationships in intermediate features through channel-spatial relationship distillation, suppresses old class feature drift during training through feature anchoring and old class relationship bridging, and calibrates the historical category prototype and relationship memory through a drift projector, thereby improving the cross-stage stability of hyperspectral image category incremental classification. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is an overall flowchart of the hyperspectral image category incremental classification method based on feature drift calibration in the present invention.

[0021] Figure 2 This is a schematic diagram of the channel-space relationship distillation framework in the method of the present invention.

[0022] Figure 3 This is a schematic diagram of feature drift calibration and memory update in the method of the present invention.

[0023] Figure 4 The diagram shows the classification results of the method of the present invention on the Houston dataset, where (a) is the classification result of the comparison method and (b) is the classification result of the method of the present invention.

[0024] Figure 5 The diagram shows the classification results of the method of the present invention on the Salinas dataset, where (a) is the classification result of the comparison method and (b) is the classification result of the method of the present invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0026] In a specific embodiment of the present invention, the hyperspectral image sample can be represented as an image patch or pixel neighborhood tensor containing spectral and spatial neighborhood dimensions. The sample set of the current incremental task is denoted as... The historical category auxiliary sample set is denoted as The frozen hyperspectral image classification model of the previous stage is denoted as The hyperspectral image classification model to be trained at this stage is denoted as . Historical category auxiliary samples are historical category alternative training evidence obtained without preserving the original historical hyperspectral image samples. This invention does not limit the specific generation method of these samples; they can be obtained from the previous stage model and compact historical memory, or from other methods of obtaining old category evidence without original samples.

[0027] The improvement of this invention focuses on relation distillation and post-training drift calibration in the current stage of classification training, rather than relying on a specific historical auxiliary sample generation algorithm as a necessary technical feature.

[0028] A hyperspectral image category incremental classification method based on feature drift calibration, as described in the specific embodiment of this invention, is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps:

[0029] Step a: Obtain the hyperspectral image samples of the current incremental task, historical category auxiliary samples, and the frozen hyperspectral image classification model from the previous stage.

[0030] Specifically, in the first In each incremental task, the hyperspectral image samples of the current incremental task contain new category samples accessible at the current stage; historical category auxiliary samples carry historical category labels and are used to provide old class classification boundaries and old class feature relationship constraints during training at the current stage; the frozen hyperspectral image classification model from the previous stage is used to provide old class output responses and intermediate feature relationship references.

[0031] Step b: Construct fusion training batches.

[0032] Specifically, samples of the current category are sampled from the hyperspectral image samples of the current incremental task according to a preset sampling ratio, and samples of the historical category are sampled from the historical category auxiliary samples according to the historical category labels. The two are then concatenated in the batch dimension to obtain a fused training batch. The fused training batch can be represented as:

[0033]

[0034] in, This indicates the current category of real sample batches. Indicates the batch of auxiliary samples for historical categories. This indicates a fused training batch that simultaneously contains real samples of the current category and auxiliary samples of the historical categories.

[0035] Step c: Input the fusion training batches into the current stage hyperspectral image classification model and the frozen previous stage hyperspectral image classification model, respectively.

[0036] Specifically, the fused training batches are input into the hyperspectral image classification model at the current stage. To obtain the current model output and intermediate feature representation Input the same fusion training batch into the frozen previous stage hyperspectral image classification model. The output of the previous stage model is obtained. and intermediate feature representation Among them, superscript Indicates the student model at the current stage, superscript This represents the teacher model from the previous stage.

[0037] Step d: Construct supervised classification constraints and old class output distillation constraints.

[0038] For samples with class labels in the fusion training batch, supervised classification constraints are constructed based on the output of the hyperspectral image classification model at the current stage:

[0039]

[0040] in, Indicates the first One hyperspectral image sample, Indicates the corresponding category label, This indicates the current stage of the hyperspectral image classification model's classification of categories. The predicted probability.

[0041] For historical class outputs, the frozen previous stage hyperspectral image classification model is used as the teacher model, and the current stage hyperspectral image classification model is used as the student model to construct distillation constraints for the old class outputs:

[0042]

[0043] in, Indicates the distillation temperature. Denotes KL divergence, This represents the output dimension corresponding to the historical category. This constraint ensures that the current model, when learning the current category, can maintain the discriminative response of the previous model to the historical categories.

[0044] Step e: Construct channel-space relationship distillation constraints.

[0045] like Figure 2As shown, in order to preserve the internal response relationships formed in the hyperspectral intermediate features of the previous stage model, this embodiment extracts intermediate feature representations from the corresponding network layers of the current stage model and the frozen previous stage model, and constructs channel relationship representations and spatial relationship representations respectively. For a certain network layer... intermediate features Organize them according to channel dimension as The channel relationship is then expressed as:

[0046]

[0047] in, This indicates the number of spectral channels or characteristic channels. and Indicates spatial dimension, This indicates a normalization operation. This is used to characterize the correlation structure between responses of different spectral channels. The same intermediate feature is organized according to the spatial location dimension. The spatial relationship is then expressed as:

[0048]

[0049] in, This is used to characterize the correlation structure between responses at different spatial locations. Based on the differences between the channel relationship representations and spatial relationship representations of the current stage model and the frozen previous stage model, channel-spatial relationship distillation constraints are constructed:

[0050]

[0051] in, This represents the set of network layers selected for relational distillation. This represents the F-norm. Through this constraint, the current stage model not only maintains the output results of the old class, but also preserves the internal relationships of the previous stage model in terms of spectral channel response and spatial structure response.

[0052] Step f: Construct feature anchoring constraints and bridging constraints for the old class relationships from the previous stage.

[0053] To suppress feature drift of historical category auxiliary samples during the current training stage, this embodiment inputs the historical category auxiliary samples into the current stage model, extracts their normalized feature representations, and constrains them with the corresponding historical category prototypes. Let the historical category be... The corresponding historical category prototype is The normalized features extracted by the model at the current stage are represented as follows: Then the feature anchoring constraint is:

[0054]

[0055] in, Represents cosine similarity. Indicates the auxiliary sample label of the historical category. The corresponding historical category prototype. Through this constraint, the historical category auxiliary samples are located closer to the center of the corresponding historical category in the feature space of the model at the current stage.

[0056] Furthermore, to maintain the old class relation structure from the previous stage, this embodiment constructs bridging constraints for the old class relations from the previous stage. Let the relation memory be... ,function Representation of features Given the similarity vector between the reference features and the previous stage's old class relation bridging constraint, the previous stage's constraint is:

[0057]

[0058] in, This represents the frozen historical category auxiliary sample features extracted by the previous stage model. Through this constraint, the current stage feature representation maintains the old class relationship structure consistent with the previous stage feature representation.

[0059] Step g: Train the hyperspectral image classification model for the current stage using multiple constraints, and output the incremental classification results.

[0060] When training the hyperspectral image classification model for the current stage, the supervised classification constraints, old class output distillation constraints, channel-space relationship distillation constraints, feature anchoring constraints, and old class relationship bridging constraints from the previous stage are weighted and summed according to preset weights to obtain the joint optimization objective:

[0061]

[0062] in, These represent the weight coefficients of the old class output distillation constraint, the channel-space relation distillation constraint, the feature anchoring constraint, and the previous stage old class relation bridging constraint, respectively. This is achieved by minimizing... Update the parameters of the hyperspectral image classification model at the current stage.

[0063] The trained hyperspectral image classification model is used to predict the category of the current incremental task or the hyperspectral image to be classified, and the incremental classification result of the hyperspectral image category is output.

[0064] Step h: Fit the drift projector and calibrate the historical category prototype and relation memory.

[0065] like Figure 3As shown, after the current stage of training is completed, the model parameters have been updated, and the reference features in the historical category prototypes and relation memory may be offset from the feature space of the current stage model. Therefore, this embodiment fits a drift projector based on the feature correspondence between historical category auxiliary samples before and after training, or between the previous stage model and the current stage model. Let the drift projector be... Its parameters are The fitting objective of the drift projector is:

[0066]

[0067] in, This represents the result after mapping the features from the previous stage to the feature space of the current stage. This represents the post-training features extracted by the model at the current stage. The drift projector can be a linear mapping or a lightweight parameterized feature map.

[0068] After the drift projector has completed its fitting, it is used to calibrate the historical category prototypes and historical category reference features in the relational memory:

[0069]

[0070] in, This represents the calibrated historical category prototype. This represents the prototype of the historical category in the previous stage. Represents the first in the relational memory. Historical category reference features, This represents the calibrated historical category reference features. The calibrated historical category prototypes and the calibrated relational memory are used as a compact historical memory for the next stage of incremental category classification, for subsequent incremental category classification of hyperspectral images without original samples.

[0071] It should be noted that the hyperspectral image classification model in the above embodiments can employ convolutional neural networks, Transformer networks, spectral-spatial joint networks, or other deep learning models suitable for hyperspectral image classification. The number of network layers, distillation temperature, constraint weights, and drift projector structure selected for channel-spatial relationship distillation can be set according to the specific dataset and model structure. Historical category auxiliary samples are only used to supplement old class evidence during the current incremental task training process and are not required to save historical original hyperspectral image samples.

[0072] This invention is validated using the Houston dataset and the Salinas dataset.

[0073] The Houston dataset is a hyperspectral remote sensing dataset of complex urban land features, containing fifteen land feature categories, including healthy grassland, stressed grassland, trees, soil, water bodies, residential areas, commercial areas, roads, highways, railways, parking lots, tennis courts, and running tracks. In the experiment, it was divided into three incremental tasks. This dataset contains many artificial land feature categories with similar spectra and complex spatial textures, making it suitable for verifying the stability of the incremental classification of this invention in scenarios with strong confusion between old classes.

[0074] The Salinas dataset is a typical agricultural hyperspectral remote sensing dataset, containing sixteen land cover categories, including various broccoli, fallow land, celery, grapes, corn, lettuce, and vineyards. In the experiment, it was divided into three incremental tasks. This dataset contains many fine-grained crop categories and similar spectral responses, making it suitable for verifying the old class preservation ability of this invention in fine-grained category incremental learning scenarios.

[0075] In the experiment, the current incremental task samples and historical category auxiliary samples together form a fusion training batch. The historical category prototypes and relation memory are calibrated using the old class output distillation constraints, channel-space relation distillation constraints, feature anchoring constraints, previous stage old class relation bridging constraints, and the trained drift projector as described in this invention. To avoid confusion with the historical category auxiliary sample generation process, this embodiment focuses on verifying the technical effectiveness of relation distillation and drift calibration for the current stage of incremental category classification training, without elaborating on the specific generation process of historical category auxiliary samples.

[0076] The experimental results are shown in Tables 1 and 2. On the Houston dataset, after completing the three incremental tasks sequentially, the method of this invention achieved an overall classification accuracy of 83.42%, an average classification accuracy of 86.59%, and a Kappa coefficient of 82.03 after the third incremental task. On the Salinas dataset, the method of this invention achieved an overall classification accuracy of 94.09%, an average classification accuracy of 97.23%, and a Kappa coefficient of 93.40 after the third incremental task. These results demonstrate that, without preserving historical original hyperspectral image samples, this invention can maintain relatively stable classification performance in complex urban landscape scenes and fine-grained agricultural scenes.

[0077] like Figure 4 and Figure 5As shown, after applying the method of this invention, relatively continuous classification results can be obtained for easily confused artificial feature areas such as roads, railways, and parking lots in the Houston dataset, as well as fine-grained crop areas such as grapes, corn, lettuce, and vineyards in the Salinas dataset. Combined with the incremental task classification accuracy, it can be seen that relation distillation can maintain the old class output response and channel-space internal relationships formed by the model in the previous stage, and drift calibration can adapt the historical class prototypes and relation memory to the updated feature space, thereby improving the old class preservation ability and cross-task stability of incremental classification without original sample classes.

[0078] Table 1. Incremental classification experimental results on the Houston dataset.

[0079]

[0080] Table 2. Incremental classification experimental results on the Salinas dataset.

[0081]

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art can modify, substitute, or alter the above embodiments without departing from the principles and spirit of the present invention, and such modifications, substitutions, or alterations should all fall within the protection scope of the present invention.

Claims

1. A hyperspectral image category incremental classification method based on feature drift calibration, characterized in that, Includes the following steps: Step a: Input the current incremental task sample, historical category auxiliary samples, and the frozen hyperspectral image classification model from the previous stage; Step b: Combine the current incremental task samples and historical category auxiliary samples into a fusion training batch containing the current category and historical categories; Step c: Input the fused training batches into the current stage hyperspectral image classification model and the frozen previous stage hyperspectral image classification model respectively to obtain the current model output, the previous stage model output and the corresponding intermediate feature representations; Step d: Construct supervised classification constraints based on the current model output, and construct old class output distillation constraints based on the current model output and the previous stage model output; Step e: Construct channel-spatial relationship distillation constraints by utilizing the intermediate feature representations of the current stage hyperspectral image classification model and the frozen previous stage hyperspectral image classification model; Step f: Construct feature anchoring constraints and old class relationship bridging constraints for historical category auxiliary samples to suppress feature drift of historical category auxiliary samples during the current training process; Step g: Combine supervised classification constraints, old class output distillation constraints, channel-space relation distillation constraints, feature anchoring constraints, and old class relation bridging constraints to train the hyperspectral image classification model for the current stage and output the incremental classification results. Step h: After training, a drift projector is fitted based on the feature changes of historical category auxiliary samples before and after training. The drift projector is then used to calibrate the historical category prototype and relation memory bank to obtain a compact historical memory for the next stage of category incremental classification task.

2. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, Step b specifically includes: sampling current category samples from the current incremental task samples according to a preset sampling ratio, sampling historical category samples from historical category auxiliary samples according to historical category labels, and concatenating the current category samples and historical category samples in the batch dimension to obtain a fused training batch.

3. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The old class output distillation constraint in step d is constructed by comparing the differences in historical class output responses between the frozen previous-stage hyperspectral image classification model and the current-stage hyperspectral image classification model, in order to maintain the discriminative response of the current-stage hyperspectral image classification model to historical classes.

4. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The channel-spatial relationship distillation constraint in step e includes channel relationship distillation and spatial relationship distillation, wherein channel relationship distillation is used to maintain the correlation structure between responses of different spectral channels, and spatial relationship distillation is used to maintain the correlation structure between responses of different spatial locations.

5. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 4, characterized in that, The channel relationship distillation is obtained by aggregating intermediate feature representations along the spatial dimension and calculating the similarity matrix between different channel responses; the spatial relationship distillation is obtained by aggregating or flattening intermediate feature representations along the channel dimension and calculating the similarity matrix between responses at different spatial locations.

6. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The feature anchoring constraint in step f is constructed by calculating the distance between the feature representation of the historical category auxiliary sample in the current stage hyperspectral image classification model and the corresponding historical category prototype, so that the historical category auxiliary sample is close to the corresponding historical category prototype in the current model feature space.

7. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The old class relationship bridging constraint in step f is constructed by comparing the feature relationships of historical class auxiliary samples in the frozen previous stage hyperspectral image classification model and the feature relationships in the current stage hyperspectral image classification model, so as to maintain the old class feature relationship structure of the previous stage.

8. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, In step g, the supervised classification constraints, old class output distillation constraints, channel-space relationship distillation constraints, feature anchoring constraints, and old class relationship bridging constraints are weighted and summed according to preset weights to obtain a joint optimization objective, and the parameters of the hyperspectral image classification model at the current stage are updated based on the joint optimization objective.

9. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The drift projector in step h is a linear mapping or a parameterized feature mapping. Its input is the feature representation of the historical category auxiliary sample in the previous stage hyperspectral image classification model before training or frozen, and its output is the feature representation of the historical category auxiliary sample in the current stage hyperspectral image classification model after training.

10. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, In step h, a drift projector is used to map the historical category prototypes and historical category reference features in the relation memory bank to obtain the calibrated historical category prototypes and the calibrated relation memory bank.

11. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The historical category auxiliary samples are only used for current stage category incremental classification training, drift projector fitting, and compact history memory calibration. They are discarded after completion, and the original historical hyperspectral image samples are not saved.

12. The hyperspectral image category incremental classification method based on feature drift calibration according to claim 1, characterized in that, The calibrated historical category prototypes and the calibrated relational memory serve as compact historical memories for the next stage of incremental category classification tasks, and are used for subsequent incremental category classification of hyperspectral images.

Images