Hyperspectral image class incremental learning method and system

Abstract

The application relates to a hyperspectral image class incremental learning method and system based on feature space expansion compression and space-spectrum joint augmentation, and relates to the field of hyperspectral image class incremental learning. The application solves the problems that traditional image augmentation methods mainly process spatial information and cannot fully utilize the spectral dimension of a hyperspectral image, and the problem of catastrophic forgetting in class incremental learning. The method comprises the following steps: constructing a lightweight classification model according to a space-spectrum joint enhancement method; in the initial stage, taking a full-precision network as a teacher model to provide online supervision for the training of the lightweight network; in the incremental stage, introducing a new full-precision network to narrow the gap between an old feature space and a target feature space, and obtaining an optimized quantization network; and realizing online updating of the learning ability of the hyperspectral classification network according to the optimized quantization network. The application is applied to the field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of hyperspectral image class incremental learning, and particularly relates to a hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation. BACKGROUND

[0002] Hyperspectral image classification (HSIC) is an important topic in pattern recognition. The ultimate goal of hyperspectral image classification is to assign each pixel to a specific class based on its spatial and spectral features, so HSIC is a pixel-level image classification task. Traditional machine learning methods, such as support vector machines and polynomial logistic regression, were initially used for HSIC. These methods mainly focus on utilizing the spectral features of hyperspectral data. To extract more representative features from spectral channels, researchers have also proposed dimensionality reduction methods, including principal component analysis (PCA) and independent component analysis. Spatial-spectral classification methods were introduced to enable simultaneous utilization of spatial and spectral features. MSMIL introduced a spatial-spectral decision fusion strategy to weigh multi-scale superpixel maps. Functional data analysis was also integrated into HSIC to extract better spatial and spectral knowledge.

[0003] Inspired by the dominance of deep learning in other optical computer vision tasks, many deep neural network-based HSIC methods have been developed to improve classification performance in hyperspectral data tasks. However, these methods are all based on the premise that the training dataset can cover all types of targets in the test scene. However, the landscape is often affected by human activities and seasonal changes, and gradually changes over time. In addition, different types of landscapes may be obtained in turn during the training process. Therefore, the model must constantly increase its capacity to adapt to dynamic landscape changes. However, if the model is updated with new input data, the catastrophic forgetting phenomenon is inevitable. If the model is generalized to new image types, the previous decision boundaries will change.

[0004] With the development of sensor technology and image processing technology, hyperspectral image classification has been widely studied in many remote sensing fields, including agricultural research, ocean exploration, military surveillance and mineral exploration. However, there are still many challenges in promoting the application of hyperspectral image classification in these fields. First, the imaging mechanism of hyperspectral image is more complex, which not only contains rich spatial information, but also contains a large amount of spectral information. Traditional image augmentation methods mainly process spatial information, and cannot fully utilize the spectral dimension of hyperspectral image. Therefore, in order to realize efficient hyperspectral image processing, a joint augmentation method that can enhance spatial and spectral information at the same time is needed to promote the deep learning network to more effectively extract and learn the features of spatial and spectral domains. Secondly, with human activities, seasonal changes and other factors, the hyperspectral information of the same region changes greatly, and new ground object categories and different states of original target categories may appear. The classification network trained by the previously collected hyperspectral image data cannot maintain consistent classification ability for images of the same scene. Therefore, the hyperspectral image classification network needs to expand its classification ability according to the scene changes. Finally, the edge platform used to deploy the hyperspectral image classification method cannot meet the inference needs of large-scale networks in terms of computing resources and storage resources. However, small networks are limited by the number of parameters, and the accuracy of the inference result is difficult to meet the requirements. SUMMARY

[0005] The present application is directed to the problem that the traditional image augmentation method mainly processes spatial information, cannot fully utilize the spectral dimension of hyperspectral image, and there is a catastrophic forgetting in class-incremental learning, and the classification network trained by the previously collected hyperspectral image data cannot maintain consistent classification ability for images of the same scene. A hyperspectral image class-incremental learning method based on feature space expansion compression and space-spectrum joint augmentation is proposed, which comprises:

[0006] Constructing a lightweight classification model according to the space-spectrum joint enhancement method;

[0007] In the initial stage, the full-precision network is used as a teacher model to provide online supervision for the training of the lightweight network;

[0008] In the incremental stage, a new full-precision network is introduced to narrow the gap between the old feature space and the target feature space, and an optimized quantization network is obtained;

[0009] According to the optimized quantization network, the learning ability of the hyperspectral classification network is updated online.

[0010] Further, an optimization method is also proposed, wherein the space-spectrum joint enhancement method is: selecting a plurality of target bands in the spectral dimension, performing random adjustment at the pixel level in the spatial region corresponding to the target bands, and combining center cropping and scaling based on random size, and random horizontal flip to enhance the spatial domain data.

[0011] Further, it is proposed that a preferred mode is provided, wherein the training of the lightweight network is supervised online by the full-precision network as the teacher model, comprising:

[0012] A single-stage framework is constructed, in which the teacher and student models are updated simultaneously;

[0013] The correlation between samples is used to capture the feature space of the model, and the output of the student model is:

[0014]

[0015] wherein, denotes the feature extraction backbone of the student model, denotes the dataset, x 1 to x n denotes the sample in the dataset;

[0016] The correlation generated by the student model is :

[0017]

[0018] wherein, denotes the cosine similarity operator. denotes ;

[0019] The correlation generated by the teacher model is:

[0020]

[0021] wherein, denotes the feature extraction backbone of the teacher model;

[0022] The difference between the student model and the teacher model is calculated, and the first third of the value of the difference is used as the coefficient of the teacher model hyper-feature aggregation loss function;

[0023] The loss function is constructed according to the coefficient of the loss function as the adversarial distillation loss;

[0024] The distillation loss function is constructed, and the distillation loss function is used to promote the transfer of knowledge from the teacher to the student model.

[0025] Further, it is proposed that a preferred mode is provided, wherein the adversarial distillation loss is:

[0026]

[0027] wherein k is a variable calculated by accumulating the square of the number of each class, is an indicator function, is a sample label, is a difference between a student model and a teacher model, are outputs of a teacher network feature extraction backbone, is a loss function.

[0028] Further, an optimal way is proposed, and the construction of the distillation loss function is:

[0029]

[0030]

[0031] wherein, represents a backbone of a network, represents a feature extraction backbone of a network comprising a feature enhancement layer, represents an adoption of a mean square error loss function, represents an aggregation of a student model and a teacher model on a higher dimension.

[0032] Further, an optimal way is proposed, and the construction of the distillation loss function is:

[0033] Training a model:

[0034]

[0035] wherein, represents a combination of a backbone network, represents a backbone network of an old model, represents a backbone network of a new model, represents a normalization of model integration, represents a loss function, represents an input and a corresponding label;

[0036] According to the compression strategy of knowledge distillation, a new quantization model is trained under the supervision of a previously trained model integration:

[0037]

[0038]

[0039] wherein, represents a feature extraction backbone of a new model, composed of a feature enhancement backbone and a backbone network , represents a cosine similarity function, represents a basic loss function of each stage, represents an overall feature extraction backbone including ​represents the feature similarity constraint function, represents the accumulation function, represents the output of the backbone network of the new network, Represents the output of the combined backbone network of the two trained models.

[0040] Furthermore, a preferred method is proposed, wherein the optimized quantized network further includes introducing a trainable factor Adjust the backbone network of the new model .

[0041] Based on the same inventive concept, the present invention also proposes a hyperspectral image incremental learning system based on feature space expansion and compression and spatial-spectral joint augmentation, the system comprising:

[0042] A lightweight classification model building unit, used to build a lightweight classification model based on a spatial-spectral joint enhancement method;

[0043] An online supervision unit is used to provide online supervision for the training of the lightweight network using the full-precision network as a teacher model in the initial stage;

[0044] The optimization unit is used to introduce a new full-precision network in the incremental phase to narrow the gap between the old feature space and the target feature space, and obtain an optimized quantized network;

[0045] A learning unit is used to achieve online update of the learning ability of the hyperspectral classification network based on the optimized quantization network.

[0046] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation as described in any one of the above items.

[0047] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the steps of a hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation as described above.

[0048] The present invention is beneficial in that:

[0049] The application provides a hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation, designs a feature space expansion, integration and compression and space-spectrum enhancement (FEICA-CIL) framework, introduces a space-spectrum mixed enhancement technology in the framework, enhances the robustness of the model by comprehensively extracting the representation from the space and spectrum dimensions, and directly alleviates the catastrophic forgetting by improving the cross-class performance. The learning process is divided into two different stages of initial and incremental, and each stage is optimized by network quantization to reduce the calculation overhead. In the initial stage, an online adversarial distillation mechanism is designed, in which the student network learns from the real label and the teacher network at the same time, and the teacher explores the knowledge beyond the scope of the student. This distillation strategy can realize more abstract representation, so that the model can learn robust knowledge and maintain strong performance on the first task, thereby alleviating the catastrophic forgetting. In the incremental stage, a new full-precision model is introduced to solve the feature space limitation, and the model expands the feature space. This expansion allows to establish a clear decision boundary for new classes through model integration, while retaining the representation of old classes. The framework alleviates catastrophic forgetting by supplementing rather than modifying the feature extractor of the previously learned data. Finally, the double model is integrated and compressed into an efficient single-branch network through knowledge distillation, optimizing the performance and computational efficiency. The incremental stage can be repeated to further expand the capacity of the model.

[0050] The application is applied to the field of image processing. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation according to the first embodiment is a flowchart;

[0052] Figure 2 An SSMA flowchart according to the second embodiment;

[0053] Figure 3 A feature space expansion embedding diagram according to the eleventh embodiment;

[0054] Figure 4 An overall accuracy diagram after the network learns the last task on all data sets according to the eleventh embodiment;

[0055] Figure 5 A performance comparison diagram of the FEICA-CIL with and without two trainable factors according to the eleventh embodiment. DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application.

[0057] Embodiment one, see Figure 1 This embodiment is described. The hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation described in this embodiment includes:

[0058] According to the space-spectrum joint enhancement method, a lightweight classification model is constructed;

[0059] In the initial stage, the full-precision network is used as a teacher model to provide online supervision for the training of the lightweight network;

[0060] In the incremental stage, a new full-precision network is introduced to reduce the gap between the old feature space and the target feature space, and an optimized quantization network is obtained;

[0061] According to the optimized quantization network, the learning ability of the hyperspectral classification network is updated online.

[0062] The method proposed in this embodiment solves the problem that traditional image augmentation methods mostly focus on the processing of spatial information and ignore the spectral dimension of hyperspectral images. Hyperspectral images have multi-dimensional spectral data, so fully utilizing spectral information to enhance the data set can improve the accuracy and robustness of the classification model. However, many existing augmentation methods fail to effectively utilize this feature. The present application proposes a space-spectrum joint enhancement method that considers both spatial information and spectral information during the augmentation process. This not only increases the diversity of training data, but also better captures the spectral features in hyperspectral images, improving classification performance.

[0063] The method proposed in this embodiment solves the problem of catastrophic forgetting in class incremental learning, and adopts a full-precision network as a teacher model to provide online supervision for the lightweight network. By guiding the training of the student model through the teacher model, the consistency of the model is maintained to prevent forgetting of old knowledge during incremental learning.

[0064] The method proposed in this embodiment addresses the problem of how to ensure the feature space expansion of the model and avoid the interference of new data introduction to the old data feature space in incremental learning. By introducing a new full-precision network to reduce the gap between the old feature space and the target feature space, the feature space is optimized and expanded. This process combines feature compression and expansion, so that incremental learning can adapt to new data and categories while ensuring the stability of the model.

[0065] Implementation method 2, see Figure 2 This embodiment further defines the incremental learning method for hyperspectral imagery based on feature space expansion and compression and spatial-spectral joint augmentation described in Implementation 1. The spatial-spectral joint enhancement method comprises the following steps: selecting several target bands in the spectral dimension, performing pixel-level random adjustments within the spatial regions corresponding to the target bands, and enhancing the spatial domain data by combining random-size center cropping and scaling with random horizontal flipping.

[0066] In this implementation, random permutations are performed in both spatial and spectral dimensions by varying the values ​​of selected points within randomly selected bands. Combined with a super-feature aggregation loss, SSMA encourages the learning of stable, representative features. Furthermore, this implementation incorporates center cropping (CenterCropResize) and horizontal flipping (Horizonflip). This allows SSMA to vary the randomly selected channels during different augmentations, forcing the model to extract robust representations in both spatial and spectral dimensions.

[0067] Implementation method 3: This implementation method further limits the incremental learning method for hyperspectral images based on feature space expansion and compression and spatial-spectral joint augmentation described in implementation method 1. The method provides online supervision for the training of the lightweight network based on the full-precision network as a teacher model, including:

[0068] Constructing a single-stage framework in which the teacher and student models are updated simultaneously;

[0069] The feature space of the model is captured by using the correlation between samples. The output of the student model is:

[0070]

[0071] in, represents the feature extraction backbone of the student model, represents a dataset, x 1 to x n Represents a sample in the dataset;

[0072] Correlations generated by the student model for:

[0073]

[0074] in, Represents the cosine similarity operator. express ;

[0075] The correlation generated by the teacher model is:

[0076]

[0077] in, represents the feature extraction backbone of the teacher model;

[0078] Calculate the difference between the student model and the teacher model, and use the first third of the difference as the coefficient of the teacher model's super-feature aggregation loss function;

[0079] Construct a loss function as adversarial distillation loss based on the coefficients of the loss function;

[0080] Construct a distillation loss function and use it to promote the transfer of knowledge from the teacher to the student model.

[0081] Lightweight classification models are limited by the number of parameters and bit width, which can lead to a decline in classification performance. In this implementation, a full-precision model is introduced as a teacher network to guide the student model online, updating both and optimizing the learning ability of the student model.

[0082] Implementation 4: This implementation further limits the incremental learning method for hyperspectral images based on feature space expansion and compression and spatial-spectral joint augmentation described in Implementation 1. The adversarial distillation loss is:

[0083]

[0084] Implementation 5: This implementation further limits the incremental learning method for hyperspectral images based on feature space expansion and compression and spatial-spectral joint augmentation described in Implementation 1. The distillation loss function is constructed as follows:

[0085]

[0086]

[0087] in, Represents the backbone of the network, represents the feature extraction backbone of the network including the feature lifting layer, Indicates the use of mean square error loss function, Represents the aggregation of student and teacher models in higher dimensions.

[0088] Where k is a variable calculated by summing the square of the number of each category, is the indicator function, is the sample label, is the difference between the student model and the teacher model, are the outputs of the teacher network feature extraction backbone, Loss function.

[0089] The adversarial distillation loss can be regarded as the positive part of the adversarial loss. This loss term, combined with the previous formula, constitutes an adversarial loss pair. Thus, the overall loss function of the network in the initial stage of the depth image classification lightweight network enhanced training is as follows:

[0090]

[0091] wherein, denotes the overall loss function in the initial stage, denotes the local similarity loss function, denotes the basic loss.

[0092] Embodiment six, this embodiment is a further limitation of the hyperspectral image class incremental learning method based on feature space expansion compression and space spectrum joint augmentation according to embodiment one, the acquisition of the optimized quantization network comprises:

[0093] Training model:

[0094]

[0095] wherein, denotes the combination of the backbone network, denotes the backbone network of the old model, denotes the backbone network of the new model, denotes the normalization of model integration, denotes the loss function, denotes the input and corresponding label;

[0096] According to the compression strategy of knowledge distillation, a new quantization model is trained under the supervision of the previously trained model integration:

[0097]

[0098]

[0099] wherein, denotes the feature extraction backbone of the new model, composed of the feature promotion backbone and the backbone network , denotes the cosine similarity function, denotes the basic loss function of each stage, denotes the overall feature extraction backbone including , denotes the feature similarity constraint function, denotes the accumulation function, denotes the output of the backbone network of the new network, an output of a combination of backbone networks of two models representing training.

[0100] In this embodiment, the network structure is expanded, the network maintains the old class knowledge and learns the new task knowledge, and the catastrophic forgetting is inhibited. At the same time, additional network parameters are introduced for the feature space expansion algorithm, which causes the network to continuously expand during the incremental process. The feature space compression algorithm is introduced to control the network parameter amount and inhibit the expansion of the network parameters.

[0101] Embodiment seven, this embodiment is a further limitation of the hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation according to embodiment one, wherein the introduction of the trainable factor Adjusting the backbone network of the new model .

[0102] Embodiment eight, a hyperspectral image class incremental learning system based on feature space expansion compression and space-spectrum joint augmentation according to the embodiment, the system comprises:

[0103] A lightweight classification model construction unit is configured to construct a lightweight classification model according to a space-spectrum joint enhancement method.

[0104] An online supervision unit is configured to provide online supervision for the training of the lightweight network by taking the full-precision network as a teacher model in the initial stage.

[0105] An optimization unit is configured to introduce a new full-precision network to narrow the gap between the old feature space and the target feature space in the incremental stage, and obtain an optimized quantization network.

[0106] A learning unit is configured to realize online update of the learning ability of the hyperspectral classification network according to the optimized quantization network.

[0107] Embodiment nine, a computer device according to the embodiment, comprising a memory and a processor, the memory stores a computer program, when the processor runs the computer program stored in the memory, the processor executes the hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation according to any one of embodiments one to seven.

[0108] Embodiment ten, a computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, the computer program is run by a processor to execute the steps of the hyperspectral image class incremental learning method based on feature space expansion compression and space-spectrum joint augmentation according to any one of embodiments one to seven.

[0109] Embodiment eleven, seeFigure 3 、 Figure 4 and Figure 5 This embodiment is described. This embodiment is a specific embodiment of the hyperspectral image class incremental learning method based on feature space expansion compression and space spectrum joint augmentation described in embodiment one, and is also used to explain embodiments two to seven. Specifically:

[0110] Data augmentation techniques in deep learning can help neural networks obtain more general feature representations by randomly transforming original images. However, due to the significant differences between hyperspectral data and traditional visible light images in data characteristics, it is necessary to specially design augmentation strategies suitable for hyperspectral data. Existing hyperspectral augmentation methods mainly include two categories: one only focuses on the transformation of the spectral dimension (such as random band replacement, etc.), and the other considers the joint transformation of the spatial and spectral dimensions (such as random masking, etc.).

[0111] Considering the dual characteristics of hyperspectral data - location distribution information in the spatial dimension and band response characteristics in the spectral dimension, a new spatial-spectral joint enhancement method (SSMA) is proposed in this embodiment. This method enriches the training samples by performing collaborative random transformation in the spatial and spectral dimensions while maintaining the overall structure of the data. Specifically, SSMA first selects several target bands in the spectral dimension, and then performs pixel-level random adjustment in the spatial regions corresponding to these bands.

[0112] To further enhance the diversity of data, this method also combines center cropping and scaling based on random size, as well as random horizontal flipping and other auxiliary augmentation techniques. When SSMA is used in conjunction with high-dimensional feature aggregation loss functions, it can effectively guide the network to learn more robust and discriminative feature representations, thereby improving the overall performance of hyperspectral image classification. As shown in Figure 2 , the images processed by SSMA exhibit more diverse data changes while maintaining the original semantic information.

[0113] The three rows in the figure respectively present different aspects of hyperspectral data enhancement. The top row highlights the rich information in the spectral dimension of hyperspectral data by showing the imaging results of the same sample at different bands, and intuitively reflects the different response characteristics of different bands to ground feature characteristics.

[0114] The middle row focuses on the effects of two basic spatial domain data augmentation: one is center cropping and scaling with random size, which enhances sample diversity by dynamically adjusting the center region of the image; the other is the random horizontal flipping technique, which simulates the scenario of observing the same ground object from different angles.

[0115] The bottom row highlights the enhanced effect of the SSMA technique proposed in this embodiment. By selectively enhancing different wavebands in each processing, SSMA generates a rich variety of training sample variants while preserving the original data structure. This diversified data augmentation strategy prompts the network to pay attention to the feature expression in both spatial and spectral domains during training, thereby constructing a more robust feature extractor. Experiments have verified that this dual-domain joint enhancement strategy significantly improves the performance of the model in complex hyperspectral scene classification tasks, especially in terms of the model's generalization performance.

[0116] In the initial stage, the model is trained to perform a general classification task. Therefore, a distillation strategy is adopted to improve the classification performance. The conventional distillation strategy only transfers knowledge from the teacher to the student, requiring a well-trained teacher model. Inspired by adversarial distillation, this embodiment proposes to build a single-stage framework in which the teacher and student models can be updated simultaneously, making it easy to migrate to different hyperspectral scenes. Through the adversarial relationship established between their outputs, the teacher model is forced to emphasize the differences with the student model during training, enabling it to explore knowledge beyond the current understanding of the student and provide meaningful supervision. These knowledge is then transferred to the student model through the conventional distillation constraint. In this way, the teacher model can effectively guide the student's learning without pre-training.

[0117] In the initial stage and the incremental stage, the network is trained with a high-dimensional feature aggregation loss function and a local similarity classifier; the expression of the feature aggregation loss function is as follows:

[0118]

[0119] where the indicator function is denoted as , which returns a value when its input satisfies the specified condition. is a variable calculated by accumulating the square of the number of each class. This function aggregates the sample representations from the same class in the high-dimensional feature space. The features are not explicitly aggregated into the same region, preserving the diversity of the representations, which is beneficial for the training of the classifier. The loss function of the local similarity classifier is as follows:

[0120]

[0121] where the parameter is a learnable scaling factor, and is a constant used to induce stronger inter-class separation. The activation function is denoted as . The base loss function in each stage is as follows:

[0122] .

[0123] The implementation uses the correlation between samples to capture the feature space of the model. Specifically, assume that is a dataset containing all samples belonging to class The output of the student model can be organized as:

[0124]

[0125] Here, denotes the feature extraction backbone of the student model. Then, the correlation for this particular class can be computed as:

[0126]

[0127] is the cosine similarity operator. denotes . Similarly, the correlation generated by the teacher model is as follows:

[0128]

[0129] Let denote the feature extraction backbone of the teacher model. and The difference between the two is calculated as Although This computation initially focuses on a particular class, but the actual training process involves independently computing the intra-class correlation matrix for each class to obtain the difference matrix. In the implementation, the top third of the values in are reserved as coefficients for the teacher model hyper-feature aggregation loss function. The modified loss function formula is as follows:

[0130]

[0131] This loss serves as an adversarial distillation loss, highlighting the differences between the teacher and student models. Through this mechanism, the teacher model explores knowledge beyond the current understanding of the student model. At the same time, a distillation loss function is introduced to facilitate the transfer of knowledge from the teacher to the student model:

[0132]

[0133]

[0134] denotes the backbone of the network, while denotes the feature extraction backbone of the network containing feature enhancement layers. denotes the adoption of the mean square error loss function. includes augmented samples. The representations of the student and teacher models are aggregated in higher dimensions, while The output is constrained in lower dimensions. The overall loss function in the initial stage is as follows:

[0135]

[0136] While the old model trained in the previous stage can represent the features of the samples from the old classes well, the new classes are beyond its view. Therefore, a new model is introduced to complement the representation ability of the old model. Let represent the input and its label, and represent the backbone networks of the old model and the new model, respectively. The combination of these backbone networks is denoted as . The training process can be represented as:

[0137]

[0138] where denotes the normalized logits of the model ensemble, denotes the loss function. Then, the ideal training result should be:

[0139]

[0140] Let denote the cosine similarity function, denote the growing class proxy in the local similarity classifier. Although the new model reduces the difference between the old features and the ideal features, simply accumulating the outputs of the two models will limit the updating ability of the equation. This limitation occurs because only can be updated during incremental training, while the influence of remains static.

[0141] For new tasks, the model ensemble is enhanced to adapt to additional classes, implementing a trainable factor to independently adjust each channel of the output of the old model. is implemented as a factor rather than a parameter because the disproportionate amount of new class data compared to the old class data can drive the trainable parameters to trivial values during training. For the old classes, the feature representation is effectively captured by the output of the old model. The model ensemble is trained to reconstruct the feature patterns using selected samples. Therefore, a trainable factor is introduced in this embodiment to adjust the influence of . The final optimization objective formula is as follows:

[0142]

[0143] Let the overall feature extraction backbone including be The overall loss function of the feature expansion and integration step is as follows:

[0144]

[0145] Although the model ensemble achieves impressive classification results, it introduces significant computational overhead in the inference process. Since a single model can usually handle more classes, there can be parameter redundancy. To address this issue, a compression strategy using knowledge distillation is proposed in the present embodiment. Under the supervision of the previously trained model ensemble, a new quantized model is trained, adopting a multi-level feature constraint similar to the initial stage. The optimization objective formula is as follows:

[0146]

[0147]

[0148] Here, represents the feature extraction backbone of the new model, which is composed of the feature lifting backbone and the backbone network . represents the cosine similarity function. Unlike the loss in the initial stage, the cosine similarity loss is chosen instead of the mean square error loss to constrain the shallow features output by .

[0149] To verify the effectiveness of the method proposed in the present application, four classic hyperspectral datasets are used for evaluation experiments, namely: Salinas Valley (SA), Indian Pines (IP), Pavia University (PU), and Kennedy Space Center (KSC). The SA dataset is land cover data captured by the AVIRIS sensor in the Salinas Valley of California, covering pixels and 224 spectral bands. After removing 20 water absorption bands, a total of 204 bands and 54129 labeled samples are used for training. The ground truth of this dataset contains a total of 16 different classes. The IP dataset is land cover data captured by the AVIRIS sensor in the northwest of Indiana. It consists of pixels and 220 spectral bands. After removing 20 water absorption bands and background pixels, a total of 10249 land cover pixels and 200 bands are used for training. There are 16 types of land cover in the IP dataset. The PU dataset is land cover data obtained by the ROSIS sensor flying over Pavia in northern Italy, with a spectral range from 430nm to 860nm. The dataset contains The SA dataset contains 103 spectral bands and 103 spatial pixels, and is divided into 9 classes. After removing the water absorption band and background, a total of 42776 samples are available for training.

[0150] The training set partition ratio is selected according to the size of different datasets: 2% of the samples are used as training data for the SA dataset, 10% of the samples are used as training data for the IP and KSC datasets, and 3% of the samples are used as training data for the PU dataset. In order to reduce the complexity of the data, principal component analysis is used to reduce the spectral dimension of all datasets to 8. Then the entire data block is divided into a three-dimensional cube with a size of , where 8 corresponds to the reduced spectral dimension, representing the spatial dimension.

[0151] In the training phase, the Adam optimizer is used to update the network parameters. During the entire training process, the learning rate is fixed at 0.001 and the weight decay is set to 0.005. The batch size for network training is set to 256. For the memory budget of saving old class samples in incremental learning, the memory size of all datasets except KSC is 480. Considering the small amount of data in the KSC dataset, its memory size is adjusted to 120. The memory size of each class in the dataset is calculated according to the proportion of that class in the training samples. It is worth noting that as the training progresses and more classes are introduced, the number of samples saved for each class will decrease accordingly.

[0152] Four-stage class incremental learning experiments are conducted on the SA, IP, and KSC datasets, and three-stage incremental learning experiments are conducted on the PU dataset. Except for the SA dataset, the training epoch of all other datasets in the initial stage is 200. Considering the characteristics of the SA dataset, the training period of the initial stage is set to 50. In the incremental learning stage, whether it is feature space expansion or compression process, the training period is uniformly set to 200. The convolutional neural network used in the experiment consists of four convolutional layers and three pooling layers, with strong feature extraction capability.

[0153] For hyperspectral image classification experiments, three indicators are generally used to measure the classification ability of the network, namely overall accuracy (OA), average accuracy (AA), and Kappa coefficient. OA can reflect the classification accuracy of the network, and AA further calculates the average accuracy of all classes. Compared with OA, AA is more objective in measuring the classification ability of the network in the case of unbalanced class data volume. Kappa is mainly used to evaluate the consistency of the classification task. This coefficient is particularly suitable for fields such as remote sensing image classification.

[0154] Experiment 1: Comparative experiment

[0155] The FEICA-CIL method was tested in this embodiment to test the performance of the classification network in the deep image classification lightweight network enhanced training process on four data sets. In the experiment, the random number seed was set to 1 to obtain reproducible experimental results. Tables 1 and 2 show the classification results on each data set.

[0156] The FEICA-CIL method proposed in the present application exhibits excellent class-incremental learning performance on multiple hyperspectral data sets. On the SA data set, the overall accuracy (OA) of the network in the four tasks reached 99.98%, 99.96%, 99.95% and 99.74%, respectively. In the first three tasks, the accuracy only decreased slightly. However, as the amount of training data increases, the performance decline in the last task becomes inevitable. Compared with the previous task, the network classification performance decreased by 0.21%. The results of the four tasks on the IP data set are 98.60%, 99.19%, 98.82% and 98.35%, with excellent overall performance, and the result of the final task only decreased by 0.25% compared with the initial task. In the three tasks of the PU data set, the accuracy decreased slightly from 100.00% to 99.80% and 99.24%, with a decrease of 0.20% and 0.56% respectively. Even on the KSC data set with smaller memory space, the proposed method still helps the network to maintain strong performance, with an accuracy of 100.00%, 98.51%, 99.14% and 98.57% for the four tasks, and the result of the final task decreased by 1.43% compared with the initial task.

[0157] Table 1 Classification results on SA and IP data sets (%)

[0158]

[0159] Table 2 Classification results on PU and KSC data sets (%)

[0160]

[0161] These results show that although the catastrophic forgetting problem still exists, the method proposed in the present application can effectively balance the requirements of preserving old knowledge and learning new knowledge. The trained incremental network can achieve excellent hyperspectral image classification ability on multiple data sets, providing a state-of-the-art solution for class-incremental learning of hyperspectral data.

[0162] To demonstrate the superiority of the proposed FEICA-CIL method, other CIL frameworks are deployed on the same backbone network, and the state-of-the-art hyperspectral data-based incremental learning method, GS$_2$FIN-CIL, is also included in the comparison to analyze the effectiveness of each method. All the compared methods use the same training settings as FEICA-CIL. It is worth noting that the methods marked with $*$ adopt the same memory budget management strategy as FEICA-CIL, otherwise the default memory budget management strategy of their original code will be retained. Tables 3 and 4 show the overall accuracy (OA) of different methods during the process of class incremental learning.

[0163] Table 3. Comparison of results of algorithm library on SA and IP datasets (%)

[0164]

[0165] Table 4. Comparison of results of algorithm library on PU and KSC datasets (%)

[0166]

[0167] On the SA dataset, the proposed method achieved significant accuracy improvement in all tasks. The largest performance gap appears in task 4, with a 0.72% accuracy gap from the suboptimal method. On the IP dataset, FEICA-CIL also achieved excellent results in tasks 2-4. As the number of tasks increases, FEICA-CIL achieved a stable accuracy improvement of about 0.3% compared to other methods in tasks 3 and 4. On the PU dataset, FEICA-CIL also outperformed all other methods, achieving 100.00%, 99.80%, and 99.24% accuracy in three tasks, while the best results of the compared methods were 99.98%, 99.64%, and 98.98%, respectively. On the KSC dataset, even though FEICA-CIL's accuracy in task 2 was slightly lower than other methods, it achieved significant accuracy improvement in subsequent tasks. The largest accuracy gap appears in the last task of the KSC dataset, where the proposed method improves by 3.92% compared to the suboptimal method, significantly outperforming other methods. In summary, the FEICA-CIL method outperforms the compared algorithms in each training stage.

[0168] Experiment II: Ablation experiment of the augmentation technique of spectral-spatial joint

[0169] To study the impact of SSMA, the experiment removed the SSMA strategy in data preprocessing, but retained the random size center crop scaling and random horizontal flip techniques. Figure 4The overall accuracy (OA) of the network after learning the last task on all datasets is shown. Obviously, the network shows a significant performance drop on all datasets. The largest drop occurs on the SA dataset, and the network trained without SSMA has a 1.62% drop in classification performance. The above results show that SSMA, together with other data augmentation techniques, promotes the network to learn abstract features from the perspective of joint space and spectrum, thereby improving the classification performance of the network.

[0170] Experiment three: ablation experiment of quantization strategy

[0171] The quantization strategy used in the FEICA-CIL method is deep projection \cite{wu2022adaptive}, which uses a projection layer and a normalization function to optimize the update process of quantized weights. In order to study the influence of the quantized network, the FEICA-CIL method is reconstructed using a full-precision network, and the distillation function is removed in the initial stage. The overall accuracy results of the full-precision network are shown in the following table. From the results, it can be seen that although the parameter quantity is smaller, the performance of the quantized network is comparable to that of the full-precision network. On the three datasets, the OA of the quantized network even exceeds that of the full-precision network. On the SA dataset, the quantized network even achieves a 1.60% improvement over the full-precision network. In addition to network accuracy, the parameter quantity of the full-precision and quantized networks is also listed in the table. Since the size of the classifier will increase in a series of incremental tasks, only the parameter quantity in the backbone network is calculated in the table.

[0172] Table 5 Performance comparison of FEICA-CIL method based on quantized network and full-precision network (%)

[0173]

[0174] Experiment four: ablation experiment of trainable parameters

[0175] The FEICA-CIL method uses trainable factors and to adjust the influence of old and new networks on the final output. In order to study the influence of these two factors, the experiment removes these two parameters and directly accumulates the outputs of the two networks. The experimental results are shown in the following table Figure 5 From the results, it can be seen that after using and , the FEICA-CIL achieves better classification accuracy on the three datasets, and the improvement brought by the trainable parameters is the largest on the SA dataset, reaching 1.25%. However, on the IP dataset, the network trained by the FEICA-CIL method with trainable parameters shows a slight performance drop, which is due to the characteristics of different datasets. The above results show that in most cases, the introduction of trainable factors and Both can help the network achieve better classification ability.

[0176] Those skilled in the art will understand that the embodiments of the present disclosure can be provided as a method, a system or a computer program product. Therefore, the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0177] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams. These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that implements the functions specified in the flowcharts and / or block diagrams.

[0179] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present disclosure and not to limit the protection scope thereof, although the present disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that: after reading the present disclosure, the skilled person can make various changes, modifications or equivalent replacements to the specific embodiments of the present disclosure, but these changes, modifications or equivalent replacements are all within the protection scope of the disclosed claims.

Claims

1. A hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation, characterized by: The method comprises: Construct a lightweight classification model based on the spatial-spectral joint enhancement method; In the initial stage, the full-precision network is used as a teacher model to provide online supervision for the training of the lightweight network; In the incremental phase, a new full-precision network is introduced to narrow the gap between the old feature space and the target feature space, obtaining an optimized quantized network. The online update of the learning ability of the hyperspectral classification network is achieved based on the optimized quantization network; The full-precision network is used as a teacher model to provide online supervision for the training of the lightweight network, including: Constructing a single-stage framework in which the teacher and student models are updated simultaneously; The feature space of the model is captured by using the correlation between samples. The output of the student model is: in, represents the feature extraction backbone of the student model, represents a dataset, x 1 to x n Represents a sample in the dataset; Correlations generated by the student model for: in, represents the cosine similarity operator, express ; The correlation generated by the teacher model is: in, represents the feature extraction backbone of the teacher model; Calculate the difference between the student model and the teacher model, and use the first third of the difference as the coefficient of the teacher model's super-feature aggregation loss function; Construct a loss function as adversarial distillation loss based on the coefficients of the loss function; Construct a distillation loss function and use it to promote the transfer of knowledge from the teacher to the student model; The anti-distillation loss is: Where k is a variable calculated by summing the square of the number of each category, is the indicator function, is the sample label, is the difference between the student model and the teacher model, are the outputs of the teacher network feature extraction backbone, Loss function; The distillation loss function is constructed as follows: in, Represents the backbone of the network, represents the feature extraction backbone of the network including the feature lifting layer, Indicates the use of mean square error loss function, Represents the aggregated student and teacher models in higher dimensions; The step of obtaining an optimized quantization network includes: Training the model: in, represents the combination of backbone networks, represents the backbone network of the old model, represents the backbone network of the new model, represents the normalization of the model ensemble, represents the loss function, Represents input and corresponding labels; According to the compression strategy of knowledge distillation, a new quantized model is trained under the supervision of the previously trained model ensemble: in, Represents the feature extraction backbone of the new model, consisting of the feature enhancement backbone and the backbone network composition, represents the cosine similarity function, Represents the basic loss function of each stage, Indicates including The overall feature extraction backbone, represents the feature similarity constraint function, represents the accumulation function, represents the output of the backbone network of the new network, Represents the output of the combined backbone network of the two trained models.

2. The hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation according to claim 1 is characterized in that: The spatial-spectral joint enhancement method is as follows: several target bands are selected in the spectral dimension, and pixel-level random adjustment is performed in the spatial region corresponding to the target bands; and the spatial domain data is enhanced by combining central cropping and scaling based on random size, as well as random horizontal flipping.

3. The hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation according to claim 1 is characterized in that: The optimized quantized network also includes introducing a trainable factor Adjust the backbone network of the new model .

4. A hyperspectral image incremental learning system based on feature space expansion and compression and spatial-spectral joint augmentation, characterized by: The system is implemented based on the hyperspectral image incremental learning method according to claim 1, and includes: A lightweight classification model building unit, used to build a lightweight classification model based on a spatial-spectral joint enhancement method; An online supervision unit is used to provide online supervision for the training of the lightweight network using the full-precision network as a teacher model in the initial stage; The optimization unit is used to introduce a new full-precision network in the incremental phase to narrow the gap between the old feature space and the target feature space, and obtain an optimized quantized network; A learning unit is used to achieve online update of the learning ability of the hyperspectral classification network based on the optimized quantization network.

5. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes a hyperspectral image class incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a hyperspectral image incremental learning method based on feature space expansion and compression and spatial-spectral joint augmentation as claimed in any one of claims 1 to 3.

Images