Freezing feature enhancement-based few-sample continuous learning method

Abstract

The invention discloses a few-sample continuous learning method based on frozen feature enhancement, and belongs to the field of image recognition. Few-sample continuous learning is the necessary ability of a general artificial intelligence model based on a pre-training model, and although a small amount of research proposes a method based on data enhancement to alleviate an overfitting problem, no research and exploration are provided to realize the same target by using feature enhancement. With wide application of a pre-training model in continuous learning, exploration of how to enhance output features (namely frozen features) of the pre-training model becomes crucial. The method comprises the following steps: 1) innovatively providing a few-sample continuous learning method based on frozen feature enhancement, and enhancing frozen features output by a pre-training model to improve data diversity so as to relieve an overfitting problem in few-sample continuous learning; and 2) the method provided by the invention is combined with a current advanced few-sample continuous learning framework, and the effectiveness of the method provided by the invention is verified on a classical few-sample continuous learning reference. And 3) by using a decision attribution stability index, the interpretability of using the frozen features to enhance in a few-sample continuous learning training process is verified.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and is mainly used to improve the few-shot continuous learning performance of deep learning models; it is mainly applied to the field of image classification. Background Art

[0002] In recent years, deep learning (DL) has made remarkable progress in the fields of computer vision (CV) and natural language processing (NLP); its goal is to train deep neural networks offline by using a fixed dataset, so that they can show excellent performance in corresponding tasks; in the early stage of deep learning, due to the limitations of data and computing resources, it was very difficult to train large-scale deep neural networks. With the improvement of big data and computing power, researchers began to try to train models on large-scale datasets and apply these models to other tasks, and pre-trained models emerged as a result. Pre-trained models are models trained on large-scale datasets, and these models can be widely applied to downstream tasks.

[0003] However, the data and tasks faced by the model are constantly changing, and the above-mentioned static models trained using fixed datasets cannot show excellent performance in some specific tasks. A possible solution is to retrain the network when the data distribution changes. However, since pre-trained models often have a large number of parameters, the computational cost of using the extended dataset for full training is unacceptable in the case of limited computing resources in the real world. In order to achieve general artificial intelligence, it is more desirable for the model to have the ability to gradually acquire, update, accumulate, and utilize knowledge throughout its entire life cycle, and this ability is called continuous learning or lifelong learning.

[0004] Continuous learning endows artificial intelligence systems with human-like learning abilities and is a key research direction in the field of artificial intelligence. In this field, few-shot class-incremental learning (FSCIL) is an important challenge. Different from traditional continuous learning methods that rely on a large amount of training data, FSCIL needs to learn knowledge by gradually integrating a small number of samples of new classes. Usually, after learning in a new stage, neural network models tend to forget the knowledge learned in the previous stage, and this phenomenon is called catastrophic forgetting. In FSCIL, the training data in each incremental stage is very limited, which not only exacerbates catastrophic forgetting but also causes overfitting problems. To solve these problems, recent research has focused on feature learning, aiming to improve the discrimination between basic classes while reserving enough feature space to support future new classes. Reference: Noor, A., Kukleva, A., and Schiele, B., "Orco: Towards better generalization via orthogonality and contrast for few-shot class-incremental learning," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28762-28771. However, in FSCIL, although a small number of studies have proposed data augmentation-based methods to alleviate the overfitting problem, no study has explored using feature augmentation to achieve the same goal. With the wide application of pre-trained models in continuous learning, it becomes crucial to explore how to enhance the output features (i.e., frozen features) of pre-trained models. In the few-shot incremental learning stage, through frozen feature augmentation, the diversity and scale of the training set can be improved to a certain extent, thereby enhancing the generalization ability of the few-shot continuous learning model and effectively alleviating the overfitting problem. Summary of the Invention

[0005] The present invention is a method for few-shot class-incremental learning (FSCIL). By introducing and improving the frozen feature augmentation technology, the pre-trained model can complete few-shot continuous learning in the sequential stage. This method enhances the processing by using the frozen features of the pre-trained model in each incremental learning stage, improves the diversity of few-shot class data, effectively alleviates the overfitting problem, and enhances the generalization ability of the model.

[0006] To conveniently describe the content of the present invention, some terms are first defined.

[0007] Definition 1: Transformer. A Transformer is a deep learning framework. Its core feature is the use of self-attention mechanism, which allows the model to consider all positions in the sequence simultaneously when processing sequential data, thus effectively capturing long-range dependencies. Compared with traditional methods, the Transformer provides higher parallel processing capabilities and efficiency. Since its introduction, it has become the mainstream framework for handling various complex tasks.

[0008] Definition 2: Vision Transformer. Vision Transformer (ViT) is a deep learning model for handling computer vision tasks. It innovatively applies the Transformer architecture to the field of image recognition. In ViT, the input image is first divided into a series of image patches, and then through a linear transformation, it is converted into a one-dimensional vector sequence (tokens). After adding positional information to these vectors, they are fed into the subsequent Transformer encoder to capture the complex relationships between image patches through the self-attention mechanism. Compared with traditional convolutional neural networks, ViT is completely based on the self-attention mechanism and demonstrates excellent image classification performance. It is the mainstream network framework used in current computer vision tasks.

[0009] Definition 3: Multi-Head Self-Attention mechanism. The Multi-Head Self-Attention mechanism is a core component of the Transformer architecture. In this mechanism, attention is decomposed into multiple "heads", and each "head" learns different aspects of the input data. Specifically, first, the input sequence is transformed into queries (Q), keys (K), and values (V); the self-attention of a single "head" can be expressed as: where d k represents the feature dimension. For multi-head attention, the model performs the above attention calculations in parallel, then concatenates their outputs, and finally performs a transformation through a linear layer. The multi-head self-attention mechanism can help the model focus on different parts of the sequence simultaneously, thereby improving the model's ability to capture complex data distributions.

[0010] Definition 4: Shapley value. The Shapley value is a concept from cooperative game theory used to measure the contribution of each participant (feature) to the final outcome in a cooperation process. In machine learning, the Shapley value is used to evaluate the impact of each feature on the model's prediction results, taking into account the marginal contribution of each feature in different feature combinations. The Shapley value has fairness as it considers the order and combination of features, thus assigning a contribution value to each feature. This method can be used to explain complex machine learning models, facilitating an understanding of which features play a key role in the model's decision-making.

[0011] The technical solution of the present invention is a few-shot continual learning method based on frozen feature enhancement, and the method includes:

[0012] Step 1: Preprocess the experimental data images;

[0013] Obtain real object images, label the images according to the image content; scale and crop the images to a specified size, and normalize the pixel values of all images; then divide the images into multiple data sets, and each data set corresponds to a few-shot continual learning stage;

[0014] Step 2: Construct a few-shot continual learning model with a pre-trained model as the backbone network;

[0015] The constructed few-shot continual learning model is mainly divided into three modules: a feature extraction module with a pre-trained model as the backbone network, a feature fusion module composed of multi-head attention blocks, and a feature mapping module composed of multi-layer perceptrons; use Vision Transformer as the backbone network, and its first layer divides the image into multiple image patches, and after linear mapping, a series of one-dimensional sequence vectors are obtained; then there are 12 Transformer blocks, and each Transformer block includes a self-attention calculation module and a multi-layer perceptron module; the input of the backbone network is the image W, H, and C respectively represent the width, height, and number of channels of the feature space, and the output is the frozen feature of the image Where N represents the number of image patches, D represents the feature dimension, represents the class label, represents the feature of the corresponding image patch; the multi-head attention block is composed of 1 layer of Transformer block, and is used to fuse and enhance the frozen feature to obtain an enhanced class label, and its input is the enhanced frozen feature The output is the class label The multi-layer perceptron is composed of two linear layers and a ReLU activation function, and is used to map the class label from a high-dimensional space to a low-dimensional space, and the input is the class label The output is a d-dimensional feature The classifier is used to classify the features output by the model; therefore, the backbone network is represented as: The multi-head attention block is represented as: The multi-layer perceptron is represented as

[0016] Step 3: Implement frozen feature enhancement;

[0017] First, the frozen features Z = [z cls ; z1; z2; … z N obtained through the backbone network are split into two parts: z cls represents the class token, represents the patch features;

[0018] Among them, Z patch is reshaped into a three-dimensional form of Then, min-max normalization is used for each of the D feature channels to map each element to the interval as follows:

[0019]

[0020] where z max and z min are the maximum and minimum values of each channel respectively. Subsequently, an independent random value is added to each of the D feature channels, where represents a uniform distribution;

[0021] Then, the enhanced result is mapped back to the value range of the original features to obtain the final representation:

[0022]

[0023] where a(·) represents the brightness enhancement operation. Then, is reshaped back into a two-dimensional form, denoted as and concatenated with the original class token z cls to obtain the enhanced frozen features [z cls ; Z ′ patch , where [·;·] represents the row-wise concatenation operation;

[0024] Step 4: Use the feature fusion module to obtain the enhanced class token;

[0025] Step 5: Use the feature mapping module to implement dimensionality reduction of the class token;

[0026] The enhanced class token z ′ cls obtained in Step 4 is fed into the feature mapping module to obtain the d-dimensional feature h for subsequent classification tasks:

[0027]

[0028] Step 6: Construct a constraint mechanism using the OrCo framework;

[0029] The OrCo loss consists of three parts: perturbation supervised contrastive loss, cross-entropy loss, and orthogonality loss;

[0030] Step 7: First, process the dataset according to Step 1 to obtain datasets at multiple stages; then, construct a few-shot continual learning model with the pre-trained model as the backbone network according to Step 2; finally, use the loss function constructed in Step 6 to perform constraints, and train the model on the training set of each stage in the order of stage arrival;

[0031] Step 8: After training the model at each stage, use the test samples of all stages for classification prediction to evaluate the performance of the model; first, extract the class label z of the test sample X using the backbone network and the multi-head attention block cls ; Next, input the class label z cls into the MLP feature mapping module g(·) to obtain a d-dimensional feature h = g(z cls ); Then, calculate the cosine similarity between h and the pseudo-target vector

[0032]

[0033] where denotes the set of all classes, cos(·,·) denotes the cosine similarity, and the output class is the class with the maximum cosine similarity.

[0034] Furthermore, the specific method of Step 4 is as follows:

[0035] Send the enhanced frozen features [z cls ; Z ′ patch obtained in Step 3 into the feature fusion module, which fuses the enhanced frozen features with the original class label to construct an enhanced class label:

[0036]

[0037] where [·;·] denotes concatenation by rows. For the same few-shot class sample X, at the τ-th stage, τ > 0, apply frozen feature enhancement to it to generate M enhanced features {Z ′ 1, Z ′ 2, … Z ′ M}, and use these features to generate M enhanced class labels {z′ cls,1 , z ′ cls,2 , …, z ′ cls,M}; Since the dimension D of the class label is relatively high, which is not convenient for the classification of subsequent models, a feature mapping module is used to map these enhanced features from the high-dimensional feature space to the low-dimensional feature space.

[0038] Furthermore, the method of introducing the perturbation supervised contrastive loss in step 6 is as follows: Generate and fix a set of mutually orthogonal vectors, called the pseudo-target vector set, where is equal to the total number of classes; each class will be assigned a pseudo-target vector as the class prototype for subsequent classification;

[0039] Subsequently, apply uniform perturbations to these pseudo-target vectors to generate a series of perturbed pseudo-target vectors:

[0040]

[0041] where, represents the uniform distribution, and λ represents the sampling range; in the perturbation supervised contrastive loss, the positive sample set not only includes the other sample features from the same class, but also includes the pseudo-target vector assigned to this class and its perturbed versions, further expanding the diversity of the positive sample set; the negative sample set includes the sample features from other classes and other pseudo-target vectors; when introducing frozen feature enhancement, the different enhanced features of the same sample will be included in the positive sample set, while the negative sample set will include the enhanced features of the samples from other classes.

[0042] Furthermore, the method of introducing the cross-entropy loss in step 6 is that the cross-entropy loss is used to align all the enhanced features h of the samples of a certain class i = g(z ′ cls,i ), i ∈ [1, M] with the pseudo-target vector t c assigned in the current stage τ; project the enhanced sample features onto the pseudo-target vector, encouraging the features of the same class to be close to the corresponding pseudo-target vector, ensuring classification consistency.

[0043] Furthermore, the method of introducing the orthogonal loss in step 6 is as follows: In stage τ, the orthogonal loss ensures that the class mean and other pseudo-target vectors are orthogonal in the

[0044] feature space, reducing the interference between different classes. N k represents the number of samples in class k, and the orthogonal loss is as follows:

[0045]

[0046] Among them, τ o is the temperature parameter.

[0047] Furthermore, the following method is used to evaluate the model:

[0048] The harmonic mean is used as the evaluation index to measure the classification accuracy in the few-shot incremental learning stage; the calculation method of the harmonic mean accuracy is where a0 represents the accuracy of the model on the base classes, and a 1:τ represents the average accuracy of the model on the few-shot incremental classes from the first stage to the τ-th stage; the harmonic mean accuracy of the final stage is defined as and the average harmonic mean accuracy of all stages is denoted as

[0049] The innovation of the present invention lies in:

[0050] (1) An innovative few-shot continuous learning method based on frozen feature enhancement is proposed. This method improves data diversity by enhancing the frozen features output by the pre-trained model, thereby alleviating the overfitting problem in few-shot continuous learning.

[0051] (2) The method proposed in the present invention is combined with the current advanced few-shot continuous learning framework, and the effectiveness of the method proposed in the present invention is verified on the classical few-shot continuous learning benchmark.

[0052] (3) Using the decision attribution stability index, the interpretability of using frozen feature enhancement in the few-shot continuous learning training process is verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is the Transformer structure diagram of the method of the present invention

[0054] Figure 2 is the Vision Transformer structure diagram used in the method of the present invention

[0055] Figure 3 is the framework schematic diagram of the method of the present invention DETAILED DESCRIPTION OF THE INVENTION

[0056] This method first proposes a frozen feature enhancement method based on brightness adjustment for FSCIL. In each incremental learning stage, first use a frozen pre-trained model (Vision Transformer, ViT) to extract the frozen features of the samples, and then enhance the brightness of these features to generate diverse enhanced features. These enhanced features are then fed into a multi-head attention block (MAB) to produce enhanced class tokens for subsequent classification.

[0057] The focus of this method is to introduce a more suitable enhancement method for few-shot continuous learning of pre-trained models, namely frozen feature enhancement, and improve it. At the same time, three special losses are introduced for constraint, further improving the performance of the model. In addition, through decision attribution stability analysis, it is proved that this feature enhancement scheme does not affect the interpretability of the few-shot continuous learning model.

[0058] From the perspective of method design and application, the present invention mainly does the following work: 1) Innovatively propose a few-shot continuous learning method based on frozen feature enhancement, which improves data diversity by enhancing the frozen features output by the pre-trained model, thus alleviating the overfitting problem in few-shot continuous learning. 2) Combine the method proposed by the present invention with the current advanced few-shot continuous learning framework, and verify the effectiveness of the method proposed by the present invention on a classic few-shot continuous learning benchmark. 3) Use the decision attribution stability index to verify the interpretability of using frozen feature enhancement in the process of few-shot continuous learning training.

[0059] Step 1: Preprocess the experimental data images;

[0060] The experiments are evaluated on an FSCIL benchmark dataset and two multi-modal datasets: CIFAR-100, UESTC-MMEA-CL, and ARIC.

[0061] CIFAR-100: Follow the data processing method of the OrCo framework, with an image resolution of 224*224. Set 60 categories as base categories and 40 categories as few-shot incremental categories. There is a total of 1 base stage and 8 few-shot incremental stages in the experiment. In the base stage, each base category contains 500 samples. In each incremental stage, there are 5 incremental categories, and each incremental category has 5 samples.

[0062] UESTC-MMEA-CL: This is a multi-modal continuous learning dataset, which includes three modalities: video, accelerometer data, and gyroscope data, with a total of 32 daily activity categories. There are 16 base categories and 16 incremental categories. The incremental stage is set to 4 categories per stage, and each category has 5 samples.

[0063] ARIC: This is an activity recognition dataset extracted from actual classroom monitoring, containing 32 classroom activities in three modalities: images, text, and audio. 20 categories are basic categories, and the remaining 12 categories are divided into 4 stages. The incremental stage is set to 3 categories per stage, with 5 samples per category.

[0064] Step 2: Construct a few-shot continual learning model with a pre-trained model as the backbone network

[0065] The few-shot continual learning model constructed in the present invention is mainly divided into three modules: a feature extraction module with a pre-trained model as the backbone network, a feature fusion module composed of multi-head attention blocks (MAB), and a feature mapping module composed of multi-layer perceptrons (MLP). Vision Transformer is used as the backbone network. Its first layer divides the image into multiple image patches, and after linear mapping, a series of one-dimensional sequence vectors are obtained; then there are 12 Transformer blocks, and each Transformer block includes a self-attention calculation module and a multi-layer perceptron (MLP) module. The input of the backbone network is an image The output is the frozen features of the image where N represents the number of image patches (Patch), D represents the feature dimension, represents the class token, represents the features of the corresponding image patch. The multi-head attention block is composed of 1 layer of Transformer block, which is used to fuse and enhance the frozen features to obtain the augmented class token. Its input is the augmented frozen features The output is the class token The multi-layer perceptron is composed of two linear layers and a ReLU activation function, which is used to map the class token from a high-dimensional space to a low-dimensional space. The input is the class token The output is the d-dimensional feature The classifier is used to classify the features output by the model. Therefore, the backbone network can be expressed as: The multi-head attention block is expressed as: The multi-layer perceptron is expressed as See the structure of the Transformer block in Figure 1 and see the structure of Vision Transformer in Figure 2 .

[0066] Step 3: Implement frozen feature enhancement

[0067] Freezing Feature Enhancement, as a feature processing method, its core idea is to introduce data augmentation into feature processing. For example, image data augmentation includes brightness enhancement, contrast enhancement, sharpening, etc. Similar to data augmentation, feature enhancement can also improve data diversity, thus alleviating the overfitting problem caused by insufficient samples in the few-shot continuous learning stage. The present invention introduces freezing feature enhancement to improve data diversity in the few-shot continuous learning stage. First, the frozen feature Z = [z cls ; z1; z2; … z N obtained through the backbone network is split into two parts: z cls represents the class label, represents the image patch feature.

[0068] Reshape the Z patch among them into a three-dimensional form of Then, perform min-max normalization on each of the D feature channels respectively, so as to map each element to the interval :

[0069]

[0070] where z max and z min are the maximum and minimum values of each channel respectively. Subsequently, add an independent random value to each of the D feature channels, where represents a uniform distribution. This process is similar to the brightness enhancement of images.

[0071] Then, map the enhanced result back to the value range of the original feature to obtain the final representation:

[0072]

[0073] where a(·) represents the brightness enhancement operation. Then, reshape back to a two-dimensional form, denoted as and concatenate it with the original class label z cls to obtain the enhanced frozen feature [z cls ; Z ′ patch , where [·; ·] represents the row-wise concatenation operation.

[0074] Step 4: Obtain the enhanced class label using the feature fusion module

[0075] Use the enhanced frozen feature [z cls ; Z ′ patchFeed into the MAB, i.e., the feature fusion module, which can fuse the enhanced frozen features with the original class labels to construct enhanced class labels:

[0076]

[0077] where [·;·] denotes concatenation by rows. For the same few-shot class sample X (at the τ-th stage, τ > 0), apply frozen feature enhancement to it to generate M enhanced features {Z ′ 1,Z ′ 2,…Z ′ M}, and use these features to generate M enhanced class labels {z ′ cls,1 ,z ′ cls,2 ,…,z ′ cls,M}. Since the dimension D of the class labels is relatively high and not convenient for subsequent model classification, a feature mapping module is used to map these enhanced features from the high-dimensional feature space to the low-dimensional feature space.

[0078] Step 5: Implement dimensionality reduction of class labels using the feature mapping module

[0079] Feed the enhanced class labels z ′ cls obtained in Step 4 into the MLP, i.e., the feature mapping module, to obtain d-dimensional features for subsequent classification tasks:

[0080]

[0081] However, relying solely on traditional few-shot continual learning models is difficult to effectively address the problems of catastrophic forgetting and inter-class interference. In the setting of few-shot continual learning, the model needs to be continuously updated to adapt to the data distribution in the new stage, which may cause the model parameters to gradually deviate from their optimal solutions in the historical stages, leading to an exacerbation of the model's forgetting of the knowledge in the old stages. At the same time, with the introduction of new classes, the representations of different classes in the feature space may be confused, weakening the discriminative ability of the model. Therefore, a new constraint mechanism needs to be introduced to impose more stringent structural constraints on the learning process of the model to ensure that the model retains a good memory of the historical stages while adapting to the current stage, and optimizes the separability and consistency of the feature space.

[0082] Step 6: Construct a constraint mechanism using the OrCo framework

[0083] As a typical FSCIL framework, OrCo has achieved the current best performance in few-shot continual learning by constraining the alignment between class features and pseudo-target vectors. In the present invention, frozen feature enhancement will be combined with the OrCo framework. The OrCo loss consists of three parts: perturbation supervised contrast loss, cross-entropy loss, and orthogonality loss.

[0084] To maximize the margin between different classes in the feature space, perturbation supervised contrast loss is first introduced. A set of mutually orthogonal vectors, called the pseudo-target vector set, is generated and fixed first, where is equal to the total number of classes. Each class will be assigned a pseudo-target vector as the class prototype for subsequent classification. Subsequently, uniform perturbations are applied to these pseudo-target vectors to generate a series of perturbed pseudo-target vectors:

[0085]

[0086] where represents the uniform distribution, and λ represents the sampling range. In the perturbation supervised contrast loss, the positive sample set includes not only the other sample features from the same class, but also the pseudo-target vector assigned to this class and its perturbed versions, further expanding the diversity of the positive sample set. The negative sample set includes the sample features from other classes and other pseudo-target vectors. When frozen feature enhancement is introduced, different enhanced features of the same sample will be included in the positive sample set, while the negative sample set will include the enhanced features of samples from other classes.

[0087] The perturbation supervised contrast loss promotes the separation of class features by maximizing the margin between classes in the feature space, and at the same time enhances the discriminative ability of the model by using the diversity of positive samples, enabling the model to not only learn to distinguish different classes, but also capture the complex differences between classes in the few-shot continual learning scenario.

[0088] To further improve the proximity of features of the same class to the pseudo-target vector, cross-entropy loss is also introduced. The cross-entropy loss is used to align all enhanced features h of samples of a certain class i = g(z ′ cls,i ), i ∈ [1, M] with the pseudo-target vector t c assigned in the current stage τ. This loss projects the enhanced sample features onto the pseudo-target vector, encouraging the features of the same class to be close to the corresponding pseudo-target vector and ensuring classification consistency.

[0089] In addition, to minimize the interference between different categories as much as possible, an orthogonal loss is introduced. In stage τ, the orthogonal loss ensures, through geometric constraints, that the class means (where N k represents the number of samples in class k) and other pseudo-target vectors remain orthogonal in the feature space, reducing the interference between different categories. The form of the orthogonal loss is as follows:

[0090]

[0091] where τ o is the temperature parameter. In practice, this constraint is applied to all augmented features in the current training batch. By optimizing the OrCo loss, the boundaries between classes in the feature space can be expanded, thereby reducing class confusion.

[0092] Step 7: (1) Process the dataset according to Step 1 to obtain datasets for multiple stages; (2) Construct a few-shot continual learning model with the pre-trained model as the backbone network according to Step 2; (3) Use the loss function constructed in Step 6 for constraint, and train the model on the training set of each stage in the order of stage arrival.

[0093] Step 8: After training the model in each stage, use the test samples of all stages for classification prediction to evaluate the performance of the model. First, use the backbone network (pre-trained ViT) and the multi-head attention block (MAB) to extract the class token z cls of the test sample X. Next, input the class token z cls into the MLP feature mapping module g(·) to obtain a d-dimensional feature h = g(z cls ). Then, calculate the cosine similarity between h and the pseudo-target vectors assigned to each class:

[0094]

[0095] where represents the set of all classes, and cos(·,·) represents the cosine similarity. The output class is the class with the maximum cosine similarity. The present invention uses the harmonic mean as the evaluation metric to measure the classification accuracy in the few-shot incremental learning stage. The calculation method of the harmonic mean accuracy is where a0 represents the accuracy of the model on the base classes, and a 1:τ represents the average accuracy of the model on the few-shot incremental classes from the first stage to the τ-th stage. In addition, define the harmonic mean accuracy of the final stage as and denote the average harmonic mean accuracy of all stages as See Tables 1 and 2 for the test results.

[0096] Step 9: Decision Attribution Stability Analysis

[0097] The frozen feature augmentation perturbs the original image features. Therefore, it is further explored whether this perturbation will affect the interpretability of the few-shot continual learning model. Two Shapley value-based metrics have been proposed in the research to measure the decision attribution stability of the continual learning model. These two metrics first calculate the SHAP values of each feature component of the sample to quantify the importance of this component. Subsequently, the decision attribution stability of the continual learning model is measured by evaluating the decision attribution consistency of the same sample at different stages. The metrics are as follows: (1) Shapley Decision Consistency (SHAPC Mean): Measures the decision consistency of the model on the same sample, and the higher the value, the better. (2) Shapley Decision Variance (SHAPC Var): Evaluates the volatility of the model's decision attribution, and the lower the value, the better. In this study, these two metrics are used to evaluate the impact of frozen feature augmentation on the interpretability of the few-shot continual learning model.

[0098] Experiments on the CIFAR-100, UESTC-MMEA-CL, and ARIC datasets show that there is no significant difference in decision consistency and variance between FAOrCo-ViT and OrCo-ViT. This indicates that the feature augmentation method proposed in this paper does not harm the interpretability of the few-shot continual learning model. The experimental results are shown in Table 3.

[0099] Table 1 shows the comparison between the proposed method and previous methods in the benchmark phase and incremental phase on the CIFAR-100 dataset. All results are presented in the form of percentages (%).

[0100] Table 1

[0101]

[0102] Table 2 shows the results of the proposed method on two multi-modal datasets. All results are presented in the form of percentages (%).

[0103] Table 2

[0104]

[0105] Table 3 shows the decision stability evaluation results of the proposed method on three datasets. All results are presented in the form of percentages (%).

[0106] Table 3

[0107]

[0108] As can be seen from Table 1 above, the experimental results of FAOrCo-ViT with frozen feature enhancement added to OrCo-ViT show that the harmonic mean accuracy of the present invention in the final stage is improved by 1.6% compared to OrCo-ViT, and remains leading in the average harmonic mean accuracy at all stages . As can be seen from Table 2, frozen feature enhancement can still effectively improve the diversity of few-shot class training samples on other datasets, thereby improving the performance of the few-shot continual learning model. As can be seen from Table 3, the experiments on the CIFAR-100, UESTC-MMEA-CL, and ARIC datasets show that the differences in decision consistency and variance between FAOrCo-ViT and OrCo-ViT are not significant. This indicates that the frozen feature enhancement method proposed by the present invention does not damage the interpretability of few-shot continual learning.

Claims

1. A few-shot continual learning method based on frozen feature enhancement, the method comprising: Step 1: Preprocess the experimental data images; Obtain real object images, and perform class annotation on the images according to the image content; Scale and crop the images to a specified size, and normalize the pixel values of all images; Then divide the images into multiple data sets, each data set corresponding to a few-shot continual learning stage; Step 2: Construct a few-shot continual learning model with a pre-trained model as the backbone network; The constructed few-shot continuous learning model is mainly divided into three modules: a feature extraction module with a pre-trained model as the backbone network, a feature fusion module composed of multi-head attention blocks, and a feature mapping module composed of multi-layer perceptrons; Vision Transformer is used as the backbone network, and its first layer divides the image into multiple image patches, and a series of one-dimensional sequence vectors are obtained after linear mapping; then there are 12 Transformer blocks, each of which includes a self-attention calculation module and a multi-layer perceptron module; the input of the backbone network is the image W, H, and C represent the width, height, and number of channels of the feature space respectively, and the output is the frozen feature of the image where N represents the number of image patches and D represents the feature dimension represents the class token represents the feature of the corresponding image patch The multi-head attention block consists of 1 layer of Transformer blocks, which is used to fuse and enhance the frozen features to obtain enhanced class tokens. Its input is the enhanced frozen features The output is the class token The multi-layer perceptron consists of two linear layers and a ReLU activation function, which is used to map the class tokens from a high-dimensional space to a low-dimensional space. Its input is the class token The output is the d-dimensional feature The classifier is used to classify the features output by the model; Therefore, represent the backbone network as: f(·): Represent the multi-head attention block as: h(·): Represent the multi-layer perceptron as g(·): Step 3: Implement frozen feature enhancement; First, the frozen feature Z = [z cls ; z1; z2; … z N obtained through the backbone network is split into two parts: z cls represents the class label, represents the image patch feature; Reshape the Z among them patch into a three-dimensional form Then, perform min-max normalization on each of the D feature channels, mapping each element to the interval as follows: where z max and z min are the maximum and minimum values of each channel, respectively. Subsequently, an independent random value is added to each of the D feature channels where represents a uniform distribution; Then, map the enhanced result back to the value range of the original feature to obtain the final representation: where a(·) represents the brightness enhancement operation, and then is reshaped back to a two-dimensional form, denoted as and concatenated with the original class label z cls to obtain the enhanced frozen feature [z cls ; Z′ patch , where [·;·] represents the row-wise concatenation operation; Step 4: Use a feature fusion module to obtain enhanced class labels; Step 5: Use a feature mapping module to implement dimensionality reduction of class labels; Send the enhanced class label \(z'\) obtained in step 4 cls into the feature mapping module to obtain the \(d\)-dimensional feature \(h\) for subsequent classification tasks: Step 6: Use the OrCo framework to construct a constraint mechanism; The OrCo loss consists of three parts: perturbation supervised contrast loss, cross-entropy loss, and orthogonality loss; Step 7: First, process the data set according to Step 1 to obtain data sets for multiple stages; then construct a few-shot continual learning model with a pre-trained model as the backbone network according to Step 2; finally, use the loss function constructed in Step 6 for constraint, and train the model on the training set of each stage in the order of stage arrival; Step 8: After training the model in each stage, use the test samples of all stages for classification prediction to evaluate the performance of the model; first, use the backbone network and the multi-head attention block to extract the class token z of the test sample X cls ; Next, input the class token z cls into the MLP feature mapping module g(·) to obtain a d-dimensional feature h = g(z cls ); Then, calculate the cosine similarity between h and the pseudo target vector assigned to each class Among them, represents the set of all categories, cos(·,·) represents the cosine similarity, and the output category is the category with the maximum cosine similarity.

2. The few-shot continuous learning method based on enhanced frozen features as claimed in claim 1, wherein, The specific method of Step 4 is: Feed the enhanced frozen feature [z cls ; Z′ patch into the feature fusion module, which fuses the enhanced frozen feature with the original class label to construct an enhanced class label: where [·;·] denotes concatenation by rows. For the same few-shot class sample X, at the τ-th stage, τ > 0, frozen feature augmentation is applied to it to generate M augmented features {Z′1, Z′2, … Z′ M}, and these features are used to generate M augmented class labels {z′ cls,1 , z′ cls,2 , …, z′ cls,M}; Since the dimension D of the class label is relatively high and not convenient for the subsequent classification of the model, a feature mapping module is used to map these augmented features from the high-dimensional feature space to the low-dimensional feature space.

3. A few-shot continuous learning method based on freeze feature enhancement according to claim 1, characterized in that The method of the perturbation supervised contrastive loss introduced in step 6 is as follows: generate and fix a set of mutually orthogonal vectors, called the pseudo-target vector set, where is equal to the total number of categories; each category will be assigned a pseudo-target vector as the category prototype for subsequent classification; Subsequently, apply uniform perturbations to these pseudo-target vectors to generate a series of perturbed pseudo-target vectors: Among them, represents a uniform distribution, and λ represents the sampling range; in the perturbed supervised contrastive loss, the positive sample set not only includes the other sample features from the same category, but also includes the pseudo-target vector assigned to this category and its perturbed versions, further expanding the diversity of the positive sample set; the negative sample set includes the sample features from other categories and other pseudo-target vectors; when introducing frozen feature enhancement, different enhanced features of the same sample will be included in the positive sample set, while the negative sample set will include the enhanced features of samples from other categories.

4. A few-shot continuous learning method based on enhanced frozen features as claimed in claim 1, wherein, The method of the cross-entropy loss introduced in Step 6 is that the cross-entropy loss is used to align all the enhanced features h of the samples of a certain category i = g(z′ cls,i ), i ∈ [1, M] with the pseudo-target vector t c assigned in the current stage τ; project the enhanced sample features onto the pseudo-target vector, encourage the features of the same category to be close to the corresponding pseudo-target vector, and ensure classification consistency.

5. A few-shot continuous learning method based on enhanced freezing features as described in claim 1, characterized in that, The method of the orthogonal loss introduced in step 6 is as follows: in stage τ, the orthogonal loss ensures, through geometric constraints, that the class means and other pseudo-target vectors remain orthogonal in the feature space, reducing the interference between different classes. N k represents the number of samples of class k, and the orthogonal loss is as follows: Among them, τ o is the temperature parameter.

6. The few-shot continuous learning method based on enhanced freezing features as claimed in claim 1, wherein The model is evaluated using the following method: The harmonic mean is used as an evaluation metric to measure the classification accuracy in the few-shot incremental learning phase; the calculation method of the harmonic mean accuracy is where a0 represents the accuracy of the model on the base classes, and a 1:τ represents the average accuracy of the model on the few-shot incremental classes from the first stage to the τ-th stage; define the harmonic mean accuracy of the final stage as and denote the average harmonic mean accuracy of all stages as

Images