An image classification method based on important parameter constraint continuous learning

Abstract

The application discloses an image classification method based on important parameter constraint continuous learning, and belongs to the field of image processing. It is found that in a very high dimensional parameter space, even if the LoRA module is constrained to be orthogonal, it may also lead to a suboptimal solution, so that the model cannot completely alleviate the forgetting problem. In the application, (1) it is found that even under the orthogonal LoRA constraint condition, the model parameters sensitive to the loss of historical tasks will still change significantly on each task; (2) the importance of each trainable model parameter in the current task is defined according to the sensitivity of the model parameter to the loss; (3) based on the importance of the model parameter, an important parameter constraint method is proposed, which effectively alleviates the forgetting of the model to the knowledge of the current task by constraining the parameters of the model of the current task from changing in subsequent tasks. Experiments prove that the method proposed in the application exhibits excellent performance on multiple continuous learning data sets.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of machine learning, mainly used to improve the continuous learning performance of deep learning models; mainly applied in the field of image classification. BACKGROUND

[0002] In the field of artificial intelligence today, deep learning (DL) technology has become the core technology to promote computer vision and natural language processing. Through the training of deep neural networks (DNNs), it can exhibit superior performance to traditional algorithms on complex tasks such as machine translation, image classification, object detection, and instance segmentation. The basic training method of deep neural networks is to apply optimization algorithms such as stochastic gradient descent on pre-collected and processed fixed-size data sets, optimize network model parameters, and make them achieve the expected effect on specific tasks. This training method has achieved great success in offline training phase, making deep neural networks reach or even surpass human level on specific tasks. However, this offline training method also has some limitations. Once the training is completed, the parameters of the network model are fixed and cannot adapt to the subsequent changing data distribution or new environmental conditions, which makes the model perform poorly on these unexposed tasks. If this static model is deployed to real application scenarios where data or tasks are constantly changing, it is obviously unable to respond effectively. For example, in the fields of autonomous driving, video surveillance, intelligent medical treatment, etc., due to the dynamic nature of the environment, the data distribution may change over time, which requires the model to continuously learn and adapt to the changing data distribution. In the face of new data distribution, we can consider retraining the model, but this requires a lot of computing resources, and as the amount of data increases, the cost of retraining the entire network will also increase. In the case of limited storage space and computing resources, this is unrealistic. At the same time, traditional deep neural networks will have a "catastrophic forgetting" phenomenon when facing dynamic environments, that is, the model will forget the old task knowledge when learning new tasks. The reason for this phenomenon is that when the model is trained on new tasks, the model parameters will be adjusted again to achieve optimal performance on new tasks; due to the change of model parameters, the model's performance on old tasks may decline. Therefore, a new learning method is needed to enable the model to continuously learn new task knowledge while retaining old task knowledge under limited resources.

[0003] Continual Learning (CL) is a method to solve the above problems, which aims to enable the model to acquire, update, accumulate and utilize knowledge step by step throughout its life cycle, like humans, so as to cope with changing data distribution and environmental conditions. One of the key challenges of continual learning is how to learn new knowledge without forgetting the old task knowledge, that is, how to avoid the "catastrophic forgetting" problem. In the past few years, researchers have proposed various strategies to overcome the above problems, such as regularization-based methods, dynamic architecture adjustment methods, memory replay methods, etc. Regularization-based methods limit the impact of network parameter updates on old knowledge by adding additional terms to the loss function. Dynamic architecture adjustment dynamically expands the network structure when the network needs to learn new tasks, so that new and old task knowledge can coexist. Memory replay strategy stores part of the historical data and replays these data while learning new tasks to reduce forgetting.

[0004] The above continuous learning strategies usually start from the initial state of the model and test all the test samples of the tasks using the model learned from the last task. In recent years, with the emergence of pre-trained models, combining continuous learning with powerful pre-trained models has become the mainstream in this field. Pre-trained models use the idea of transfer learning to first learn the general features of the samples on a large amount of data, and then apply these features to specific tasks. Existing work has shown that with the powerful representation ability learned by pre-trained models on a large amount of data, directly using pre-trained models to extract sample features can achieve good performance on the test tasks of continuous learning. With the emergence of various parameter-efficient fine-tuning (PEFT) methods, these fine-tuning methods are improved to combine pre-trained models to complete continuous learning, and their performance on various continuous learning benchmark datasets has exceeded traditional methods. Past work mostly uses Prompt tuning, a fine-tuning method, to combine pre-trained models to complete continuous learning, and its main idea is to select a set of learnable parameter vectors, called prompts, for each task's samples through a query mechanism during training, and concatenate them with the image vectors input to the model for training. The disadvantage of this approach is that the learning ability of the model on each task is limited by the length of the parameter vector. In addition, there are also works that use Adapter, a lightweight fine-tuning strategy, to complete continuous learning. Existing work points out that low rank adaptation tuning (LoRA tuning) has become a more suitable fine-tuning strategy for pre-trained models for continuous learning due to its low computational cost and high representation quality.In recent years, some works have shown that the forgetting of historical task knowledge of the continuous learning model can be significantly reduced by orthogonal LoRA fine-tuning; Ref: Wang X, Chen T, Ge Q, et al. Orthogonal subspace learning for language model continual learning [J]. arXiv preprint arXiv:2310.14152, 2023. These works are inspired by the orthogonal gradient descent algorithm; Ref: Farajtabar M, Azizan N, Mott A, et al. Orthogonal gradient descent for continual learning [C] / / International Conference on Artificial Intelligence and Statistics. PMLR, 2020: 3762-3773. It is believed that merging the orthogonal LoRA module into the parameters of the pre-trained model will not change the training loss of the historical task; Therefore, since the training loss of the historical task will not be changed in the continuous learning process, the forgetting of the model to the historical task knowledge can be significantly alleviated. However, it is found that under the orthogonal LoRA fine-tuning, the model parameters sensitive to the loss of the historical task will still change significantly in the continuous learning process; This means that the loss of the model to the historical task will still change significantly, and the forgetting of the model to the historical task knowledge has not been completely alleviated. It is believed that the reason for this phenomenon is that in the extremely high-dimensional parameter space, there are too many potential orthogonal solutions, which may eventually lead to the emergence of suboptimal solutions. Based on the above findings, the present application proposes an important parameter constraint method; This method defines the importance of the parameters according to the sensitivity of the parameters to the training loss, and by constraining the model parameters important to the current task from changing in subsequent tasks, the forgetting of the model to the current task knowledge is alleviated, and the model performance is improved; And combined with the orthogonal LoRA fine-tuning strategy, a more stable and robust continuous learning model is constructed, and the continuous learning technology based on the pre-trained model is continuously developed. SUMMARY

[0005] The application is a continuous learning image classification method based on important parameter constraint. By constraining the model parameters important to the current task from changing in subsequent tasks and combining the orthogonal LoRA fine-tuning strategy, the pre-trained model can perform more stable and efficient continuous learning on sequential tasks. The method first discovers that under orthogonal LoRA fine-tuning, model parameters sensitive to historical task loss will still change significantly. Then, according to the sensitivity of model parameters to loss, the parameter importance is defined. Combined with the parameter importance, the method constrains the model parameters important to the current task from changing in subsequent tasks, and alleviates the forgetting of model knowledge about the current task. On the basis of orthogonal LoRA fine-tuning, the method adds the important parameter constraint strategy, effectively improving the stability and robustness of the model.

[0006] The method first designs a pre-trained model continuous learning paradigm based on orthogonal LoRA fine-tuning. For sequential tasks, the method learns a LoRA module specific to the task on each task, and after the current task training is completed, the LoRA weight learned by the previous task is combined with the LoRA weight of the current task, where the combination coefficient is a series of learnable parameters. In order to alleviate the knowledge interference between different tasks, the method constrains the LoRA modules corresponding to different tasks to be mutually orthogonal, thereby further improving the adaptability of the model. In addition, since there are too many potential orthogonal solutions in high-dimensional parameter space, it may lead to suboptimal solution of orthogonal LoRA fine-tuning, the method further proposes to constrain the model parameters important to the current task, effectively improving the model performance. The focus of the method is to find that the model parameters sensitive to the loss of historical tasks will still change in subsequent tasks, and then propose to constrain the important parameters from changing, in order to alleviate the forgetting of model knowledge about historical tasks, and enhance the stability and robustness of the model. From the method design and application point of view, the following work is mainly done: 1) a continuous learning method based on pre-trained model is proposed, which uses orthogonal LoRA module combination to retain previous knowledge and incorporate new knowledge in the continuous learning process; by assigning learnable weight parameters to different LoRA modules, the adaptability of orthogonal LoRA fine-tuning is further enhanced. 2) It is revealed that under orthogonal LoRA fine-tuning, important parameters sensitive to the training loss of the current task will change significantly in different tasks; and based on this discovery, important parameter constraint is proposed to enhance the stability of the model. 3) The method achieves optimal performance on multiple classical continuous learning benchmarks in the replay-free class-incremental learning scene, effectively proving the effectiveness of the method proposed in the application.

[0007] In order to conveniently describe the content of the application, some terms are defined first.

[0008] Definition 1: Low-Rank Adaption. Low-Rank Adaption (LoRA) is a way to fine-tune pre-trained models. The core idea is that pre-trained models exhibit low intrinsic dimensionality when adapting to downstream tasks, i.e., the weight changes during adaptation have intrinsic low-rankness. Therefore, the incremental parameters of the model can be decomposed into the product of two low-rank matrices. When fine-tuning the pre-trained model, only the parameters of these two low-rank matrices are updated, while other model parameters remain unchanged. Its expression is:

[0009] W0+ΔW=W0+AB

[0010] where A and B are both low-rank matrices with rank r.

[0011] Definition 2: Vision Transformer. Vision Transformer (ViT) is a mainstream deep learning model for processing computer vision tasks. Its model structure mainly includes three parts: image block embedding, transformer module, and linear classification head. The transformer module can achieve good sample feature extraction effect through multi-layer stacking. ViT divides the image into fixed-size image blocks (patches) and treats each image block as a "word embedding", similar to the word vector in natural language processing. By inputting these embeddings into the transformer module for processing, ViT can capture global features in images and achieve efficient image classification and recognition. Compared with traditional deep learning models, ViT performs well in processing large-scale image data, especially when training on large datasets.

[0012] Definition 3: Transformer Module. The transformer module in ViT mainly includes two parts: multi-head self-attention mechanism and feed-forward neural network. First, the image block embedding is processed by the multi-head self-attention mechanism to capture global context information; the feed-forward neural network contains two linear layers W fc1 and W fc2 , and a nonlinear activation function, which enhances the nonlinear representation ability of the model. Multiple transformer modules are stacked together, allowing ViT to extract and integrate image features layer by layer, achieving excellent image understanding capabilities.

[0013] Definition 4: Multi-Head Self-Attention Mechanism. Multi-Head Self-Attention mechanism (Multi-Head Self-Attention) can capture features in different subspaces by performing multiple self-attention mechanisms (multiple heads). Each head of self-attention calculation is first input through three learnable parameter matrices W Q , WK and W V Get query matrix Q, key value matrix K and value matrix V respectively, and calculate self-attention: Multi-head self-attention is to splice the outputs of multiple self-attentions, and pass through a linear layer W o Get the final output. Multi-head self-attention mechanism can help the model focus on different parts of the input sequence at the same time, thereby improving the model's ability to capture complex distribution of data.

[0014] Definition 5: Orthogonal Gradient Descent. Orthogonal Gradient Descent is an optimization algorithm that aims to reduce interference between gradients by keeping the gradient direction of each update orthogonal to the gradient direction of the previous update. Through this method, Orthogonal Gradient Descent can more effectively explore the optimization space, improving the stability and convergence speed of the training process.

[0015] Definition 6: Nearest Class Mean Algorithm. Nearest Class Mean (NCM) is a classification algorithm that assigns test samples to the nearest class center by calculating the mean of feature vectors of each class (i.e. class center). Specifically, NCM first calculates the mean vector of each class in the training phase, and then compares the distance (usually using Euclidean distance) between the test sample and these mean vectors in the test phase.

[0016] Definition 7: Mahalanobis Distance. Mahalanobis distance is a method of measuring the distance between a point and a distribution. Unlike Euclidean distance, Mahalanobis distance takes into account the covariance structure of the data, so it can measure the correlation of data in different dimensions. It is particularly important in multivariate analysis because it can standardize the scales of different features, making the differences between features more comparable. This distance has wide applications in fields such as anomaly detection, clustering analysis and classification.

[0017] The technical scheme of the present application is an image classification method based on important parameter constraint continuous learning, which comprises:

[0018] Step 1: Preprocess the data image;

[0019] Obtain the real image dataset, and divide the dataset according to the number of tasks, each subset corresponds to a continuous learning task; wherein the image data in each subset has corresponding class label, the image is scaled and cropped to the same size, and the pixel value is normalized;

[0020] Step 2: Construct a continuous learner with a pre-trained model as a feature extraction network;

[0021] A pre-trained Vision Transformer model is taken as a backbone network to extract data features, the backbone network comprising 1 image embedding layer and 12 stacked transformer modules; the input of the feature extraction network is an image The output is a feature representation of the image Wherein D represents the feature dimension; suppose that the updated Θ, of the backbone network after learning the task t is Φ, the classifier is: Φ, the classifier is:

[0022] M represents the number of categories, and the goal of continuous learning is to learn a feature extraction network f(·, Θ t ) and a classifier h(·, Φ), which can predict the label of an unseen test sample from any task seen so far, wherein Φ = [Φ1, Φ2, … Φ t ];

[0023] Suppose that the pre-trained model parameter is W0, and for task t, the pre-trained model parameter is represented as:

[0024] W t =W0+ω1ΔW1+ω2ΔW2+…+ω t ΔW t

[0025] Wherein ΔW t =A t B t represents learning knowledge on each task with the LoRA module, ω = (ω1, ω2, … ω t ) is a set of learnable weight parameters;

[0026] First, the low-rank matrix A t in the LoRA module is QR decomposed, so that the weight update of the model on task t is represented in a low-rank space:

[0027] ΔW t =A t B t =Q t R t B t

[0028] Wherein Q t is an orthogonal matrix, R t is an upper triangular matrix, and each column vector of the LoRA module ΔW t is a linear combination of the column vectors of Q t , so that the ΔW torthogonal to each other, and only need to make the Q t orthogonal to each other, and thus establish the following orthogonal loss:

[0029]

[0030] wherein indicates the Q t concatenated by column; I indicates a unit matrix, and ||·||2is a two-norm operator; this constraint can make the each column vector has an inner product of 1 with itself and an inner product of 0 with other column vectors, thereby ensuring the orthogonality between the Q t of different tasks; combined with the cross-entropy loss used for classification, the loss function used for training the model is obtained:

[0031]

[0032] wherein indicates the cross-entropy loss, wherein indicates the data set of task t, y indicates the label of a sample, and λ indicates a weight;

[0033] Step 4: Establish an important parameter constraint mechanism;

[0034] Step 5: Fine-tune the parameters of task-adaptive prediction;

[0035] Step 6: Design a task identification inference strategy;

[0036] In the prediction phase, first infer the task to which the test sample belongs, and then use the fine-tuned classifier in step 5 to predict the label of the test sample;

[0037] Step 7: (1) Process the experimental data set according to step 1;

[0038] (2) Construct a continuous learner according to steps 2 and 3 and train the model on the training set of each task;

[0039] Step 8: (1) After training the model on each task, obtain the important model parameters for the current task according to step 4, and constrain them from changing in subsequent tasks;

[0040] (2) Fine-tune the linear classifier according to step 5 so that it can accurately classify the test samples of the current task and the previous tasks;

[0041] Step 9: (1) After completing the learning of all tasks, infer the task to which each test sample belongs according to step 6, and extract the features of the test sample using the model parameters specific to the task;

[0042] (2) Using the linear classifier fine-tuned in step 5 to classify the test samples.

[0043] Further, the specific method of step 4 is:

[0044] The orthogonal loss proposed in step 3 can constrain the LoRA module ΔW τ = A τ B τ to maintain orthogonality, τ = 1, 2 ， …, t.

[0045] First, define the importance of trainable parameters w t，ij for task t as follows:

[0046]

[0047] Calculate the smoothed importance score:

[0048]

[0049] where I(ω t，ij ) represents the parameter importance score based on sensitivity; is the smoothed importance score obtained by exponential moving average; is used to quantify the local uncertainty of I(ω t，ij ) and ; is the result of exponential moving average on U(w t，ij ); β1>0, β2<1 are two adjustable hyperparameters; for trainable parameters w t，ij , the importance is ultimately the product of and :

[0050]

[0051] Calculate the average importance of parameter matrix Wt:

[0052]

[0053] ω t，ij represents the element of the i-th row and j-th column of W t ; the higher S(W t ), the more important the parameter matrix is for task t; ω t represents the corresponding weight, in the process of continuous learning, the model parameters most important for the current task are selected to constrain them from changing in subsequent tasks, thereby alleviating the forgetting of the model.

[0054] Further, the specific method of step 5 is:

[0055] First, the prototype of the corresponding task class is obtained using the frozen feature extractor The data set representing class c under task t is sampled from the class prototype to obtain pseudo-features, and the classifier is fine-tuned using the pseudo-features, and the optimization objective is as follows:

[0056]

[0057] where f' represents the pseudo-feature, where f' represents the pseudo-feature,

[0058] Further, the specific method of step 6 is that after the pre-trained model is trained on task 1, it has good representation ability for all task samples. Therefore, first, the prototype of each class is extracted using the feature extractor f(·, Θ1) trained on task 1 For a test sample x, first, the feature of the test sample is extracted using f(·, Θ1), and the distance between the feature of the test sample and each class prototype is calculated by combining the nearest class mean algorithm, and the task corresponding to the class prototype with the smallest distance is selected as the task to which the test sample belongs:

[0059]

[0060] where The Mahalanobis distance is represented.

[0061] Innovations of the present application:

[0062] (1) The present application first discloses that even if continuous learning is completed using orthogonal LoRA fine-tuning combined with a pre-trained model, the loss-sensitive model parameters of the historical tasks will still change significantly on each task; The present application believes that the reason for this phenomenon is that in the extremely high-dimensional parameter space, there are too many potential orthogonal solutions, which ultimately leads to the model being able to only obtain a suboptimal solution on the current task.

[0063] (2) The present application proposes an important parameter constraint method based on loss sensitivity. The sensitivity of the model parameters to the loss is calculated as the importance of the parameter to the current task; by constraining the model parameters important to the current task from changing in subsequent tasks, the forgetting of the model's knowledge of the current task is effectively alleviated. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The figure is a framework diagram of a Vision Transformer model.

[0065] Figure 2 The figure is a framework diagram of the method of the present application.

[0066] Figure 3 A parameter fine-tuning process framework diagram for task-adaptive prediction. Implementation method

[0067] Step 1: Preprocessing the experimental data images

[0068] The present application selects four data sets for experimental verification.

[0069] The CIFAR100 data set contains 100 classes of real-world color images, each class containing 500 training samples and 100 test samples, and the image resolution size is 32*32. The CIFAR100 data set is divided into 10 tasks, each task containing 10 random class image samples, and the image classes of each task data set are mutually exclusive.

[0070] 5-datasets is constructed by 5 data sets, including SVHN, MNIST, CIFAR-10, Not-MNIST and Fashion-MNIST; each data set contains 10 classes of images, and the number of samples of each data set is different; wherein the image resolution size of CIFAR-10 and SVHN is 28*28, and the image resolution size of the remaining three data sets is 32*32; in the experiment, each data set is regarded as a task, and the difference between the tasks is large.

[0071] ImageNet-R selects 200 classes in the ImageNet data set, a total of 30000 color images, and the image resolution is 256*256. The ImageNet-R data set is divided into 10 tasks, each task containing 20 random class image samples.

[0072] DomainNet is a cross-domain data set, containing 345 classes and a total of 409832 images; since the images come from different domains, their original sizes are also different, in order to facilitate use, the images are cropped to a standard resolution of 224*224. This data set is considered more challenging because it has a large number of classes, and the number of samples in each class differs greatly. In the present application, DomainNet is divided into 5 tasks, each task containing 69 classes of images.

[0073] Step 2: Construct a continuous learner with a pre-trained model as a feature extraction network;

[0074] The pre-trained Vision Transformer model is used as a backbone network to extract data features, and the backbone network mainly includes one image embedding layer for dividing the input image into fixed-size image blocks (patches), and 12 stacked transformer modules for extracting image features. The output is a feature representation of the image where D represents the feature dimension. Let the updated parameters of the backbone network after learning task t be The parameters of the classifier are represented as: M represents the number of categories. The overall goal of continuous learning is to learn a feature extraction network f(·, Θ t ) and a classifier h(·, Φ) (here, the classifiers trained on each task are concatenated, i.e. Φ = [Φ1, Φ2, … Φ t ]) that can predict the label of an unseen test sample from any task seen so far.

[0075] Step 3: Constructing the orthogonal LoRA fine-tuning continuous learning paradigm

[0076] LoRA, as a fine-tuning paradigm for pre-trained models, decomposes the incremental parameters into the product of two low-rank matrices, and only updates these two low-rank matrices when learning downstream tasks, while keeping the pre-trained model parameters frozen. The present application improves this fine-tuning method, using a pre-trained ViT as the base model, and proposes a LoRA module combination method for continuous learning. For task t, the parameter matrix in any ViT model (such as W Q , W K , and W V in the transformer module) can be represented as W t :

[0077] W t = W0+ ω1ΔW1+ ω2ΔW2+ … + ω t ΔW t

[0078] where ΔW t = A t B t represents the LoRA module used to learn the knowledge on each task; ω = (ω1, ω2, … ω t ) is a set of learnable weight parameters; when learning task t, the pre-trained model parameters and the LoRA modules ΔW τ (τ = 1, 2, … t-1) trained on historical tasks are frozen, and only ΔW t corresponding to the current task and the weight coefficient ω = (ω1, ω2, … ω tThis weighted combination method allows the model to effectively incorporate knowledge from new tasks while preserving knowledge from historical tasks, thus enhancing the model's adaptability.

[0079] In theory, an efficient continuous learner can acquire different knowledge on different tasks and reduce the mutual interference between existing knowledge and newly learned knowledge. To achieve this goal, a method is designed to constrain the LoRA modules of different tasks to be orthogonal. The core idea is to constrain the LoRA modules learned in the current task to be orthogonal to the LoRA modules of the historical task pairs, so as not to change the loss of the model on the historical tasks.

[0080] The specific steps are as follows:

[0081] First, consider the low-rank matrix A in the LoRA module. t Therefore, by performing QR decomposition, the model's weight updates on task t can be represented in a low-rank space:

[0082] ΔW t =A t B t =Q t R t B t =Q t K t ,

[0083] Q t It is an orthogonal matrix, R t It is an upper triangular matrix. From the above formula, we know that ΔW t Each column vector can be obtained from Q. t The linear combination of column vectors is obtained; to make ΔW corresponding to different tasks t They are orthogonal to each other; we only need to make Q corresponding to different tasks... t Since they are mutually orthogonal, the following orthogonal loss was designed:

[0084]

[0085] in This indicates that Q represents different tasks. t Concatenate columns; I represents the identity matrix, ||·||2 is the L2 norm operator; this constraint allows... Each column vector has an inner product of itself of 1 and an inner product of all other column vectors of 0; thus ensuring Q-values ​​across different tasks. torthogonality between the gradients of the loss function of the previous tasks and the current task. This loss function makes the update direction of the model parameters orthogonal to the gradient descent direction of the loss function on the previous tasks when the model is trained on the current task, to some extent, reducing the influence of learning the loss of the current task on the historical tasks, and maintaining the performance of the model on the historical tasks. Combined with the cross-entropy loss for classification, the final loss function for training the model is obtained:

[0086]

[0087] wherein represents the cross-entropy loss, wherein represents the data set of task t.

[0088] Step 4: Designing an important parameter constraint mechanism

[0089] The orthogonal loss proposed in step 3 can constrain the LoRA modules ΔW τ = A τ B τ (τ = 1, 2, …, t) to be orthogonal; the principle of orthogonal gradient descent reveals that updating the parameters along the orthogonal direction of the gradient will not change the loss, so the training loss of the historical tasks may not change significantly. Since the incremental parameters learned on each task are assumed to be low-rank, i.e., the rank of ΔW τ is much smaller than the dimension of the incremental parameters under full fine-tuning, even if ΔW τ = A τ B τ (τ = 1, 2, …, t) are orthogonal to each other, the orthogonality cannot be strictly guaranteed, and thus the loss of the historical tasks cannot be guaranteed not to change significantly. In the present invention, it is found that the model parameters sensitive to the loss of the historical tasks still change significantly in each task, which is suspected to be because there are a large number of potential orthogonal solutions in the extremely high-dimensional parameter space, ultimately leading to the emergence of suboptimal solutions. Based on this finding, an important parameter constraint method is proposed, which constrains the model parameters important to the current task from changing in subsequent tasks, alleviating the forgetting of the knowledge of the current task by the model.

[0090] First, define the importance of a trainable parameter w t，ij for task t as follows:

[0091]

[0092] This formula essentially evaluates the importance of model parameters according to the change in loss when the parameters are zeroed. Since the estimate of the importance of model parameters is made on small batches of samples, the random sampling of small batches and the complex training dynamics can lead to high uncertainty in the evaluation results; this problem can be solved by sensitivity smoothing and uncertainty quantification, and the specific formula is as follows:

[0093]

[0094] where I(ω t,ij ) represents the parameter importance score based on sensitivity; is the smoothed importance score obtained by using exponential moving average; is used to quantify the local uncertainty of I(w t,ij ) and ; is the result of exponential moving average on U(w t,ij ). β1>0, β2<1 are two adjustable hyperparameters. For trainable parameters w t,ij , the importance can be finally defined as the product of and :

[0095]

[0096] Next, the average importance of the parameter matrix Wtis defined:

[0097]

[0098] Here, the parameter matrix Wtcan represent W Q , W K , W V and W O in the multi-head self-attention mechanism of the transformer module of ViT, and the two linear layers W fc1 and W fc2 in the feedforward neural network. w t，ij represents the element in the i-th row and j-th column of W t . The higher S(W t ) is, the more important the parameter matrix is for task t. In the process of continuous learning, for each task, the parameter matrices with importance in the top 5% or 10% are selected as the most important model parameters of the current task, and are constrained not to change in subsequent tasks, thereby alleviating the forgetting of the model. Specifically, for those important parameter matrices, no new LoRA module is added to update them in subsequent tasks, and the LoRA modules of historical tasks and the pre-training model parameters are frozen. For details, please refer to Figure 2 .

[0099] Step 5: Design a parameter fine-tuning method for task-adaptive prediction

[0100] When the model is trained on each task, the classifier h(·, Φ t ) is only trained on the training samples of this task, so it can only acquire the knowledge of the current task; after completing the entire continuous learning process, we hope that the classifier can predict the labels of the test samples on all tasks, so we concatenate the classifiers learned on all tasks Φ = [Φ1,..., Φ T ] and fine-tune them. The specific process is as follows: first, use the feature extractor f(x, Θ t ) learned on each task to obtain the prototype of the corresponding task and calculate the covariance according to the real data; next, Gaussian sampling is performed from the class prototype to obtain pseudo-features f', which are used to fine-tune the classifier, and the optimization objective is as follows:

[0101]

[0102] where f' represents the pseudo-features, and the sampled data set. The specific process can be seen in Figure 3 .

[0103] Step 6: Design a task identification inference strategy

[0104] In the prediction stage, a task identification inference strategy is designed, which first infers the task to which the test sample belongs, and then uses the classifier fine-tuned in step 5 to predict the label of the test sample. Existing work proposes that after the pre-trained model is trained on task 1, it has good representation ability for samples of all tasks. Therefore, first use the feature extractor f(·, Θ1) trained on task 1 to extract the prototype of each class For a test sample x, first extract its features f = f(x, Θ1) using f(·, Θ1), and combine the nearest class mean algorithm to first calculate the distance between the features of the test sample and each class prototype, and select the task corresponding to the class prototype with the smallest distance as the task to which the test sample belongs:

[0105]

[0106] where denotes the Mahalanobis distance.

[0107] Step 7: (1) Process the experimental dataset according to step 1; (2) Construct a continuous learner according to steps 2 and 3 and train the model on the training set of each task;

[0108] Step 8: (1) After training the model on each task, obtain the model parameters important for the current task according to step 4, and constrain them from changing in subsequent tasks; (2) fine-tune the linear classifier according to step 5 so that it can accurately classify test samples of the current task and previous tasks;

[0109] Step 9: (1) After completing the learning of all tasks, infer the task to which each test sample belongs according to step 6, and extract test sample features using task-specific model parameters; (2) classify the test sample using the linear classifier fine-tuned in step 5. The results on the classic continuous learning dataset are shown in Tables 1 and 2. The present application performs class-incremental continuous learning experiments without replay on four datasets; from the experimental results, the method proposed by the present application can significantly alleviate the problem of catastrophic forgetting and improve the accuracy of the model on each task.

[0110] Table 1 shows the experimental results of the method of the present application on CIFAR-100 and 5-datasets; the higher the accuracy, the lower the forgetting rate, representing better model performance.

[0111] Table 1

[0112]

[0113] Table 2 shows the experimental results of the method of the present application on ImageNet-R and DomainNet

[0114] Table 2

[0115]

[0116] As can be seen from the above two tables, the method proposed by the present application outperforms the current best continuous learning method on four datasets. On the CIFAR-100 dataset, the method proposed by the present application improves the accuracy by 6.35% and reduces the forgetting rate by 3.24% compared to the current best method; thus proving that the method proposed by the present application, by constraining the model parameters important for the current task from changing in subsequent tasks, in combination with the orthogonal LoRA fine-tuning method, effectively alleviates the forgetting of historical task knowledge by the model, improves the stability and robustness of the model, and thus makes the model achieve relatively ideal results on each dataset.

Claims

1. An image classification method based on continuous learning with important parameter constraints, the method comprising: Step 1: Preprocess the data image; Obtain a real image dataset and divide the dataset according to the number of tasks, with each subset corresponding to a continuous learning task; the image data in each subset will have corresponding category labels, the images will be scaled and cropped to the same size, and the pixel values ​​will be normalized. Step 2: Construct a continuous learner using a pre-trained model as the feature extraction network; A pre-trained Vision Transformer model is used as the backbone network to extract data features. This backbone network consists of one image embedding layer and 12 stacked transformer modules; the input to the feature extraction network is the image. The output is a feature representation of the image. Where D represents the feature dimension; assuming the backbone network updates the feature dimension after completing task t. Θ t The parameters of the feature extraction network are: Φ t Here are the classifier parameters; M represents the number of categories. The goal of continuous learning is to learn a feature extraction network f(·,Θ). t The classifier h(·,Φ) is capable of predicting the label of an unseen test sample from any task seen so far, where Φ=[Φ1,Φ2,…Φ]. t ]; Step 3: Construct a continuous learning paradigm for orthogonal LoRA fine-tuning; Let the pre-trained model parameters be W0. For task t, the pre-trained model parameters are expressed as: W t =W0+ω1ΔW1+ω2ΔW2+…+ω t ΔW t Where ΔW t =A t B t This indicates that the LoRA module is used to learn knowledge for each task, ω = (ω1, ω2, ... ω t ( ) is a set of learnable weight parameters; First, consider the low-rank matrix A in the LoRA module. t QR decomposition is performed, so the model's weight updates on task t are represented in a low-rank space: △W t =A t B t =Q t R t B t Q t It is an orthogonal matrix, R t It is an upper triangular matrix, LoRA module ΔW t Each column vector is Q t A linear combination of column vectors, such that ΔW corresponds to different tasks. t They are orthogonal to each other; we only need to make Q corresponding to different tasks... t Since they are mutually orthogonal, the following orthogonal loss is established: in This indicates that Q represents different tasks. t Concatenate columns; I represents the identity matrix, ||·||2 is the L2 norm operator; this constraint allows... Each column vector has an inner product of itself of 1 and an inner product of all other column vectors of 0; thus ensuring Q-values ​​across different tasks. t The orthogonality between them; combined with the cross-entropy loss used for classification, the loss function used to train the model is obtained: in Represents the cross-entropy loss, where Let represent the dataset for task t, y represent the labels of the samples, and λ represent the weights; Step 4: Establish a constraint mechanism for important parameters; Step 5: Fine-tune the parameters of the task adaptation prediction; Step 6: Design a task identifier inference strategy; In the prediction phase, the task to which the test sample belongs is first inferred, and then the classifier fine-tuned in step 5 is used to predict the label of the test sample. Step 7: (1) Process the experimental dataset according to Step 1; (2) Construct a continuous learner according to steps 2 and 3 and train the model on the training set for each task; Step 8: (1) After training the model on each task, obtain the model parameters that are important for the current task according to step 4, and constrain them not to change in subsequent tasks; (2) Fine-tune the linear classifier according to step 5 so that it can accurately classify the test samples of the current task and previous tasks. Step 9: (1) After completing the learning of all tasks, infer the task to which each test sample belongs according to step 6, and extract the features of the test sample using task-specific model parameters; (2) Use the linear classifier fine-tuned in step 5 to classify the test samples.

2. The image classification method based on continuous learning with important parameter constraints as described in claim 1, characterized in that, The specific method for step 4 is as follows: The orthogonal loss proposed in step 3 can constrain the LoRA module ΔW corresponding to different tasks. τ =A τ B τ Maintain orthogonality, τ = 1, 2, ..., t; First, define the trainable parameters ω for task t. t，ij Its importance is as follows: Calculate the smoothing importance score: Where I(ω) t，ij ) represents the sensitivity-based parameter importance score; It is a smoothed importance score obtained by using an exponential moving average; Used to quantize I(ω) t，ij )and Local uncertainties; It is for U(ω) t，ij The result after performing an exponential moving average; β1 > 0, β2 < 1 are two adjustable hyperparameters; for the trainable parameter ω t，ij Its importance is ultimately and The product of: Calculate the parameter matrix W t Average importance: ω t，ij W t The element in the i-th row and j-th column; S(W t The higher the value of ω, the more important this parameter matrix is ​​for task t; t This represents the corresponding weight. During continuous learning, the most important model parameters for the current task are selected and constrained so that they do not change in subsequent tasks, thereby mitigating model forgetting.

3. The image classification method based on continuous learning with important parameter constraints as described in claim 1, characterized in that, The specific method for step 5 is as follows: First, the frozen feature extractor is used to obtain the prototype of the corresponding task category. Let represent the dataset for class c under task t. Next, Gaussian sampling is performed from the class prototypes to obtain pseudo-features. The classifier is then fine-tuned using these pseudo-features, with the following optimization objective: Where f′ represents a pseudo-feature. Let T represent the sampled dataset, and T represent the total number of tasks.

4. The image classification method based on continuous learning with important parameter constraints as described in claim 1, characterized in that, The specific method of step 6 is as follows: After the pre-trained model is trained on Task 1, it can have good representation ability for samples of all tasks; therefore, the feature extractor f(·, Θ1) trained on Task 1 is first used to extract the prototype of each category. For a test sample x, its features are first extracted using f(·, Θ1), and the nearest class mean algorithm is used to calculate the distance from the features of the test sample to each class prototype. The task corresponding to the class prototype with the smallest distance is selected as the task to which the test sample belongs. in This indicates the distance to Maharanobis.

Images