A kind of cross-task information exchange incremental method

Abstract

The application relates to an incremental method for cross-task information exchange, relates to the field of artificial intelligence incremental learning, and discloses an application of a deep network model to a large number of tasks without huge calculation and space costs. The method uses a two-section incremental learning network to adjust the residual cluster space between a target model and an original model by using a cross-task clustering expansion method, removes redundant parameters and feature dimensions of the network model by using a distillation strategy, and keeps the single backbone. Finally, a dynamic alignment weight scheme for solving high weight offset is used to solve the weight offset problem of the last full connection layer. Experiments prove that the method can effectively reduce the catastrophic forgetting.

Description

Technical Field

[0001] This invention relates to the field of incremental learning in artificial intelligence, and more specifically to a deep learning-based incremental learning method that combines knowledge distillation and replay strategies. Background Technology

[0002] Incremental learning is a model that continuously learns and accumulates knowledge and skills over time, receiving training data strictly in chronological order. The model typically transitions from a well-understood data distribution to a completely unfamiliar one. This distribution change could be a shift in the input domain or the learning of new categories of knowledge. The main challenge of incremental learning models is overcoming the problem of forgetting previously learned knowledge after learning new knowledge—a problem known as catastrophic forgetting.

[0003] In recent years, various methods have been proposed from different perspectives to address catastrophic forgetting. However, the most widely used incremental learning strategies to date are knowledge distillation and replay. Icarl et al.'s method adds a knowledge distillation constraint model to replay, using the output of the new model to fit the old model to prevent forgetting old knowledge categories. However, these methods only have a single network backbone and lack sufficient plasticity to cope with the addition of new knowledge categories. Even with the knowledge distillation constraint, the single network still suffers from feature degradation of old knowledge due to the limited access to old data. Recently, methods based on dynamic architectures in incremental learning have achieved superior results.

[0004] Existing research has frozen parameters by retaining some modules, aiming to preserve important parts of the knowledge from older classes and then expand new trainable modules to improve performance in learning new classes. However, these studies have two problems. First, expanding to new modules leads to a sharp increase in the number of parameters and increases storage and computation costs, making these methods unsuitable for long-term incremental learning. Second, because information sharing between old and new modules is blocked, retaining only old knowledge in older modules may affect the performance of new classes. Therefore, research on incremental learning models that facilitate communication between old and new tasks while maintaining a lightweight network is particularly important.

[0005] In incremental learning, the continuous emergence of new categories can lead to distribution drift and discrepancies between the target label and the existing model output. To address this issue, this method employs a two-stage incremental learning paradigm and cross-task feature clustering to achieve the goals of incremental learning. While cross-task information sharing methods use progressive convergence to the true target to fit the difference between the model and the actual model, the two-stage incremental learning paradigm offers a more general and dynamic structural framework. This framework offers two advantages: firstly, it helps highly malleable new models learn new class knowledge; secondly, it maintains the model's lightweight nature and stability by utilizing compression and other methods while preserving previous feature parameters. Summary of the Invention

[0006] The purpose of this invention is to address the problem of incremental learning in class classification scenarios and reduce the catastrophic forgetting effect caused by incremental learning. Considering that knowledge distillation is a method for transferring knowledge and can effectively reduce the number of network parameters, this invention proposes an effective cross-task communication method for incremental learning in class classification. One of the main technical highlights is cross-task clustering information exchange and balanced decoupling distillation. This method is highly effective in classification scenarios in incremental learning of class classification in dynamic network architectures.

[0007] The technical solution of this invention is:

[0008] An incremental method for cross-task information exchange, comprising the following modules:

[0009] Step 1: Dataset partitioning;

[0010] The pre-divided dataset is input into the classification model in chronological order. Each category is encoded using a random seed, and N already encoded categories are input each time.

[0011] Step 1.1: Dataset partitioning and task queueing. This involves encoding all classes in the dataset using a random seed, then dividing the dataset into N sub-tasks according to the incremental requirements, and learning the task as each increment arrives.

[0012] Step 2: Cross-task information exchange;

[0013] Based on the encoded category data from step 1, the network model is expanded to learn new category knowledge, which is then gradually input into the classification model according to the time series. The clustering module is used to facilitate cross-task communication between the old and new category models, and to aggregate each category instance into its respective network space.

[0014] Step 2.1: Learn new category knowledge using augmentation learning in a two-stage learning paradigm. This involves expanding the network with the same architecture as the old network while freezing the old network during the period when a new task arrives. The incremental learning process is as follows:

[0015] 1) Use the RestNet residual network to extract features from the image.

[0016] 2) When the number of tasks T = 1, the network uses only a single network and adopts the simplest cross-entropy loss function to learn all categories in the current task.

[0017] 3) When the number of tasks T>1, the network adopts an expansion strategy, and the network uses a clustering module to enhance the information exchange between the old and new classes.

[0018] 4) When the number of tasks T>1, combine the category of the current task and the category in memory to form a new training set and train the new class network.

[0019] Step 2.2: Store the samples in memory. Use random sampling to store the current new class instances in memory according to the memory size.

[0020] Step 3, Model Compression:

[0021] Based on the dual-network architecture obtained in step 2, the dual-network architecture is compressed using a balanced decoupling distillation module. This dual-network architecture consists of the concatenation of the new class network and the old class network from step 2, and the new and old class networks have the same structure.

[0022] Step 3.1: Use the category balancing algorithm to calculate the weight factor for each category.

[0023] Step 3.2, Balanced Decoupling Distillation: Create a network identical to the dual network architecture as the student network. Multiply the class weight factor calculated in Step 3.1 with the logit value of the softmax layer in the student network. Use the dual network architecture obtained in Step 2 as the teacher network to decouple and distill the student network.

[0024] Step 4: Resolve weight offset:

[0025] Based on the compressed model obtained in step 3, a dynamic alignment weight scheme is used to solve the knowledge offset problem.

[0026] The advantages and positive effects of this invention are:

[0027] This invention proposes a class-incremental method for cross-task information exchange. By using a cross-task clustering module, each task network stores both new and old class knowledge, reducing knowledge category distribution drift and thus mitigating catastrophic forgetting. Building upon the advantages and disadvantages of existing dynamic network structures, this method further reduces computational cost and accuracy loss during compression by incorporating Balanced Decoupling Knowledge Distillation. Inspired by weight alignment, this method proposes DynamicsWeight Alignment, which addresses the weight shift problem in the final fully connected layers of the model to some extent. Comparative experiments on multiple datasets demonstrate that this invention performs well in class-incremental learning. Furthermore, this method achieves excellent performance in scenarios using currently popular class-incremental learning methods. Currently, most solutions to AI tasks rely on closed-dataset training methods, which struggle to cope with the ever-increasing data processing demands. This method provides significant assistance for further research into more intelligent AI paradigms. Attached Figure Description

[0028] Figure 1 This is a structural diagram of the overall network of the present invention;

[0029] Figure 2 This is a diagram illustrating the cross-task clustering process during the expansion phase of this invention. Figure 2 a shows the clustering distribution that the old-class network has already learned. Figure 2 b shows the clustering distribution of the new network after 80 iterations. Figure 2 c shows the clustering distribution of the new network after 170 iterations of learning;

[0030] Figure 3 These are the incremental learning results of various methods in different environments on the CIFAR-100 dataset;

[0031] Figure 4 These are the incremental learning results of various methods on different environments in the S-Tiny-ImageNet dataset;

[0032] Figure 5 It represents the incremental learning results of various methods in different environments on the ImageNet-100 / 1000 dataset; Detailed Implementation

[0033] Example:

[0034] This example strictly adheres to the incremental learning framework and considers a series of tasks. in The task representing time i, and carrying a sample set D specific to that time task. i To maintain a unequal distribution of categories in each task, Categories and The categories in the data are strictly disjoint, that is... Each dataset at a specific time consists of input sample X i and its one-hot encoding Y i Composition (X) i ,Y i At any time t, a new sample set D is obtained. t and a fixed-size sample set M t , Due to memory space limitations, typically |M t |<<|D1∪···∪D t-1 |. When t>1, D t =D t ∪M t Assume F t-1 Indicates training from T1 to T t-1 The neural network, for each new task T t The arrival of D t Retraining F with the help of t-1 The model obtained in this way is model F t For (T1∪····∪T) t All classes in the class can achieve good classification results.

[0035] This example demonstrates an incremental approach based on knowledge distillation, dynamic structure, and replay strategies. Built using Python and the PyTorch framework, it consists of two parts, with the overall structure shown in the diagram below. Figure 1 As shown, Part 1 below represents the expansion stage in the structure diagram, and Part 2 represents the compression stage. The "Step 1" and "Step 2" below describe in detail the operation steps performed on the image:

[0036] Before starting "Step 1" and "Step 2", this method needs to define each task sequence. Each task contains different categories and is fed into the model for training according to the chronological order.

[0037] Step 1, Expansion Phase:

[0038] 1.1 Use the ResNet residual module to extract features from the image. When the number of tasks T=1, the network only uses a single network and adopts the simplest cross-entropy loss function to learn all categories in the current task.

[0039] Given cross-entropy loss function for:

[0040]

[0041] in This represents the images contained in the current task, where Composed of images arriving in the current time series and images in memory, D t =D t ∪M t x i Represents the image for the current task, y i Represents x i The corresponding image labels, among which Represents x i In the current model, y is generated. i The predicted probability.

[0042] Step 1.2: Freeze the old class model and create a new network with the same architecture to learn new class knowledge. Assume that in stage t, model F... t It can be further decomposed into feature embedding f t and linear classifier f t By constructing and feature extractors Same feature extractor This is achieved through [the means]. For example... Figure 1 As shown, given an image x belonging to D t Feature f t The following is an expression:

[0043]

[0044] Then the predicted value pass The conclusion is that in step t, the classifier is designed by matching the new input and output dimensions (i.e., synchronizing the number of classes added each time to the dimensions of the final classifier). The classifier parameters for old features inherit from the classifier of the previous stage to retain old knowledge, and its newly added parameters are randomly initialized.

[0045] To reduce catastrophic forgetting, step t is frozen. The parameters are set to ensure that the feature extractor is not updated in the last step. The parameters are then processed in batches to obtain normalized statistical information. In addition, this invention will also... use The parameters are used as initialization values, reusing previous knowledge to quickly adapt to the new network and move forward to the new network.

[0046] To further enable networks to learn diverse and discriminative features of new knowledge, DER proposes a new feature-based approach. The auxiliary loss is used (see: S. Yan, J. Xie, and X. He, “Der: Dynamically expandable representation for class incremental learning,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3014–3023.). Specifically, an auxiliary classifier is utilized in the newly developed network. The predicted probability is Therefore, the loss can be represented in an extensible way. The formula is as follows:

[0047]

[0048] Where λ t λ is a hyperparameter that controls the performance of the auxiliary classifier. Note that when t = 1 in the first step, λ... t =0.

[0049] Section 1.3, Cross-task information exchange: By extracting the clustering spatial location of each task, the feature set is divided into multiple clusters based on the correlation between features and feature groups between categories. According to the class incremental learning setting, cross-task feature clustering utilizes the old model feature extractor. The feature space points are used to determine the region where the optimized category is located, preventing these categories from drifting to useless regions. Figure 2 The text explains the cross-task feature clustering objective and shows T in the task flow. t-1 To T t The three old classes in and The instance location. Each old class in It can aggregate representations well, but in the early stages of incremental learning, the new class will exhibit a scattered state, such as... Figure 2 b is shown in the upper right corner. This invention adds constraints on instances (as shown by the dashed lines). Model F is defined. t-1 In D t-1 The training is performed on the cross-task spatial clustering model, and the loss function is as follows:

[0050]

[0051]

[0052] Where q(x) is the softening probability of the distillation function with respect to the temperature variable t. and x represents i From F t-1 The soft labels are obtained from the reference model, and α is the hyperparameter balancing factor of the loss.

[0053] Step 2, Compression Stage

[0054] Section 2.1 addresses the long-tail effect and incremental learning. Over time, a persistent long-tail effect often emerges, where a minority class constitutes the majority of the data, while the majority class is insufficiently representative. During distillation, the long-tail effect significantly impacts knowledge transfer within the teacher network. To mitigate this effect, this example calculates the influence weight E for each category. n The specific formula is as follows:

[0055]

[0056] Where n equals the number of instances of the current class, and β is an adjustable hyperparameter. Accordingly, the weight factors for each class are obtained again through normalization as follows:

[0057]

[0058] 2.2 Balanced Decoupled Distillation (BDKD): BDKD decouples the classic knowledge distillation loss into two parts: target-class knowledge distillation (TCKD) and non-target-class knowledge distillation (NCKD), such as... Figure 1 As shown, this decoupling method effectively improves the effectiveness of knowledge transfer. To alleviate classification bias caused by imbalanced training data, a balanced decoupling distillation is performed using pre-calculated class weights, as shown in the following formula:

[0059]

[0060] Where b T Represents the real target logit. This represents the logit after masking the true target category. This simple distillation method has proven to achieve excellent performance in ablation experiments.

[0061] 2.3. Resolving Weight Shift: When training on a new task, due to a single imbalanced training set D... t =D t ∪M tClass imbalance can lead to a strong classification bias in the model. Furthermore, incrementally learned models are more prone to ignoring a small number of class residuals. To mitigate classification bias and encourage the model to learn new and old classes in a balanced manner, Dynamic Weight Alignment (DWA) is used. Assuming the classifier in stage t uses W... t W indicates t It can be further broken down into in

[0062] The final logit is as follows:

[0063]

[0064] Where γ is the scaling factor, composed of γ1 and γ2. Since the new class weight vector has a larger norm in incremental learning, the logarithm of the new class output is usually also larger, and the trained model output may be more biased towards the new class. Therefore, this invention uses dynamically set parameters. Where C represents the number of categories, this scaling strategy reduces the number of new categories while increasing the number of old categories.

[0065] The above is an operational example of an incremental method for cross-task information exchange. Ultimately, this results in a network that can achieve good classification results for categories across all task sequences.

[0066] The present invention has undergone corresponding experiments to verify its actual effects. The specific contents of the experiments are as follows.

[0067] CIFAR-100: CIFAR-100 contains 50,000 training images (500 images per class) and 10,000 test images (100 images per class). S-Tiny-ImageNet is a medium-sized dataset consisting of nearly 20,000 images across 200 classes. The training and test sets each contain approximately 10,000 images, with about 50 images per class. ImageNet-100: ImageNet-1000 is a large dataset consisting of approximately 1.28 million images for training and 50,000 images for validation, with 500 images per class. ImageNet-100 consists of 100 classes randomly selected from the original ImageNet-1000.

[0068] On the ImageNet dataset, the standard ResNet-18 was used as the feature extractor, with a batch size of 256. For CIFAR-100, a modified ResNet32 was used as the feature extractor, with a batch size of 128. For all ImageNet and CIFAR-100 datasets, a cosine annealing scheduler was used to gradually decay the learning rate from 0.1 to zero (for a total of 170 epochs). During the scalable representation learning phase, SGD with a momentum of 0.9 and weight decay of 5e-4 were employed to preserve the optimal results. For data augmentation settings, strategies such as AutoAugment, random cropping, horizontal flipping, and normalization were used to augment the training images.

[0069] This invention is compared with various methods on the CIFA-100, imagnet-100 / 1000, and S-Tiny-ImageNet datasets, and the results are as follows: Figure 3 As can be seen from Figures 4 and 5, this invention achieves good results in most scenarios. In classification tasks involving incremental learning, this invention effectively reduces catastrophic forgetting and improves overall accuracy.

Claims

1. An incremental method for cross-task information exchange, comprising the following steps: Step 1: Dataset partitioning; The pre-divided dataset is input into the classification model in a time sequence, and each category is encoded using a random seed. Each input consists of N already encoded categories. Step 2: Cross-task clustering and communication; Based on the encoded category data from step 1, the network model is expanded to learn new category knowledge, which is then gradually input into the classification model according to the time series. The clustering module is used to facilitate cross-task communication between the old and new category models, and to aggregate each category instance into its respective network space. Step 3, Model Compression: Based on the overlapping dual-network architecture obtained in step 2, the dual-network architecture is compressed using balanced decoupling distillation; Step 3.1: Use the category balancing algorithm to calculate the weight factor for each category; Step 3.2, Balanced Decoupling Distillation: Create a network identical to the dual network architecture as the student network. Multiply the class weight factor calculated in Step 3.1 with the logit value of the softmax layer in the student network, and use the dual network architecture obtained in Step 2 as the teacher network to decouple and distill the student network. Step 4: Resolve weight offset: Based on the compressed model obtained in step 3, a dynamic alignment weight scheme is used to solve the knowledge offset problem; The first step, dataset partitioning, involves encoding each category in the given dataset using a random number seed, then shuffling the category codes and feeding them into the training network. When the number of tasks T=1, the network uses only a single network and employs the simplest cross-entropy loss function to learn all categories in the current task. Cross-entropy loss function for: in This represents the images included in the current task. It consists of images arriving in the current time series and images in memory. , Image representing the current task. represent The corresponding image label, represent In the current model generation The predicted probability.

2. The incremental method for cross-task information exchange according to claim 1, characterized in that, The second step, cross-task clustering communication, uses ResNet as the network infrastructure. Under each time series branch T, the model creates a new ResNet network to learn new category knowledge. When T=1, the model has only one ResNet network and only learns the knowledge under the current time series. When T>1, the model has two ResNet networks. During learning, the parameters in one network are frozen to save the old category knowledge, while the other is not frozen to learn new category knowledge. While learning new category knowledge, the clustering module and replay strategy are used to reduce data distribution drift between tasks and enhance information exchange.

3. The incremental method for cross-task information exchange according to claim 1, characterized in that, The student network used in decoupled distillation is a ResNet architecture.

4. The incremental method for cross-task information exchange according to claim 1, characterized in that, The fourth step addresses weight offset by calculating the last fully connected layer in the compressed network from the third step, using the number of class instances arriving in the time series and the number of class instances in memory.

5. The incremental method for cross-task information exchange according to claim 1, characterized in that, The final logit value obtained from the dual-network architecture is obtained by multiplying the class weight factor by the logit value of the student network after passing through the Softmax layer. The dual-network architecture is used as the teacher network to perform balanced decoupling distillation on the student network. The category weighting factor is calculated as follows: Calculate the influence weight for each category The specific formula is as follows: in, Equal to the number of instances of the current category, These are adjustable hyperparameters. Accordingly, the weight factors for each category are obtained again through normalization as follows: 。 6. The incremental method for cross-task information exchange according to claim 1 or 5, characterized in that, The specific formula for equilibrium decoupled distillation is as follows: in Represents the real target logit. This represents the logit after masking the actual target category.

7. The incremental method for cross-task information exchange according to claim 1, characterized in that, The dynamic alignment weight scheme is as follows: assuming the classifier for stage t uses... express, It can be further broken down into ,in , The final as follows: in It is the scaling factor, determined by... and Because the new class weight vector has a larger norm in incremental learning, the logarithm of the new class output is usually also larger. Therefore, the trained model output may be more biased towards the new class. Thus, dynamically setting the parameters is used. ,in This represents the number of categories.

Images