An image classification method based on incremental learning of a multi-branch network architecture

Abstract

A kind of incremental learning method based on multi-branch network architecture, comprising the following steps;Step 1: for each new task T i , the similarity of task and the old task learned is calculated, and is sorted;Step 2: for the highest similarity value after sorting and the set similarity threshold value are compared, if less than the threshold value, then expand new network model to learn, otherwise use the most similar network model to learn and update;Step 3: for each network model, an auxiliary network is introduced inside, the auxiliary network is used to learn the task that is not similar in network model, network model learns similar task using the way of knowledge distillation, after learning, auxiliary network is integrated into network model, and the final model after expansion is obtained;Step 4: using the final model after expansion, test sample is predicted, and the final classification accuracy is calculated.The present application not only reduces the forgetting of old class due to learning new class, but also controls the size of parameter quantity.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image classification, and particularly relates to an image classification method based on incremental learning of a multi-branch network architecture. BACKGROUND

[0002] The current mainstream machine learning method is generally learning and reasoning. Learning refers to learning a network model for a specified data set and then using the network model for reasoning. When a new task is encountered, the model generally starts training from scratch, which results in low efficiency and time consumption of repeated training of the model. Humans have strong memory and continuous learning capabilities. Humans can gradually expand from simple task learning to complex task learning. Many scholars also want to enable deep neural networks to imitate this process. Deep neural networks gradually expand from single-task learning to multi-task learning without training from scratch each time, which saves a large amount of resource and time cost. However, when the neural network expands from single-task learning to multi-task learning, catastrophic forgetting occurs, that is, the network model performs well on new classes but performs poorly on old classes. This is because the network model parameters change when the deep neural network learns a new task, causing the model to forget the previous task, which limits the continuous learning ability of the deep neural network.

[0003] Incremental learning methods are mainly divided into three categories: replay-based methods, regularization-based methods, and architecture-based methods.

[0004] The replay-based method mainly simulates the human memory mechanism, stores the learned knowledge, and learns the stored knowledge and new tasks together when learning new tasks, so as to slow down the catastrophic forgetting. However, storing original data will have memory limitations and data privacy problems.

[0005] The regularization-based method mainly limits the update amplitude of part of the weights when updating the weights of the neural network through a certain strategy, and usually measures the influence of the sample on the model.

[0006] However, in the context of a large number of tasks, the class shift is more serious. Because the weights important to the old task change little, the model's ability to learn new tasks decreases as the number of tasks increases.

[0007] The architecture-based method refers to adding a new module (i.e., network structure) corresponding to different tasks in the neural network, so the module of the new task and the module of the old task are independent of each other and do not interfere with the previously learned knowledge. Although the architecture-based method largely solves the problem of catastrophic forgetting, it produces a huge amount of parameters and requires a high memory requirement.

[0008] The invention discloses an image classification method and system based on CLIP category incremental learning. First, according to the similarity of the text features of the newly input training images and the old category text features, the new and old category pairs of adjacent categories are screened, and for the old category text features in each pair of adjacent categories, normal distribution is used for sampling to construct a hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task. The trained CLIP model is used to obtain the classification result of the image to be processed, reduce the forgetting of old categories due to learning new categories, and improve the image classification accuracy.

[0009] This method belongs to the regularization algorithm, which mainly focuses on reducing the forgetting of old categories, but when dealing with new categories, the learning ability of new categories may be limited, especially when the difference between new categories and old categories is large, the classification performance may decrease. SUMMARY

[0010] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide an image classification method based on incremental learning of multi-branch network architecture, which quantifies the similarity between new and old image classification tasks by calculating the Wasserstein distance between them, and decides whether to expand the new network model according to the similarity. At the same time, an auxiliary network is also introduced into the main network to learn dissimilar tasks, and the learned parameters are integrated into the main network to reduce the size of the network parameter scale. This method not only reduces the forgetting of old categories due to learning new categories, but also controls the size of the parameter scale.

[0011] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0012] An image classification method based on incremental learning of multi-branch network architecture, comprising the following steps:

[0013] Step 1: For each new image classification task T i (For example, classify the data set containing cats and dogs), calculate the similarity of the task with the old image classification tasks that have been learned, and sort them;

[0014] Step 2: Compare the highest similarity value with the set similarity threshold value, if it is less than the threshold value, expand the new network model for learning, otherwise use the most similar network model for learning and updating;

[0015] Step 3: Introducing an auxiliary network inside each network model in step 2, which is used to learn dissimilar tasks inside the network model, and the backbone network of the network model learns similar tasks using knowledge distillation, after learning, the auxiliary network is integrated into the backbone network, and the expanded final model is obtained, which controls the size of the parameters and prevents catastrophic forgetting;

[0016] Step 4: Using the expanded final model to predict the test sample and calculate the final classification accuracy.

[0017] The specific process of step 1 task similarity calculation is as follows:

[0018] Step 1.1: For T class incremental image classification tasks is the t-th class incremental image classification task, including N t sample images, is the i-th sample of the t-th class incremental image classification task, is the label value of the corresponding i-th sample of the t-th class incremental image classification task, and the class set of the image classification task T i is C i , T represents the number of tasks, Ti represents the i-th task, for each class c∈C i , a prototype feature is maintained, which is defined as the mean of the image sample features belonging to the class c:

[0019]

[0020] where w i represents the weight of the sample X i , which is used to define according to the class balance or sample confidence;

[0021] Step 1.2: Calculate the nonlinear distance between the sample image and the prototype feature based on the kernel function method:

[0022]

[0023] where k(·,·) is the kernel function, for example, Gaussian kernel, defined as:

[0024]

[0025] In addition, the minimum distance of the new sample image X l to all class prototypes of previous image classification tasks is defined as:

[0026]

[0027] where C j is the task T ja set of categories, is a task T j prototype feature of category c in task T

[0028] Step 1.3: Calculate the Wasserstein distance, the dissimilarity between the distributions of image classification tasks T i and T j is measured by the Wasserstein distance, defined as:

[0029]

[0030] where Π(q i , q j ) represents the joint distribution between the task distributions;

[0031] To simplify the calculation, Sinkhorn approximation is used to estimate the Wasserstein distance:

[0032]

[0033] Step 1.4: Design a dynamic mapping function to map the Wasserstein distance to the interval [0, 1], as follows:

[0034] s i,g = 1-exp(-α·W ε (q i , q j ))

[0035] where s i is the similarity measure between the new task T t and the old image classification task group g, with a value range of [0, 1], and α is a scaling factor adjusted through experiments.

[0036] The specific process of model expansion in step 2 is as follows:

[0037] After step 1 similarity calculation, N t similarity values are obtained, and the similarity values are mapped to the range of [0, 1]. Sort the N t highest value T max and the threshold δ, δ is in [0-1], if the highest similarity value is greater than the threshold, it means that the current task is very similar to the old task, at this time the task is assigned to the network N old corresponding to the highest similarity task, and the network is learned; this approach can maximize the reduction of training cost and alleviate catastrophic forgetting.

[0038] If the highest similarity value is less than the threshold value, it indicates that the current image classification task is not similar to all learned image classification tasks, at which time the model needs to be expanded, and the mathematical formula is as follows:

[0039]

[0040] wherein G(T cur ) represents the network to which the current image classification task T cur is assigned, the expansion strategy is to copy the network with the highest similarity to the learned task, and the copied network N new is used to learn the new image classification task. This way can reduce the cost of starting from zero and shorten the training time.

[0041] When step 2 decides to expand a new network model, the new network will become part of the multi-branch network architecture. Specifically: the new network is copied based on the network with the highest similarity to the learned task to reduce the cost of starting from zero. After learning the new task, the new network will be included in the model library of the framework for subsequent task reuse or expansion.

[0042] The step 3 is specifically:

[0043] Step 3.1: First stage (initial training stage)

[0044] Use complete supervised data (training data X1 and label Y1) for standard classification training; train a standard classification model A standard classification model Contains the following components:

[0045] (1) Feature extractor Responsible for extracting high-dimensional feature representation of samples;

[0046] (2) Classifier g c1 : Map features to classification labels; use the feature extractor to extract features for the task, and then use the classifier to map these extracted features to specific labels such as cats, dogs, etc. The feature extractor and the classifier are the necessary components of a classification model and are in a sequential relationship, first feature extraction and then classification using the classifier.

[0047] Use cross-entropy loss L ce to train the model and optimize the target; improve classification performance.

[0048] Incremental stage (nth stage)

[0049] The input data of the current image classification task T n only contains the images of the new task Does not contain old image classification task samples, and a new feature extractor is used Extracting feature representation for new task:

[0050]

[0051] To learn discriminative features for new image classification task, supervised optimization is performed using new task labels (from Y n ) using classifier mapping features to label space:

[0052]

[0053] Computing cross-entropy loss L ce :

[0054]

[0055] Meanwhile, to preserve knowledge of old tasks, knowledge distillation is used to preserve feature representation of old classes, extracting feature representation of previous stage n-1:

[0056]

[0057] Computing distillation loss L kd :

[0058]

[0059] where F kd denotes Euclidean distance;

[0060] Step 3.2: Since the classification performance of new task and old tasks is not considered in step 3.1, step 3.2 further optimizes the feature representation by using prototype vectors and balance loss to balance the classification performance of new task and old task. In the NECIL scenario, a prototype is stored in the feature space for each class: the classification performance of new image classification task and old image classification task is balanced by these prototypes; prototypes are oversampled to match batch size B, so as to calibrate the classifier:

[0061] p B = U p B(Prototype)

[0062] where U p B is an oversampling function;

[0063] Computing prototype balance loss L proto :

[0064] L proto = F ce (p B ,yB )

[0065] where y B is the over-sampled label set;

[0066] Step 3.3: To ensure that the network in steps 3.1 and 3.2 does not destroy the feature representation of the old image classification task while learning the new image classification task, while keeping the model lightweight, to learn new classes without bias while maintaining the old class feature representation, a dynamic structure reorganization strategy is adopted:

[0067] Structure expansion: Insert a residual adapter in each convolutional block of the fixed feature extractor in the last stage. The residual adapter only updates the most discriminative part while preserving the old features:

[0068]

[0069] Structure reparameterization: After training is complete, integrate the side branch information into the main branch without loss, reducing the parameter size:

[0070]

[0071] Delete the adapter to keep the network structure unchanged for the next stage and control the parameter size.

[0072] Step 3.4: Comprehensive optimization of steps 3.1 to 3.3, further reduce feature confusion through prototype selection mechanism and mask strategy, ensure that the feature representations of new image classification tasks and old image classification tasks can coexist in harmony; To reduce feature confusion during distillation, a prototype selection mechanism based on scalable embedding space is adopted: Calculate the cosine similarity of all new samples and old prototypes:

[0073]

[0074] Set the similarity threshold σ, select the update strategy according to the sample similarity: dissimilar samples: participate in the update of the residual adapter, used to learn new features. Similar samples: participate in the distillation process, preserve the discriminative features of old classes. Optimize different loss functions by adding masks (Masks): If the similarity is greater than σ, add a mask to the distillation loss L kd ; If the similarity is less than σ, add a mask to the cross-entropy loss L ce . The final loss function is defined as:

[0075] L = Mask ce (L ce ) + λMask kd (L kd ) + γL proto

[0076] where λ and γ are loss weights.

[0077] The step 4: for a new test sample x, the pre-trained model Resnet-18 is used to extract its features, and the similarity value of the old image classification task is calculated to determine which model is used for testing, and the main network in a single model is used for testing, which can be mapped to the corresponding category. An image classification system based on the incremental learning of the multi-branch network architecture comprises a task similarity calculation module, a model expansion module and an auxiliary network module.

[0078] The task similarity calculation module adopts the Wasserstein distance to measure the similarity between the distributions of image classification tasks T i and T j The calculated image classification task similarity is used as the basis for model expansion of the model expansion module.

[0079] The model expansion module compares the maximum similarity T max with the threshold value δ, if it is greater than the threshold value, it does not need to be expanded, otherwise a new model is expanded for learning, the task similarity is calculated by using the image classification task similarity calculation module to determine whether to expand the model.

[0080] The auxiliary network module realizes structural expansion and reparameterization through the residual adapter, and considers new feature learning and old feature reservation, and performs structural expansion and reparameterization on the expanded model in the model expansion module, thereby slowing down the catastrophic forgetting and controlling the parameters of the expanded network model.

[0081] An image classification device based on the incremental learning of the multi-branch network architecture comprises:

[0082] A memory for storing a computer program for implementing the image classification method based on the incremental learning of the multi-branch network architecture.

[0083] A processor for implementing the image classification method based on the incremental learning of the multi-branch network architecture when the computer program is executed.

[0084] A computer readable storage medium comprises:

[0085] The computer readable storage medium stores a computer program, and the computer program can implement the image classification method based on the incremental learning of the multi-branch network architecture when the processor is executed.

[0086] The beneficial effects of the present application are:

[0087] The application quantifies the task similarity by calculating the Wasserstein distance between new and old tasks, and can intelligently determine whether to expand a new network model or reuse an existing model. This strategy not only reduces the time and computational cost of repeated training, but also enhances the adaptability and flexibility of the model in the face of diverse and dynamically changing task environments. In image classification tasks, when the new task involves similar image categories to the existing task (such as expanding from "cat" and "dog" classification to "tiger" and "wolf" classification), the application can effectively reuse the existing model, significantly reducing the training time.

[0088] The application adopts a multi-branch network architecture and integrates the parameters of the auxiliary network into the backbone network through a dynamic mapping function and a structure reparameterization strategy, thereby realizing learning of new tasks without significantly increasing the number of model parameters. This method effectively controls the size of the model parameters, reduces the consumption of memory and computing resources, and is suitable for resource-constrained practical application scenarios. In image classification tasks, when new categories (such as "bird") need to be added, the application can efficiently complete the learning of new tasks without significantly increasing the complexity of the model.

[0089] The application introduces a residual adapter and a dynamic structure recombination strategy to ensure that the model can focus on discriminative learning of new features while retaining important features of old tasks when learning new tasks. This balancing mechanism enables the model to fully exploit information from new tasks while stabilizing knowledge from old tasks when handling multiple tasks, thereby achieving comprehensive and balanced performance improvement. In image classification tasks, when a new "fish" classification task is added, the model can quickly learn the features of "fish" without forgetting the existing "cat" and "dog" classification capabilities, thereby achieving balanced optimization of multiple tasks. BRIEF DESCRIPTION OF DRAWINGS

[0090] Figure 1 The method flowchart of the application.

[0091] Figure 2 The task similarity calculation module schematic diagram of the application.

[0092] Figure 3 The model expansion module schematic diagram of the application.

[0093] Figure 4 The auxiliary network module schematic diagram of the application.

[0094] Figure 5 The classification accuracy curve of the application and other SOTA baselines on CIFAR100, TinyImageNet, mini-ImageNet, and subImageNet.

[0095] Figure 6The figure of classification performance decline curves of the present application and other SOTA baselines on CIFAR100, TinyImageNet, mini-ImageNet, subImageNet.

[0096] Figure 7 The figure of parameter quantity of the present application in the experiment of different incremental learning methods on SubImageNet 10.

[0097] Figure 8 The figure of results of the present application in the experiment of different thresholds on SubImageNet 10.

[0098] Figure 9 The figure of the framework of the present application. DETAILED DESCRIPTION

[0099] The present application will be further described in detail below with reference to the accompanying drawings.

[0100] Referring to Figure 1 , an image classification method based on a multi-branch network architecture incremental learning includes the following steps:

[0101] The data set adopts CIFAR100, TinyImageNet, mini-ImageNet, subImageNet. Through random seed, the classes of the data set are shuffled. In order to compare with the classical method, each image classification task includes the same number of classes, such as CIFAR100-20, which means that 20 tasks are divided, each task includes 5 classes. In the training process, batch_size=64, the Adam optimizer is used for optimization, the learning rate is set to 0.01, the learning rate decay adopts the cosine annealing strategy, and 80 epochs are trained.

[0102] Referring to Figure 2 , step 1.1: for T class incremental image classification tasks {D 1 ,D 2 ,…,D T}, is the tth class incremental task, including N t samples, is the ith sample of the tth class incremental task, is the label value of the corresponding ith sample of the tth class incremental task, assuming that the class set of task T i is C i , for each class c∈C i , a prototype feature is maintained, which is defined as the mean of the sample features belonging to the class c:

[0103]

[0104] where w i is the weight of sample X i , which can be defined according to class balance or sample confidence. In CIFAR10, CIFAR100 dataset, since the data is balanced, the value of w i is 1, and the features here are extracted using the pre-trained model Resnet-18.

[0105] Step 1.2: Nonlinear distance between samples and prototypes, the distance between a new sample and a certain prototype is calculated by a kernel function-based method:

[0106]

[0107] where k(·,·) is the kernel function, for example, Gaussian kernel, defined as:

[0108]

[0109] σ is the bandwidth parameter of the Gaussian kernel, which controls the width of the kernel function, set to 1 in the experiment.

[0110] In addition, the minimum distance of the new sample X l to all class prototypes of the previous task is defined as:

[0111]

[0112] where C j is the class set of task T j , and j is the prototype of class c in task T .

[0113] Step 1.3: Wasserstein distance, the dissimilarity between the distributions of tasks T i and T j can be measured by the Wasserstein distance, defined as:

[0114]

[0115] where Π(q i ,q j ) represents the joint distribution between the task distributions.

[0116] To simplify the calculation, Sinkhorn approximation is used to estimate the Wasserstein distance:

[0117]

[0118] Step 1.4: Define a dynamic mapping function to map the Wasserstein distance to the interval [0,1], as shown in the following formula:

[0119] s i,g =1-exp(-α·W ε (q i ,q j ))

[0120] Among them, this is the new task T i The similarity measure between the task group and the existing task group g is in the range of [0,1]. In the experiment, α was set to 0.65.

[0121] In the experiment, after the above steps, a set of similarity values ​​between the new task and the old task will be obtained. Based on the similarity values, the model can be flexibly selected for learning, which enhances the generalization and adaptability of the model.

[0122] See Figure 3 Step 2: For N obtained in Step 1 t Generate a similarity value for these N t Sort the values ​​and select the maximum value T. max The highest similarity value is compared with a threshold δ. If the highest similarity value is greater than the threshold, it indicates that the current task is very similar to a task we have learned before. In this case, the task is assigned to the network N corresponding to the previously learned task with the highest similarity. old Learning is then performed. This approach minimizes training costs and mitigates catastrophic forgetting. If the highest similarity value is less than a threshold, it indicates that the current task is dissimilar to all previously learned tasks, and the model needs to be expanded. The mathematical formula is as follows:

[0123]

[0124] Among them, G(T) cur ) indicates the current task T cur The assigned network. Our proposed expansion strategy is to copy the network with the highest similarity to the learned task, and use the copied network N. new Learn new tasks. This approach reduces the cost of learning from scratch and shortens training time.

[0125] In the experiment, the threshold δ was 0.6, which determined the expansion speed and performance of the model.

[0126] See Figure 4 Step 3.1: First Stage (Initial Training Stage)

[0127] Perform standard classification training using complete supervised data (training data X1 and label Y1). Train a classification model. Includes the following components:

[0128] (1) Feature Extractor Responsible for extracting high-dimensional feature representation of samples.

[0129] (2) Classifier Maps features to classification labels.

[0130] Optimization objective: Use cross-entropy loss L ce Train the model to improve classification performance.

[0131] Incremental stage (stage n)

[0132] Current task T n Input data only contains images of new task Does not contain old task samples. Use a new feature extractor Extract feature representation of new task:

[0133]

[0134] To learn discriminative features for new tasks, use new task labels (Supervised optimization from Y n ) to supervise the classifier Maps features to label space:

[0135]

[0136] Calculate cross-entropy loss L ce :

[0137]

[0138] At the same time, in order to retain the knowledge of the old task, use knowledge distillation to maintain the feature representation of the old class, extract the feature representation of the previous stage n-1:

[0139]

[0140] Calculate distillation loss L kd :

[0141]

[0142] Where F kd represents the Euclidean distance.

[0143] Step 3.2: In the NECIL scenario, store a prototype vector (Prototype) for each class in the feature space: Balance the classification performance of new tasks and old tasks through these prototypes.

[0144] The prototype is oversampled to match the batch size B, so as to calibrate the classifier:

[0145] p B = U p B(Prototype)

[0146] where U p B is the oversampling function.

[0147] The prototype balancing loss L proto is calculated:

[0148] L proto = F ce (p B , y B )

[0149] where y B is the oversampled label set.

[0150] Step 3.3: In order to learn new classes without bias while maintaining the feature representation of old classes, a dynamic structure reorganization strategy is adopted:

[0151] Structure expansion: insert a residual adapter in each convolutional block of the fixed feature extractor in the previous stage. The residual adapter only updates the most discriminative part while preserving the old features:

[0152]

[0153] Structure reparameterization: after training is complete, integrate the side branch information into the main branch without loss, reducing the parameter amount:

[0154]

[0155] Delete the adapter to keep the network structure unchanged for the next stage.

[0156] Step 3.4: In order to reduce feature confusion during distillation, a prototype selection mechanism based on scalable embedding space is adopted: calculate the cosine similarity between all new samples and old prototypes:

[0157]

[0158] Set the similarity threshold σ, and select the update strategy according to the sample similarity: dissimilar samples: participate in the update of the residual adapter, used to learn new features. Similar samples: participate in the distillation process, preserving the discriminative features of old classes. Optimize different loss functions by adding a mask: if the similarity is greater than σ, add a mask to the distillation loss L kd ; if the similarity is less than σ, add a mask to the cross-entropy loss L ceAdd mask. The final loss function is defined as:

[0159] L = Mask ce (L ce )+λMask kd (L kd )+γL proto

[0160] Where λ and γ are loss weights.

[0161] In the experiment, σ = 0.45, λ = 0.7, γ = 0.3, the threshold σ determines the selection of the update strategy, and λ and γ affect the training of the model.

[0162] Step 4: For a new test sample x, use the pre-trained model Resnet-18 to extract its features, and then calculate its similarity value with the old task to determine which model to use for testing. Within a single model, the main network is used for testing, which can be mapped to the corresponding category.

[0163] As Figure 9 shown above, the method is based on the incremental learning framework of the multi-branch network architecture, which consists of the following core components:

[0164] Task similarity calculation module: used to calculate the similarity between new graphical classification tasks and learned image classification tasks, based on the Wasserstein distance and dynamic mapping function.

[0165] Model expansion and reuse decision module: according to the similarity calculation result, decide whether to expand a new network model or reuse an existing model.

[0166] Multi-branch network architecture: after multiple model expansions, there are multiple network branches, and auxiliary networks are introduced in each network branch. Among them, the main network is used to learn similar tasks, and the auxiliary network is used to learn dissimilar tasks, and the auxiliary network is integrated into the main network through a dynamic mapping function and a structure reparameterization strategy.

[0167] Test and evaluation: use the expanded final model to predict the test sample and calculate the classification accuracy.

[0168] To evaluate the performance of the proposed pre-trained model based class-incremental learning method, the method of the present application (Ours) is compared with a variety of advanced incremental learning methods, namely EWC, MAS, OWM, Adam-NSCL, iCaRL, PackNet, RPSNet, WSN, Genifer. For the data sets CIFAR100, TinyImageNet, mini-ImageNet, subImageNet, all are divided into 10 tasks, 20 tasks, 25 tasks. The average accuracy and the forgetting degree BWT, the experimental results are shown in Table 1.

[0169] Table 1 Accuracy comparison of different methods on multiple data sets

[0170]

[0171]

[0172] From the results, the method of the present application (Ours) achieves the highest classification accuracy on all data sets: CIFAR100-20: 81.53%, TinyImageNet20: 60.89%, Mini-ImageNet-20: 69.21%, SubImageNet-10: 50.79%. Compared with the second best method (Genifer), the accuracy of the method of the present application on all data sets is significantly improved: CIFAR100-20 is improved by 1.43%, TinyImageNet20 is improved by 1.94%, Mini-ImageNet-20 is improved by 1.76%, and SubImageNet-10 is improved by 3.66%.

[0173] The performance drop reflects the degree of forgetting of the model when learning new tasks, and the value closer to zero indicates less forgetting. The method of the present application (Ours) has the smallest performance drop on all data sets: CIFAR100-20: -3.11%, TinyImageNet20: -4.58%, Mini-ImageNet-20: -3.67%, SubImageNet-10: -4.88%. Compared with other methods, the method of the present application significantly reduces the performance drop. For example, on SubImageNet-10, the performance drop of the method of the present application is -4.88%, which is significantly improved compared with -12.01% of Genifer.

[0174] See Figure 5 , Figure 6The method can effectively slow down the forgetting of the model to the old task while maintaining high classification accuracy. Although the EWC and MAS methods perform well on some data sets, they have a large performance decline on other data sets, indicating that they are not stable in the multi-task environment. The iCaRL and Genifer methods perform well in accuracy, but have a large problem in performance decline, especially on Mini-ImageNet-20 and SubImageNet-10. The OWM and PathNet methods have general performance in balancing the accuracy and performance decline, and fail to achieve the best performance on all data sets. The Adam-NSCL and WSN methods perform well on some data sets, but the overall performance is not as good as the method.

[0175] Referring to Figure 7 Since the method uses model expansion, the total parameter amount of the network used by different methods is compared, and it can be seen that, compared with the methods WSN and RPSNet which also use the same architecture, the method (Ours) has smaller total parameter amount under the same conditions, although the parameter amount is larger than the EWC and MAS methods based on regularization, but the performance is much higher, which proves that the method (Ours) is more suitable for real environment and reduces the consumption of memory.

[0176] Referring to Figure 8 Since the method uses model expansion, the influence of different thresholds on the model is compared, and with the increase of the threshold, the ACC and the parameter amount are increasing, because the threshold increases, the number of tasks less than the threshold increases, which means that the number of similar tasks increases, the number of model expansion increases, and therefore the parameter amount increases, the performance of the single model is improved, and therefore the ACC is also increasing, and by comprehensively considering the ACC and the parameter amount, it is appropriate to select δ=0.6, which has smaller parameter amount and higher performance than similar methods.

[0177] An image classification device based on a multi-branch network architecture incremental learning method, comprising:

[0178] A memory for storing a computer program for implementing the image classification device based on a multi-branch network architecture incremental learning method;

[0179] A processor for executing the computer program to implement the image classification device based on a multi-branch network architecture incremental learning method.

[0180] A computer readable storage medium, comprising:

[0181] The computer readable storage medium stores a computer program, and the computer program can implement an image classification method based on a multi-branch network architecture incremental learning when executed by a processor.

Claims

1. An image classification method based on incremental learning of a multi-branch network architecture, characterized in that, Comprising the following steps: Step 1: For each new image classification task T i , compute similarity to already learned old image classifications and rank them; Step 2: Compare the highest similarity value with the set similarity threshold value, if less than the threshold value, expand the new network model for learning, otherwise use the most similar network model for learning and updating; Step 3: Introduce an auxiliary network inside each network model in step 2, which is used to learn the dissimilar tasks inside the network model, and the backbone network of the network model learns the similar tasks using the knowledge distillation method, after learning, the auxiliary network is integrated into the backbone network, and the expanded final model is obtained, the control parameter size; Step 4: Use the expanded final model to predict the test sample, and calculate the final classification accuracy; The specific process of step 1 task similarity calculation is: Step 1.1: For T class-incremental image classification tasks {D 1 , 2 ,…,D T}, is the t-th class-incremental image classification task, including N t sample images, is the i-th sample of the t-th class-incremental image classification task, is the label value of the corresponding i-th sample of the t-th class-incremental image classification task, and the class set of the image classification task T i is C i , for each class c ∈ C i , a prototype feature is maintained, which is defined as the mean value of the image sample features belonging to the class c: where w i represents the weight of the sample X i , which is used to define according to the category balance or sample confidence; Step 1.2: Calculate the nonlinear distance between the sample image and the prototype feature based on the kernel function method: Where k(·,·) is the kernel function, defined as: Further, the new sample image X l The minimum distance to the prototype features of all classes of the previous image classification task is defined as: wherein C j is a set of classes of the task T j , is a prototype feature of the class c in the task T j . Step 1.3: Compute the Wasserstein distance, image classification task T i and T j The dissimilarity between the distributions of and T is measured by the Wasserstein distance, defined as: where Π(q i , q j ) denotes the joint distribution between task distributions; Sinkhorn approximation is used to estimate the Wasserstein distance: Step 1.4: Design a dynamic mapping function to map the Wasserstein distance to the interval [0,1], the formula is as follows: s i,g = 1 - exp(-a · W ε (q i , q j )) wherein the new image classification task T i a similarity measure between the new image classification task T and the old image classification task group g, with a value range of [0, 1], and a is a scaling factor adjusted by experiment.

2. The image classification method based on the incremental learning of the multi-branch network architecture according to claim 1, characterized in that, The specific process of step 2 is: After similarity calculation in step 1, N was obtained. t Group similarity values ​​and map them to the range [0,1]. For these N... t Sort the values ​​and select the maximum value T. max The task is compared with a threshold δ, which is between 0 and 1. If the highest similarity value is greater than the threshold, it indicates that the current task is very similar to the old task. In this case, the task is assigned to the network N corresponding to the learned task with the highest similarity. old To learn; If the highest similarity value is less than the threshold value, it indicates that the current image classification task is not similar to all learned image classification tasks, at this time the model needs to be expanded, the formula is as follows: Among them, G(T) cur ) represents the current image classification task T cur The assigned network is expanded using a strategy that involves copying the network with the highest similarity to the learned task and using the copied network N. new Learn new image classification tasks.

3. The image classification method based on the incremental learning of the multi-branch network architecture according to claim 2, characterized in that, The specific process of step 3 is: Step 3.1: Initial training stage: Use complete supervised data (training data X1 and label Y1) for standard classification training; training a standard classification model a standard classification model comprising the following components: (1) feature extractor responsible for extracting a high-dimensional feature representation of the sample; (2) classifier mapping the features to a classification label; Use the feature extractor to extract the features of the task, and then use the classifier to map these extracted features to specific labels; Using cross-entropy loss L ce Training the model, performing target optimization; Incremental stage: The input data for the current image classification task T n has only images of the new task does not contain samples of old image classification tasks, using a new feature extractor extracts a feature representation for the new task: Utilizing new image classification task labels Supervised optimization with classifier n ) using classifier Mapping features to label space: computing the cross-entropy loss L ce : At the same time, use knowledge distillation to maintain the feature representation of the old class, and extract the feature representation of the last stage n-1: Computing the distillation loss L kd : where F kd denotes the Euclidean distance; Step 3.2: Further optimize the feature representation by using prototype vectors and balanced loss to balance the classification performance of new tasks and old tasks; In the NECIL scenario, store a prototype vector (Prototype) for each class in the feature space, and balance the classification performance of new image classification tasks and old image classification tasks through these prototypes; Step 3.3: Adopt a dynamic structure reorganization strategy to ensure that the network in steps 3.1 and 3.2 does not destroy the feature representation of the old image classification task when learning the new image classification task, and to maintain the old class feature representation while learning the new class without bias; Step 3.4: Comprehensive optimization of steps 3.1 to 3.3, using a prototype selection mechanism based on an expandable embedding space and a mask strategy to reduce feature confusion and ensure that the feature representations of new image classification tasks and old image classification tasks can coexist.

4. The image classification method based on the incremental learning of the multi-branch network architecture according to claim 3, characterized in that, The specific process of step 3.2 is: Over-sampling the prototype to match the batch size B, thereby calibrating the classifier: p B = U p B (Prototype) where U p B is an oversampling function; Computing the prototype balancing loss L proto : L proto = F ce (p B , y B ) where y B is the set of labels after oversampling.

5. The image classification method based on the incremental learning of the multi-branch network architecture according to claim 3, characterized in that, The specific process of step 3.3 is: Structure expansion: Insert a residual adapter (ResidualAdapter) in each convolution block of the feature extractor, which only updates the most discriminative part while preserving the old features: Structure reparameterization: After training is completed, integrate the side branch information into the main branch without loss, reducing the parameter amount: Delete the adapter, keep the network structure unchanged for the next stage, control the size of the parameter; The step 3.4 is specifically: Calculate the cosine similarity of all new samples and old prototypes: Set the similarity threshold σ, and select the update strategy according to the sample similarity: Dissimilar samples: participate in the update of the residual adapter and are used to learn new features; Similar Samples: participate in the distillation process and retain discriminative features of old categories; Optimize different loss functions by adding masks: If similarity is greater than σ, distillation loss L kd Add mask; if similarity is less than σ, cross-entropy loss L ce Add mask, final loss function defined as: L = Mask ce (L ce )+λMask kd (L kd )+γL proto Where λ and γ are loss weights.

6. The image classification method based on incremental learning of multi-branch network architecture according to claim 3, characterized in that, In step 4: for a new test sample x, use the pre-trained model Resnet-18 to extract its features, then calculate its similarity value with the old image classification task, and decide which model to use for testing. In a single model, the main network can be used for testing to map to the corresponding category.

7. A multi-branch network architecture based incremental learning image classification system for implementing the method of any one of claims 1-6, characterized in that, Including task similarity calculation module, model expansion module and auxiliary network module; The task similarity calculation module adopts a Wasserstein distance to measure the similarity between the distributions of image classification tasks T i and T j , and the calculated image classification task similarity serves as the basis for model expansion by the model expansion module. The model expansion module adopts maximum similarity T max The threshold value δ is compared, if greater than the threshold value, no expansion is needed, otherwise, the new model is expanded and learned, the image classification task similarity calculation module is used to calculate the similarity of the task, so as to determine whether to expand the model. The auxiliary network module realizes structure expansion and reparameterization through the residual adapter, and considers new feature learning and old feature retention. The expanded model in the model expansion module is expanded and reparameterized to control the parameters of the expanded network model while slowing down the catastrophic forgetting.

8. An image classification device based on incremental learning of a multi-branch network architecture, characterized by, Including: Memory: for storing the computer program of the image classification method based on the multi-branch network architecture incremental learning according to any one of claims 1-6; Processor: for executing the computer program to realize the image classification method based on the multi-branch network architecture incremental learning.

9. A computer-readable storage medium, characterized in that, Including: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the image classification method based on the multi-branch network architecture incremental learning according to any one of claims 1-6.

Images