Image classification method for continual learning based on dual-branch network

Abstract

The application discloses a kind of based on the image classification method of continuous learning of double-branch network, belong to image processing technical field.The application includes: constructing double-branch network, including main branch, memory branch;Using memory branch learns the first batch of categories, and generates the prediction result of the first batch of categories;From the second batch, using main branch learns the t batch of categories, updates memory branch parameter, generates the prediction result of the first batch to the t batch of categories;Subsequent batch uses the same method of step three to learn, until learning all batches.The application can effectively avoid the forgetting of old categories caused by learning new categories and effectively overcome the prediction bias phenomenon caused by class imbalance, and does not additionally increase model parameter quantity or network structure.The application effectively improves the performance of model processing class incremental data.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to an image classification method based on continuous learning of a dual-branch network. Background Technology

[0002] With the development of deep learning technology, image classification based on deep learning has become increasingly popular. Traditional deep learning methods follow a "train once" strategy, using all training data at once. While this paradigm has achieved excellent results in many fields, "train once" is often impractical in most real-world scenarios. These scenarios require models to handle continuously arriving data streams, i.e., to continuously learn new categories.

[0003] Existing incremental class learning methods typically impose constraints on the model when learning new classes to avoid forgetting older classes. However, this approach often leads to a significant drop in accuracy for older classes when learning new ones. Another approach is to assign unused parameters to new classes or add parts of the network structure. However, these methods result in a continuous increase in the number of model parameters, leading to greater computational costs. Existing incremental class learning methods often suffer from prediction bias caused by class imbalance, where predictions favor the most recently learned class, again causing a significant drop in accuracy for older classes. Summary of the Invention

[0004] This invention provides an image classification method based on continuous learning of a dual-branch network, which effectively avoids the forgetting of old categories when learning new categories during image classification and effectively overcomes the prediction bias caused by class imbalance.

[0005] The technical solution adopted in this invention is as follows:

[0006] An image classification method based on continuous learning using a dual-branch network includes the following steps:

[0007] Step 1: Construct a dual-branch network for image classification, which includes a main branch feature extraction network, a memory branch feature extraction network, a main classifier, a memory classifier, and an auxiliary classifier;

[0008] The main branch feature extraction network consists of, in sequence: an input convolutional block, multiple main branch residual layers, an output convolutional block, and a global average pooling layer; the memory branch feature extraction network consists of, in sequence: multiple memory branch residual layers, an output convolutional block, and a global average pooling layer; and the input of the memory branch is the output of the first main branch residual layer of the main branch.

[0009] The input to the main classifier includes: the output features of the memory branch feature extraction network, and the combined features obtained by concatenating the output features of the main branch feature extraction network and the output features of the memory branch feature extraction network;

[0010] The input to the memory classifier is the output feature of the memory branch feature extraction network;

[0011] The auxiliary classifier is input to the output features of the main branch feature extraction network, and the auxiliary classifier is only used for training the dual-branch network.

[0012] Step 2: Divide the image classification task into multiple batches according to the order of class increment;

[0013] Step 3: Based on the memory branch feature extraction network, learn the first batch of categories and generate image classification prediction results for the first batch of categories:

[0014] Set the predicted categories of the main classifier to: the image categories of the first batch of categories;

[0015] The input image is processed through an input convolutional block, the first main branch residual layer of the main branch, and a memory branch feature extraction network to obtain the output features of the first batch of categories.

[0016] Output features Input the main classifier, and obtain the prediction results of the first batch of categories based on the output of the main classifier;

[0017] Step 4: Learn the current batch category based on the dual-branch network, update the network parameters of the dual-branch network, and generate prediction results for the categories from batch 1 to the current batch:

[0018] Step 401: Update the network parameters of the dual-branch network:

[0019] Let t represent the current batch category number, where the initial value of t is 2;

[0020] For the t-th batch category, extract the output features of the main branch feature extraction network. Output features of memory branch feature extraction network The combined features are denoted as combined features.

[0021] Based on the total number of image categories from batch 1 to batch t, ​​the number of predicted categories for the auxiliary classifier, the main classifier, and the memory classifier is set. The auxiliary classifier, the main classifier, and the memory classifier are used to output the image classification probability for each category (image category).

[0022] Output features Input the auxiliary classifier and obtain the prediction results of the auxiliary classifier from the 1st to the tth batch based on its output;

[0023] Combined features Input the main classifier and obtain the prediction results of the main classifier for batches 1 to t based on its output;

[0024] Output features Input the memory classifier and obtain the memory classifier prediction results for batches 1 to t based on its output;

[0025] The cross-entropy loss of the main branch feature extraction network is calculated based on the prediction results of the auxiliary classifier, the main classifier, and the memory classifier; and the network parameters of the main branch feature extraction network are updated based on the cross-entropy loss.

[0026] Based on the combination characteristics of the t-th batch Combined features with batch t-1 Calculate the distillation loss of the memory branch feature extraction network and update the network parameters of the memory branch feature extraction network based on the distillation loss;

[0027] When the preset network parameter update termination condition is met (e.g., the number of training iterations, or the total loss (the sum or weighted sum of the cross-entropy loss of the main branch feature extraction network and the distillation loss of the memory branch feature extraction network) meets the specified condition), the update of the network parameters of the main branch feature extraction network and the memory branch feature extraction network is stopped, and the updated dual-branch network is obtained.

[0028] Step 402: Generate image category prediction results for batches 1 to t:

[0029] The target image is input into the updated dual-branch network, and the output features of the main branch feature extraction network and the output features of the memory branch feature extraction network are concatenated to obtain the combined features. The combined features are then input into the main classifier, and the main classifier prediction result of the target image is obtained based on its output.

[0030] Simultaneously, the output features of the memory branch feature extraction network are input into the memory classifier, and the memory classifier prediction result of the target image is obtained based on its output;

[0031] For each image classification probability of batches 1 to t-1, the image classification probability of the same category in the main classifier prediction result of the target image and the memory classifier prediction result is weighted and summed to obtain the image classification fusion probability of batches 1 to t-1. Based on the image classification fusion probability of each category and the image classification probability of each batch of t in the main classifier prediction result of the target image, the image classification ensemble probability of batches 1 to t is generated.

[0032] Step 5: If the current batch category number t is less than the maximum batch category number, then t is incremented by 1, and step 4 is executed again.

[0033] Furthermore, in step 4, when calculating the cross-entropy loss of the main branch feature extraction network based on the prediction results of the auxiliary classifier, the main classifier, and the memory classifier, the true labels corresponding to the auxiliary classifier, the main classifier, and the memory classifier are respectively set as:

[0034] The auxiliary classifier summarizes each image category in the first batch to the (t-1)th batch into a single category label, and assigns each image category in the tth batch as a separate category label.

[0035] The main classifier assigns each image category from the first batch to the t-th batch as a separate category label.

[0036] The memory classifier assigns each image category in the first to the (t-1)th batches as a separate category label, and summarizes each image category in the tth batch as a single category label.

[0037] Furthermore, in step 4, a cosine-rising ensemble strategy is used to set the weights of the prediction results of the main classifier and the prediction results of the memory classifier.

[0038] Furthermore, in step 4, the weight of the main classifier's prediction result is set to 1, and the weight of the memory classifier's prediction result is set to α(t): Where t max This indicates the number of batches classified in step 2, i.e., the largest batch category number.

[0039] Furthermore, the main branch feature extraction network includes four main branch residual layers, which are configured as follows:

[0040] The network structures of the first residual layer and the second residual layer of the main branch are the same, consisting of two stacked residual blocks;

[0041] The network structures of the third residual layer and the fourth residual layer of the main branch are the same, both consisting of a residual block.

[0042] The memory branch feature extraction network includes two branch residual layers, configured as follows:

[0043] The first residual layer of the memory branch consists of four stacked residual blocks;

[0044] The second residual layer of the memory branch consists of one residual block.

[0045] Furthermore, the main classifier, memory classifier, and auxiliary classifier are all composed of a fully connected linear classifier and a SoftMax layer, wherein the SoftMax layer is used to output the image classification probability of each category.

[0046] The technical solution provided by this invention brings at least the following beneficial effects:

[0047] This invention effectively avoids forgetting old categories when learning new categories and overcomes prediction bias caused by class imbalance, without increasing the number of model parameters or network structure. This invention effectively improves the performance of the model in processing incremental data. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of an image classification method based on continuous learning of a dual-branch network provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the dual-branch network structure used in an embodiment of the present invention;

[0051] Figure 3 This is a comparison chart of the confusion matrix when using integration and when not using integration in an embodiment of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0053] like Figure 1 As shown, this embodiment provides an image classification method based on continuous learning of a dual-branch network, which includes the following steps:

[0054] Step 1: Construct a two-branch network for image classification, which includes a main branch feature extraction network (referred to as the main branch), a memory branch feature extraction network (referred to as the memory branch), a main classifier, a memory classifier, and an auxiliary classifier.

[0055] like Figure 2As shown, the main branch consists of: an input convolutional block, four residual layers, an output convolutional block, and a global average pooling layer; the memory branch consists of: two residual layers, an output convolutional block, and a global average pooling layer. The four residual layers in the main branch are the first, second, third, and fourth residual layers. The first and second residual layers each contain two residual blocks, while the third and fourth contain one residual block. These residual blocks use the basic residual blocks from ResNet, consisting of two stacked convolutional blocks and a lateral identity connection. The memory branch contains two residual layers: the first and second residual layers. The first residual layer consists of four stacked residual blocks, while the second residual layer consists of one residual block. The main classifier, memory classifier, and auxiliary classifier are constructed using fully connected linear classifiers and SoftMax layers. The input and output convolutional blocks have the same network structure, consisting of convolutional layers, batch normalization layers, and ReLU activation functions.

[0056] Step 2: Use memory branches to learn the first batch of categories and generate prediction results for the first batch of categories;

[0057] The first batch of category learning uses an input convolutional block, the first residual layer of the main branch, a memory branch, and the main classifier. The input image, after passing through the input convolutional block and the first residual block of the main branch, is fed into the memory branch. The features generated by the memory branch... Input to the main classifier FC main And generate the prediction results for the first batch of categories.

[0058]

[0059]

[0060] Where q1 represents the main classifier FC main For the output of the first batch of categories (i.e., the image classification probability vector), q1[y] represents the image classification probability of the y-th image category in q1.

[0061] Step 3: Use a two-branch network to learn the t-th batch of categories (t initialized to 2, t≥2), and generate prediction results for the 1st to t-th batches of categories, specifically:

[0062] (a) Features generated by the main branch Features generated by memory branches splicing to form combined features

[0063] (b) Update the parameters of the two-branch network:

[0064] The parameter updates for the main branch and the memory branch employ two different loss functions: cross-entropy loss and distillation loss.

[0065] The cross-entropy loss function used in the main branch is as follows:

[0066]

[0067] in, This represents the number of images in the dataset; y t These are the real labels for categories 1 through t, y t,i y′ represents the true label of the i-th image, which corresponds to the true label of the main classifier; t The true labels of the t-th batch of categories are included, and the categories from the 1st batch to the (t-1th batch) are grouped into one category, which corresponds to the true label of the auxiliary classifier, y′. t,i Indicates the i-th image in the true label y′ t The value in y″. t It contains the true labels of categories from batch 1 to batch t-1, and groups the categories of batch t into one category, which corresponds to the true label of the memory classifier, y″. t,i This indicates that the i-th image is in the true label y″ t The value in q. t ,q′ t and q″ t The outputs of the main classifier, the auxiliary classifier, and the memory classifier are respectively, q t,i ,q′ t,i and q″ t,i These represent the outputs of the main classifier, auxiliary classifier, and memory classifier for the i-th image, respectively. The auxiliary classifier is only used during training.

[0068] Specifically, the distillation loss function used in the memory branch is as follows:

[0069]

[0070] in, Let represent the combined features of the i-th image in the t-th and t-1-th batches, respectively.

[0071] The total loss function during training is:

[0072]

[0073] (c) Integrate the outputs of the main classifier and the memory classifier to generate prediction results for the first to t batches of categories. That is, after the training of the current batch (batch t) is completed, the prediction results of the categories from batch 1 to batch t can be generated based on the trained network;

[0074] The integration employs a weighted summation method. Preferably, the integrated output can be adjusted using a scalar function with an increasing cosine.

[0075]

[0076]

[0077]

[0078]

[0079] in, The number of categories from batch 1 to batch (t-1) Let t represent the number of categories from batch 1 to batch t. max This represents the maximum batch size. Concat indicates channel concatenation.

[0080] Step 4: Return to Step 3 and increment t by 1. Repeat this process until all categories have been learned, i.e., the maximum batch size t is reached. max .

[0081] To further verify the image classification performance of the image classification method based on continuous learning with a dual-branch network provided in this invention, the method of this invention is compared with previous state-of-the-art methods on CIFAR-100 (a miniature image dataset), as shown in Tables 1 to 5. As shown in Table 1, the method of this invention significantly outperforms all baselines. This significant improvement demonstrates the effectiveness of the invention and its strong generalization ability in various incremental scenarios.

[0082] Table 1: Results on CIFAR-100 (Top-1 accuracy %), averaged in order of the three different categories.

[0083]

[0084] Table 2: Results of two batches of incremental data on CIFAR-100 (Top-1 accuracy %), averaged in three different categories.

[0085]

[0086]

[0087] Table 3: Results of 5 batches of incremental data on CIFAR-100 (Top-1 accuracy %), averaged in three different categories.

[0088]

[0089] Table 4: Results of 10 incremental batches on CIFAR-100 (Top-1 accuracy %), averaged in three different categories.

[0090]

[0091] Table 5-1: Results of 20 incremental batches on CIFAR-100 (Top-1 accuracy %), averaged in three different categories.

[0092]

[0093]

[0094] Table 5-2: Results of 20 incremental batches on CIFAR-100 (Top-1 accuracy %), averaged in three different categories.

[0095]

[0096] As can also be observed in Tables 2 to 5, the embodiments of the present invention consistently outperform other existing methods (iCaRL, UCIR, BiC, WA, S&B) in every batch across all batch settings. This advancement demonstrates the effectiveness of the present invention in complex incremental scenarios. Figure 3 The confusion matrices with and without ensemble are shown. Without ensemble, the model is susceptible to prediction bias, tending to classify samples into new categories. With ensemble, this phenomenon is mitigated on both the diagonal (i.e., correct predictions) and off-diagonal (i.e., incorrect predictions) sides. These improvements demonstrate that the present invention can effectively handle prediction bias.

[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

[0098] The above descriptions are merely some embodiments of the present invention. Those skilled in the art can make various modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the scope of protection of the present invention.

Claims

1. A method of image classification based on continual learning of a dual-branch network, characterized in that, The method comprises the following steps: Step 1: constructing a double-branch network for image classification, the double-branch network comprising a main branch feature extraction network, a memory branch feature extraction network, a main classifier, a memory classifier and an auxiliary classifier; The main branch feature extraction network comprises, in sequence, an input convolution block, a plurality of main branch residual layers, an output convolution block and a global average pooling layer; the memory branch feature extraction network comprises, in sequence, a plurality of memory branch residual layers, an output convolution block and a global average pooling layer; and the input of the memory branch is the output of the first main branch residual layer of the main branch; The input of the main classifier comprises the output feature of the memory branch feature extraction network and a combined feature obtained by splicing the output feature of the main branch feature extraction network and the output feature of the memory branch feature extraction network; The input of the memory classifier is the output feature of the memory branch feature extraction network; The input of the auxiliary classifier is the output feature of the main branch feature extraction network, and the auxiliary classifier is only used for training of the double-branch network; Step 2: dividing the classes of the image classification task into a plurality of class batches according to the order of class increment; Step 3: learning the first batch of classes based on the memory branch feature extraction network and generating image classification prediction results of the first batch of classes: setting the prediction classes of the main classifier to be each image class of the first batch of classes; The input picture is input into an input convolution block, a first main branch residual layer of a main branch, and a memory branch feature extraction network to obtain output features of a first batch of categories output features input the main classifier, and obtain a prediction result of the first batch of categories based on an output of the main classifier; Step 4: learning the current batch of classes based on the double-branch network, updating the network parameters of the double-branch network and generating prediction results of the first batch to the current batch of classes: Step 401: updating the network parameters of the double-branch network: defining t to represent the current batch class number, wherein the initial value of t is 2; For the t-th batch of categories, the output feature of the main branch feature extraction network is taken as the main branch feature and the output feature of the memory branch feature extraction network is taken as the memory branch feature The combined feature is formed by splicing, denoted as combined feature based on the number of all image classes from the first batch to the tth batch, setting the number of prediction classes of the auxiliary classifier, the main classifier and the memory classifier, the auxiliary classifier, the main classifier and the memory classifier being used to output image classification probabilities of each class respectively; output features input the auxiliary classifier, and obtain the first to the t-th batch of auxiliary classifier prediction results based on the output thereof Combination features inputting the main classifier, and obtaining the first to the t-th batch of main classifier prediction results based on the output of the main classifier; output features input memory classifier, and obtain the first to the t-th batch of memory classifier prediction results based on the output thereof; calculating the cross-entropy loss of the main branch feature extraction network based on the prediction results of the auxiliary classifier, the prediction results of the main classifier and the prediction results of the memory classifier; and updating the network parameters of the main branch feature extraction network based on the cross-entropy loss; combination features of the t-th batch combination features of the t-1-th batch calculate a distillation loss of the memory branch feature extraction network, and update network parameters of the memory branch feature extraction network based on the distillation loss; when a preset network parameter update end condition is met, stopping the update of the network parameters of the main branch feature extraction network and the memory branch feature extraction network, to obtain an updated double-branch network; Step 402: generating image class prediction results from the first batch to the tth batch: inputting a target image into the updated double-branch network, splicing the output feature of the main branch feature extraction network and the output feature of the memory branch feature extraction network to obtain a combined feature, inputting the combined feature into the main classifier, and obtaining the main classifier prediction result of the target image based on the output thereof; at the same time, inputting the output feature of the memory branch feature extraction network into the memory classifier, and obtaining the memory classifier prediction result of the target image based on the output thereof; The image classification fusion probability of the first batch to the t-1th batch of categories is obtained by weighting and summing the image classification probability of the same category in the main classifier prediction result of the target image and the memory classifier prediction result; the image classification integrated probability of the first batch to the tth batch of categories is generated based on the image classification fusion probability of each category and the image classification probability of the tth batch of categories in the main classifier prediction result of the target image, and the prediction result of the kth batch is obtained based on the maximum item in the image classification integrated probability, wherein k=1, 2, …, t; Step 5: if the current batch category number t is less than the maximum batch category number, then t is increased by 1, and step 4 is continued to be executed.

2. The method of claim 1, wherein, In step 4, when calculating the cross-entropy loss of the main branch feature extraction network based on the auxiliary classifier prediction result, the main classifier prediction result and the memory classifier prediction result, the true labels corresponding to the auxiliary classifier, the main classifier and the memory classifier are respectively set as: The memory classifier takes each image category in the first batch to the t-1th batch of categories as a category label respectively, and the image categories in the tth batch of categories are summarized as a category label; The main classifier takes each image category in the first batch to the tth batch of categories as a category label respectively; The auxiliary classifier takes each image category in the tth batch of categories as a category label respectively, and the image categories in the first batch to the t-1th batch of categories are summarized as a category label.

3. The method of claim 1, wherein, In step 4, the weight of the main classifier prediction result and the memory classifier prediction result is set by using the cosine rising integration strategy.

4. The method of claim 3, wherein, In step 4, the weight of the main classifier prediction result is set to 1, and the weight of the memory classifier prediction result is set to a(t): where t max represents the number of batches of classes divided in step 2.

5. The method according to any one of claims 1 to 4, characterized in that, The main branch feature extraction network includes four main branch residual layers, which are respectively set as: The network structure of the main branch first residual layer and the main branch second residual layer is the same, including two stacked residual blocks; The network structure of the main branch third residual layer and the main branch fourth residual layer is the same, including one residual block; The memory branch feature extraction network includes two branch residual layers, which are respectively set as: The memory branch first residual layer includes four stacked residual blocks; The memory branch second residual layer includes one residual block.

6. The method according to any one of claims 1 to 4, wherein The main classifier, the memory classifier and the auxiliary classifier all adopt a fully connected linear classifier and a SoftMax layer. The main classifier, the memory classifier and the auxiliary classifier all adopt a fully connected linear classifier and a SoftMax layer.

Images