Online image increment classification method based on multi-domain distribution multiplexing network

Abstract

The invention discloses an online image increment classification method based on a multi-domain distribution multiplexing network, and the method comprises the steps: taking RGB image data as input, and carrying out the increment division of a data set; sending the divided image data into a multi-domain distribution multiplexing image classification network, and training the established model parameters; adding new image data to train the whole model again, and repeating the step until no new data can be added; and finally, testing the trained model, and outputting an online image increment classification result. According to the method, the defects of a traditional image increment classification method can be effectively overcome; meanwhile, a frequency domain distribution multiplexing module is designed, and hybrid factors of different high and low frequency combinations are introduced to obtain orthogonal feature aggregation information of different channels to highlight causal related features. In each iteration, causal biased features are implicitly converted into unbiased features, so that feature distribution offset between training and memory buffer data is relieved.

Description

Technical Field

[0001] The invention discloses an online image incremental classification method based on a multi-domain distribution multiplexing network, and belongs to the technical field of computer vision. Background Art

[0002] Modern artificial intelligence models are expected to have the ability to learn continuously, just like humans, so incremental learning has become a hot topic of research. Incremental learning mainly explores the problem of data stream learning under non-stationary data distribution, where the data comes from various tasks and the input domain is constantly evolving. Traditionally, incremental learning includes three paradigms: task incremental learning (incrementally learning a set of tasks, and the task ID is visible during testing), domain incremental learning (learning the same type of problem in different domains), and class incremental learning (incrementally learning a set of tasks, and the task ID is invisible during testing). The primary challenge facing incremental learning is catastrophic forgetting, that is, the historical knowledge in the old data is replaced by new knowledge. In recent years, thanks to the rapid development of deep learning, a large number of image incremental classification methods have been proposed, which can be generally divided into the following three categories: memory replay-based methods, regularization-based methods, and parameter isolation-based methods.

[0003] However, the aforementioned incremental image classification methods lack flexibility in practical applications because real-world data usually arrives in the form of online streams without task identifiers. Summary of the invention

[0004] The technical purpose of the present invention is to propose a more general incremental learning setting, namely an online incremental image classification method, in which data arrives gradually in the form of a stream, and training is performed whenever a small batch of training samples accumulates. This method requires that the model can classify images of all categories it has seen at each stage. However, due to the fast pace of the data stream, the learning algorithm can only be trained once for each data batch, and the task identifiers are not available. Formally, let T be the total number of tasks, and the online incremental image classification model processes a continuous sequence of tasks derived from a single-pass data stream. in represents the training data for task t, t = 1, 2, ..., T. Online image incremental classification allows And it is required that the task boundary is unknown during the entire training and testing phase. The training data for each task t arrives gradually in the form of a stream. Whenever a small batch of data for task t is accumulated from the data stream, When , it is trained in one iteration. After seeing all the data of a task, the next task starts. Online image incremental classification faces more serious catastrophic forgetting, mainly due to inter-task bias and intra-task bias.

[0005] (1) Inter-task bias: Due to online learning and unknown task boundaries during the online incremental image classification training process, inter-task bias occurs and is more difficult to eliminate than traditional incremental image classification.

[0006] (2) Intra-task bias: Since data arrives in the form of a stream, each batch involves only a small amount of data, resulting in huge intra-task bias caused by model overfitting.

[0007] Many studies have alleviated inter-task bias by balancing inter-class relations or learning representative features of all classes. CoCa adopts collaborative distillation to solve the problem of relation bias in online incremental image classification, and uses pretext tasks and supervised contrastive learning to learn complete and discriminative features of all categories through a collaborative self-supervised network. CBA targets the problem of posterior probability shift in online incremental image classification and proposes a two-layer learning framework with a continuous bias adapter to enhance the classifier network to adapt to catastrophic distribution changes during training, so that the classifier network can learn a stable integration of previously learned tasks. Although these methods have good performance, most of them ignore the problem of intra-task bias in online incremental image classification.

[0008] The present invention aims to solve the problems of inter-task bias and intra-task bias in online incremental image classification at the same time, but the main challenge is that the form of streaming data easily makes the model fall into the trap of false correlation. Therefore, it becomes crucial to resist false correlation and reduce the feature bias of streaming data. Some researchers have proposed two methods, counterfactual data enhancement and causal intervention, to resist false correlation from a causal perspective. Specifically, counterfactual data enhancement adds additional counterfactual data to training to eliminate spurious correlations. CGN, with the help of the concept of causal independent mechanism, constructs a counterfactual image generation model by artificially separating the image generation process into separate mechanisms of background, shape and texture, and improves the robustness of the image classification model by adding the generated fake images to the training. Causal effect calibration reduces bias by adjusting the effect on confounding factor variables. CONTA uses structural causal relationship models to deeply analyze the causal relationship between image features, context and class labels, and achieves state-of-the-art performance in weakly supervised semantic segmentation tasks through context adjustment. However, both causal inference methods require a deep understanding of potential confounding factors, which are difficult to identify in online incremental image classification. For example, in offline mode, CafeBoost proposed the concept of task-induced bias, which introduces class features from previous tasks into new class features in the current task, and uses backdoor adjustments to confounding factors to reduce the model's focus on biased features. However, these two methods require a certain understanding of confounding factors. Since data enters in the form of streams, the model cannot repeatedly iterate on a type of data to obtain a stable source of false correlation, and therefore cannot actually intervene in confounding factors.

[0009] Therefore, the present invention does not explicitly model the confounding factors, but attempts to perturb the confounding factors while keeping the causal factors unchanged. Some studies have shown that frequency representation contains rich image understanding patterns, and frequency changes can be used to separate entangled feature information in images. Inspired by these works, we propose a pseudo-causal intervention strategy based on frequency transformation, which intervenes in two stages: intervention on confounding factors and intervention on causal factors, and combines the two to form a frequency domain distribution multiplexing module (Frequency Division Multiplexer, referred to as FDM) to resist false correlations. In addition, we found that relying solely on frequency domain features cannot fully emphasize causal features, especially for images with rich semantics. Therefore, the spatial distribution multiplexing module (Spatial Division Multiplexer, referred to as SDM) is introduced to emphasize the representation ability of different target areas in the spatial domain as a supplement to the former. Therefore, the present invention proposes a dual-domain division multiplexer (D3M) as a plug-and-play module for online incremental image classification, which is integrated with the convolutional network into a multi-domain distribution multiplexing network. The multi-domain distribution and reuse module divides and reuses features in the frequency and spatial domains, which effectively alleviates the impact of spurious correlations and helps to learn invariant causal features to mitigate catastrophic forgetting in online incremental image classification.

[0010] The present invention aims to solve the technical problems in online incremental image classification, mainly including the simultaneous processing of inter-task bias and intra-task bias, and how to use causal intervention methods to resist false correlations, thereby alleviating catastrophic forgetting and improving classification accuracy. The present invention proposes a multi-domain distribution multiplexing module, which consists of a frequency domain distribution multiplexing module and a spatial distribution multiplexing module, and is used to intervene in confounding factors and multiple causal factors in the frequency domain and the spatial domain. The frequency domain distribution multiplexing module divides and reuses the features of different frequency components, and prioritizes the causal features related to the target through a two-stage pseudo-causal intervention strategy. The first intervention is called frequency distribution, which represents the image by selecting different frequency transforms and adaptively adjusts the contributions of different frequencies to perturb different parts of the confounding factors. The second intervention is called frequency multiplexing, which eliminates the dependencies between features based on the first intervention. The spatial distribution multiplexing module does similar work in the spatial domain as a supplement to the frequency domain distribution multiplexing module. As a lightweight model enhancement module, the multi-domain distribution and reuse module can be seamlessly integrated into most current online image incremental classification methods to form a multi-domain distribution and reuse network, thereby effectively solving problems such as inter-task bias, intra-task bias and false correlation, and improving the accuracy and robustness of online image incremental classification.

[0011] The online image incremental classification method based on the multi-domain distribution multiplexing network proposed in the present invention can effectively make up for the shortcomings of the traditional image incremental classification method. The overall framework of the multi-domain distribution multiplexing network introduces a multi-domain distribution multiplexing unit into the traditional convolutional network, applies the frequency domain and space modules in parallel, and eliminates confounding factors by learning which information to emphasize or suppress, thereby highlighting the feature information causally related to the target. Specifically, the present invention designs a frequency domain distribution multiplexing module, introduces confounding factors of different high and low frequency combinations, to obtain orthogonal feature aggregation information of different channels, thereby highlighting causally related features. At the same time, the spatial distribution multiplexing module focuses on capturing the aggregation information of different target areas in the feature map. In essence, the proposed method implicitly converts causally biased features into unbiased features at each iteration, thereby alleviating the feature distribution offset between training and memory buffer data.

[0012] The present invention is achieved through the following technical solutions:

[0013] The online image incremental classification method based on the multi-domain distribution multiplexing network takes RGB image data as input and incrementally divides the data set; then the divided image data is sent to the multi-domain distribution multiplexing image classification network to train the established model parameters; then new image data is added to train the entire model again, and this step is repeated until no new data can be added; finally, the trained model is tested and the online image incremental classification results are output. The specific steps are as follows:

[0014] Step 1: Establish a multi-domain distribution multiplexing network model

[0015] Step 11. Training data set input and general increment division;

[0016] In order to simulate the real scene of the continuous increase of new images on the Internet, the RGB image dataset will be randomly divided into different sub-datasets by category to represent the category image data added at different times, and the data will be divided into tasks according to the sequential and smooth benchmarks. The sequential benchmark must ensure that the categories between different tasks are not repeated, while the smooth benchmark allows the same category to be repeated between different tasks.

[0017] Step 12. Design a multi-domain distribution multiplexing network;

[0018] The multi-domain distribution multiplexing module D3M is a plug-and-play module. Given an input image, it first passes through the ResNet18 convolutional layer to obtain a preprocessed feature map. Subsequently, the feature map passes through the multi-domain distribution multiplexing module to generate an updated new feature map, which is causally adaptive. This updated feature map is then propagated to subsequent layers.

[0019] Step 13. Design frequency domain distribution multiplexing FDM module;

[0020] Step 13.1 frequency domain distribution submodule;

[0021] In the frequency domain distribution submodule, the feature map is divided into different components. First, the original feature map after the fixed convolution layer is Divided into J groups along the channel dimension, namely

[0022]

[0023] in is the number of groups, satisfying J|C', J ≤ H'W". Using different 2DDCT bases (u j ,v j )∈{1…H'}×{1…W'} for each group of channel components Perform frequency transformation to obtain frequency domain components Then connect them together to get the multi-frequency domain components c l :

[0024]

[0025] Step 13.2 Frequency domain multiplexing submodule:

[0026] Construct a frequency domain multiplexing submodule, each of which consists of a multi-layer perceptron, i.e., two FC layers, a ReLU activation function, and a sigmoid function. l Passed to the shared network to learn frequency domain channel scores for

[0027]

[0028] Where W0, W1 represent the weights of the multi-layer perceptron, σ represents the Sigmoid function; the frequency domain enhancement representation is obtained for:

[0029]

[0030] Where: Indicates channel-wise multiplication.

[0031] Step 13. Design the spatial distribution multiplexing SDM module;

[0032] Given an input feature Z l , use average pooling and maximum pooling operations to aggregate the channel information of each spatial region feature and cascade them to obtain the aggregated feature representation

[0033]

[0034] Then, convolutional filters are applied to the aggregated feature representation Get the regional information aggregation vector and generate the spatial causal score To measure the sensitivity of different spatial regions to capturing causal features, namely:

[0035]

[0036] Finally, the calculation formula for spatial enhancement is as follows:

[0037]

[0038] Step 2: Adjust network parameters and train and test the neural network;

[0039] Step 21. Parameter setting;

[0040] The model is implemented in PyTorch framework. To evaluate the effectiveness of the multi-domain distribution reuse module, it is applied to two benchmarks, each of which includes four prominent baselines.

[0041] Step 22. Input the test data set;

[0042] For the CIFAR10 dataset, 10,000 images from all 10 categories are used for testing; for the CIFAR100 dataset, 10,000 images from all 100 categories are used for testing; for the MiniImageNet dataset, 5,000 images from all 100 categories are used for testing; for the MNIST-360 dataset, 27,000 images from 9 categories are used for testing.

[0043] Further, in step 11, under the sequential benchmark, for the CIFAR10 dataset, which has 100 category images, it is divided into 10 non-overlapping tasks, each task has 10 classes, and then the sub-datasets of each task are sequentially input to the model for incremental image classification training. Under the smooth benchmark, for the MNIST dataset, samples belonging to two consecutive categories at the same time are provided in the form of a stream input to the model for incremental image classification training, and each example of the stream is rotated by an increasing angle, and after a fixed number of steps, the smaller of the two-digit numbers is switched to the lower number.

[0044] Four datasets are used for experiments. The sequential benchmark includes CIFAR10, CIFAR100, and MiniImageNet, and the smooth benchmark includes the MNIST-360 dataset. The CIFAR10 dataset is divided into 5 disjoint tasks, each containing 2 classes. The CIFAR100 and MiniImageNet datasets are divided into 10 disjoint tasks, each containing 10 classes. The MNIST-360 dataset is visited 3 times for nine possible category pairs, with a total of 18 steps in the incremental setting.

[0045] Furthermore, in step 12, the multi-domain distribution multiplexing module is combined with the convolutional network to form a multi-domain distribution multiplexing network including L combination blocks Each combination block consists of two convolution blocks A multi-domain distribution multiplexing module And two ReLU activation functions; the convolution block Including convolutional layers and normalization layers.

[0046]

[0047] Where: Represents a function mapping operation.

[0048] In each batch of tasks t, the training data After L combination blocks and a classifier Get the final output Right now,

[0049]

[0050] Where C, H, and W represent X respectively. 0 The number of channels, height, and width of the

[0051] The corresponding training objective is consistent with the online image incremental classification method, and the final loss is:

[0052]

[0053] Where: is the cross entropy loss, is the distillation loss of the corresponding online image incremental classification method, and α is a hyperparameter.

[0054] The multi-domain distribution multiplexing module adopts a dual-branch network consisting of a frequency domain distribution multiplexing FDM module and a spatial distribution multiplexing SDM module. The frequency domain distribution multiplexing module and the spatial distribution multiplexing module are placed in parallel or sequentially. The input features are enhanced by the parallel frequency domain distribution multiplexing module and the spatial distribution multiplexing module, and then the two features are combined by element-by-element addition to obtain the final output of the multi-domain distribution multiplexing module. For the output of the frequency domain distribution multiplexing module and the spatial distribution multiplexing module The fusion process is achieved by element-by-element addition, namely:

[0055]

[0056] in Represents element-wise addition.

[0057] For each multi-domain distribution multiplexing module, record the following symbols: Z l , is the input and output of each multi-domain distribution multiplexing module in the lth block, that is,

[0058]

[0059] Further, step 21 includes that the sequential benchmarks include DER++, ASER, DVC and PCR. The smooth benchmarks include DER, CLS-ER, CoCa, and MMGCL. All baselines are reproduced in the same environment using the baseline source code and default settings, and each module is applied separately. For all baselines, following their respective data augmentation methods, untrained ResNet18 is used as the backbone, and then the same batch size and replay batch size as in the same benchmark setting are used, and the rest of the settings of the optimizer and hyperparameters are consistent with the original settings of the baseline. Among them, ASER, DVC, and PCR use a streamlined ResNet18 with C=20 as the backbone, and DER, DER++, CLS-ER, CoCa, and MMGCL use a full Res Net18 with C=64. For the sequential benchmark, set the batch size |Bt|=10 and the replay batch size |M|=10. For the smooth benchmark, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, the frequency domain distribution multiplexing and spatial distribution multiplexing modules are organized in parallel, and the 2DDCT basis function of size 8×8 is used in the frequency domain distribution multiplexing module. For the MNIST dataset, the channels are divided into 4 groups, and for the CIFAR10, CIFAR100, and MiniImageNet datasets, the channels are divided into 10 groups.

[0060] Compared with the prior art, the present invention experimentally verifies the above method and evaluates the effect of online incremental image classification on two benchmarks: sequential and smooth. The sequential benchmark refers to the fact that categories are not repeated between different tasks, including three datasets: CIFAR10, CIFAR100, and MiniImageNet. The smooth benchmark refers to the fact that the same categories appear repeatedly between different tasks, including the MNIST-360 dataset. In order to evaluate the effectiveness of the multi-domain distribution reuse module, we apply it to two benchmarks, each of which includes four excellent baselines. The sequential benchmarks include DER++, ASER, DVC, and PCR. The smooth benchmarks include DER, CLS-ER, CoCa, MMGCL, etc. In addition, in order to understand how the multi-domain distribution reuse module captures causal features well, the feature maps of the four datasets are visualized. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Online image incremental classification process

[0062] Figure 2 The overall framework of the multi-domain distribution multiplexing network: It consists of L combination blocks, each of which consists of two convolution blocks, a multi-domain distribution multiplexing module and a ReLU function.

[0063] Figure 3 Multi-domain distribution multiplexing module flow chart: It adopts a plug-and-play dual-stream structure, consisting of a frequency domain distribution multiplexing module and a spatial distribution multiplexing module

[0064] Figure 4 Frequency domain distribution multiplexing network flow chart: frequency domain components are obtained through learnable 2DDCT

[0065] Figure 5 Visualization of the progressive debiasing effect of the multi-domain distribution reuse module; (a) debiasing process on “car” in CIFAR10; (b) debiasing process on “fish” in CIFAR100; (c) debiasing process on “setter” in MiniImageNet; (d) debiasing process on “0” in MNIST360. DETAILED DESCRIPTION

[0066] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0067] The model proposed in this invention takes RGB image data as input, first divides the data set incrementally; then feeds the divided image data into a multi-domain distribution multiplexing image classification network to train the established model parameters; then adds new image data to train the entire model again, repeating this step until no new data can be added; finally, tests the trained model and outputs the online image incremental classification results. The specific steps are as follows:

[0068] Step 1: Establish a multi-domain distribution multiplexing network model

[0069] Step 11. Training data set input and general incremental division

[0070] In order to simulate the real scene of the increasing number of new images on the network, the RGB image data set will be randomly divided into different sub-data sets by category to represent the category image data added at different times. The present invention divides the data into tasks according to the sequential and smooth benchmarks. Under the sequential benchmark, it is necessary to ensure that the categories between different tasks are not repeated, and the smooth benchmark allows the same category to be repeated between different tasks. Specifically, under the sequential benchmark, for the CIFAR10 data set, it has 100 category images, which are divided into 10 non-overlapping tasks, each task has 10 classes, and then the sub-data set of each task will be sequentially input to the model for incremental image classification training. Under the smooth benchmark, for the MNIST data set, samples belonging to two consecutive categories at the same time are provided in the form of a stream to the model for incremental image classification training, such as {0,1}, {1,2}, {2,3}, etc., and each example of the stream is rotated by an increasing angle, and after a fixed number of steps, the smaller number in the two-digit number is switched to the following number.

[0071] The present invention uses four data sets for experiments, including CIFAR10, CIFAR100, and MiniImageNet under the sequential benchmark, and the MNIST-360 data set under the smooth benchmark. The CIFAR10 data set is divided into 5 non-intersecting tasks, each task contains 2 classes. The CIFAR100 and MiniImageNet data sets are divided into 10 non-intersecting tasks, each task contains 10 classes. The MNIST-360 data set is visited 3 times for nine possible category pairs, and the incremental setting is 18 steps in total.

[0072] Step 12. Design a multi-domain distribution multiplexing network;

[0073] The multi-domain distribution multiplexing module (D3M) is a plug-and-play module that can be integrated after each fixed convolutional layer of the existing architecture (such as ResNet18) to form a multi-domain distribution multiplexing network. Given an input image, it first passes through the ResNet18 convolutional layer to obtain a preprocessed feature map. Subsequently, the feature map passes through the multi-domain distribution multiplexing module to produce an updated new feature map, which is causally adaptive. This updated feature map is then propagated to subsequent layers. The multi-domain distribution multiplexing network formed by combining the multi-domain distribution multiplexing module with the convolutional network includes L combination blocks Each combination block consists of two convolution blocks ( Including convolutional layers and normalization layers), a multi-domain distribution multiplexing module And two ReLU activation functions.

[0074]

[0075] Where: ° represents the function mapping operation. In each batch of tasks t, the training data After L combination blocks and a classifier Get the final output Right now,

[0076]

[0077] Where C, H, and W represent X respectively. 0 The number of channels, height, and width of the

[0078] The corresponding training objective is consistent with most online image incremental classification methods (such as DER). The final loss is:

[0079]

[0080] Where: is the cross entropy loss, is the distillation loss of the corresponding online image incremental classification method, such as the distillation loss of DER, and α is a hyperparameter.

[0081] The multi-domain distribution multiplexing module adopts a dual-branch network consisting of a frequency domain distribution multiplexing (FDM) module and a spatial distribution multiplexing (SDM) module. The frequency domain distribution multiplexing module and the spatial distribution multiplexing module can be placed in parallel or in sequence. Our experimental results show that parallel arrangement provides better results than sequential arrangement. Figure 3 As shown in Figure 1, the input features are enhanced by the parallel frequency domain distribution multiplexing module and the spatial distribution multiplexing module, and then the two features are combined by element-by-element addition to obtain the final output of the multi-domain distribution multiplexing module. The fusion process is achieved by element-by-element addition, namely:

[0082]

[0083] in Represents element-wise addition.

[0084] For each multi-domain distribution multiplexing module, we record the following symbols: Z l , is the input and output of each multi-domain distribution multiplexing module in the lth block, that is,

[0085]

[0086] Step 13. Design a frequency domain distribution multiplexing (FDM) module;

[0087] Due to the spurious correlations in the original feature maps, the learned features may be biased and insufficient. Each frequency distribution multiplexing module aims to obtain frequency enhanced representations. In each frequency distribution multiplexing module, the input feature map passes through a frequency distribution submodule and a frequency multiplexing submodule to learn causal scores on different channels, and these scores are used to weight the input feature map to obtain causal enhanced features for multiplexing.

[0088] Step 13.1 frequency domain distribution submodule;

[0089] In the frequency domain distribution submodule, the feature map is divided into different components. We select different frequency transforms (i.e., 2DDCT basis) to simulate various perturbations caused by confounding information while preserving causal information. This intervention encourages the multi-domain distribution multiplexing module to extract features only contained in causal information and discard confounding features.

[0090] like Figure 4 As shown, first, the original feature map after the fixed convolution layer is Divided into J groups along the channel dimension, namely

[0091]

[0092] where J is the number of groups, satisfying J|C', J ≤ H'W". We use different 2DDCT bases (u j ,v j )∈{1…H'}×{1…W'} for each group of channel components Perform frequency transformation to obtain frequency domain components Then connect them together to get the multi-frequency domain components c l :

[0093]

[0094] Step 13.2 Frequency domain multiplexing submodule:

[0095] In order to enhance the causal sufficiency of features, we construct a frequency domain multiplexing submodule to emphasize high-quality components (containing sufficient potential causal information for classification) and reduce the influence of low-quality components (carrying relatively little causal information). Each frequency domain multiplexing submodule consists of a multi-layer perceptron (i.e., two FC layers and a ReLU activation function) and a sigmoid function. l Passed to the shared network to learn frequency domain channel scores for

[0096]

[0097] Where W0, W1 represent the weights of the multilayer perceptron, σ represents the Sigmoid function, and The larger component of corresponds to the high-quality component, and the rest is considered to be the low-quality component. Then, the frequency domain enhanced representation is obtained for:

[0098]

[0099] Where: Indicates channel-wise multiplication.

[0100] Step 13. Design a spatial distribution multiplexing (SDM) module;

[0101] Frequency domain intervention can easily lead to over-adjustment problems, resulting in target deviation and treating other mixed objects as causal information. The spatial distribution multiplexing module aims to guide target debiasing by adaptively adjusting the causal scores of spatial regions, making up for the lack of frequency domain intervention strategies in fully capturing specific target location information in feature maps. The core idea of ​​the spatial distribution multiplexing module is to use convolution operations to divide different spatial regions. Subsequently, each spatial region is evaluated separately for its final prediction impact on a specific category. Regions that emphasize key causal features receive high scores, while regions that do not emphasize key causal features receive low scores. Ultimately, this process allows us to retrieve causally enhanced features for reuse to obtain spatially enhanced representations. Specifically, such as Figure 3 As shown at the bottom, given an input feature Z l , we first aggregate the channel information of each spatial region feature using average pooling and maximum pooling operations and concatenate them to obtain the aggregated feature representation That is to say,

[0102]

[0103]

[0104] Then, convolutional filters are applied to the aggregated feature representation Get the regional information aggregation vector and generate the spatial causal score To measure the sensitivity of different spatial regions to capturing causal features, namely:

[0105]

[0106] Finally, the calculation formula for spatial enhancement is as follows:

[0107]

[0108] Step 2: Adjust network parameters and train and test the neural network;

[0109] Step 21. Parameter setting;

[0110] Our model is implemented in PyTorch and trained on an NVIDIA GeForce RTX3090 with 24G video memory. To evaluate the effectiveness of the multi-domain distribution reuse module, we apply it to two benchmarks, each of which includes four prominent baselines. The sequential benchmarks include DER++, ASER, DVC, and PCR. The smoothing benchmarks include DER, CLS-ER, CoCa, and MMGCL. We reproduce all baselines in the same environment using the baseline source code and default settings, and apply each module separately. For all baselines, we follow their respective data augmentation methods, use untrained ResNet18 as the backbone, and then use the same batch size and replay batch size as in the same benchmark settings. The rest of the settings such as optimizer and hyperparameters are consistent with the original settings of the baseline. Among them, ASER, DVC, and PCR use a reduced ResNet18 with C=20 as the backbone, and DER, DER++, CLS-ER, CoCa, and MMGCL use a full Res Net18 with C=64. For the sequential benchmark, we set the batch size |Bt|=10 and the replay batch size |M|=10. For the smooth benchmark, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, we organize the frequency domain distribution multiplexing and spatial distribution multiplexing modules in a parallel arrangement, and use the 8×8 size 2DDCT basis function in the frequency domain distribution multiplexing module. For the MNIST dataset, the channels are divided into 4 groups, and for the CIFAR10, CIFAR100, MiniImageNet datasets, the channels are divided into 10 groups.

[0111] Step 22. Input the test data set;

[0112] For the CIFAR10 dataset, 10,000 images from all 10 categories are used for testing; for the CIFAR100 dataset, 10,000 images from all 100 categories are used for testing; for the MiniImageNet dataset, 5,000 images from all 100 categories are used for testing; for the MNIST-360 dataset, 27,000 images from 9 categories are used for testing.

[0113] 1. Qualitative evaluation

[0114] (1) Sequential Benchmark - CIFAR10

[0115] Table 1 reports the average accuracy of the model for three different buffer sizes on the CIFAR10 dataset. We can observe that on the DER++ baseline, the frequency domain distribution multiplexing and spatial distribution multiplexing modules consistently improve the performance of the baseline model in all playback situations. In contrast, the spatial distribution multiplexing module achieves higher gains. Since our modules are compatible with each other, we observe significant enhancements when they are all adopted, with accuracies of 3.39%, 6.48%, and 4.81% when the buffer size is 200, 500, and 1000, respectively. On the ASER and DVC baselines, except for the case where the buffer size is 500, the performance improvement of the frequency domain distribution multiplexing module is the largest. The performance is more beneficial when the frequency domain distribution multiplexing and spatial distribution multiplexing modules are used in combination. Specifically, the accuracy of the frequency domain distribution multiplexing module in the case of buffer size equal to 500 is 1.13% and 5.10% higher than that of the ASER and DVC baselines, respectively. On the other hand, the multi-domain distribution multiplexing module provides 3.43% and 4.42% performance improvements for both baselines when the buffer size is 200, and 1.61% and 2.66% performance improvements when the buffer size is 1000. On the PCR baseline, except for the case where the buffer size is 1000, the state-of-the-art scores are achieved when the frequency-domain distribution multiplexing and spatial distribution multiplexing modules are applied together. Specifically, the accuracy rates are 51.88% and 56.10% when the buffer size is 200 and 500, respectively.

[0116] (2) Sequential Benchmark - CIFAR100

[0117] Table 1 reports the average accuracy of the model for three different buffer sizes on the CIFAR100 dataset. Our modules can be used individually or in combination, and show a consistent trend across different choices of modules, improving the performance of all baselines regardless of the size of the replay pool. On the DER++ baseline, the performance is even better when used in combination than when using only a single module, with average accuracies of 16.69%, 22.05%, and 29.97% for buffer sizes of 200, 500, and 1000, respectively. On the ASER baseline, performance improvements of 2.70%, 1.58%, and 1.33% are achieved for buffer sizes of 200, 500, and 1000, respectively. On the DVC baseline, performance improvements of 1.36%, 1.60%, and 1.65% are achieved for buffer sizes of 200, 500, and 1000, respectively. On the PCR baseline, for buffer sizes of 200, 500, and 1000, performance improvements of 0.86%, 1.39%, and 1.81% can be achieved, respectively.

[0118] (3) Sequential Benchmark - MiniImageNet

[0119] Table 1 reports the average accuracy of the models for three different buffer sizes on the MiniImageNet dataset. Except for DER++ when the buffer size is equal to 200, the frequency domain distribution multiplexing and spatial distribution multiplexing modules can generally improve the performance of the baseline model. On DER++, the multi-domain distribution multiplexing module has a small gain effect in the case of smaller buffer sizes, but it brings a huge improvement with the increase of buffer size, and achieves a 4.01% performance improvement when the buffer size is equal to 4k. On the ASER baseline, the effect of using a single module is the best, and the performance improvement can be achieved by 2.22%, 3.04%, and 1.83% for buffer sizes of 200, 500, and 1000, respectively. On the DVC baseline, the performance improvement can be achieved by 2.93%, 3.86%, and 4.10% for buffer sizes of 200, 500, and 1000, respectively. On the PCR baseline, for buffer sizes of 200, 500, and 1000, performance improvements of 0.65%, 0.69%, and 0.57% can be achieved, respectively.

[0120] Table 1 Quantitative results on three sequential benchmarks (average accuracy unit: %, bold font indicates the highest)

[0121]

[0122] (4) Smooth benchmark - MNIST360

[0123] Table 2 reports the average accuracy of the model for three different buffer sizes on the MNIST360 dataset. We can observe that the frequency domain distribution multiplexing and spatial distribution multiplexing modules consistently and significantly improve the performance of the baseline model in all cases, while the frequency domain distribution multiplexing module works better, and in some cases they reach the state-of-the-art scores when applied together. On the DER baseline, for buffer sizes of 200, 500, and 1000, performance improvements of 13.51%, 18.95%, and 15.86% can be achieved, respectively. On the CLS-ER baseline, for a buffer size of 200, the combination of the two modules works best, and the multi-domain distribution multiplexing module can achieve an 11.48% performance improvement. For buffer sizes of 500 and 1000, the frequency domain distribution multiplexing module works best alone, achieving 17.73% and 18.45% performance improvements, respectively. On the CoCa baseline, the combination of the two modules works best, and for buffer sizes of 200, 500, and 1000, performance improvements of 17.87%, 14.50%, and 12.48% can be achieved, respectively. On the OCDNet baseline, for buffer sizes of 200 and 500, the combination of the two modules works best, achieving 15.91% and 12.85% performance improvements, respectively. For a buffer size of 1000, the frequency domain distribution multiplexing module works best alone, achieving 11.06% performance improvement.

[0124] Table 2 Quantitative results on the smoothing benchmark (average accuracy unit: %, bold font indicates the highest)

[0125]

[0126] 2. Subjective effect

[0127] Figure 5 The feature maps of the four datasets on different benchmarks are visualized to better demonstrate the effect of the multi-domain distribution reuse module in capturing causal features in online image incremental classification, including the baseline model, + frequency domain distribution reuse module, + spatial distribution reuse module, and + multi-domain distribution reuse module. For each benchmark, we selected an image and visualized the feature maps of the same channel of the first layer of each model in four stages. These four stages can vividly show the progressive debiasing effect of the multi-domain distribution reuse module. Figure 5(a) shows the feature map of the "car" at the beginning of task 1, the end of task 1, the end of task 3, and the end of task 5 (the total number of tasks is 5). As the training progresses, the baseline tends to focus on the offset features due to spurious correlations (the top sky of the image). In contrast, the addition of the multi-domain distribution multiplexing module effectively strengthens the focus of the feature map on the "car" part. Similarly, according to Figure 5 (b)5(d), compared with other settings, the model with the multi-domain distribution reuse module focuses on the target object region and does not produce significant changes during incremental learning. In other words, the multi-domain distribution reuse module effectively utilizes information from the target object region and aggregates causal features, thereby improving model generalization and alleviating catastrophic forgetting.

Claims

1. An online image incremental classification method based on a multi-domain distribution multiplexing network, characterized in that: Taking RGB image data as input, the data set is incrementally divided; the divided image data is then sent to the multi-domain distribution multiplexing image classification network to train the established model parameters; Then, new image data is added to train the entire model again, and this step is repeated until no new data can be added; finally, the trained model is tested and the online image incremental classification results are output.

2. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 1 is characterized in that: The steps include: Step 1: Establish a multi-domain distribution multiplexing network model; Step 11. Training data set input and general increment division; In order to simulate the real scene of the continuous increase of new images on the Internet, the RGB image dataset will be randomly divided into different sub-datasets by category to represent the category image data added at different times, and the data will be divided into tasks according to the sequential and smooth benchmarks. The sequential benchmark must ensure that the categories between different tasks are not repeated, while the smooth benchmark allows the same category to be repeated between different tasks. Step 12. Design a multi-domain distribution multiplexing network; The multi-domain distribution multiplexing module D3M is a plug-and-play module. Given an input image, it first passes through the ResNet18 convolutional layer to obtain a preprocessed feature map. Subsequently, the feature map passes through the multi-domain distribution multiplexing module to generate an updated new feature map, which is causally adaptive. This updated feature map is then propagated to subsequent layers. Step 13. Design frequency domain distribution multiplexing FDM module; Step 13.1 frequency domain distribution submodule; In the frequency domain distribution submodule, the feature map is divided into different components. First, the original feature map after the fixed convolution layer is Along the channel dimension, it is divided into J Group, that is Where J is the number of groups, satisfying J|C′, J≤H′W′′. Using different 2DDCT bases (u j , v j )∈{1…H′}×{1…W′} for each group of channel components Perform frequency transformation to obtain frequency domain components Then connect them together to get the multi-frequency domain components c l : Step 13.2 Frequency domain multiplexing submodule: Construct a frequency domain multiplexing submodule, each of which consists of a multi-layer perceptron, i.e., two FC layers, a ReLU activation function, and a sigmoid function. l Passed to the shared network to learn frequency domain channel scores for Where W0, W1 represent the weights of the multi-layer perceptron, and σ represents the Sigmoid function; Get frequency domain enhanced representation for: Where: Indicates channel-wise multiplication. Step 13. Design the spatial distribution multiplexing SDM module; Given an input feature Z l , use average pooling and maximum pooling operations to aggregate the channel information of each spatial region feature and cascade them to obtain the aggregated feature representation Then, convolutional filters are applied to the aggregated feature representation Get the regional information aggregation vector and generate the spatial causal score To measure the sensitivity of different spatial regions to capturing causal features, namely: Finally, the calculation formula for spatial enhancement is as follows: Step 2: Adjust network parameters and train and test the neural network; Step 21. Parameter setting; The model is implemented in PyTorch framework. To evaluate the effectiveness of the multi-domain distribution reuse module, it is applied to two benchmarks, each of which includes four prominent baselines. Step 22. Input the test data set; For the CIFAR10 dataset, 10,000 images from all 10 categories are used for testing; For the CIFAR100 dataset, 10,000 images from all 100 categories are used for testing; for the MiniImageNet dataset, 5,000 images from all 100 categories are used for testing; for the MNIST-360 dataset, 27,000 images from 9 categories are used for testing.

3. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 2 is characterized in that: In step 11, under the sequential benchmark, for the CIFAR10 dataset, which has 100 category images, it is divided into 10 non-overlapping tasks, each task has 10 classes, and then the sub-datasets of each task are sequentially input to the model for incremental image classification training. Under the smooth benchmark, for the MNIST dataset, samples belonging to two consecutive categories at the same time are provided in the form of a stream to the model for incremental image classification training, and each example of the stream is rotated by an increasing angle, and after a fixed number of steps, the smaller of the two-digit numbers is switched to the lower number. Four datasets are used for experiments. The sequential benchmark includes CIFAR10, CIFAR100, and MiniImageNet, and the smooth benchmark includes the MNIST-360 dataset. The CIFAR10 dataset is divided into 5 disjoint tasks, each containing 2 classes. The CIFAR100 and MiniImageNet datasets are divided into 10 disjoint tasks, each containing 10 classes. The MNIST-360 dataset is visited 3 times for nine possible category pairs, with a total of 18 steps in the incremental setting.

4. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 2 is characterized in that: In step 12, the multi-domain distribution multiplexing module is combined with the convolutional network to form a multi-domain distribution multiplexing network including L combination blocks Each combination block consists of two convolution blocks A multi-domain distribution multiplexing module And two ReLU activation functions; the convolution block Including convolutional layers and normalization layers. Where: Represents a function mapping operation. In each batch of tasks t, the training data After L combination blocks and a classifier Get the final output Right now, Where C, H, and W represent X respectively. 0 The number of channels, height, and width of the The corresponding training objective is consistent with the online image incremental classification method, and the final loss is: Where: is the cross entropy loss, is the distillation loss of the corresponding online image incremental classification method, and α is a hyperparameter. The multi-domain distribution multiplexing module adopts a dual-branch network consisting of a frequency domain distribution multiplexing FDM module and a spatial distribution multiplexing SDM module. The frequency domain distribution multiplexing module and the spatial distribution multiplexing module are placed in parallel or sequentially. The input features are enhanced by the parallel frequency domain distribution multiplexing module and the spatial distribution multiplexing module, and then the two features are combined by element-by-element addition to obtain the final output of the multi-domain distribution multiplexing module. For the output of the frequency domain distribution multiplexing module and the spatial distribution multiplexing module The fusion process is achieved by element-by-element addition, that is: in Represents element-wise addition. For each multi-domain distribution multiplexing module, remember the following symbols: is the input and output of each multi-domain distribution multiplexing module in the lth block, that is, 5. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 2 is characterized in that: Step 21 includes, sequential benchmarks include DER++, ASER, DVC and PCR. Smooth benchmarks include DER, CLS-ER, CoCa, MMGCL. All baselines are reproduced in the same environment using the baseline source code and default settings, and each module is applied separately. For all baselines, follow their respective data augmentation methods, use untrained ResNet18 as the backbone, and then use the same batch size and replay batch size as in the same benchmark setting, and the optimizer, hyperparameters and other settings are consistent with the original settings of the baseline. ASER, DVC, and PCR use a streamlined ResNet18 with C=20 as the backbone, and DER, DER++, CLS-ER, CoCa, and MMGCL use a full Res Net18 with C=64. For the sequential benchmark, the batch size |Bt|=10 and the replay batch size |M|=10 are set. For the smooth benchmark, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, the frequency domain distribution multiplexing and spatial distribution multiplexing modules are organized in a parallel arrangement, and the 2DDCT basis function of size 8×8 is used in the frequency domain distribution multiplexing module. For the MNIST dataset, the channels are divided into 4 groups, and for the CIFAR10, CIFAR100, and MiniImageNet datasets, the channels are divided into 10 groups.

Images