Online image incremental classification method based on multi-domain distribution multiplexing network

Abstract

The application discloses an online image incremental classification method based on a multi-domain distribution multiplexing network. The online image incremental classification method takes RGB image data as input, divides the data set incrementally, sends the divided image data into a multi-domain distribution multiplexing image classification network, trains the established model parameters, then increases new image data to train the whole model again, repeats the step until there is no new data to be added, finally tests the trained model, and outputs the online image incremental classification result. The application can effectively make up for the shortcomings of the traditional image incremental classification method. Meanwhile, a frequency domain distribution multiplexing module is designed, a mixed factor of different high and low frequency combinations is introduced, and orthogonal characteristic aggregation information of different channels is obtained to highlight the causal correlation characteristics. At each iteration, the causal biased characteristics are implicitly converted into unbiased characteristics, thereby relieving the feature distribution deviation between the training and the memory buffer data.

Description

Technical Field

[0001] This invention is an online incremental image classification method based on a multi-domain distribution multiplexing network, belonging to the field of computer vision technology. Background Technology

[0002] Modern artificial intelligence models are expected to possess the ability to continuously learn, similar to humans, making incremental learning a hot research topic. Incremental learning primarily explores the problem of learning from data streams under non-stationary data distributions, where data originates from various tasks and the input domain is constantly evolving. Traditionally, incremental learning includes three paradigms: task incremental learning (incrementally learning a set of tasks, with task IDs visible during testing), domain incremental learning (learning the same type of problem in different domains), and class incremental learning (incrementally learning a set of tasks, with task IDs not visible during testing). The primary challenge facing incremental learning is catastrophic forgetting, where historical knowledge in old data is replaced by new knowledge. In recent years, thanks to the rapid development of deep learning, numerous image incremental classification methods have been proposed, generally falling into three categories: memory replay-based methods, regularization-based methods, and parameter isolation-based methods.

[0003] However, the above-mentioned incremental image classification methods lack flexibility in practical applications because real-world data usually arrives in the form of online streams and lacks task identifiers. Summary of the Invention

[0004] The technical objective of this invention is to propose a more general incremental learning setup, namely an online image incremental classification method, where data arrives gradually in a streaming manner, and training is performed whenever a small batch of training samples accumulates. This method requires the model to classify images of all seen categories at each stage. However, due to the rapid pace of the data stream, the learning algorithm can only train once per data batch, and task identifiers are unavailable. Formally, let T be the total number of tasks, and the online image incremental classification model processes a continuous sequence of tasks derived from a single data stream. in Let represent the training data for task t, where t = 1, 2, ..., T. Online incremental image classification allows... Furthermore, the task boundaries must be unknown throughout the training and testing phases. Training data for each task t arrives gradually in a streaming manner, with small batches of task t accumulated from the data stream each time. Training is performed in a single iteration. After reviewing all the data for one task, the next task begins. Online image incremental classification faces more severe catastrophic forgetting, primarily due to inter-task bias and intra-task bias.

[0005] (1) Inter-task bias: Due to online learning and unknown task boundaries during the online image incremental classification training process, inter-task bias is generated and is more difficult to eliminate than in traditional image incremental classification.

[0006] (2) Intra-task bias: Since the data arrives in the form of a stream, each batch involves only a small amount of data, resulting in a huge intra-task bias caused by model overfitting.

[0007] Many studies have mitigated inter-class bias by balancing inter-class relationships or learning representative features of all classes. CoCa addresses the relationship bias problem in online image incremental classification through collaborative distillation and leverages predicate tasks and supervised contrastive learning to learn complete and distinctive features of all classes via a collaborative self-supervised network. CBA addresses the posterior probability shift problem in online image incremental classification by proposing a two-layer learning framework with a continuous bias adapter to enhance the classifier network to adapt to catastrophic distribution changes during training, enabling the classifier network to learn stable integrations of previously learned tasks. Despite these methods achieving good performance, most neglect the intra-task bias problem in online image incremental classification.

[0008] This invention aims to simultaneously address the inter-task bias and intra-task bias issues in online incremental image classification. However, a major challenge lies in the fact that streaming data can easily trap models in spurious correlations. Therefore, resisting spurious correlations and reducing feature bias in streaming data becomes crucial. Some researchers have proposed counterfactual data augmentation and causal intervention methods from a causal perspective to combat spurious correlations. Specifically, counterfactual data augmentation eliminates spurious correlations by constructing additional counterfactual data for training. CGN, leveraging the concept of causal independence, constructs a counterfactual image generation model by artificially separating the image generation process into separate mechanisms for background, shape, and texture. By incorporating generated fake images into the training, it improves the robustness of the image classification model. Causal effect calibration mitigates bias by adjusting the effects of confounding variables. CONTA utilizes a structural causal relationship model to deeply analyze the causal relationships between image features, context, and class labels, achieving state-of-the-art performance in weakly supervised semantic segmentation tasks through context adjustment. However, both of these causal inference methods require a deep understanding of potential confounding factors, which are difficult to identify in online incremental image classification. For example, in offline mode, CafeBoost introduces the concept of task-induced bias. By introducing class features from previous tasks into new class features in the current task, it uses backdoor adjustment to mitigate confounding factors and reduce the model's focus on biased features. However, both methods require some understanding of confounding factors. Since the data enters in a streaming manner, the model cannot iterate repeatedly on a single class of data to obtain stable sources of spurious correlations, thus making it impossible to effectively intervene in confounding factors.

[0009] Therefore, this invention does not explicitly model confounding factors, but instead attempts to perturb them while keeping causal factors constant. Some studies have shown that frequency representation contains rich image understanding patterns, and frequency changes can be used to separate entangled feature information in an image. Inspired by these works, we propose a pseudo-causal intervention strategy based on frequency transformation, intervening in two stages: intervention on confounding factors and intervention on causal factors, combining the two to form a Frequency Division Multiplexer (FDM) to resist spurious correlations. Furthermore, we find that relying solely on frequency domain features cannot adequately emphasize causal features, especially for images with rich semantics. Therefore, a Spatial Division Multiplexer (SDM) is introduced to emphasize the representational power of different target regions in the spatial domain, as a supplement to the former. Thus, this invention proposes a Dual-Domain Division Multiplexer (D3M) as a plug-and-play module for online incremental image classification, integrated with convolutional networks to form a Dual-Domain Division Multiplexer network. The multi-domain distribution and reuse module partitions and reuses features in the frequency and spatial domains, effectively mitigating the impact of spurious correlations and helping to learn invariant causal features to reduce catastrophic forgetting in online image incremental classification.

[0010] This invention aims to address technical problems in online incremental image classification, primarily including simultaneously handling inter-task bias and intra-task bias, and how to utilize causal intervention methods to resist spurious correlations, thereby mitigating catastrophic forgetting and improving classification accuracy. This invention proposes a multi-domain distribution and reuse module, composed of a frequency domain distribution and reuse module and a spatial distribution and reuse module, used to intervene in confounding factors and multiple causal factors in the frequency and spatial domains. The frequency domain distribution and reuse module divides and reuses features of different frequency components, prioritizing causal features relevant to the target through a two-stage spurious causal intervention strategy. The first intervention, called frequency distribution, selects different frequency transformations to represent the image and adaptively adjusts the contributions of different frequencies to perturb different parts of the confounding factors. The second intervention, called frequency reuse, eliminates dependencies between features based on the first intervention. The spatial distribution and reuse module performs similar work in the spatial domain, serving as a complement to the frequency domain distribution and reuse module. As a lightweight model enhancement module, the multi-domain distribution reuse module can be seamlessly integrated into most current online image incremental classification methods to form a multi-domain distribution reuse network. This effectively solves problems such as inter-task bias, intra-task bias, and spurious correlation, thereby improving the accuracy and robustness of online image incremental classification.

[0011] This invention proposes an online image incremental classification method based on a multi-domain distribution multiplexing network, which effectively overcomes the shortcomings of traditional image incremental classification methods. The overall framework of the multi-domain distribution multiplexing network introduces a multi-domain distribution multiplexing unit into a traditional convolutional network, applying frequency and spatial modules in parallel. It learns to emphasize or suppress certain information to eliminate confounding factors and highlight features causally related to the target. Specifically, this invention designs a frequency domain distribution multiplexing module that introduces confounding factors of different high- and low-frequency combinations to obtain orthogonal feature aggregation information from different channels, thereby highlighting causally related features. Simultaneously, the spatial distribution multiplexing module focuses on capturing aggregation information from different target regions in the feature map. Essentially, the proposed method implicitly transforms causally biased features into unbiased features at each iteration, thereby mitigating the feature distribution offset between training and memory buffer data.

[0012] This invention is achieved through the following technical solution:

[0013] This online incremental image classification method, based on a multi-domain distribution and reuse network, takes RGB image data as input and incrementally partitions the dataset. The partitioned image data is then fed into a multi-domain distribution and reuse image classification network to train the model parameters. Next, new image data is added, and the entire model is trained again, repeating this process until no more new data can be added. Finally, the trained model is tested, and the online incremental image classification results are output. The specific steps are as follows:

[0014] Step 1: Establish a multi-domain distribution and multiplexing network model

[0015] Step 11. Input the training dataset and perform a general incremental partition;

[0016] To simulate the real-world scenario of a continuous increase in new images on the internet, the RGB image dataset will be randomly divided into different subsets according to category to represent the category image data added at different times. The data will be divided into tasks using both sequential and smoothed baselines. Under the sequential baseline, it is required that categories not overlap between different tasks, while the smoothed baseline allows the same category to appear repeatedly across different tasks.

[0017] Step 12. Design a multi-domain distribution and multiplexing network;

[0018] The Multi-Domain Distribution Multiplexing (D3M) module is a plug-and-play module. Given an input image, it first passes through ResNet18 convolutional layers to obtain a preprocessed feature map. This feature map is then passed through the D3M module to generate an updated, causally adaptive feature map. This updated feature map is then propagated to subsequent layers.

[0019] Step 13. Design the frequency domain distribution multiplexing (FDM) module;

[0020] Step 13.1 Frequency Domain Distribution Submodule;

[0021] In the frequency domain distribution submodule, the feature map is divided into different components. First, the original feature map after fixing the convolutional layers is... Divided into J groups along the channel dimension, i.e.

[0022]

[0023] in It is a set of groups that satisfy J|C', J≤H'W". Using different 2DDCT bases (u j ,v j For each group of channel components, )∈{1…H'}×{1…W'} Perform frequency transformation to obtain frequency domain components. Then they are concatenated to obtain the multi-frequency domain component c. l :

[0024]

[0025] Step 13.2 Frequency Domain Multiplexing Submodule:

[0026] Construct a frequency domain multiplexing submodule. Each frequency domain multiplexing submodule consists of a multilayer perceptron (i.e., two fully connected (FC) layers), a ReLU activation function, and a sigmoid function. Represent the multi-frequency domain components c... l The data is transmitted to the shared network to learn the frequency domain channel scores. for

[0027]

[0028] Where W0 and W1 represent the weights of the multilayer perceptron, and σ represents the sigmoid function; thus, the frequency domain enhancement representation is obtained. for:

[0029]

[0030] In the formula: This indicates multiplication by channel.

[0031] Step 13. Design the Spatial Distribution Multiplexing (SDM) module;

[0032] Given an input feature Z l The channel information of each spatial region feature is aggregated using average pooling and max pooling operations, and then concatenated to obtain the aggregated feature representation.

[0033]

[0034] Then, the convolutional filter is applied to the aggregated feature representation. Obtain the regional information aggregation vector and generate spatial causal scores. To measure the sensitivity of different spatial regions to capturing causal features, namely:

[0035]

[0036] Finally, the formula for calculating the spatial augmentation representation is as follows:

[0037]

[0038] Step 2: Adjust network parameters and train and test the neural network;

[0039] Step 21. Parameter settings;

[0040] The model is implemented in the PyTorch framework. To evaluate the effectiveness of the multi-domain distribution reuse module, it is applied to two benchmarks, each of which includes four prominent baselines.

[0041] Step 22. Input the test dataset;

[0042] For the CIFAR10 dataset, 10,000 images across all 10 classes were used for testing; for the CIFAR100 dataset, 10,000 images across all 100 classes were used for testing; for the MiniImageNet dataset, 5,000 images across all 100 classes were used for testing; and for the MNIST-360 dataset, 27,000 images across 9 classes were used for testing.

[0043] Further, in step 11, under the sequential benchmark, for the CIFAR10 dataset, which has 100 categories of images, it is divided into 10 non-overlapping tasks, each with 10 classes. Sub-datasets from each task are then sequentially input into the model for incremental image classification training. Under the smooth benchmark, for the MNIST dataset, samples belonging to two consecutive classes at the same time are provided to the model in a streaming manner for incremental image classification training. Each example in the stream is rotated by an increasing angle, and after a fixed number of steps, the smaller of the two digits is switched to the next lower digit.

[0044] Experiments were conducted using four datasets: the sequential benchmark included CIFAR10, CIFAR100, and MiniImageNet; the smooth benchmark included the MNIST-360 dataset. The CIFAR10 dataset was divided into five disjoint tasks, each containing two classes. The CIFAR100 and MiniImageNet datasets were divided into ten disjoint tasks, each containing ten classes. The MNIST-360 dataset was accessed three times, with nine possible class pairs, and the incremental setup consisted of 18 steps.

[0045] Furthermore, in step 12, the multi-domain distribution multiplexing network formed by combining the multi-domain distribution multiplexing module with the convolutional network includes L combination blocks. Each combined block consists of two convolutional blocks. A multi-domain distribution reuse module The convolutional block consists of two ReLU activation functions. It includes convolutional layers and normalization layers.

[0046]

[0047] In the formula: This represents function mapping operations.

[0048] In each batch of tasks t, the training data After L combination blocks and a classifier Get the final output Right now,

[0049]

[0050] Where C, H, and W represent X respectively. 0 The number of channels, height, and width.

[0051] The corresponding training objective is consistent with the online image incremental classification method, and the final loss is:

[0052]

[0053] In the formula: For cross-entropy loss, Let α be the distillation loss of the corresponding online image incremental classification method, and α be a hyperparameter.

[0054] The multi-domain distribution multiplexing module employs a dual-branch network consisting of a frequency-domain distribution multiplexing (FDM) module and a spatial distribution multiplexing (SDM) module, which are placed in parallel or sequentially. Input features are enhanced representations by the parallel frequency-domain and spatial distribution multiplexing modules respectively, and then the two features are combined using an element-wise addition operation to obtain the final output of the multi-domain distribution multiplexing module. The outputs of the frequency-domain and spatial distribution multiplexing modules are then analyzed. The fusion process is achieved through element-wise addition, that is:

[0055]

[0056] in This represents element-wise addition.

[0057] For each multi-domain distribution multiplexing module, denote the following symbol: Z l , Distribute the inputs and outputs of the multiplexing module to each multi-domain component in the l-th block, i.e.

[0058]

[0059] Further, step 21 includes the following: sequential benchmarks include DER++, ASER, DVC, and PCR. Smoothing benchmarks include DER, CLS-ER, CoCa, and MMGCL. All baselines are reproduced in the same environment using the baseline source code and default settings, with each module applied individually. For all baselines, following their respective data augmentation methods, an untrained ResNet18 is used as the backbone, followed by the same batch size and replay batch size as in the same benchmark settings. The optimizer and other hyperparameter settings remain consistent with the original baseline settings. Specifically, ASER, DVC, and PCR use a simplified ResNet18 with C=20 as the backbone, while DER, DER++, CLS-ER, CoCa, and MMGCL use a full ResNet18 with C=64. For sequential benchmarks, the batch size |Bt|=10 and the replay batch size |M|=10 are set. For smoothing benchmarks, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, the frequency domain distribution multiplexing and spatial distribution multiplexing modules are organized in parallel. In the frequency domain distribution multiplexing module, 8×8 2DDCT basis functions are used. For the MNIST dataset, the channels are divided into 4 groups. For the CIFAR10, CIFAR100 and MiniImageNet datasets, the channels are divided into 10 groups.

[0060] Compared with existing technologies, this invention experimentally validates the above-mentioned method, evaluating the performance of online incremental image classification on two benchmarks: sequential and smoothed. Sequential benchmarks refer to datasets where classes do not overlap across different tasks, including CIFAR10, CIFAR100, and MiniImageNet. Smoothed benchmarks refer to datasets where the same class appears repeatedly across different tasks, including the MNIST-360 dataset. To evaluate the effectiveness of the multi-domain distribution multiplexing module, we apply it to two benchmarks, each including four excellent baselines. Sequential benchmarks include DER++, ASER, DVC, and PCR. Smoothed benchmarks include DER, CLS-ER, CoCa, and MMGCL. Furthermore, to understand how well the multi-domain distribution multiplexing module captures causal features, feature maps from the four datasets are visualized. Attached Figure Description

[0061] Figure 1 Online image incremental classification process

[0062] Figure 2 Overall framework diagram of the multi-domain distribution multiplexing network: It consists of L combination blocks, each combination block consisting of two convolutional blocks, a multi-domain distribution multiplexing module, and a ReLU function.

[0063] Figure 3 Multi-domain distribution multiplexing module flowchart: It adopts a plug-and-play dual-stream structure and consists of a frequency domain distribution multiplexing module and a spatial distribution multiplexing module.

[0064] Figure 4 Frequency domain distribution multiplexing network flowchart: Frequency domain components are obtained through learnable 2DDCT.

[0065] Figure 5 Visualization of the progressive biasing effect of the multi-domain distribution reuse module; (a) the biasing process on "car" in CIFAR10; (b) the biasing process on "fish" in CIFAR100; (c) the biasing process on "setter" in MiniImageNet; (d) the biasing process on "0" in MNIST360. Detailed Implementation

[0066] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0067] The model proposed in this invention takes RGB image data as input. First, it incrementally partitions the dataset. Then, it feeds the partitioned image data into a multi-domain distribution and reuse image classification network to train the established model parameters. Next, it adds new image data and trains the entire model again, repeating this process until no more new data can be added. Finally, it tests the trained model and outputs online incremental image classification results. The specific steps are as follows:

[0068] Step 1: Establish a multi-domain distribution and multiplexing network model

[0069] Step 11. Input of training dataset and general incremental partitioning

[0070] To simulate the real-world scenario of a continuous increase in new images on the internet, the RGB image dataset is randomly divided into different subsets according to category to represent the category image data added at different times. This invention uses sequential and smoothing benchmarks to perform task partitioning on the data. Under the sequential benchmark, it is necessary to ensure that the categories do not overlap between different tasks, while the smoothing benchmark allows the same category to appear repeatedly between different tasks. Specifically, under the sequential benchmark, for the CIFAR10 dataset, which has 100 categories of images, it is divided into 10 non-overlapping tasks, each with 10 classes. Then, the subset of each task is sequentially input into the model for incremental image classification training. Under the smoothing benchmark, for the MNIST dataset, samples belonging to two consecutive classes at the same time are provided to the model in the form of a stream for incremental image classification training, such as {0,1}, {1,2}, {2,3}, etc., and each example in the stream is rotated by an increasing angle. After a fixed number of steps, the smaller digit of the two-digit number is switched to the next lower digit.

[0071] This invention employed four datasets for experiments: the sequential benchmark included CIFAR10, CIFAR100, and MiniImageNet; the smooth benchmark included the MNIST-360 dataset. The CIFAR10 dataset was divided into five disjoint tasks, each containing two classes. The CIFAR100 and MiniImageNet datasets were divided into ten disjoint tasks, each containing ten classes. The MNIST-360 dataset was accessed three times, representing nine possible class pairs, with an incremental setup of 18 steps.

[0072] Step 12. Design a multi-domain distribution and multiplexing network;

[0073] The Multi-Domain Distribution Multiplexing Module (D3M) is a plug-and-play module that can be integrated after each fixed convolutional layer in an existing architecture (such as ResNet18) to form a Multi-Domain Distribution Multiplexing Network. Given an input image, it is first passed through ResNet18 convolutional layers to obtain a preprocessed feature map. This feature map is then passed through the Multi-Domain Distribution Multiplexing Module to generate an updated, causally adaptive feature map. This updated feature map is then propagated to subsequent layers. The Multi-Domain Distribution Multiplexing Network, formed by combining the Multi-Domain Distribution Multiplexing Module with a convolutional network, consists of L combination blocks. Each combined block consists of two convolutional blocks ( Including convolutional layers and normalization layers, a multi-domain distribution multiplexing module It consists of two ReLU activation functions.

[0074]

[0075] In the formula: ° represents the function mapping operation. In each batch of tasks t, the training data After L combination blocks and a classifier Get the final output Right now,

[0076]

[0077] Where C, H, and W represent X respectively. 0 The number of channels, height, and width.

[0078] The corresponding training objective is consistent with most online incremental image classification methods (e.g., DER). The final loss is:

[0079]

[0080] In the formula: For cross-entropy loss, α represents the distillation loss of the corresponding online image incremental classification method, such as the distillation loss of DER, where α is a hyperparameter.

[0081] The multi-domain distribution multiplexing module employs a dual-branch network consisting of a frequency domain distribution multiplexing (FDM) module and a spatial distribution multiplexing (SDM) module. These modules can be placed in parallel or sequentially. Our experimental results show that the parallel arrangement provides better results than the sequential arrangement. Figure 3 As shown, the input features are enhanced through parallel frequency domain distribution multiplexing modules and spatial distribution multiplexing modules, respectively. Then, element-wise addition is used to combine the two types of features to obtain the final output of the multi-domain distribution multiplexing module. Formally, the outputs of the frequency domain distribution multiplexing module and the spatial distribution multiplexing module... The fusion process is achieved through element-wise addition, that is:

[0082]

[0083] in This represents element-wise addition.

[0084] For each multi-domain distribution multiplexing module, we denote the following symbol: Z l , Distribute the inputs and outputs of the multiplexing module to each multi-domain component in the l-th block, i.e.

[0085]

[0086] Step 13. Design the Frequency Domain Distribution Multiplexing (FDM) module;

[0087] Because the original feature maps contain spurious correlations, the learned features may be biased and insufficient. Each frequency domain distribution multiplexing module aims to obtain a frequency-enhanced representation. Within each frequency domain distribution multiplexing module, the input feature map learns causal scores on different channels through a frequency domain distribution submodule and a frequency domain multiplexing submodule, and these scores are used to weight the input feature map to obtain causal-enhanced features for multiplexing.

[0088] Step 13.1 Frequency Domain Distribution Submodule;

[0089] In the frequency domain distribution submodule, the feature map is divided into different components. We select different frequency transforms (i.e., 2DDCT basis) to simulate various perturbations caused by profanity while preserving causal information. This intervention encourages the multi-domain distribution multiplexing module to extract features contained only in causal information, discarding profanity features.

[0090] like Figure 4 As shown, firstly, the original feature map after fixing the convolutional layer is... Divided into J groups along the channel dimension, i.e.

[0091]

[0092] Where J is the number of groups, satisfying J|C', J≤H'W". We use different 2DDCT bases (u j ,v j For each group of channel components, )∈{1…H'}×{1…W'} Perform frequency transformation to obtain frequency domain components. Then they are concatenated to obtain the multi-frequency domain component c. l :

[0093]

[0094] Step 13.2 Frequency Domain Multiplexing Submodule:

[0095] To enhance the causal sufficiency of features, we construct a frequency domain multiplexing submodule to emphasize high-quality components (containing sufficient latent causal information for classification) and reduce the impact of low-quality components (carrying relatively little causal information). Each frequency domain multiplexing submodule consists of a multilayer perceptron (i.e., two fully connected layers and a ReLU activation function) and a sigmoid function. The multi-frequency domain representation components c... l The data is transmitted to the shared network to learn the frequency domain channel scores. for

[0096]

[0097] Where W0 and W1 represent the weights of the multilayer perceptron, σ represents the sigmoid function, and The larger components correspond to high-quality components, while the rest are considered low-quality components. Then, the frequency domain enhancement representation is obtained. for:

[0098]

[0099] In the formula: This indicates multiplication by channel.

[0100] Step 13. Design the Spatial Distribution Multiplexing (SDM) module;

[0101] Frequency domain intervention is prone to over-adjustment, leading to target bias and the misinterpretation of other cluttered objects as causal information. The spatial distribution multiplexing module aims to guide target de-biasing by adaptively adjusting the causal scores of spatial regions, compensating for the shortcomings of frequency domain intervention strategies in fully capturing specific target location information in feature maps. The core idea of ​​the spatial distribution multiplexing module is to use convolutional operations to divide different spatial regions. Subsequently, each spatial region is individually evaluated for its impact on the final prediction of a specific category. Regions emphasizing key causal features receive high scores, while regions not emphasizing key causal features receive low scores. Ultimately, this process allows us to retrieve causally enhanced features for reuse, resulting in a spatially enhanced representation. Specifically, such as... Figure 3 As shown at the bottom, given an input feature Z l We first aggregate the channel information of each spatial region feature using average pooling and max pooling operations, and then concatenate them to obtain the aggregated feature representation. In other words,

[0102]

[0103]

[0104] Then, the convolutional filter is applied to the aggregated feature representation. Obtain the regional information aggregation vector and generate spatial causal scores. To measure the sensitivity of different spatial regions to capturing causal features, namely:

[0105]

[0106] Finally, the formula for calculating the spatial augmentation representation is as follows:

[0107]

[0108] Step 2: Adjust network parameters and train and test the neural network;

[0109] Step 21. Parameter settings;

[0110] Our model was implemented in the PyTorch framework and trained on an NVIDIA GeForce RTX 3090 with 24GB of VRAM. To evaluate the effectiveness of the multi-domain distribution multiplexing module, we applied it to two benchmarks, each consisting of four prominent baselines. The sequential benchmarks included DER++, ASER, DVC, and PCR. The smoothed benchmarks included DER, CLS-ER, CoCa, and MMGCL. We reproduced all baselines in the same environment using the baseline source code and default settings, applying each module individually. For all baselines, we followed their respective data augmentation methods, using an untrained ResNet18 as the backbone, and then using the same batch size and replay batch size as in the same benchmark settings, with the optimizer, hyperparameters, and other settings consistent with the original baseline settings. Specifically, ASER, DVC, and PCR used a simplified ResNet18 with C=20 as the backbone, while DER, DER++, CLS-ER, CoCa, and MMGCL used a full ResNet18 with C=64. For sequential benchmarks, we set the batch size |Bt| = 10 and the replay batch size |M| = 10. For smooth benchmarks, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, we organize the frequency domain distribution multiplexing and spatial distribution multiplexing modules in parallel, and use 8×8 2DDCT basis functions in the frequency domain distribution multiplexing module. For the MNIST dataset, the channels are divided into 4 groups, and for the CIFAR10, CIFAR100, and MiniImageNet datasets, the channels are divided into 10 groups.

[0111] Step 22. Input the test dataset;

[0112] For the CIFAR10 dataset, 10,000 images across all 10 classes were used for testing; for the CIFAR100 dataset, 10,000 images across all 100 classes were used for testing; for the MiniImageNet dataset, 5,000 images across all 100 classes were used for testing; and for the MNIST-360 dataset, 27,000 images across 9 classes were used for testing.

[0113] 1. Qualitative assessment

[0114] (1) Sequence Standard - CIFAR10

[0115] Table 1 reports the average accuracy of the model on the CIFAR10 dataset with three different buffer sizes. We can observe that on the DER++ baseline, both the frequency domain distribution multiplexing and spatial distribution multiplexing modules consistently improve the performance of the baseline model across all playback scenarios. The spatial distribution multiplexing module achieves a higher gain, and due to the compatibility of our modules, we observe significant enhancements when both are used, with accuracies of 3.39%, 6.48%, and 4.81% at buffer sizes of 200, 500, and 1000, respectively. On the ASER and DVC baselines, the performance improvement is greatest when using the frequency domain distribution multiplexing module, except for the case with a buffer size of 500. The combined use of the frequency domain distribution multiplexing and spatial distribution multiplexing modules is more beneficial. Specifically, the frequency domain distribution multiplexing module improves accuracy by 1.13% and 5.10% compared to the ASER and DVC baselines, respectively, when the buffer size is 500. On the other hand, the multi-domain distribution multiplexing module provided performance improvements of 3.43% and 4.42% for the two baselines when the buffer size was 200, and 1.61% and 2.66% for the buffer size was 1000. On the PCR baseline, except for the case of buffer size 1000, the frequency domain distribution multiplexing and spatial distribution multiplexing modules were used together to achieve state-of-the-art scores. Specifically, the accuracy reached 51.88% and 56.10% for buffer size 200 and buffer size 500, respectively.

[0116] (2) Sequence Standard - CIFAR100

[0117] Table 1 reports the average accuracy of the model on the CIFAR100 dataset with three different buffer sizes. Our modules can be used individually or in combination, and a consistent trend emerges across all baselines regardless of the playback pool size: On the DER++ baseline, combined use demonstrates superior performance compared to using a single module, achieving average accuracies of 16.69%, 22.05%, and 29.97% for buffer sizes of 200, 500, and 1000, respectively. On the ASER baseline, performance improvements of 2.70%, 1.58%, and 1.33% are achieved for buffer sizes of 200, 500, and 1000, respectively. On the DVC baseline, performance improvements of 1.36%, 1.60%, and 1.65% are achieved for buffer sizes of 200, 500, and 1000, respectively. At the PCR baseline, performance improvements of 0.86%, 1.39%, and 1.81% were achieved for buffer sizes of 200, 500, and 1000, respectively.

[0118] (3) Sequential benchmark - MiniImageNet

[0119] Table 1 reports the average accuracy of the model on the MiniImageNet dataset with three different buffer sizes. Except for DER++ with a buffer size of 200, the frequency domain distribution multiplexing and spatial distribution multiplexing modules generally improve the performance of the baseline model. On DER++, the multi-domain distribution multiplexing module shows little gain with smaller buffer sizes, but delivers a significant improvement with increasing buffer size, achieving a 4.01% performance improvement with a buffer size of 4k. On the ASER baseline, using a single module yields the best results, achieving performance improvements of 2.22%, 3.04%, and 1.83% for buffer sizes of 200, 500, and 1000, respectively. On the DVC baseline, performance improvements of 2.93%, 3.86%, and 4.10% are achieved for buffer sizes of 200, 500, and 1000, respectively. At the PCR baseline, performance improvements of 0.65%, 0.69%, and 0.57% were achieved for buffer sizes of 200, 500, and 1000, respectively.

[0120] Table 1. Quantitative results on three ordinal references (average accuracy in %, bold text indicates highest accuracy).

[0121]

[0122] (4) Smoothing benchmark - MNIST360

[0123] Table 2 reports the average accuracy of the model on the MNIST360 dataset with three different buffer sizes. We can observe that the frequency domain distribution multiplexing and spatial distribution multiplexing modules consistently and significantly improve the performance of the baseline model in all cases, with the frequency domain distribution multiplexing module performing better, and achieving state-of-the-art scores in some cases when applied together. On the DER baseline, performance improvements of 13.51%, 18.95%, and 15.86% are achieved for buffer sizes of 200, 500, and 1000, respectively. On the CLS-ER baseline, for a buffer size of 200, combining two modules yields the best results, with the multi-domain distribution multiplexing module achieving an 11.48% performance improvement. For buffer sizes of 500 and 1000, using the frequency domain distribution multiplexing module alone is the best approach, achieving performance improvements of 17.73% and 18.45%, respectively. On the CoCa baseline, combining two modules yields the best results, achieving performance improvements of 17.87%, 14.50%, and 12.48% for buffer sizes of 200, 500, and 1000, respectively. On the OCDNet baseline, for buffer sizes of 200 and 500, combining two modules yields the best results, achieving performance improvements of 15.91% and 12.85%, respectively. For a buffer size of 1000, using the frequency domain distribution multiplexing module alone is the best approach, achieving an 11.06% performance improvement.

[0124] Table 2 shows the quantitative results on a smoothed baseline (average accuracy in %, bold text indicates the highest accuracy).

[0125]

[0126] 2. Subjective effect

[0127] Figure 5 Feature maps from four datasets on different benchmarks were visualized to better demonstrate the effectiveness of the multi-domain distribution multiplexing module in capturing causal features in online image incremental classification. These benchmarks include the baseline model, the frequency domain distribution multiplexing module, the spatial distribution multiplexing module, and the multi-domain distribution multiplexing module. For each benchmark, one image was selected, and the feature maps of the same channel in the first layer of each model were visualized in four stages. These four stages visually illustrate the asymptotic biasing effect of the multi-domain distribution multiplexing module. Figure 5(a) Feature maps of the "car" at the four stages of Task 1 start, Task 1 end, Task 3 end, and Task 5 end (total of 5 tasks). As training progresses, the baseline tends to focus on offset features due to spurious correlations (top sky of the image). In contrast, the addition of the multi-domain distribution reuse module effectively enhances the focus of the feature maps on the "car" components. Similarly, according to Figure 5 (b)5(d), compared to other settings, the model with the multi-domain distribution reuse module focuses on the target object region and does not show significant changes during incremental learning. In other words, the multi-domain distribution reuse module effectively utilizes information from the target object region and aggregates causal features, thereby improving model generalization and mitigating catastrophic forgetting.

Claims

1. An online image incremental classification method based on a multi-domain distribution multiplexing network, characterized in that, The dataset is incrementally partitioned using RGB image data as input; the partitioned image data is then fed into a multi-domain distribution and reuse image classification network to train the established model parameters. Then, new image data is added to train the entire model again, and this step is repeated until no new data can be added; finally, the trained model is tested and online image incremental classification results are output. The implementation steps also include: Step 11. Input and incremental partitioning of the training dataset; To simulate the real-world scenario of a continuous increase in new images on the internet, the RGB image dataset will be randomly divided into different subsets according to categories to represent the category image data added at different times. The data will be divided into tasks based on either sequential or smoothed baselines. Under the sequential baseline, it is necessary to ensure that the categories do not overlap between different tasks, while the smoothed baseline allows the same category to appear repeatedly between different tasks. Step 12. Design a multi-domain distribution and multiplexing network; The multi-domain distribution multiplexing module (D3M) is a plug-and-play module. Given an input image, it first passes through a ResNet18 convolutional layer to obtain a preprocessed feature map. Then, this feature map is passed through the multi-domain distribution multiplexing module to generate an updated new feature map, which is causally adaptive. This updated feature map is then propagated to subsequent layers. Step 13. Design the frequency domain distribution multiplexing (FDM) module; Step 13.1 Frequency Domain Distribution Submodule; In the frequency domain distribution submodule, the feature map is divided into different components; firstly, the original feature map after fixing the convolutional layer is... Divided along the channel dimension Group, i.e. (8) in It is a set of numbers that satisfy... For each group of channel components Perform frequency conversion. Obtain frequency domain components Then they are concatenated to obtain multi-frequency domain components. : (9) (10) Step 13.2 Frequency Domain Multiplexing Submodule: Construct a frequency domain multiplexing submodule, each consisting of a multilayer perceptron (i.e., two fully connected layers), a ReLU activation function, and a sigmoid function; represent the multi-frequency domain components. The data is transmitted to the shared network to learn the frequency domain channel scores. for (11) in Represents the weights of the multilayer perceptron. Represent the Sigmoid function; obtain the frequency domain enhancement representation. for: (12) In the formula: This indicates multiplication by channel; Step 14. Design the Spatial Distribution and Reuse (SDM) module; Given an input feature The channel information of each spatial region feature is aggregated using average pooling and max pooling operations, and then concatenated to obtain the aggregated feature representation. ; (13) (14) (15) Then, the convolutional filter is applied to the aggregated feature representation. This yields a region information aggregation vector and generates a spatial causal score. To measure the sensitivity of different spatial regions to capturing causal features, i.e.: (16) Finally, the formula for calculating the spatial augmentation representation is as follows: (17)； In step 12, the multi-domain distribution multiplexing network formed by combining the multi-domain distribution multiplexing module with the convolutional network includes L combination blocks. Each composite block consists of two convolutional blocks. A multi-domain distribution reuse module The convolutional block consists of two ReLU activation functions. Includes convolutional layers and normalization layers; (1) In the formula: This represents function mapping operations; In each batch of tasks t, the training data After L combination blocks and a classifier To obtain the final output ;Right now, (2) (3) (4) Where C, H, and W represent respectively The number of channels, height, and width; The corresponding training objective is consistent with the online image incremental classification method, and the final loss is: (5) In the formula: For cross-entropy loss, The distillation loss is the value of the corresponding online image incremental classification method, where α is a hyperparameter. The multi-domain distribution multiplexing module employs a dual-branch network consisting of a frequency-domain distribution multiplexing (FDM) module and a spatial distribution multiplexing (SDM) module. These modules are placed in parallel or sequentially. Input features are enhanced representations by the parallel frequency-domain and spatial distribution multiplexing modules, respectively. Then, element-wise addition is used to combine the two features to obtain the final output of the multi-domain distribution multiplexing module. The outputs of the frequency-domain and spatial distribution multiplexing modules are then analyzed. , The fusion process is achieved through element-wise addition, that is: (6) in This indicates element-wise addition. For each multi-domain distribution reuse module, the following symbols are used: Distribute the inputs and outputs of the multiplexing module to each multi-domain component in the l-th block, i.e. (7)。 2. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 1, characterized in that, Includes the following steps: Step 1: Establish a multi-domain distribution and multiplexing network model; Step 2: Adjust network parameters and train and test the neural network; Step 21. Parameter settings; The model is implemented in the PyTorch framework. To evaluate the effectiveness of the multi-domain distribution reuse module, it is applied to two benchmarks, each of which includes four prominent baselines. Step 22. Input the test dataset; For the CIFAR10 dataset, a total of 10,000 images from all 10 classes were used for testing; For the CIFAR100 dataset, 10,000 images across all 100 classes were used for testing; for the MiniImageNet dataset, 5,000 images across all 100 classes were used for testing; and for the MNIST-360 dataset, 27,000 images across 9 classes were used for testing.

3. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 2, characterized in that, In step 11, under the sequential benchmark, for the CIFAR100 dataset, which has 100 categories of images, it is divided into 10 non-overlapping tasks, each with 10 classes. Then, the subset of the dataset for each task is sequentially input into the model for incremental image classification training. Under the smooth benchmark, for the MNIST-360 dataset, samples belonging to two consecutive classes at the same time are provided to the model in the form of a stream for incremental image classification training. Each example in the stream is rotated by an increasing angle, and after a fixed number of steps, the smaller of the two digits is switched. Experiments were conducted using four datasets: the CIFAR10, CIFAR100, and MiniImageNet datasets for the sequential benchmark, and the MNIST-360 dataset for the smooth benchmark. The CIFAR10 dataset was divided into 5 non-overlapping tasks, each containing 2 classes; the CIFAR100 and MiniImageNet datasets were divided into 10 non-overlapping tasks, each containing 10 classes.

4. The online image incremental classification method based on a multi-domain distribution multiplexing network according to claim 2, characterized in that, Step 21 includes: sequential benchmarks including DER++, ASER, DVC, and PCR; smoothed benchmarks including DER, CLS-ER, CoCa, and MMGCL; reproducing all baselines in the same environment using the baseline source code and default settings, and applying each module individually; for all baselines, following their respective data augmentation methods, using untrained ResNet18 as the backbone, and then using the same batch size and replay batch size, optimizer, hyperparameters, and other settings consistent with the original baseline settings as in the same benchmark settings; ASER, DVC, and PCR use a simplified ResNet18 with C=20 as the backbone, while DER, DER++, CLS-ER, CoCa, and MMGCL use a full ResNet18 with C=64. For sequential benchmarks, the batch size |Bt|=10 and the replay batch size |M|=10 are set. For smooth benchmarks, the batch size and replay batch size are consistent with DER. For the multi-domain distribution multiplexing unit, the frequency domain distribution multiplexing and spatial distribution multiplexing modules are organized in parallel, and 8×8 2DDCT basis functions are used in the frequency domain distribution multiplexing module. For the MNIST dataset, the channels are divided into 4 groups, and for the CIFAR10, CIFAR100, and MNIST-360 datasets, the channels are divided into 10 groups.

Images