Image classification method and system based on CLIP category incremental learning

Abstract

The application discloses a kind of based on CLIP category incremental learning's image classification method and system, comprising: constructing pre-trained CLIP model, obtains training image and corresponding text label, respectively using image encoder and text encoder extracts image feature and text feature, to train adapter;Wherein, according to the similarity of the text feature of new input training image and old category text feature, filter new and old category pairs of adjacent categories, to the old category text feature in each pair of adjacent categories, normal distribution is sampled, to construct hinge loss function;After completing the t training task, the adapter parameters of previous training task are fused with the adapter parameters of current training task, to obtain the final adapter parameters of current training task;The classification result of the image to be processed is obtained using the trained CLIP model. Reduce the forgetting of old class due to learning new class, improve image classification accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image classification method and system based on CLIP category incremental learning. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In image classification, such as in robotic object detection tasks, traditional object detection frameworks like neural network models are only suitable for static environments. They generate feature vectors of unknown targets from environmental images and compare them with feature vectors of known targets. While this can identify unknown targets, all unknown targets are grouped into one class, and the system cannot incrementally learn new classes, lacking the flexibility to adapt and update in dynamically changing environments. In real-world scenarios, however, we often encounter unknown target categories that haven't appeared in the training data. This necessitates that image classification training can both adapt to new knowledge and retain old knowledge.

[0004] If models are not updated, they become outdated over time, degrading their performance. Furthermore, due to privacy and storage constraints, older data may be unavailable or only partially accessible, leading to a scarcity of data from older categories and a severe imbalance in data distribution. Consequently, models become more biased towards current data and forget knowledge gained from older data—a phenomenon known as catastrophic forgetting. Therefore, the challenge of continuous learning lies in balancing plasticity and stability—that is, enabling models to learn new knowledge without forgetting old knowledge and to reuse and expand upon experiential knowledge across different tasks.

[0005] Class-incremental learning (CIL) is a continuous learning scenario involving learning from a data stream consisting of new classes added over time. It encompasses three main approaches: regularization-based, replay-based, and parameter isolation-based methods. These methods aim to preserve the memory of previous knowledge by adding a regularization term to the loss function, replaying old data samples, or assigning dedicated parameters to each class. However, most of these methods rely on models trained from scratch, which may not be optimal for learning from incremental data.

[0006] Pre-trained models on large-scale datasets demonstrate excellent generalization ability and robustness against catastrophic forgetting in downstream tasks. Furthermore, Contrastive Language-Image Pre-Training (CLIP) models exhibit strong zero-shot capability and high adaptability to downstream tasks during continuous learning. Leveraging the excellent feature extraction capabilities of pre-trained models, each incremental step requires only updating a small number of parameters, reducing the risk of forgetting. In contrast, models trained from scratch do not offer this advantage and may suffer severe performance degradation.

[0007] Currently, there are two main strategies for continuous learning of pre-trained models: one is to fine-tune the model; the other is to continuously expand a small number of parameters on the model, such as through cue-based methods or by adding adapters. Even with regularization constraints to avoid unnecessary adjustments, fine-tuning the model may impair the feature extraction capabilities of the original model and lead to catastrophic forgetting. Expanding parameters can mitigate the interference with the original model, but it increases time and space costs over time. Meanwhile, for visual-language pre-trained models, the language encoder provides rich information that is beneficial for continuous learning. However, in most existing methods, text features are only used for classification purposes, and their potential to help reduce forgetting has not been fully explored. Therefore, the poor training performance and low classification accuracy of pre-trained models can lead to problems such as low object detection accuracy, false positives, and false negatives when performing specific tasks such as robot object detection. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes an image classification method and system based on CLIP category incremental learning. This method selects and filters adjacent categories based on the text feature similarity of the images, adjusts the feature representations of old categories affected by similar new categories, and combines a decomposition parameter fusion strategy to reduce the forgetting of old categories due to learning new categories, thereby improving image classification accuracy.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, the present invention provides an image classification method based on CLIP category incremental learning, comprising:

[0011] Construct a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively;

[0012] The training images and their corresponding text labels are obtained, and image features and text features are extracted using an image encoder and a text encoder, respectively, to train the adapter.

[0013] During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task.

[0014] The CLIP model trained on the adapter is used to obtain the classification result for the image to be processed.

[0015] As an alternative implementation, the Euclidean distance between the text features of the old and new categories is used as the similarity, and pairs of old and new categories with similarity less than a set threshold are regarded as pairs of old and new categories of adjacent categories; for the text features of the old category in each pair of adjacent categories, the centroid and covariance matrix are calculated, and sampling is performed from a Gaussian distribution based on this.

[0016] As an alternative implementation, the sum of the hinge loss function and the cross-entropy loss function is used as the total loss function for the training process; the hinge loss function is:

[0017]

[0018] Where m is a constant, c and They represent belonging to adjacent category sets respectively. The old and new categories in the k-th pair of adjacent categories; f text (t c )and These represent the old category text features and the new category text features, respectively; A(·) represents the adapter; This represents the sampled data of the old category text features; dist(·) represents the distance function.

[0019] As an optional implementation, the adapter parameter fusion process includes: decomposing the adapter parameters of the current training task and the adapter parameters of the previous training task into the same orthonormal basis; calculating fusion weights based on the difference between the decomposed parameters of the current training task and the decomposed parameters of the previous training task; fusing the decomposed parameters of the current training task and the decomposed parameters of the previous training task based on the fusion weights to obtain fusion parameters; and obtaining the final adapter parameters based on the fusion parameters and the orthonormal basis.

[0020] As an alternative implementation method, the fusion weight M is:

[0021]

[0022] Among them, R t R represents the decomposition parameters for the current training task. t-1is the decomposition parameter of the previous training task, and b is a constant.

[0023] As an alternative implementation, the fusion parameter R and the final adapter parameter W are respectively:

[0024] R=(JM)⊙R t-1 +M⊙R t ;

[0025] W = BR;

[0026] Where J represents a matrix of all 1s; ⊙ represents element-wise multiplication; B is an orthonormal basis; R t R represents the decomposition parameters for the current training task. t-1 represents the decomposition parameters of the previous training task; M represents the fusion weights.

[0027] Secondly, the present invention provides an image classification system based on CLIP category incremental learning, comprising:

[0028] The model building module is configured to build a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively.

[0029] The training module is configured to acquire training images and corresponding text labels, and extract image features and text features using an image encoder and a text encoder, respectively, to train the adapter;

[0030] During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task.

[0031] The classification module is configured to use the CLIP model trained on the adapter to obtain the classification result for the image to be processed.

[0032] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0033] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0034] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0036] This invention proposes an image classification method and system based on CLIP (Category Incremental Learning), which utilizes textual features of images to enhance the classification ability of adjacent categories in category incremental learning, reducing the forgetting of old categories due to learning new categories. When a new category appears, the new decision boundary may classify some old category samples into the new category. To address this issue, this invention uses textual features to infer the relationship between new and old categories, strengthening the separation of adjacent categories. This invention selects adjacent category pairs by calculating the distance between the textual features of new and old categories. Since there is sufficient data to learn the new category, it is not necessary to modify its representation; the focus is on adjusting the representation of the old category affected by the new category, thereby reducing the forgetting of old categories due to learning new categories.

[0037] This invention proposes an image classification method and system based on CLIP category incremental learning. For the linear layer adapter of the pre-trained model, a decomposition parameter fusion method is proposed, which does not increase the number of parameters with the increase of training tasks. Unlike directly calculating the average value of parameters, the fusion strategy of this invention is more fine-grained and considers the shared knowledge between tasks. To balance stability and plasticity, the parameters before and after the incremental task are merged according to the parameter changes caused by learning the current task. This eliminates the need to add additional distillation loss during training to constrain parameter changes, thereby reducing training costs.

[0038] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0039] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0040] Figure 1 This is a flowchart of the image classification method based on CLIP category incremental learning provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a framework diagram of the image classification method based on CLIP category incremental learning provided in Embodiment 1 of the present invention;

[0042] Figure 3The graphs show the accuracy of the method of this invention compared to other state-of-the-art (SOTA) baselines on CIFAR100, ImageNet100, and ImageNet-R.

[0043] Figure 4 A diagram showing the comparison results of various methods in terms of accuracy and learnable parameters;

[0044] Figure 5 This is a diagram illustrating the confusion matrix and differences of the first five tasks in the ImageNet100 B0 Inc10 experiment.

[0045] Figure 6 This is a schematic diagram showing the results of experiments with different thresholds α on ImageNet100 B0 Inc10. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0048] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0049] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0050] Terminology Explanation

[0051] 1. Category Incremental Learning

[0052] Based on the main strategies used, incremental class learning methods can be divided into three types. (1) Regularization-based methods suppress the forgetting of old classes by imposing constraints on the classifier output. Some studies calculate the importance of each parameter of the old task and then add a regularization term to avoid catastrophic forgetting; others share knowledge between tasks by using knowledge distillation based on optimal transfer; these methods use knowledge distillation loss as a regularization term. (2) Replay-based methods retain the memory of the old classes by saving some samples or features of the old classes and then replaying these samples or features with samples of the new classes when training the new classes. Some studies explore how to reduce the storage footprint of samples, some studies look for better strategies to select samples to be added to the samples, and some studies can save additional models and samples to assist the training of the current model. (3) Dynamic architecture-based methods dynamically adjust the structure of the model to adapt to the learning of new classes.

[0053] 2. Pre-trained category incremental learning

[0054] There are two main approaches to using pre-trained models in incremental learning. One is to fine-tune the model's parameters to adjust the feature representation; for example, using different learning rates to update the pre-trained backbone and classifier. The other is to keep the pre-trained model unchanged and add parameters to adjust the feature representation; for example, introducing an adapter into the pre-trained CLIP model to adapt to the incremental task; or training an adapter for each task and using cross-modal attention to fuse the language and visual information of CLIP. There is also a cue-based approach, which selects appropriate cues based on the feature output of the pre-trained model, adds them to the model, and then re-acquires features for classification.

[0055] 3. Incremental learning of categories without replay samples

[0056] Sometimes, due to privacy and memory constraints, it is not possible to store old class samples. Some existing methods that do not use replay samples use a Gaussian distribution to model the data and aid in classification. Other methods include oversampling prototypes or augmented prototypes to simulate replay samples. Some recent research uses models to synthesize data from older tasks as an alternative to replay samples. Cue-based methods use a frozen backbone and relative isolation of cue parameters to avoid using replay samples.

[0057] Example 1

[0058] The goal of incremental class learning is to train a model that can classify data from an increasing number of classes over time. With the development of visual language pre-trained models (such as CLIP), they have demonstrated good generalization capabilities, enabling incremental class learning with completely fixed parameters. However, simply fine-tuning the model to further adapt it to downstream tasks can lead to significant forgetting. Most existing work on pre-trained models assumes that forgetting of old classes is consistent as the model acquires new knowledge.

[0059] Therefore, this embodiment proposes an image classification method based on CLIP category incremental learning, which uses the textual features of class names to measure the influence of new classes on old classes and adjusts their representations accordingly to reduce forgetting; it also proposes a decomposition parameter fusion method for adapters, which greatly reduces forgetting caused by fine-tuning the adapters with new data.

[0060] like Figure 1 As shown, it specifically includes:

[0061] Construct a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively;

[0062] The training images and their corresponding text labels are obtained, and image features and text features are extracted using an image encoder and a text encoder, respectively, to train the adapter.

[0063] During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task.

[0064] The CLIP model trained on the adapter is used to obtain the classification result for the image to be processed.

[0065] like Figure 2 As shown, the framework of this embodiment includes a pre-trained CLIP model and a linear adapter set in the CLIP model. For incremental learning of downstream tasks, only the adapter needs to be trained. Image features are transmitted through the image encoder and the adapter, and label features are transmitted through the text encoder to transmit class labels. The classification result is determined by measuring the similarity between them.

[0066] The categorical incremental learning algorithm trains a model M on each training task t. tThis model can classify data in new classification tasks where the class sets are disjoint. Model M t Use only the current dataset D t The data in the previous model M t-1 Update without accessing previous datasets.

[0067] By acquiring training images, such as robot inspection images and corresponding text labels, image encoders and text encoders are used to extract image features and text features of targets in the inspection images, respectively, to train the adapter.

[0068] One manifestation of catastrophic forgetting during training is the model's misidentification of old category data as new categories, a phenomenon exacerbated when new categories are similar to old ones. These categories belong to adjacent categories. They often have similar semantics and appearances in natural language, such as "macaw" and "parrot." Distinguishing between them is challenging for the model, especially as new categories are added over time.

[0069] Therefore, this embodiment introduces a neighbor separation module after the text encoder. The neighbor separation module identifies adjacent categories by calculating the similarity of text features, samples the distribution of text features of the old category, and calculates the hinge loss function to guide the adapter learning process.

[0070] Semantic information can filter out adjacent old and new classes when encountering new data, and then adjust the image features of the old classes accordingly. The image features of the old classes are generated by sampling from their respective Gaussian distributions and input into the adapter along with the new data to calculate the classification loss. Simultaneously, pairs of old and new classes with adjacent classes are selected based on text feature similarity. For each old class in the pair, more data is sampled from its Gaussian distribution and input into the adapter to calculate the separation loss. This adjusts the feature representations of the old classes affected by the similar new classes, thereby reducing the forgetting of old classes due to learning new classes.

[0071] The specific process includes:

[0072] To filter pairs of new and old categories that are adjacent to each other, the distance between the new and old categories is calculated using normalized text features of the category names:

[0073] D = dist(f text (t new ), f text (t old )) (1)

[0074] Where D is the Euclidean distance between the normalized text features of the old and new categories, and f text (t new ) and ftext (t old ) are the new categories t new Text features and old categories t old Textual features.

[0075] Focus on measuring the distance between the old and new categories, as this reflects the degree of interference. Then, filter the adjacent category sets.

[0076] P = {(i, j)|D} ij <α} (2)

[0077] Where α represents the threshold; D ij It is the distance between the text features of the old category i and the new category j. This criterion reduces the complexity of many-to-many relationships by selecting a subset of one-to-one relationships.

[0078] This embodiment then introduces a hinge loss by sampling features of the old class from the normal distribution of each pair of adjacent classes. Since there is no real data for the old class, excessive adjustment could harm its classification performance; therefore, only a small but effective adjustment is made using the hinge loss. The hinge loss function allows the adapter to focus more on the more easily confused adjacent classes while minimizing interference with the representation of the old classes.

[0079] The hinge loss function is:

[0080]

[0081] Where m is a constant, c and Indicates belonging to adjacent category sets The old and new categories in the k-th pair of adjacent categories; A represents the adapter; The sampled data representing the text features c of the old category are approximated using a Gaussian distribution, and the feature distribution of each category is based on the real image embedding e of category c. c =f img (x c To calculate the centroid μ c The sum of the covariance matrix ∑ c Then sample from a Gaussian distribution. Generate image embeddings for class C

[0082] The final total loss function is the sum of the hinge loss function and the cross-entropy loss. merge:

[0083]

[0084] Cross-entropy loss for:

[0085] Let the image encoder of the CLIP model be f img (·), the text encoder is f text (·), adapter is A(·), given image input x i , class y i It has a fixed prompt template, such as "a photo of a [CLS]", using t i The output result is as follows:

[0086]

[0087] Where τ is temperature.

[0088] If the text encoder is frozen, only the text embeddings, not the class names, need to be preserved in the CIL. The cross-entropy loss criterion is used to fine-tune the adapter parameters.

[0089]

[0090] In this embodiment, a decomposed parameter fusion strategy is further employed to reduce forgetting and maintain the stability of the linear adapter. After completing the t-th training task, the adapter parameters from the previous training task are fused with the adapter parameters from the current training task to obtain the final adapter parameters for the current training task. The parameters learned from two consecutive training tasks are decomposed into shared knowledge and task-specific knowledge. The parameters are then fused based on this decomposition, which better balances the plasticity and stability of the system.

[0091] Specifically:

[0092] Evaluate the importance of each new parameter. The gradient represents the direction of parameter updates based on new data, and the parameter differences between training tasks are a weighted sum of gradients. Therefore, parameters that change more significantly are more important for learning new knowledge. To obtain an importance matrix as fine-grained weights for parameter fusion, calculate the difference between the parameters of the current training task and the parameters of the previous training task, and take the maximum value for normalization:

[0093]

[0094] Where M represents the importance of each new parameter, W t W represents the adapter parameters for the current training task. t-1 This represents the adapter parameters for the previous training task, where b is a constant.

[0095] To compare W t and W t-1 Differences under the same standard will W t and W t-1 Decompose to the same orthonormal basis B; that is, decompose W through SVD. t-1Decompose to an orthonormal basis B and compute from matrix W t Projection to B:

[0096] W t-1 →BR old (8)

[0097] R new =B T W t (9)

[0098] Wherein, the parameter matrix W t-1 and W t Represented as a linear combination of orthonormal basis B, i.e., R t-1 and R t These represent task-specific knowledge; matrix B represents the shared knowledge space across parameter matrices.

[0099] Therefore, calculating the matrix difference can be transformed into calculating the difference between two different weights of the same orthonormal basis; based on the R obtained from the decomposition... t and R t-1 The difference is used to calculate the soft mask M, which is then substituted into W in (7). t and W t-1 The soft mask M is calculated as the fusion weight; finally, the fusion parameters R and the final adapter parameters W are calculated.

[0100] R=(JM)⊙R t-1 +M⊙R t (10)

[0101] W = BR (11)

[0102] Where J represents a matrix of all 1s, and ⊙ represents element-wise multiplication.

[0103] Finally, based on the CLIP model obtained after adapter training, the images to be processed are classified, such as target classification and recognition of robot inspection images.

[0104] This embodiment uses the following datasets for experiments: CIFAR00, ImageNet1K, ImageNet100, ImageNet-R, and CUB200. The CIFAR100 dataset consists of 100 categories, each containing 600 color images at a resolution of 32×32 pixels. 500 images are allocated to the training set, and 100 images to the test set. The ImageNet1K dataset consists of 1000 categories, and the ImageNet100 dataset is a subset of the ImageNet1K dataset, consisting of 100 selected classes. The ImageNet-R dataset is a collection of various image categories derived from the ImageNet dataset, including images of various styles such as art, comics, graffiti, embroidery, and video games. These are different representations of the 200 categories in the ImageNet dataset. Following previous work, the datasets are split into training and test sets. The CUB200 dataset is widely used for fine-grained visual classification tasks and includes 11,788 images of 200 subclasses of different birds.

[0105] Comparison of CIL methods: L2P++, DualPrompt, CODA, SLCA, ADAM-Adapter, and PROOF. Continual-CLIP is a zero-shot test of CLIP. PROOF is based on the CLIP-based method with replay samples. To ensure fairness in the comparison, all methods use the same OpenAI CLIP pre-trained weights. Results for DualPrompt, L2P++, and CODA are obtained by running the publicly available code implementation of the CODA method. Results for the SLCA, ADAM-Adapter, PROOF, and Continual-clip methods are derived from their respective publicly available code.

[0106] The evaluation metric is: the average accuracy of training the t-th task on the test data for tasks 1 to t, denoted as A. t Avg is the average accuracy across all tasks, and Last is the average accuracy after the last task.

[0107] The backbone network in this embodiment is the ViT-B / 16 version of CLIP, trained for 15 epochs using the Adam optimizer, starting with a learning rate of 0.001. The MultiStepLR scheduler was used, reducing the learning rate to 0.1x at epochs 4 and 10. A default threshold of 0.65 was used to select adjacent class pairs based on textual feature distance. Approximately 2000 samples per epoch were used to simulate replay samples, matching the increased data volume required in traditional replay settings. In each iteration, 20 additional features were sampled for each class selected by the threshold. Due to insufficient data for some classes, a full-rank covariance matrix could not be obtained. Following previous work and using covariance shrinkage to obtain the full-rank matrix, experiments were conducted on several different shuffling orders, and the average values ​​of these orders are reported.

[0108] 1. Overall Comparison Results

[0109] Table 1 shows the experimental results for continuous learning on CIFAR100, Table 2 shows the experimental results on ImageNet100, and Table 3 shows the experimental results on ImageNet-R. B represents the number of base classes, and Inc represents the number of incremental classes. All baseline-based results are replicated using the published code and CLIP pre-trained VIT-B / 16 weights. Experiments were conducted on several different shuffling orders, and the average values ​​of these orders are reported.

[0110] Table 1. Experimental results of continuous learning on CIFAR100

[0111]

[0112] Table 2 Experimental results on ImageNet100

[0113]

[0114] Table 3 Experimental results on ImageNet-R

[0115]

[0116] Tables 1, 2, and 3 present the comparison results between the proposed method and existing methods on the CIFAR100, ImageNet100, and ImageNet-R datasets. The proposed method significantly outperforms various competing methods in most cases. On the ImageNet100 dataset, the proposed method achieves at least 1.58% higher final accuracy than other methods. In the base0 setting experiments on the ImageNet-R dataset, the proposed method achieves at least 1.08% higher final accuracy than other methods, indicating that the proposed method effectively utilizes textual information and parameter fusion to reduce forgetting and maintain model stability. Compared to CLIP zero-shot capability, the proposed method only adds one linear layer adapter, yet still achieves good results, demonstrating that the proposed method does not solely rely on the generalization of the pre-trained model to achieve good performance. Figure 3 The results show a downward trend in average accuracy as the number of classes increases with different datasets and settings. The method in this embodiment can significantly mitigate this trend by reducing forgetting.

[0117] These three cue-based methods exhibit a severe cue imbalance in the base50 and base100 settings because they use the same number of cue points per task by default. This causes the model's predictions to be biased towards the base class and makes it difficult to infuse new knowledge. L2P and DualPrompt have outperformed the original model pre-trained on ImageNet21k on the ImageNet-R dataset. However, CODA performs poorly on all three datasets, suggesting that its approach may be highly coupled to specific pre-trained models, such as ViT pre-trained on ImageNet21k.

[0118] SLCA easily achieves higher performance on initial tasks because it fine-tunes the backbone and classifier. Therefore, it performs well in both base50 and base100 settings. However, in long sequence settings, the cumulative forgetting caused by backbone fine-tuning is more pronounced.

[0119] ADAM primarily trains the adapter model on the initial task and then does not train the model on subsequent tasks. Therefore, its performance depends on the proportion of data in the initial task. Since the model is not trained on subsequent tasks, it maintains model stability but does not learn new knowledge. The model's low plasticity affects performance. Due to the significant differences in image styles in ImageNet-R, the dataset requires the model to learn more knowledge to adapt to rich image styles, and the method performs worse on this dataset than on the unified style CIFAR100 and ImageNet100.

[0120] The PROOF adapter designed for CLIP extends with the task and fuses information from text and images using cross-modal attention, but it does not consider the influence of neighboring categories in CLIP classification. Even without the paradigm and extended parameters, the method in this embodiment still has advantages.

[0121] This embodiment was also evaluated on the large dataset ImageNet1K and the fine-grained dataset CUB200. The results are shown in Table 4. Experimental results show that the method still has advantages on large datasets, especially on the more difficult fine-grained task CUB200, where the performance of the method in this embodiment is greatly improved compared to the zero-shot performance of CLIP.

[0122] Table 4 shows the results on CUB200 and ImageNet1K, divided into 10 tasks.

[0123]

[0124] In summary, the method in this embodiment does not expand the model as the number of tasks increases, while maintaining learning from new data and mitigating forgetting in incremental tasks. It achieves a good balance between the plasticity and stability of the model while maintaining the capabilities of the pre-trained model.

[0125] 2. Parameter quantity analysis

[0126] Figure 4 This section compares the incremental parameter size of the method in this embodiment with other methods on the ImageNet100 dataset. Using the same backbone CLIP pre-trained VIT-B / 16, the method of freezing the backbone parameters has significantly lower parameter update costs than the SLCA method, which fine-tunes the entire model. Among methods that freeze the backbone and add learnable parameters, the method in this embodiment adds the fewest parameters. Although it adds a text encoder to the pure visual encoder method, since the label text is fixed, the text features of the labels only need to be calculated once during the entire training process, and the number of labels is far less than the amount of data. For example, ImageNet100 only has 100 labels to be calculated. Therefore, the cost of the text encoder is negligible compared to the cost of processing a large number of images and multiple iterations of the visual encoder.

[0127] 3. Ablation test

[0128] 3.1 Module Ablation

[0129] Table 5 shows the module ablation experiment results on ImageNet100 using B0 Inc10. SG represents the extraction of some old class features from a Gaussian distribution. It is a random selection of class pairs. The meaning of PF w / o MD is to use only Equation (7) for parameter fusion. MD represents matrix decomposition.

[0130] Table 5 shows the experimental results of module ablation on ImageNet100 using B0 Inc10.

[0131]

[0132] Table 5 shows the results for different components in the method of this embodiment. (random) refers to randomly selecting class pairs instead of using the distance of text features. In this case, the accuracy is very close to the baseline, while using the full module with text features improves the accuracy by 2.24%. This indicates that text-guided nearest-class selection is effective. The performance of the parameter fusion method without matrix factorization and the full parameter fusion method with matrix factorization demonstrates that the parameter fusion method using fine-grained importance matrices is effective, and matrix factorization further enhances its performance.

[0133] 3.2. Classification by Proximity

[0134] Table 6 shows the prediction results of the model under different ablation experimental settings on 50 experimental images. The other three images are adjacent new categories selected based on text feature similarity.

[0135] Table 6 shows the prediction results of the model under different ablation experimental settings for 50 experimental images.

[0136]

[0137] like Figure 5 As shown, the method in this embodiment effectively corrects old categories that have been incorrectly classified into new categories. Figure 5 (c) shows that the number of samples misclassified to the new category decreased, the number of correctly classified samples increased, and the negative impact on other categories was minimal. Table 6 shows some examples of affected categories. The results indicate that the new category selected based on text feature similarity, which is close to the old category, is the main cause of prediction errors in the old affected categories. The selected category accounts for 23 out of 25 incorrect predictions. The selected category uses... It can effectively reduce erroneous predictions of affected old categories.

[0138] 3.3 Threshold ablation

[0139] exist Figure 6The effect of the threshold α on neighboring categories was investigated. Since most distances were greater than 0.5, the method in this embodiment gradually increased the threshold starting from 0.5. As the threshold increased, more neighboring categories were selected, and performance improved accordingly. However, an excessively high threshold would select many different category pairs, which would interfere with classification and increase computational cost. Selecting more different categories is not beneficial. Therefore, the method in this embodiment used 0.65 as the threshold for all experiments.

[0140] 3.4. A prompt-based approach with CLIP text encoder.

[0141] For a fair comparison, the CLIP text encoder was added as a classifier to the cue-based method, which is independent of the classifier. The results are shown in Table 7. In L2P++ and DualPrompt, the key and value for different tasks are relatively independent, which makes the representation of the old class change very little. Using fixed CLIP text features instead of trainable linear layers as a classifier may lead to misclassification of old class samples when the text features of the new class are similar to those of the old class. Therefore, the performance of these two methods degrades when using the CLIP text encoder. CODA uses weights to combine all cue components, allowing the old class to benefit from the newly expanded parameters. Using an informative text classifier can improve performance. Compared to these methods, the method in this embodiment can better utilize the CLIP text encoder.

[0142] Table 7 Final accuracy on ImageNet-R BO Inc20

[0143]

[0144] 4. Comparison with traditional methods

[0145] Prior to this, traditional methods using examples without pre-trained models also achieved good results in classroom incremental learning. Table 8 compares the traditional methods using a VIT-B / 16 model pre-trained on ImageNet21K for initialization, with the results reproduced from PILOT using pre-trained weights on ImageNet21K. This embodiment also evaluates these baselines using the same initialization as this embodiment, but the performance is much worse. Even without using examples, the method in this embodiment still has a significant advantage over these state-of-the-art methods.

[0146] Table 8 Comparison results of traditional methods on ImageNet-R BO Inc20

[0147]

[0148] This embodiment proposes an image classification method based on CLIP category incremental learning. It studies the incremental learning problem based on pre-trained visual language models, introduces text features of classes to adjust the representation of classes that are greatly affected by new data, and effectively alleviates forgetting. The model's stability and forgetting are effectively maintained through adapter and parameter fusion strategies.

[0149] Example 2

[0150] This embodiment provides an image classification system based on CLIP category incremental learning, including:

[0151] The model building module is configured to build a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively.

[0152] The training module is configured to acquire training images and corresponding text labels, and extract image features and text features using an image encoder and a text encoder, respectively, to train the adapter;

[0153] During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task.

[0154] The classification module is configured to use the CLIP model trained on the adapter to obtain the classification result for the image to be processed.

[0155] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0156] In further embodiments, the following is also provided:

[0157] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0158] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0159] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0160] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0161] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0162] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0163] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0164] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0165] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and so on. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0166] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0167] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. An image classification method based on CLIP category incremental learning, characterized in that, include: Construct a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively; The training images and their corresponding text labels are obtained, and image features and text features are extracted using an image encoder and a text encoder, respectively, to train the adapter. During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task. The adapter parameter fusion process includes: decomposing the adapter parameters of the current training task and the adapter parameters of the previous training task into the same orthonormal basis; calculating the fusion weight based on the difference between the decomposed parameters of the current training task and the decomposed parameters of the previous training task; fusing the decomposed parameters of the current training task and the decomposed parameters of the previous training task according to the fusion weight to obtain the fusion parameters; and obtaining the final adapter parameters based on the fusion parameters and the orthonormal basis. Fusion weights for: ； Fusion parameters and final adapter parameters They are respectively: ； ； in, These are the decomposition parameters for the current training task. These are the decomposition parameters from the previous training task. It is a constant; A matrix representing all 1s; This is element-wise multiplication; It is an orthonormal basis; The CLIP model trained on the adapter is used to obtain the classification result for the image to be processed.

2. The image classification method based on CLIP category incremental learning as described in claim 1, characterized in that, The Euclidean distance between the text features of the old and new categories is used as the similarity. Pairs of old and new categories with similarity less than a set threshold are considered as pairs of old and new categories that are adjacent to each other. For the text features of the old category in each pair of adjacent categories, the centroid and covariance matrix are calculated, and samples are taken from the Gaussian distribution based on these matrices.

3. The image classification method based on CLIP category incremental learning as described in claim 1, characterized in that, The sum of the hinge loss function and the cross-entropy loss function is used as the total loss function for the training process; the hinge loss function is: ； in, It is a constant. and They represent belonging to adjacent category sets respectively. The old and new categories in the k-th pair of adjacent categories; and These are text features for the old category and text features for the new category, respectively. Indicates adapter; Sampling data representing the text features of the old category; Represents the distance function.

4. An image classification system based on CLIP category incremental learning, characterized in that, include: The model building module is configured to build a pre-trained CLIP model, including an image encoder, a text encoder, and adapters set after the image encoder and the text encoder, respectively. The training module is configured to acquire training images and corresponding text labels, and extract image features and text features using an image encoder and a text encoder, respectively, to train the adapter; During training, based on the similarity between the text features of the newly input training image and the text features of the old category, pairs of new and old categories in adjacent categories are selected. For the text features of the old category in each pair of adjacent categories, a normal distribution is used for sampling to construct the hinge loss function. After completing the t-th training task, the adapter parameters of the previous training task are fused with the adapter parameters of the current training task to obtain the final adapter parameters of the current training task. The adapter parameter fusion process includes: decomposing the adapter parameters of the current training task and the adapter parameters of the previous training task into the same orthonormal basis; calculating the fusion weight based on the difference between the decomposed parameters of the current training task and the decomposed parameters of the previous training task; fusing the decomposed parameters of the current training task and the decomposed parameters of the previous training task according to the fusion weight to obtain the fusion parameters; and obtaining the final adapter parameters based on the fusion parameters and the orthonormal basis. Fusion weights for: ； Fusion parameters and final adapter parameters They are respectively: ； ； in, These are the decomposition parameters for the current training task. These are the decomposition parameters from the previous training task. It is a constant; A matrix representing all 1s; This is element-wise multiplication; It is an orthonormal basis; The classification module is configured to use the CLIP model trained on the adapter to obtain the classification result for the image to be processed.

5. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, implement the method as described in any one of claims 1-3.

7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method as described in any one of claims 1-3.

Images