Image classification method and system capable of reserving and compensating modal difference

Abstract

The invention relates to the technical field of image processing, and provides an image classification method and system capable of reserving and compensating modal difference, and the method comprises the steps: carrying out the fine adjustment of a CLIP model through a first task data set under different epochs for a first task, calculating the variable quantity of negative class output, measuring the variable quantity of the modal difference, and carrying out the calculation of the variable quantity of the modal difference; when the variable quantity of the modal difference exceeds a selected threshold value, performing fixed fine adjustment on epoch; any task except the first task is traversed, according to a fixed fine tuning epoch, a corresponding task data set is used to perform fine tuning on the CLIP model, an auxiliary cosine classifier is trained, and the auxiliary cosine classifier only accepts visual modal input obtained by the CLIP model; and obtaining a text and an image to be classified, and obtaining a classification result through the fine-tuned CLIP model and the auxiliary cosine classifier. The task specificity is enhanced while the modal gap is reserved, and the classification performance of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an image classification method and system for retaining and compensating for modal gaps. Background Art

[0002] The statements in this section merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] The goal of continual learning is to enable the model to continuously acquire new knowledge and adapt to the ever-changing real world. Traditional methods of continual learning focus on training the model from scratch, aiming to reduce catastrophic forgetting of old task knowledge while maintaining plasticity to adapt to new data, which is commonly referred to as the stability-plasticity trade-off. However, with the recent development of large-scale pre-trained models, these models provide stronger stability and generalization ability for continual learning. Among pre-trained models, vision-language pre-trained models such as the Contrastive Language-Image Pre-training (CLIP) using contrastive learning show excellent generalization ability for downstream tasks and demonstrate impressive zero-shot capabilities. Therefore, CLIP-based continual learning (CL) has become a promising new direction, attracting increasing attention from researchers.

[0004] Existing CLIP-based continual learning methods can be roughly divided into two approaches: one is to fine-tune the backbone to modify the feature table, and the other is to freeze the backbone and introduce learnable modules for continual learning. They usually regard CLIP as a feature extractor and approximate it to a vision model enhanced by text. Based on this view, they focus on better feature fusion to improve performance or utilize text information to guide the adaptation of visual features. However, these methods often neglect the unique cross-modal characteristics of CLIP, namely the modal gap, which differentiates it from single-modal systems.

[0005] Downstream datasets are usually much smaller than pre-training datasets, making them insufficient to fully train the model. Therefore, modifying the basic properties of the pre-trained model, such as the modal gap, according to downstream data may disrupt the pre-trained knowledge, which may affect the performance of continual learning. During the training process of the current task, the cross-entropy optimization objective tends to widen this gap, and this phenomenon will intensify with the continuous arrival of incremental learning data. In addition, directly aligning the two modalities of the downstream task will also significantly change the pre-trained knowledge. Summary of the Invention

[0006] To solve the technical problems existing in the above-mentioned background art, the present invention provides an image classification method and system for retaining and compensating the modality gap, which stops training when the modality gap deviation exceeds the stable threshold to prevent cross-modal knowledge distortion. In order to compensate for the modality gap, an auxiliary cosine classifier is introduced in the visual space to enhance task specificity while retaining the modality gap, thereby further improving the learning ability of CLIP and improving the classification performance of the model.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] The first aspect of the present invention provides an image classification method for retaining and compensating the modality gap, which includes:

[0009] Obtain a number of tasks, each task corresponding to a task data set composed of a number of pieces of training data, and each piece of training data includes a piece of text and an image;

[0010] For the first task, at different epochs, fine-tune the CLIP model using the first task data set, and calculate the change amount of the negative class output to measure the change amount of the modality gap. When the change amount of the modality gap exceeds the selected threshold, fix the fine-tuning epoch;

[0011] Traverse any task except the first task. After fine-tuning the CLIP model using the corresponding task data set according to the fixed fine-tuning epoch, train the auxiliary cosine classifier, and the auxiliary cosine classifier only accepts the visual modality input obtained by the CLIP model;

[0012] Obtain the text and image to be classified, and obtain the classification result through the fine-tuned CLIP model and the auxiliary cosine classifier.

[0013] Further, the negative class output is:

[0014]

[0015] where x i is the image feature, represents the text feature of the non-matching class of the image x i , N is the number of images, and K is the number of texts.

[0016] Further, the change amount of the modality gap is:

[0017]

[0018] where neg e represents the negative class output after the e-th epoch, and neg 0 represents the initial negative class output.

[0019] Furthermore, during the training process of the auxiliary classifier, the weights of the old tasks are frozen, and only the weights of the new tasks are trained.

[0020] The second aspect of the present invention provides an image classification system for retaining and compensating for the modality gap, which includes:

[0021] A data acquisition module, which is configured to: acquire a number of tasks, each task corresponding to a task dataset composed of a number of pieces of training data, and each piece of training data includes a piece of text and an image;

[0022] A parameter determination module, which is configured to: for the first task, at different epochs, fine-tune the CLIP model using the first task dataset, and calculate the change in the negative class output to measure the change in the modality gap. When the change in the modality gap exceeds the selected threshold, fix the fine-tuning epoch;

[0023] A fine-tuning module, which is configured to: traverse any task except the first task, fine-tune the CLIP model using the corresponding task dataset according to the fixed fine-tuning epoch, and then train an auxiliary cosine classifier, and the auxiliary cosine classifier only accepts the visual modality input obtained by the CLIP model;

[0024] An inference module, which is configured to: acquire the text and image to be classified, and obtain the classification result through the fine-tuned CLIP model and the auxiliary cosine classifier.

[0025] Furthermore, the negative class output is:

[0026]

[0027] where x i is the image feature, represents the text feature of the non-matching class of image x i , N is the number of images, and K is the number of texts.

[0028] Furthermore, the change in the modality gap is:

[0029]

[0030] where neg e represents the negative class output after the e-th epoch, and neg 0 represents the initial negative class output.

[0031] Furthermore, during the training process of the auxiliary classifier, the weights of the old tasks are frozen, and only the weights of the new tasks are trained.

[0032] The third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps in an image classification method for retaining and compensating for modal gaps as described above are implemented.

[0033] The fourth aspect of the present invention provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, and when the processor executes the program, the steps in an image classification method for retaining and compensating for modal gaps as described above are implemented.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] When the modal gap deviation of the present invention exceeds the stability threshold, the training is stopped to prevent cross-modal knowledge distortion, and in order to compensate for the modal gap, an auxiliary cosine classifier is introduced in the visual space to enhance task specificity while retaining the modal gap, thereby further improving the learning ability of CLIP and improving the classification performance of the model.

[0036] The present invention analyzes the change of the modal gap during the fine-tuning process of CLIP. The modal gap effectively reflects the degree to which the pre-trained knowledge is retained. Using modal gap preservation to mitigate forgetting and modal gap compensation to enhance the ability of new data, a new modal-gap-based perspective is introduced for continuous learning. A large number of experiments conducted on multiple benchmarks show that while improving performance, no additional replay data is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specification drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.

[0038] Figure 1 is a flowchart of an image classification method for retaining and compensating for modal gaps according to Embodiment 1 of the present invention;

[0039] Figure 2 is a schematic diagram of the modal gap according to Embodiment 1 of the present invention;

[0040] Figure 3 is a schematic diagram of the naive fine-tuning in Embodiment 1 of the present invention to enlarge the modal gap;

[0041] Figure 4 is a schematic diagram of the direct alignment loss in Embodiment 1 of the present invention to reduce the modal gap;

[0042] Figure 5 is a schematic diagram of the change of the mean of the positive class cosine and the negative class cosine on ImageNet-R according to Embodiment 1 of the present invention;

[0043] Figure 6 It is a schematic diagram of the change of the positive cosine and negative cosine means on ImageNet-100 in the first embodiment of the present invention;

[0044] Figure 7 It is a comparison chart of the accuracy rate and learnable parameters of different methods in the first embodiment of the present invention;

[0045] Figure 8 It is a schematic structural diagram of a computer device in the fourth embodiment of the present invention. Detailed implementation manners

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0047] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further descriptions of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0048] Term explanation:

[0049] One epoch refers to the process of sending all data into the network to complete a forward calculation and a backpropagation process.

[0050] Embodiment 1

[0051] This embodiment provides an image classification method for retaining and compensating for the modality gap.

[0052] As Figure 2 shown, the modality gap is a phenomenon in which the feature distances within a modality are small and the feature distances between modalities are large, and the feature distributions of the two modalities are in two different cones. In order to make full use of the cross-modal characteristics and pre-trained knowledge of CLIP, this embodiment explores strategies for retaining the inherent modality gap while addressing its limitations in continual learning.

[0053] This embodiment studies CLIP-based class-incremental learning from the perspective of the modality gap. In existing research on multi-modal model training and downstream task adaptation, the modality gap is often regarded as a source of sub-optimal performance. Researchers focus on reducing the modality gap in pre-trained multi-modal models. However, in the context of using CLIP for continual learning, the goal of this embodiment is to retain the strong generalization ability of the model while learning new data. Therefore, how to handle the modality gap in continual learning remains an open problem, and the work of this embodiment is based on the assumption that this modality gap reflects the inherent knowledge of the pre-trained model.

[0054] Downstream datasets are usually much smaller than the pre-training dataset, making them insufficient to fully train the model. Therefore, modifying the basic properties of the pre-trained model according to the downstream data, such as the modality gap, may destroy the pre-trained knowledge, which may affect the performance of continual learning. During the training process of the current task, the cross-entropy optimization objective tends to widen this gap, and this phenomenon will intensify with the continuous arrival of incremental learning data. In addition, directly aligning the two modalities of the downstream task will also significantly change the pre-trained knowledge. Therefore, this embodiment keeps the modality gap relatively stable, thus maintaining the stability of the model. This embodiment analyzes the change of the modality gap during the training process and proposes an adjustment strategy that perceives the modality gap: by tracking the change of the modality gap, adjusting the training process to maintain a stable modality gap, so as to retain the pre-trained knowledge and reduce forgetting; in addition, the impact of retaining the modality gap is also analyzed.

[0055] As Figure 2 shown, when using text features as the classifier, the modality gap may limit the model's ability to learn new data in continual learning, which limits the model's adaptability and reduces plasticity. To solve this problem, this embodiment proposes to build a classifier in the visual space where the modality gap does not pose a limitation. By integrating the output of this classifier with the output of the text classifier, the modality gap can be compensated, thus further improving the learning ability of CLIP.

[0056] (1) Class-incremental learning.

[0057] Class-incremental learning methods are generally divided into three categories. Regularization methods limit the changes to the model. Some methods use distillation to reduce the bias of the model features or penalize the changes in the model parameters. Dynamic network methods allow the model structure to evolve with the introduction of new tasks. Replay-based methods retain the original samples or obtain relevant information from them. When learning a new task, these methods replay the original samples or recover the old samples from the retained information. The latter usually involves techniques such as memory compression or feature replay. Dynamic network or feature replay methods tend to increase the number of parameters and memory requirements. With the widespread use of pre-trained models, incremental learning methods for visual pre-trained models have also emerged. Parameter-efficient fine-tuning methods gradually expand a small number of parameters as the tasks increase. Ape maintains stability by only training the model on the first task. SLC uses a small learning rate to fine-tune the backbone network for continual learning. They show that maintaining the stability of the pre-trained visual model can enhance generalization and is beneficial to class-incremental learning of downstream tasks.

[0058] (2) CLIP-based class-incremental learning.

[0059] Without specific training, CLIP performs very well in class-incremental learning tasks with downstream data. Continuous learning based on CLIP has attracted increasing attention from researchers. Some methods add learnable modules to the original CLIP features to modify them for better adaptation to new tasks. PROOF and CLAP add learnable modules to the CLIP output features to promote cross-modal interaction. RAPF introduces a linear layer after the CLIP visual encoder for downstream adaptation and uses text-modal information to guide feature replay. Other methods fine-tune CLIP to change its output features. ZSCL extracts additional datasets during training. MOE4CL fine-tunes CLIP with experts and introduces a selection mechanism to selectively use the original CLIP model. Magmax fine-tunes the entire CLIP model sequentially and uses the task vector algorithm to merge models during inference. These methods mainly focus on leveraging the prior knowledge of natural language to assist continuous learning and often neglect the retention of zero-shot capabilities when emphasizing the performance of sequential tasks. Focus on retaining the capabilities of the model while enhancing its continuous learning ability, starting from the inherent properties of CLIP.

[0060] (3) Modal gap.

[0061] In contrastive vision-language models, the modal gap is reflected in the fact that text and image features lie in two separate narrow cones. Features of different modalities exhibit obvious separability, while features within the same modality tend to cluster more closely. Existing research has explored the impact of the modal gap on tasks such as domain adaptation, few-shot classification, model pre-training, and retrieval tasks. Narrowing the modal gap can improve the performance of the model in these tasks. However, in continuous learning tasks, the stability of pre-trained knowledge over time is more critical for downstream tasks. Therefore, this embodiment proposes using the modal gap as an indicator of the change in the feature space of the pre-trained model, with the goal of maintaining the modal gap in consecutive tasks.

[0062] (4) Definition of class-incremental learning.

[0063] This embodiment considers a class-incremental learning setting based on the pre-trained CLIP model M, with the goal of sequentially training the model on a series of classification tasks. Each task t consists of a set of classes C t such that the classes between tasks do not overlap, and during the training of task t, the model has no access to information related to the previous t - 1 tasks. After training on the t-th task, the model M t is required to correctly classify all previously learned classes C1 ∪ C2 ∪ … ∪ C t without access to the task identifier.

[0064] (5) Modal gap measurement.

[0065] In the classification task, given N image and K text class names, the cross-modal similarity is measured as the average cosine similarity between all image and text features: where x i is the image feature and t j is the text feature, which captures the overall similarity between the image and text modalities and reflects the modality gap in the feature space. To analyze the impact of image-text similarity in a more fine-grained manner, the average similarity of positive image-text pairs is defined as:

[0066]

[0067] where t yi represents the class text feature corresponding to the image x i .

[0068] Similarly, the average similarity of negative image-text pairs is measured as:

[0069]

[0070] where represents the text feature of the non-matching class of the image x i .

[0071] In this embodiment, from the perspective of modality, the continuous learning based on CLIP is studied, and its key role in maintaining the generalization ability of CLIP in continuous learning is recognized. Uncontrolled training can disrupt this inherent modality gap, leading to suboptimal performance in downstream incremental learning. To address this issue, this embodiment proposes a two-stage strategy: ① Retain the modality gap to maintain model stability, regularize the training process to ensure a stable modality gap across tasks; ② Compensate for the modality gap to improve model plasticity. The constraint of the modality gap may limit the adaptability of the model. This embodiment introduces a complementary classifier to improve the performance of specific tasks without changing the retained modality gap. During the inference process, the two outputs are integrated to balance stability and task plasticity.

[0072] (6) The role of modality gap in continuous learning.

[0073] First, analyze the evolution of the modality gap during the continuous learning process.

[0074] The cross-entropy loss enlarges the modality gap. The optimization objective of cross-entropy is inconsistent with the modality gap of the original segments. When optimizing through the cross-entropy loss, the training objective aligns the cosine similarity of matching image-text pairs to 1 and non-matching pairs to -1. However, this optimization contradicts the moderate similarity distribution of the original CLIP model, which is between approximately 0 and 0.3, reflecting the inherent modality gap. This optimization leads to an enlarged modality gap in the trained model.

[0075] In addition, downstream classification tasks introduce a structural imbalance that is lacking in CLIP pre-training. Different from the symmetric text-image pairs in pre-training, classification tasks pair multiple images of the same class with a single text embedding. In a dataset with C classes, each text only forms a positive pair with an image, while being mutually exclusive with the remaining negative pairs. The imbalance causes the optimization process to mainly reject non-matching samples, further enlarging the modality gap. The modality gap implicitly reflects the pre-trained knowledge, and enlarging the modality gap may lead to forgetting previous knowledge. As Figure 3 shown, the experimental results confirm this effect: as the task progresses, the cosine similarity between image and text representations steadily decreases, showing an increasingly large modality gap. This coincides with the continuous decline in accuracy, highlighting its adverse impact on CLIP's pre-trained knowledge and overall performance in class-incremental learning.

[0076] The direct alignment loss reduces the modality gap but disrupts the pre-trained knowledge. As Figure 4 shown, introducing an alignment loss in downstream tasks can reduce the modality gap by minimizing the Euclidean distance between matching image and text features. The key problem lies in the subspace mismatch: the downstream dataset spans a limited region of the original CLIP feature space, making direct alignment at risk of over-specialization. Directly aligning the two modalities significantly changes the pre-trained representation, resulting in forgetting.

[0077] Maintaining CLIP's lifelong learning ability. Previous analyses have revealed a phenomenon: simple fine-tuning often enlarges the modality gap, while direct alignment reduces it. As class-incremental learning progresses, this problem becomes increasingly severe, gradually eroding CLIP's pre-training ability. Since CLIP's pre-trained knowledge plays a crucial role in providing stability for downstream tasks, it is necessary to maintain a relatively stable modality gap. Ensuring this stability is crucial for allowing CLIP to incorporate new knowledge without affecting its fundamental visual-language correspondences.

[0078] (7) Adaptive modality gap maintenance.

[0079] In this embodiment, the asymmetric evolution of the modality gap in continuous learning is described, and an adaptive modality preservation strategy is proposed to enhance the stability of the model.

[0080] The asymmetric phenomenon of the change in the modality gap during the training process. As Figure 5 and Figure 6 shown, an asymmetric change in the cosine similarity between text and image features was observed during single-task training. pos and neg are calculated using Equation 1 and Equation 2. Before training, i.e., at epoch 0, pos was already larger than neg, indicating the zero-shot ability of the model. Even though the similarity between the image and its corresponding class text was still relatively low, which is a reflection of the modality gap.

[0081] In the early stage of training, the change in the output was mainly due to the increase in the similarity of the positive image-text pairs, indicating that the model learned new knowledge and became more confident in correct classification. This corresponds to the part before the green dashed line in Figure 5 and Figure 6 As long as the positive output was still greater than the negative output, the model could still make correct predictions even in the presence of loss. However, in the later stage of training, the positive output tended to stabilize and the negative output began to decrease. This indicates that the distance between the image and most other text features increased. Therefore, the optimal training stage can retain the knowledge it has learned while keeping the inter-modal distance relatively stable.

[0082] Adaptive training to maintain the modality gap. Based on these observations, we propose to determine the number of training epochs by monitoring the average negative output. As Figure 1 shown, using the data of the first task of the dataset, the number of epochs required for subsequent tasks is estimated. Specifically, in this embodiment, the average value of the negative class output is first calculated using Equation 2 (denoted as neg 0 ), and the original CLIP model is used for the first task; then, the model is trained using LoRA, and after the e-th epoch, the neg e of the negative output of all data is calculated.

[0083] Among them, the relative difference between the current and the original is calculated as follows:

[0084]

[0085] When the difference Δ exceeds the predefined threshold α, the value e of the previous epoch is recorded, where Δ is still lower than α. Finally, for all tasks in the downstream dataset, the model is trained with epoch = max(e, 1).

[0086] (8) Inter-modal compensation to bridge the modality gap.

[0087] In this embodiment, it is explained how the modality gap limits the capabilities of the model, and this gap must be maintained. Therefore, an intra-modal classifier is constructed to compensate for these limitations.

[0088] Limitations of the modality gap on the capabilities of the text classifier. To achieve the minimum cross-entropy loss, there must exist an optimal classifier in the image feature space. Specifically, for any classifier that minimizes the classification error, there exists an equivalent form W opt , which is completely contained within the spanned space of the image features. This conclusion stems from the decomposition of any classifier into components parallel and orthogonal to the image space, where the orthogonal component makes no contribution to classification.

[0089] In view of this, this embodiment analyzes the impact of the modality gap on the text classifier. The text feature matrix T can be decomposed as: T = T ∥ + T ⊥ , where T ∥ lies within the subspace spanned by the visual features X, while T ⊥ is orthogonal to it. Since text features usually cannot fully span the image feature space, the best achievable text classifier is restricted to a low-rank subspace, resulting in an alignment error. The lower bound of the distance from T ∥ to the optimal image space classifier is determined by the singular value s 2 outside the text feature subspace:

[0090]

[0091] where r′ is the rank of W opt , and r is the rank of T ∥ .

[0092] This result reveals an inherent limitation: perfect alignment cannot be achieved unless the text feature space has sufficient capacity to represent the optimal classifier. Due to the existence of the modality gap, text classifiers typically operate in a low-rank subspace, which limits their classification performance.

[0093] Compensation for the modality gap through an intra-modal classifier. To compensate for the modality gap, an auxiliary classifier is introduced in the visual space. The fine-tuned CLIP model f clip (·) and the classifier weights for the old classes are kept frozen. For the classes introduced in the current task, their classifier weights are initialized in the cosine classifier W v using their class prototypes, and they are trained using image features (without using text). Since the gradients of the classifier are kept within the input space (i.e., the visual space), this ensures that the classifier operates within the visual subspace.

[0094] As Figure 1As shown, in the model inference stage, the prediction results from the text and visual classifiers are combined, and the final prediction score is calculated as follows:

[0095]

[0096] where β is a constant hyperparameter and t is the text description of all the classes to be classified.

[0097] An image classification method for preserving and compensating for the modality gap provided by this embodiment, as Figure 1 shown, includes the following steps:

[0098] S1. Prepare the input dataset D = {X1, X2, …, X T}, which contains T tasks (Task1, Task2, …, Task T ), and each task Task t corresponds to a task dataset X t composed of several pieces of training data. Each piece of training data contains an image and the text of the class name.

[0099] S2. Let e = 0, and load the original pre-trained CLIP model f clip (·) as the base model for subsequent fine-tuning.

[0100] Among them, the CLIP model includes a text encoder and an image encoder.

[0101] The goal is to obtain the CLIP model fine-tuned continuously on T tasks and train an auxiliary classifier W to compensate for the modality gap.

[0102] Among them, Eq.2 represents formula (2).

[0103] S3. At the first task t = 1, fine-tune the backbone of the CLIP model using the current task dataset X1.

[0104] That is, Among them, represents the model after fine-tuning for e + 1 epochs through the first task at the epoch, and FINETUNE represents fine-tuning.

[0105] S4. Through formula (2), calculate the negative class output of the model at different epochs (i.e., e) to measure the modality gap, and use equation (3) to calculate the change amount △ of the modality gap relative to the original model.

[0106] S5. Determine whether the change amount △ of the current modal gap exceeds the selected threshold. If it exceeds, select the previous epoch (i.e., e - 1) as the fixed fine-tuning epoch for all subsequent tasks, i.e., e = max(1, e); otherwise, increment e by 1 and return to step S3.

[0107] S6. For other tasks where t > 1, fine-tune the CLIP model according to the epoch determined by the first task to keep the modal gap stable.

[0108] S7. When the CLIP backbone network is fine-tuned on the dataset X of task t t and completed, freeze its parameters and train the auxiliary cosine classifier W in the second stage, i.e., The auxiliary classifier W freezes the weights of the old tasks, only trains the weights of the new tasks, and only accepts visual modality inputs to compensate for the modal gap maintained during the fine-tuning process.

[0109] S8. Repeat steps S6 - S7 until all tasks are completed to obtain the finally fine-tuned CLIP model and the optimized auxiliary classifier.

[0110] S9. In the inference stage, combine two different classifier subspaces for prediction.

[0111] In the model inference stage, combine the prediction results from the text and visual classifiers. The final prediction score is calculated as follows:

[0112] Dataset. In this embodiment, our method is evaluated through continual learning tasks on five benchmark datasets: CIFAR-100, ImageNet-, ImageNet-10, ImageNet-1, and VTAB. All datasets except VTAB are evenly divided into 10 consecutive tasks. Following previous work, a subset is extracted from VTAB, containing 5 tasks, and each task contains ten categories.

[0113] Comparison algorithms. The experiments in this embodiment compare two types of methods: (1) pure vision methods, including L2P++, DualPromp, CODA, SLCA, and Aper-Adapte; (2) CLIP-based methods, including PROOF, CLAP, RAP, MOE4C, MagMax, and the zero-shot CLIP baseline Continual-CLI. All methods default to using the OpenAI's ViT-B / 16 weights. Although the original MagMax implementation uses enhanced data augmentation and optimized text templates, for fair comparison, following previous work, basic image augmentation and fixed prompt templates are used for standardized evaluation.

[0114] Evaluation metrics. After training the t-th task, the average accuracy of the test data of the first t tasks is denoted by A t . "Avg" represents the average of the accuracies of all tasks, that is "Last" represents the average accuracy after completing the last task T, that is A T .

[0115] Implementation details. The method of this embodiment was developed on an A40 GPU. Unless otherwise specified, the pre-trained CLIP model provided by OpenAI was used, specifically the ViT-B / 16 version. When training the backbone network, LoRA (Low-Rank Adaptation) was applied to adapt the model, and its default rank was set to 8. Since the focus of the work is not on the fine-tuning process itself, the implementation was simplified by applying LoRA only to the key and value parts of the attention module. Different fine-tuning model implementations are left for future work. The Adam optimizer was used, and a cosine learning rate scheduler was adopted, with an initial learning rate of 0.001. The number of training epochs was determined by the first part of the method, where the threshold α was set to 10%. In the training phase of the image space classifier, we trained for 3 epochs with an initial learning rate of 0.0005. The classifier used a cosine classifier. In the inference phase, the hyperparameter β for integrating the two classification results was default set to 4. The impact of the hyperparameters was described in the supplementary materials.

[0116] Table 1. Comparison results

[0117]

[0118] Table 1 shows the performance comparison of different methods. DR represents using real data for replay, FR represents generating features of old classes for replay, and NR represents not using replay. The results mainly come from the references and were reproduced using their open-source code. The performance of the method of this embodiment is based on the average of three different class orders. Except that VTAB was divided into 5 tasks, other datasets were divided into 10 tasks. CLAP* represents the non-replay version provided in the CLAP paper.

[0119] Table 1 shows the comparison of the method of this embodiment with other methods. On most datasets, the method of this embodiment outperforms all other methods, including replay-dependent methods. Specifically, on CIFAR-100, the method of this embodiment is at least 1.53% higher than all competitors in terms of the "Last" accuracy. On ImageNet-R, the method of this embodiment has at least increased by 1.82% in terms of the "Last" accuracy. On ImageNet-100, although the method of this embodiment is slightly inferior to RAPF and CLAP (both of which use replay), the method of this embodiment does not rely on replay and is still at least 1.72% higher than all non-replay methods in terms of the "Last" accuracy. In addition, on ImageNet-100, the performance of CLAP* is significantly lower than that of CLAP, indicating that the performance improvement of CLAP largely comes from data replay. On the larger-scale ImageNet-1K dataset, the method of this embodiment has achieved results equivalent to or slightly better than the best alternative method.

[0120] The VTAB dataset poses a major challenge to CLIP, as can be seen from the zero-shot accuracy (i.e., Continual_CLIP, only 31.5%). In this challenging cross-domain benchmark, the performance of most methods has dropped significantly. The method of this embodiment is 1.86% higher than the best replay-based CLAP method in terms of the "Last" accuracy, and significantly outperforms its non-replay version. This shows that the method of this embodiment has obvious advantages in such challenging cross-domain scenarios.

[0121] Table 2. Comparison of zero-shot capabilities

[0122]

[0123] Table 2 shows the zero-shot performance of the model on different downstream datasets after completing all class incremental learning tasks on CIFAR-100. CLIP refers to the original CLIP pre-trained model without any downstream task-specific training.

[0124] The goal of continual learning is to enable the model to retain existing capabilities while gradually acquiring new knowledge. Compared with traditional pre-trained visual models, the CLIP model demonstrates superior zero-shot generalization ability. Therefore, continual learning based on CLIP should not only minimize forgetting on new downstream tasks but also retain its original zero-shot ability. This ensures that continual learning based on CLIP is not just a task-specific initialization method that leads to overfitting, which is a limitation of previous work in this field. These works usually follow traditional evaluation protocols and neglect the retention of the model's inherent capabilities. This embodiment proposes to evaluate the zero-shot generalization ability on an independent benchmark after fine-tuning to address this issue.

[0125] As shown in Table 2, after continuous learning on CIFAR-100, zero-shot performance is evaluated on three standard datasets. The method of this embodiment is slightly better than the original CLIP on Food101 and Oxford Pet, which are significantly different from CIFAR-100, while all baseline methods show a performance decline. On the more similar ImageNet-1K, the three methods outperform the original CLIP, and the method of this embodiment achieves the most significant improvement (an increase of 2.85%). These results indicate that the method of this embodiment successfully retains the pre-trained knowledge while effectively integrating new information.

[0126] It is worth noting that the performance decline of replay-based methods (PROOF, RAPF, and CLAP) in zero-shot tasks is more significant than that of non-replay methods. This indicates that the replay mechanism may lead to overfitting to downstream tasks, while non-replay methods must retain the original representation to prevent forgetting. This further highlights the possible negative impact of the replay strategy on pre-trained models.

[0127] Training cost analysis.

[0128] Taking CIFAR-100 as an example, the parameter overhead of different methods is analyzed by comparing the additional learnable parameters introduced by the method of this embodiment and other baselines, as Figure 7 shown. The method of this embodiment only introduces 0.54M additional trainable parameters. Although the RAPF method requires fewer learnable parameters, it still needs to store a covariance matrix for each category, resulting in additional storage consumption proportional to the square of the feature dimension, i.e., nd 2 . For example, for 100 categories, this will result in an additional storage overhead of more than 26M parameters. In contrast, the method of this embodiment completely avoids replay and maintains the minimum storage consumption.

[0129] During the training process, the only additional cost of the method of this embodiment occurs after each training epoch of the first task, where an additional forward pass of the data of the first task is performed to evaluate the change in the modality gap. However, this step only needs to be performed in the first task, does not involve backpropagation, and is negligible compared to the training cost.

[0130] Table 3, Ablation experiments

[0131]

[0132] Table 3 shows the ablation study of the modules on ImageNet-100. MGP refers to the adaptive modality gap preservation of this embodiment, while MGC represents the intra-modal compensation to make up for the modality gap.

[0133] Table 3 shows the ablation study of the proposed module in this embodiment. The baseline setting involves a simple fine-tuning of the CLIP model for each task. Following the settings of previous work, each task was fine-tuned for 10 epochs, and the widely adopted cross-entropy loss and the old-class output mask were used. MGC represents modal gap compensation, while MGP represents modal gap preservation. As shown in Table 3, both components of the method in this embodiment can bring performance improvements when used alone, and the modal gap preservation (MGP) provides a more significant improvement. When the two components are used together, the best performance is achieved.

[0134] When MGC is used alone in the baseline setting, the improvement in the "Last" accuracy is 1.84%, which is greater than the improvement (1.52%) when MGC is added on top of MGP. A similar trend was also observed for the "Avg" accuracy. This indicates that the simple fine-tuning in the baseline setting enlarges the modal gap, and thus the role of MGC in bridging this gap becomes more prominent.

[0135] Image space and classification space. The relationship between the image feature space, the text classifier space, and the classifier space is studied by examining their linear structures. To quantify the differences between them, matrix factorization is applied to extract the orthonormal bases of each space. B i : The orthonormal basis matrix of the image feature space; B t : The orthonormal basis of the text classifier; B vc : The orthonormal basis of the visual space classifier; B t+vc : The combined space spanned by the two classifiers. To measure the extent to which the text classifier space B t covers the image feature space B t , calculate the average norm of the orthogonal components of B i with respect to B t :

[0136]

[0137] This metric reflects how much of B i lies outside of B i : If the spaces are orthogonal, this value equals 1; if one space is a subspace of the other, this value equals 0. Similarly, calculate d(B i ,B vc ) and d(B i ,B t+vc ) to evaluate the extent to which the visual classifier and the combined classifier spaces capture the image features.

[0138] Table 4. Difference metrics between the image subspace and different classifier subspaces on different datasets

[0139]

[0140] As shown in Table 4, due to the existence of the modality gap, the text classifier space significantly deviates from the image feature space. In contrast, the visual classifier is better aligned with the image space. Combining the two classifiers can further improve the coverage of the image space and highlight their complementary roles.

[0141] Table 5. Comparison of cosine similarity and last accuracy (Last Acc) under different experimental settings on ImageNet-100

[0142]

[0143] Analyze the impact of the method of this embodiment on the modality gap. The impact of the method of this implementation on the modality gap was analyzed. Table 5 shows the average cosine similarity between the positive and negative samples of the final task and the last accuracy (Last Acc) under different experimental settings. "Base" refers to the result of simple fine-tuning. It can be observed that compared with the original CLIP, the average similarity of the negative samples has decreased significantly. This indicates that the modality gap is enlarged, resulting in a decrease in the last accuracy. "Distill" represents the traditional distillation method, which explicitly limits the amplitude of the model output. It limits the change of the modality gap and achieves some positive effects. However, completely restricting the modality gap will obviously lead to a decline in the learning ability of the model, resulting in suboptimal performance. It can be seen that its performance is close to the original CLIP. In contrast, the method of this implementation maintains a relatively stable modality gap, allowing a moderate increase in the positive sample similarity and a moderate decrease in the negative sample similarity. This balance ensures that the model can retain the pre-trained knowledge while appropriately learning new knowledge, ultimately achieving optimal performance.

[0144] This implementation studied the impact of the modality gap on the performance of the vision-language pre-trained model in class incremental learning. It was found that maintaining a relatively stable modality gap helps to retain the pre-trained knowledge and prevent its degradation. Under the condition of a stable modality gap, training a visual space classifier not restricted by the modality gap can compensate for some of its negative impacts, thereby further enhancing the model's ability. The experimental results prove the effectiveness of the method of this implementation.

[0145] Embodiment 2

[0146] This embodiment provides an image classification system for retaining and compensating the modality gap, which specifically includes:

[0147] A data acquisition module, which is configured to: acquire a number of tasks, each task corresponding to a task data set composed of a number of pieces of training data, and each piece of training data includes a piece of text and an image;

[0148] A parameter determination module, which is configured to: for the first task, fine-tune the CLIP model using the first task dataset at different epochs, and calculate the change in the negative class output to measure the change in the modality gap. When the change in the modality gap exceeds the selected threshold, fix the fine-tuning epoch.

[0149] A fine-tuning module, which is configured to: traverse any task other than the first task, fine-tune the CLIP model using the corresponding task dataset according to the fixed fine-tuning epoch, and then train an auxiliary cosine classifier, and the auxiliary cosine classifier only accepts the visual modality input obtained by the CLIP model.

[0150] An inference module, which is configured to: obtain the text and image to be classified, and obtain the classification result through the fine-tuned CLIP model and the auxiliary cosine classifier.

[0151] Further, the negative class output is:

[0152]

[0153] where x i is the image feature, represents the text feature of the non-matching class of image x i , N is the number of images, and K is the number of texts.

[0154] Further, the change in the modality gap is:

[0155]

[0156] where neg e represents the negative class output after the e-th epoch, and neg 0 represents the initial negative class output.

[0157] Further, during the training process of the auxiliary classifier, freeze the weights of the old tasks and only train the weights of the new tasks.

[0158] It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1 one by one, and the specific implementation process is the same, so it will not be repeated here.

[0159] Embodiment 3

[0160] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps in a method for image classification that preserves and compensates for the modality gap as described in Embodiment 1 above.

[0161] Embodiment 4

[0162] This embodiment provides a computer device, such as Figure 8 shown, including a display device, an input device, a computer-readable storage medium (volatile memory and non-volatile storage medium), a processor, a communication interface (i.e., a network interface), and a computer program stored on the computer-readable storage medium and executable on the processor. Among them, the processor, the communication interface, and the computer-readable storage medium can be connected through a bus or other means. Among them, the communication interface is used to receive and send data, and when the processor executes the program, it implements the steps in an image classification method for retaining and compensating for modal gaps as described in the first embodiment above.

[0163] Among them, any reference to a memory, storage, database, or other medium provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or an external cache. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0164] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in Figure 1 one or more of the processes or multiple processes and / or blocks Figure 1 one or more of the blocks or multiple blocks.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions in the processFigure 1 one process or multiple processes and / or blocks Figure 1 the functions specified in one block or multiple blocks

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple processes and / or the steps of the functions specified in one block or multiple blocks

[0167] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image classification method for retaining and compensating for modal gaps, characterized in that, Including: Obtain a number of tasks, each task corresponding to a task dataset composed of a number of pieces of training data, and each piece of training data includes a piece of text and an image; For the first task, at different epochs, fine-tune the CLIP model using the first task dataset, and calculate the change in the negative class output to measure the change in the modality gap. When the change in the modality gap exceeds the selected threshold, fix the fine-tuning epoch; Traverse any task except the first task. After fine-tuning the CLIP model using the corresponding task dataset according to the fixed fine-tuning epoch, train an auxiliary cosine classifier, and the auxiliary cosine classifier only accepts the visual modality input obtained by the CLIP model; Obtain the text and image to be classified, and obtain the classification result through the fine-tuned CLIP model and the auxiliary cosine classifier.

2. The image classification method for retaining and compensating for modal gaps as described in claim 1, wherein, The negative class output is: where x i is an image feature, represents the text feature of the non-matching class of image x i , N is the number of images, and K is the number of texts.

3. The image classification method for retaining and compensating for modal gaps as described in claim 1, characterized in that, The change in the modality gap is: Among them, neg e represents the negative class output after the e-th epoch, and neg 0 represents the initial negative class output.

4. The image classification method for retaining and compensating for modal gaps as described in claim 1, characterized in that During the training process of the auxiliary classifier, freeze the weights of the old tasks and only train the weights of the new tasks.

5. An image classification system for retaining and compensating for modal gaps, characterized in that, Including: A data acquisition module, which is configured to: obtain a number of tasks, each task corresponding to a task dataset composed of a number of pieces of training data, and each piece of training data includes a piece of text and an image; A parameter determination module, which is configured to: for the first task, at different epochs, fine-tune the CLIP model using the first task dataset, and calculate the change in the negative class output to measure the change in the modality gap. When the change in the modality gap exceeds the selected threshold, fix the fine-tuning epoch; A fine-tuning module, which is configured to: traverse any task except the first task. After fine-tuning the CLIP model using the corresponding task dataset according to the fixed fine-tuning epoch, train an auxiliary cosine classifier, and the auxiliary cosine classifier only accepts the visual modality input obtained by the CLIP model; An inference module, which is configured to: obtain the text and image to be classified, and obtain the classification result through the fine-tuned CLIP model and the auxiliary cosine classifier.

6. An image classification system for retaining and compensating for modal gaps as described in claim 5, characterized in that The negative class output is: Among them, x i is an image feature, indicating the text feature of the non-matching class of image x i where N is the number of images and K is the number of texts.

7. An image classification system for retaining and compensating for modal gaps as claimed in claim 5, characterized in that The change in the modality gap is: Among them, neg e represents the negative class output after the e-th epoch, and neg 0 represents the initial negative class output.

8. An image classification system for retaining and compensating for modal gaps as claimed in claim 5, characterized in that, During the training process of the auxiliary classifier, freeze the weights of the old tasks and only train the weights of the new tasks.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps in an image classification method for retaining and compensating for modality gaps as described in any one of claims 1-4.

10. A computer device, comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in an image classification method for retaining and compensating for modality gaps as described in any one of claims 1-4.

Images