A method for alleviating double forgetting of visual language model and an evaluation method

Abstract

This invention proposes a continuous learning method and evaluation method to mitigate the double forgetting of visual language models. The method includes: adding two parallel paths to the backbone network of a pre-trained visual language model—one a task-agnostic expert path and the other a task-related expert group path—to obtain a model based on heterogeneous expert hybridization; training the model using a two-stage training strategy, including: a first stage, freezing the task-related expert group path and training only the task-agnostic expert path, using contrastive learning to solidify the pre-trained general knowledge of the visual language model; a second stage, freezing the task-agnostic expert path and training only the task-related expert group path, using a cross-entropy loss function to learn task-specific knowledge; and an inference stage, dynamically fusing the outputs of the two parallel paths through a hierarchical routing mechanism to achieve collaborative learning of pre-trained knowledge and task-specific knowledge.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and deep learning technology, specifically involving the intersection of computer vision and natural language processing, and in particular a dual-forgetting continuous learning method and evaluation method for mitigating visual language models. Background Technology

[0002] Visual language models, exemplified by the CLIP model, have achieved unprecedented zero-shot generalization capabilities through pre-training on massive amounts of image and text data, enabling them to understand and process visual concepts not encountered during the training phase. However, in practical applications, we often need to fine-tune these powerful foundational models for specific downstream domains or tasks to learn new expertise. This process is known as continuous learning.

[0003] The core objective of continuous learning is to efficiently add new knowledge to the model without resorting to costly, fully pre-training. However, this process faces a significant challenge: "catastrophic forgetting." For visual language models, this forgetting manifests as a "double forgetting" phenomenon:

[0004] Incremental Knowledge Forgetting (IKF): This is a typical form of forgetting in traditional continuous learning, referring to the model forgetting the knowledge of previously learned tasks after learning a new task.

[0005] Pre-trained knowledge forgetting (PKF): This is a more subtle form of forgetting unique to visual language models during continuous learning. It refers to the gradual erosion and destruction of the broad and general zero-shot generalization ability that the model learned during the pre-training stage as it is fine-tuned to adapt to new tasks.

[0006] Current research on continuous learning methods is fragmented and fails to comprehensively address the double forgetting problem. On the one hand, most standard Class-Incremental Learning (CIL) methods focus solely on mitigating IKF (Individual Knowledge Forgetting) and completely ignore PKF (Primary Knowledge Forgetting). This one-sided evaluation can create a false sense of progress, where the model appears to have learned new categories, but its core generalization ability is unknowingly and severely degraded. On the other hand, the few frameworks that do address PKF, such as multi-domain task incremental learning, suffer from biased evaluation protocols. They typically assume that the dataset used to evaluate zero-shot performance is "completely unseen" by the model. However, when the newly learned task is conceptually highly similar to an evaluation dataset, this "unseen" assumption no longer holds. The model's performance on that dataset is more attributable to the transfer of newly learned knowledge than to the retention of its original pre-trained knowledge. This evaluation bias caused by similarity masks the true extent of PKF.

[0007] Therefore, there is an urgent need for a technical solution that can comprehensively and accurately assess and effectively alleviate the double forgetting problem of visual language models. Summary of the Invention

[0008] Purpose of the invention: To address the double forgetting problem in the continuous learning process of existing visual language models, as well as the one-sidedness and bias of evaluation methods, this invention proposes a continuous learning method for mitigating double forgetting in visual language models. This method enables the model to efficiently learn new knowledge while effectively suppressing the forgetting of incremental knowledge and pre-trained knowledge.

[0009] Technical solution: This invention proposes a continuous learning method to alleviate the double forgetting of visual language models, including:

[0010] By adding two parallel paths to the backbone network of the pre-trained visual language model—one being a task-agnostic expert path and the other a task-related expert group path—a model based on heterogeneous expert hybrid is obtained.

[0011] The model based on heterogeneous expert hybrid training is trained using a two-stage training strategy, including:

[0012] In the first stage, by freezing the expert group paths related to the task and training only the expert paths unrelated to the task, we consolidated the pre-trained general knowledge of the pre-trained visual language model through comparative learning.

[0013] In the second stage, task-independent expert paths are frozen, and only task-related expert group paths are trained. The cross-entropy loss function is used to learn specific knowledge of specific tasks.

[0014] During the inference phase, the outputs of two parallel paths are dynamically fused through a hierarchical routing mechanism to achieve collaborative learning of pre-trained knowledge and task-specific knowledge.

[0015] Furthermore, the task-agnostic expert path includes a separate task-agnostic expert module; by minimizing the contrastive loss, this separate task-agnostic expert module is incentivized to focus on pre-trained general knowledge.

[0016] Furthermore, the task-related expert group path includes multiple task-related expert modules and an internal router; the internal router is used to calculate the weight of each task-related expert module according to the input features, and to perform a weighted summation of the outputs of each task-related expert module to obtain the aggregated output of the task-related expert group path.

[0017] Furthermore, the hierarchical routing mechanism is implemented through an external router, which is used to calculate the fusion weight of the output of the task-independent expert path and the output of the task-related expert group path, and to perform weighted fusion of the outputs of the two paths.

[0018] This invention proposes a model evaluation method, comprising:

[0019] Construct a dataset set that includes multiple different domains, and divide the dataset set into a source task set for incremental class learning and an evaluation pool for evaluating zero-shot capability;

[0020] The source dataset is further divided into a task sequence;

[0021] A continuous learning approach is adopted to learn tasks in a task sequence sequentially. After each task is completed, the forgetting of specific knowledge of all historical tasks is evaluated. An evaluation pool is also used to evaluate the forgetting of general knowledge of pre-training, thereby obtaining the evaluation results.

[0022] The continuous learning method described above is a continuous learning method for mitigating the double forgetting of visual language models.

[0023] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0024] (1) Comprehensive solution to double forgetting: The method of this invention clearly proposes and systematically solves the problem of double forgetting (IKF and PKF) in the continuous learning of visual language models. Compared with the traditional method that only focuses on IKF, the method of this invention can better maintain the comprehensive ability of the model;

[0025] (2) More accurate and reliable evaluation: The evaluation method and similarity-aware transfer metric proposed in this invention take into account task similarity bias for the first time in the evaluation process, which can more realistically reflect the degree of forgetting of model pre-training knowledge and provide a more rigorous evaluation benchmark for related research.

[0026] (3) Efficient architecture design: a continuous learning method to alleviate double forgetting in visual language models. The method achieves specialized processing of knowledge retention and knowledge acquisition through heterogeneous expert design with functional decoupling, and strengthens this functional separation through a two-stage training strategy, thus achieving an industry-leading balance in solving the double forgetting problem.

[0027] (4) Efficient parameters and computation: The method of this invention is based on efficient parameter fine-tuning technology. Only a small number of parameters (expert module and router) need to be updated during the training process, which greatly reduces the computation and storage overhead while maintaining high performance. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a continuous learning method for mitigating double forgetting in visual language models proposed in this invention.

[0029] Figure 2 This is a radar chart comparing the final performance of the continuous learning method for mitigating double forgetting in visual language models proposed in this invention with various mainstream baseline methods in 22 CIL scenarios. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0031] Example 1:

[0032] like Figure 1 As shown in the figure, this invention proposes a continuous learning method to alleviate the double forgetting of visual language models, including:

[0033] By adding two parallel paths to the backbone network of a pre-trained visual language model—one a task-agnostic expert path and the other a task-related expert group path—a heterogeneous expert-based (DFA-MoE) model is obtained. The backbone parameters of the pre-trained visual language model's backbone network remain frozen throughout the process, and all trainable parameters are concentrated in the lightweight expert modules within the two parallel paths.

[0034] In this embodiment of the invention, the task-independent expert path includes a separate task-independent expert module. Its sole objective is to mitigate PKF; by minimizing the contrastive loss, this independent task-agnostic expert module is incentivized to focus on pre-trained general knowledge. It is a single expert module. Its function is "stabilization," that is, maintaining the model's general knowledge. During training for each new task, this expert module optimizes through a contrastive learning objective, forcing the model to maintain the original text-image feature alignment. To enhance learning performance, a momentum queue is also introduced to provide abundant negative samples.

[0035] In this embodiment of the invention, the task-related expert group path includes multiple task-related expert modules. and an internal router Its goal is to mitigate IKF. The role of the Task-Related Expert Group (TRIG) path is to learn task-specific knowledge for new tasks. The internal router is used to calculate weights for each TRIG based on the input features and to perform a weighted summation of the outputs of each TRIG, resulting in the aggregated output of the TRIG path.

[0036] In this embodiment of the invention, an external router is used to calculate the fusion weight of the output of the task-independent expert path and the output of the task-related expert group path, and the outputs of the two paths are weighted and fused.

[0037] For each new incremental task, a two-stage training strategy is employed to train the model based on heterogeneous expert hybrid training, including:

[0038] The first stage is used to suppress PKF (Physical Classification of Faults). By freezing task-related expert group paths and training only task-independent expert paths, contrastive learning is used to solidify the pre-trained general knowledge of the pre-trained visual language model. The training objective function is a symmetric InfoNCE contrastive loss. For the current task... Each image and text pair in the text, Positive samples are from two sources: (1) all other texts in the current batch. (j≠i); (2) Historical text features stored in a momentum queue. A momentum queue is a first-in, first-out buffer that continuously collects features from the most recent batch, thus providing a large-scale and diverse pool of negative samples, which is crucial for effective contrastive learning. By minimizing the contrastive loss, task-agnostic experts They were motivated to focus on more generalized image-text alignment knowledge that was irrelevant to specific tasks, thus strengthening the pre-training foundation of the visual language model.

[0039] The second stage is used to learn new knowledge, i.e., to mitigate IKF. By freezing the task-independent expert paths trained in the first training stage, we instead train only the task-related expert group paths and the two routers (internal and external), using the cross-entropy loss function to learn task-specific knowledge.

[0040] For example, for an input feature x, it will proceed through two paths in parallel:

[0041] On task-independent paths, the output is stable. .

[0042] In the task-related path, the input feature x first passes through the internal router. This internal router will provide services to the task-related expert group. Each expert in the process calculates a weight. Then, the weighted summation is used to obtain the aggregated output:

[0043]

[0044] external router Calculate the two fusion weights The outputs of the two paths are then finally weighted and merged:

[0045]

[0046] Trainable parameters for the entire task-related path (including expert groups) The two routers are optimized end-to-end using a standard cross-entropy classification loss.

[0047] During the inference phase, the outputs of two parallel paths are dynamically fused through a hierarchical routing mechanism based on the input, enabling collaborative learning of pre-trained knowledge and task-specific knowledge. In this embodiment of the invention, the hierarchical routing mechanism utilizes an external router. Implementation. During inference, the model operates using a unified forward propagation mechanism. For any input, a hierarchical routing mechanism is automatically activated to determine the optimal ratio of pre-trained knowledge to newly learned knowledge, without requiring any external task IDs or mode switching. This design makes the embodiments of the present invention very convenient and efficient in practical applications.

[0048] This invention employs task-agnostic experts to solidify pre-trained knowledge through contrastive learning and momentum queue mechanisms; while task-related expert groups learn incremental knowledge for new tasks through an internal routing mechanism. Combined with a hierarchical routing mechanism and a two-stage training strategy, this achieves decoupling of knowledge "retention" and "learning." This invention significantly improves the model's retention of old knowledge and pre-trained generalization ability while effectively learning new knowledge. Experimental results show that this invention significantly outperforms existing technologies on multiple benchmark datasets, both in terms of incremental learning performance and pre-trained knowledge retention.

[0049] Example 2:

[0050] Based on Example 1, a model evaluation method is proposed, which runs in a practical scenario without task IDs, including:

[0051] Construct a dataset collection comprising multiple different domains, dividing the dataset collection into a source task set for incremental class learning and an evaluation pool for assessing zero-shot capabilities. For example, construct a dataset collection containing M different domains. In each complete evaluation experiment, one dataset is selected from the dataset. The datasets serve as the source datasets for incremental class learning. The remaining M-1 datasets collectively constitute the zero-shot evaluation pool. .

[0052] The source dataset is further divided into a task sequence. For example, the source dataset is further divided into a task sequence. The model will learn these tasks sequentially, without knowing the boundaries or IDs of the tasks during the learning process. The data for each task is divided into a training set and a test set.

[0053] Using the continuous learning method disclosed in Example 1, tasks in the task sequence are learned sequentially. After each task is completed, the following evaluation is performed:

[0054] An assessment of specific knowledge forgetting for all historical tasks, known as IKF assessment, is conducted within the task. The model's accuracy was evaluated on the test set, and standard CIL metrics such as backward transfer (BWT) were calculated to measure the degree to which the model had forgotten old knowledge.

[0055] The evaluation pool is used to assess the forgetting of general knowledge from pre-training, i.e., PKF evaluation, in the zero-shot evaluation pool. Zero-shot classification tests were performed on each dataset. To correct for evaluation bias, dissimilarity weights were assigned to each dataset using a similarity-aware transfer metric. Then, the zero-shot classification test results for each dataset were weighted based on these dissimilarity weights to obtain the final PKF evaluation result. This weighted averaging of the zero-shot evaluation results more accurately reflects the model's ability to generalize to truly "unseen" domains.

[0056] In this embodiment of the invention, the core of the similarity-aware transfer metric is to evaluate the zero-sample evaluation pool. For each dataset in the dataset, a dissimilarity weight is calculated. The specific steps include:

[0057] For any two datasets in the evaluation pool (e.g., the training task set) and evaluation dataset A dataset contains data from multiple categories. First, its multimodal prototype is extracted for each category. The multimodal prototype r of a category is formed by concatenating its visual prototype v (e.g., the mean of all image features of the category) and text prototype t (e.g., the features of the category that describe text), i.e., r = [v; t].

[0058] Calculated using the directed Chamfer distance d right The multimodal feature distance. This distance measures... All categories prototype to The average distance between the nearest prototypes can effectively reflect the degree of conceptual overlap between two sets.

[0059] The calculated distance d is passed through an exponential function. This is converted into a similarity score S ranging from [0,1]. Here, β is a hyperparameter. Then, the dissimilarity weights... Defined as Therefore, the more similar a dataset is to the training task, the lower its weight.

[0060] To further verify the effectiveness of the DFA-CIL evaluation framework and DFA-MoE method proposed in this embodiment in solving the double forgetting problem of visual language models, extensive experiments were conducted on three representative datasets, and compared with several existing mainstream continuous learning methods. The three source datasets selected for the experiments were EuroSAT (remote sensing image classification), OxfordFlower (fine-grained flower classification), and FER2013 (facial expression recognition), covering various types of images, including natural images, non-natural images, and domain-specific images, which can comprehensively examine the model's generalization ability and adaptability.

[0061] The comparison methods include: the standard CLIP zero-shot baseline; traditional full-scale fine-tuning continuous learning methods LWF, iCaRL, and LWF-VR; the replay-based method GIFT; and parameter-efficient fine-tuning methods MoE4Adapter, Chordprompt, and DMNSP. To ensure the fairness of the comparison, all methods are evaluated within the DFA-CIL framework proposed in this invention, focusing on the following four core metrics:

[0062] CIL-Last (↑): The final accuracy after continuous learning ends, reflecting the ability to retain incremental knowledge;

[0063] CIL-BWT (↑): Backward transfer ability, reflecting the impact of learning a new task on old tasks;

[0064] Zero-shot Overall-Last (↑): Average zero-shot accuracy on the evaluation pool dataset;

[0065] Zero-shot Overall-SAT (↑): The similarity-aware transfer metric proposed in this invention reflects the degree of true retention of pre-trained knowledge.

[0066] Table 1 shows the experimental results on the EuroSAT dataset. In this experiment, the task was divided into 5 incremental stages.

[0067] Table 1. Results of dual forgetting assessment on the EuroSAT dataset

[0068]

[0069] As shown in Table 1, on the EuroSAT dataset, the DFA-MoE method proposed in this embodiment achieves best or near-best results across all metrics. Specifically, on the CIL-Last metric, which measures incremental learning capability, this method achieves 72.30%, significantly outperforming similar parameter-based efficient fine-tuning methods (such as MoE4Adapter's 62.52% and DMNSP's 69.64%), demonstrating the powerful ability of heterogeneous expert groups in learning new knowledge. More importantly, on the SAT metric, which measures pre-trained knowledge retention, this method achieves a high score of -0.75, greatly approaching the baseline level of the original CLIP model and far exceeding other fine-tuning methods. This indicates that the combination of task-agnostic experts and contrastive learning effectively suppresses the PKF phenomenon.

[0070] Table 2 shows the experimental results on the Oxford Flower dataset. This dataset contains 102 flower classes, divided into 17 incremental tasks, which places high demands on the model's fine-grained classification capabilities.

[0071] Table 2. Results of dual forgetting assessment on the OxfordFlower dataset

[0072]

[0073] As shown in Table 2, the method in this embodiment also performs excellently on the OxfordFlower task. Although the GIFT method is on par with our method on CIL-Last (70.21% vs 70.24%), its BWT score (-10.28) is significantly lower than our method (-6.91), indicating that it suffers from more severe forgetting of older tasks. On the SAT score, DFA-MoE reaches -0.66, outperforming all compared methods, further demonstrating the necessity of introducing similarity-aware weighted evaluation and the effectiveness of heterogeneous expert architecture in maintaining general knowledge. In contrast, methods such as Chordprompt exhibit severe knowledge collapse on this task.

[0074] Table 3 shows the experimental results on the FER2013 dataset. This dataset is divided into 4 tasks, which mainly examine the robustness of the model on non-RGB natural images (grayscale images of faces).

[0075] Table 3. Results of Double Forgetting Assessment on the FER2013 Dataset

[0076]

[0077] As shown in Table 3, FER2013 posed a challenge to many continuous learning methods; for example, LWF-VR achieved an accuracy of only 11.74%. However, the DFA-MoE method in this embodiment maintained robust performance, achieving a CIL-Last accuracy of 61.62%, significantly outperforming the second-place MoE4Adapter (60.59%) and DMNSP (56.35%). Furthermore, this method ranked first in the SAT metric with a score of -1.78, demonstrating that even with a wide range of domains, this invention can effectively protect the pre-trained foundation of the model while learning new tasks.

[0078] To further verify the universality and robustness of this invention in various heterogeneous domains, this embodiment also conducted a large-scale cross-task experiment. The 22 datasets in the evaluation pool were used sequentially as source tasks for incremental class learning, and the retention of pre-trained knowledge was evaluated on the remaining datasets. Figure 2 The radar chart compares the final performance of the DFA-MoE method proposed in this invention with that of various mainstream baseline methods in 22 CIL scenarios.

[0079] like Figure 2 As shown, each vertex in the radar chart represents an evaluation dimension, and a larger polygon area indicates better overall performance. Experimental results show that:

[0080] Pre-trained knowledge preservation (Zero-shot Last Acc. and SAT): The polygons of the DFA-MoE in this invention significantly and consistently cover all baseline methods in both the Zero-shot Last Acc. and SAT dimensions. This demonstrates that the present invention perfectly preserves the original zero-shot generalization ability of the visual language model in 22 different continuous learning scenarios, and its robustness is not affected by specific learning domains.

[0081] Incremental Knowledge Learning (CIL Last Acc. and BWT): On traditional category incremental learning metrics, the polygon area of ​​the method in this invention remains competitive, especially in the BWT dimension, where it exhibits stronger resistance to forgetting. This demonstrates that the invention can maintain highly competitive learning efficiency while protecting old knowledge.

[0082] Robustness across all scenarios: Compared to replay methods such as GIFT or parameter-based efficient fine-tuning methods such as DMNSP, which may exhibit significant performance dips in certain specific domains (such as datasets with large domain bias), the performance polygon of this invention is more complete and balanced, demonstrating the comprehensive robustness of this invention across cross-domain scenarios.

[0083] In summary, through in-depth analysis on representative datasets such as EuroSAT, OxfordFlower, and FER2013, and large-scale cross-task validation on 22 datasets, the experimental results demonstrate the superiority of this invention.

Claims

1. A continuous learning method to alleviate the double forgetting of visual language models, characterized in that, include: By adding two parallel paths to the backbone network of the pre-trained visual language model—one being a task-agnostic expert path and the other a task-related expert group path—a model based on heterogeneous expert hybrid is obtained. The model based on heterogeneous expert hybrid training is trained using a two-stage training strategy, including: In the first stage, by freezing the expert group paths related to the task and training only the expert paths unrelated to the task, we consolidated the pre-trained general knowledge of the pre-trained visual language model through comparative learning. In the second stage, task-independent expert paths are frozen, and only task-related expert group paths are trained. The cross-entropy loss function is used to learn specific knowledge of specific tasks. During the inference phase, the outputs of two parallel paths are dynamically fused through a hierarchical routing mechanism to achieve collaborative learning of pre-trained knowledge and task-specific knowledge.

2. The continuous learning method for mitigating dual forgetting in visual language models according to claim 1, characterized in that, The task-agnostic expert path includes a separate task-agnostic expert module; this separate task-agnostic expert module is incentivized to focus on pre-trained general knowledge by minimizing the contrastive loss.

3. The continuous learning method for mitigating dual forgetting in visual language models according to claim 1, characterized in that, The task-related expert group path includes multiple task-related expert modules and an internal router; the internal router is used to calculate the weight of each task-related expert module according to the input features, and to perform a weighted summation of the outputs of each task-related expert module to obtain the aggregated output of the task-related expert group path.

4. The continuous learning method for mitigating dual forgetting in visual language models according to claim 1, characterized in that, The hierarchical routing mechanism is implemented through an external router, which is used to calculate the fusion weight of the output of the task-independent expert path and the output of the task-related expert group path, and to perform weighted fusion of the outputs of the two paths.

5. A model evaluation method, characterized in that, include: Construct a dataset set that includes multiple different domains, and divide the dataset set into a source task set for incremental class learning and an evaluation pool for evaluating zero-shot capability; The source dataset is further divided into a task sequence; A continuous learning approach is adopted to learn tasks in a task sequence sequentially. After each task is completed, the forgetting of specific knowledge of all historical tasks is evaluated. An evaluation pool is also used to evaluate the forgetting of general knowledge of pre-training, thereby obtaining the evaluation results. The continuous learning method is a continuous learning method for mitigating double forgetting in visual language models as described in any one of claims 1 to 4.

Images