Method, device and medium for continual learning of visual language models for image classification

Abstract

This invention discloses a continuous learning method, device, and medium for a visual language model for image classification, relating to the field of artificial intelligence. It includes: connecting a t-th adapter to be trained in parallel at both ends of an MLP module in a visual language model that has already learned t-1 image classification tasks; obtaining, in a first stage, the first output of the t-th adapter, the first output of the MLP module, and the first outputs of the trained 1 to t-1 adapters to determine a first loss; entering a second stage when the fluctuation value of the first loss is less than or equal to a preset fluctuation value; obtaining, in the second stage, the second output of the MLP module and the second output of the t-th adapter to determine a second loss; and updating the model parameters of the t-th adapter based on the first and second losses to obtain a visual language model that has learned t image classification tasks, including the trained t-th adapter, thereby improving the prediction accuracy of the visual language model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a continuous learning method, device, and medium for a visual language model used for image classification. Background Technology

[0002] With the widespread application of pre-trained visual-language models (VLMs, such as the CLIP model), the key issue of continuous learning (CL) is how to enable the model to continuously learn new knowledge without fully retraining it, while trying to retain old knowledge as much as possible.

[0003] In related continuous learning methods, while strong orthogonal constraints can alleviate catastrophic forgetting, they significantly limit the ability of new task adapters to utilize knowledge from old tasks. Simply relying on experience replay or distillation cannot provide finer-grained constraints and sharing mechanisms for the "adapter structure." Furthermore, strong isolation between task adapters leads to insufficient cross-task knowledge sharing: adapters from different tasks are forcibly squeezed into mutually orthogonal or weakly correlated subspaces, making the model lack the ability to characterize the similarity between tasks, making it difficult to perform positive transfer to similar tasks, and even increasing the risk of negative transfer, thereby affecting the prediction accuracy of visual language models in continuous learning. Summary of the Invention

[0004] In view of the above-mentioned technical problems, the present invention provides a method, device and medium for continuous learning of visual language models for image classification, which aims to overcome the above problems or at least partially solve the above problems.

[0005] The first aspect of this invention provides a continuous learning method for a visual language model for image classification, the method comprising:

[0006] At both ends of the MLP module in the visual language model that has learned t-1 image classification tasks, connect the t-th adapter to be trained in parallel, where t is an integer greater than 1;

[0007] In the first stage, for the sample images of the t-th image classification task, the first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the 1 to t-1 adapters after training are obtained to determine the first loss and update the model parameters of the t-th adapter to be trained; the first loss is used to help the t-th adapter learn the t-th image classification task when the 1 to t-1 adapters participate in the aggregation.

[0008] If the fluctuation value of the first loss in the first stage is less than or equal to the preset fluctuation value, then proceed to the second stage;

[0009] In the second stage, for the sample images of the t-th image classification task, the second output of the MLP module and the second output of the t-th adapter updated in the first stage are obtained to determine the second loss; the second loss is used to help the t-th adapter learn the t-th image classification task when the 1st to t-1th adapters do not participate in the aggregation.

[0010] Based on the first and second losses of the second stage, the model parameters of the t-th adapter after the first stage are updated to obtain the t-th adapter after training, so as to obtain a visual language model that has learned t image classification tasks, which can be used to perform any image classification task from the 1st to the tth image classification task.

[0011] A second aspect of the present invention provides an electronic device comprising a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the continuous learning method for a visual language model for image classification as described in the first aspect of the present invention.

[0012] A third aspect of the present invention provides a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the continuous learning method for a visual language model for image classification as described in the first aspect of the present invention.

[0013] The continuous learning method for visual language models for image classification proposed in this invention, during the learning process of the visual language model for the t-th image classification task, connects the t-th adapter to be trained in parallel at both ends of the MLP module in the visual language model that has already learned t-1 image classification tasks. In the initial training phase (first stage), through conditional branching, for the sample image of the t-th image classification task, based on the first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the trained adapters from the 1st to the (t-1th)th, a first loss (conditional loss) is calculated. Based on the first loss, the model parameters of the t-th adapter to be trained are updated, prioritizing the recovery of the conditional branch. Convergence; as training stability improves, if the fluctuation value of the conditional loss is less than or equal to the preset fluctuation value, the second stage is entered to activate the marginal branch; in the second stage, through the marginal branch, for the sample image of the t-th image classification task, the second loss (marginal loss, also known as marginal loss) is determined based on the second output of the MLP module and the second output of the t-th adapter updated in the first stage. In the second stage, the first loss is determined through the conditional branch, and the model parameters of the t-th adapter updated in the first stage are updated based on the first loss and the second loss of the second stage to obtain the t-th adapter after training, and thus obtain the visual language model that has learned t image classification tasks. Thus, this invention explicitly characterizes the difference between "conditional posterior" and "marginal posterior" through a two-branch structure (conditional branch and marginal branch). By jointly optimizing the conditional loss and marginal loss, it implicitly constrains the difference between the adapter's conditional posterior and marginal posterior, improving the plasticity of the visual language model. This allows the model to efficiently absorb knowledge from new tasks without compromising the performance of already learned tasks, and enables controlled knowledge sharing among related tasks. It achieves fine-grained dynamic control of the visual language model adapter, simultaneously satisfying the requirements of "fully adapting to new tasks" and "retaining existing capabilities" during continuous learning, thereby improving the predictive accuracy of the visual language model. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating the steps of a continuous learning method for a visual language model used for image classification, as shown in an embodiment of the present invention.

[0016] Figure 2This is a schematic diagram illustrating a visual language model performing continuous learning for the tth time, as shown in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Currently, the main approaches to continuous learning of pre-trained visual language models in related technologies are as follows: 1. Classical continuous learning methods, 2. Parameter Efficient Fine-Tuning (PEFT) continuous learning methods, and 3. Instruction Fine-Tuning and Instruction-Driven Continuous Learning methods. However, this invention carefully studies the above methods and finds that the above methods generally have the following technical defects and the reasons for these defects: (1) The training perspective is mostly based on experience or heuristic design, lacking systematic analysis based on statistical learning theories such as PAC-Bayesian; (2) The adapter or LoRA for each task is usually trained to full convergence, and then mutual interference is minimized by orthogonal constraints, etc. This "full convergence + strong isolation" design is prone to forming "knowledge islands"; (3) When each task adapter is forced to occupy non-intersecting or approximately orthogonal subspaces, the ability to transfer and share knowledge across tasks is significantly suppressed, especially when there is semantic correlation between tasks (e.g., multiple fine-grained classification datasets).

[0020] Based on this, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, this invention proposes a continuous learning method for visual language models for image classification. By introducing "Phase-like Transition Constraints (PACT)" based on PAC Beyes theory, the difference between "conditional posterior" and "marginal posterior" is explicitly characterized through a two-branch structure (conditional branch and marginal branch). By jointly optimizing the conditional loss and marginal loss, implicit constraints on the difference between the adapter's conditional posterior and marginal posterior are achieved, improving the plasticity of the visual language model. This allows the model to efficiently absorb knowledge from new tasks without compromising the performance of already learned tasks, and enables controlled knowledge sharing between related tasks. It achieves fine-grained dynamic control of the visual language model adapter, and can simultaneously meet the requirements of "fully adapting to new tasks" and "retaining existing capabilities" during continuous learning, thereby improving the prediction accuracy of the visual language model.

[0021] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a continuous learning method for a visual language model used for image classification, as shown in an embodiment of the present invention. Figure 1 As shown, the continuous learning method for visual language models used for image classification provided in this embodiment includes at least the following steps:

[0022] Step S11: Connect the t-th adapter to be trained in parallel at both ends of the MLP module in the visual language model that has learned t-1 image classification tasks.

[0023] This embodiment performs continuous learning on a pre-trained visual language model. The pre-trained visual language model is used for image classification. Each time the pre-trained visual language model undergoes continuous learning, it can learn one more image classification task. For example, during the t-th continuous learning iteration, it learns the t-th image classification task based on the t-1 image classification tasks already learned, where t is an integer greater than 1.

[0024] During the t-th continuous learning iteration, the t-th adapter to be trained is connected in parallel to both ends of the MLP module in the visual language model that has already learned t-1 image classification tasks, resulting in a visual language model for learning t image classification tasks. The t-th adapter corresponds to the t-th image classification task. The visual language model that has already learned t-1 image classification tasks is the visual language model obtained after t-1 continuous learning iterations from the pre-trained visual language model. This visual language model that has learned t-1 image classification tasks includes at least: t-1 adapters that have been trained (with fixed parameters) (i.e., the first to t-1 trained adapters), and an MLP module with fixed parameters. The first to t-1 trained adapters are also connected in parallel to both ends of the MLP module in the visual language model. In other words, the t-th adapter to be trained in the visual language model for learning t image classification tasks is connected in parallel with the first to t-1 trained adapters, respectively.

[0025] Step S12: In the first stage, for the sample image of the t-th image classification task, obtain the first output of the t-th adapter to be trained, the first output of the MLP module, and the first output of the 1st to t-1th adapters after training, so as to determine the first loss and update the model parameters of the t-th adapter to be trained.

[0026] In this embodiment, during the t-th continuous learning of the visual language model that has already learned t-1 image classification tasks, in the first stage of the t-th continuous learning, the sample image of the t-th image classification task is input into the visual language model to be learned for t image classification tasks. The first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the 1st to t-1th adapters after training are obtained through conditional branching. Then, based at least on the first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the 1st to t-1th adapters after training, the conditional loss of the conditional branch, i.e., the first loss, is calculated. This first loss is used to promote the t-th adapter to learn the t-th image classification task when the 1st to t-1th adapters participate in aggregation. After obtaining the first loss, the model parameters of at least the t-th adapter to be trained are continuously updated based on the first loss to obtain the t-th adapter after the first stage update.

[0027] Step S13: If the fluctuation value of the first loss in the first stage is less than or equal to the preset fluctuation value, proceed to the second stage.

[0028] In this embodiment, the first loss calculated in the first stage is continuously monitored to determine its fluctuation value. When the fluctuation value of the first loss in the first stage is less than or equal to a preset fluctuation value, the characterization training enters a relatively stable stage, at which point it enters the second stage. In this embodiment, the marginal branch is activated in the second stage to perform two-branch loss training (marginal branch and conditional branch) to achieve phase transition adjustment in the later stage. The preset fluctuation value is the maximum value of the loss fluctuation during the relatively stable stage of the characterization training, which can be freely set according to requirements.

[0029] In one embodiment, the first loss sequence can be... (Including multiple first losses) A ​​sliding window of length W is used (the sliding window length W is used to detect training stability, and can be 50 to 1000 steps, adjusted according to the actual batch size and task difficulty), and the mean of the first loss sequence within the window is calculated. ,variance With trend items (Where, the trend term refers to the direction and magnitude of change within the window, i.e.) = t – t-1 , (as the first loss); and based on the mean of the first loss sequence. ,variance With trend items The fluctuation value of the first loss is calculated. :

[0030] Formula (1);

[0031] in, , These are the first and second weights, respectively, and t is the current training time step, which is the tth iteration. To avoid a denominator of 0, a numerical stability constant is typically chosen as 10^{-8}; then, a preset fluctuation value is used. Stability Gating : =1 . and , The adjustment is based on the fluctuation of the loss curve to achieve a balance between "activating regularization too early" and "activating regularization too late". In this embodiment, when the fluctuation value of the first loss is greater than a preset fluctuation value... (In this case, where the trend of loss is obvious) =0, at this point only the conditional loss is optimized (i.e. only the main task loss is optimized); when the fluctuation value of the first loss is less than or equal to the preset fluctuation value (At this point, the loss tends to plateau) The value changes from 0 to 1, entering the second stage to activate the PACT regular expression.

[0032] Step S14: In the second stage, for the sample image of the t-th image classification task, obtain the second output of the MLP module and the second output of the t-th adapter after the first stage update, so as to determine the second loss.

[0033] In this embodiment, to satisfy the "bounded single-sample loss" condition in the PAC Bayes assumption, the training enters a relatively stable phase in the first stage (i.e., the fluctuation value of the first loss in the first stage is less than or equal to a preset fluctuation value), and then proceeds to the second stage to activate PACT regularization. In the second stage of continuous learning for the tth time, the sample image of the tth image classification task is input into the visual language model to be learned for the tth image classification tasks (the visual language model to be learned for the tth image classification tasks in the second stage includes at least: t-1 trained adapters, the tth adapter updated in the first stage, and an MLP module with fixed parameters). The second output of the tth adapter updated in the first stage, the second output of the MLP module, and the third output of each of the 1st to t-1th trained adapters are obtained through conditional branching. Then, based at least on the second output of the tth adapter updated in the first stage, the second output of the MLP module, and the third output of each of the 1st to t-1th trained adapters, the conditional loss of the conditional branch in the second stage is calculated, which is the first loss of the second stage.

[0034] Furthermore, in the second stage of the t-th continuous learning, the sample image of the t-th image classification task is input into the visual language model to be learned for the t-th image classification tasks (the visual language model to be learned for the t-th image classification tasks in the second stage includes at least: t-1 trained adapters, the t-th adapter updated in the first stage, and an MLP module with fixed parameters). The second output of the t-th adapter updated in the first stage and the second output of the MLP module are obtained through marginal branches. Then, based at least on the second output of the t-th adapter updated in the first stage and the second output of the MLP module, the marginal loss of the marginal branches in the second stage is calculated, i.e., the second loss of the second stage. This second loss is used to facilitate the t-th adapter in learning the t-th image classification task when the 1st to t-1th adapters have not participated in aggregation.

[0035] Step S15: Based on the first and second losses of the second stage, update the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, so as to obtain a visual language model that has learned t image classification tasks, which can be used to perform any image classification task from the 1st to the tth image classification task.

[0036] In this embodiment, during the second stage of the t-th continuous learning, the model parameters of the t-th adapter after the first stage update can be updated based on the first loss of the second stage and the second loss of the second stage, until the loss converges, resulting in the trained t-th adapter. This yields a visual language model for t image classification tasks that has been learned. This visual language model for t image classification tasks includes at least t trained adapters (i.e., the 1st to tth trained adapters) and an MLP module with fixed parameters. This visual language model for t image classification tasks is used to perform any one of the 1st to tth image classification tasks. The 1st to tth image classification tasks are mutually exclusive.

[0037] In other words, after obtaining a visual language model that has been learned for t image classification tasks, the image to be classified under any image classification task from the 1st to the tth image classification task can be input into the visual language model that has been learned for t image classification tasks to obtain the classification prediction result of the image to be classified output by the visual language model that has been learned for t image classification tasks.

[0038] In a specific example, for multi-domain incremental learning, the training iterations are approximately 3000 steps per task; for class-based incremental tasks, 1000 steps per task can be selected, and this can also be adjusted according to the data scale.

[0039] In this embodiment, the generalization bound of each image classification task is analyzed using PAC Bayes theory. The key KL term is decomposed into at least a plasticity term, where the first loss and the second loss together constitute the plasticity term. This plasticity term is used to measure the difference between the posterior distribution of the adapter for the current image classification task under the given old adapter conditions and its marginal posterior distribution.

[0040] Throughout the t-th continuous learning process in this embodiment, most of the pre-trained parameters are frozen (i.e., the model parameters of the backbone network of the visual language model (including at least the MLP module) are always fixed, and the model parameters of the first to the (t-1)th adapters are always fixed). Only a small number of adapter modules are added for each new task increment (e.g., adding the t-th adapter to be trained for the t-th image classification task), restricting task-specific knowledge to the adapters, thereby maintaining the zero-shot capability of the backbone model and the existing task performance.

[0041] In this embodiment, a "Phase-like Transition Constraint (PACT)" based on PAC Bayes theory is introduced. A two-branch structure (conditional branch and marginal branch) explicitly characterizes the difference between the "conditional posterior" and the "marginal posterior." By jointly optimizing the conditional loss and marginal loss, an implicit constraint on the difference between the adapter's conditional posterior and marginal posterior is achieved, enhancing the plasticity of the visual language model. This allows the model to efficiently absorb new task knowledge without compromising the performance of already learned tasks, and enables controlled knowledge sharing between related tasks. It achieves fine-grained dynamic control of the visual language model adapter, simultaneously satisfying the requirements of "fully adapting to new tasks" and "retaining existing capabilities" during continuous learning, thus improving the predictive accuracy of the visual language model.

[0042] In conjunction with the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, in addition to the steps described above, it may further include step S21, and the step S15 above, "updating the model parameters of the t-th adapter after the first stage update based on the first and second losses of the second stage to obtain the trained t-th adapter," may specifically include step S22:

[0043] Step S21: In the second stage, for the probe set, obtain the third output of the t-th adapter after the first stage update and the third output of the MLP module to determine the third loss.

[0044] In this embodiment, to prevent the adapter from deviating too far from the initial representation during subsequent training, a prior anchoring mechanism is introduced in the second stage of the t-th continuous learning: For the probe set, this probe set is input into the t-th adapter updated in the first stage to obtain the third output of the t-th adapter updated in the first stage; and the probe set is input into the MLP module in the visual language model for learning t image classification tasks to obtain the third output of the MLP module. Then, based on the third output of the t-th adapter updated in the first stage and the third output of the MLP module, the third loss of the second stage, namely the prior anchoring (PA) loss, is calculated. This third loss is used to limit the drift amplitude of the t-th adapter, ensuring that the t-th adapter maintains compatibility with the feature space of the MLP module while learning the t-th image classification task, thereby maintaining the performance and zero-shot capability of early tasks and limiting adapter updates within a reasonable range.

[0045] In this embodiment, a probe set is pre-constructed. For example, the adapter input dimension can be set to d, and a Gaussian probe set Z can be pre-fixed. This constructed probe set is shared throughout the continuous learning process to "probe" differences in the responses of different adapters to the same input. Among them, z m For the m-th Gaussian probe (Gaussian noise); I d d×s is the identity matrix of the input dimension d; M is the number of Gaussian probes, which can be 32 to 512, determined according to the model width and computing resources. In one embodiment, M is 128 to 256.

[0046] Step S22: Based on the first loss, second loss and third loss of the second stage, update the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training.

[0047] In this embodiment, in the second stage of the t-th continuous learning, the model parameters of the t-th adapter after the first stage update can be updated based on the first loss, the second loss, and the third loss of the second stage, until the loss converges, and the t-th adapter after training is obtained, thereby obtaining the visual language model for t image classification tasks that has been learned.

[0048] In this embodiment, the decomposition forms of the "plasticity term" and "stability term" are derived from PAC-Bayesian theory. The third loss is used as the stability term, which measures the degree of drift of the marginal posterior distribution of the current image classification task adapter relative to its initial distribution. In other words, the third loss can effectively suppress the adapter from deviating too far from the pre-trained representation, avoiding the destruction of the shared feature space of the existing task. This embodiment explicitly decouples "stability (preserving old knowledge)" and "plasticity (adapting to new tasks)" into interpretable terms. Phase transition constraints are suppressed in the early stages of training, prioritizing the convergence of conditional branches. As training stability improves, PACT regularization (i.e., the third loss) and the second loss are activated only after the loss converges to a relatively stable stage (i.e., the second stage). The third loss constrains the adapter's posterior drift, achieving explicit stability constraints on old knowledge, thereby satisfying PAC. Bayes' bounded loss assumption, while avoiding over-constraining the model during the early, drastic update phase, significantly improves the stability-plasticity balance and mitigates catastrophic forgetting.

[0049] In conjunction with any of the above embodiments, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, in addition to the steps described above, steps S31 to S33 may be included, and step S22 may specifically include steps S34 to S35:

[0050] Step S31: In the second stage, for the probe set, obtain the second output of the 1st to t-1th adapters after training.

[0051] In this embodiment, in the second stage of the t-th continuous learning, the probe set is input to the 1st to t-1th adapters after training, and the second output of each of the 1st to t-1th adapters after training is obtained.

[0052] Step S32: Determine the difference between the second output of the 1st to the (t-1th)th adapters and the third output of the tth adapter after the first phase update.

[0053] In this embodiment, the third output of the t-th adapter after the first stage update can be obtained based on the second outputs of the 1st to t-1th adapters after training, and the differences between the second outputs of the 1st to t-1th adapters and the third output of the t-th adapter after the first stage update can be determined.

[0054] In an alternative example, the behavioral differences of the adapters can be measured using the normalized Frobenius norm, such as by determining the differences between the second outputs of the 1st to t-1th adapters and the third output of the tth adapter after the first-stage update, as shown in the following formula (2). :

[0055] Formula (2);

[0056] in, Let be the difference between the second output of the j-th adapter and the third output of the t-th adapter after the first phase update. To prevent small constants with a denominator of zero, such as 10 -8 ; This is the third output of the t-th adapter after the first phase update. This is the second output of the j-th adapter.

[0057] In an alternative embodiment, for the current task adapter (i.e., the t-th adapter after the first phase update) and any old task adapter The response matrices on the probe are defined as follows:

[0058] Formula (3);

[0059] Formula (4);

[0060] in, It is both the response matrix of the t-th adapter on the probe after the first stage update and the third output of the t-th adapter after the first stage update; It is both the response matrix of the j-th adapter on the probe and the second output of the j-th adapter. This indicates the forward mapping of the adapter.

[0061] Step S33: Determine the weighting factor based on the determined minimum difference.

[0062] In this embodiment, to achieve a continuous transition between the "melt" and "freeze" states, a weighting factor based on the similarity of adapter behavior is constructed: the minimum difference is determined from the differences between the second output of each of the 1st to t-1th adapters and the third output of the tth adapter after the first stage update, and the weighting factor is determined based on the determined minimum difference. The weighting factor in this embodiment is the phase transition weight, which can also be called the "freeze weight".

[0063] In an optional specific example, the weighting factor can be determined by the following formula (5). :

[0064] Formula (5);

[0065] Where exp is an exponential function with the natural constant e as its base. To determine the minimum difference; Temperature hyperparameter, controlling the freezing weight The smoothness of the transition from "melting" to "freezing" can be taken as 0.05 to 2; when When it is large, When the value approaches 0, the new adapter (i.e., the t-th adapter) is in a "free / melted" state; when... When smaller, When the value approaches 1, the new adapter is in a "constrained / frozen" state.

[0066] In another embodiment, after obtaining Then, we can find the old adapter that is most similar to the current adapter (i.e., the t-th adapter). : Then, the distance between the most similar old adapter and the current adapter (i.e., the distance between the most similar old adapter) is calculated. The difference between the second output and the third output of the t-th adapter after the first phase update The mapping is to weight factors, and the mapping method can be referred to the above formula (5).

[0067] Step S34: Weight the second loss and the third loss based on the weighting factor to obtain the fusion loss.

[0068] In this embodiment, after obtaining the weighting factor, the second loss of the second stage and the third loss of the second stage can be weighted based on the weighting factor to obtain the fusion loss. In an optional example, the fusion loss = weighting factor × second loss + weighting factor × third loss.

[0069] Step S35: Based on the first loss and fusion loss of the second stage, update the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training.

[0070] In this embodiment, after obtaining the fusion loss, the model parameters of the t-th adapter after the first stage update can be updated based on the first loss and the fusion loss of the second stage, resulting in the t-th adapter after training. In an optional example, the fusion loss and the first loss of the second stage can be added together to obtain the total loss. The model parameters of the t-th adapter after the first stage update can be updated based on the total loss until the total loss converges. The model parameters of the t-th adapter at the time of total loss convergence are then fixed, resulting in the t-th adapter after training.

[0071] In this embodiment, a phase transition weight based on Gaussian probes and adapter response distance is introduced to smoothly transition between the two states of "frozen (old task dominant)" and "melted (new task dominant)". The "frozen / melted" state is not distinguished by a hard threshold, but by a continuous, distance-aware weight function, which manifests as a smooth phase transition-like process: when there is a strong correlation between tasks, the new adapter moves closer to the old adapter under the "phase transition-like constraint", forming a "frozen" state, which promotes knowledge sharing and positive transfer; when the tasks are significantly different, the constraint is automatically weakened, and the new adapter enters a "melted" state, allowing it to freely adapt to the current task and reducing interference and catastrophic forgetting. In other words, this embodiment proposes a phase transition weight calculation based on adapter response distance, introduces a Gaussian probe library to perform unified functional detection on the adapter; for example, it uses a normalized Frobenius distance to characterize the functional similarity between the current adapter and the old adapter; it uses an exponential kernel to map the distance to phase transition weights, with the weights approaching 1 when the adapters are similar and approaching 0 otherwise, which is used to adjust the weights for stability and plasticity regularization to achieve a smooth "phase transition".

[0072] It should be noted that the "phase transition-like constraint" described in this embodiment does not refer to a real physical phase transition, but rather draws on the macroscopic characterization of water's continuous change with temperature between solid (ice) and liquid (water): near the freezing point, the system does not switch instantaneously with a hard threshold of "complete freezing / complete melting", but rather there is a transition range determined by external conditions (such as temperature, impurities, local thermal disturbances, etc.), which makes the macroscopic state exhibit a continuous and adjustable trend of change. Correspondingly, this embodiment treats "task relevance / adapter functional distance" as the "effective temperature" (or control parameter) that determines the state, and achieves a smooth transition from "frozen" to "melted" through a continuous distance-aware weighting function: when the adapter response distance is small (high relevance), the phase transition weight approaches 1, equivalent to a "low-temperature environment," and the system tends to enter the "frozen" state, causing the new adapter to move closer to the old adapter under constraints, promoting knowledge sharing and positive transfer; when the adapter response distance increases (low relevance), the phase transition weight gradually decays and approaches 0, equivalent to a "warming environment," the constraints automatically weaken, and the system enters the "melted" state, giving the new adapter greater plasticity to adapt to the current task, thereby reducing mutual interference and catastrophic forgetting. Thus, this embodiment uses a continuous kernel function to "soft-gated" the distance, achieving a "continuous parameter tuning-state gradual change" mechanism similar to water phase transition, rather than relying on discrete switching of hard thresholds.

[0073] In combination with any of the above embodiments, in one implementation, a Gaussian probe can be introduced in the first stage to determine the weighting factor (i.e., phase transition weight) in the aforementioned manner. The final regularization weights are formed by combining the training stability index. That is, combining the volatility value based on the first loss. and preset fluctuation value The obtained stability gating The comprehensive phase transition weights are obtained. :

[0074] Formula (6);

[0075] The obtained comprehensive phase transition weights For the weights used in weighted plasticity and stability regularization, we can first use the comprehensive phase transition weights. The second and third losses are weighted to obtain the fusion loss; then, based on the first and fusion losses, the model parameters of the t-th adapter are updated. (This is the integrated phase transition weight.) During training, the system automatically changes with task similarity and optimization stability, thereby achieving continuous phase transition behavior between "freeze / melt".

[0076] In one example, the overall training objective of continuous learning is: ;in, The total loss is used to update the parameters of the current task adapter (i.e., the t-th adapter). This is a conditional loss, also known as the first loss. This is the second loss, namely the marginal loss. As the third loss, To incorporate phase transition weights. When When ≈0, the model primarily optimizes the performance of the current task, plasticity dominates, and the current task adapter is in a "melted" state; when When ≈1, the model strongly constrains the new adapter to maintain consistency with the old representation, stability dominates, and the current task adapter is in a "frozen" state; under most task combinations, Taking an intermediate value between 0 and 1 represents a smooth phase transition. Repeat the above optimization until the stopping condition is met, and save the current task adapter parameters. Then freeze it and proceed to the next image classification task t+1.

[0077] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, step S11 specifically includes step S41, and, in addition to the above steps, may further include steps S42 to S45:

[0078] Step S41: Insert the t-th adapter to be trained into a portion of the Transformer layers of the visual language model that has learned t-1 image classification tasks. Each t-th adapter to be trained is connected in parallel to both ends of the MLP module in its Transformer layer and has the same structure as the MLP module. Use the parameters of the MLP module as the initialization parameters of the t-th adapter to be trained.

[0079] In this embodiment, the pre-trained visual language model includes multiple Transformer layers. The parameters in each Transformer layer (including multi-head self-attention modules, MLP modules, etc.) remain frozen during continuous learning and are not updated with the task. For example, the visual encoding part of the pre-trained visual language model adopts a multi-layer Vision Transformer structure (e.g., ViT-B / 16).

[0080] In this embodiment, during the t-th continuous learning, a t-th adapter to be trained is inserted into each Transformer layer of a portion of the visual language model that has already learned t-1 image classification tasks. Each t-th adapter to be trained is connected in parallel across the two ends of the MLP module in its Transformer layer. That is, each t-th adapter to be trained is connected in parallel with the backbone MLP module of its Transformer layer. Both receive the same input, and their outputs are fused later.

[0081] Furthermore, each t-th adapter to be trained has the same structure as its parallel MLP module. That is, the structure of the t-th adapter to be trained is completely identical to that of its parallel MLP module, including the number of layers, width, and activation function, thus ensuring a one-to-one correspondence of parameters. In addition, in this embodiment, the parameters of the MLP module connected in parallel with the t-th adapter to be trained are used as the initialization parameters of the t-th adapter to be trained. In other words, the parameters of the newly inserted adapter in this embodiment are directly copied from the parameters of its corresponding backbone MLP. As initialization: This ensures that the t-th adapter implements the same function mapping as the trunk MLP upon insertion. These are the initialization parameters for the t-th adapter to be trained.

[0082] Step S42: Insert the first adapter to be trained into a portion of the Transformer layers of the visual language model. Each first adapter to be trained is connected in parallel to both ends of the MLP module in its Transformer layer and has the same structure as the MLP module. Use the parameters of the MLP module as the initialization parameters of the first adapter to be trained.

[0083] In this embodiment, during the first continuous learning of the pre-trained visual language model, a first adapter to be trained is inserted into each Transformer layer of a subset of the pre-trained visual language model. Each first adapter to be trained is connected in parallel to both ends of the MLP module in its respective Transformer layer; that is, each first adapter to be trained is connected in parallel with the backbone MLP module of its Transformer layer. Both receive the same input, and their outputs are subsequently fused to obtain a visual language model for an image classification task. Furthermore, the structure of each first adapter to be trained is identical to that of its parallel MLP module; that is, the structure of the first adapter to be trained is completely consistent with that of its parallel MLP module, including the number of layers, width, and activation function, thus ensuring a one-to-one correspondence of parameters. In addition, this embodiment uses the parameters of the MLP module connected in parallel with the first adapter to be trained as the initialization parameters of the first adapter to be trained, ensuring that the first adapter achieves the same function mapping as the backbone MLP upon insertion.

[0084] Step S43: In the first stage, for the sample images of the first image classification task, obtain the first output of the first adapter to be trained and the fourth output of the MLP module to determine the second loss corresponding to the first image classification task, and update the model parameters of the first adapter to be trained.

[0085] In this embodiment, during the first continuous learning of the visual language model, in the first stage of the first continuous learning, the sample image of the first image classification task is input into the visual language model of the image classification task to be learned, and the first output of the first adapter to be trained and the fourth output of the MLP module are obtained.

[0086] Since there is only one adapter for image classification task in the visual language model to be trained (i.e., there is only one adapter to be trained), the conditional branches and marginal branches in the visual language model to be trained are the same. They are all based on at least the first output of the first adapter to be trained and the fourth output of the MLP module to calculate the second loss corresponding to the first image classification task. Then, based on the second loss corresponding to the first image classification task, the model parameters of the first adapter to be trained are continuously updated to obtain the first adapter after the first stage update.

[0087] Step S44: If the fluctuation value of the second loss corresponding to the first image classification task in the first stage is less than or equal to the preset fluctuation value, proceed to the second stage.

[0088] This step is similar to step S13 above, continuously monitoring the second loss corresponding to the first image classification task calculated in the first stage to determine the fluctuation value of the second loss corresponding to the first image classification task. If the fluctuation value of the second loss corresponding to the first image classification task in the first stage is less than or equal to a preset fluctuation value, it indicates that the training of the first continuous learning has entered a relatively stable stage, at which point the second stage begins. The preset fluctuation value in this step can be the same as or different from the preset fluctuation value in step S13 above; there is no limitation on this. The method for determining the fluctuation value of the second loss corresponding to the first image classification task is the same as or similar to the method for determining the fluctuation value of the first loss in step S13, as can be found in the aforementioned embodiments.

[0089] Step S45: In the second stage, for the probe set, obtain the second output of the first adapter after the first stage update and the fifth output of the MLP module to determine the third loss corresponding to the first image classification task.

[0090] In this embodiment, in the second stage of the first continuous learning, the sample images of the first image classification task are still input into the visual language model of the image classification task to be learned (the visual language model of the image classification task to be learned in the second stage includes at least: the first adapter updated in the first stage, and an MLP module with fixed parameters) to obtain the first output of the first adapter updated in the first stage and the fourth output of the MLP module. Based at least on the first output of the first adapter updated in the first stage and the fourth output of the MLP module, the second loss corresponding to the first image classification task in the second stage is calculated.

[0091] Furthermore, a prior anchoring mechanism is introduced in the second stage of the first continuous learning: for the probe set, the probe set is input into the first adapter updated in the first stage to obtain the second output of the first adapter updated in the first stage; and the probe set is input into the MLP module in the visual language model to be learned for an image classification task to obtain the fifth output of the MLP module. Then, based on the second output of the first adapter updated in the first stage and the fifth output of the MLP module, the third loss corresponding to the first image classification task in the second stage, i.e., the prior anchoring loss, is calculated. In this embodiment, the probe set is shared throughout the entire continuous learning process of the pre-trained visual language model.

[0092] Step S45: Based on the second loss and third loss corresponding to the first image classification task in the second stage, update the model parameters of the first adapter after the first stage update to obtain the first adapter after training, so as to obtain a visual language model that has learned one image classification task, which is used to perform the first image classification task.

[0093] In this embodiment, in the second stage of the first continuous learning, the model parameters of the first adapter updated in the first stage can be updated based on the second loss corresponding to the first image classification task in the second stage and the third loss corresponding to the first image classification task in the second stage, until the loss converges, thus obtaining the trained first adapter. This results in a visual language model that has learned one image classification task. This visual language model includes at least the trained first adapter and an MLP module with fixed parameters. The visual language model that has learned one image classification task is used to perform the first image classification task. That is, after obtaining the visual language model that has learned one image classification task, the image to be classified under the first image classification task can be input into the visual language model that has learned one image classification task to obtain the classification prediction result of the image to be classified output by the visual language model that has learned one image classification task.

[0094] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, S12 specifically includes steps S51 to S54:

[0095] Step S51: For the Transformer layer with the t-th adapter to be trained inserted, obtain the first average output based on the first outputs of the 1st to t-1th adapters trained in the Transformer layer and the first output of the t-th adapter to be trained; according to the first fusion coefficient, fuse the first average output with the first output of the MLP module of the Transformer layer to obtain the first output of the Transformer layer.

[0096] In this embodiment, the pre-trained visual language model includes multiple Transformer layers. The parameters in each Transformer layer (including a multi-head self-attention module, an MLP module, etc.) are frozen during continuous learning and are not updated with the task. During the t-th continuous learning, in each Transformer layer of the visual language model that has learned t-1 image classification tasks, the t-th adapter to be trained is inserted. Each t-th adapter to be trained is connected in parallel to both ends of the MLP module in its Transformer layer and has the same structure as the MLP module. The parameters of the MLP module are used as the initialization parameters of the t-th adapter to be trained.

[0097] In this embodiment, during the first stage of the t-th continuous learning, the sample image of the t-th image classification task is input into the visual language model for the t image classification tasks to be learned, and the relevant output of each Transformer layer can be obtained. Specifically, for each Transformer layer with the t-th adapter inserted for training, the first outputs corresponding to the 1st to t-1th adapters trained in that Transformer layer, the first output of the t-th adapter to be trained, and the first output of the MLP module of that Transformer layer can be obtained through conditional branching. Then, based on the first outputs corresponding to the 1st to t-1th adapters trained in that Transformer layer and the first output of the t-th adapter to be trained, a first average output is obtained. Then, according to the first fusion coefficient, the first average output is fused with the first output of the MLP module of that Transformer layer to obtain the first output of that Transformer layer. The first fusion coefficient can be freely set, but it monotonically increases with the layer number of the Transformer layer containing the t-th adapter. Optionally, the first fusion coefficient in the shallow layer is close to 0, and the first fusion coefficient in the deep layer is close to 0.5. This ensures information stability while providing stronger task specialization capabilities at higher levels.

[0098] In an optional example, the first fusion coefficient can be scalar-gated. The first output of the Transformer layer can be determined by the following formula (7). :

[0099] Formula (7);

[0100] in, This is the first output of the MLP module of the Transformer layer. This is the first average output.

[0101] In another embodiment, after obtaining the first average output, a small amount of Gaussian noise can be added to the first average output to obtain the conditional features. (Used to characterize "under the old adapter conditions"), then according to the first fusion coefficient, the conditional features are... The first output of the Transformer layer is obtained by fusing the first average output with the first output of the MLP module of the Transformer layer according to the first fusion coefficient; or, after fusing the first average output with the first output of the MLP module of the Transformer layer according to the first fusion coefficient to obtain the first fusion result, a small Gaussian noise is added to the first fusion result to obtain the first output of the Transformer layer; there are no restrictions on this.

[0102] In this embodiment, in the first stage, for the Transformer layer that does not have the t-th adapter to be trained inserted, the first output of the previous Transformer layer is directly used as the input of the Transformer layer that does not have the t-th adapter to be trained inserted. After processing by the Transformer layer that does not have the t-th adapter to be trained inserted (i.e., processing by the normal Transformer layer), the first output of the Transformer layer that does not have the t-th adapter to be trained inserted is obtained.

[0103] Step S52: Process the first output of the last Transformer layer using the classification heads of the t-1 image classification tasks that have been learned to obtain the first classification prediction result.

[0104] In this embodiment, the pre-trained visual language model includes a classification head. During the continuous learning process of the pre-trained visual language model, in addition to updating the parameters of the adapter corresponding to each image classification task, the parameters of the classification head are also continuously updated. For example, after the (t-1)th continuous learning, the visual language model that has learned t-1 image classification tasks includes the classification heads of the t-1 image classification tasks. That is, the visual language model to be learned for t image classification tasks includes at least: the t-th adapter to be trained, the 1st to (t-1)th adapters that have been trained, the MLP module with fixed parameters, and the classification heads of the t-1 image classification tasks.

[0105] In this embodiment, after obtaining the first output of the last Transformer layer, the first output of the last Transformer layer can be input into the classification head of the learned t-1 image classification tasks. The first output of the last Transformer layer is then processed by the classification head of the learned t-1 image classification tasks to obtain the first classification prediction result.

[0106] Step S53: Obtain the first loss based on the first classification prediction result and the category label carried by the sample image.

[0107] In this embodiment, the sample image of the t-th image classification task carries a category label. After obtaining the first classification prediction result, the first loss can be calculated based on the first classification prediction result and the category label carried by the sample image of the t-th image classification task.

[0108] Step S54: Based on the first loss, in addition to updating the model parameters of the t-th adapter to be trained, the model parameters of the classification heads of the t-1 image classification tasks that have been learned are also updated.

[0109] In this embodiment, based on the first loss, in addition to updating the model parameters of the t-th adapter to be trained, resulting in the t-th adapter after the first stage update, the model parameters of the classification heads of the t-1 image classification tasks already learned are also updated, resulting in the classification heads after the first stage update. In other words, this embodiment updates the model parameters of the t-th adapter to be trained and the model parameters of the classification heads of the t-1 image classification tasks already learned, based on the first loss, to obtain the classification heads and the t-th adapter after the first stage update.

[0110] In one alternative example, in the first phase, i.e. the pre-convergence phase, the main task is trained via conditional branching: for each training sample (i.e., the sample image of the t-th image classification task). By freezing the backbone network (such as including multi-head self-attention modules and MLP modules). Extract features; in each Transformer layer with the t-th adapter to be trained, calculate the first output of the 1st to t-1th adapters after training, and average them to obtain the first average output. The conditional features are obtained by adding Gaussian noise. Then, according to the first fusion coefficient, the conditional features are... The first output of the Transformer layer is obtained by fusing it with the first output of the MLP module of the Transformer layer; the first output of the last Transformer layer is then predicted using the classification heads of t-1 image classification tasks already learned, to obtain the first prediction distribution. First loss defined by cross-entropy As shown in the following formula (8):

[0111] Formula (8);

[0112] Where, N t Let be the number of sample images for the t-th image classification task. The class label carried by the sample image for the t-th image classification task. The parameters of the t-th adapter to be trained are updated using gradient descent (e.g., the AdamW optimizer). The parameters of the corresponding classifier heads for the learned t-1 image classification tasks are used until the conditional loss stabilizes and meets the preset convergence criterion (i.e., the fluctuation value of the first loss is less than or equal to the preset fluctuation value).

[0113] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, step S14 specifically includes steps S61 to S63, and the step S15, "updating the model parameters of the t-th adapter after the first stage update based on the first and second losses of the second stage to obtain the trained t-th adapter, thereby obtaining a visual language model that has learned t image classification tasks," specifically includes step S64:

[0114] Step S61: For the Transformer layer with the t-th adapter to be trained inserted, according to the second fusion coefficient, fuse the second output of the t-th adapter updated in the first stage of the Transformer layer with the second output of the MLP module of the Transformer layer to obtain the second output of the Transformer layer.

[0115] In this embodiment, during the second stage of the t-th continuous learning, the sample image of the t-th image classification task is input into the visual language model for the t image classification tasks to be learned, yielding the relevant output of each Transformer layer. Specifically, for each Transformer layer with the t-th adapter inserted for training, the second output of the t-th adapter updated in the first stage and the second output of the MLP module of that Transformer layer are obtained through marginal branches. Then, according to the second fusion coefficient, the second output of the t-th adapter updated in the first stage and the second output of the MLP module of that Transformer layer are fused to obtain the second output of the Transformer layer. The second fusion coefficient can be freely set and monotonically increases with the layer number of the Transformer layer containing the t-th adapter. The second fusion coefficient can be the same as or different from the first fusion coefficient; this is not restricted.

[0116] In an optional example, the second fusion coefficient can be scalar-gated. Similarly, the second output of the Transformer layer can be determined using the above formula (7), where the value in formula (7) is... This is the second output of the MLP module of the Transformer layer. This is the second output of the t-th adapter after the first phase update in this Transformer layer. In an optional example, during initialization, since... Therefore, regardless The value chosen ensures that the overall output remains consistent with that before insertion, thus guaranteeing the continuity of network functionality.

[0117] In another embodiment, after the second output of the t-th adapter updated in the first stage of the Transformer layer, Gaussian noise of the same scale (the same Gaussian noise added in the conditional branch) can be added to the second output of the t-th adapter updated in the first stage of the Transformer layer to obtain edge features. Then, according to the second fusion coefficient, the edge features are... The second output of the Transformer layer is obtained by fusing it with the second output of the MLP module of the Transformer layer; alternatively, after fusing the second output of the t-th adapter updated in the first stage of the Transformer layer with the second output of the MLP module of the Transformer layer according to the second fusion coefficient to obtain the second fusion result, Gaussian noise of the same scale is added to the second fusion result to obtain the second output of the Transformer layer; there are no restrictions on this.

[0118] In this embodiment, in the second stage, for the Transformer layer that does not have the t-th adapter inserted for training, the output of the previous Transformer layer is directly used as the input of the current Transformer layer that does not have the t-th adapter inserted for training. After processing by the current Transformer layer that does not have the t-th adapter inserted for training (i.e., processing by a normal Transformer layer), the output of the current Transformer layer that does not have the t-th adapter inserted for training is obtained. Wherein, if the previous Transformer layer has the t-th adapter inserted for training, the output of the previous Transformer layer includes: a first output and a second output of the previous Transformer layer. For example, the first output and the second output of the previous Transformer layer can be fused and used as the input of the current Transformer layer that does not have the t-th adapter inserted for training.

[0119] Step S62: Process the second output of the last Transformer layer using the updated classification head from the first stage to obtain the second classification prediction result.

[0120] In this embodiment, the last Transformer layer is the Transformer layer into which the t-th adapter to be trained is inserted. After obtaining the second output of the last Transformer layer, the second output of the last Transformer layer can be input into the classification head updated in the first stage of the t-th continuous learning. The second output of the last Transformer layer is then processed by the classification head updated in the first stage of the t-th continuous learning to obtain the second classification prediction result.

[0121] Step S63: Obtain the second loss based on the second classification prediction result and the category label carried by the sample image.

[0122] In this embodiment, after obtaining the second classification prediction result, the second loss of the second stage can be calculated based on the second classification prediction result and the category label carried by the sample image of the t-th image classification task.

[0123] In this embodiment, the conditional branch and the marginal branch share the same frozen backbone network and the current task adapter (i.e., the t-th adapter to be trained), and the only difference is whether the old adapter (the 1st to t-1th adapters that have been trained) is used in the aggregation, thereby constructing an achievable approximation of the conditional posterior and the marginal posterior in PAC Bayes.

[0124] In one embodiment, in the second stage, in addition to training the main task through conditional branches, edge branches and plasticity regularization are also constructed, that is, a "edge branch (i.e. marginal branch)" is copied on the same main structure, and only the current task adapter is retained for each identical training sample (i.e., the sample image of the t-th image classification task). In each Transformer layer with the t-th adapter to be trained, the second output of the t-th adapter after the first stage update is obtained through the t-th adapter after the first stage update. Adding Gaussian noise yields the marginal features (i.e., edge features). Then, according to the second fusion coefficient, the marginal features are... The second output of the Transformer layer is obtained by fusing it with the second output of the MLP module of the Transformer layer; the second output of the last Transformer layer is then predicted using the classification head updated in the first stage to obtain the second prediction distribution. The second loss is defined according to cross-entropy. As shown in the following formula (9):

[0125] Formula (9);

[0126] Where, N t Let be the number of sample images for the t-th image classification task. Let the class label be the class label carried by the sample image for the t-th image classification task. Within the PAC-Bayes framework, minimize... Approximate reduction The goal is to improve plasticity.

[0127] Furthermore, in the second phase of the t-th continuous learning, for each Transformer layer with the t-th adapter inserted for training, the second outputs corresponding to the 1st to t-1th adapters trained in that Transformer layer, the second output of the t-th adapter updated in the first phase, and the second output of the MLP module of that Transformer layer can be obtained through conditional branches. Then, based on the second outputs corresponding to the 1st to t-1th adapters trained in that Transformer layer and the second output of the t-th adapter updated in the first phase, the first average output of the second phase is obtained. Then, according to the first fusion coefficient, the first average output of the second phase is fused with the second output of the MLP module of that Transformer layer in the second phase to obtain the first output of that Transformer layer in the second phase. After obtaining the first output of the last Transformer layer in the second stage, the first output of the last Transformer layer can be input into the classification head updated in the first stage. The first output of the last Transformer layer is processed by the classification head updated in the first stage to obtain the first classification prediction result of the second stage. Finally, the first loss of the second stage is calculated based on the first classification prediction result of the second stage and the category label carried by the sample image of the t-th image classification task.

[0128] Step S64: Based on the first and second losses of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, the model parameters of the classification head after the first stage update are also updated to obtain the classification head of the learned t image classification tasks, so as to obtain the visual language model of the learned t image classification tasks.

[0129] In this embodiment, based on the first loss of the second stage and the second loss of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, the model parameters of the classification head after the first stage update are also updated to obtain the classification head for t image classification tasks. That is, this embodiment updates the model parameters of the t-th adapter after the first stage update and the model parameters of the classification head after the first stage update in the t-th continuous learning process based on the first loss and the second loss of the second stage, obtaining the t-th adapter after training and the classification head for t image classification tasks after training, thereby obtaining the visual language model for t image classification tasks. In this embodiment, the visual language model for t image classification tasks after training includes at least: t adapters after training (i.e., the 1st to tth adapters after training), an MLP module with fixed parameters, and the classification head for t image classification tasks after training.

[0130] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In this method, step S21 may specifically include steps S71 to S72, and step S22 may specifically include step S73:

[0131] Step S71: Based on the probe set, for the Transformer layer with the t-th adapter to be trained, obtain the KL divergence between a pair of third outputs corresponding to the Transformer layer.

[0132] In this embodiment, during the second stage of the t-th continuous learning, for each Transformer layer with the t-th adapter to be trained inserted, a probe set is input to the t-th adapter updated in the first stage of the Transformer layer and the MLP module of the Transformer layer, respectively, to obtain a pair of third outputs corresponding to the Transformer layer. This pair of third outputs includes the third output of the t-th adapter updated in the first stage of the Transformer layer and the third output of the MLP module in the Transformer layer. Then, based on this pair of third outputs, the KL divergence between them is determined, that is, the KL divergence between the third output of the t-th adapter updated in the first stage of the Transformer layer and the third output of the MLP module in the Transformer layer is determined.

[0133] Step S72: Obtain the third loss based on the KL divergence of the Transformer layer corresponding to each of the multiple inserted t-th adapters to be trained.

[0134] In this embodiment, after obtaining the KL divergence between a pair of third outputs corresponding to each Transformer layer inserted into the t-th adapter to be trained, the third loss of the second stage can be determined based on the KL divergence of each Transformer layer inserted into the t-th adapter to be trained. In an optional embodiment, the third loss of the second stage is obtained by summing the KL divergences of each Transformer layer inserted into the t-th adapter to be trained.

[0135] Step S73: Based on the first loss, second loss and third loss of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, the model parameters of the classification head after the first stage update are also updated to obtain the classification head of the learned t image classification tasks, so as to obtain the visual language model of the learned t image classification tasks.

[0136] In this embodiment, based on the first loss of the second stage, the second loss of the second stage, and the third loss of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the trained t-th adapter, the model parameters of the classification head after the first stage update are also updated to obtain the classification head for t image classification tasks. In other words, this embodiment updates the model parameters of the t-th adapter after the first stage update and the model parameters of the classification head after the first stage update in the t-th continuous learning process based on the first loss, the second loss of the second stage, and the third loss of the second stage, to obtain the trained t-th adapter and the classification head for t image classification tasks, thereby obtaining the visual language model for t image classification tasks.

[0137] In an optional example, the KL divergence between a pair of third outputs corresponding to the Transformer layer containing the t-th adapter to be trained can be obtained by... It means that, among them, Let be the distribution of the third output of the t-th adapter after the first stage update in this Transformer layer. This is the distribution of the third output of the MLP module in this Transformer layer. In one embodiment, the adapter is initialized with the distribution. As a priori for the current task adapter, the initialization distribution of the MLP modules in the backbone network is regarded as the prior distribution. During training, parameter drift is used as a metric for estimation. This yields the stability regularization loss (i.e., the third loss). It can be represented as: This is achieved by adding a KL regularization term (i.e., a stability regularization term) to the training objective. In implementation, this KL regularization term can be estimated using methods such as L2 distance in the parameter space and Fisher approximation, and is implemented as a penalty term for adapter parameters deviating from their initial values.

[0138] In one embodiment, based on the above embodiments, a separate set of adapter parameters is set for each image classification task t. Each task adapter forms a set after insertion. (3) After training is completed for image classification task t, the parameters of the corresponding t-th adapter are... Freeze, as a condition for subsequent tasks.

[0139] In one embodiment, to reduce the number of parameters and computational overhead, thus facilitating large-scale continuous learning deployment, the adapter is not inserted at every Transformer layer, but rather sparsely: one adapter is inserted every k Transformer layers, where k is a positive integer. In a preferred embodiment, k is the adapter insertion interval, and k is 3, meaning the t-th adapter to be trained is inserted at the 3rd, 6th, 9th, and 12th Transformer layers. Experiments show that k = 1 to 3 can achieve good performance, balancing performance and the number of parameters, while performance significantly decreases when k ≥ 4.

[0140] In another embodiment, the last Transformer layer is the Transformer layer into which the t-th adapter to be trained is inserted.

[0141] In one embodiment, each Transformer layer includes at least: a multi-head self-attention module and an MLP module, wherein the multi-head self-attention module includes: a normalization submodule, a multi-head self-attention submodule and a residual connection; the MLP module includes: a normalization submodule, an MLP submodule and a residual connection.

[0142] In one embodiment, to avoid disrupting the existing good performance of the pre-trained model when inserting a new adapter, a Stable Adapter Initialization (SAI) is used to achieve parallel insertion, structure matching, parameter cloning, and convex hybrid gating: the new adapter is completely isomorphic to the parallel MLP module during insertion, and its parameters are directly copied. Since the gating output is a convex combination, the network output remains consistent before and after insertion, ensuring that the adapter's posterior is strictly aligned with the prior at the initial time, and the stability term S (i.e., stability regularization) reaches its theoretical minimum. Combined with the aforementioned sparse insertion strategy, the newly inserted adapter is obtained by copying the MLP module only in some Transformer layers, reducing the size of parameters and KL terms. Thus, in this embodiment, the new adapter has almost no perturbation to the model output during insertion, and its posterior distribution is highly consistent with the initialization prior, ensuring that the stability term S in PACBEayes reaches its theoretical minimum at the initial time, which is beneficial for subsequent training convergence and generalization.

[0143] In a specific example, a task sequence The t-th task For example, let's initialize the visual language model and integrate it with the task: Input the current task dataset. and the already trained backbone parameters Compared with old adapter parameters Insert the new task adapter to be trained into the predetermined Transformer layers (e.g., layers 3, 6, 9, and 12). It is initialized via SAI: parallel connection, structure matching, parameter cloning, and gating settings.

[0144] In conjunction with any of the above embodiments, in one implementation, the present invention also provides a continuous learning method for a visual language model for image classification. In addition to the steps described above, this method may further include steps S81 to S83:

[0145] Step S81: Input the image to be classified from any of the image classification tasks from the 1st to the tth into the visual language model that has learned t image classification tasks.

[0146] In this embodiment, after obtaining the visual language model that has learned t image classification tasks, the image to be classified under any of the image classification tasks from the 1st to the tth can be input into the visual language model that has learned t image classification tasks to obtain the target output of each Transformer layer in the visual language model that has learned t image classification tasks.

[0147] Step S82: For a Transformer layer with t trained adapters inserted, obtain the first target average output based on the first target output of the 1st to tth trained adapters; according to the first fusion coefficient, fuse the first average target output with the first target output of the MLP module of the Transformer layer to obtain the target output of the Transformer layer.

[0148] In this embodiment, the visual language model, having learned t image classification tasks, performs predictions only through conditional branches in practical applications. Specifically, for each Transformer layer with t trained adapters, the conditional branches can be used to obtain the first target output corresponding to each of the 1st to tth trained adapters in that Transformer layer, as well as the first target output of the MLP module of that Transformer layer. Then, the first target outputs corresponding to each of the 1st to tth trained adapters in that Transformer layer are averaged to obtain the first target average output. Then, according to the first fusion coefficient, the first target average output is fused with the first target output of the MLP module of that Transformer layer to obtain the target output of that Transformer layer.

[0149] In this embodiment, for a Transformer layer that has not inserted t fully trained adapters, the target output of the previous Transformer layer is directly used as the input of the current Transformer layer that has not inserted t fully trained adapters. After processing by the current Transformer layer that has not inserted t fully trained adapters (i.e., processing by a normal Transformer layer), the target output of the current Transformer layer that has not inserted t fully trained adapters is obtained.

[0150] Step S83: Process the target output of the last Transformer layer using the classification heads of the t image classification tasks that have been learned, and obtain the classification prediction result of the image to be classified.

[0151] In this embodiment, after obtaining the target output of the last Transformer layer, the target output of the last Transformer layer can be input into the classification head of the learned t image classification tasks. The target output of the last Transformer layer is then processed by the classification head of the learned t image classification tasks to obtain the classification prediction result of the image to be classified. The classification prediction result of the image to be classified is the prediction result of the visual language model that has learned t image classification tasks for the image to be classified.

[0152] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the t-th continuous learning process of a visual language model according to an embodiment of the present invention. Figure 2 In the diagram, the large solid box on the right represents a Transformer layer in a visual language model for t image classification tasks, where the t-th adapter to be trained is inserted. This Transformer layer containing the t-th adapter includes a backbone network and t adapters connected in parallel with the MLP module in the backbone network. The first t-1 adapters (i.e., the 1st adapter, the 2nd adapter, ..., etc.) are the 1st to t-1th adapters that have been trained, and the t-th adapter is the t-th adapter to be trained. The backbone network is a standard Transformer layer, including a multi-head self-attention module and a multilayer perceptron module (i.e., an MLP module). The multi-head self-attention module includes a normalization submodule, a multi-head self-attention submodule, and residual connections; the multilayer perceptron module includes a normalization submodule, a multilayer perceptron submodule, and residual connections.

[0153] Figure 2 The dashed line A in the diagram represents the processing path of the conditional branch: In the conditional branch, after obtaining the output of the multi-head self-attention module of this layer, the output of the multi-head self-attention module is input into the multilayer perceptron module, the first to t-1 trained adapters, and the t-th adapter to be trained, respectively, to obtain the first outputs corresponding to the first to t-1 trained adapters in this Transformer layer, the first output of the t-th adapter to be trained, and the first output of the multilayer perceptron module of this Transformer layer; based on the first outputs corresponding to the first to t-1 trained adapters and the first output of the t-th adapter to be trained, the first average output is calculated, and then the first average output is fused with the first output of the multilayer perceptron module of this Transformer layer to obtain the first output of this Transformer layer to calculate the conditional loss (i.e., the first loss).

[0154] The two branches (edge ​​branch and condition branch) share the same frozen backbone network and current task adapter, differing only in whether the old adapter is used for aggregation. Figure 2The dashed lines in the diagram represent the processing path of the edge branch: In the edge branch, after obtaining the output of the multi-head self-attention module of the layer, the output of the multi-head self-attention module is input into the multilayer perceptron module and the t-th adapter to be trained, respectively, to obtain the second output of the t-th adapter to be trained (such as the second output of the t-th adapter after the first stage update), and the second output of the multilayer perceptron module of the Transformer layer; then the second output of the t-th adapter to be trained is fused with the second output of the multilayer perceptron module of the Transformer layer to obtain the second output of the Transformer layer to calculate the edge loss (i.e., the second loss).

[0155] Figure 2 The dashed line B in the diagram represents the method used to calculate the loss.

[0156] exist Figure 2 In the diagram, the dashed box B on the right represents the calculation process of the prior anchoring loss (i.e., the third loss): the probe set is input into the t-th adapter to be trained and the multilayer perceptron module inserted into the Transformer layer of the t-th adapter to be trained, respectively, to obtain the third output of the t-th adapter to be trained (such as the third output of the t-th adapter after the first stage update) and the third output of the multilayer perceptron module; then, the KL divergence is calculated based on the third output of the t-th adapter to be trained and the third output of the multilayer perceptron module; finally, the KL divergences corresponding to each Transformer layer inserted into the t-th adapter to be trained are added together to obtain the prior anchoring loss (i.e., the third loss). This embodiment differs from stabilization strategies that rely solely on parameter initialization. Considering that good initialization alone is insufficient to prevent later training from going astray, a Stable Adapter Initialization (SAI) is introduced when inserting an adapter to avoid disrupting the existing good performance of the pre-trained model. Based on SAI, a prior anchoring mechanism is introduced to prevent the adapter from deviating too far from its initial representation during subsequent training. This third loss, as mentioned above, theoretically constrains posterior drift and, combined with phase transition-like constraints (i.e., phase transition weights), plays a role in the later convergence phase. Figure 2 "in This means that an adapter is inserted only once every three Transformer layers.

[0157] exist Figure 2In the diagram, the dashed box B within the large solid box on the right illustrates the calculation process of the total loss in continuous learning: first, based on the edge loss and prior anchoring loss, combined with the comprehensive phase transition weights (i.e., phase transition weights) or weight factors provided by phase transition-like constraints, the fusion loss is obtained; then, based on the fusion loss and conditional loss, the total loss (i.e., PCAT loss) is determined. Here, the edge loss and conditional loss represent the plasticity term, while the prior anchoring loss represents the stability term, balancing stability and plasticity to achieve systematic constraints based on the PAC-Bayesian perspective. In other words, this invention provides a novel adapter continuous learning method that is theoretically more grounded, more efficient in implementation, and capable of achieving smooth "phase transition" stability-plasticity adjustment.

[0158] In one embodiment, the present invention provides an adapter continuous learning method based on phase-like trAnsitionConsTraints (PACT), which mainly includes the following technical points: (1) Derivation of the decomposition forms of "plasticity term" and "stability term" from PAC-Bayesian theory; (2) A two-branch visual Transformer structure (conditional branch + marginal branch) suitable for continuous learning scenarios; (3) 4. Stable adapter initialization (SAI) to ensure that the insertion of the adapter does not destroy the pre-training capability; (4) Prior anchoring (PA) to limit the adapter update within a reasonable range, so as to control the posterior drift of the adapter by using stable adapter initialization and prior anchoring; (5) A phase-like constraint mechanism and adaptive weight scheduling that takes effect in the post-convergence stage: introducing phase-change weights based on Gaussian probes and adapter response distance, and smoothly transitioning between the two states of "frozen (old task dominant)" and "melted (new task dominant)"; (6) Reducing additional trainable parameters by sparsely inserting the adapter.

[0159] In summary, this invention differs from adapter methods based on "orthogonal constraints / subspace isolation": such methods often reduce interference by forcing different task adapters to be orthogonal in the parameter or feature space, but this suppresses positive transfer between tasks and easily leads to "knowledge silos"; this invention, however, does not force orthogonality, but instead uses the plasticity-stability term derived from PAC Bayes decomposition, combined with phase transition-like weights, to achieve soft constraint coupling and selective sharing of the adapters. Furthermore, this invention differs from simple freeze-fine-tuning or single-branch PEFT methods: traditional PEFT trains the adapter only on a single branch, failing to explicitly characterize the difference between "conditional posterior" and "marginal posterior"; this invention explicitly implements both through a two-branch structure and approximates the KL term through a joint loss, forming a theory-driven regularization objective.

[0160] Taking typical experiments as an example, the technical effects of this invention include at least the following: In multi-domain task incremental learning scenarios, this technical solution outperforms many existing methods in terms of indicators such as "forward transfer", "average accuracy" and "final task accuracy", while reducing the number of training parameters by about 30-40%, significantly reducing memory usage and iteration time; In incremental learning scenarios, this solution achieves high average accuracy and final incremental accuracy under various partitioning strategies, indicating that its advantage in the stability-plasticity trade-off has good task universality; Training logs show that in each task, the method of this invention presents a two-stage loss curve of "rapid decline followed by slight recovery and stabilization", the latter corresponding to the stage where PACT takes effect, proving that it effectively suppresses overfitting and catastrophic forgetting as a post-convergence regularization.

[0161] In summary, this technical solution achieves fine dynamic control of the visual language model adapter through a two-branch structure guided by PAC Bayes theory, stable adapter initialization, prior anchoring, and phase transition constraints. It can simultaneously meet the requirements of "fully adapting to new tasks" and "retaining existing capabilities" during continuous learning, and achieves an excellent balance between parameter quantity, computational overhead, and performance.

[0162] The following section, based on the core technical points proposed in this invention, details the beneficial effects that this invention can achieve:

[0163] 1. A phase transition-constrained continuous learning framework based on PAC-Bayesian decomposition. The complexity term of the adapter posterior in continuous learning is decomposed into a plasticity term P and a stability term S, which are explicitly reflected in the training objective. By introducing phase transition-weighted marginal loss and stability loss in the later training stage, the overall training process theoretically corresponds to the optimization of the PAC-Bayes bound.

[0164] 2. Conditional / Marginal Two-Branch ViT Structure. Under the same visual backbone, a conditional branch (aggregating all task adapters) and a marginal branch (using only the current adapter) are constructed. By jointly optimizing the cross-entropy loss of the two branches, implicit constraints on the difference between the adapter's conditional posterior and marginal posterior are implemented, thereby improving plasticity.

[0165] 3. Stable Adapter Initialization (SAI) Mechanism. The adapter structure is fully matched with the parallel MLP structure, and functional consistency is achieved through parameter copying; gated convex combination is used to ensure that the output remains basically unchanged in the early stage of adapter insertion, achieving prediction preservation and posterior consistency; this mechanism ensures that the stability term S reaches its minimum value at the beginning of the task, providing a good starting point for subsequent training.

[0166] 4. Prior Anchoring (PA) Stability Regularization. The adapter initialization parameters (or backbone MLP parameters) are used as the prior distribution; during training, the adapter posterior drift is constrained by the KL term to achieve explicit stability constraints on prior knowledge.

[0167] 5. Phase transition weight calculation based on adapter response distance. A Gaussian probe library is introduced to perform unified functional detection on the adapters; the functional similarity between the current adapter and the old adapter is characterized by a normalized Frobenius distance; and an exponential kernel is used to map the distance to phase transition weights. When the adapter is similar, the weight is close to 1, and vice versa. The weights are used to adjust the stability and plasticity regularization to achieve a smooth "phase transition".

[0168] 6. Phase transition coefficient scheduling strategy combined with training stability. In the early stages of training, suppress phase transition constraints and prioritize ensuring the convergence of conditional branches; as training stability improves, gradually activate phase transition coefficients. and This enables phase change adjustments in the later stages.

[0169] 7. Sparse adapter insertion and efficient parameter design. Adapters are inserted only in a subset of Transformer layers, and only one new set of adapters is added for each task; compared to related adapters or MoE schemes, this significantly reduces trainable parameters and resource overhead without sacrificing performance.

[0170] 8. Specific implementation process and device module division applicable to continuous learning of visual language models. This includes the combination and collaborative working methods of feature extraction module, SAI module, conditional branch module, marginal branch module, prior anchoring module, phase transition weight module, training and update module, etc.

[0171] Therefore, the beneficial effect of the present invention is that it is similar to that of related vision-related inventions. Compared to continuous learning techniques for language models (such as task-specific adapter fine-tuning only, adapter methods based on orthogonality / subspace isolation, MoE, Adapters, etc.), this method significantly improves stability while maintaining parameter efficiency. The plasticity balance, cross-task knowledge sharing ability, and training efficiency are specifically demonstrated as follows:

[0172] (a) Significantly improved stability Plasticity balance, mitigating catastrophic forgetting:

[0173] 1. This invention is based on PAC Within the Bayesian theoretical framework, the KL term for each task is decomposed into a “plasticity term P” and a “stability term S”. The adapter is then updated through a bi-branch ViT structure, with plasticity loss (conditional / marginal branches) and stability loss (prior anchoring) as explicit constraints. This ensures that the new task adapter absorbs new knowledge while being constrained by the old adapter and the initial prior, thus preventing unbounded drift.

[0174] 2. On the Multi-Domain Task Incremental Learning (MTIL) benchmark consisting of 11 datasets, the method of this invention achieves [performance] in Order [performance]. Under the training sequence I, the Transfer / Avg / Last metrics reached 71.76%, 81.05%, and 90.54, respectively, which are about 2.46, 3.75, and 3.14 percentage points higher than the best existing adapter-type methods. This indicates that a better compromise has been achieved in both "forward transfer" and "retention of learned tasks", and catastrophic forgetting has been significantly reduced.

[0175] 3. In Order II. Under different task sequences, this invention also maintains a leading position in three indicators, proving that the proposed phase transition-like constraint can still stably constrain the adapter update when the task sequence changes, and has good robustness and practical value.

[0176] (ii) Enhance cross-task knowledge sharing capabilities and form an interpretable "phase transition" structure:

[0177] 1. By using a Gaussian probe library and a normalized Frobenius distance metric adapter to measure behavioral similarity, this invention encodes inter-task correlations as continuous "frozen weights". ", and used as a unified weight for plasticity / stability regularization in the later convergence phase, to achieve a smooth transition of the adapter between the two behaviors of "free / melting" and "constrained / frozen", rather than hard threshold switching.

[0178] 2. In the adapter confusion matrix after multi-task training, adapters from similar domains (such as fine-grained classification and scene recognition) exhibit significantly high similarity, while those from fields such as handwritten digits are basically orthogonal, forming a cluster structure similar to a "phase transition boundary." This indicates that the phase transition-like constraint of this invention can automatically promote parameter convergence between tasks that are "suitable for sharing" and maintain separation between tasks that are "significantly different," thereby balancing positive transfer and interference suppression.

[0179] (iii) Maintain and enhance pre-training zero-shot capability and early task performance:

[0180] 1. This invention achieves parallel insertion, structure matching, parameter cloning, and convex hybridization through stable adapter initialization (SAI): the new adapter is completely isomorphic to the parallel MLP during insertion and its parameters are directly copied; since the gated output is a convex combination, the network output remains consistent before and after insertion, so that the adapter posterior is strictly aligned with the prior at the initial moment, and the stability term S reaches the theoretical minimum value.

[0181] 2. Building upon this, Prior Anchoring (PA) treats the MLP-induced initialization as a prior distribution and uses KL regularization to limit the extent to which the adapter deviates from the initialization during subsequent training. This ensures that the new task adapter adapts to the current data while maintaining compatibility with the pre-trained feature space. In experiments, the accuracy of the final multi-domain task incremental learning model for each early task (such as Aircraft, Caltech101, etc.) remained relatively stable after its first learning, demonstrating that pre-trained knowledge and previous task capabilities were effectively preserved.

[0182] 3. Compared with related technologies that rely solely on task-specific adapters and are prone to creating "knowledge silos", this invention can continuously absorb new task knowledge without sacrificing zero-sample capability, and its overall performance is higher than that of direct full-parameter fine-tuning and ordinary adapter fine-tuning schemes.

[0183] (iv) Excellent performance is also achieved in scenarios with few samples and incremental class processing:

[0184] 1. In 5 shot and 16 Incremental small-sample multi-domain task (FS) In the MTIL setting, this invention outperforms other methods in three metrics: Transfer, Avg, and Last. Furthermore, compared to the zero-sample baseline, it not only shows no degradation but also achieves a significant gain, demonstrating that the proposed PAC... Bayes' post-convergence constraint does not depend on large-scale data and can still function stably even with very few samples.

[0185] 2. In CIFAR 100 and TinyImageNet In the 100-class incremental learning (CIL) scenario, this invention achieves the best average accuracy and last incremental accuracy across all partitioning steps (10 / 20 / 50 or 5 / 10 / 20 steps), demonstrating that class phase transition constraints and the SAI / PA mechanism can also balance the plasticity of new classes and the stability of old classes in the practical application of "multi-class expansion in the same domain".

[0186] (v) Significantly reduce the number of parameters, memory usage, and training time, improving project usability:

[0187] 1. By employing a sparse insertion strategy of "inserting an adapter every 3 layers," and freezing the trunk and old adapters while updating only the current adapter and lightweight head, this invention significantly reduces training overhead while maintaining or improving accuracy. Experiments show that, compared to typical MoE... The Adapters method of this invention can reduce the number of trainable parameters by about 36.96%, GPU memory usage by about 31.26%, and single-step iteration time by about 13.92%.

[0188] 2. In the comparative experiment of adapter insertion interval k, when k increases from 1 to 3, the average accuracy of the present invention hardly decreases, while when k≥4, the performance drops sharply. This shows that the "sparse but not too sparse" insertion strategy proposed in this invention achieves a near-optimal trade-off between performance and resources, and is very suitable for deployment in memory-constrained or large-scale task sequence scenarios.

[0189] (vi) The training process is stable and controllable, and has a good post-convergence regularization effect:

[0190] 1. This invention constructs a stability index by statistically analyzing the mean, variance, and trend of the training loss using a sliding window. PACT regularization is activated only after the loss converges to a relatively stable phase, thus satisfying PACT. Bayes' bounded loss assumption avoids over-constraining the model during the early, drastic update phase.

[0191] 2. Training logs show that the loss curve for each task exhibits a "two-stage" characteristic: the conditional branch loss initially drops rapidly to a low level, then briefly rises after PACT is enabled, before reconverging to a slightly higher but stable level. The final model's accuracy on each task is significantly improved compared to the model that converged using only conditional loss, indicating that PACT, as a post-convergence regularization, effectively suppresses overfitting and catastrophic forgetting while maintaining numerical stability in the optimization process and preventing training divergence.

[0192] (vii) Modular, scalable, and easy to integrate with other vision systems Language Model:

[0193] 1. In this invention, the two-branch ViT structure, SAI module, prior anchoring, and phase transition weight calculation are all inserted using an adapter as a carrier, thus improving the original visual... The language backbone network is minimally invasive, does not alter the backbone structure or pre-trained weights, and achieves continuous learning enhancement through only a small number of parallel modules. It is easily integrated into existing large-scale vision systems such as CLIP. Direct integration into the language model.

[0194] 2. Because the core constraints of PACT come from PAC The theoretical decomposition of Bayes' KL terms is not strongly bound to specific network structures, therefore, except for ViT... In addition to B / 16, this can be generalized to other Transformers or CNNs. Visual Transformer Hybrid Structure The language model possesses good versatility and extensibility.

[0195] In summary, this invention, through a collaborative design of "stable adapter initialization + prior anchoring + phase transition-like constraints + dual-branch ViT structure", simultaneously achieves the following in continuous learning scenarios: significantly improved cross-task performance, maintained pre-training and old task capabilities, greatly reduced training overhead, stable training process, and enhanced interpretability. In terms of technical effectiveness, it is significantly superior to existing similar adapter continuous learning technologies.

[0196] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0197] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0198] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory, a processor, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps in the continuous learning method for a visual language model for image classification described in any of the above embodiments of the present invention.

[0199] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices described above.

[0200] Based on the same inventive concept, another embodiment of the present invention provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the continuous learning method for a visual language model for image classification as described in any of the above embodiments of the present invention. The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0203] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A continuous learning method for a visual language model for image classification, characterized in that, The method includes: At both ends of the MLP module in the visual language model that has learned t-1 image classification tasks, connect the t-th adapter to be trained in parallel, where t is an integer greater than 1; In the first stage, for the sample images of the t-th image classification task, the first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the 1 to t-1 adapters after training are obtained to determine the first loss and update the model parameters of the t-th adapter to be trained; the first loss is used to help the t-th adapter learn the t-th image classification task when the 1 to t-1 adapters participate in the aggregation. If the fluctuation value of the first loss in the first stage is less than or equal to the preset fluctuation value, then proceed to the second stage; In the second stage, for the sample images of the t-th image classification task, the second output of the MLP module and the second output of the t-th adapter updated in the first stage are obtained to determine the second loss; the second loss is used to help the t-th adapter learn the t-th image classification task when the 1st to t-1th adapters do not participate in the aggregation. Based on the first and second losses of the second stage, the model parameters of the t-th adapter after the first stage are updated to obtain the t-th adapter after training, so as to obtain a visual language model that has learned t image classification tasks, which can be used to perform any image classification task from the 1st to the tth image classification task.

2. The continuous learning method for visual language models for image classification according to claim 1, characterized in that, The method further includes: In the second stage, for the probe set, the third output of the t-th adapter after the first stage update and the third output of the MLP module are obtained to determine the third loss; the third loss is used to limit the drift amplitude of the t-th adapter, ensuring that the t-th adapter maintains compatibility with the feature space of the MLP module while learning the t-th image classification task. Based on the first and second losses of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training, including: Based on the first, second, and third losses of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training.

3. The continuous learning method for visual language models for image classification according to claim 2, characterized in that, The method further includes: In the second stage, for the probe set, the second outputs of the 1st to t-1th adapters after training are obtained; Determine the differences between the second outputs of the 1st to the (t-1th)th adapters and the third output of the tth adapter after the first phase update; The weighting factor is determined based on the identified minimum difference. Based on the first, second, and third losses of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training, including: The fusion loss is obtained by weighting the second and third losses based on the weighting factors. Based on the first loss and fusion loss of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training.

4. The continuous learning method for visual language models for image classification according to claim 1, characterized in that, At both ends of the MLP module in the visual language model that has already learned t-1 image classification tasks, the t-th adapter to be trained is connected in parallel, including: Insert the t-th adapter to be trained into a portion of the Transformer layers of the visual language model that has learned t-1 image classification tasks. Each t-th adapter to be trained is connected in parallel to both ends of the MLP module in its Transformer layer and has the same structure as the MLP module. Use the parameters of the MLP module as the initialization parameters of the t-th adapter to be trained. The method further includes: Insert the first adapter to be trained into a portion of the Transformer layers of the visual language model. Each first adapter to be trained is connected in parallel to both ends of the MLP module in its Transformer layer and has the same structure as the MLP module. Use the parameters of the MLP module as the initialization parameters of the first adapter to be trained. In the first stage, for the sample images of the first image classification task, the first output of the first adapter to be trained and the fourth output of the MLP module are obtained to determine the second loss corresponding to the first image classification task and update the model parameters of the first adapter to be trained. If the fluctuation value of the second loss corresponding to the first image classification task in the first stage is less than or equal to the preset fluctuation value, then proceed to the second stage. In the second stage, for the probe set, the second output of the first adapter after the first stage update and the fifth output of the MLP module are obtained to determine the third loss corresponding to the first image classification task; Based on the second and third losses corresponding to the first image classification task in the second stage, the model parameters of the first adapter after the first stage are updated to obtain the first adapter after training, so as to obtain a visual language model that has learned one image classification task, which is used to perform the first image classification task.

5. The continuous learning method for a visual language model for image classification according to claim 2, characterized in that, In the first stage, for the sample images of the t-th image classification task, the first output of the t-th adapter to be trained, the first output of the MLP module, and the first outputs of the 1st to t-1th adapters after training are obtained to determine the first loss. The model parameters of the t-th adapter to be trained are then updated, including: For a Transformer layer with the t-th adapter to be trained inserted, a first average output is obtained based on the first outputs of the 1st to t-1th adapters trained in the Transformer layer and the first output of the t-th adapter to be trained. The first average output is then fused with the first output of the MLP module of the Transformer layer according to a first fusion coefficient to obtain the first output of the Transformer layer. The first fusion coefficient increases monotonically with the number of layers in the Transformer layer where the t-th adapter is located. The first output of the last Transformer layer is processed by the classification heads of the t-1 image classification tasks that have been learned, to obtain the first classification prediction result; Based on the first classification prediction result and the category label carried by the sample image, a first loss is obtained; Based on the first loss, in addition to updating the model parameters of the t-th adapter to be trained, the model parameters of the classification heads that have been learned for t-1 image classification tasks are also updated.

6. The continuous learning method for a visual language model for image classification according to claim 5, characterized in that, In the second stage, for the sample image of the t-th image classification task, the second output of the MLP module and the second output of the t-th adapter updated in the first stage are obtained to determine the second loss, including: For the Transformer layer with the t-th adapter to be trained inserted, the second output of the t-th adapter updated in the first stage of the Transformer layer is fused with the second output of the MLP module of the Transformer layer according to the second fusion coefficient to obtain the second output of the Transformer layer. The second fusion coefficient increases monotonically with the number of layers in the Transformer layer where the t-th adapter is located. The second output of the last Transformer layer is processed by the updated classification head in the first stage to obtain the second classification prediction result; The second loss is obtained based on the second classification prediction result and the category label carried by the sample image; Based on the first and second losses of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training, thus obtaining a visual language model that has been learned for t image classification tasks, including: Based on the first and second losses of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, the model parameters of the classification head after the first stage update are also updated to obtain the classification head of the learned t image classification tasks, so as to obtain the visual language model of the learned t image classification tasks.

7. The continuous learning method for a visual language model for image classification according to claim 6, characterized in that, In the second phase, for the probe set, the third output of the t-th adapter after the first phase update and the third output of the MLP module are obtained to determine the third loss, including: Based on the probe set, for the Transformer layer with the t-th adapter to be trained inserted, the KL divergence between a pair of third outputs corresponding to the Transformer layer is obtained; the pair of third outputs corresponding to the Transformer layer includes: the third output of the t-th adapter after the first stage update in the Transformer layer, and the third output of the MLP module in the Transformer layer. The third loss is obtained based on the KL divergence of the Transformer layer corresponding to the t-th adapter to be trained from multiple insertions. Based on the first, second, and third losses of the second stage, the model parameters of the t-th adapter after the first stage update are updated to obtain the t-th adapter after training, including: Based on the first, second, and third losses of the second stage, in addition to updating the model parameters of the t-th adapter after the first stage update to obtain the t-th adapter after training, the model parameters of the classification head after the first stage update are also updated to obtain the classification head for t image classification tasks that have been learned, so as to obtain the visual language model for t image classification tasks that have been learned.

8. The continuous learning method for a visual language model for image classification according to any one of claims 1 to 7, characterized in that, The method further includes: Input the image to be classified from any image classification task from the 1st to the tth image classification task into the visual language model that has learned t image classification tasks. For a Transformer layer with t trained adapters inserted, the first target average output is obtained based on the first target output of the 1st to tth trained adapters; the first average target output is fused with the first target output of the MLP module of the Transformer layer according to the first fusion coefficient to obtain the target output of the Transformer layer. By processing the target output of the last Transformer layer using the classification heads of t image classification tasks that have already been learned, the classification prediction result of the image to be classified is obtained.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the continuous learning method for a visual language model for image classification as described in any one of claims 1 to 8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the continuous learning method for a visual language model for image classification as described in any one of claims 1 to 8.

Images