Visual language model continuous learning method based on dynamic hybrid expert adapter

Abstract

The invention belongs to the technical field of efficient fine tuning and continuous learning of visual language models, and discloses a visual language model continuous learning method based on a dynamic hybrid expert adapter. The method comprises the following steps: constructing a dynamic hybrid expert adapter on a part of layers of a pre-trained visual language model, wherein the expert adapter and a routing network dynamically expand along with an incremental task; whether a new expert adapter is added or not is adaptively decided through the dynamic expert extension controller, and parameter redundancy of static hybrid experts is avoided; a potential embedded self-selector is integrated in a model, potential features output by a freezing layer are utilized to automatically judge data distribution and select corresponding routes, an independent external distribution discriminator is replaced, and a unified framework is formed. According to the method, the problem of disastrous forgetting in continuous learning is effectively relieved, parameter redundancy and calculation burden are remarkably reduced, and meanwhile, the zero sample generalization ability of the model for unseen data is kept.

Description

Technical Field

[0001] The present invention belongs to the technical field of efficient fine-tuning and continuous learning of visual language models, and relates to a continuous learning method of a visual language model based on a dynamic hybrid expert adapter. Background Art

[0002] In the field of artificial intelligence, visual language models (VLMs) demonstrate powerful open-world understanding capabilities by fusing visual and textual information. However, when models need to continuously learn new knowledge in dynamic environments, traditional fully supervised training paradigms face significant challenges: integrating new data with historical datasets requires extremely high computational costs. To address this issue, continuous learning (CL) has emerged as an effective solution by incrementally learning new tasks. However, its core bottleneck lies in catastrophic forgetting—the loss of previously acquired knowledge when learning a new task.

[0003] Existing continuous learning methods have significant limitations. Mainstream dynamic extension frameworks (such as Lifelong Learning with a Network of Experts, published by Aljundi R, et al.) mitigate the forgetting problem by incrementally adding task-specific components. However, they are unable to distinguish unseen data and completely ignore the model's zero-shot transfer capabilities.

[0004] Recent approaches to pre-training VLMs (such as Learning transferable visual models from natural language supervision by Radford A et al.) attempt to preserve zero-shot generalization capabilities, but rely on computationally intensive knowledge distillation and parameter regularization techniques, resulting in limited long-term memory and difficulty in practical deployment. A representative approach is Preventing zero-shot transfer degradation in continual learning of vision-language models by Zheng Z et al., which uses parameter regularization in the continual learning of vision-language models.

[0005] Although efficient parameter fine-tuning methods based on adapters (such as Class incremental learning with pre-trained vision-language models published by Liu X et al.) or prompt tuning (such as Learning toprompt for continual learning published by Wang Z et al.) can adapt to new tasks with a small number of parameters, they still suffer from the problem of zero-shot reasoning performance degradation and cannot achieve effective fusion of knowledge between tasks.

[0006] A further improved mixture-of-experts approach (Boosting continual learning of vision-language models via mixture-of-experts adapters, published by Yu J et al.) improves efficiency by activating predefined expert adapters through incremental routing. However, its static mixture-of-experts architecture has inherent flaws. The fixed number of expert pools results in low utilization of early task experts and parameter redundancy. The independent distribution discriminant selector module requires an additional feature extractor, increasing computational complexity. Its parallel design with the backbone network further hinders unified model optimization. Therefore, a dynamic, scalable, parameter-efficient, and zero-shot capable continuous learning framework is urgently needed. Summary of the Invention

[0007] In response to the problems of severe catastrophic forgetting, significant parameter redundancy and degradation of zero-sample generalization ability in existing continuous learning methods of visual language models, the present invention proposes a continuous learning method of visual language models based on dynamic hybrid expert adapters (Dynamic MoE-Adapters). This method solves the parameter efficiency problem by constructing a dynamically scalable hybrid expert architecture on the frozen layer of the pre-trained visual language model, and uses an incremental routing mechanism and an adaptive expert expansion strategy; at the same time, a built-in latent embedding self-selector (LEAS) is designed to replace the external distribution discrimination module, and the coordinated optimization of task perception and zero-sample generalization is achieved through a unified architecture. The present invention significantly reduces the computational burden and the risk of knowledge forgetting in the continuous learning process, effectively overcomes the modal expansion limitations of the static expert pool, and provides a new paradigm for efficient continuous learning of visual language models.

[0008] The technical solution of the present invention:

[0009] A continuous learning method for visual language models based on dynamic hybrid expert adapters, the steps are as follows:

[0010] Step 1: Build a visual language model;

[0011] (1.1) The visual language model is a dual-tower structure model that includes an image encoder and a text encoder. All pre-trained parameters are set to a frozen state to preserve the original zero-shot capability.

[0012] (1.2) During the continuous learning process of the visual language model, the visual language model is trained on multiple tasks in a predetermined order. Each task includes a set of image-text pairs, each consisting of an image and its corresponding text category label. During the continuous learning process, the trained tasks are defined as seen tasks, and the remaining tasks are defined as unseen tasks.

[0013] (1.3) During the training and inference process of the visual language model, the visual language model receives input images and text; the image is subjected to feature extraction by an image encoder to obtain image features; the text is subjected to feature extraction by a text encoder to obtain text features; the cosine similarity between the image features and the text features is calculated to obtain a similarity score indicating the degree of match between the image and text;

[0014] (1.4) The text encoder uses the Transformer architecture, which consists of a series of Transformer layers;

[0015] (1.5) The image encoder includes multiple layers arranged in a processing order, and the multiple layers include, from front to back, one or more layers defined as frozen layers, one layer defined as a recognition layer, and one or more layers defined as subsequent layers; each layer of the image encoder has a self-attention module, a layer normalization module, a feedforward layer module, or other modules at the same time; the image features from the previous layer are processed by the self-attention module and the layer normalization module of the current layer in sequence, and the visual embedding feature tokens of the current layer are output, where the first token contains global information about the image features and is defined as a category token; the visual embedding feature tokens are fed into the feedforward layer to obtain the feedforward layer module of the current layer, and then the features are output;

[0016] (1.6) Each of the one or more frozen layers is frozen during training, and each frozen layer consists of a self-attention module, a layer normalization module, and a feed-forward layer module;

[0017] (1.7) The recognition layer consists of the following modules: self-attention module, layer normalization module, feedforward layer module, dynamic mixture of experts adapter module and potential embedding self-selector module;

[0018] (1.7.1) The dynamic hybrid expert adapter module includes a series of task-specific routing networks and a series of expert adapters;

[0019] (1.7.2) Each task-specific routing network corresponds to each task and is used to determine the expert selection weights of all expert adapters in the current dynamic hybrid expert adapter module; the total number of task-specific routing networks is the number of all currently seen tasks;

[0020] (1.7.3) Each expert adapter is a low-rank adapter LoRA, which consists of a downsampling and upsampling layer; the number of expert adapters increases with the number of tasks;

[0021] (1.7.4) The visual embedding feature tokens in the current recognition layer receive a task ID signal before being sent to the dynamic hybrid expert adapter module to select the task-specific routing network. During training, the task ID is the sequence number of the current task. During inference, the task ID comes from the latent embedding selector module in the recognition layer.

[0022] (1.7.5) After the dynamic hybrid expert adapter module receives the task ID signal, it accepts visual embedding feature tokens and takes the first token, which contains global information about the image features, namely the category token. The category token is fed into the task-specific routing network to generate expert selection weights, activating the k expert adapters most relevant to the current task. Each expert adapter receives the visual embedding feature tokens of the current layer, and the output features are weighted and fused using the expert selection weights to form adapted features. The features are then added to the output features of the feedforward layer of the current layer and fed into the next layer.

[0023] (1.7.6) For the first task, initialize a trainable routing network and a preset number k of expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all k expert adapters are trained end-to-end based on the current task data;

[0024] (1.7.7) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of the recognition layer. All expert adapter parameters remain frozen. At the beginning of training, an expert expansion signal is sent to the dynamic hybrid expert adapter module of the recognition layer. This initializes a new trainable expert adapter within the dynamic hybrid expert adapter module, and the task-specific routing network is also expanded with a set of parameters to facilitate compatibility with the newly added expert adapter.

[0025] (1.7.9) To enable the visual language model to adaptively perceive the distribution characteristics of input data, a self-attention module, a layer normalization module, a feedforward layer module, and a latent embedding self-selector are integrated in the recognition layer. The latent embedding self-selector consists of a series of autoencoders; each autoencoder consists of a downsampling layer and an upsampling layer. Each autoencoder corresponds one-to-one to an expert adapter in the dynamic hybrid expert adapter module of the recognition layer, and each autoencoder in the latent embedding self-selector receives the category token of the current recognition layer.

[0026] (1.7.10) For the first task, the latent embedding self-selector initializes k autoencoders. Each autoencoder is supervised by the reconstruction loss and trained synchronously with the corresponding k expert adapters. After training, the convergence value of the reconstruction loss of each autoencoder is recorded as the feature fingerprint of the task.

[0027] (1.7.11) For each subsequent task, at the beginning of training, the latent embedding autoselector initializes one trainable autoencoder, while the other autoencoders remain frozen. Each autoencoder in the latent embedding autoselector receives the class token of its layer, and after passing it through the downsampling layer and the upsampling layer, obtains the reconstructed token. A reconstruction loss is applied to the class token and the reconstructed token. The newly initialized autoencoder is supervised using the reconstruction loss, and the other frozen autoencoders also calculate the reconstruction loss simultaneously. After training, the convergence value of the reconstruction loss of each autoencoder is recorded as the feature fingerprint of the task.

[0028] (1.8) Each of the one or more subsequent layers is composed of the following modules: a self-attention module, a layer normalization module, a feed-forward layer module, a dynamic mixture of experts adapter module, and a dynamic expert extension controller;

[0029] (1.8.1) The structure and processing of the dynamic hybrid expert adapter module in each subsequent layer are exactly the same as those in the recognition layer in step (1.7);

[0030] (1.8.2) For the first task, initialize a trainable routing network and a preset number k of expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all k expert adapters are trained end-to-end based on the current task data;

[0031] (1.8.2) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of each subsequent layer. All expert adapter parameters remain frozen. During training, if the dynamic hybrid expert adapter module of a subsequent layer receives an expert extension signal, a new trainable expert adapter is initialized in the dynamic hybrid expert adapter module, and the task-specific routing network is also extended with a set of parameters to facilitate compatibility with the newly added expert adapter. The expert extension signal in the dynamic hybrid expert adapter module of each subsequent layer is independent.

[0032] (1.8.3) To eliminate parameter redundancy across a predefined number of expert adapters, a dynamic expert expansion controller is integrated into each subsequent layer. The dynamic expert expansion controller consists of a series of autoencoders, each consisting of a downsampling layer and an upsampling layer. The dynamic expert expansion controller automatically determines whether to trigger the expert expansion signal by analyzing the deviation between the visual embedding feature tokens of the current layer and all the autoencoders within the dynamic expert expansion controller. Once the deviation between the visual embedding feature tokens of the current layer and the existing expert reconstruction capability continuously exceeds a set threshold, the expert expansion signal is triggered, adding a new trainable expert adapter and simultaneously expanding the parameter dimensions of the task-specific routing network.

[0033] (1.8.4) Each autoencoder in the dynamic expert extension controller corresponds one-to-one to each expert adapter in the dynamic hybrid expert adapter module, and maintains a consistent parameter freezing state with the corresponding expert adapter; each autoencoder receives the category token defined in step (1.5) of the current layer. The category token is sequentially passed through the downsampling layer and the upsampling layer to obtain a reconstructed token, and a reconstruction loss is applied to the category token and the reconstructed token;

[0034] (1.8.5) For the first task, each dynamic expert expansion controller initializes k autoencoders, each of which is supervised by a reconstruction loss and trained synchronously with the corresponding k expert adapters;

[0035] (1.8.6) For each subsequent task, at the beginning of training, all current autoencoders remain frozen. As the number of training samples increases, each autoencoder calculates the reconstruction loss and uses the Z-score to continuously monitor the degree of deviation of all visual embedding feature tokens in the autoencoder. The Z-score is calculated as follows:

[0036]

[0037] in, represents the Z score of the i-th autoencoder under the t-th task, Indicates the reconstruction loss of the current data sample after the autoencoder, and Respectively represent the mean and standard deviation of the reconstruction loss obtained by the autoencoder for all current samples of the task;

[0038] (1.8.7) When the Z scores of all autoencoders in the dynamic expert expansion controller exceed the set threshold, it indicates that the current visual embedding feature tokens deviate significantly from the existing autoencoders. Therefore, each expert corresponding to the existing autoencoder cannot adapt to the data distribution of the current task. This triggers the expert expansion signal in the current dynamic hybrid expert adapter module, expanding the new expert adapter and trainable routing network dimensions. In addition, to ensure a one-to-one correspondence between autoencoders and expert adapters, a trainable autoencoder is also synchronously expanded in the dynamic expert expansion controller, also using reconstruction loss for supervision.

[0039] Step 2: Task distribution discrimination and zero-sample reasoning capability preservation;

[0040] (2.1) In order for the visual language model to adaptively perceive the distribution characteristics of the input data, a latent embedding self-selector is integrated in the recognition layer. The feature fingerprints of all tasks seen during training are recorded through steps (1.7.10) and (1.7.11);

[0041] (2.2) During inference, the latent embedding self-selector in the recognition layer compares the features of the current test sample with the feature fingerprints of all previously seen tasks to determine whether the current sample belongs to a previously seen task. If so, it further determines the task ID signal corresponding to the sample and sends it to all dynamic hybrid expert adapter modules according to steps (1.7.4) and (1.8.1). If it belongs to an unseen task, it directly uses the frozen pre-trained parameters for inference, preserving the original zero-shot inference capability of the visual language model.

[0042] (2.3) During inference, the latent embedding self-selector calculates the reconstruction loss of each autoencoder based on the category token of the current test sample in the recognition layer, and then calculates the deviation of the current test sample for all seen tasks. The deviation calculation formula is as follows:

[0043]

[0044] Among them, s t is the scalar value of the distribution deviation of the sample for task t; is a vector of reconstruction losses of all autoencoders associated with the sample on task t; is the feature fingerprint vector of task t, that is, the convergence value of the encoder reconstruction loss recorded after the training of each task; max(·,·) is the convergence value of Element-by-element maximum operation; α is a weighted vector, the value Used to amplify the discriminative contribution of the low reconstruction loss autoencoder; ||·||2 is the L2 norm operation, which calculates the Euclidean distance of the weighted bias vector; β is the normalization factor, which takes the value is the number of autoencoders associated with task t, which is used to eliminate the effect of the difference in the number of experts for different tasks on the bias;

[0045] (2.4) After calculating the deviation of the current test sample for all seen tasks, compare it with the set zero-shot inference threshold. If all deviations exceed the zero-shot inference threshold, it means that the current test sample belongs to a data distribution that has not been seen by the visual language model. Therefore, the dynamic hybrid expert adapter module is not activated and only the pre-trained parameters are used for inference.

[0046] (2.5) If there is a deviation that does not exceed the zero-sample inference threshold, the task ID signal corresponding to the lowest deviation is selected and sent to all dynamic hybrid expert adapter modules to facilitate the selection of task-specific routing networks.

[0047] Beneficial effects of the present invention:

[0048] 1. This invention utilizes the incremental freezing mechanism of dynamic expert adapters and inter-task collaboration strategies, achieving flexible expansion while leveraging the freeze-activate strategy in continuous learning to prevent forgetting of knowledge in each task. Compared to existing continuous learning methods, this method has better generalization and stronger forgetting resistance. In the 11-task MTIL benchmark test, the final accuracy (Last indicator) of historical tasks reached 86.2%, an improvement of 1.1% over the static MoE-Adapters and 4.4% over the traditional ZSCL method, effectively solving the problem of knowledge degradation in continuous learning.

[0049] 2. Based on the threshold routing mechanism of the latent embedding self-selector, this invention strictly isolates unseen data to the original frozen backbone network, achieving a zero-shot transfer accuracy (Transfer metric) of 69.0%, a 3.7% improvement over the baseline CLIP model. Furthermore, compared to external task ID recognition schemes, the latent embedding self-selector further improves task ID recognition by leveraging features within the visual language model, achieving a task matching accuracy of 99.9% on the MTIL benchmark across 11 tasks.

[0050] 3. The dynamic expert expansion controller of this invention enables on-demand allocation of expert resources, significantly reducing redundant parameters in the static hybrid expert solution MoE-Adapters. In an 11-task MTIL benchmark, the number of learnable parameters was only 1.1M, a 98% reduction compared to the static MoE-Adapters and a 99.3% reduction compared to the full parameter fine-tuning solution, breaking through the deployment bottleneck under resource constraints.

[0051] 4. This invention eliminates the external distribution discrimination module in the static MoE-Adapters solution and achieves collaborative optimization of task perception and zero-shot protection by reusing backbone visual features through a latent embedded self-selector. In the MTIL benchmark test of 11 tasks, single-task training GPU memory usage was reduced by 70% and iteration time by 57%, significantly improving efficiency and model consistency.

[0052] 5. The present invention performs well in continuous learning scenarios such as class incremental learning, task incremental learning, and domain incremental learning, and shows good performance under pre-trained backbone networks of various scales and types (CLIP-ViT-B, CLIP-ViT-L, Siglip). In addition, compared with traditional continuous learning schemes and static MoE-Adapters schemes, the present invention also achieves performance improvement in few-sample continuous learning scenarios. In the 5-shot setting of the MTIL benchmark test of 11 tasks, the final accuracy of the historical task (Last index) reaches 76.3%, which is 4.5% higher than the traditional scheme (ZSCL[3]) and 0.6% higher than the static MoE-Adapters[6]. Based on the above reasons, the present invention can be widely promoted in the field of efficient fine-tuning and continuous learning technology of visual language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall framework diagram of the present invention;

[0054] Figure 2 The structure and flow chart of the dynamic expert hybrid adapter, potential embedded self-selector, and dynamic expert extension controller of the present invention during the training process;

[0055] Figure 3 Flowchart of the present invention for discriminating task distribution and retaining zero-sample reasoning capability by embedding a latent self-selector during reasoning DETAILED DESCRIPTION

[0056] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0057] Example

[0058] To verify the forgetting of the visual language model in the process of continual learning, an MTIL test benchmark is constructed using 11 tasks. The order of the test benchmark is: Aircraft, Caltech101, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, SUN397. Each task contains a certain number of image-text data samples, which are divided into training set and test set. Following the setting of CLIP-ViT-B / 16, the image input is preprocessed to a size of 224x224.

[0059] In the process of model training, the AdamW optimizer, the label smoothing is 0.2, and the learning rate is searched between 0.0001 and 0.01. Each task is trained for 1000 iterations, and the number of samples in each iteration batch is 64. All experiments are carried out on a 1*NVIDIA RTX 3090 GPU.

[0060] A visual language model continual learning method based on dynamic hybrid expert adapter, as shown in Figure 1 The steps are as follows:

[0061] Step 1, constructing a visual language model;

[0062] (1.1) The visual language model of the present application is based on CLIP-ViT-B / 16, which uses a double-tower model pre-trained on image-text as an initial frozen visual language model, and sets all pre-training parameters to a frozen state to preserve the original zero-shot capability;

[0063] (1.2) In the process of continual learning of the visual language model, the visual language model is trained on 11 tasks in a predetermined order; wherein each task includes a group of image-text pairs, and the image-text pair is composed of an image and its corresponding text category label; in the process of continual learning, the tasks that have been trained are defined as seen tasks, and the other tasks are defined as unseen tasks;

[0064] (1.3) In the process of training and inference of the visual language model, the visual language model receives input images and text; the image is extracted by the image encoder to obtain the image feature; the text is extracted by the text encoder to obtain the text feature; the cosine similarity between the image feature and the text feature is calculated to obtain the similarity score representing the matching degree of the image and the text;

[0065] (1.4) The text encoder adopts the Transformer architecture, which is composed of 12 Transformer layers;

[0066] (1.5) The image encoder is built based on the ViT-B / 16 structure and contains 12 layers arranged in processing order. These layers include, from front to back: 6 layers defined as frozen layers (i.e., layers 1 to 6), 1 layer defined as the recognition layer (i.e., layer 7), and 5 layers defined as subsequent layers (i.e., layers 8 to 12). Each layer of the image encoder has a self-attention module, a layer normalization module, a feedforward layer module, or other modules at the same time. The image features from the previous layer are processed by the self-attention module and layer normalization module of the current layer in sequence, and the visual embedding feature tokens of the current layer are output. The total number of tokens is 197, and the dimension of each token is 512. The first token contains global information of the image features and is defined as a category token. The visual embedding feature tokens are sent to the feedforward layer to obtain the feedforward layer module of the current layer, and then the features are output.

[0067] (1.6) Each of the 6 frozen layers is frozen during training, and each frozen layer consists of a self-attention module, a layer normalization module, and a feed-forward layer module;

[0068] (1.7) The recognition layer consists of the following modules: self-attention module, layer normalization module, feedforward layer module, dynamic mixture of experts adapter module and latent embedding self-selector module;

[0069] (1.7.1) The dynamic hybrid expert adapter module consists of a set of task-specific routing networks and a set of expert adapters; their number increases as the number of tasks seen increases;

[0070] (1.7.2) Each task-specific routing network corresponds to each task and is used to determine the expert selection weights of all expert adapters in the current dynamic hybrid expert adapter module; each task-specific routing network is a single linear layer, with an input dimension of the visual embedding Tokens dimension of the image encoder of 512, and an output dimension of the number of expert adapters corresponding to the task t. The total number of task-specific routing networks is the number of all currently seen tasks;

[0071] (1.7.3) The expert adapter is a low-rank adapter LoRA with rank = 8, consisting of a 512×8 downsampling linear layer that maps high-dimensionality to low-dimensionality and an 8×512 upsampling linear layer that maps low-dimensionality back to high-dimensionality; the number of expert adapters increases with the number of tasks seen;

[0072] (1.7.4) The visual embedding feature tokens in the current recognition layer receive a task ID signal before being sent to the dynamic hybrid expert adapter module to select the task-specific routing network. During training, the task ID is the sequence number of the current task. During inference, the task ID comes from the latent embedding selector module in the recognition layer.

[0073] (1.7.5) After the dynamic hybrid expert adapter module receives the task ID signal, it accepts visual embedding feature tokens and takes the first token, which contains global information about the image features, namely the category token. The category token is fed into the task-specific routing network to generate expert selection weights, activating the two expert adapters most relevant to the current task. Each expert adapter receives the visual embedding feature tokens of the current layer, and the output features are weighted and fused using the expert selection weights to form adapted features. These features are then added to the output features of the feedforward layer of the current layer and fed into the next layer.

[0074] (1.7.6) For the first task, initialize a trainable routing network and a preset number of two expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all two expert adapters are trained end-to-end based on the current task data;

[0075] (1.7.7) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of the recognition layer. All expert adapter parameters remain frozen. At the beginning of training, an expert expansion signal is sent to the dynamic hybrid expert adapter module of the recognition layer. This initializes a new trainable expert adapter within the dynamic hybrid expert adapter module, and the task-specific routing network is also expanded with a set of parameters to facilitate compatibility with the newly added expert adapter.

[0076] (1.7.9) To enable the visual language model to adaptively perceive the distribution characteristics of input data, a self-attention module, layer normalization module, feedforward layer module, and latent embedding self-selector are integrated in the recognition layer. The latent embedding self-selector consists of a series of autoencoders. Each autoencoder consists of a 512×32 downsampling linear layer that maps high-dimensionality to low-dimensionality and a 32×512 upsampling linear layer that maps low-dimensionality back to high-dimensionality. Each autoencoder corresponds one-to-one with an expert adapter in the dynamic hybrid expert adapter module of the recognition layer. Each autoencoder in the latent embedding self-selector receives the 512-dimensional category token of the current recognition layer, passes it through the downsampling layer and the upsampling layer in sequence, and obtains a 512-dimensional reconstruction token. A mean square error loss is applied to the category token and the reconstruction token. This mean square error loss is the reconstruction loss of the autoencoder. The smaller the reconstruction loss, the stronger the reconstruction capability of the autoencoder.

[0077] (1.7.10) For the first task, the latent embedding self-selector initializes two autoencoders. Each autoencoder is supervised using the reconstruction loss as in step (1.7.9) and trained synchronously with the corresponding two expert adapters. After training, the convergence value of the reconstruction loss of each autoencoder is recorded as the feature fingerprint of the task.

[0078] (1.7.11) For each subsequent task, at the beginning of training, the latent embedding self-selector initializes one trainable autoencoder, while the other autoencoders remain frozen. The newly initialized autoencoder is supervised using the reconstruction loss as in step (1.7.9), and the other frozen autoencoders also synchronously calculate the reconstruction loss as in step (1.7.9). After training, the convergence value of the reconstruction loss for each autoencoder is recorded as the characteristic fingerprint of the task.

[0079] (1.8) Each of all subsequent layers consists of the following modules: a self-attention module, a layer normalization module, a feed-forward layer module, a dynamic mixture of experts adapter module, and a dynamic expert extension controller;

[0080] (1.8.1) The structure and processing of the dynamic hybrid expert adapter module in each subsequent layer are exactly the same as those in the recognition layer in step (1.7);

[0081] (1.8.2) For the first task, initialize a trainable routing network and a preset number of two expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all two expert adapters are trained end-to-end based on the current task data;

[0082] (1.8.2) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of each subsequent layer. All expert adapter parameters remain frozen. During training, if the dynamic hybrid expert adapter module of a subsequent layer receives an expert extension signal, a new trainable expert adapter is initialized in the dynamic hybrid expert adapter module, and the task-specific routing network is also extended with a set of parameters to facilitate compatibility with the newly added expert adapter. The expert extension signal in the dynamic hybrid expert adapter module of each subsequent layer is independent.

[0083] (1.8.3) To eliminate parameter redundancy across a predefined number of expert adapters, a dynamic expert expansion controller is integrated into each subsequent layer. The dynamic expert expansion controller consists of a series of autoencoders, each of which consists of a 512×32 downsampling linear layer that maps from high dimensions to low dimensions and a 32×512 upsampling linear layer that maps from low dimensions back to high dimensions. The dynamic expert expansion controller automatically decides whether to trigger the expert expansion signal by analyzing the deviation between the visual embedding feature tokens of the current layer and all the autoencoders within the dynamic expert expansion controller. Once the deviation between the visual embedding feature tokens of the current layer and the existing expert reconstruction capability continues to exceed the set expert expansion threshold, the expert expansion signal is triggered, adding a new trainable expert adapter and synchronously expanding the parameter dimensions of the task-specific routing network.

[0084] (1.8.4) Each autoencoder in the dynamic expert extension controller corresponds one-to-one to each expert adapter in the dynamic hybrid expert adapter module, and maintains a consistent parameter freezing state with the corresponding expert adapter; each autoencoder receives the category token defined in step (1.5) of the current layer. The category token is sequentially passed through the downsampling layer and the upsampling layer to obtain a reconstructed token, and a reconstruction loss is applied to the category token and the reconstructed token;

[0085] (1.8.5) For the first task, each dynamic expert expansion controller initializes two autoencoders, each of which is supervised by a reconstruction loss and trained synchronously with the corresponding two expert adapters;

[0086] (1.8.6) For each subsequent task, at the beginning of training, all current autoencoders remain frozen. As the number of training samples increases, each autoencoder calculates the reconstruction loss and uses the Z-score to continuously monitor the degree of deviation of all visual embedding feature tokens in the autoencoder. The Z-score is calculated as follows:

[0087]

[0088] in, represents the Z score of the i-th autoencoder under the t-th task, Indicates the reconstruction loss of the current data sample after the autoencoder, and Respectively represent the mean and standard deviation of the reconstruction loss obtained by the autoencoder for all current samples of the task;

[0089] (1.8.7) When the Z scores of all autoencoders in the dynamic expert expansion controller exceed the set expert expansion threshold, it indicates that the current visual embedding feature tokens deviate significantly from the existing autoencoders. Therefore, each expert corresponding to the existing autoencoder cannot adapt to the data distribution of the current task. This triggers the expert expansion signal in the current dynamic hybrid expert adapter module, expanding the parameters of the new expert adapter and trainable routing. In addition, to ensure a one-to-one correspondence between autoencoders and expert adapters, a trainable autoencoder is also synchronously expanded in the dynamic expert expansion controller, also using reconstruction loss for supervision.

[0090] Step 2: Task distribution discrimination and zero-sample reasoning capability preservation;

[0091] (2.1) In order for the visual language model to adaptively perceive the distribution characteristics of the input data, a latent embedding self-selector is integrated in the recognition layer. The feature fingerprints of all tasks seen during training are recorded through steps (1.7.10) and (1.7.11);

[0092] (2.2) During inference, the latent embedding self-selector in the recognition layer compares the features of the current test sample with the feature fingerprints of all previously seen tasks to determine whether the current sample belongs to a previously seen task. If so, it further determines the task ID signal corresponding to the sample and sends it to all dynamic hybrid expert adapter modules according to steps (1.7.4) and (1.8.1). If it belongs to an unseen task, it directly uses the frozen pre-trained parameters for inference, preserving the original zero-shot inference capability of the visual language model.

[0093] (2.3) During inference, the latent embedding self-selector calculates the reconstruction loss of each autoencoder based on the category token of the current test sample in the recognition layer, and then calculates the deviation of the current test sample for all seen tasks. The deviation calculation formula is as follows:

[0094]

[0095] Among them, s t is the scalar value of the distribution deviation of the sample for task t; is a vector of reconstruction losses of all autoencoders associated with the sample on task t; is the feature fingerprint vector of task t, that is, the convergence value of the encoder reconstruction loss recorded after the training of each task; max(·,·) is the convergence value of Element-by-element maximum operation; α is a weighted vector, the value Used to amplify the discriminative contribution of the low reconstruction loss autoencoder; ||·||2 is the L2 norm operation, which calculates the Euclidean distance of the weighted bias vector; β is the normalization factor, which takes the value is the number of expert adapters associated with task t, which is used to eliminate the impact of the difference in the number of expert adapters for different tasks on the bias degree;

[0096] (2.4) After calculating the deviation of the current test sample for all seen tasks, compare it with the set zero-shot inference threshold. If all deviations exceed the zero-shot inference threshold, it means that the current test sample belongs to a data distribution that has not been seen by the visual language model. Therefore, the dynamic hybrid expert adapter module is not activated, and only the pre-trained parameters are used for inference. The zero-shot inference threshold of the model after each task training is searched between 2.0 and 3.0.

[0097] (2.5) If there is a deviation that does not exceed the zero-sample inference threshold, the task ID signal corresponding to the lowest deviation is selected and sent to all dynamic hybrid expert adapter modules to facilitate the selection of task-specific routing networks.

[0098] This method demonstrates its technical advantages in an 11-task MTIL benchmark: 1) In terms of forgetfulness resistance, the final accuracy (Last metric) on historical tasks reaches 86.2%, an improvement of 1.1% and 2.6% over the static MoE-Adapters (85.1%) and the traditional ZSCL (83.6%), respectively. 2) In terms of zero-shot capability, the latent embedding self-selector achieves a transfer accuracy (Transfer metric) of 69.0%, surpassing the baseline CLIP (65.3%) by 3.7 percentage points, while achieving a task matching accuracy of 99.9%. 3) In terms of parameter efficiency, the dynamic expert expansion mechanism compresses the number of learnable parameters to 1.1M, a 98% reduction compared to the static solution (59.8M). 4) In terms of computational efficiency, single-task training memory usage is reduced by 70%, and iteration time is shortened by 57%. In particular, the method maintains a Last metric accuracy of 76.3% in the 5-shot few-shot scenario, a 4.5% improvement over ZSCL (71.8%). Experimental results show that the present invention maintains stable performance advantages under various continuous learning settings, providing an efficient solution for the continuous learning of visual language models.

[0099] The present invention uses the incremental freezing mechanism of the dynamic expert adapter and the inter-task collaboration strategy to achieve anti-forgetting of each task knowledge with the help of the freeze-activation strategy in continuous learning while flexibly expanding. Compared with the existing continuous learning methods, it has better generalization and stronger anti-forgetting. The present invention is based on the threshold routing mechanism of the latent embedded self-selector, which strictly isolates the unseen data to the original frozen backbone network. In addition, compared with the external task ID recognition scheme, the latent embedded self-selector further improves the recognition ability of the task ID in the unified model vein by utilizing the internal features of the visual language model. Finally, the dynamic expert expansion controller of the present invention realizes the on-demand allocation of expert resources, significantly reduces the redundant parameters in the static expert hybrid scheme MoE-Adapters, and significantly improves the computational efficiency.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for continuous learning of visual language models based on dynamic hybrid expert adapters, characterized by: Here are the steps: Step 1: Build a visual language model; (1.1) The visual language model is a dual-tower structure model that includes an image encoder and a text encoder. All pre-trained parameters are set to a frozen state to preserve the original zero-shot capability. (1.2) During the continuous learning process of the visual language model, the visual language model is trained on multiple tasks in a predetermined order. Each task includes a set of image-text pairs, each consisting of an image and its corresponding text category label. During the continuous learning process, the trained tasks are defined as seen tasks, and the remaining tasks are defined as unseen tasks. (1.3) During the training and inference process of the visual language model, the visual language model receives input images and text; the image is subjected to feature extraction by an image encoder to obtain image features; the text is subjected to feature extraction by a text encoder to obtain text features; the cosine similarity between the image features and the text features is calculated to obtain a similarity score indicating the degree of match between the image and text; (1.4) The text encoder uses the Transformer architecture, which consists of a series of Transformer layers; (1.5) The image encoder includes multiple layers arranged in a processing order, and the multiple layers include, from front to back, one or more layers defined as frozen layers, one layer defined as a recognition layer, and one or more layers defined as subsequent layers; each layer of the image encoder has a self-attention module, a layer normalization module, a feedforward layer module, or other modules at the same time; the image features from the previous layer are processed by the self-attention module and the layer normalization module of the current layer in sequence, and the visual embedding feature tokens of the current layer are output, where the first token contains global information about the image features and is defined as a category token; the visual embedding feature tokens are fed into the feedforward layer to obtain the feedforward layer module of the current layer, and then the features are output; (1.6) Each of the one or more frozen layers is frozen during training, and each frozen layer consists of a self-attention module, a layer normalization module, and a feed-forward layer module; (1.7) The recognition layer consists of the following modules: self-attention module, layer normalization module, feedforward layer module, dynamic mixture of experts adapter module and potential embedding self-selector module; (1.8) Each of the one or more subsequent layers is composed of the following modules: a self-attention module, a layer normalization module, a feed-forward layer module, a dynamic mixture of experts adapter module, and a dynamic expert extension controller; Step 2: Task distribution discrimination and zero-sample reasoning capability preservation; (2.1) In order for the visual language model to adaptively perceive the distribution characteristics of the input data, a latent embedding self-selector is integrated in the recognition layer. The feature fingerprints of all tasks seen during training are recorded through steps (1.7.10) and (1.7.11); (2.2) During inference, the latent embedding self-selector in the recognition layer compares the features of the current test sample with the feature fingerprints of all previously seen tasks to determine whether the current sample belongs to a previously seen task. If so, it further determines the task ID signal corresponding to the sample and sends it to all dynamic hybrid expert adapter modules according to steps (1.7.4) and (1.8.1). If it belongs to an unseen task, it directly uses the frozen pre-trained parameters for inference, preserving the original zero-shot inference capability of the visual language model. (2.3) During the inference process, the latent embedding self-selector calculates the reconstruction loss of each autoencoder based on the category token of the current test sample in this recognition layer, and then calculates the deviation of the current test sample with respect to all historical tasks. The calculation formula of the deviation is as follows: Among them, s t is the scalar value of the distribution deviation of the sample for task t; is a vector of reconstruction losses of all autoencoders associated with the sample on task t; is the feature fingerprint vector of task t, that is, the convergence value of the encoder reconstruction loss recorded after the training of each task; max(·,·) is the convergence value of Element-by-element maximum operation; α is a weighted vector, the value Used to amplify the discriminative contribution of the low reconstruction loss autoencoder; ||·||2 is the L2 norm operation, which calculates the Euclidean distance of the weighted bias vector; β is the normalization factor, which takes the value is the number of expert adapters associated with task t, which is used to eliminate the impact of the difference in the number of expert adapters for different tasks on the bias degree; (2.4) After calculating the deviation of the current test sample for all seen tasks, compare it with the set zero-shot inference threshold. If all deviations exceed the zero-shot inference threshold, it means that the current test sample belongs to a data distribution that has not been seen by the visual language model. Therefore, the dynamic hybrid expert adapter module is not activated and only the pre-trained parameters are used for inference. (2.5) If there is a deviation that does not exceed the zero-sample inference threshold, the task ID signal corresponding to the lowest deviation is selected and sent to all dynamic hybrid expert adapter modules to facilitate the selection of task-specific routing networks.

2. The method for continuous learning of a visual language model based on a dynamic hybrid expert adapter according to claim 1, characterized in that: Step (1.7) is as follows: (1.7.1) The dynamic hybrid expert adapter module includes a series of task-specific routing networks and a series of expert adapters; (1.7.2) Each task-specific routing network corresponds to each task and is used to determine the expert selection weights of all expert adapters in the current dynamic hybrid expert adapter module; the total number of task-specific routing networks is the number of all currently seen tasks; (1.7.3) Each expert adapter is a low-rank adapter LoRA, which consists of a downsampling and upsampling layer; the number of expert adapters increases with the number of tasks; (1.7.4) The visual embedding feature tokens in the current recognition layer receive a task ID signal before being sent to the dynamic hybrid expert adapter module to select the task-specific routing network. During training, the task ID is the sequence number of the current task. During inference, the task ID comes from the latent embedding selector module in the recognition layer. (1.7.5) After the dynamic hybrid expert adapter module receives the task ID signal, it accepts visual embedding feature tokens and takes the first token, which contains global information about the image features, namely the category token. The category token is fed into the task-specific routing network to generate expert selection weights, activating the k expert adapters most relevant to the current task. Each expert adapter receives the visual embedding feature tokens of the current layer, and the output features are weighted and fused using the expert selection weights to form adapted features. The features are then added to the output features of the feedforward layer of the current layer and fed into the next layer. (1.7.6) For the first task, initialize a trainable routing network and a preset number k of expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all k expert adapters are trained end-to-end based on the current task data; (1.7.7) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of the recognition layer. All expert adapter parameters remain frozen. At the beginning of training, an expert expansion signal is sent to the dynamic hybrid expert adapter module of the recognition layer. This initializes a new trainable expert adapter within the dynamic hybrid expert adapter module, and the task-specific routing network is also expanded with a set of parameters to facilitate compatibility with the newly added expert adapter. (1.7.9) To enable the visual language model to adaptively perceive the distribution characteristics of the input data, a self-attention module, a layer normalization module, a feedforward layer module, and a latent embedding self-selector are integrated in the recognition layer. The latent embedding self-selector consists of a series of autoencoders, each of which consists of a downsampling layer and an upsampling layer. Each autoencoder corresponds one-to-one to an expert adapter in the dynamic hybrid expert adapter module of the recognition layer, and each autoencoder in the latent embedding self-selector receives the category token of the current recognition layer. (1.7.10) For the first task, the latent embedding self-selector initializes k autoencoders. Each autoencoder is supervised by a reconstruction loss and trained synchronously with the corresponding k expert adapters. After training, the convergence value of the reconstruction loss of each autoencoder is recorded as the feature fingerprint of the task. (1.7.11) For each subsequent task, at the beginning of training, the latent embedding autoselector initializes one trainable autoencoder, and the other autoencoders remain frozen; each autoencoder in the latent embedding autoselector accepts the category token of this layer, and after passing through the downsampling layer and upsampling layer in sequence, obtains the reconstructed token, and performs reconstruction loss on the category token and the reconstructed token; the newly initialized autoencoder uses the reconstruction loss for supervision, and the other frozen autoencoders also calculate the reconstruction loss synchronously; after training, the convergence value of the reconstruction loss of each autoencoder is recorded as the feature fingerprint of the task.

3. The method for continuous learning of a visual language model based on a dynamic hybrid expert adapter according to claim 1, characterized in that: Step (1.8) is as follows: (1.8.1) The structure and processing of the dynamic hybrid expert adapter module in each subsequent layer are exactly the same as those in the recognition layer in step (1.7); (1.8.2) For the first task, initialize a trainable routing network and a preset number k of expert adapters in the dynamic hybrid expert adapter module of the recognition layer; all k expert adapters are trained end-to-end based on the current task data; (1.8.2) For each subsequent new task, a new trainable routing network, i.e., the task-specific routing network, is initialized in the dynamic hybrid expert adapter module of each subsequent layer. All expert adapter parameters remain frozen. During training, if the dynamic hybrid expert adapter module of a subsequent layer receives an expert extension signal, a new trainable expert adapter is initialized in the dynamic hybrid expert adapter module, and the task-specific routing network is also extended with a set of parameters to facilitate compatibility with the newly added expert adapter. The expert extension signal in the dynamic hybrid expert adapter module of each subsequent layer is independent. (1.8.3) To eliminate parameter redundancy across a predefined number of expert adapters, a dynamic expert expansion controller is integrated into each subsequent layer. The dynamic expert expansion controller consists of a series of autoencoders, each consisting of a downsampling layer and an upsampling layer. The dynamic expert expansion controller automatically determines whether to trigger the expert expansion signal by analyzing the deviation between the visual embedding feature tokens of the current layer and all the autoencoders within the dynamic expert expansion controller. Once the deviation between the visual embedding feature tokens of the current layer and the existing expert reconstruction capability continuously exceeds the set expert expansion threshold, the expert expansion signal is triggered, adding a new trainable expert adapter and simultaneously expanding the parameter dimensions of the task-specific routing network. (1.8.4) Each autoencoder in the dynamic expert extension controller corresponds one-to-one to each expert adapter in the dynamic hybrid expert adapter module, and maintains a consistent parameter freezing state with the corresponding expert adapter; each autoencoder receives the category token defined in step (1.5) of the current layer. The category token is sequentially passed through the downsampling layer and the upsampling layer to obtain a reconstructed token, and a reconstruction loss is applied to the category token and the reconstructed token; (1.8.5) For the first task, each dynamic expert expansion controller initializes k autoencoders, each supervised by a reconstruction loss, and trained synchronously with the corresponding k expert adapters. (1.8.6) For each subsequent task, at the beginning of training, all current autoencoders remain frozen. As the number of training samples increases, each autoencoder calculates the reconstruction loss and uses a Z-score to continuously monitor the degree of deviation of all visual embedding feature tokens in the autoencoder. The Z-score is calculated as follows: in, represents the Z score of the i-th autoencoder under the t-th task, Indicates the reconstruction loss of the current data sample after the autoencoder, and Respectively represent the mean and standard deviation of the reconstruction loss obtained by the autoencoder for all current samples of the task; (1.8.7) When the Z scores of all autoencoders in the dynamic expert expansion controller exceed the set expert expansion threshold, it indicates that the current visual embedding feature tokens deviate significantly from the existing autoencoders. Therefore, each expert corresponding to the existing autoencoder cannot adapt to the data distribution of the current task. This triggers the expert expansion signal in the current dynamic hybrid expert adapter module, expanding the parameters of the new expert adapter and trainable routing. In addition, to ensure a one-to-one correspondence between autoencoders and expert adapters, a trainable autoencoder is also synchronously expanded in the dynamic expert expansion controller, also using reconstruction loss for supervision.

Images