Methods for preserving image and text knowledge from pre-trained vision-language models in continual learning

Abstract

The present invention provides a method for retaining the pre-trained image and text knowledge of a visual-language model during continuous learning, designs a distribution-aware interference-free knowledge integration mechanism, and provides an innovative solution for the continuous learning application of the visual-language model, effectively solving the backward forgetting problem while retaining the model's pre-trained image and text knowledge. In the present invention, by modifying the self-attention mechanism, training the residual attention branch without changing the original attention score of the full residual design and the zero-initialization strategy, the impact of new task learning on pre-trained knowledge is reduced and noise interference is minimized. At the same time, a distribution-aware integration and calibration mechanism is introduced, which can accurately identify and adapt to new image and text distributions and protect pre-trained image and text knowledge. The present invention enables the visual-language model to efficiently adapt to new image and text data and maintain high accuracy.

Description

Technical Field

[0001] The present invention relates to an application technology of a computer vision-language model, and in particular to a method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning. Background Art

[0002] Supervised learning techniques train networks by accessing all data, which can lack flexibility when expanding to acquire knowledge from new tasks. Continual learning has emerged as a solution, enabling models to continuously train on continuously arriving data while retaining previously learned information. Traditional continuous learning settings consider the introduction of new categories or changes in domain distribution, which are known as class-incremental learning and domain-incremental learning. However, by considering only one type of increment, these existing works limit their applicability in complex real-world scenarios.

[0003] Consider a more challenging Domain-Class Incremental Learning setting, where the domain data distribution and the categories to be classified in all tasks can remain variable, such as Figure 1 (a) shows that in this case, techniques based on ordinary image encoders are not feasible because their classification head designs are not scalable. Recently, the emergence of contrastively trained vision-language models (VLMs), such as CLIP, has made it possible to solve this demanding but practical problem. Vision-language models are trained on web-scale image-text pairs and have strong zero-shot generalization capabilities, capable of recognizing a virtually unlimited number of categories, making them capable of handling this severe task variation scenario.

[0004] However, using vision-language models introduces new challenges for incremental training. Traditional continual learning schemes aim to prevent the model from forgetting previously learned knowledge, which is called "forward forgetting". Existing work has explored the potential of regularization mechanisms, rehearsal buffers, and architectural designs to mitigate backward forgetting, with encouraging results. However, when these methods are applied to vision-language models, a different form of catastrophic forgetting occurs: the models tend to forget what they learned during the pre-training phase, thereby compromising their strong zero-shot generalization capabilities. This problem is called "backward forgetting" because it occurs when vision-language models make backward predictions on data from an unseen distribution. Figure 1 (a) shows these two types of forgetting.

[0005] Recent work ZSCL attempts to address the forward forgetting problem on CLIP. They introduced a large-scale reference dataset for knowledge distillation and adopted a weight integration scheme. However, this approach requires a lot of computation and external data, which may not be feasible in real-world scenarios. At the same time, existing parameter-efficient continuous learning methods based on vision-language models mostly use a prompt fine-tuning mechanism, which fails to retain pre-training knowledge and leads to a decline in zero-shot capabilities, such as Figure 1 As shown in (b).

[0006] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0007] Through in-depth research, the inventors attributed the aforementioned problem of backward forgetting in the continuous learning process of the visual-language model to the information interference caused by the introduction of new task-specific parameters on the pre-trained image and text knowledge.

[0008] To solve the above problems, the present invention provides a method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning.

[0009] To achieve the above object, the present invention adopts the following technical solutions:

[0010] A method for preserving image and text knowledge from pre-trained vision-language models during continuous learning, including:

[0011] Set up multiple tasks that arrive sequentially, each task contains an image dataset and corresponding one-hot labels, as well as a class name set that associates the label index with the class name;

[0012] Prepare a pre-trained vision-language model, which contains an image encoder and a text encoder to generate aligned feature representations of image-text pairs;

[0013] For each newly arrived task, a set of lightweight prompts is created to assist the model in processing images and texts, forming a prompt pool. The prompt pool is used to store the lightweight prompts for use during reasoning and to assist the model in recalling and utilizing previously learned image and text knowledge during the continuous learning phase.

[0014] By using the knowledge injection mechanism without information interference and modifying the self-attention mechanism, the residual attention branch is trained without changing the original attention score;

[0015] Initialize the parameters of the residual attention branch to zero so as not to affect the pre-trained knowledge of the model at the beginning of training;

[0016] A multivariate Gaussian distribution model is built for each task to capture and represent the distribution of image and text features of the task;

[0017] When a new image or text test sample arrives, a multivariate Gaussian model is used to determine which learned image or text task the sample is most likely to come from.

[0018] According to the distribution recognition results of image or text test samples, the weight of the residual attention output is adjusted to assign lower weights to unfamiliar samples to protect pre-training knowledge;

[0019] For each new image or text task, select the most relevant prompts for the current task; attach the selected prompts to a pre-trained and frozen vision-language model; perform inference, using the attached prompts to help the model recall and utilize previously learned image and text knowledge; adjust the model's output based on a multivariate Gaussian distribution and residual attention weight adjustment mechanism to optimize learning of new image or text tasks and retain old knowledge.

[0020] Further:

[0021] The image encoder and text encoder of the visual-language model are jointly trained to generate tightly aligned feature representations between images and text samples; the image encoder is responsible for encoding the input image into a feature vector, and the text encoder embeds the potential class name into a template and encodes it to form a text embedding; during the inference phase, the visual-language model converts the input image into a feature vector through the image encoder, and the text encoder converts the class name into a text embedding; the prediction of the visual-language model is determined based on the maximum similarity score between the image feature vector and all text embeddings, where the similarity score is calculated by cosine similarity.

[0022] During inference, the most relevant prompts for the current task are selected and attached to the pre-trained visual-language model to recover the learned knowledge; the selected prompt sequence is concatenated with the feature vector of the layer in the model to generate a new input vector to improve feature embedding and enhance classification accuracy; in the continuous learning phase, the average feature representation of each task is determined by maximizing cosine similarity or clustering algorithm to provide a basis for prompt selection; the similarity between the new test sample and the learned task is determined through the key lookup mechanism to select the most relevant prompt; the selected prompt is attached to the frozen visual-language model to perform the inference process.

[0023] The operation of the key lookup mechanism is expressed as:

[0024]

[0025] I sis the key most relevant to the test sample, x is the test sample, <f(x),I i > is used to calculate the feature representation and key I of sample x after being processed by encoder f i The similarity function between them, I represents the set of keys, It means finding the key that maximizes the similarity function.

[0026] The information-free knowledge injection mechanism includes:

[0027] The input tokens are first transformed into query Q, key K, and value V matrices via linear projection:

[0028] Q e =x e W Q +b Q ;K e =x e W K +b K ; V e =x e W V +b V

[0029] where W∈R d×d and b∈R d are pre-trained parameters; self-attention calculation is then performed to produce the output matrix in the following way:

[0030]

[0031] in Constrained attention result Attn(Q e ,K e )∈R L×L The sum of the elements in is 1;

[0032] Calculate the self-attention output within the input token and the cross-attention output between the prompt and the input token respectively, train only one residual attention branch, and keep the original attention score unchanged; through the newly introduced key K r Sum V r , the output of the residual attention branch is:

[0033]

[0034] Residual output O r ∈R L×d is obtained by comparing it with the original output O L Derived in an orthogonal way;

[0035] By adding rThe learning knowledge stored in is embedded in the output; in the continuous training phase, only the learnable key K is updated r Sum V r .

[0036] By passing the value parameter V of the residual attention branch r Initialized to zero, the addition of residual attention is forced to become the identity function:

[0037]

[0038] Only at the beginning constrained to zero while maintaining K r Random initialization.

[0039] The establishment of the multivariate Gaussian distribution model specifically includes:

[0040] During the training phase, a

[0041]

[0042] in is the image feature extracted by the frozen encoder, μ i is the image feature extracted by the frozen encoder Expected value Σ i is the covariance matrix;

[0043] The multivariate Gaussian model is used to determine which learned image or text task the sample is most likely to come from, specifically including:

[0044] Using the estimated multivariate Gaussian distribution, calculate the probability that the test sample x is drawn from each The likelihood, which is calculated by calculating the logarithm of the probability density S i As the scoring function for sample x on each learned task:

[0045]

[0046] in is the probability density function;

[0047] The weight of the residual attention output is adjusted, specifically including:

[0048] Adjust the weight of the residual attention output according to the distribution recognition results of the test samples;

[0049] Among them, the maximum score is used To weight the residual attention output:

[0050]

[0051] Where N is the total number of tasks, Is a fraction Mapping function that scales to the range [0,1].

[0052] Mapping Function Using Sigmoid function

[0053] A computer-readable storage medium stores a computer program, which implements the method when executed by a processor.

[0054] A computer program product comprises a computer program, which implements the method when executed by a processor.

[0055] The present invention has the following beneficial effects:

[0056] The present invention provides a method for retaining the image and text knowledge pre-trained by a vision-language model during continuous learning. A distribution-aware interference-free knowledge integration (DIKI) mechanism is designed, providing an innovative solution for the continuous learning application of vision-language models, effectively solving the backward forgetting problem while retaining the model's pre-trained image and text knowledge. The main advantage of the present invention lies in its unique knowledge integration mechanism. Among them, by modifying the self-attention mechanism, training the residual attention branch without changing the original attention score, and the full residual design and zero initialization strategy, the impact of new task learning on pre-trained knowledge is reduced and noise interference is minimized. At the same time, a distribution-aware integration calibration mechanism is introduced, which can accurately identify and adapt to new image and text distributions and protect pre-trained image and text knowledge. Through comprehensive experimental verification, the present invention shows significant advantages in reducing the number of trainable parameters and shortening training time, while maintaining industry-leading performance. The present invention is suitable for continuous fine-tuning tasks of various vision-language models such as defect detection, object classification, and scene recognition, enabling the model to efficiently adapt to new image and text data and maintain high accuracy.

[0057] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Schematic diagram for analyzing the forgetting problem of vision-language models in existing solutions.

[0059] Figure 2 Schematic diagram of the self-attention structure and residual attention branch of an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0061] An embodiment of the present invention provides a method for retaining image and text knowledge pre-trained by a vision-language model during continuous learning, including:

[0062] Set up multiple tasks that arrive sequentially, each task contains an image dataset and corresponding one-hot labels, as well as a class name set that associates the label index with the class name;

[0063] Prepare a pre-trained vision-language model, which contains an image encoder and a text encoder to generate aligned feature representations of image-text pairs;

[0064] For each newly arrived task, a set of lightweight prompts is created to assist the model in processing images and texts, forming a prompt pool. The prompt pool is used to store the lightweight prompts for use during reasoning and to assist the model in recalling and utilizing previously learned image and text knowledge during the continuous learning phase.

[0065] By using the knowledge injection mechanism without information interference and modifying the self-attention mechanism, the residual attention branch is trained without changing the original attention score;

[0066] Initialize the parameters of the residual attention branch to zero so as not to affect the pre-trained knowledge of the model at the beginning of training;

[0067] A multivariate Gaussian distribution model is built for each task to capture and represent the distribution of image and text features of the task;

[0068] When a new image or text test sample arrives, a multivariate Gaussian model is used to determine which learned image or text task the sample is most likely to come from.

[0069] According to the distribution recognition results of image or text test samples, the weight of the residual attention output is adjusted to assign lower weights to unfamiliar samples to protect pre-training knowledge;

[0070] For each new image or text task, select the most relevant prompts for the current task; attach the selected prompts to a pre-trained and frozen vision-language model; perform inference, using the attached prompts to help the model recall and utilize previously learned image and text knowledge; adjust the model's output based on a multivariate Gaussian distribution and residual attention weight adjustment mechanism to optimize learning of new image or text tasks and retain old knowledge.

[0071] Embodiments of the present invention provide a method for preserving pre-trained image and text knowledge in vision-language models during continuous learning. A distribution-aware, non-interference knowledge integration mechanism is designed, offering an innovative solution for continuous learning applications of vision-language models. This mechanism effectively addresses the backward forgetting problem while preserving the model's pre-trained image and text knowledge. The main advantage of the present invention lies in its unique knowledge integration mechanism. This mechanism utilizes a knowledge injection mechanism without information interference, a fully residual design that modifies the self-attention mechanism, and a zero-initialization strategy to train the residual attention branch without changing the original attention score. This reduces the impact of new task learning on pre-trained knowledge and minimizes noise interference. Furthermore, a distribution-aware integration calibration mechanism is introduced. By using a multivariate Gaussian distribution to capture and represent the distribution of image and text features of a task, the mechanism assesses which previously learned image or text task a new image or text test sample is most likely to have come from. Based on the distribution recognition results of the image or text test sample, the weights of the residual attention output are adjusted to optimize learning of the new task. This mechanism enables accurate recognition and adaptation to new image and text distributions while preserving pre-trained image and text knowledge. Comprehensive experimental verification demonstrates that the present invention demonstrates significant advantages in reducing the number of trainable parameters and shortening training time, while maintaining industry-leading performance. The present invention performs continuous fine-tuning of the vision-language model, enabling the model to efficiently adapt to new data and maintain high accuracy.

[0072] Specific embodiments of the present invention are further described below.

[0073] (1) Incremental learning settings

[0074] Incremental learning aims to learn different tasks continuously without forgetting previously learned knowledge. Consider N tasks that arrive in sequence. Each task Contains a dataset in is an image, is the corresponding one-hot label in the current dataset, N i is the number of image samples. In addition, it also includes a set of class names The label index is associated with the category name used by the vision-language model. Different from the previous class incremental learning and domain incremental learning settings, a more practical continuous learning setting is used: domain-class incremental learning. In this setting, the domain distribution and the categories to be recognized are continuously changing between different tasks, i.e., C i ≠C j and P(D i )≠P(D j ) for any i≠j, where P represents the data distribution of the task dataset.

[0075] (2) Vision-Language Model Structure

[0076] For the challenging domain-class incremental learning task, training models based on conventional image encoders (such as ResNets and ViTs) is not practical for gradually learning strongly changing domains and categories. Therefore, pre-trained visual-language models are introduced for this task because of their strong zero-sample transfer capabilities. CLIP consists of an image encoder f and a text encoder g, which are trained to generate tightly aligned feature representations for paired image-text samples. At inference time, f first encodes the input image x into a feature vector f(x). At the same time, the potential class name c is embedded in the template, such as "a photo of c", and then encoded by g to form a text embedding. The model prediction is determined by the maximum similarity score between the image embedding and all text embeddings, as s j = <f(x),t j >, where <·,·> represents the cosine similarity.

[0077] (3) Task-independent cue learning

[0078] Learn and store a set of lightweight prompts for each task to form a prompt pool for the continuous learning phase. The formula is:

[0079] P={P1,P2,…,P N},whereP i ∈R l×d ,

[0080] where N is the number of tasks, l and d are the prompt length and feature embedding dimension.

[0081] At inference time, the trained hints are selected and appended to the pre-trained frozen model to recover the learned knowledge. Assume x e ∈R L×d is the feature vector of the hth layer of the Transformer, and then the hint can be added to x e Before, generate the prompt input:

[0082]

[0083] in Yes Select prompt P s The embedding vector of , where “;” represents the concatenation operation along the token length dimension. By embedding knowledge in this way, better image and text feature embedding is generated, improving the final classification accuracy.

[0084] The above-mentioned prompt selection process is achieved through query-key matching. In the continuous training phase, the average feature representation of each task is learned by maximizing cosine similarity or applying clustering algorithms. When a test sample x arrives, the key lookup mechanism is executed:

[0085]

[0086] Most relevant keys I s , select the corresponding prompt P s And attached to the frozen vision-language model to perform the reasoning process.

[0087] (4) Knowledge injection without information interference

[0088] Instead of training a set of cue vectors for each task, we focus on following the modification of the self-attention mechanism, which is similar to the parameter-efficient fine-tuning method in natural language processing. e ∈R L×d The multi-head self-attention mechanism is performed. For simplicity, the multi-head design is omitted here and only the single-head case is considered, which can be naturally extended to the multi-head case. The input token is first converted into the query Q, key K and value V matrices through linear projection:

[0089] Q e =x e W Q +b Q ;K e =x e W K +b K ; V e =x e W V +b V ,

[0090] where W∈R d×d and b∈R d are pre-trained parameters. Then the self-attention calculation is performed to produce the output matrix in the following way:

[0091]

[0092] in The attention result Attn(Q e ,K e )∈R L×L The sum of the elements in is 1.

[0093] The conventional hint adjustment method appends a trainable hint before the input token, e ∈R L×d Expand to x p ∈R (l +L)×d Then calculate And passed to the softmax function. In the softmax calculation, the attention scores of the input tokens and the prompts interact and affect each other, resulting in the inevitable loss of pre-training knowledge, such as Figure 2 As shown in (a).

[0094] To solve this problem, we calculate the self-attention output within the input token and the cross-attention output between the prompt and the input token separately, as Figure 2 In other words, only one residual attention branch is trained, and the existing attention score is kept unchanged. r Sum V r , the output of the residual attention branch can be expressed as:

[0095]

[0096] Here the residual output is O r ∈R L×d is obtained by comparing it with the original output O L It is derived in an orthogonal way and will not affect the original attention process. Finally, by adding r The learned knowledge stored in is embedded in the output. During the continuous training phase, only the learnable key K is updated r Sum V r , instead of the commonly used hint P. In order to keep the sequence length unchanged, no query parameters are introduced here.

[0097] Ideally, an ideal residual block should not affect the original branch before training on the downstream dataset, i.e., at initialization. Widely used protocols use uniform or normal distribution to initialize cues, which injects random noise into the pre-trained vision-language model even if no knowledge is learned. Specifically, by setting the parameter V r Initialized to zero, forcing the addition of residual attention to become the identity function:

[0098]

[0099] Here, only the value is set at the beginning constrained to zero while maintaining K r Random initialization. This is because K r and V r Initializing the matrix to zero will prevent K r Update through the gradient flow and make V r degenerates into vectors with the same value.

[0100] Zero-initialization, traditionally used to ensure a stable and progressive training mechanism, is applied in a domain-like incremental learning scenario by incorporating zero-initialization into a residual attention design that can inject new knowledge into a pre-trained vision-language model with minimal noise.

[0101] (5) Distribution-based knowledge injection correction

[0102] In order to determine whether a test sample belongs to a learned task, different from existing methods, instead of maintaining a single key vector, a feature distribution is maintained. Multivariate Gaussian distribution is used here. Specifically, during the training phase, a

[0103]

[0104] in is the image feature extracted by the frozen encoder. Through these estimated distributions, we can calculate the probability that a test sample is drawn from each Here, the scoring function for the input x on each learned task is calculated by calculating the logarithm of the probability density:

[0105]

[0106]

[0107] in is the probability density function. Intuitively, a score S i Samples with higher σ are more likely to be drawn from task i and should be introduced into the parameters predicted by the model. In addition, it should be considered that the input sample x may come from some new distribution. If all S i Therefore, the maximum score is used here. To weight the residual attention output:

[0108]

[0109] in Is a fraction A mapping function that scales to the range [0,1]. The Sigmoid function is found here Through this distribution-aware ensemble calibration mechanism, the pre-trained zero-shot capability of VLMs can be better preserved by assigning lower weights to unfamiliar images, further addressing the forward forgetting problem.

[0110] Experimental evaluation

[0111] Evaluation Metrics

[0112] To assess the forward and backward forgetting issues mentioned in the introduction, we use the Transfer, Avg, and Last metrics. The Last score represents the model performance after all continuous training runs, representing the degree of forward forgetting and is widely used in conventional continuous learning. To address backward forgetting, i.e., the loss of zero-shot capability, the model's average accuracy on tasks i+1, i+2, ..., N is evaluated after training for task i. This metric is called Transfer. Finally, Avg refers to the average accuracy across all time steps.

[0113] Comparison method

[0114] The present invention (hereafter referred to as DIKI) was compared with full-parameter fine-tuning and parameter-efficient fine-tuning methods. For full-parameter fine-tuning, ZSCL, ZSCL*, LwF, iCaRL, LwF-VR, and WiSE-FT were selected. For parameter-efficient methods, L2P, DualPrompt, and S-Prompts were selected, as these methods have similar task-specific parameter training procedures as DIKI. Note that the original L2P and DualPrompt methods were designed for ViT and are replicated here on CLIP.

[0115] Comparison results

[0116] Table 1 contains the Transfer, Avg, and Last scores of all methods on the MTIL benchmark. "Extradata" includes the memory buffer and reference dataset used in the distillation, and "#Params" refers to the number of trainable parameters. The "Zero-shot" results are obtained by only utilizing the original CLIP weights for each task and are used as a comparison reference for transfer metrics. Note that the transfer score may be higher than the zero-shot result because the knowledge from the current task i may contain some task-invariant information, which can improve the performance of future tasks i+1,i+2,...,N. The "upper bound" is calculated by applying the full parameter fine-tuning technique on each independent dataset and serves as a guide for the final score.

[0117] As shown in bold, DIKI surpasses the previous best methods on all three metrics with only 0.86% of trainable parameters while alleviating the need for any external data. Thanks to the task-specific parameter training technique, DIKI can memorize knowledge from previous tasks without the need for a rehearsal buffer and parameter set, while maintaining a high final score and low computational complexity. In addition, compared with task-specific prompt adjustment methods (L2P, DualPrompt, and S-Prompts), DIKI achieves significant improvements on transfer metrics, demonstrating that the DIKI mechanism can effectively inject new information into a fixed backbone network without interfering with pre-trained knowledge.

[0118] Table 1 MTIL experimental results

[0119]

[0120] We also conducted experiments on the 16-shot MTIL-FS benchmark. The results are shown in Table 2. Because only a small number of parameters are updated, DIKI achieves significant improvements over ZSCL in full-parameter training. Furthermore, due to minimal noise introduced, the full-residual knowledge injection design demonstrates stronger competitiveness compared to other interruptive prompt adjustment methods when training data is insufficient.

[0121] Table 2 MTIL-FS experimental results

[0122]

[0123] The effectiveness of multiple modules of the embodiments of the present invention is verified here, and an ablation study of each module of DIKI is performed on the MTIL benchmark, as shown in Table 3. Prompt refers to the basic task-independent prompt learning scheme, ResAttn refers to the residual attention mechanism, Z-init refers to zero initialization, and Calib refers to distribution-based knowledge injection correction.

[0124] First, consider the Transfer score, or zero-shot capability: As can be seen from the first two rows, zero-initialization is ineffective against hint adjustment methods, as they can still perturb pre-trained knowledge via the softmax function in the attention computation. However, through the residual attention design, the effect of zero-initialization is activated. Together, they prevent the introduction of irrelevant information into the frozen backbone. Due to the fully residual property, distribution-aware calibration can be exploited to further improve performance by identifying unseen distributions.

[0125] Considering the Last metric, DIKI's non-interference mechanism, due to its clear information injection process, stores more task knowledge, thereby improving the accuracy of the final state. However, since DIKI's zero-initialization and distribution-aware calibration are designed to improve the retention of pre-training knowledge, their addition did not lead to an increase in the Last score. This demonstrates the correctness of the theory.

[0126] Table 3 Ablation experiment results

[0127]

[0128] In summary, this paper proposes an innovative distribution-aware interference-free knowledge integration (DIKI) solution, which effectively addresses the backward forgetting problem encountered in the continuous learning process of vision-language models. Through the following key technical innovations, it effectively retains and integrates pre-trained image and text knowledge without sacrificing computational and parameter efficiency:

[0129] 1. Knowledge Integration Mechanism: This paper uses a fully residual design and zero-initialization strategy to create a mechanism that can inject new task-specific knowledge without interfering with pre-trained image and text knowledge. Compared with fine-tuning based on hint learning, this approach significantly reduces noise interference with the pre-trained model.

[0130] 2. Distribution-Aware Calibration: We further introduce a calibration mechanism that explicitly identifies and handles samples from unseen image and text distributions. This mechanism improves the model's generalization to new image and text situations by controlling the amount of information implanted into these samples.

[0131] 3. Performance improvement: Through extensive experimental verification, the present invention proves the advantages of its method in reducing the number of trainable parameters (only 0.86% of training parameters are used) and shortening training time, while achieving industry-leading performance.

[0132] 4. Practical Application Scenarios: This invention has broad practical application value and can be applied to various vision-language model continuous fine-tuning tasks, such as defect detection, object classification, and scene recognition. It enables the model to continuously adapt to new image and text data while maintaining high accuracy.

[0133] Furthermore, compared to existing methods such as ZSCL, which require extensive computation and external data, the DIKI framework provides a more efficient and practical solution. It avoids the information interference caused by the introduction of new tasks on pre-trained image and text knowledge, while maintaining the model's parameter efficiency and computational efficiency, which is particularly important for real-world applications.

[0134] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0135] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0136] An embodiment of the present invention further provides a processor, which executes a computer program and at least performs the method described above.

[0137] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a magnetic disk memory or a magnetic tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0138] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0139] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0141] Those skilled in the art will appreciate that all or part of the steps of the above-mentioned method embodiments may be implemented by hardware associated with program instructions, and the aforementioned program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the above-mentioned method embodiments. The aforementioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0142] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0143] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0144] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0145] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0146] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art will recognize that, without departing from the scope of the present invention, several equivalent substitutions or obvious variations can be made, and the performance or use of the same should be considered to fall within the scope of protection of the present invention.

Claims

1. A method for preserving image and text knowledge from pre-trained vision-language models in continuous learning, characterized in that include: Set up multiple tasks that arrive sequentially, each task contains an image dataset and corresponding one-hot labels, as well as a class name set that associates the label index with the class name; Prepare a pre-trained vision-language model, which contains an image encoder and a text encoder to generate aligned feature representations of image-text pairs; For each newly arrived task, a set of lightweight prompts is created to assist the model in processing images and texts, forming a prompt pool. The prompt pool is used to store the lightweight prompts for use during reasoning and to assist the model in recalling and utilizing previously learned image and text knowledge during the continuous learning phase. By using the knowledge injection mechanism without information interference and modifying the self-attention mechanism, the residual attention branch is trained without changing the original attention score; Initialize the parameters of the residual attention branch to zero so as not to affect the pre-trained knowledge of the model at the beginning of training; A multivariate Gaussian distribution model is built for each task to capture and represent the distribution of image and text features of the task; When a new image or text test sample arrives, a multivariate Gaussian model is used to determine which learned image or text task the sample is most likely to come from. Based on the distribution recognition results of image or text test samples, the weights of the residual attention output are adjusted to assign lower weights to unfamiliar samples to protect pre-trained knowledge; For each new image or text task, select the cue that is most relevant to the task at hand; The selected prompts are attached to a pre-trained and frozen vision-language model; an inference process is performed, using the attached prompts to help the model recall and utilize previously learned image and text knowledge; the output of the model is adjusted according to the multivariate Gaussian distribution and residual attention weight adjustment mechanism to optimize the learning of new image or text tasks and retain old knowledge.

2. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 1, wherein: The image encoder and text encoder of the visual-language model are jointly trained to generate tightly aligned feature representations between images and text samples; the image encoder is responsible for encoding the input image into a feature vector, and the text encoder embeds the potential class name into a template and encodes it to form a text embedding; during the inference phase, the visual-language model converts the input image into a feature vector through the image encoder, and the text encoder converts the class name into a text embedding; the prediction of the visual-language model is determined based on the maximum similarity score between the image feature vector and all text embeddings, where the similarity score is calculated by cosine similarity.

3. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 1 or 2, characterized in that During inference, the most relevant prompts for the current task are selected and attached to the pre-trained visual-language model to recover the learned knowledge; the selected prompt sequence is concatenated with the feature vector of the layer in the model to generate a new input vector to improve feature embedding and enhance classification accuracy; in the continuous learning phase, the average feature representation of each task is determined by maximizing cosine similarity or clustering algorithm to provide a basis for prompt selection; the similarity between the new test sample and the learned task is determined through the key lookup mechanism to select the most relevant prompt; the selected prompt is attached to the frozen visual-language model to perform the inference process.

4. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 3, wherein: The operation of the key lookup mechanism is expressed as: ， is the key most relevant to the test sample, For the test sample, For calculation of samples Through the encoder Processed feature representation and key The similarity function between represents a collection of keys, It means finding the key that maximizes the similarity function.

5. The method for retaining image and text knowledge pre-trained by a visual-language model in continuous learning according to any one of claims 1 to 2, characterized in that The information-free knowledge injection mechanism includes: The input tokens are first transformed into query Q, key K, and value V matrices via linear projection: ， in and are pre-trained parameters; self-attention calculation is then performed to produce the output matrix in the following way: ， in Constrained Attention Results The sum of the elements in is 1; Calculate the self-attention output within the input token and the cross-attention output between the prompt and the input token respectively, train only one residual attention branch, and keep the original attention score unchanged; through the newly introduced key Sum , the output of the residual attention branch is: ， Residual output is obtained by comparing with the original output Derived in an orthogonal way; By adding The learned knowledge stored in is embedded in the output; during the continuous training phase, only the learnable keys are updated Sum .

6. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 5, characterized in that By initializing the value parameter of the residual attention branch to zero, we force the addition of the residual attention to become the identity function: ， Only at the beginning constrained to zero while maintaining Random initialization.

7. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 5, characterized in that The establishment of the multivariate Gaussian distribution model specifically includes: During the training phase, a : ， in are the image features extracted by the frozen encoder, is the image feature extracted by the frozen encoder Expected value , is the covariance matrix; The multivariate Gaussian model is used to determine which learned image or text task the sample is most likely to come from, specifically including: Using the estimated multivariate Gaussian distribution, calculate the test sample Extracted from each The likelihood of As a sample Scoring function on each learned task: ， in is the probability density function; The weight of the residual attention output is adjusted, specifically including: Adjust the weight of the residual attention output according to the distribution recognition results of the test samples; Among them, the maximum score is used To weight the residual attention output: ， in is the total number of tasks, Is a fraction Mapping function that scales to the range [0,1].

8. The method for retaining image and text knowledge pre-trained by a vision-language model in continuous learning according to claim 7, wherein: Mapping Function Using Sigmoid function .

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for retaining image and text knowledge pre-trained by a visual-language model in continuous learning is implemented as described in any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for retaining image and text knowledge pre-trained by a visual-language model in continuous learning is implemented as described in any one of claims 1 to 8.

Images