Incremental learning method and system based on visual language model

Abstract

The application relates to a visual language model-based incremental learning method and system, and relates to the technical field of computer vision. The method comprises the following steps: fine-tuning a visual language pre-training model according to training sample data and historical sample data of a current task to obtain a target visual language model; extracting discriminative image features of all sample data according to the target visual language model; determining class feature means and a shared covariance matrix of a plurality of known categories involved in the current all sample data respectively according to the discriminative image features; dynamically calibrating the experience frequency of the plurality of known categories according to an inverse frequency adjustment mechanism to obtain calibrated prior probability; determining a debiased classification parameter according to the calibrated prior probability, the class feature means and the shared covariance matrix to construct a target debiased classifier; and performing weighted summation on the output result of the target visual language model corresponding to a to-be-classified image and the output result of the debiased classifier to obtain a target classification result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a class incremental learning method and system based on visual language models. Background Technology

[0002] In the dynamic and ever-changing real world, information is generated as a continuous data stream, requiring intelligent systems to adapt to new environments while retaining previously learned knowledge. This ability is known as continuous learning, and its core challenge is the problem of catastrophic forgetting, which is the decline in a model's ability to recall previously learned knowledge when learning new information.

[0003] Research has shown that class-incremental learning methods based on visual language models (CLIPs) exhibit strong transfer capabilities and classification performance by adding lightweight adapter modules to a frozen visual language-image pre-training (CLIP) model. However, VLM-based CIL methods remain susceptible to catastrophic forgetting. Because CLIP models undergo continuous parameter adjustments when learning new tasks, the fixed-size memory must accommodate samples from an increasing number of classes as tasks are introduced. This leads to a gradual decrease in the number of samples from each class in older tasks. This imbalance between the number of samples from old and new classes causes the adjusted CLIP model to become increasingly biased towards the newly introduced classes. Currently, CIL methods typically employ long-tail learning techniques (such as FocalLoss and SMOTE) to address class imbalance. However, these methods were originally designed for static class imbalance and ignore the inherent dynamic changes in class distribution within the CIL setting. They cannot effectively handle dynamic class imbalance between new and old tasks, resulting in model bias and severe forgetting of old knowledge. Summary of the Invention

[0004] This invention provides a class incremental learning method and system based on a visual language model, to at least address the problem of inaccurate classification predictions caused by severe model bias and forgetting of old knowledge due to dynamic class imbalance between new and old tasks. The technical solution of this invention is as follows: According to a first aspect of the present invention, a class incremental learning method based on a visual language model is provided. The method includes: fine-tuning a pre-trained visual language model based on training sample data and historical sample data for a current task to obtain a target visual language model; extracting discriminative image features from the training sample data and historical sample data based on the target visual language model; determining the mean class features and shared covariance matrix of multiple known categories involved in the training sample data and historical sample data based on the discriminative image features; dynamically calibrating the empirical frequencies of multiple known categories according to an inverse frequency adjustment mechanism to obtain calibrated prior probabilities; determining debiased classification parameters for multiple known categories based on the calibrated prior probabilities, the mean class features, and the shared covariance matrix; constructing a target debiased classifier based on the debiased classification parameters; the target debiased classifier being used to correct the bias in the output of the target visual language model; inputting the image to be classified into the target visual language model and the target debiased classification model respectively, and weighted summing the output of the target visual language model with the output of the debiased classifier to obtain a target classification result.

[0005] One implementation involves fine-tuning a visual language pre-trained model based on training sample data and historical sample data to obtain a target visual language model. This includes: initializing the visual language pre-trained model and constructing a memory bank with a preset sample capacity; storing multiple historical sample data in the memory bank; freezing the backbone parameters of the image encoder and text encoder in the visual language pre-trained model and inserting a lightweight hybrid expert adapter module into the visual language pre-trained model; performing sample fusion processing on the training sample data and historical sample data to obtain fused sample data; determining the cross-entropy loss function based on the fused sample data; optimizing and adjusting the parameters of the lightweight hybrid expert adapter module based on the cross-entropy mixing function to obtain an optimized lightweight hybrid expert adapter module; and generating the target visual language model based on the frozen backbone parameters and the optimized lightweight hybrid expert adapter module.

[0006] In this implementation, by freezing the backbone parameters, the general visual semantic space of the visual language model is solidified into a set of unshakeable coordinate axes, ensuring that the underlying logic of old category feature extraction remains unchanged and avoiding feature drift. Simultaneously, by inserting a lightweight hybrid expert adapter module, the visual language model is given the plasticity to learn new categories, enabling new categories to open dedicated feature enhancement channels without interfering with the expert paths of old categories.

[0007] As one implementation method, discriminative image features are extracted from training sample data and historical sample data based on the target visual language model. Based on the discriminative image features, the mean class features and shared covariance matrix of multiple known categories involved in the training sample data and historical sample data are determined, including: performing forward inference on the training sample data and historical sample data based on the target visual language model to extract the discriminative image features corresponding to each sample data; classifying the discriminative image features according to multiple known categories to determine the mean class features of each known category; and determining the shared covariance matrix of multiple known categories based on the deviation vector between the discriminative image features of each sample data and the corresponding mean class features.

[0008] In this implementation, the mean vector of the old category is calculated entirely from the historical features extracted from the frozen backbone, unaffected by any new task gradient contamination, exhibiting extremely strong anti-forgetting stability. Simultaneously, a category-shared covariance matrix is ​​estimated by assuming all known categories have similar distribution shapes in the feature space and using the biases of all samples to estimate a common covariance. This is a powerful regularization strategy that ensures the covariance matrix maintains full rank and invertibility even with a finite number of samples.

[0009] As one implementation method, based on the empirical Bayesian ridge estimator, the deviation vector between each sample data and the corresponding class feature mean is shrunk and estimated to obtain a shared covariance matrix of multiple known classes.

[0010] In this implementation, the empirical Bayesian ridge estimator is used to ensure the numerical invertibility of the high-dimensional covariance matrix without any human intervention, suppress the estimation variance caused by small samples, and enable the subsequent Gaussian discriminant classifier to maintain a stable, smooth and highly generalizable decision boundary when facing new and old categories with vastly different sample sizes.

[0011] As one implementation, the multiple known categories include multiple new categories in the training sample data and multiple old categories in the historical sample data; the calibrated prior probabilities include the prior probabilities of the old categories and the prior probabilities of the new categories; the prior probabilities of the multiple known categories are dynamically calibrated according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities, including: determining the first empirical frequency of the old categories based on the ratio of the preset sample capacity of the memory to the total number of old categories in the historical sample data; determining the second empirical frequency of the new categories based on the ratio of the number of samples in the training sample data to the total number of new categories in the training sample data; and determining the prior probabilities of the old categories and the new categories respectively based on the reciprocals of the first empirical frequency and the second empirical frequency.

[0012] In this implementation, the mechanism dynamically calibrates the prior probabilities of the new and old classes by using the reciprocal of the empirical frequency, thereby offsetting the systematic classification bias introduced by the imbalance of data flow during incremental learning, explicitly handling the dynamic class imbalance problem, effectively alleviating the model's bias towards the new class, and improving the model's stability.

[0013] As one implementation, the bias-reducing classifier parameters include a weight vector and a bias vector. Based on the calibrated prior probabilities, the mean of the class features, and the shared covariance matrix, the bias-reducing classifier parameters for multiple known classes are determined. Based on these parameters, a target bias-reducing classifier is constructed, including: determining the weight vector for each known class based on the feature mean and the class shared covariance matrix; determining the bias vector for each known class based on the calibrated prior probabilities, the mean of the class features, and the shared covariance matrix; and constructing the target bias-reducing classifier based on the weight vector and bias vector for each known class.

[0014] In this implementation, the discriminant function of Gaussian discriminant analysis is explicitly decomposed into a weight vector and a bias term, and then analytically calculated using the calibrated prior probability, class mean, and shared covariance matrix. The constructed bias-free classifier is non-parametric, relying only on frequency statistics and not on gradient optimization, thus requiring no additional learnable parameters and having low computational overhead.

[0015] One implementation involves inputting the image to be classified into a target visual language model and a target debiasing classification model, respectively. The output of the target visual language model and the output of the debiasing classifier are then weighted and summed to obtain the target classification result. This includes: extracting target image features from the image to be classified based on the target visual language model; inputting the target image features into the target visual language model to obtain a first original prediction score; inputting the target image features into the target debiasing classifier to obtain a second original prediction score; and obtaining the target classification result based on the weighted sum of the first original prediction score, the second original prediction score, and the debiasing contribution hyperparameter.

[0016] In this implementation, the strong semantic generalization ability of the visual language model is organically combined with the statistical unbiasedness of the debiased Gaussian classifier in a weighted summation form during the inference phase. By dynamically adjusting the correction intensity through hyperparameters, a balance is achieved between the retention of old category memory and the accurate discrimination of new categories in incremental learning scenarios without significantly increasing the computational burden, while maintaining the model's robustness to open environments.

[0017] As one implementation, after determining the target classification result of the image to be classified based on the target visual language model and the biased classifier, the method further includes: based on multiple known types, selecting multiple high-confidence pseudo-labeled inference samples with the same known types from multiple inference sample data of the current task, and determining the inference sample features of multiple inference samples; the inference sample data represents the unlabeled data used for testing on the current incremental learning node; based on the inference samples, determining the historical sample data to be updated with the same category as the inference samples from the memory, and determining the historical sample features of the historical sample data to be updated; comparing the semantic similarity between the inference sample features and the historical sample features to obtain a cosine similarity matrix; based on the cosine similarity matrix, determining the similarity score of each historical sample data to be updated; based on the similarity score, retaining historical sample data in the memory with a similarity score exceeding a preset retention threshold, and deleting historical sample data in the memory with a similarity score below the preset retention threshold to update the memory.

[0018] In this implementation, reliable test samples, i.e., samples with high confidence pseudo-labels and old task samples, are used to adaptively select samples that are most consistent with the distribution of previous task tests, thereby promoting the long-term retention of knowledge of previous tasks.

[0019] As one implementation method, based on multiple known types, multiple high-confidence pseudo-label inference samples with the same known types are selected from multiple inference sample data of the current task, and the inference sample features of multiple inference samples are determined, including: performing forward inference on multiple inference sample data according to the target visual language model to obtain the inference sample features corresponding to multiple inference sample data; determining the predicted probability distribution value of each known category according to the multiple inference sample features; and selecting inference sample data with predicted probability distribution values ​​greater than a preset threshold from multiple inference sample data according to the predicted probability distribution values, and determining them as high-confidence pseudo-label inference samples.

[0020] According to a second aspect of the present invention, a class incremental learning system based on a visual language model is provided, the system comprising: The storage module is configured to build and store a memory with a preset sample capacity.

[0021] The feature extraction module is configured to fine-tune the visual language pre-trained model based on the training sample data and historical sample data of the current task to obtain the target visual language model; extract discriminative image features from the training sample data and historical sample data based on the target visual language model; and determine the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and historical sample data based on the discriminative image features.

[0022] The classifier building module is configured to dynamically calibrate the empirical frequencies of multiple known categories according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities; and to determine the debiased classifier parameters for multiple known categories based on the calibrated prior probabilities, the mean of category features, and the shared covariance matrix to construct the debiased classifier.

[0023] The fusion inference module is configured to determine the target classification result of the image to be classified based on the target visual language model and the biased classifier.

[0024] The memory bank update module is configured to, after the current task training is completed, select multiple inference samples with high confidence pseudo-labels that are the same as the known types from the multiple inference sample data of the current task, based on the multiple known types; and update the memory bank based on the inference samples with high confidence pseudo-labels.

[0025] The class incremental learning system based on the visual language model is configured to perform a class incremental learning method based on the visual language model, as in the first aspect and any of its possible implementations.

[0026] According to a third aspect of the present invention, a class incremental learning device based on a visual language model is provided, the device being configured to perform a class incremental learning method based on a visual language model as described in the first aspect and any possible implementation thereof.

[0027] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of a class incremental learning device based on a visual language model, the class incremental learning device based on a visual language model is able to perform a class incremental learning method based on a visual language model as described in the first aspect and any possible implementation thereof.

[0028] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product including computer instructions, which, when executed on a class incremental learning device based on a visual language model, cause the class incremental learning device based on a visual language model to execute the class incremental learning method based on a visual language model described in the first aspect and any possible implementation thereof.

[0029] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: In order to ensure that the model maintains the recognition ability of the old category while adapting to the new category during continuous learning, the visual language model is fine-tuned by simultaneously using the training sample data of the current task and the historical sample data. This can effectively alleviate the catastrophic forgetting of the old category in incremental learning. By using the discriminative image features extracted from the fine-tuned target visual language model, the mean of the category features of all known categories and the unified shared covariance matrix are determined, thereby more stably depicting the distribution of the new and old categories. This reduces the distribution shift caused by the dynamic reduction of old category samples due to the increase of new category samples brought about by the incremental task, thus protecting the discriminative ability of the old category. Furthermore, in order to overcome the imbalance of sample data between the new and old categories, an inverse frequency adjustment mechanism is adopted to dynamically calibrate the empirical frequency of each category. This can suppress the excessive influence of high-frequency categories and increase the weight of minority categories, thereby effectively improving the fairness of the recognition of each category. Based on this, a bias-reducing classifier is constructed by combining the calibrated prior probability, the mean of category features and the shared covariance matrix. This can effectively reduce the bias of the majority class in the classification decision and improve the recognition accuracy of the minority class. Thus, by using the finely tuned target visual language model and the bias-reducing classifier to infer the images to be classified, this invention can output accurate classification results that take into account both new and old categories and balance the minority and majority categories. In the case of class incremental learning, it can effectively balance the plasticity and stability of the model, ensuring the accuracy and fairness of the classification results.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0032] Figure 1 This is a schematic diagram illustrating an incremental learning system based on a visual language model according to an exemplary embodiment; Figure 2 This is a flowchart illustrating an incremental learning method based on a visual language model according to an exemplary embodiment; Figure 3 This is a flowchart illustrating the principle of biased memory calibration Gaussian discriminant analysis according to an exemplary embodiment; Figure 4 This is a flowchart illustrating a bank update strategy according to an exemplary embodiment; Figure 5This is a chart showing the performance comparison results of different methods on the CIFAR100 dataset according to an exemplary embodiment; Figure 6 This is a schematic diagram illustrating an incremental learning device based on a visual language model according to an exemplary embodiment. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0035] Before providing a detailed introduction to the class incremental learning method based on visual language models provided in the embodiments of this application, let's first briefly introduce the application scenarios and implementation environment involved in the embodiments of this application.

[0036] In the dynamic and ever-changing real world, information is generated as a continuous data stream, requiring intelligent systems to adapt to new environments while retaining previously learned knowledge. This ability is known as continuous learning, and its core challenge is the problem of catastrophic forgetting, which is the decline in a model's ability to recall previously learned knowledge when learning new information.

[0037] Class Incremental Learning (CIL) is a representative paradigm of continuous learning. Its goal is to incrementally learn new classes in a task-agnostic manner, maintaining high classification performance across all known classes even without providing task labels for samples during testing. Existing class incremental learning methods can be broadly categorized into rehearsal-based methods and sample-free methods. Rehearsal-based methods retain a limited number of samples from older tasks in memory and replay these samples during subsequent training to enhance the model's retention of previous knowledge.

[0038] In recent years, visual-language models (VLMs), such as contrastive language-image pre-training (CLIP), have demonstrated strong transfer capabilities by jointly utilizing textual and visual information. VLM-based incremental learning methods have achieved state-of-the-art performance by adding lightweight adapter modules to frozen CLIP models. However, VLM-based CIL methods remain susceptible to catastrophic forgetting, primarily because CLIPs undergo continuous parameter tuning when learning new tasks.

[0039] In this context, replay mechanisms remain a strategy to mitigate catastrophic forgetting. However, through experimental observation, the inventors found that the performance improvement brought about by introducing replay mechanisms into a VLM-based CIL framework is limited. In-depth analysis revealed a key problem: the replay mechanism fails to consider the dynamic class imbalance between new and old tasks. As tasks are introduced sequentially, the fixed-size memory must accommodate samples from an increasing number of classes, leading to a gradual decrease in the number of samples from each class in older tasks. This imbalance between the number of samples from old and new classes causes the adjusted CLIP model to become increasingly biased towards the newly introduced classes.

[0040] Existing CIL methods typically employ long-tail learning techniques (such as Focal Loss and SMOTE) to address class imbalance, but these methods were originally designed for static class imbalance and ignore the inherent dynamic nature of class distribution in CIL settings.

[0041] To address the aforementioned issues, this application proposes a class-incremental learning method based on a visual language model. To ensure the model maintains its ability to recognize old classes while adapting to new classes during continuous learning, the visual language model is simultaneously fine-tuned using both training and historical sample data from the current task. This effectively mitigates the catastrophic forgetting of old classes during class-incremental learning. Furthermore, by utilizing the discriminative image features extracted from the fine-tuned target visual language model, the mean of class features for all known classes and a unified shared covariance matrix are determined. This provides a more stable characterization of the distribution of new and old classes, reducing the impact of incremental tasks. The distribution shift caused by the increase in new category samples and the dynamic decrease in old category samples protects the discriminative ability of the old categories. Furthermore, to overcome the imbalance between new and old category sample data, an inverse frequency adjustment mechanism is used to dynamically calibrate the empirical frequencies of each category. This suppresses the excessive influence of high-frequency categories while increasing the weight of minority categories, effectively improving the fairness of category recognition. Based on this, a bias-reducing classifier is constructed by combining the calibrated prior probabilities, category feature means, and shared covariance matrix. This effectively reduces the bias towards the majority class in classification decisions and improves the recognition accuracy of minority categories. Thus, this application, through the above-mentioned fine-tuned target visual language model and bias-reducing classifier, can output accurate classification results that balance new and old categories and minority and majority categories when inferring from images to be classified. In the class incremental learning scenario, it effectively balances the plasticity and stability of the model, ensuring the accuracy and fairness of the classification results.

[0042] Secondly, the implementation architecture involved in this application will be briefly introduced below.

[0043] Figure 1 This is a schematic diagram of a visual language model-based incremental learning system provided in this application. Figure 1 As shown, the class incremental learning system based on the visual language model includes a storage module 11, a feature extraction module 12, a classifier construction module 13, a fusion inference module 14, and a memory bank update module 15.

[0044] The aforementioned modules 11, 12, 13, 14, and 15 are connected via communication.

[0045] Storage module 11 is configured to build and store a memory with a preset sample capacity.

[0046] The feature extraction module 12 is configured to fine-tune the visual language pre-trained model based on the training sample data and historical sample data of the current task to obtain the target visual language model; extract discriminative image features from the training sample data and historical sample data based on the target visual language model; and determine the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and historical sample data based on the discriminative image features.

[0047] The classifier building module 13 is configured to dynamically calibrate the empirical frequencies of multiple known categories according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities; and to determine the debiased classifier parameters of multiple known categories based on the calibrated prior probabilities, the mean of category features and the shared covariance matrix to construct a debiased classifier.

[0048] The fusion reasoning module 14 is configured to determine the target classification result of the image to be classified based on the target visual language model and the bias classifier.

[0049] The memory bank update module 15 is configured to, after the current task training is completed, select multiple inference samples with high confidence pseudo-labels that are the same as the known types from the multiple inference sample data of the current task, based on the multiple known types; and update the memory bank based on the inference samples with high confidence pseudo-labels.

[0050] This incremental learning system based on a visual language model is configured to fine-tune a pre-trained visual language model using training and historical sample data for the current task to obtain a target visual language model; extract discriminative image features from the training and historical sample data based on the target visual language model; determine the mean class features and shared covariance matrix of multiple known categories involved in the training and historical sample data based on the discriminative image features; dynamically calibrate the empirical frequencies of multiple known categories using an inverse frequency adjustment mechanism to obtain calibrated prior probabilities; determine the debiasing classification parameters for multiple known categories based on the calibrated prior probabilities, mean class features, and shared covariance matrix; construct a target debiasing classifier based on the debiasing classification parameters; the target debiasing classifier is used to correct the bias in the output of the target visual language model; input the image to be classified into the target visual language model and the target debiasing classification model respectively, and weight the output of the target visual language model and the output of the debiasing classifier to obtain the target classification result.

[0051] As one implementation method, the incremental learning system based on the visual language model is specifically configured to fine-tune the visual language pre-trained model according to training sample data and historical sample data to obtain the target visual language model. This includes: initializing the visual language pre-trained model and constructing a memory bank with a preset sample capacity; storing multiple historical sample data in the memory bank; freezing the backbone parameters of the image encoder and text encoder in the visual language pre-trained model and inserting a lightweight hybrid expert adapter module into the visual language pre-trained model; performing sample fusion processing on the training sample data and historical sample data to obtain fused sample data; determining the cross-entropy loss function based on the fused sample data; optimizing and adjusting the parameters of the lightweight hybrid expert adapter module according to the cross-entropy mixing function to obtain an optimized lightweight hybrid expert adapter module; and generating the target visual language model based on the frozen backbone parameters and the optimized lightweight hybrid expert adapter module.

[0052] As one implementation method, the class incremental learning system based on the visual language model is specifically configured to: extract discriminative image features from training sample data and historical sample data according to the target visual language model; and determine the mean class features and shared covariance matrix of multiple known categories involved in the training sample data and historical sample data according to the discriminative image features, including: performing forward inference on the training sample data and historical sample data according to the target visual language model to extract the discriminative image features corresponding to each sample data; classifying the discriminative image features according to multiple known categories to determine the mean class features of each known category; and determining the shared covariance matrix of multiple known categories according to the deviation vector between the discriminative image features of each sample data and the corresponding mean class features.

[0053] As one implementation, the class incremental learning system based on the visual language model is specifically configured to use an empirical Bayesian ridge estimator to perform a shrinkage estimation of the deviation vector between each sample data and the corresponding class feature mean, thereby obtaining a shared covariance matrix for multiple known classes.

[0054] As one implementation, the class incremental learning system based on the visual language model is specifically configured with multiple known categories, including multiple new categories in the training sample data and multiple old categories in the historical sample data; the calibrated prior probabilities include the prior probabilities of the old categories and the prior probabilities of the new categories; the prior probabilities of the multiple known categories are dynamically calibrated according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities, including: determining the first empirical frequency of the old categories based on the ratio of the preset sample capacity of the memory to the total number of old categories in the historical sample data; determining the second empirical frequency of the new categories based on the ratio of the number of samples in the training sample data to the total number of new categories in the training sample data; and determining the prior probabilities of the old categories and the new categories respectively based on the reciprocals of the first empirical frequency and the reciprocals of the second empirical frequency.

[0055] As one implementation, the class incremental learning system based on the visual language model is specifically configured as follows: the debiased classifier parameters include weight vectors and bias vectors; the debiased classifier parameters for multiple known classes are determined based on the calibrated prior probabilities, class feature mean, and shared covariance matrix; based on the debiased classification parameters, a target debiased classifier is constructed, including: determining the weight vectors for each known class based on the feature mean and class shared covariance matrix; determining the bias vectors for each known class based on the calibrated prior probabilities, class feature mean, and shared covariance matrix; and constructing the target debiased classifier based on the weight vectors and bias vectors of each known class.

[0056] As one implementation method, the incremental learning system based on the visual language model is specifically configured to input the image to be classified into the target visual language model and the target debiasing classification model respectively, and to obtain the target classification result by weighted summation of the output of the target visual language model and the output of the debiasing classifier. This includes: extracting the target image features of the image to be classified based on the target visual language model; inputting the target image features into the target visual language model to obtain a first original prediction score; inputting the target image features into the target debiasing classifier to obtain a second original prediction score; and obtaining the target classification result by weighted summation of the first original prediction score, the second original prediction score, and the debiasing contribution hyperparameter.

[0057] As one implementation method, the incremental learning system based on visual language models is specifically configured as follows: after determining the target classification result of the image to be classified based on the target visual language model and the biased classifier, the method further includes: based on multiple known types, selecting multiple high-confidence pseudo-labeled inference samples with the same known types from multiple inference sample data of the current task, and determining the inference sample features of multiple inference samples; the inference sample data represents the unlabeled data used for testing at the current incremental learning node; based on the inference samples, determining the historical sample data to be updated with the same category as the inference samples from the memory, and determining the historical sample features of the historical sample data to be updated; comparing the semantic similarity between the inference sample features and the historical sample features to obtain a cosine similarity matrix; based on the cosine similarity matrix, determining the similarity score of each historical sample data to be updated; based on the similarity score, retaining historical sample data in the memory with a similarity score exceeding a preset retention threshold, and deleting historical sample data in the memory with a similarity score below the preset retention threshold to update the memory.

[0058] As one implementation method, the class incremental learning system based on the visual language model is specifically configured to, based on multiple known types, select multiple inference samples with high confidence pseudo-labels that are the same as the known types from multiple inference sample data of the current task, and determine the inference sample features of multiple inference samples, including: performing forward inference on multiple inference sample data according to the target visual language model to obtain the inference sample features corresponding to multiple inference sample data; determining the predicted probability distribution value of each known category according to the multiple inference sample features; and selecting inference sample data with predicted probability distribution values ​​greater than a preset threshold from multiple inference sample data according to the predicted probability distribution values, and determining them as inference samples with high confidence pseudo-labels.

[0059] For ease of understanding, the class incremental learning method based on visual language models provided in this application will be described in detail below with reference to the accompanying drawings.

[0060] Figure 2 This is a flowchart illustrating an incremental learning method based on a visual language model according to an exemplary embodiment, such as... Figure 2 As shown, the class incremental learning method based on the visual language model includes the following steps.

[0061] S21. Based on the training sample data and historical sample data of the current task, fine-tune the visual language pre-trained model to obtain the target visual language model.

[0062] The training sample data for the current task represents a new category that the model has not learned before in the current class incremental learning step, and it is an image dataset with manually annotated real labels.

[0063] To combat catastrophic forgetting in the incremental learning process of visual language models, the general visual semantic space of the visual language model is solidified into an unshakeable set of coordinate axes by freezing the backbone parameters. This ensures that the underlying logic of old category feature extraction remains unchanged and avoids feature drift. Simultaneously, by inserting a lightweight hybrid expert adapter module, the visual language model is given the plasticity to learn new categories, allowing new categories to open up dedicated feature enhancement channels without interfering with the expert paths of old categories.

[0064] Specifically, a visual language pre-trained model is initialized, and a memory bank with a preset sample size is constructed. The memory bank stores multiple historical sample data. The backbone parameters of the image encoder and text encoder in the visual language pre-trained model are frozen, and a lightweight hybrid expert adapter module is inserted into the visual language pre-trained model. Sample fusion processing is performed on the training sample data and historical sample data to obtain fused sample data; and the cross-entropy loss function is determined based on the fused sample data. The parameters of the lightweight hybrid expert adapter module are optimized and adjusted according to the cross-entropy mixing function to obtain the optimized lightweight hybrid expert adapter module. Based on the frozen backbone parameters and the optimized lightweight hybrid expert adapter module, the target visual language model is generated.

[0065] In one implementation, before acquiring training sample data for the current task, the visual language model and memory are initialized. A pre-trained CLIP model is loaded, and its image encoder and text encoder backbone parameters are frozen. A memory with a fixed total capacity of N is constructed to store historical task samples.

[0066] For example, the total capacity is a fixed size N set to 2000, and the initial tasks contain two categories.

[0067] Building upon this, the training sample data for the current task, containing multiple new categories, is received. This training sample data is then mixed with historical sample data from the memory bank. A hybrid expert adapter module (MoE-Adapter) is inserted into the CLIP model, and the adapter parameters are fine-tuned using the cross-entropy loss function.

[0068] In this implementation, if the backbone parameters of the visual language model are directly fine-tuned during incremental learning, the gradient of new samples for the current task will globally update the weights of the image encoder and text encoder, causing the feature mapping relationship originally established by the model on the old categories to drift. Therefore, the backbone parameters are frozen to ensure that the underlying logic of old category feature extraction remains unchanged. Furthermore, a lightweight hybrid expert adapter module is inserted to give the model the plasticity to learn new categories. The cross-entropy loss function is calculated based on the mixed data of training and historical samples, and the lightweight hybrid expert adapter module is optimized to simultaneously consider the discriminative power of both new and old categories, effectively preventing implicit forgetting within the adapter. Thus, the target visual language model generated by using frozen backbone parameters and the optimized lightweight hybrid expert adapter module can effectively resist catastrophic forgetting.

[0069] S22, Based on the target visual language model, extract discriminative image features from the training sample data and historical sample data; based on the discriminative image features, determine the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and historical sample data, respectively.

[0070] In one implementation, forward reasoning is performed on training sample data and historical sample data based on the target visual language model to extract discriminative image features corresponding to each sample data. The discriminative image features are then classified into multiple known categories to determine the mean value of the category features for each known category. Furthermore, a shared covariance matrix for multiple known categories is determined based on the deviation vector between the discriminative image features of each sample data and the corresponding mean value of the category features.

[0071] Among them, the category feature mean is the center position of each category in the feature space, representing the typical discriminative features of that category.

[0072] The shared covariance matrix is ​​the common feature distribution shape for all categories, representing the correlation of feature dimensions.

[0073] In this implementation, the mean vector of the old category is calculated entirely from the historical features extracted from the frozen backbone, unaffected by any new task gradient contamination, exhibiting extremely strong anti-forgetting stability. Simultaneously, a category-shared covariance matrix is ​​estimated by assuming all known categories have similar distribution shapes in the feature space and using the biases of all samples to estimate a common covariance. This is a powerful regularization strategy, ensuring the covariance matrix maintains full rank and invertibility even with a finite number of samples. The category feature mean and the shared covariance matrix are used to construct the category statistics for the classifier, transforming the classifier construction problem in incremental learning into a robust statistical estimation problem in the feature space. This achieves the construction of unbiased discriminative boundaries between the old and new categories with extremely low computational cost, serving as a crucial bridge connecting feature learning and fair reasoning in the incremental learning framework of visual language models.

[0074] In one implementation, to mitigate high-dimensional bias, an empirical Bayesian ridge estimator is used to estimate the shared covariance matrix.

[0075] Specifically, based on the empirical Bayesian ridge estimator, the deviation vector between each sample data and the corresponding class feature mean is shrunk and estimated to obtain a shared covariance matrix for multiple known classes.

[0076] In this implementation, the empirical Bayesian ridge estimator is used to ensure the numerical invertibility of the high-dimensional covariance matrix without any human intervention, suppress the estimation variance caused by small samples, and enable the subsequent Gaussian discriminant classifier to maintain a stable, smooth and highly generalizable decision boundary when facing new and old categories with vastly different sample sizes.

[0077] S23, dynamically calibrate the empirical frequencies of multiple known categories according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities.

[0078] Among them, the multiple known categories include multiple new categories in the training sample data and multiple old categories in the historical sample data.

[0079] The calibrated prior probabilities include the prior probabilities of the old class and the prior probabilities of the new class.

[0080] Specifically, the first empirical frequency of the old class is determined based on the ratio of the preset sample size of the memory to the total number of old classes in the historical sample data. The second empirical frequency of the new class is determined based on the ratio of the number of training samples to the total number of new classes in the training sample data. The prior probabilities of the old class and the new class are determined based on the reciprocals of the first and second empirical frequencies, respectively.

[0081] In one implementation, the prior probability is calibrated using an inverse frequency adjustment mechanism, as specifically represented by the following formula (1).

[0082] (1).

[0083] in, The prior probability after calibration for the old category; The prior probability after calibration for the new category; is the empirical frequency of the old categories in the memory; N is the preset sample size of the memory; This represents the total number of items in the old category. For the old category set; The empirical frequency of the new category in the training data; The sample size for the training data of the new class; Total number of new categories; For the new category set.

[0084] In this implementation, the mechanism dynamically calibrates the prior probabilities of the new and old classes by using the reciprocal of the empirical frequency, thereby offsetting the systematic classification bias introduced by the imbalance of data flow during incremental learning, explicitly handling the dynamic class imbalance problem, effectively alleviating the model's bias towards the new class, and improving the model's stability.

[0085] S24. Based on the calibrated prior probabilities, class feature mean, and shared covariance matrix, determine the debiasing classification parameters for multiple known classes; construct a target debiasing classifier based on the debiasing classification parameters; the target debiasing classifier is used to correct the bias in the output of the target visual language model.

[0086] The debiased classification parameters include the weight vector and the bias vector.

[0087] In one implementation, the weight vector and bias vector for each known category are determined; and a target debiasing classifier is constructed based on the weight vector and bias vector.

[0088] The weight vectors for each known class are determined based on the feature mean and the class-shared covariance matrix. The specific representation is shown in formula (2).

[0089] (2).

[0090] in, For the first The weight vector of the class; To share the covariance matrix; For each category The characteristic mean.

[0091] Based on the calibrated prior probabilities, the mean of class features, and the shared covariance matrix, the bias vector for each known class is determined. Specifically, this is represented by formula (3).

[0092] (3).

[0093] in, For the first The class's bias vector; To share the covariance matrix; For each category The characteristic mean; This is the calibrated prior probability.

[0094] It is understandable that each category has a new category and an old category. When calculating the bias vector of the new category, the prior probability of the new category obtained in the above steps is used for calculation. When calculating the bias vector of the old category, the prior probability of the old category obtained in the above steps is used for calculation.

[0095] Based on steps S22 to S24 above, the Debiased Memory-Calibrated Gaussian Discriminant Analysis (DMGDA) method is introduced to dynamically calibrate the prior probabilities of the old and new classes through an inverse frequency adjustment mechanism. This mechanism uses the reciprocal of the frequency of old class samples in memory and the frequency of new class samples in the current training data to calculate the calibrated prior probability, thereby improving the classification probability of underrepresented old classes. Based on the calibrated prior probability, the mean of class features, and the shared covariance matrix, a non-parametric debiased classifier is obtained. The specific process is as follows: Figure 3 As shown.

[0096] S301, Data Input: Memory bank, current task data.

[0097] S302, the finely tuned image encoder extracts image features from the current task and memory samples.

[0098] S303, Statistical estimation: Class mean estimation, using an empirical Bayesian ridge estimator to estimate the shared covariance matrix.

[0099] S304, Inverse Frequency Adjustment: Dynamically calibrate the prior probabilities of each category based on the empirical frequencies of the old category in the memory bank and the empirical frequencies of the new category in the current training data.

[0100] S305, Debiased Memory Calibration GDA Classifier: Based on the calibrated prior probabilities, class mean vectors, and shared covariance matrix, a non-parametric classifier constructed using Bayes' theorem is used to calculate the classifier weight matrix and bias vector.

[0101] S306, Output: Partially modified classifier parameters. The weight matrix and bias vector are sent to the fusion inference module and fused with the baseline classifier output.

[0102] The adjusted target visual language model is also called the baseline classifier, or MoE-Adapters model, which represents the insertion of a hybrid expert adapter module (MoE-Adapter) into the CLIP model.

[0103] In this implementation, the discriminant function of Gaussian discriminant analysis is explicitly decomposed into a weight vector and a bias term, and then analytically calculated using the calibrated prior probability, class mean, and shared covariance matrix. The constructed bias-free classifier is non-parametric, relying only on frequency statistics and not on gradient optimization, thus requiring no additional learnable parameters and having low computational overhead.

[0104] S25. Input the image to be classified into the target visual language model and the target debiased classification model respectively, and sum the weighted results of the target visual language model and the debiased classifier to obtain the target classification result.

[0105] Specifically, based on the target visual language model, target image features of the image to be classified are extracted; the target image features are input into the target visual language model to obtain the first original prediction score; the target image features are input into the target debiasing classifier to obtain the second original prediction score; and the target classification result is obtained by weighting the first original prediction score, the second original prediction score and the debiasing contribution hyperparameter.

[0106] In one implementation, during the inference phase, the image to be classified is input into the image encoder of the fine-tuned target visual language model. The target image features are extracted through forward propagation, and the output logits of the baseline model MoE-Adapters and the output logits of the target debiased classifier are calculated to obtain the final predicted logits. Specifically, the following formula (4) represents this.

[0107] (4).

[0108] in, For the final prediction of logits; The output logits of the baseline model; The output logits of the bias-reduced classifier; This is a hyperparameter used to control the contribution of the debiased logits, and is used to balance the plasticity and stability of the model. Set to 1.9.

[0109] Among them, the target visual language model, namely the baseline MoE-Adapter model, is based on strong semantic priors pre-trained by image-text comparison to understand the visual concepts of categories. However, it may still be affected by the new category bias of new task data and has a slight suppression of the old category logits. The target debias classifier that introduces Gaussian discriminant analysis can explicitly correct the prior imbalance based on the accurate closed set discrimination of feature statistics.

[0110] Therefore, during the inference phase, the strong semantic generalization ability of the visual language model is organically combined with the statistical unbiasedness of the debiased Gaussian classifier using a weighted summation method. By dynamically adjusting the correction intensity through hyperparameters, a balance is achieved between preserving old category memory and accurately distinguishing new categories in incremental learning scenarios without significantly increasing the computational burden, while maintaining the model's robustness to open environments.

[0111] Optionally, after obtaining the target classification result and completing model training, the memory is adaptively updated. This involves the following four steps.

[0112] First, based on multiple known types, multiple inference samples with high confidence pseudo-labels that are the same as the known types are selected from multiple inference sample data of the current task, and the inference sample features of multiple inference samples are determined.

[0113] Among them, the inference sample data represents the unlabeled data used for testing on the current incremental learning node.

[0114] Specifically, based on the target visual language model, forward reasoning is performed on multiple reasoning sample data to obtain the reasoning sample features corresponding to the multiple reasoning sample data; based on the multiple reasoning sample features, the predicted probability distribution value of each known category is determined; based on the predicted probability distribution value, reasoning sample data with a predicted probability distribution value greater than a preset threshold is selected from the multiple reasoning sample data and determined as reasoning samples with high confidence pseudo-labels.

[0115] In one implementation, a predicted probability distribution for each category is calculated based on the features. The category corresponding to the maximum probability is taken as the pseudo-label, and the maximum probability is used as the confidence level of the sample. When the confidence level is greater than a preset threshold, the sample is marked as a high-confidence pseudo-label inference sample. The preset threshold is set to 0.8.

[0116] Secondly, based on the inference samples, historical sample data of the same category as the inference samples are identified from the memory bank, and the historical sample features of the historical sample data to be updated are determined. The semantic similarity of the inference sample features and the historical sample features is compared to obtain a cosine similarity matrix.

[0117] Third, based on the cosine similarity matrix, determine the similarity score of each historical sample data to be updated.

[0118] Fourth, based on the similarity score, historical sample data in the memory with similarity scores exceeding the preset retention threshold are retained, while historical sample data in the memory with similarity scores below the preset retention threshold are deleted to update the memory.

[0119] Optionally, a preset number of samples with the highest similarity scores in each category are retained, and the remaining samples are deleted.

[0120] Based on the above steps, an Adaptive Memory Iteration Strategy (AMI) was designed and integrated into the rehearsal mechanism. This strategy adaptively selects samples that best match the distribution of previous task tests by leveraging the semantic similarity between reliable test samples (with high-confidence pseudo-labels) and old task samples, thereby promoting long-term retention of knowledge from previous tasks. The specific process is as follows: Figure 4 As shown.

[0121] S401, Input: Features of high-confidence pseudo-label test samples.

[0122] S402, extract features of similar samples matching pseudo-labels from the memory.

[0123] S403, calculate the cosine similarity matrix.

[0124] S404 calculates the similarity score for each sample.

[0125] S405, similarity score sorting (descending order).

[0126] S406, retains highly similar samples.

[0127] S407, Delete samples with low similarity.

[0128] S408, update the memory bank.

[0129] Optionally, a structured memory design is adopted, which organizes memory into a set of task-specific memory blocks. Each memory block is further divided into block slots corresponding to categories, which facilitates category-level organization and retrieval, and supports more effective memory calibration and controlled rehearsal playback mechanisms.

[0130] For example, to verify the effectiveness of the method of the present invention, experiments were conducted on the standard CIL benchmark (CIFAR100). Fifty incremental steps were set, and the memory size was 2000. Performance comparison results of different methods on this CIFAR100 dataset are as follows: Figure 5 As shown.

[0131] The accuracy rate is the average of three independent repeated trials.

[0132] The baseline method is the MoE-Adapters model, which does not use the DMGDA debiased memory calibration Gaussian discriminant analysis method and the AMI adaptive memory iteration strategy.

[0133] The average accuracy of MoE-Adapters is 83.60%, with a final step accuracy of 75.24%. The average accuracy of the old class is 85.64%, with a final step accuracy of 71.67%.

[0134] The average accuracy of MoE-Adapters+iCaRL is 85.11%, with a final step accuracy of 76.68%.

[0135] The average accuracy of MoE-Adapters+iCaRL+DMGDA was 88.15%, with the final step achieving an accuracy of 82.19%.

[0136] The average accuracy of MoE-Adapters+iCaRL+AMI was 86.58%, with a final step accuracy of 79.96%.

[0137] The average accuracy of the method of this invention is 90.10%, and the accuracy of the last step is 83.69%. The average accuracy of the old method is 91.76%, and the accuracy of the last step of the old method is 82.78%.

[0138] Among them, MoE-Adapters are hybrid expert adapters.

[0139] iCaRL is an incremental classifier and representation learning method.

[0140] DMGDA is a debiased memory-calibrated Gaussian discriminant analysis classifier.

[0141] AMI is an adaptive memory iteration strategy.

[0142] The results show that the method of this invention achieves an average accuracy of 90.10% in a 50-step setting, which is 6.50% higher than the baseline model. The accuracy of the last step for the old class is improved by 11.11%, demonstrating that the method of this invention effectively enhances model stability and alleviates the forgetting problem caused by dynamic class imbalance. It is evident that this application achieves state-of-the-art performance on both standard CIL and long-tailed CIL (LT-CIL) benchmarks, exhibiting excellent generalization ability.

[0143] In specific applications, the system can be used in industrial vision inspection scenarios, such as product quality classification, defect detection, and component identification. When new defect types appear in the production environment, the model training process from steps S21 to S25 is repeated to enable the model to continuously learn new defect types while maintaining a high detection rate for old defect types. This supports the model in continuously learning new product categories in the production environment without forgetting already learned categories.

[0144] Figure 6 This is a schematic diagram of a visual language model-based incremental learning device provided in this application. Figure 6 The monitoring device 50 includes: a first processor 501, a communication bus 502, a memory 503, a communication interface 504, an output device 505, an input device 506, and a second processor 507.

[0145] The class incremental learning device 50 based on the visual language model may include at least one first processor 501 and a memory 503 for storing processor-executable instructions. The first processor 501 is configured to execute the instructions in the memory 503 to implement the class incremental learning method based on the visual language model in the following embodiments.

[0146] In addition, the class incremental learning device 50 based on the visual language model may also include a communication bus 502, at least one communication interface 504, an input device 506, and an output device 505.

[0147] The first processor 501 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present application.

[0148] The communication bus 502 may include a path for transmitting information between the aforementioned components.

[0149] Communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0150] Input device 506 is used to receive input signals and output device 505 is used to output signals.

[0151] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0152] The memory 503 stores instructions for executing the scheme of this application, and the execution is controlled by the first processor 501. The first processor 501 executes the instructions stored in the memory 503 to realize the functions of the method of this application.

[0153] In a specific implementation, as one example, the first processor 501 may include one or more CPUs, for example... Figure 6 CPU0 and CPU1 in the CPU.

[0154] In a specific implementation, as one example, the class incremental learning device 50 based on the visual language model may include multiple processors, such as... Figure 6 The first processor 501 and the second processor 507 are described. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0155] This incremental learning device based on visual language models, such as Figure 6 The diagram includes a first processor 501 and a memory 503 for storing executable instructions of the first processor 501. The first processor 501 is configured to execute the executable instructions to implement the class incremental learning method based on a visual language model as described in any of the possible embodiments above. Furthermore, it achieves the same technical effect, and to avoid repetition, will not be elaborated further here.

[0156] This application also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of a class incremental learning device based on a visual language model, the class incremental learning device based on a visual language model is able to perform the class incremental learning method based on a visual language model as described in any of the possible implementations above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0157] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as described in any of the possible implementations above, representing a class incremental learning method based on a visual language model. This achieves the same technical effect, and to avoid repetition, will not be repeated here.

[0158] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0159] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A class-based incremental learning method based on a visual language model, characterized in that, The method includes: Based on the training sample data and historical sample data of the current task, the visual language pre-trained model is fine-tuned to obtain the target visual language model. Based on the target visual language model, discriminative image features are extracted from the training sample data and the historical sample data; based on the discriminative image features, the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and the historical sample data are determined respectively. The empirical frequencies of the multiple known categories are dynamically calibrated according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities. Based on the calibrated prior probabilities, the mean values ​​of the class features, and the shared covariance matrix, the bias removal classification parameters for the multiple known classes are determined; based on the bias removal classification parameters, a target bias removal classifier is constructed; the target bias removal classifier is used to correct the bias in the output of the target visual language model. The image to be classified is input into the target visual language model and the target debiased classification model, respectively. The output of the target visual language model and the output of the debiased classifier are weighted and summed to obtain the target classification result.

2. The class incremental learning method based on a visual language model according to claim 1, characterized in that, The step of fine-tuning the visual language pre-trained model based on training sample data and historical sample data to obtain the target visual language model includes: The visual language pre-trained model is initialized, and a memory bank with a preset sample size is constructed; the memory bank stores multiple historical sample data. Freeze the backbone parameters of the image encoder and text encoder in the visual language pre-trained model, and insert a lightweight hybrid expert adapter module into the visual language pre-trained model; The training sample data and the historical sample data are subjected to sample fusion processing to obtain fused sample data; and the cross-entropy loss function is determined based on the fused sample data. Based on the cross-entropy mixing function, the parameters of the lightweight hybrid expert adapter module are optimized and adjusted to obtain the optimized lightweight hybrid expert adapter module. The target visual language model is generated based on the frozen backbone parameters and the optimized lightweight hybrid expert adapter module.

3. The class incremental learning method based on visual language models according to claim 2, characterized in that, The step of extracting discriminative image features from the training sample data and the historical sample data based on the target visual language model, and determining the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and the historical sample data based on the discriminative image features, includes: Based on the target visual language model, forward reasoning is performed on the training sample data and the historical sample data to extract the discriminative image features corresponding to each sample data. The discriminative image features are classified according to the multiple known categories to determine the mean value of the category features for each of the known categories; Furthermore, the shared covariance matrix of the plurality of known categories is determined based on the deviation vector between the discriminative image features of each sample data and the mean of the corresponding category features.

4. The class incremental learning method based on a visual language model according to claim 3, characterized in that, The method further includes: Based on the empirical Bayesian ridge estimator, the deviation vector between each sample data and the corresponding class feature mean is shrunk and estimated to obtain the shared covariance matrix of the multiple known classes.

5. The class incremental learning method based on a visual language model according to claim 1, characterized in that, The multiple known categories include multiple new categories in the training sample data and multiple old categories in the historical sample data; the calibrated prior probabilities include prior probabilities of old categories and prior probabilities of new categories; The step of dynamically calibrating the prior probabilities of the multiple known categories according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities includes: The first empirical frequency of the old category is determined based on the ratio of the preset sample size of the memory bank to the total number of old categories in the historical sample data. The second empirical frequency of the new category is determined based on the ratio of the number of samples in the training sample data to the total number of new categories in the training sample data; The prior probability of the old category and the prior probability of the new category are determined based on the reciprocal of the first empirical frequency and the reciprocal of the second empirical frequency, respectively.

6. The class incremental learning method based on a visual language model according to claim 5, characterized in that, The parameters of the debiased classifier include a weight vector and a bias vector; The parameters of the bias-reducing classifier for the multiple known categories are determined based on the calibrated prior probabilities, the mean values ​​of the category features, and the shared covariance matrix. Based on the aforementioned bias-reduction classification parameters, a target bias-reduction classifier is constructed, including: The weight vector for each of the known categories is determined based on the feature mean and the category shared covariance matrix. The bias vector for each of the known classes is determined based on the calibrated prior probabilities, the mean of the class features, and the shared covariance matrix. The target debias classifier is constructed based on the weight vector and the bias vector of each known category.

7. The class incremental learning method based on a visual language model according to claim 6, characterized in that, The process involves inputting the image to be classified into the target visual language model and the target bias-reduction classification model, respectively, and then weighted and summing the output of the target visual language model with the output of the bias-reduction classifier to obtain the target classification result, including: Based on the target visual language model, the target image features of the image to be classified are extracted; The target image features are input into the target visual language model to obtain the first original prediction score; The target image features are input into the target debiasing classifier to obtain the second original prediction score; The target classification result is obtained by weighting the first original predicted score, the second original predicted score, and the debiasing contribution hyperparameter.

8. The class incremental learning method based on a visual language model according to any one of claims 1 to 7, characterized in that, After determining the target classification result of the image to be classified based on the target visual language model and the bias-reducing classifier, the method further includes: Based on the known types, multiple high-confidence pseudo-labeled inference samples with the same known types are selected from the multiple inference sample data of the current task, and the inference sample features of the multiple inference samples are determined; the inference sample data represents the unlabeled data used for testing on the current incremental learning node; Based on the inference sample, identify historical sample data of the same category as the inference sample from the memory bank, and determine the historical sample features of the historical sample data to be updated; The semantic similarity of the inference sample features and the historical sample features is compared to obtain a cosine similarity matrix; Based on the cosine similarity matrix, determine the similarity score of each of the historical sample data to be updated; Based on the similarity score, historical sample data in the memory bank whose similarity score exceeds a preset retention threshold are retained, and historical sample data in the memory bank whose similarity score is lower than the preset retention threshold are deleted, so as to update the memory bank.

9. The class incremental learning method based on a visual language model according to claim 8, characterized in that, Based on the multiple known types, the process involves selecting multiple high-confidence pseudo-labeled inference samples with the same known types from multiple inference sample data of the current task, and determining the inference sample features of the multiple inference samples, including: Based on the target visual language model, forward reasoning is performed on the multiple reasoning sample data to obtain the reasoning sample features corresponding to the multiple reasoning sample data; Based on the features of the multiple inference samples, determine the predicted probability distribution value for each of the known categories; Based on the predicted probability distribution value, inference sample data with a predicted probability distribution value greater than a preset threshold are selected from the multiple inference sample data and determined as the inference sample with high confidence pseudo-label.

10. A class-incremental learning system based on a visual language model, characterized in that, The system includes: The storage module is configured to build and store a memory of a preset sample size; The feature extraction module is configured to fine-tune the visual language pre-training model based on the training sample data and historical sample data of the current task to obtain a target visual language model; extract discriminative image features from the training sample data and the historical sample data based on the target visual language model; and determine the mean of category features and the shared covariance matrix of multiple known categories involved in the training sample data and the historical sample data based on the discriminative image features. The classifier construction module is configured to dynamically calibrate the empirical frequencies of the multiple known categories according to the inverse frequency adjustment mechanism to obtain the calibrated prior probabilities; and to determine the debiased classifier parameters of the multiple known categories according to the calibrated prior probabilities, the mean of the category features and the shared covariance matrix, so as to construct the debiased classifier. The fusion reasoning module is configured to determine the target classification result of the image to be classified based on the target visual language model and the bias-reducing classifier. The memory bank update module is configured to, after the current task training is completed, select multiple inference samples with high confidence pseudo-labels that are the same as the known types from multiple inference sample data of the current task, based on the multiple known types; and update the memory bank based on the inference samples with high confidence pseudo-labels. The system is configured to perform class incremental learning based on a visual language model as described in any one of claims 1-9.

Images