Visual language model classification adaptive method based on transduction information maximization

Abstract

The invention discloses a visual language model classification adaptive method based on transduction information maximization, and solves the problems of high calculation cost and poor universality caused by unstable classifier initialization and model parameter updating in zero-sample and few-sample scenes in the prior art. The classification performance of the classifier in a few-sample scene is effectively improved; the method comprises the following steps: extracting visual features and category text features of a support set and a query set by utilizing a pre-trained visual language large model; and constructing a classifier prediction probability model taking the optimizable category prototype as a parameter based on the visual features. And combining the real label of the support set with the prediction category indication information of the query set to construct a joint category prototype as an initial parameter of the classifier. And by taking the query set as a transduction object, constructing a transduction information maximization objective function containing entropy regularization and KL divergence regularization, and carrying out iterative optimization on classifier parameters to obtain a final model. And performing classification prediction on the query set based on the final model and outputting a result.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a visual language model classification adaptive method based on maximizing transduction information. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning-based computer vision methods have made significant progress in tasks such as image classification and object recognition. In recent years, large-scale visual-language models (VLMs) have been pre-trained on large-scale image and text data by jointly modeling visual information and natural language semantics, giving the models strong cross-modal representation and generalization capabilities. Typical examples include CLIP models. These models have shown great potential in zero-shot classification and few-shot learning tasks, becoming important foundational models in current research and applications. However, in real-world applications, target tasks often face problems such as scarce or even completely missing labeled samples, and significant shifts in data distribution compared to the pre-training stage. Directly using pre-trained visual-language models for inference can easily lead to a decline in classification performance.

[0003] To improve the adaptability of models to target data, researchers have proposed various downstream adaptation methods based on visual language models. In existing technologies, one type of method fine-tunes or provides cue learning using a small number of labeled samples to guide the model to better adapt to the target task. These methods typically rely on real label information from the support set and can achieve some performance improvement in scenarios with few samples. However, their adaptability is significantly limited when labeled samples are extremely limited or nonexistent. Furthermore, some methods require parameter updates to the visual language model itself, resulting in shortcomings in computational cost, stability, and model reusability, and are unsuitable for applications involving frozen or black-box models. Another type of method focuses on transductive learning or adaptive learning during testing, using unlabeled data in the target domain to adaptively optimize the model and mitigate performance degradation caused by distribution shifts. For example, some methods use cluster analysis and pseudo-label generation to adjust the model using structural information from unlabeled samples. Although these methods have certain advantages under unlabeled conditions, their performance is highly dependent on the accuracy of pseudo-labels or clustering results. In cases of complex data distribution or high noise, error accumulation can easily occur, affecting model stability and the final classification result.

[0004] In addition, some existing methods typically rely on a small support set to build the classifier during the model initialization phase, failing to fully utilize the overall distribution information of the target query data. This results in a deviation between the initialization parameters and the true target distribution, thus limiting the effectiveness of subsequent adaptive optimization.

[0005] In summary, existing classification methods based on large visual language models still have certain limitations in zero-shot and few-shot scenarios. On the one hand, these methods typically rely heavily on labeled samples and have limited adaptability to the distribution of target data in unlabeled or weakly supervised scenarios. On the other hand, existing classifier initialization methods are often unstable and fail to fully characterize the true class structure of the target data, thus affecting subsequent prediction performance. Furthermore, some methods require updating the parameters of the pre-trained visual language model, which not only incurs high computational costs but also reduces the generality and scalability of the methods to some extent. Summary of the Invention

[0006] This invention provides a visual language model classification adaptive method based on maximizing transduction information, which solves the problems of excessive dependence on labeled samples, unstable classifier initialization, high computational cost and poor versatility caused by model parameter updates in zero-sample and few-sample scenarios. It effectively improves the initial discrimination ability and stability of the classifier in few-sample scenarios.

[0007] This invention provides a visual language model classification adaptive method based on maximizing transduction information, the method comprising: Extracting support sets using pre-trained large visual language models and query set Visual feature vectors and the text feature representations corresponding to each category And based on the visual feature vector Construct a classifier prediction probability model with optimizable class prototype vectors as parameters; Based on the support set and the query set Visual feature vectors corresponding to each sample image and support set The true category label corresponding to the sample, combined with the query set The predicted category information of the samples is used to construct a joint category prototype, which serves as the initial parameters for the classifier's prediction probability model. With the query set For the transduction object, a transduction information maximization objective function is constructed, and the initial parameters of the classifier prediction probability model are iteratively optimized according to the objective function to obtain the final classifier prediction probability model; wherein, the transduction information maximization objective function includes: an entropy regularization term and a KL divergence regularization term; The query set is predicted based on the final classifier prediction probability model. Perform classification prediction and output the classification results.

[0008] In one possible implementation, the support set is extracted using a pre-trained large visual language model. and query set Visual feature vectors and the text feature representations corresponding to each category ,include: Using the visual encoder of the aforementioned large visual language model, the support set is... and the query set Each sample image is encoded to obtain the corresponding visual feature vector. ; The text encoder of the aforementioned visual language large model is used to encode text descriptions for each category, thereby obtaining the corresponding text feature representations. Wherein, the visual feature vector Text feature representation They are in the same feature space and have been normalized.

[0009] In one possible implementation, the visual feature vector Represented as: ; The text feature representation Represented as: ; in, Indicates the first One input image; This refers to the visual encoder in the pre-trained large visual language model; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text descriptions for each category; This refers to the text encoder in the pre-trained visual language large model; Indicates the first Text feature representation for each category.

[0010] In one possible implementation, the zero-shot prediction probability model corresponding to the pre-trained large visual language model is represented as: ; in, Indicates the temperature scaling factor; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text feature representation for each category; This indicates that the pre-trained visual language model is for the first... The input image is identified as the first... The predicted probability results for the class; Indicates the first Text feature representation for each category; This indicates the total number of categories.

[0011] In one possible implementation, the classifier prediction probability model is expressed as: ; in, A random variable representing the category label; Indicates a category index; A random variable representing the input image; Indicates the first One input image; This represents the weight matrix of the classifier's prediction probability model; This refers to the visual encoder in the pre-trained large visual language model; Indicates temperature parameter; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The input image belongs to the first The posterior probability of a class.

[0012] In one possible implementation, the support set is used... and the query set Visual feature vectors corresponding to each sample image and support set The true category label corresponding to the sample, combined with the query set The predicted class information of the samples is used to construct a joint class prototype, which serves as the initial parameters for the classifier's prediction probability model, including: The query set is processed using the pre-trained visual language large model. Perform zero-shot inference to obtain the predicted category indication information; Based on the true category labels and predicted category indication information, the support set and query set Visual feature vectors Weighted aggregation is performed to obtain a joint category feature prototype, which is then used as the initial parameters of the classifier prediction probability model.

[0013] In one possible implementation, in a few-shot learning scenario, the joint category prototype with few samples is represented as: ; in, Indicates a support set; Represents a query set; Indicates support set The Middle For the input image of the th time Real labels for each category; Indicates the first Visual feature vectors corresponding to each input image; Represents query set The Middle For the input image of the th time Hard-coded predictions for each category; Indicates the total number of categories; In the case of few samples, the first The initial joint class prototype vector of the class.

[0014] In one possible implementation, in a zero-shot learning scenario, the joint category prototype of zero-shot learning is represented as: ; in, Represents a query set; Represents query set The Middle For the input image of the th time Soft-coded predictions for each category; Indicates the first Visual feature vectors corresponding to each input image; Indicates the total number of categories; In the zero-sample scenario, the first The initial class prototype vector.

[0015] In one possible implementation, the objective function for maximizing the transduction information is expressed as: ; in, Indicates the first trade-off coefficient; Indicating support set Cross-entropy loss; Indicates in the query set Mutual information items; Represents query set A set of visual features; Represents query set The set of predicted labels; Indicates the second trade-off coefficient; Indicates KL divergence; This represents the output probability distribution of the classifier's prediction probability model; This represents the predicted probability distribution in zero-shot scenarios based on a pre-trained visual language large model.

[0016] In one possible implementation, the iterative optimization of the initial parameters of the classifier prediction probability model based on the objective function to obtain the final classifier prediction probability model includes: The objective function that minimizes transduction information is minimized using the alternating direction multiplier method, by introducing auxiliary variables. The objective function is solved iteratively. Each iteration of the solution process includes alternating execution of updating auxiliary variables. and classifier prediction probability model parameters .

[0017] One or more technical solutions provided in this invention have at least the following technical effects or advantages: This invention utilizes a pre-trained visual language model to extract visual feature vectors and text feature representations corresponding to each category from the support set and query set. Based on these visual feature vectors, a classifier prediction probability model is constructed using optimizable category prototype vectors as parameters. This step fully leverages the powerful representational capabilities of the pre-trained model, ensuring efficient feature extraction and cross-modal consistency, providing a unified and reliable foundation for subsequent optimization, while avoiding the computational cost of updating model parameters. This approach is suitable for frozen models or black-box scenarios. Secondly, based on the visual feature vectors of each sample image in the support and query sets, the true category labels of the support set, and the predicted category indication information of the query set, a joint category prototype is constructed as the initial parameters for the classifier prediction probability model. This integrates the distribution information of limited labeled and unlabeled data, enabling the initial parameters to more comprehensively reflect the characteristics of the target data, effectively alleviating the instability problem of parameter initialization under conditions of few samples or no labels, and improving the model's discriminative ability. Then, using the query set as the transduction object, a transduction information maximization objective function containing entropy regularization and KL divergence regularization terms is constructed, and the initial parameters are iteratively optimized to obtain the final model. The entropy regularization term constrains prediction uncertainty, while the KL divergence regularization term introduces textual semantic priors. This optimization strategy fully utilizes the internal structural information of the query set, achieving adaptive updates without additional annotations, significantly enhancing the model's classification accuracy and robustness under zero-shot, few-shot, and cross-domain distribution shifts. Finally, the query set is classified and predicted based on the final classifier prediction probability model, and the results are output. This directly supports applications such as zero-shot classification, few-shot learning, and cross-dataset transfer, demonstrating the method's practicality and versatility, ensuring reliable classification results and easy deployment. Overall, these steps work together to solve the problems of strong dependence on annotations, unstable initialization, and high computational cost in existing technologies. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the steps of the visual language model classification adaptive method based on maximizing transduction information provided in this embodiment of the invention; Figure 2 This diagram illustrates a comparison of the classification performance of the method of the present invention with various existing few-shot learning methods under different few-shot settings. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a visual language model classification adaptive method based on maximizing transduction information, which includes the following steps S101 to S104.

[0021] S101 utilizes a pre-trained large visual language model to extract the support set. and query set Visual feature vectors and the text feature representations corresponding to each category And based on visual feature vectors Construct a classifier prediction probability model with optimizable class prototype vectors as parameters; Specifically, in step S101, the support set is extracted using a pre-trained large visual language model. and query set Visual feature vectors and the text feature representations corresponding to each category The steps include S1011 to S1012.

[0022] S1011 utilizes the visual encoder of the large visual language model to analyze the support set. and query set Each sample image is encoded to obtain the corresponding visual feature vector. ; S1012 utilizes the text encoder of the visual language large-scale model to encode text descriptions for each category, obtaining the corresponding text feature representations. Among them, visual feature vectors Text feature representation They are in the same feature space and have been normalized.

[0023] Here, visual feature vectors Represented as: (1.1) Text feature representation Represented as: (1.2) in, Indicates the first One input image; This represents the visual encoder in a pre-trained large visual language model. Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text descriptions for each category; This represents the text encoder in a pre-trained large visual language model. Indicates the first Text feature representation for each category.

[0024] The zero-shot prediction probability model corresponding to the pre-trained large visual language model is represented as: (1.3) in, Indicates the temperature scaling factor; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text feature representation for each category; This indicates that the pre-trained visual language model is for the first... The input image is identified as the first... The predicted probability results for the class.

[0025] The classifier prediction probability model in this invention is expressed as follows: (1.4) in, A random variable representing the category label; Indicates a category index; A random variable representing the input image; Indicates the first One input image; This represents the weight matrix of the classifier's prediction probability model; This represents the visual encoder in a pre-trained large visual language model. Indicates temperature parameter; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The input image belongs to the first The posterior probability of a class.

[0026] For example, a support set can be extracted using a pre-trained visual language model. With query set Visual feature vectors and text feature representation A unified feature representation space is constructed; and based on visual features, the prediction probability form of the classifier is defined, providing a foundation for subsequent adaptive optimization of classification. For a pre-trained Visual-Language Model (VLM), such as CLIP, let... This represents an input image. Indicates the first The text description corresponding to the class, where The input image is extracted using a pre-trained VLM visual encoder and text encoder. With text description The features are embedded and L2 normalized to obtain the visual feature vector. With text feature vectors .

[0027] S102, based on support set and query set Visual feature vectors corresponding to each sample image and support set The true category label corresponding to the sample, combined with the query set The predicted class information of the sample is used to construct a joint class prototype as the initial parameters of the classifier's prediction probability model; Specifically, step S102 includes the following steps S1021 and S1022.

[0028] S1021, using a pre-trained large visual language model to process the query set Perform zero-shot inference to obtain predicted category indication information; S1022, Based on the true category label and predicted category indication information, the support set is... and query set Visual feature vectors We perform weighted aggregation to obtain a joint category feature prototype, and use the joint category prototype as the initial parameters of the classifier prediction probability model.

[0029] Here, in a few-shot learning scenario, the joint class prototype for few shots is represented as: (1.5) in, Indicates a support set; Represents a query set; Indicates support set The Middle For the input image of the th time Real labels for each category; Indicates the first Visual feature vectors corresponding to each input image; Represents query set The Middle For the input image of the th time Hard-coded predictions for each category; Indicates the total number of categories; In the case of few samples, the first The initial joint class prototype vector of the class.

[0030] In zero-shot learning scenarios, the class prototype of zero-shot learning is represented as: (1.6) in, Represents a query set; Represents query set The Middle For the input image of the th time Soft-coded predictions for each category; Indicates the first Visual feature vectors corresponding to each input image; Indicates the total number of categories; In the zero-sample scenario, the first The initial class prototype vector.

[0031] For example, after completing the support set With query set Visual feature vectors and text feature representation Next, the parameters of the classifier prediction probability model are initialized.

[0032] The parameters of the classifier prediction probability model that needs to be initialized are defined as the weight matrix. Each of them Can be regarded as the corresponding category The characteristic prototype.

[0033] In few-shot learning scenarios, the following joint initialization strategy is proposed to support the set With query set The information is also incorporated into the prototype construction process, as shown in Equation (1.5). This invention adopts hard-coded few-sample prediction because it is more consistent in form with the true labels of the support set, thereby helping to improve the stability and convergence of the optimization process.

[0034] In zero-shot scenarios, the prototype initialization method also needs to be adjusted accordingly to eliminate the dependence on support set supervision information. Specifically, the initialization strategy is redefined as: support set If empty, see formula (1.6). In the zero-sample scenario, this paper uses soft-coded prediction for initialization. This is because in the absence of support set supervision, hard-coded prediction is prone to error accumulation, which leads to poor initialization results. In contrast, soft probability distribution can provide richer and more stable semantic information, which helps to obtain more reliable performance.

[0035] S103, with query set For the transduction object, a transduction information maximization objective function is constructed, and the initial parameters of the classifier prediction probability model are iteratively optimized according to the transduction information maximization objective function to obtain the final classifier prediction probability model; wherein, the transduction information maximization objective function includes: entropy regularization term and KL divergence regularization term; Specifically, in step S103, the objective function for maximizing transduction information is expressed as: (1.7) in, Indicates the first trade-off coefficient; Indicating support set Cross-entropy loss; Indicates in the query set Mutual information items; Represents query set A set of visual features; Represents query set The set of predicted labels; Indicates the second trade-off coefficient; Indicates KL divergence; This represents the output probability distribution of the classifier's prediction probability model; This represents the predicted probability distribution in zero-shot scenarios based on a pre-trained visual language large model.

[0036] Specifically, in step S103, the initial parameters of the classifier prediction probability model are iteratively optimized according to the objective function of maximizing transduction information to obtain the final classifier prediction probability model, including: (1) The alternating direction multiplier method is used to minimize the transduction information and maximize the objective function by introducing auxiliary variables. The objective function is solved iteratively. (2) In each iteration of the solution process, including alternating execution: updating auxiliary variables and classifier prediction probability model parameters .

[0037] For example, after the classifier initialization is complete, for the query set The classifier parameters are adaptively optimized using transduction. By constructing an optimization objective that maximizes transduction information centered on the query set, the uncertainty of single-sample prediction is reduced while constraining the overall class distribution structure to avoid class collapse or imbalanced prediction distribution. Furthermore, a KL divergence regularization term is introduced into the optimization objective to explicitly integrate the textual information of the visual language model into the adaptive update process of the classifier.

[0038] Visual feature vectors extracted from S101 Based on the corresponding classifier prediction probability model, a transduction information maximization optimization objective is constructed. Solving the above classifier prediction probability model, and adjusting the classifier weight matrix... Iterative optimization and updates will be carried out.

[0039] The objective function is defined by formula (1.7). This represents the cross-entropy loss on the support set. This represents the mutual information between query set features and their predicted labels. The regularization term of the KL divergence is used to explicitly incorporate textual information from the visual language model into the optimization objective. (The last part, "weighting coefficient," appears to be an unrelated fragment and is left untranslated.) , and These are used to control the relative importance of the supervised loss term, entropy regularization term, and KL divergence regularization term in the overall objective function, respectively.

[0040] Specifically, the cross-entropy loss is defined as: (1.8) in, Indicates support set samples Does it belong to a category? .

[0041] The mutual information term can be further decomposed into two parts: marginal entropy and conditional entropy. (1.9) in, Used to control the ratio of marginal entropy to conditional entropy. The marginal distribution of categories on the query set is defined as follows: .

[0042] The KL divergence term is defined as: (1.10) It represents the average KL divergence between the model's predicted posterior distribution and the CLIP zero-sample predicted distribution. For a single query sample The KL divergence is defined as: (1.11) in Used to adjust the relative weights between the entropy term and the cross-entropy term. When this happens, the expression degenerates into the standard KL divergence form.

[0043] To adapt to zero-shot learning scenarios, by... Setting it to 0 removes the supervision term, relying solely on query-set-based entropy regularization and KL divergence regularization for optimization. The objective function for maximizing transduction information in zero-shot learning scenarios is: (1.12) When solving the above classifier prediction probability model, since the objective function simultaneously includes the prediction probability... The linear and logarithmic terms make directly solving this problem quite difficult. Therefore, we introduce auxiliary variables. , used to represent the latent class assignment of the query sample, such that It appears only in the logarithmic term, thus simplifying the optimization process. The objective function (1) can be approximately minimized by the following constrained optimization problem: (1.13) Among them, auxiliary variables This indicates that it corresponds to the query set. The category assignment probability matrix is ​​used to approximate the query set. Prediction probability matrix ; For the corresponding number Marginal distribution of each category.

[0044] For the constrained optimization problem, i.e., Equation (1.13), the alternating direction multiplier method is used for solution. In each iteration, the auxiliary variables are updated alternately. and classifier weights . No. In this iteration, the update rules are as follows: (1.14) Among them, the parameters must satisfy ,and: (1.15) S104, Based on the final classifier prediction probability model, the query set Perform classification prediction and output the classification results.

[0045] For example, after completing the transductive adaptive optimization, the query set samples are finally classified and predicted based on the updated classifier parameters, and the corresponding category results are output. The classification results can be directly applied to application scenarios such as zero-shot classification, few-shot learning, cross-dataset transfer, and domain generalization, thereby verifying the robustness and versatility of the method of the present invention under different data distribution conditions.

[0046] This invention introduces the real label information of the support set and the predicted category indication information of the query set samples simultaneously during the classifier initialization stage to construct a joint category prototype. This enables the initial classifier parameters to more comprehensively reflect the distribution characteristics of the target data, effectively alleviating the problem of initialization instability caused by insufficient samples under conditions of few or no samples.

[0047] This invention employs a transductive information maximization optimization strategy based on unlabeled query sets to adaptively update the classifier. This fully utilizes the internal structural information of the query set without the need for additional manual annotation, thereby significantly improving the classification performance and robustness of the model in scenarios with zero samples, few samples, and cross-domain distribution shifts.

[0048] This invention does not require updating the parameters of the pre-trained visual language model; it only optimizes the classifier in the feature space. It is suitable for frozen model or black-box model scenarios and can be flexibly applied to different visual language models and various classification tasks. It has good versatility and engineering application value.

[0049] In a simulation experiment provided by this invention, the simulation experiment platform of this invention is based on PyTorch 2.0.1 and CUDA 12.2, and is completed in the Ubuntu 22.04 operating system environment. The hardware configuration includes NVIDIA RTX A5000 GPU, 64GB memory and 13th generation Intel® Core™ i9-13900K processor.

[0050] In our experimental evaluation, we selected 11 publicly available datasets: ImageNet, SUN397, FGVC-Aircraft, EuroSAT, Stanford Cars, Food101, Oxford Pets, Oxford Flowers, Caltech101, DTD, and UCF101. These datasets cover a variety of visual tasks, including general object recognition, scene classification, fine-grained recognition, remote sensing image analysis, and action recognition, demonstrating strong diversity and representativeness.

[0051] In all experiments, a pre-trained CLIP ViT-B / 16 was used as the visual encoder, and its parameters were kept frozen throughout the experiments. To ensure reproducibility and fairness, a fixed text prompt template was used for each dataset. All experiments were repeated under three different random seeds (1, 2, and 3), and the average Top-1 classification accuracy was reported as the final performance metric.

[0052] In all experiments, the hyperparameter values ​​were fixed: τ=120, λ=0.4, γ=0.05. Note that the iterative optimization process in this invention is independent of the specific values ​​of α and β, and only depends on α. The difference in β is relevant, therefore α is set. β=0.1, and it satisfies α The stability condition β+1>0. The optimization parameter updates of the method were performed T=150 times.

[0053] Simulation 1: Comparative experimental results with existing methods in a few-shot learning scenario.

[0054] To verify the effectiveness and stability of the method of this invention, simulation experiments were conducted on 11 publicly available classification datasets, comparing it with several existing representative methods, including Zero-shot CLIP, TF, BD-CSPN, LaplacianShot, PT-MAP, LP, LP++, and TransCLIP. In the experimental settings, different small sample sizes (1, 2, 4, 8, and 16-shot) were used, and the experiments were repeated under three different random seed conditions. The mean and standard deviation of the Top-1 classification accuracy of each method were statistically analyzed.

[0055] Simulation results are shown in Table 1 and Figure 2 As shown, the method of this invention achieved the highest classification accuracy under all the aforementioned few-sample settings, regardless of the dataset and sample size, demonstrating its good stability and robustness. Figure 2 This diagram illustrates a comparison of the classification performance of the method of this invention with several existing few-shot learning methods under different few-shot settings (1-shot, 2-shot, 4-shot, 8-shot, and 16-shot). The figure shows the average Top-1 classification accuracy obtained from testing on 11 public datasets.

[0056] Especially in extremely limited sample scenarios such as 1-shot, the method of this invention shows a more significant performance improvement compared to existing visual feature-driven methods, indicating that it can effectively utilize existing prior information for model inference when sample information is extremely limited.

[0057] Simulation results fully demonstrate that the method of the present invention has good generalization ability and stability under different small sample sizes and diverse datasets, and can effectively improve the prediction accuracy of small sample classification tasks, thus verifying the feasibility and superiority of the technical solution of the present invention.

[0058] Table 1 compares the present invention with existing methods. The results in the table are the average Top-1 classification accuracy (%) and its standard deviation obtained on 11 public datasets based on 3 random seed experiments. The best results are highlighted in bold.

[0059] Table 1

[0060] Simulation 2: Experimental results and analysis in a zero-sample scenario.

[0061] In this simulation, the performance of the proposed method was verified in a zero-shot learning scenario where no labeled samples are used for training. On the aforementioned 11 public datasets, the average Top-1 classification accuracy of the Zero-shot CLIP method without any adaptation is 65.25%. Building upon this, by introducing the zero-shot adaptation strategy of this invention, without using any labeled support samples, and relying solely on an unsupervised objective function to optimize model predictions, the average Top-1 classification accuracy is improved to 69.81%, representing a significant performance improvement compared to the original CLIP.

[0062] Table 2 shows the cross-dataset transferability evaluation. Three cue-based optimization methods were trained on the 16-shot ImageNet dataset and evaluated on ten other fine-grained datasets.

[0063] Table 2

[0064] Furthermore, the zero-shot adaptive method proposed in this invention can be used in conjunction with existing cue learning methods to further improve classification performance. Table 2 presents the cross-dataset transfer experiment results in zero-shot scenarios, based on different cue learning methods (including CoOp, CoCoOp, and KgCoOp) and combined with the zero-shot adaptive method of this invention. Specifically, each cue learning method first completes 16-shot few-shot training on the ImageNet dataset, and then directly transfers to the remaining 10 public datasets for cross-dataset evaluation. Experimental results show that after introducing the zero-shot adaptive mechanism of this invention on the basis of cue learning methods, the classification accuracy is significantly improved in both the original dataset and the cross-dataset transfer scenario. Among them, in terms of the average classification accuracy on the 10 cross datasets, the CoOp method improved from 60.18% to 64.71%, the CoCoOp method improved from 64.79% to 70.28%, and the KgCoOp method improved from 65.29% to 69.34%. The above results fully demonstrate that the method of the present invention can effectively enhance the adaptability of the model under different data distributions and migration scenarios, and further verify its good robustness and versatility.

[0065] In summary, the experimental results of Simulation 2 fully demonstrate that the method proposed in this invention not only performs excellently in scenarios with few samples, but also has significant performance advantages and good generalization ability in zero-sample learning scenarios, reflecting its practical value and advanced nature in the adaptive optimization of visual language models.

[0066] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. All or part of this invention can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. A visual language model classification adaptive method based on maximizing transduction information, characterized in that, include: Extracting support sets using pre-trained large visual language models and query set Visual feature vectors and the text feature representations corresponding to each category And based on the visual feature vector Construct a classifier prediction probability model with optimizable class prototype vectors as parameters; Based on the support set and the query set Visual feature vectors corresponding to each sample image and support set The true category label corresponding to the sample, combined with the query set The predicted category information of the sample is used to construct a joint category prototype as the initial parameters of the classifier's prediction probability model; With the query set For the transduction object, an objective function for maximizing transduction information is constructed, and the initial parameters of the classifier prediction probability model are iteratively optimized according to the objective function to obtain the final classifier prediction probability model; wherein, the objective function for maximizing transduction information includes: an entropy regularization term and a KL divergence regularization term; The query set is predicted based on the final classifier prediction probability model. Perform classification prediction and output the classification results.

2. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The method utilizes a pre-trained large visual language model to extract the support set. and query set Visual feature vectors and the text feature representations corresponding to each category ,include: Using the visual encoder of the aforementioned large visual language model, the support set is... and the query set Each sample image is encoded to obtain the corresponding visual feature vector. ; The text encoder of the aforementioned visual language big data model is used to encode text descriptions for each category, thereby obtaining the corresponding text feature representations. Wherein, the visual feature vector Text feature representation They are in the same feature space and have been normalized.

3. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The visual feature vector Represented as: ； The text feature representation Represented as: ； in, Indicates the first One input image; This refers to the visual encoder in the pre-trained large visual language model; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text descriptions for each category; This refers to the text encoder in the pre-trained visual language large model; Indicates the first Text feature representation for each category.

4. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The zero-shot prediction probability model corresponding to the pre-trained large visual language model is represented as follows: ； in, Indicates the temperature scaling factor; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first Text feature representation for each category; This indicates that the pre-trained visual language model is for the first... The input image is identified as the first... The predicted probability results for the class; Indicates the first Text feature representation for each category; This indicates the total number of categories.

5. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The classifier prediction probability model is expressed as follows: ； in, A random variable representing the category label; Indicates a category index; A random variable representing the input image; Indicates the first One input image; This represents the weight matrix of the classifier's prediction probability model; This refers to the visual encoder in the pre-trained large visual language model; Indicates temperature parameter; Indicates the first Visual feature vectors corresponding to each input image; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The feature prototype vector corresponding to the class; Indicates the first The input image belongs to the first The posterior probability of the class.

6. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The support set and the query set Visual feature vectors corresponding to each sample image and support set The true category label corresponding to the sample, combined with the query set The predicted class information of the samples is used to construct a joint class prototype, which serves as the initial parameters for the classifier's prediction probability model, including: The query set is processed using the pre-trained visual language large model. Perform zero-shot inference to obtain the predicted category indication information; Based on the true category labels and predicted category indication information, the support set and query set Visual feature vectors Weighted aggregation is performed to obtain a joint category feature prototype, which is then used as the initial parameters of the classifier prediction probability model.

7. The visual language model classification adaptive method based on maximizing transduction information according to claim 6, characterized in that, In few-shot learning scenarios, the joint category prototype of few samples is represented as: ； in, Indicates a support set; Represents a query set; Indicates support set The Middle For the input image of the th time Real labels for each category; Indicates the first Visual feature vectors corresponding to each input image; Represents query set The Middle For the input image of the th time Hard-coded predictions for each category; Indicates the total number of categories; In the case of few samples, the first The initial joint class prototype vector of the class.

8. The visual language model classification adaptive method based on maximizing transduction information according to claim 6, characterized in that, In a zero-shot learning scenario, the joint category prototype for zero-shot learning is represented as: ； in, Represents a query set; Represents query set The Middle For the input image of the th time Soft-coded predictions for each category; Indicates the first Visual feature vectors corresponding to each input image; Indicates the total number of categories; In the zero-sample scenario, the first The initial class prototype vector.

9. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The objective function for maximizing the transduction information is expressed as: ； in, Indicates the first trade-off coefficient; Indicating support set Cross-entropy loss; Indicates in the query set Mutual information items; Represents query set A set of visual features; Represents query set The set of predicted labels; Indicates the second trade-off coefficient; Indicates KL divergence; This represents the output probability distribution of the classifier's prediction probability model; This represents the predicted probability distribution in zero-shot scenarios based on a pre-trained visual language large model.

10. The visual language model classification adaptive method based on maximizing transduction information according to claim 1, characterized in that, The step of iteratively optimizing the initial parameters of the classifier prediction probability model according to the objective function to obtain the final classifier prediction probability model includes: The objective function that minimizes transduction information is minimized using the alternating direction multiplier method, by introducing auxiliary variables. The objective function is solved iteratively. Each iteration of the solution process includes alternating execution of updating auxiliary variables. and classifier prediction probability model parameters .

Images