Efficient adaptation method during visual language model testing in field environment
By combining soft-cue tuning and meta-learning, and dynamically constructing meta-training tasks, the problems of distribution offset and open-class data in the field environment of visual language models are solved, and the model is able to adapt efficiently and be accurately evaluated in the field environment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- YUNNAN UNIV
- Filing Date
- 2026-01-08
- Publication Date
- 2026-04-10
AI Technical Summary
Visual language models face challenges in field environments, such as distribution shifts and open-class data, and existing technologies struggle to effectively adapt and evaluate them on out-of-distribution data.
We employ a soft-cue-based tuning approach, combined with the Reptile algorithm, which integrates domain hybridization and meta-learning. By dynamically constructing meta-training tasks, we enhance the model's adaptability to distribution shifts and open-class data.
This improved the model's adaptability and accuracy in the field, enabling real-time evaluation and efficient adaptation of out-of-distribution, open-class data.
Smart Images
Figure CN121837779A_ABST
Abstract
Description
Technical Field
[0001] This invention provides a method for efficient adaptation during visual language model testing in a field environment, belonging to the field of machine learning technology. Background Technology
[0002] In the complex and ever-changing open real world, machines inevitably encounter out-of-distribution problems, meaning that the data distribution encountered during testing is always different from the data distribution encountered during training, i.e., there is a distribution shift. Especially in field environments, test data is often open-class and has covariance shift, which poses a challenge to machine learning models in practical applications. Recently, the visual-text pre-trained model CLIP has demonstrated strong zero-shot inference performance by using contrastive learning methods for cross-modal unsupervised alignment training. During inference, it uses a text prompt "an image of [category]" as the input text encoder as a semantic query signal, instead of using traditional closed one-hot labels for classification, enabling visual language models to recognize various concepts in the open world.
[0003] However, even visual language models pre-trained on large-scale data and parameter-efficient transfer methods based on them face a significant challenge when evaluating distribution-biased data. To ensure good performance on downstream tasks, visual language models still require fine-tuning on task-specific data. Directly fine-tuning CLIP models results in the loss of original pre-training knowledge, and performance on out-of-distribution data is even worse than zero-shot inference. To better transfer CLIP to downstream tasks, some efficient transfer techniques based on soft-cue tuning have emerged. These techniques freeze the original pre-trained model parameters and only adjust a small subset of additional cue parameters, reducing parameter tuning and mitigating the catastrophic forgetting caused by fully fine-tuning CLIP models. However, these methods can only utilize known-class data within the distribution during soft-cue tuning training, and still face challenges when evaluating out-of-distribution, open-class data that is unavailable in the wild. Summary of the Invention
[0004] To address the covariate shifts and open-class data present in field environments, this paper proposes a method for adapting a visual language model to field environment data during testing. This method enables the model to adapt to out-of-distribution and open-class target data during testing. Based on existing soft-cue tuning techniques, this method has a relatively small number of trainable parameters. Test-time adaptation on a single target dataset allows for real-time evaluation in real-world environments, enhancing the model's adaptability and accuracy to field data with distribution shifts and open-class characteristics.
[0005] To achieve the above-mentioned objectives, this invention proposes a method for efficient adaptation during visual language model testing in a field environment, comprising the following steps:
[0006] (1) Collect a small number of samples from M source domains to obtain category images. After random centering and cropping, the resolution is 224×224, and after random horizontal flipping, the pixel range of the image is changed from [0, 255] to [0.0, 1.0]. Then, the mean and variance of each channel are standardized to align with the data distribution of the pre-trained CLIP input;
[0007] (2) Based on the data obtained in step (1) A domain mixing approach is used to dynamically construct meta-training tasks in each meta-training cycle to simulate various distribution shifts, thus making the constructed meta-training tasks diverse. For the few-sample data of each source domain, it is shuffled and divided into two parts. One part remains unchanged and serves as a meta-training task, while the other part serves as candidate data for domain mixing.
[0008] (3) After obtaining the candidate data of all source domains according to step (2), for the candidate data of each source domain, randomly select the candidate data of another source domain, and then from the uniform distribution Randomly sample a mixing factor and perform a domain mixing operation to form a new meta-training task;
[0009] (4) For steps (2) and (3), dynamically construct a meta-training task set for each meta-training cycle. There are a total of Each meta-training task will be used to train domain-independent meta-cues. The model participates in each meta-training task within the inner loop to achieve rapid adaptation to each domain-specific task.
[0010] (5) In the outer loop, based on the specific parameters updated in step (4) on each meta-task, Ultimately, updates are achieved on specific meta-task parameters across multiple domains;
[0011] (6) Based on steps (4) and (5), after multiple meta-training cycles of meta-cue parameter updates, a domain-independent meta-cue parameter is finally obtained as follows: This will be used to prompt during initialization testing. This allows for rapid adaptation to the target data. For each target data set... Using random enhancements including random cropping and random horizontal flipping. Enhance target data A series of different random views are selected, and then the views with higher prediction probabilities and entropy are filtered using a confidence selection method.
[0012] (7) Based on the self-entropy view filtering in step (6), minimize the self-entropy of the filtered view to enhance the consistency of the prediction, so that Adapt the target domain data during testing;
[0013] (8) Finally, utilize the optimized predict The result is the classification prediction of the target data.
[0014] Furthermore, the CLIP pre-trained model in step (1) can achieve zero-shot inference for open categories by using natural language to describe visual concepts. Specifically, given an image to be inferred... Extract its corresponding image features as ,in This represents the visual encoder. The candidate classification category name is... ,in This refers to the number of categories. Each candidate category is entered into the text prompt template "a [ In the picture, as text Input to text encoder Obtaining text features The prediction belongs to the first... The probability of each category is calculated as follows:
[0015]
[0016] in, Represents the calculation of cosine similarity. This represents the fixed temperature parameters learned during the pre-training phase.
[0017] Furthermore, in step (2), the dynamic construction method of the meta-training task based on domain hybridity can simulate various distribution shifts, making the constructed meta-training task... It has diversity. The specific construction consists of data from various few-sample source domains. It is a combination of dynamically constructed meta-tasks in each outer loop meta-cycle. Follow these rules:
[0018] A. For each source domain with few sample data After scrambling it, it is divided into two parts, namely .in Remain unchanged as a meta-training task ;and Candidate data for domain mixing .
[0019] B. Obtain candidate data from all source domains. Then, for each source domain, candidate data Randomly select the first Candidate data from each source domain Then from a uniform distribution Randomly sample a mixing factor Perform domain-mixed operations as shown in formula (2). and After modulation and mixing, new candidate data are formed for each source domain. :
[0020]
[0021] A new meta-training task can be constituted .
[0022] After the above process of dynamically constructing the meta-training task in each meta-cycle, each meta-cycle... A source domain can constitute This allows for the expansion of the number and diversity of meta-training tasks.
[0023] Furthermore, in step (4), the constructed meta-training task set This will be used to train domain-independent meta-cues, one of the meta-training tasks. From a small amount of labeled data The system is composed of components used for learning domain-specific tasks within the inner loop. Specifically, the training algorithm for this meta-cue is based on the Reptile algorithm, a first-order two-layer optimization in meta-learning, used to learn domain-independent cue parameters. The model participates in each meta-training task within the inner loop to achieve rapid adaptation to each domain-specific task. Assume the initial meta-cue parameters are... For the first Domain-specific meta-training tasks ,from Copy a prompt parameter For the Learning domain-specific methods in the group loop In each inner loop, In the mission Data Previous iteration The update process is represented as follows:
[0024]
[0025] in, express exist The above utilizes cross-entropy loss Updated One of these updates can be written as: , This is the learning rate for the inner loop. In the outer meta-loop, Ultimately, updates are implemented on specific meta-task parameters across multiple domains. The update process is as follows:
[0026]
[0027] Furthermore, in step (6), it is assumed that the meta-cue parameters obtained after training with domain-independent meta-cues are as follows: It will be used to prompt during test initialization. This allows for rapid adaptation to target data in the field environment. For each target data... Data augmentation The algorithm selects several different random views and filters out some views with high prediction probability self-entropy. Then, it enhances prediction consistency by minimizing self-entropy. Specifically, it uses confidence-based selection to filter views with high prediction probability self-entropy. ,in It is an indicator function. Indicates the first The self-entropy calculated for each enhanced view. It is a dynamic threshold, sorted from low to high. An enhanced view with self-entropy The percentile is used to determine this. Assume the stochastic augmentation process is... This indicates that it includes random cropping and random horizontal flipping. Therefore, the adaptation process of domain-independent meta-cues on the target data during testing can be represented as follows:
[0028]
[0029] in, or Representation model pairs No. The probability predicted by an augmented view.
[0030] Furthermore, in step (7), in order to minimize the self-entropy of the filtered view, The single-step optimization process for adapting to the target domain data during testing is as follows:
[0031]
[0032] Furthermore, in step (8), the predicted probability is calculated according to formula (1) using CLIP, and the optimized method is employed. predict The result is:
[0033]
[0034] This invention enhances the model's adaptability and accuracy to distributional shifts and open-class data in field environments. Attached Figure Description
[0035] Figure 1 This is a flowchart of the present invention. Detailed Implementation
[0036] The specific technical solutions of the present invention will be described with reference to the embodiments.
[0037] A method for efficient adaptation of visual language models during testing in a field environment, the process is as follows: Figure 1 As shown, it includes the following steps:
[0038] (1) Collect a small number of samples from M source domains to obtain category images. After random centering and cropping, the resolution is 224×224, and after random horizontal flipping, the pixel range of the image is changed from [0, 255] to [0.0, 1.0]. Then, the mean and variance of each channel are standardized to align with the data distribution of the pre-trained CLIP input;
[0039] (2) Based on the data obtained in step (1) A domain mixing approach is used to dynamically construct meta-training tasks in each meta-training cycle to simulate various distribution shifts, thus making the constructed meta-training tasks diverse. For the few-sample data of each source domain, it is shuffled and divided into two parts. One part remains unchanged and serves as a meta-training task, while the other part serves as candidate data for domain mixing.
[0040] The domain-hybrid meta-training task dynamic construction method can simulate various distribution shifts, enabling the constructed meta-training task to achieve the desired results. It has diversity. The specific construction consists of data from various few-sample source domains. It is a combination of dynamically constructed meta-tasks in each outer loop meta-cycle. Follow these rules:
[0041] A. For each source domain with few sample data After scrambling it, it is divided into two parts, namely .in Remain unchanged as a meta-training task ;and Candidate data for domain mixing .
[0042] B. Obtain candidate data from all source domains. Then, for each source domain, candidate data Randomly select the first Candidate data from each source domain Then from a uniform distribution Randomly sample a mixing factor Perform domain-mixed operations as shown in formula (2). and After modulation and mixing, new candidate data are formed for each source domain. :
[0043]
[0044] A new meta-training task can be constituted .
[0045] After the above process of dynamically constructing the meta-training task in each meta-cycle, each meta-cycle... A source domain can constitute This allows for the expansion of the number and diversity of meta-training tasks.
[0046] (3) After obtaining the candidate data of all source domains according to step (2), for the candidate data of each source domain, randomly select the candidate data of another source domain, and then from the uniform distribution Randomly sample a mixing factor and perform a domain mixing operation to form a new meta-training task;
[0047] (4) For steps (2) and (3), dynamically construct a meta-training task set for each meta-training cycle. There are a total of Each meta-training task will be used to train domain-independent meta-cues. The model participates in each meta-training task within the inner loop to achieve rapid adaptation to each domain-specific task.
[0048] (5) In the outer loop, based on the specific parameters updated in step (4) on each meta-task, Ultimately, updates are achieved on specific meta-task parameters across multiple domains;
[0049] Constructed meta-training task set This will be used to train domain-independent meta-cues, one of the meta-training tasks. From a small amount of labeled data The system is composed of components used for learning domain-specific tasks within the inner loop. Specifically, the training algorithm for this meta-cue is based on the Reptile algorithm, a first-order two-layer optimization in meta-learning, used to learn domain-independent cue parameters. The model participates in each meta-training task within the inner loop to achieve rapid adaptation to each domain-specific task. Assume the initial meta-cue parameters are... For the first Domain-specific meta-training tasks ,from Copy a prompt parameter For the Learning domain-specific methods in the group loop In each inner loop, In the mission Data Previous iteration The update process is represented as follows:
[0050]
[0051] in, express exist The above utilizes cross-entropy loss Updated One of these updates can be written as: , This is the learning rate for the inner loop. In the outer meta-loop, Ultimately, updates are implemented on specific meta-task parameters across multiple domains. The update process is as follows:
[0052]
[0053] (6) Based on steps (4) and (5), after multiple meta-training cycles of meta-cue parameter updates, a domain-independent meta-cue parameter is finally obtained as follows: This will be used to prompt during initialization testing. This allows for rapid adaptation to the target data. For each target data set... Using random enhancements including random cropping and random horizontal flipping. Enhance target data A series of different random views are selected, and then the views with higher prediction probabilities and entropy are filtered using a confidence selection method.
[0054] Suppose that the meta-cue parameters obtained after training with domain-independent meta-cues are It will be used to prompt during test initialization. This allows for rapid adaptation to target data in the field environment. For each target data... Data augmentation The algorithm selects several different random views and filters out some views with high prediction probability self-entropy. Then, it enhances prediction consistency by minimizing self-entropy. Specifically, it uses confidence-based selection to filter views with high prediction probability self-entropy. ,in It is an indicator function. Indicates the first The self-entropy calculated for each enhanced view. It is a dynamic threshold, sorted from low to high. An enhanced view with self-entropy The percentile is used to determine this. Assume the stochastic augmentation process is... This indicates that it includes random cropping and random horizontal flipping. Therefore, the adaptation process of domain-independent meta-cues on the target data during testing can be represented as follows:
[0055]
[0056] in, or Representation model pairs No. The probability predicted by an augmented view.
[0057] (7) Based on the self-entropy view filtering in step (6), minimize the self-entropy of the filtered view to enhance the consistency of the prediction, so that Adapt the target domain data during testing;
[0058] To minimize the self-entropy of the filtered view, The single-step optimization process for adapting to the target domain data during testing is as follows:
[0059]
[0060] (8) Finally, utilize the optimized predict The result is the classification prediction of the target data.
[0061] Based on formula (7), CLIP calculates the prediction probability using the optimized method. predict The result is:
[0062] .
Claims
1. A method for efficient adaptation during visual language model testing in a field environment, characterized in that, Includes the following steps: (1) Collect a small number of samples from M source domains to obtain category images. After random center cropping, the resolution is 224×224, and after random horizontal flipping, the range of image pixels is changed from [0, 255] to [0.0, 1.0]. Then, the mean and variance of each channel are standardized to align with the data distribution of the pre-trained CLIP input. (2) Based on the data obtained in step (1) The domain mixing approach is used to dynamically construct meta-training tasks in each meta-training cycle to simulate various distribution shifts, thus making the constructed meta-training tasks diverse. For the few-sample data of each source domain, it is shuffled and divided into two parts. One part remains unchanged as a meta-training task, and the other part is used as candidate data for domain mixing. (3) After obtaining the candidate data of all source domains according to step (2), for the candidate data of each source domain, randomly select the candidate data of another source domain, and then from the uniform distribution Randomly sample a mixing factor and perform a domain mixing operation to form a new meta-training task; (4) For steps (2) and (3), dynamically construct a meta-training task set for each meta-training cycle. There are a total of Each meta-training task will be used to train domain-independent meta-cues; the model participates in each meta-training task in the inner loop to achieve rapid adaptation to each domain-specific task. (5) In the outer loop, based on the specific parameters updated in step (4) on each meta-task, Ultimately, updates are achieved on specific meta-task parameters across multiple domains; (6) Based on steps (4) and (5), after multiple meta-training cycles of meta-cue parameter updates, a domain-independent meta-cue parameter is finally obtained as follows: This will be used to prompt during initialization testing. To quickly adapt to target data; for each target data Using random enhancements including random cropping and random horizontal flipping. Enhance target data A series of different random views are selected, and then the views with higher prediction probabilities and entropy are filtered using a confidence selection method. (7) Based on the self-entropy view filtering in step (6), minimize the self-entropy of the filtered view to enhance the consistency of the prediction, so that Adapt the target domain data during testing; (8) Finally, utilize the optimized predict The result is the classification prediction of the target data.
2. The method for efficient adaptation during visual language model testing in a field environment according to claim 1, characterized in that, In step (1), the CLIP pre-trained model achieves zero-shot inference for open categories by using natural language to describe visual concepts; specifically: Given an image to be deduced Extract its corresponding image features as ,in Indicates a visual encoder; The candidate classification category name is ,in This refers to the number of categories; each candidate category is filled into the text prompt template "a picture [ In the picture, as text Input to text encoder Obtaining text features Prediction belongs to the first The probability of each category is calculated as follows: ； in, Indicates the prediction is the first The result of the class, Represents the calculation of cosine similarity. This represents the fixed temperature parameters learned during the pre-training phase.
3. The method for efficient adaptation during visual language model testing in a field environment according to claim 2, characterized in that, In step (2), the constructed meta-training task Data from various small sample source domains It is a combination of dynamically constructed meta-tasks in each outer loop meta-cycle. Follow these rules: A. For each source domain with few sample data After scrambling it, it is divided into two parts, namely ;in Remain unchanged as a meta-training task ;and Candidate data for domain mixing ; B. Obtain candidate data from all source domains. Then, for each source domain, candidate data Randomly select the first Candidate data from each source domain Then from a uniform distribution Randomly sample a mixing factor Perform domain-mixed operations as shown in formula (2); where and After modulation and mixing, new candidate data are formed for each source domain. : ； A new meta-training task constitutes ; After the above process of dynamically constructing the meta-training task in each meta-cycle, each meta-cycle... Each source domain constitutes This allows for the expansion of the number and diversity of meta-training tasks.
4. The method for efficient adaptation during visual language model testing in a field environment according to claim 3, characterized in that, In step (4), the constructed meta-training task set This will be used to train domain-independent meta-cues, one of the meta-training tasks. From a small amount of labeled data Composition, used for learning domain-specific tasks in the inner loop; Specifically, the training algorithm for this meta-cue is based on the Reptile algorithm, a first-order two-layer optimization in meta-learning, used to learn domain-independent cue parameters. The model participates in each meta-training task in the inner loop to achieve rapid adaptation to each domain-specific task. Assume the initial meta-cue parameters are... For the first Domain-specific meta-training tasks ,from Copy a prompt parameter For the Learning domain-specific methods in the group loop In each inner loop, In the mission Data Previous iteration The update process is represented as follows: ； in, express exist The above utilizes cross-entropy loss Updated One of the updates was written as: , The learning rate is used for the inner loop; in the outer meta-loop, Ultimately, updates are implemented on specific meta-task parameters across multiple domains. The update process is as follows: 。 5. The method for efficient adaptation during visual language model testing in a field environment according to claim 4, characterized in that, In step (6), let the meta-cue parameters obtained after training with domain-independent meta-cues be: It will be used to prompt during test initialization. To quickly adapt to target data in the field environment; for each target data Data augmentation The system generates several different random views, filters out some views with high prediction probabilities of self-entropy, and then enhances the consistency of predictions by minimizing self-entropy. Specifically, a confidence-based selection method is used to filter views with higher prediction probabilities and entropy, i.e. ,in It is an indicator function. Indicates the first The self-entropy calculated for each enhanced view; It is a dynamic threshold, sorted from low to high. An enhanced view with self-entropy Determined by percentile; Assume the stochastic enhancement process is composed of This indicates that it includes random cropping and random horizontal flipping. Therefore, the adaptation process of domain-independent meta-cues on the target data during testing can be represented as follows: ； in, or Representation model pairs No. The probability predicted by an augmented view.
6. The method for efficient adaptation during visual language model testing in a field environment according to claim 5, characterized in that, In step (7), in order to minimize the self-entropy of the filtered view, The single-step optimization process for adapting to the target domain data during testing is as follows: 。 7. The method for efficient adaptation during visual language model testing in a field environment according to claim 6, characterized in that, In step (8), the predicted probability is calculated according to formula (1) using CLIP, and the optimized method is applied. predict The result is: 。