A visual language model test adaptation method based on logarithmic calibration and consistency cache

Abstract

The application relates to a visual language model test adaptation method based on logarithmic calibration and consistency cache, and relates to the technical fields of computer vision, pattern recognition, machine learning and artificial intelligence. The method aims at the problems of class prediction bias, insufficient cache sample coverage and low utilization rate of boundary samples of a visual language model in a target domain test stage, and constructs an online adaptation framework composed of an image encoder, a text encoder, a dynamic logarithmic calibration module, a consistency guide exploration cache module and a cross-modal joint optimization module. The method improves the identification opportunities of difficult classes and low-frequency classes through a dynamic logarithmic calibration mechanism, and improves the overall class balance and classification stability. Through the consistency guide exploration cache mechanism, the coverage range of the cache to the real distribution of the target domain is expanded, and the perception ability of the model to the decision boundary region is enhanced, so that the adaptation effect in a complex distribution offset scene is improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, pattern recognition, machine learning and artificial intelligence, and in particular to a method for testing and adapting a visual language model based on logarithmic calibration and consistency caching. Background Technology

[0002] Visual language models are a class of multimodal models that are pre-trained on large-scale image and text data to learn a unified representational relationship between image information and text semantics. These models can map visual features and linguistic features to the same semantic space, thereby achieving cross-modal alignment between images and text. Based on these technical characteristics, visual language models have been widely applied to tasks such as zero-shot classification, open-vocabulary recognition, visual retrieval, multimodal understanding, image-text matching, and visual question answering, demonstrating good versatility and transferability in various downstream application scenarios.

[0003] Visual language models, exemplified by CLIP (Contrastive Language-Image Pre-trained Model), jointly train image and text encoders, enabling them to identify target categories solely based on textual descriptions without requiring additional supervised training for specific tasks. This demonstrates strong generalization performance in zero-shot recognition. However, in practical deployments, the test data often differs significantly in distribution from the training data, exhibiting a distribution shift problem. This shift can be caused by various factors, such as changes in weather conditions, lighting conditions, imaging equipment, image style, shooting angle, background complexity, target occlusion, and differences in category semantic granularity. All of these factors can degrade the cross-modal alignment established during training, leading to a significant decline in the zero-shot prediction performance of visual language models in the target domain.

[0004] To improve the adaptability of models in scenarios with distribution shifts, test-time adaptation techniques have gained increasing attention. Test-time adaptation refers to dynamically adjusting model parameters, auxiliary modules, or statistical information during the inference phase when only unlabeled test samples in the target domain are available and the source domain training data cannot be reused. This allows the model to gradually adapt to the target domain data distribution, thereby improving the model's prediction accuracy and robustness in real-world deployment environments. Compared to traditional offline domain adaptation methods, test-time adaptation techniques better meet the needs of real-world applications, especially those where source domain data cannot be back-transferred, labeling costs are high, or the target environment is constantly changing.

[0005] Existing test-time adaptive methods for visual language models mainly include cue word optimization, cache memory, and distribution statistical calibration. Cue word optimization enhances the matching between text semantics and target domain visual features by adjusting category text prompts or introducing learnable text residual parameters. Cache memory stores high-confidence samples, category features, or category prototypes encountered during testing and uses this historical information to assist in current sample prediction. Distribution statistical calibration estimates and recalibrates the category distribution, output confidence distribution, or feature distribution in the test stream to reduce the impact of distribution shift on classification results. In particular, cache memory, due to its ability to fully utilize historical sample information in streaming testing scenarios while balancing implementation complexity and adaptive performance, has become one of the important technical approaches in test-time adaptive methods for visual language models.

[0006] However, existing caching-based adaptive learning methods typically employ sample selection strategies based on entropy or confidence levels. This means prioritizing samples with high prediction confidence or low output entropy values ​​for caching, and then updating the category prototype or auxiliary model predictions based on these cached samples. While this approach can mitigate the noise risk introduced by unreliable pseudo-labels to some extent, it also easily introduces the model's original category prediction bias into the learning process during testing. Specifically, since visual language models are generally more likely to generate high-confidence predictions for some easily categorized classes, these samples are more likely to be prioritized for caching and repeatedly used in subsequent adaptive processes, leading to these classes dominating the pseudo-label distribution. Consequently, difficult categories, low-frequency categories, and samples near the decision boundary, due to their higher prediction uncertainty, are often difficult to cache, resulting in the ineffective utilization of information with significant adaptive value in the target domain.

[0007] On the other hand, given a fixed cache capacity, existing methods typically employ a minimum entropy-priority retention mechanism, which tends to retain a small number of the easiest-to-classify samples for a long time, while excluding a large number of samples in low-confidence regions or near the decision boundary. While this strategy helps maintain the surface purity of the cached samples, it also leads to insufficient coverage of the true distribution of the target domain. Especially when the target domain has an unbalanced class distribution, blurred class boundaries, or continuously changing environmental conditions, if the cache consists of only a small number of high-confidence samples for a long time, the model will struggle to fully perceive the complex structural information in the target domain, thus limiting further improvements in class prototype update effectiveness and adaptive performance during testing.

[0008] Therefore, existing technologies still lack an adaptive method for visual language model testing that can simultaneously suppress class prediction bias, fully exploit boundary sample information, and maintain the reliability of cached samples in streaming testing scenarios. How to improve the cache's coverage of the true distribution of the target domain while ensuring cache update stability, and enhance the model's adaptability to distribution shift environments, has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0009] To address the technical problems of existing visual language models (VLMs) during test-time adaptation (TTA), such as over-reliance on high-confidence samples based on entropy filtering leading to implicit inheritance of inherent class prediction bias, insufficient exploration of target data distribution, difficulty in covering decision boundary regions, and susceptibility to pseudo-label collapse, this invention proposes a test-time adaptation method for visual language models based on Dynamic LogitsAdjustment and Exploration (DLAE).

[0010] This invention provides an adaptive method for testing visual language models based on dynamic logic adjustment and exploration. It employs an end-to-end sequential processing architecture that includes feature extraction, logic recalibration, consistent cache filtering, and multimodal optimization. The specific technical solution is as follows:

[0011] A visual language model testing adaptation method based on logarithmic calibration and consistency caching includes the following steps:

[0012] S1. Obtain a continuous unlabeled test image data stream, extract image features and text prototype features of each category using an image encoder and a text encoder, calculate the similarity between the current image features and the text prototypes of each category, obtain the original semantic category logarithm, and generate the original classification probability, original pseudo-label, and uncertainty accordingly.

[0013] S2. During the online inference process, maintain two types of statistical information for each category: category count and historical average confidence. Construct empirical category prior probability based on historical category count. Construct category balance factor based on category prediction frequency statistics and historical average confidence statistics in historical test stream. Dynamically recalibrate the original category score based on empirical category prior probability and category balance factor to obtain calibrated category score and calibrated pseudo-label.

[0014] S3. Use the consistency-guided exploratory caching module to filter the current test samples and write the samples that meet the caching admission criteria into the corresponding category cache. The samples that meet the caching admission criteria include at least high-confidence samples and boundary samples whose pseudo-labels change before and after calibration and simultaneously meet the semantic consistency constraint and temporal consistency constraint. And construct category visual prototypes based on the visual features in the same category cache.

[0015] S4. Perform joint prediction and test-time optimization based on the visual prototype and calibrated category scores, and output the final classification result.

[0016] Furthermore, S1 is specifically as follows:

[0017] For any unlabeled image in the input data stream during the testing phase Visual features are extracted using a pre-trained visual encoder with frozen parameters, and then normalized to obtain image features. Meanwhile, regarding the inclusion The task involves classifying the data into categories c and constructing text prompts for each category. The data is then input into a pre-trained text encoder with frozen parameters to extract text prototype features for each category. ;

[0018] Subsequently, the inner product between image features and various text prototype features is calculated to obtain the basic raw category scores. Then, the similarity between the current image features and the text prototypes of each category is calculated to obtain the original semantic category logarithm, and the original zero-sample classification probability is generated accordingly.

[0019]

[0020] in, Here, y represents the temperature parameter, y represents the label, and c represents the category index. For any of the C categories, For category The original category score, The original classification probabilities are given; based on the above probability distribution, the category corresponding to the maximum probability is selected as the initial pseudo-label for the image. And use Shannon entropy to calculate the entropy value of the current original classification probability. This is used to characterize the uncertainty in prediction.

[0021] Furthermore, S2 is specifically as follows:

[0022] During the online inference process, two types of statistical information are maintained for each category: category count and historical average confidence. The category count is used to reflect the cumulative frequency of the corresponding category being predicted as a false label in the test stream, and the historical average confidence is used to reflect the average reliability of the prediction results of that category over history.

[0023] For the current test sample, the prediction confidence level corresponding to the original pseudo-label is determined based on the original classification probability obtained in S1. And construct the empirical class prior probability based on historical class counts. :

[0024]

[0025] in, This represents the category count variable with category c. Indicates category as The category count variable;

[0026] For the current test sample, calculate the deviation between the predicted confidence level corresponding to the current original pseudo-label and the historical average confidence level of this category. ;

[0027] Based on this, a category-related dynamic equilibrium function is constructed. :

[0028]

[0029] in, The dynamic logarithmic calibration intensity hyperparameter is used; the dynamic balancing function calibrates the original category scores to obtain the calibrated category scores. Based on the calibrated category scores, the calibrated probability distribution is further obtained, and the calibration pseudo-labels are identified. ,in, The category with the highest category score after calibration;

[0030] After the current sample is processed, the historical statistics are updated incrementally: the category count is updated by incrementing the cumulative count of the corresponding calibration pseudo-label category by one, and the average confidence is updated recursively based on the current calibration confidence.

[0031] Furthermore, S3 is specifically as follows:

[0032] Perform a flip test to determine whether the predicted label of the sample changes before and after the logical adjustment. When the above conditions are not met, the current sample is regarded as a regular high-confidence candidate sample, and whether to write it into the cache is determined based on its post-calibration confidence and uncertainty. When the above conditions are met, it indicates that the category of the current sample has changed before and after dynamic logarithmic calibration, triggering the semantic consistency filter.

[0033] Semantic consistency filter: The initial prediction entropy value of the sample is down-modulated by using the cosine similarity between the text features corresponding to the original predicted category and the text features corresponding to the calibrated predicted category in the text embedding space.

[0034] Then, all samples stored in the cache are evaluated based on the time consistency filter; specifically, let... This indicates the text features of the corresponding predicted category when the cached sample is initially written to the cache; This indicates the text features of the corresponding predicted category for the cached sample at the current time; if

[0035]

[0036] Then, by increasing its entropy value, a penalty is imposed to force its elimination:

[0037]

[0038] in, For the current time step, This refers to the time step when the sample is initially stored in the cache. Parameters used to control attenuation;

[0039] After semantic and temporal entropy modulation, the samples are sent to a priority queue with a fixed capacity, and a low-entropy priority strategy is used to insert samples and replace high-entropy samples.

[0040] After the cache update is complete, visual features in the same category of cache are aggregated to form a category visual prototype. The visual prototypes of each category are represented as follows:

[0041]

[0042] in, Indicate category The corresponding cached sample set, This indicates the number of samples in the cache set.

[0043] Furthermore, S4 is specifically as follows:

[0044] An affinity modulation function is introduced to process the inner product of image features and visual prototypes, which is then added to the calibrated category score to calculate the final prediction score. Therefore, the final classification result can be determined:

[0045]

[0046] During the optimization phase at test time, a joint optimization objective is constructed that updates only the lightweight residual parameters without updating the parameters of the image encoder and text encoder. The overall joint loss function includes confidence-aware calibration loss and symmetric cross-modal alignment loss.

[0047] Furthermore, the deviation is calculated as follows:

[0048]

[0049] in, This indicates that the current test sample was predicted as the original pseudo-label by the original classifier. The probability value, This represents the historical average confidence level corresponding to the original pseudo-label category;

[0050] when When the prediction confidence score is lower than the historical average for that category, it indicates that the category may be over-predicted, or the current sample is near the category decision boundary. When the confidence level is 1, it indicates that the current prediction confidence level is basically consistent with or higher than the historical level of the category, suggesting that the current prediction result is relatively more reliable.

[0051] Furthermore, the lightweight residual parameters include learnable text residuals. and learnable visual residuals Lightweight modifications were made to both the text prototype and the visual prototype.

[0052] The beneficial effects of this invention are as follows:

[0053] Firstly, by using a dynamic logarithmic calibration mechanism to comprehensively utilize the historical prediction frequency and historical average confidence of the categories, the original category logarithms are recalibrated online. This can suppress the excessive dominance of high-frequency categories in the test stream, alleviate the problem of imbalanced pseudo-label distribution, thereby increasing the recognition opportunities of difficult and low-frequency categories and improving the overall category balance and classification stability.

[0054] Secondly, by exploring the caching mechanism through consistency guidance, while retaining high-confidence samples, we can selectively introduce boundary samples that have been flipped before and after calibration but satisfy semantic consistency and temporal consistency constraints. This can expand the coverage of the cache to the true distribution of the target domain, enhance the model's ability to perceive the decision boundary region, and thus improve the adaptation effect in complex distribution offset scenarios.

[0055] Third, by constructing category visual prototypes based on cached samples and performing cross-modal joint optimization by combining learnable text residual parameters and learnable visual residual parameters, the alignment between text semantic representation and target domain visual representation can be continuously enhanced. This improves the robustness, generalization ability, and engineering deployability of visual language models in scenarios such as open environment recognition, autonomous driving, intelligent security, industrial inspection, and edge visual analysis without requiring access to source domain data and manual annotation information. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments of the present invention are briefly described below. Obviously, the drawings described below are merely illustrative drawings of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0057] Figure 1 This is a general framework diagram of the method according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic diagram of the data flow and online statistical update of the dynamic logarithmic calibration module provided in this embodiment of the invention;

[0059] Figure 3 This is a schematic diagram of the processing flow of the consistency-guided exploration cache module provided in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of semantic consistency constraints provided in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of time consistency constraints provided in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the joint prediction and testing optimization process provided in the embodiments of the present invention. Detailed Implementation

[0063] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Equivalent substitutions, modifications, improvements or combinations made by those skilled in the art to the present invention without departing from the concept and essence of the present invention should all fall within the scope of protection of the present invention.

[0064] Example 1.

[0065] like Figure 1As shown, this embodiment provides a visual language model test adaptation method based on logarithmic calibration and consistency caching, applicable to unlabeled test flow scenarios in the target domain. The method generally includes the following steps: First, perform zero-shot visual language inference on the input test samples to obtain the original class logarithms, original pseudo-labels, and their uncertainties; then, perform dynamic logarithmic calibration based on historical online statistical information to obtain a more balanced calibrated class score; next, update the exploratory cache by combining the prediction consistency relationship before and after calibration to introduce boundary samples with adaptation value while controlling noise risk; finally, construct a class visual prototype and combine the visual prototype with the calibrated semantic score for prediction and test-time optimization, thereby achieving continuous online adaptation of the visual language model in a distributed offset environment. Specific sub-steps are as follows:

[0066] S1. Obtain a continuous stream of unlabeled test image data and perform zero-shot inference feature extraction and probability calculation for the basic visual language model.

[0067] For any unlabeled image in the input data stream during the testing phase Visual features are extracted using a pre-trained and parameter-frozen visual encoder, and then... Normalization process to obtain image features Meanwhile, regarding the inclusion The task involves classifying the data into categories c and constructing text prompts for each category. The data is then input into a pre-trained text encoder with frozen parameters to extract text prototype features for each category. .

[0068] Subsequently, the inner product between image features and various text prototype features is calculated to obtain the basic raw category scores. , Calculate the similarity between the current image features and the text prototypes of each category to obtain the original semantic category logarithms, and generate the original zero-sample classification probabilities based on these logarithms.

[0069]

[0070] in, Here, is the temperature parameter used to adjust the smoothness of the probability distribution; y is the label; and c is the category index. For any of the C categories, For category The original category score, The original classification probabilities are given. Based on the above probability distribution, the category corresponding to the maximum probability is selected as the initial pseudo-label for the image, i.e.:

[0071]

[0072] The entropy of the current original prediction distribution is calculated using Shannon entropy to characterize the prediction uncertainty.

[0073]

[0074] Through step S1, the basic zero-shot inference of the current test sample on the frozen visual language model is completed, and the original category score, original pseudo-label and original uncertainty are output, providing a basis for subsequent dynamic calibration and cache update.

[0075] S2. Based on the historical statistical information of the test flow, the initial category scores are recalibrated online according to the statistical distribution to alleviate the category prediction bias and expose the decision boundary samples, so as to obtain the calibrated category scores and calibrated pseudo-labels.

[0076] like Figure 2 As shown, to avoid certain high-frequency categories from continuously dominating the distribution of false labels in the test stream, this implementation maintains two types of statistical information for each category during online inference: a category count and a historical average confidence score. The category count reflects the cumulative frequency with which the corresponding category is predicted as a false label in the test stream, and the historical average confidence score reflects the average reliability of the prediction results for that category over history. The category count variable... Initialize to 1; and run the average confidence variable. It is initialized to 0.

[0077] For the current test sample, first determine the prediction confidence level corresponding to the original pseudo-label based on the original classification probability obtained in S1. And construct the empirical class prior probability based on historical class counts:

[0078]

[0079] in, This represents the category count variable with category c. Indicates category as The category count variable; the category count variable Used to characterize categories The cumulative number of times a class is determined as the final predicted class during historical testing. Based on the count variables for each class, the empirical frequency distribution of each class in the test stream can be characterized, and the prior probability of the empirical class can be constructed accordingly. .

[0080] Meanwhile, to quantify the reliability of the current sample prediction results, a variable representing the historical average confidence level of the category is introduced. .in, Indicate category The average of the original classification confidence scores used as the final prediction result during historical testing.

[0081] For the current test sample, calculate the deviation between the predicted confidence level corresponding to the current original pseudo-label and the historical average confidence level of this category. :

[0082]

[0083] in, This indicates that the current test sample was predicted as the original pseudo-label by the original classifier. The probability value, This represents the historical average confidence level corresponding to the original pseudo-label category.

[0084] when When the prediction confidence score is lower than the historical average for that category, it indicates that the category may be over-predicted, or the current sample is near the category decision boundary. When the confidence level is 1, it indicates that the current prediction confidence level is basically consistent with or higher than the historical level of the category, suggesting that the current prediction result is relatively more reliable.

[0085] Based on this, a category-related dynamic equilibrium function is constructed. The class score is nonlinearly modulated based on empirical class frequency and current prediction reliability.

[0086]

[0087] in, The intensity hyperparameter is dynamically logarithmically calibrated. For categories with high historical frequency and low current confidence, the corresponding balance function value is smaller, thus providing stronger suppression of the original score of that category; for categories with low historical frequency, the suppression is relatively weaker, thus preventing tail categories from being further suppressed.

[0088] Furthermore, the original category scores are calibrated using the dynamic balancing function to obtain the calibrated category scores:

[0089]

[0090] Based on calibrated category score This allows for the further acquisition of the post-calibration probability distribution and the identification of calibration pseudo-labels. ,in, The category with the highest calibrated category score is assigned to this category. To form a closed-loop online adaptation mechanism, incremental updates are performed on historical statistics after the current sample is processed. Specifically, the category count is updated by incrementing the cumulative count of the corresponding calibrated pseudo-label category by one, and the average confidence score is recursively updated based on the current calibration confidence score to ensure that the statistical prior dynamically evolves with the test flow. Through step S2, lightweight online logarithmic calibration can be performed on the data arriving in the test flow sample by sample without updating the backbone parameters of the image encoder and text encoder, thereby reducing high-frequency category bias and providing a more reliable basis for the controlled introduction of subsequent boundary samples.

[0091] S3. Use the Consistency-Guided Exploratory Caching Module (CGEC) to filter the current test samples and write the samples that meet the caching admission criteria into the corresponding category cache. The samples that meet the caching admission criteria include at least high-confidence samples and boundary samples whose pseudo-labels change before and after calibration and simultaneously meet the semantic consistency constraint and temporal consistency constraint. And construct the category visual prototype.

[0092] like Figure 3 As shown, traditional cache-based testing adaptation methods typically retain only high-confidence samples. While this effectively controls noise, it results in a narrow cache coverage area, making it difficult to reflect information from boundary samples and low-confidence regions within the target domain. This implementation not only retains high-confidence samples but also explores samples whose predictions have flipped before and after dynamic logarithmic calibration, thereby improving the cache's coverage of the true distribution of the target domain.

[0093] Specifically, a flip test is performed to determine whether the predicted label of the sample changes before and after the logical adjustment. When the above conditions are not met, the current sample is treated as a regular high-confidence candidate sample, and its post-calibration confidence and uncertainty determine whether to write it to the cache. When the above conditions are met, it indicates that the current sample's category has changed before and after dynamic logarithmic calibration, suggesting that the sample is near the category competition boundary. For boundary samples that have flipped, the semantic consistency filter (SCF) is triggered. Specifically, This represents the text features corresponding to the original predicted category. This represents the text features corresponding to the predicted category after calibration. For example... Figure 5 As shown, the initial prediction entropy of the sample is calculated using the cosine similarity between the two in the text embedding space. Down-modulation:

[0094]

[0095] in, This is a scaling parameter. If the semantics of the categories before and after the flip are similar, the cosine similarity between the corresponding text features is high, and the sample entropy value decreases accordingly, making it easier for the sample to be cached and retained as a high-value exploration sample.

[0096] Meanwhile, to prevent feature drift during the adaptation process, all samples stored in the cache are evaluated based on a Time Consistency Filter (TCF). For example... Figure 4 As shown, for historical image features in the cache The dot product response between the text features before and after evolution is detected. This indicates the text features of the corresponding predicted category when the cached sample is initially written to the cache; This indicates the text features of the corresponding predicted category of the cached sample at the current time. This represents the historical image features of the cached samples. If...

[0097]

[0098] This indicates that the matching degree between the historical image features and the current text features is lower than the matching degree between the historical image features and the corresponding text features when they were written to the cache. This suggests a deviation between the sample and the current model's adaptation direction, i.e., a temporal inconsistency has occurred. In this case, increasing its entropy value is used to penalize and force its elimination.

[0099]

[0100] in, For the current time step, This refers to the time step when the sample is initially stored in the cache. The parameters are used to control the decay. After semantic and temporal dual entropy modulation, the samples are sent to a priority queue with a fixed capacity, and a low-entropy priority strategy is used for sample insertion and high-entropy sample replacement.

[0101] In a preferred embodiment, each category maintains a fixed capacity. A priority queue cache is used. Cache elements must include at least image features, sample evaluation values, and the write time. When the target category cache is not full, candidate samples are written directly. When the cache is full, replacement is performed only if the current candidate sample is better than the worst sample in the cache. Let the current candidate sample's corresponding category cache be... Then, if the following conditions are met: If the condition is met, write the sample directly to the cache. Otherwise, select the sample with the largest modulation entropy value from the cache. Replacement is performed only if the modulation entropy value of the current candidate sample is less than that of the worst sample.

[0102] After the cache update is complete, visual features in the same category of cache are aggregated to form a category-specific visual prototype. For example... Figure 1 As shown, Indicate category The corresponding cached sample set, This indicates the number of samples in the cache set, the th... The visual prototype of each category can be represented as:

[0103]

[0104] To enhance cross-modal alignment capabilities within the target domain, learnable text residuals can also be introduced for each category. and learnable visual residuals The text prototype and visual prototype are then lightly modified. Preferably, the modified text prototype and visual prototype can be represented as the sum of the prototype and the corresponding residual, followed by normalization.

[0105] Through the above S3 steps, not only can controlled exploratory cache updates be achieved, but also a visual prototype that is more consistent with the target domain distribution can be built based on the cache.

[0106] S4. Perform joint prediction and test-time optimization based on visual prototype and calibrated semantic scores, and output the final classification result.

[0107] like Figure 1 , Figure 6 As shown, during the inference phase, the final category score preferably consists of two parts: the first part is the semantic score after dynamic logarithmic calibration, used to alleviate the category bias in the original zero-shot inference; the second part is the similarity score between the visual features of the current test sample and the visual prototypes of each category, used to supplement the discrimination using historical cached information from the target domain. By fusing the two, the final joint prediction result for the current sample is obtained.

[0108] To integrate semantic information with cached visual distribution information, an affinity modulation function is introduced. The inner product of image features and visual prototypes is processed to map the similarity between the current image features and the category visual prototypes into a visual aid discrimination signal. This signal is then added to the calibrated semantic logical value from step S2 to calculate the final prediction score. :

[0109]

[0110] Therefore, the final classification result can be determined:

[0111]

[0112] During the optimization phase at test time, to further improve prediction stability and cross-modal consistency in the target domain, a joint optimization objective is constructed, updating only the lightweight residual parameters and not the backbone parameters of the image encoder and text encoder. Preferably, the overall joint loss function can be expressed as:

[0113]

[0114] in, This is the loss balance coefficient. Confidence-aware calibration loss. Used to constrain the uncertainty of joint prediction results in the target domain, enabling the model to gradually form a more stable and lower-entropy prediction distribution; symmetric cross-modal alignment loss. This is used to promote consistent alignment between textual and visual prototypes in a shared embedding space, thereby enhancing cross-modal matching relationships and improving the model's classification robustness and generalization ability in target domain distribution offset scenarios.

[0115] The confidence-aware calibration loss minimizes entropy using the following formula:

[0116] Cross-modal alignment loss Committed to bridging the gap between adaptive text features and visual prototypes:

[0117]

[0118] Finally, through backpropagation of the joint objective function, the lightweight residual parameters attached to the model are driven by the unsupervised signal from the test data stream. and Iterative updates are performed to complete the adaptive learning process for the current test sample.

[0119] Finally, through backpropagation of the joint objective function, the lightweight residual parameters are iteratively updated using the unsupervised signal from the test data stream. Specifically, after the joint prediction for the current test sample is completed, the joint loss is first calculated based on the joint prediction results for that sample. And minimize the joint loss, then apply the text residual parameters separately. and visual residual parameters The loss gradient is calculated and updated using gradient descent or an adaptive optimization algorithm, while the backbone parameters of the image encoder and text encoder remain frozen. The update process can be represented as follows:

[0120]

[0121]

[0122] in, The learning rate is represented by [value]. After the update, the new text residual parameters and visual residual parameters are superimposed on the corresponding text prototype and visual prototype, respectively, for joint prediction of subsequent test samples and adaptive optimization during testing. In this way, the model can continuously adjust the alignment between cross-modal representations using the unlabeled test flow of the target domain without changing the parameters of the pre-trained backbone network, thereby gradually improving its adaptability to the distribution of the target domain.

[0123] In summary, this invention suppresses high-frequency class bias in the test stream through a dynamic logarithmic calibration mechanism, introduces boundary samples with high adaptation value while controlling noise risk through a consistency-guided exploration caching mechanism, and further combines class visual prototypes with calibration semantic scores to achieve joint prediction and lightweight test-time optimization. Therefore, even in real-world deployment scenarios where source domain data is inaccessible, target domain is unlabeled, and the environment is constantly changing, this invention can still effectively improve the stability, robustness, and cross-domain adaptability of visual language models, demonstrating significant engineering application value.

[0124] Finally, it should be noted that the above specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention; any equivalent substitutions, improvements or modifications made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0125] Example 2: Project configuration example.

[0126] The above method operates in a single-sample streaming test mode, performing prediction, calibration, cache update, and parameter optimization sequentially for each sample in the test stream; furthermore, it generates one or more enhanced views for each test sample, through multi-view... Figure 1 The method improves the stability of pseudo-labels and the robustness of test-time adaptation. It can be plugged into the test-time adaptive framework of retrieval-based visual language models or the test-time adaptive framework of cached visual language models. Without changing the offline training process of the model backbone, it improves the generalization ability across datasets, class balance and classification robustness in natural distribution offset scenarios.

[0127] In one specific implementation, a pre-trained CLIP model can be used as the visual language model and image encoder. ResNet-50, ViT-B / 16, or similar visual backbone networks can be used for the text encoder. To support this, a single-sample streaming approach is used to process the target domain samples sequentially during the testing phase, and multiple enhanced views are constructed for each sample to improve robustness.

[0128] For example, random cropping, scaling, flipping, or color perturbation can be performed on each test sample to generate multiple augmented views, and the consistency of prediction results under these multiple views can be used to improve the stability of pseudo-labels. For multi-augmented-view scenarios, the prediction probabilities of different augmented views can also be averaged first, and then the average result can be used to calculate... , , , as well as This reduces the propagation of errors caused by noise in a single view.

[0129] Regarding cache configuration, the cache capacity for each category It can be set to 3 to 5 to balance online efficiency and sample representativeness. Semantic consistency modulation parameter. Dynamic logarithmic calibration intensity parameters Time consistency decay parameter Loss balance coefficient and temperature parameters The parameters can be set according to the target task, model backbone, and test scenario.

[0130] In this embodiment, the dynamic logarithmic calibration module mainly introduces additional category statistics updates and logarithmic adjustment operations, while the consistency-guided exploration cache module mainly introduces sample entropy modulation and priority queue admission judgment. Therefore, the overall additional computational overhead is relatively small, making it suitable for deployment in online continuous inference scenarios. Experiments show that in cross-dataset generalization and natural distribution offset scenarios, this invention has higher average accuracy, more stable category-level performance, and stronger robustness compared to existing methods that only use high-confidence sample caching.

[0131] Example 3: Extended deployment method.

[0132] This invention is not only applicable to pure zero-sample visual language models, but can also be integrated as a plug-in module into existing retrieval-based test-time adaptive methods, cache-based test-time adaptive methods, or prototype-update test-time adaptive methods. Integration requires no modification to the original image encoder, text encoder, or offline pre-training process. Only a dynamic logarithmic calibration step needs to be added after the original prediction, and consistency-guided exploration of cache replacement or parallel connection of existing cache update modules is required to significantly improve the system's ability to utilize boundary samples and its coverage of the target domain distribution.

[0133] Furthermore, when this invention is integrated into an existing visual language model testing framework in a modular manner, the basic loss, prototype update process, or retrieval aggregation mechanism in the original framework can be retained, only the category semantic score is replaced with the calibrated one. Furthermore, the update strategy for cached samples is replaced with an exploratory admission mechanism based on flip detection, semantic consistency, and temporal consistency, thereby improving the system's ability to model difficult, weak, and boundary samples without disrupting the overall structure of the original framework.

[0134] In summary, this invention, through a combination of "dynamic calibration, controlled exploration, consistency maintenance, and cross-modal joint optimization," simultaneously solves key problems in adaptive testing of visual language models, such as class bias, insufficient cache representativeness, and inefficient use of boundary samples. It can achieve stable, reliable, and efficient online adaptation in an unlabeled streaming testing environment of the target domain, and has good prospects for engineering applications.

[0135] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention; any equivalent substitutions, improvements or modifications made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for testing and adapting a visual language model based on logarithmic calibration and consistency caching, characterized in that, Includes the following steps: S1. Obtain a continuous unlabeled test image data stream, extract image features and text prototype features of each category using an image encoder and a text encoder, calculate the similarity between the current image features and the text prototypes of each category, obtain the original semantic category logarithm, and generate the original classification probability, original pseudo-label, and uncertainty accordingly. S2. During the online inference process, maintain two types of statistical information for each category: category count and historical average confidence. Construct empirical category prior probability based on historical category count. Construct category balance factor based on category prediction frequency statistics and historical average confidence statistics in historical test stream. Dynamically recalibrate the original category score based on empirical category prior probability and category balance factor to obtain calibrated category score and calibrated pseudo-label. S3. Use the consistency-guided exploratory caching module to filter the current test samples and write the samples that meet the caching admission criteria into the corresponding category cache. The samples that meet the cache admission criteria include at least high-confidence samples and boundary samples whose pseudo-labels change before and after calibration and simultaneously satisfy semantic consistency constraints and temporal consistency constraints; and a category visual prototype is constructed based on the visual features in the same category of cache. S4. Perform joint prediction and test-time optimization based on the visual prototype and calibrated category scores, and output the final classification result.

2. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 1, characterized in that, S1 is specifically as follows: For any unlabeled image in the input data stream during the testing phase Visual features are extracted using a pre-trained visual encoder with frozen parameters. Then, normalization processing is performed to obtain image features. Meanwhile, regarding the inclusion The task involves classifying the data into categories c and constructing text prompts for each category. The data is then input into a pre-trained text encoder with frozen parameters to extract text prototype features for each category. ; Subsequently, the inner product between image features and various text prototype features is calculated to obtain the basic raw category scores. Then, the similarity between the current image features and the text prototypes of each category is calculated to obtain the original semantic category logarithm, and the original zero-sample classification probability is generated accordingly. in, Here, y represents the temperature parameter, y represents the label, and c represents the category index. For any of the C categories, For category The original category score, The original classification probabilities are given; based on the above probability distribution, the category corresponding to the maximum probability is selected as the initial pseudo-label for the image. And use Shannon entropy to calculate the entropy value of the current original classification probability. This is used to characterize the uncertainty in prediction.

3. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 2, characterized in that, S2 is specifically as follows: During the online inference process, two types of statistical information are maintained for each category: category count and historical average confidence. The category count is used to reflect the cumulative frequency of the corresponding category being predicted as a false label in the test stream, and the historical average confidence is used to reflect the average reliability of the prediction results of that category over history. For the current test sample, the prediction confidence level corresponding to the original pseudo-label is determined based on the original classification probability obtained in S1. And construct the empirical class prior probability based on historical class counts. : in, This represents the category count variable with category c. Indicates category as The category count variable; For the current test sample, calculate the deviation between the predicted confidence level corresponding to the current original pseudo-label and the historical average confidence level of this category. ; Based on this, a category-related dynamic equilibrium function is constructed. : in, The dynamic logarithmic calibration intensity hyperparameter is used; the dynamic balancing function calibrates the original category scores to obtain the calibrated category scores. Based on the calibrated category scores, the calibrated probability distribution is further obtained, and the calibration pseudo-labels are identified. ,in, The category with the highest category score after calibration; After the current sample is processed, the historical statistics are updated incrementally: the category count is updated by incrementing the cumulative count of the corresponding calibration pseudo-label category by one, and the average confidence is updated recursively based on the current calibration confidence.

4. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 3, characterized in that, S3 is specifically as follows: Perform a flip test to determine whether the predicted label of the sample changes before and after the logical adjustment. When the above conditions are not met, the current sample is regarded as a regular high-confidence candidate sample, and whether to write it into the cache is determined based on its post-calibration confidence and uncertainty. When the above conditions are met, it indicates that the category of the current sample has changed before and after dynamic logarithmic calibration, triggering the semantic consistency filter. Semantic consistency filter: The initial prediction entropy value of the sample is down-modulated by using the cosine similarity between the text features corresponding to the original predicted category and the text features corresponding to the calibrated predicted category in the text embedding space. Then, all samples stored in the cache are evaluated based on the time consistency filter; specifically, let... This indicates the text features of the corresponding predicted category when the cached sample is initially written to the cache; This indicates the text features of the corresponding predicted category of the cached sample at the current time. like Then, by increasing its entropy value, a penalty is imposed to force its elimination: in, For the current time step, This refers to the time step when the sample is initially stored in the cache. Parameters used to control attenuation; After semantic and temporal entropy modulation, the samples are sent to a priority queue with a fixed capacity, and a low-entropy priority strategy is used to insert samples and replace high-entropy samples. After the cache update is complete, visual features in the same category of cache are aggregated to form a category visual prototype. The visual prototypes of each category are represented as follows: in, Indicate category The corresponding cached sample set, This indicates the number of samples in the cache set.

5. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 4, characterized in that, S4 is specifically as follows: An affinity modulation function is introduced to process the inner product of image features and visual prototypes, which is then added to the calibrated category score to calculate the final prediction score. Therefore, the final classification result can be determined: During the optimization phase at test time, a joint optimization objective is constructed that updates only the lightweight residual parameters without updating the parameters of the image encoder and text encoder. The overall joint loss function includes confidence-aware calibration loss and symmetric cross-modal alignment loss.

6. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 5, characterized in that, The deviation is calculated as follows: in, This indicates that the current test sample was predicted as the original pseudo-label by the original classifier. The probability value, This represents the historical average confidence level corresponding to the original pseudo-label category; when When the prediction confidence score is lower than the historical average for that category, it indicates that the category may be over-predicted, or the current sample is near the category decision boundary. When the confidence level is 1, it indicates that the current prediction confidence level is basically consistent with or higher than the historical level of the category, suggesting that the current prediction result is relatively more reliable.

7. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 6, characterized in that, The lightweight residual parameters include learnable text residuals. and learnable visual residuals Lightweight modifications were made to both the text prototype and the visual prototype.

8. The visual language model testing adaptation method based on logarithmic calibration and consistency caching according to claim 7, characterized in that, The confidence-perceived calibration loss Entropy is minimized by the following formula: The cross-modal alignment loss Committed to bridging the gap between adaptive text features and visual prototypes: Finally, through backpropagation of the overall joint loss function, the lightweight residual parameters attached to the model are driven by the unsupervised signal of the test data stream for iterative updates, thus completing the adaptive learning process for the current test sample.

Images