Visual language model zero-sample distribution out-of-distribution detection methods, media and computer equipment

Abstract

This invention discloses a zero-sample out-of-distribution detection method, medium, and computer device for visual-language models. The main steps of the method include: optimal transmission OT probability alignment, adaptive pseudo-label generation, lightweight detector training, and self-reinforcing feedback loop. The detection method based on self-reinforcing optimal transmission provided by this invention, without relying on any label data or fine-tuning the model's backbone network, significantly improves the out-of-distribution detection performance of visual-language models on target tasks by aligning posterior probabilities through optimal transmission and correcting the alignment process using a feedback loop of a lightweight detector.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer vision, and more specifically to a method, medium, and computer device for detecting zero-sample distribution out-of-distribution visual language models. Background Technology

[0002] Pre-trained vision-language models (VLMs, such as CLIP) are widely used due to their powerful zero-shot classification capabilities. In real-world open-world deployments, models not only need to identify known categories (In-Distribution, ID) but also need to be able to detect and reject unknown out-of-distribution (OOD) samples.

[0003] Existing OOD detection methods (such as the Maximum Softmax Probability Thresholding method, MSP) perform reasonably well on supervised learning models, but their performance is extremely poor when directly applied to zero-shot Virtual Learning Models. The main reason is:

[0004] 1. Feature misalignment: VLM is trained based on image-text matching rather than discriminative training for a specific task, which leads to blurred inter-class boundaries and severe overlap in the confidence distribution of ID and OOD samples.

[0005] 2. Distribution skew: The distribution of downstream test data often deviates from the pre-trained distribution of VLM, making the original posterior probability unreliable.

[0006] 3. Limitations of existing Test-Time Adaptive (TTA) methods: Existing TTA methods (such as Tent and TPT) mainly focus on improving classification accuracy, often neglecting the calibration of prediction confidence, and may even lead to the model making high-confidence erroneous predictions for OOD samples. Summary of the Invention

[0007] The present invention provides a method, medium, and computer equipment for high-precision out-of-distribution detection of visual language models with zero sample distribution by means of a self-enhancing mechanism that can calibrate the output of VLM in real time during the testing phase, thereby solving at least one of the above-mentioned technical problems.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A method for detecting out-of-distribution zero-samples in a visual language model includes the following steps:

[0010] S1, Model Input: The pre-trained visual-language model (VLM) is used to receive the unlabeled image data stream during the testing phase. This unlabeled image data stream is input in mini-batch form.

[0011] S2. Feature Extraction: Use the frozen pre-trained visual-language model (VLM) to extract image features and text prompt features from the unlabeled image data stream;

[0012] S3, Optimal Transmission OT Probability Alignment: Construct a cost matrix based on the original output of the pre-trained visual-language model (VLM), and use the entropy regularization optimal transmission mechanism to calculate the posterior probability of the pre-trained visual-language model (VLM) after prediction alignment, so as to alleviate distribution mismatch.

[0013] S4. Adaptive pseudo-label generation: Calculate the out-of-distribution (OOD) sample score based on the aligned posterior probability, and dynamically determine the threshold using a sliding window mechanism to generate binary pseudo-labels;

[0014] S5. Lightweight Detector Training: A lightweight detector for out-of-distribution (OOD) samples is trained using binary pseudo-labels. This lightweight detector directly outputs the probability of binary pseudo-labels based on image features and text prompt features.

[0015] S6. Self-reinforcing feedback loop: Feedback the out-of-distribution OOD sample probability output by the trained lightweight detector back to the optimal transmission OT process, correct the cost matrix of the optimal transmission OT, and introduce open set priors to form a closed-loop optimization.

[0016] S7. Model Output: Based on the final trained lightweight detector, output the out-of-distribution OOD sample score to determine whether the current detected sample is an out-of-distribution OOD sample of unknown category.

[0017] Furthermore, the pre-trained visual-language model (VLM) in S1 includes four operating phases, namely optimal transport OT alignment, adaptive pseudo-label generation, lightweight detector training, and self-reinforcing feedback loop.

[0018] Furthermore, in the probability alignment process based on optimal transmission OT, S3 selects a multimodal pre-trained neural network model CLIP from the pre-trained visual-language model VLM. For a batch of test image samples x, it calculates the similarity between x and the text prototypes of the ID sample categories within the distribution of all known categories, thereby obtaining the posterior probability of the original multimodal pre-trained neural network model CLIP. This forces the pre-trained visual-language model (VLM) to predict in accordance with marginal distribution constraints, thus correcting the probability bias caused by distribution shift.

[0019] Furthermore, in S3, optimal transmission OT probability alignment is performed to obtain the posterior probability of the aligned multimodal pre-trained neural network model CLIP. The process includes:

[0020] S3.1 Construct the cost matrix M, ;

[0021] S3.2, Set uniform sample weights and category prior To encourage sample class balance;

[0022] S3.3 Solving the optimal transmission problem based on the entropy regularization optimal transmission mechanism to obtain the transmission scheme. The calculation expression is:

[0023]

[0024] in, Indicates a feasible transmission plan. This represents the plan that minimizes the total transmission cost among all feasible transmission plans. Represents the Frobenius inner product. represents the regularization coefficient, used to control the influence of the KL divergence term on the optimization results. KL divergence is used to measure the distribution difference, and T represents the conjugate transpose.

[0025] S3.4, Transmission Scheme Normalization is performed to obtain the aligned posterior probabilities. .

[0026] Furthermore, the S4 process of adaptive thresholding and pseudo-label generation based on a sliding window includes:

[0027] S4.1 Calculate the out-of-distribution OOD sample score based on optimal transmission OT. :

[0028]

[0029] Where x represents the test image sample, This represents the posterior probability after alignment;

[0030] S4.2 Store the sample scores obtained in the most recent batches in a first-in-first-out queue to maintain a sliding window containing the scores of the most recent historical samples. ;

[0031] S4.3, in the sliding window Find the optimal segmentation threshold within the context The expression is as follows: This minimizes the sum of the within-class variances of the two groups of samples divided by the threshold.

[0032]

[0033] in, This represents the optimal threshold that minimizes the sum of the variances of the ID group and the OOD group among all possible thresholds λ. Indicates the current threshold Below, the fractional variances of the samples divided into within-distribution groups, Indicates the current threshold The fractional variance of the samples divided into out-of-distribution groups;

[0034] S4.4 Utilizing the optimal segmentation threshold Generate pseudo-labels for the current batch of samples. Where 0 represents an in-distribution ID sample of a known category, and 1 represents an out-of-distribution OOD sample of an unknown category.

[0035] Furthermore, in S5, the lightweight detector training process includes:

[0036] S5.1, Construct a lightweight detector Its input is frozen image features. ;

[0037] S5.2, Based on generated pseudo-tags The parameters of this lightweight detector are updated online using the cross-entropy loss function;

[0038] S5.3 The output of the lightweight detector is the probability of the obtained out-of-distribution (OOD) samples. and the probability of ID samples within the distribution .

[0039] Furthermore, in S6, the self-reinforcing feedback loop process includes:

[0040] S6.1 Construct the corrected cost matrix based on the out-of-distribution OOD sample probabilities output by the trained lightweight detector. The corresponding probability distribution The CLIP original distribution and open set prior of the multimodal pre-trained neural network model. The weighted mixture is expressed as:

[0041]

[0042] in, If it is a uniform distribution or other prior distribution, The probability of an out-of-distribution (OOD) sample. The original posterior probability of CLIP for the multimodal pre-trained neural network model;

[0043] S6.2 Logical Judgment: If the lightweight detector believes that the sample is an out-of-distribution OOD sample more likely than the historical experience probability value, then the uniform prior is referenced when performing optimal transmission OT alignment; otherwise, if the lightweight detector believes that the sample is an out-of-distribution OOD sample less likely than the historical experience probability value, then the original prediction of the multimodal pre-trained neural network model CLIP is referenced when performing optimal transmission OT alignment.

[0044] S6.3, Using the modified cost matrix Re-execute the optimal transmission problem in the optimal transmission OT probability alignment, update the pseudo-labels, and continue to optimize the lightweight detector in the next batch of samples.

[0045] A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described visual language model zero-sample out-of-distribution detection method.

[0046] A computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the above-described visual language model zero-sample out-of-distribution detection method.

[0047] The beneficial effects of this invention are reflected in:

[0048] 1. High-precision out-of-distribution (OOD) sample detection: On multiple standard benchmark datasets (such as ImageNet, iNaturalist, etc.), this method significantly outperforms existing Zero-Shot and TTA methods in multiple evaluation metrics.

[0049] 2. No labeled data required: This method enables pure unsupervised test-time adaptation without requiring any manually labeled target domain data.

[0050] 3. High computational efficiency: This method only requires training a very lightweight linear layer, while the backbone network (such as ViT-B / 16) remains frozen, resulting in extremely low computational overhead, making it suitable for deployment on edge devices.

[0051] 4. Strong robustness: This method effectively alleviates the distribution mismatch problem between the pre-trained model and the downstream task through optimal transmission OT alignment and self-enhancement mechanism. It not only improves the detection capability of out-of-distribution OOD samples of unknown categories, but also maintains the accuracy of classification of in-distribution ID samples of known categories. Attached Figure Description

[0052] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0053] Figure 1 This is a schematic diagram of the overall process of the visual language model zero-sample distribution out-of-distribution detection method according to an embodiment of the present invention.

[0054] Figure 2 This is a structural block diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] It should be noted that the meaning of "and / or" throughout the text includes three parallel solutions. Taking "A and / or B" as an example, it includes solution A, solution B, or a solution that simultaneously satisfies A and B. Furthermore, "multiple" refers to two or more. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0057] See Figure 1 This invention provides a method for detecting out-of-distribution zero-samples in a visual language model, comprising the following steps:

[0058] S1, Model Input: The pre-trained visual-language model (VLM) is used to receive the unlabeled image data stream during the testing phase. This unlabeled image data stream is input in mini-batch form.

[0059] S2. Feature Extraction: Use the frozen pre-trained visual-language model (VLM) to extract image features and text prompt features from the unlabeled image data stream;

[0060] S3, Optimal Transmission OT Probability Alignment: Construct a cost matrix based on the original output of the pre-trained visual-language model (VLM), and use the entropy regularization optimal transmission mechanism to calculate the posterior probability of the pre-trained visual-language model (VLM) after prediction alignment, so as to alleviate distribution mismatch.

[0061] S4. Adaptive pseudo-label generation: Calculate the out-of-distribution (OOD) sample score based on the aligned posterior probability, and dynamically determine the threshold using a sliding window mechanism to generate binary pseudo-labels;

[0062] S5. Lightweight Detector Training: A lightweight detector for out-of-distribution (OOD) samples is trained using binary pseudo-labels. This lightweight detector directly outputs the probability of binary pseudo-labels based on image features and text prompt features.

[0063] S6. Self-reinforcing feedback loop: Feedback the out-of-distribution OOD sample probability output by the trained lightweight detector back to the optimal transmission OT process, correct the cost matrix of the optimal transmission OT, and introduce open set priors to form a closed-loop optimization.

[0064] S7. Model Output: Based on the final trained lightweight detector, output the out-of-distribution OOD sample score to determine whether the current detected sample is an out-of-distribution OOD sample of unknown category.

[0065] In this embodiment, the pre-trained visual-language model (VLM) in S1 includes four operating phases, namely optimal transport OT alignment, adaptive pseudo-label generation, lightweight detector training, and self-reinforcing feedback loop.

[0066] In this embodiment, during the probability alignment process based on optimal transmission OT, step S3 selects a multimodal pre-trained neural network model CLIP from the pre-trained visual-language model VLM. For a batch of test image samples x, it calculates the similarity between x and the text prototypes of the ID sample categories within the distribution of all known categories, thereby obtaining the posterior probability of the original multimodal pre-trained neural network model CLIP. This forces the pre-trained visual-language model (VLM) to predict in accordance with marginal distribution constraints, thus correcting the probability bias caused by distribution shift.

[0067] CLIP, a multimodal pre-trained neural network model, is the first large-scale unsupervised visual language pre-training model proposed by OpenAI. It has achieved the universal alignment capability of "directly matching text descriptions with image features" for the first time. Its core structure of dual encoders (image tower + text tower) + contrastive learning has become the design blueprint for all subsequent visual language models (such as ALBEF, BLIP, and FLAVA), and is the "basic paradigm" in this field. Therefore, this invention selected this model.

[0068] In this embodiment, in step S3, optimal transmission OT probability alignment is performed to obtain the posterior probability of the aligned multimodal pre-trained neural network model CLIP. The process includes:

[0069] S3.1 Construct the cost matrix M, ;

[0070] S3.2, Set uniform sample weights and category prior To encourage sample class balance;

[0071] S3.3 Solving the optimal transmission problem based on the entropy regularization optimal transmission mechanism to obtain the transmission scheme. The calculation expression is:

[0072]

[0073] in, Indicates a feasible transmission plan. This represents the plan that minimizes the total transmission cost among all feasible transmission plans. Represents the Frobenius inner product. represents the regularization coefficient, used to control the influence of the KL divergence term on the optimization results. KL divergence is used to measure the distribution difference, and T represents the conjugate transpose.

[0074] S3.4, Transmission Scheme Normalization is performed to obtain the aligned posterior probabilities. .

[0075] In this embodiment, the S4 process of adaptive thresholding and pseudo-label generation based on a sliding window includes:

[0076] S4.1 Calculate the out-of-distribution OOD sample score based on optimal transmission OT. :

[0077]

[0078] Where x represents the test image sample, This represents the posterior probability after alignment;

[0079] S4.2 Store the sample scores obtained in the most recent batches in a first-in-first-out queue to maintain a sliding window containing the scores of the most recent historical samples. ;

[0080] S4.3, in the sliding window Find the optimal segmentation threshold within the context The expression is as follows: This minimizes the sum of the within-class variances of the two groups of samples divided by the threshold.

[0081]

[0082] in, This represents the optimal threshold that minimizes the sum of the variances of the ID group and the OOD group among all possible thresholds λ. Indicates the current threshold Below, the fractional variances of the samples divided into within-distribution groups, Indicates the current threshold The fractional variance of the samples divided into out-of-distribution groups;

[0083] S4.4 Utilizing the optimal segmentation threshold Generate pseudo-labels for the current batch of samples. Where 0 represents an in-distribution ID sample of a known category, and 1 represents an out-of-distribution OOD sample of an unknown category.

[0084] In this embodiment, the lightweight detector training process in step S5 includes:

[0085] S5.1, Construct a lightweight detector For example, a linear layer classifier whose input is frozen image features. ;

[0086] S5.2, Based on generated pseudo-tags The parameters of this lightweight detector are updated online using the cross-entropy loss function;

[0087] S5.3 The output of the lightweight detector is the probability of the obtained out-of-distribution (OOD) samples. and the probability of ID samples within the distribution .

[0088] In this embodiment, in step S6, to further improve the alignment effect, the prediction results of the lightweight detector are used to correct the cost matrix of the optimal transmission OT. Specifically, the self-reinforcing feedback loop process includes:

[0089] S6.1 Construct the corrected cost matrix based on the out-of-distribution OOD sample probabilities output by the trained lightweight detector. The corresponding probability distribution The CLIP original distribution and open set prior of the multimodal pre-trained neural network model. The weighted mixture is expressed as:

[0090]

[0091] in, If it is a uniform distribution or other prior distribution, The probability of an out-of-distribution (OOD) sample. The original posterior probability of CLIP for the multimodal pre-trained neural network model;

[0092] S6.2 Logical Judgment: If the lightweight detector believes that the sample is an out-of-distribution OOD sample more likely than the historical experience probability value, then the uniform prior is referenced when performing optimal transmission OT alignment; otherwise, if the lightweight detector believes that the sample is an out-of-distribution OOD sample less likely than the historical experience probability value, then the original prediction of the multimodal pre-trained neural network model CLIP is referenced when performing optimal transmission OT alignment.

[0093] S6.3, Using the modified cost matrix Re-execute the optimal transmission problem in the optimal transmission OT probability alignment, update the pseudo-labels, and continue to optimize the lightweight detector in the next batch of samples.

[0094] As can be seen from the overall method flow, in this method, the flow of unlabeled image data for testing is as follows: CLIP features of multimodal pre-trained neural network model → optimal transmission OT alignment → pseudo-labels → lightweight detector training → lightweight detector output feedback to the optimal transmission OT cost matrix, thereby optimizing the loop.

[0095] To further verify the feasibility and superiority of this method, the present invention is illustrated with the following actual experimental case.

[0096] I. Experimental Setup

[0097] 1. Model Setup: The multimodal pre-trained neural network model CLIP (ViT-B / 16) is used as the backbone network, and the parameters are frozen.

[0098] 2. Lightweight detector structure: A simple fully connected layer that maps the feature dimension to 2D (ID / OOD).

[0099] 3. Hyperparameter settings:

[0100] Sliding window size: Set to the 3 most recent batches;

[0101] Optimal transmission OT regularization coefficient : Set according to the validation set or empirical values;

[0102] Optimizer: Update detector parameters using the Adam optimizer, with the learning rate set to... .

[0103] 4. Reasoning process:

[0104] For newly arrived samples, a lightweight detector first determines the probability that they are out-of-distribution (OOD) samples. ,if (threshold) If the sample is determined by the sliding window, it is rejected (considered an out-of-distribution (OOD) sample of unknown class); otherwise, the sample is accepted and the original text matching result from CLIP is used as the classification prediction.

[0105] II. Dataset Configuration

[0106] This invention is evaluated under the Test Time Adaptation (TTA) task for out-of-distribution (OOD) detection.

[0107] Following existing research paradigms, the following datasets are used as in-distribution (ID) datasets in sequence: Caltech Birds Dataset (CUB-200-2011), Stanford Cars Dataset, Food-101 Dataset, Oxford-IIIT Pet Dataset, ImageNet Dataset, and ImageNet derivative datasets—Sketch (ImageNet-Sketch / ImageNet-K), Adversarial (ImageNet-A), Variant (ImageNet-V2), and Render (ImageNet-R).

[0108] The out-of-distribution (OOD) dataset uses four standard benchmarks: iNaturalist, Large Scale Scene Understanding Dataset (LSUN), Textures, and Places.

[0109] The core evaluation metrics for the experiment include: clean ID sample accuracy. OOD sample accuracy and the harmonic mean of the two. The above metrics collectively reveal the performance trade-offs of closed sets that arise during model adaptation. Unless otherwise stated, all comparative methods employ completely consistent preprocessing procedures, batch settings, and evaluation codes to ensure fair comparability of experiments.

[0110] Unless otherwise specified, the backbone network used in this invention is the CLIP model, with a visual Transformer-B / 16 (ViT-B / 16) image encoder and a Transformer structure for text encoder. During the adaptation phase of testing, the parameters of both encoders are frozen. The model adaptation process optimizes only a lightweight linear classifier head used to output the OOD detection score, and the optimizer is Adam. To improve the stability of the update process, this invention injects small-amplitude random perturbations into the CLIP model input at a fixed frequency. This design, while keeping the backbone network frozen, can directly address zero-sample OOD detection tasks, exhibiting not only extremely low computational overhead but also stable performance under data distribution shift scenarios.

[0111] Furthermore, this invention selects the following representative methods as baseline models for comparison:

[0112] 1. Zero-shot CLIP (ZS-CLIP, a zero-shot inference method without adaptation).

[0113] 2. Adaptation method (TENT) during entropy minimization testing;

[0114] 3. A self-training test-time adaptation method for zero-shot visual language models (SoTTA).

[0115] 4. During testing, the system will provide optimization methods (TPT).

[0116] 5. Adaptive noise detection method combined with online threshold adjustment (AdaND).

[0117] III. Experimental Results

[0118] Experimental results for fine-grained ID datasets (CUB-200-2011, Stanford Cars, Food-101, Oxford-IIIT Pet) and ImageNet series ID datasets are summarized in Tables 1, 1 (continued), 2 and 2 (continued) below.

[0119] Analysis of Tables 1, 1 (continued), 2, and 2 (continued) shows that, across all nine ID datasets, compared to the current best-performing test-time adapted baseline model (AdaND), the method proposed in this invention improves the accuracy of the harmonic mean while achieving... and The synchronous growth. The average performance improvement across all ID datasets is: The absolute value increased by 2.53 percentage points. An increase of 1.8 percentage points. An improvement of 4.1 percentage points. The only exception occurred on the Food-101 dataset, where the average for this method was [missing information]. The difference compared to AdaND is only 0.1–0.4. This phenomenon stems from the performance saturation effect: for CLIP-style zero-shot recognition tasks, the Food-101 dataset is relatively easy (its... (Highest across all datasets), the model's performance is nearing its limit, thus room for improvement is limited. Methods relying solely on similarity calculations are often vulnerable to severe distribution shifts, and such methods sometimes sacrifice... However, this approach, while costing significant resources, failed to achieve a corresponding improvement in OOD detection performance; and methods based solely on prompt-based tuning, while able to maintain performance relatively well... However, the correction effect in model calibration is unsatisfactory. In contrast, the method proposed in this invention combines the self-enhancing mechanism of optimal transfer (OT) with the ID / OOD detector, linking the feature alignment strength to the degree of OOD predicted by the model. This not only regularizes pseudo-labels but also effectively suppresses the model's overconfident prediction of OOD samples. This mechanism achieves [the desired effect] while keeping the visual language backbone network frozen. Continuous improvement and Steady growth.

[0120] Overall, the experimental results verify the effectiveness and rationality of the alignment-aware test-time adaptation method for OOD detection: while maintaining zero-sample generalization ability, this method can achieve a significant performance leap in scenarios where there is a significant shift in data distribution.

[0121] Table 1. Zero-shot test-time adaptation (TTA) and out-of-distribution (OOD) detection results using CUB-200-2011, Stanford Automotive, Food-101, and Oxford University-IIIT Pet datasets as in-distribution (ID) datasets.

[0122]

[0123] Table 1 (continued). Zero-shot test-time adaptation (TTA) out-of-distribution (OOD) detection results using CUB-200-2011, Stanford Automotive, Food-101, and Oxford-IIIT Pet datasets as in-distribution (ID) datasets.

[0124]

[0125] Table 2. Zero-shot Test-Time Adaptation (TTA) and Out-of-Distribution (OOD) Detection Results Using ImageNet and its variant datasets as In-Distribution (ID) Datasets

[0126]

[0127] Table 2 (continued). Zero-shot test-time adaptation (TTA) and out-of-distribution (OOD) detection results using ImageNet and its variant datasets as in-distribution (ID) datasets.

[0128]

[0129] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-described visual language model zero-sample out-of-distribution detection method.

[0130] See Figure 2 The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the above-described visual language model zero-sample distribution out-of-distribution detection method.

[0131] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to perform the steps of the above-described visual language model zero-sample distribution out-of-distribution detection method.

[0132] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above-described visual language model zero-sample distribution out-of-distribution detection method.

[0133] It should be noted that those skilled in the art will understand that all or part of the steps implemented in the embodiments of the present invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in hardware, it can be implemented entirely or partially by purchasing standard parts or modifications. When implemented in software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0134] In summary, to address the poor performance of existing OOD detection methods when directly applied to zero-sample VLMs, this invention provides a detection method based on Self-Reinforced Optimal Transport (SROT). This method aims to significantly improve the out-of-distribution detection performance of visual-language models on target tasks by aligning posterior probabilities through optimal transport without relying on any labeled data or fine-tuning the model's backbone network. Furthermore, it utilizes the feedback loop of a lightweight detector to correct the alignment process, thereby significantly enhancing the out-of-distribution detection performance of visual-language models on target tasks. This method offers advantages such as high-precision OOD detection, no need for labeled data, high computational efficiency, and strong robustness.

[0135] It should be understood that the examples and embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Those skilled in the art can make various modifications or changes based on them. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for detecting out-of-distribution zero-samples in a visual language model, characterized in that, Includes the following steps: S1. Utilize the pre-trained visual-language model (VLM) to receive the unlabeled image data stream during the testing phase; S2. Use the frozen pre-trained visual-language model (VLM) to extract image features and text prompt features from the unlabeled image data stream; S3. Construct a cost matrix based on the original output of the pre-trained visual-language model (VLM), and use the entropy regularization optimal transmission mechanism to calculate the posterior probability of the pre-trained visual-language model (VLM) prediction after alignment, so as to alleviate distribution mismatch. S4. Calculate the out-of-distribution (OOD) sample score based on the aligned posterior probability, and dynamically determine the threshold using a sliding window mechanism to generate binary pseudo-labels; S5. Train a lightweight detector for out-of-distribution (OOD) samples using binary pseudo-labels. This lightweight detector directly outputs the probability of binary pseudo-labels based on image features and text prompt features. S6. Self-reinforcing feedback loop: Feedback the out-of-distribution OOD sample probability output by the trained lightweight detector back to the optimal transmission OT process, correct the cost matrix of the optimal transmission OT, and introduce open set priors to form a closed-loop optimization. S7. Based on the output score of the out-of-distribution OOD sample from the finally trained lightweight detector, determine whether the current detected sample is an out-of-distribution OOD sample of an unknown category; In step S6, the self-reinforcing feedback loop process includes: S6.1 Construct the corrected cost matrix based on the out-of-distribution OOD sample probabilities output by the trained lightweight detector. The corresponding probability distribution The CLIP original distribution and open set prior of the multimodal pre-trained neural network model. The weighted mixture is expressed as: in, If it is a uniform distribution or other prior distribution, The probability of an out-of-distribution (OOD) sample. The original posterior probability of CLIP for the multimodal pre-trained neural network model; S6.2 Logical Judgment: If the lightweight detector considers the sample to be an out-of-distribution OOD sample, then the uniform prior is referenced during optimal transmission OT alignment; otherwise, if the lightweight detector considers the sample to be an in-distribution ID sample, then the original prediction of the multimodal pre-trained neural network model CLIP is referenced during optimal transmission OT alignment. S6.3, Using the modified cost matrix Re-execute the optimal transmission problem in the optimal transmission OT probability alignment, update the pseudo-labels, and continue to optimize the lightweight detector in the next batch of samples.

2. The method for detecting zero-sample distribution out-of-distribution visual language models as described in claim 1, characterized in that, The pre-trained visual-language model (VLM) in S1 includes four operational phases: optimal transport OT alignment, adaptive pseudo-label generation, lightweight detector training, and self-reinforcing feedback loop.

3. The method for detecting out-of-distribution zero-sample visual language models as described in claim 1, characterized in that, In the probability alignment process based on optimal transmission OT, S3 selects a multimodal pre-trained neural network model CLIP from the pre-trained visual-language model VLM. For a batch of test image samples x, it calculates the similarity with the text prototypes of ID sample categories within the distribution of all known categories, and obtains the posterior probability of the original multimodal pre-trained neural network model CLIP. This forces the pre-trained visual-language model (VLM) to predict in accordance with marginal distribution constraints, thus correcting the probability bias caused by distribution shift.

4. The method for detecting out-of-distribution zero-sample visual language models as described in claim 3, characterized in that, In step S3, optimal transmission OT probability alignment is performed to obtain the posterior probability of the aligned multimodal pre-trained neural network model CLIP. The process includes: S3.1 Construct the cost matrix M, ; S3.2, Set uniform sample weights and category prior To encourage sample class balance; S3.3 Solving the optimal transmission problem based on the entropy regularization optimal transmission mechanism to obtain the transmission scheme. The calculation expression is: in, Indicates a feasible transmission plan. This represents the plan that minimizes the total transmission cost among all feasible transmission plans. Represents the Frobenius inner product. represents the regularization coefficient, used to control the influence of the KL divergence term on the optimization results. KL divergence is used to measure the distribution difference, and T represents the conjugate transpose. S3.4, Transmission Scheme Normalization is performed to obtain the aligned posterior probabilities. .

5. The method for detecting zero-sample distribution out-of-distribution visual language models as described in claim 1, characterized in that, The S4 process, which involves adaptive thresholding and pseudo-label generation based on a sliding window, includes: S4.1 Calculate the out-of-distribution OOD sample score based on optimal transmission OT. : Where x represents the test image sample, This represents the posterior probability after alignment; S4.2 Store the sample scores obtained in the most recent batches in a first-in-first-out queue to maintain a sliding window containing the scores of the most recent historical samples. ; S4.3, in the sliding window Find the optimal segmentation threshold within the context The expression is as follows: This minimizes the sum of the within-class variances of the two groups of samples divided by the threshold. in, This represents the optimal threshold that minimizes the sum of the variances of the ID group and the OOD group among all possible thresholds λ. Indicates the current threshold Below, the fractional variances of the samples divided into within-distribution groups, Indicates the current threshold The fractional variance of the samples divided into out-of-distribution groups; S4.4 Utilizing the optimal segmentation threshold Generate pseudo-labels for the current batch of samples. Where 0 represents an in-distribution ID sample of a known category, and 1 represents an out-of-distribution OOD sample of an unknown category.

6. The method for detecting out-of-distribution zero-sample visual language models as described in claim 1, characterized in that, In S5, the lightweight detector training process includes: S5.1, Construct a lightweight detector Its input is frozen image features. ; S5.2, Based on generated pseudo-tags The parameters of this lightweight detector are updated online using the cross-entropy loss function; S5.3 The output of the lightweight detector is the probability of the obtained out-of-distribution (OOD) samples. and the probability of ID samples within the distribution .

7. A computer-readable storage medium, characterized in that, The system stores a computer program that, when executed by a processor, causes the processor to perform the steps of the out-of-distribution detection method for a visual language model as described in any one of claims 1-6.

8. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the out-of-distribution detection method for the visual language model as described in any one of claims 1-6.

Images