A zero-shot open-set identification method based on probability distribution matching

Abstract

The present application provides a zero-shot open set recognition method based on probability distribution matching, which overcomes the limitations of traditional methods relying only on point-to-point maximum similarity matching in sparse semantic space. By introducing a large language model with fine-grained attributes and approximate unknown class interpolation, the decision benchmark is expanded from sparse points to a dense and complete semantic reference system. At the same time, the statistical characteristics of the test data are used to construct a distribution prototype reflecting the essence of the class, and the recognition process is reconstructed as a global distribution pattern matching. In inference, the distribution similarity is used instead of a single similarity score, effectively balancing the accurate classification of known classes and the robust rejection of unknown classes, which is beneficial to the efficient and secure deployment of the model in an open and dynamic scene without labeled data.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and image processing technology, and specifically relates to a method for zero-shot open set recognition based on probability distribution matching without training samples. Background Technology

[0002] With the widespread application of artificial intelligence (AI) technology in open-world scenarios such as autonomous driving, security monitoring, and industrial quality inspection, the deployment environments faced by models are becoming increasingly complex and dynamic. Therefore, zero-shot open-set classification tasks are particularly urgent in practical applications. Their core value lies in their extremely high practicality: First, they do not require collecting labeled images for retraining; they can quickly transfer and adapt to different business domains based solely on textual semantics, significantly reducing the cost of model deployment. Second, they endow the system with the ability to reject unknown categories, ensuring that the model can reject unfamiliar interference items as unknown classes, rather than blindly classifying them as known classes. This characteristic of accurately identifying targets and robustly intercepting unknowns in open and dynamic environments is the cornerstone of building safe and reliable AI systems.

[0003] Existing related technologies mainly focus on two aspects. Firstly, they enhance the model's ability to distinguish known classes, thereby indirectly improving its ability to reject unknown classes. For example, Liu et al., in their paper "X. Liu and C. Zach, 'Tag: Textprompt augmentation for zero-shot out-of-distribution detection,' in Proceedings of the European Conference on Computer Vision, pp.364-380, 2024," performed text prompt augmentation on known classes. Miyai et al., in their paper "A. Miyai, Q. Yu, G. Irie, and K. Aizawa, 'Gl-mcm: Global and local maximum concept matching for zero-shotout-of-distribution detection,' International Journal of Computer Vision, vol.133, no.6, pp.3586-3596, 2025," improved the model's classification performance by fusing global and local features. Secondly, they explicitly model the unknown open space to distinguish between known and unknown classes. For example, Wang et al., in the paper "H.Wang, Y.Li, H.Yao, and X.Li, “Clipn for zero-shot ooddetection: Teaching clip to say no,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, pp.1802-1812, 2023," used a pre-trained negative text encoder to assist in the identification of unknown samples.Cao et al., in their paper "C. Cao, Z. Zhong, Z. Zhou, Y. Liu, T. Liu, and B. Han, 'Envisioning outlier exposure by large language models for out-of-distribution determination,'" in Proceedings of the 41st International Conference on Machine Learning, pp. 5629-5659, 2024, utilize large language models to generate potential unknown categories in order to construct more explicit known-unknown decision boundaries.

[0004] The methods described above primarily rely on the maximum similarity matching strategy. Specifically, these methods map image features and text features to the same space, and classify samples into the category with the highest similarity by calculating point-to-point cosine similarity. However, this strategy relies solely on a sparse semantic space constructed from category names, lacking fine-grained discriminative information, making it difficult for the model to distinguish subtle differences between categories. Furthermore, it focuses only on a single category semantic anchor, ignoring the probability distribution characteristics of samples in the feature space. When an unknown sample happens to be similar to a local feature of a known class, misclassification is highly likely. Therefore, there is an urgent need for a new method that can overcome the limitations of maximum similarity matching and utilize global distribution information to improve the robustness of zero-shot open set recognition. Summary of the Invention

[0005] This invention discloses a zero-sample open set identification method based on probability distribution matching, which mainly solves the problems of high false positive rates and inability to effectively reject unknown classes caused by existing technologies relying on maximum similarity matching in sparse semantic spaces. Specifically, the purpose of this invention is: 1. By introducing fine-grained attributes of categories and synthesizing approximate unknown classes, the problem of sparse semantic space is solved, and a dense and complete semantic coordinate system is constructed.

[0006] 2. Abandoning the traditional maximum similarity matching, we propose a distribution matching paradigm based on statistical characteristics, which uses the distribution prototype of categories to improve the robustness of recognition.

[0007] This invention proposes a zero-sample open set identification method based on probability distribution matching, which specifically includes the following steps: Step 1: Use a text encoder to extract the text feature vector for each category in the known category set, and use a visual encoder to extract the image feature vector for all unlabeled samples to be tested; Step 2: Use the large language model for each known class ,generate Each known class is described with fine-grained attributes, and its textual features are extracted; for each known class The corresponding approximate unknown class features are generated through hybrid interpolation; Step 3: Calculate the cosine similarity between all test samples and known categories, for each known category. Filter out the most similar A set of samples And calculate the distribution prototype; Step 4, for the test sample The feature vector is obtained through a visual encoder. Then, the probability distribution of the test samples in the extended semantic space is calculated; Step 5, calculate the sample Category The matching score is calculated, and the maximum matching score is obtained. If the maximum matching score is higher than a preset threshold, the sample is classified into the corresponding known class. Otherwise, reject the sample and mark it as an unknown class.

[0008] Furthermore, in step 1, the text encoder of the pre-trained visual-language model CLIP is used to extract text feature vectors for each category.

[0009] Furthermore, approximately unknown features The calculation formula is as follows:

[0010] in, For interpolation hyperparameters, Given the number of known classes, This indicates other known categories besides the current category.

[0011] Furthermore, the distribution prototype The calculation formula is as follows:

[0012] Here, softmax represents the normalization exponential function, which maps the elements of the input vector to a probability distribution that sums to 1. Representative set The first in The visual feature vectors of each sample, the known class feature vectors and their attribute feature vectors, and the approximately unknown class feature vectors and their attribute feature vectors together construct the expanded semantic feature vector set. The distribution prototype of a category represents the global probability distribution pattern of that category.

[0013] Furthermore, probability distribution The calculation formula is as follows:

[0014] Among them, the known class feature vectors and their attribute feature vectors, and the approximately unknown class feature vectors and their attribute feature vectors together construct the expanded semantic feature vector set. .

[0015] Further, calculate the sample Category Matching score The formula is as follows:

[0016] in It is the dimension of the vector. and Let the nth element in the test sample probability distribution vector and the class prototype probability distribution vector be respectively represented. Each component.

[0017] Furthermore, the maximum matching score is calculated using the following formula: .

[0018] Furthermore, the known category set comes from a general object dataset in the field of image processing.

[0019] Furthermore, the matching score ranges from 0 to 1.

[0020] The present invention also provides a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the zero-sample open set identification method based on probability distribution matching as described in the above technical solution.

[0021] This invention has the following significant advantages over existing zero-sample open set identification methods: Significantly improved recognition accuracy and robustness: This invention reshapes the decision paradigm from maximum similarity matching to global probability distribution matching. The model is no longer limited by the matching error of a single feature point and can effectively resist noise interference. On large-scale object classification datasets (ImageNet) and general object classification datasets (Caltech101), which test the performance of image classification, the overall accuracy of this method is improved by 2.3% and 6.84% respectively compared with the previous best model. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the various modules involved in the embodiment description are briefly introduced below. Obviously, the accompanying drawings in the following description are merely flowcharts of the present invention. For those skilled in the art, these drawings can be extended to generate other related drawings without any creative effort.

[0023] Figure 1 This is a flowchart of an embodiment of the present invention. Detailed Implementation

[0024] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0025] This invention proposes a zero-shot open set identification method based on probability distribution matching. This method constructs a complete semantic benchmark through semantic space expansion without requiring any labeled training samples, mines statistical patterns in the data through distribution prototype construction, and finally uses the Bach coefficient to measure the distributional similarity between test samples and category prototypes to achieve classification and rejection. The specific technical solution is as follows: 1. Expanding the Sparse Semantic Space. To address the problem of sparse known information, the knowledge base of a large language model is utilized to generate several fine-grained attribute descriptions (such as color, shape, texture, etc.) for each known category, expanding the single category center into a rich cluster of semantic features. To provide the model with prior knowledge about the unknown, a hybrid interpolation strategy is employed. Features of known classes are used to generate approximate features of unknown classes. This transforms the semantic space from sparse to dense, providing a complete reference coordinate system for subsequent distribution matching.

[0026] 2. Constructing Category Distribution Prototypes. First, preliminary predictions are made on the samples to be tested, and the set of samples with the highest confidence in each known category is selected as the pseudo-label sample set. For each known class, its classification probability distribution is calculated using the selected sample set, serving as the category distribution prototype for that class. These prototypes characterize the typical distribution pattern of that class under the current data distribution.

[0027] 3. Distribution Similarity Calculation and Classification Decision. During the inference phase, decisions are made by measuring the similarity between the sample distribution and the prototype distribution. For any test sample, its class probability distribution in the extended semantic space is calculated. The Bach coefficient is used as the metric to calculate the similarity between the test sample distribution and the prototype distributions of each known class, which is then used as the final matching score. If the maximum matching score is higher than a preset open set threshold, it is classified as the corresponding known class; otherwise, it is classified as an unknown class.

[0028] In summary, this invention presents a zero-shot open set recognition method based on probability distribution matching that requires no training. It primarily overcomes the limitations of traditional methods that rely solely on point-to-point maximum similarity matching in sparse semantic spaces. This invention expands the decision-making benchmark from sparse points to a dense and complete semantic reference system by introducing fine-grained attributes of a large language model and interpolation for approximate unknown classes. Simultaneously, it utilizes the statistical characteristics of test data to construct a distribution prototype reflecting the essence of the categories, reconstructing the recognition process into a matching of global distribution patterns. During inference, it uses distributional similarity instead of a single similarity score, effectively balancing accurate classification of known classes with robust rejection of unknown classes, facilitating efficient and secure model deployment in open, dynamic scenarios without labeled data.

[0029] Reference Figure 1 The specific implementation steps of this invention are as follows: Step 1: First, use the pre-trained visual-language model (CLIP) text encoder to extract text feature vectors for each category. The visual encoder was used to extract the image feature vectors of all unlabeled samples to be tested. Suppose that the set of known categories in a general object dataset in the field of image processing is . .

[0030] Step 2: Utilize the large language model for each known class .generate Fine-grained attribute description (For example, describing birds as having "yellow feathers" or "pointed beaks"), and extracting their textual features using a text encoder. To simulate unknown classes, for each known class... Approximate unknown features are generated through hybrid interpolation. The calculation formula is as follows:

[0031] in, For interpolation hyperparameters, Given the number of known classes, This represents other known categories besides the current category. Therefore, the known class feature vectors and their attribute feature vectors, along with the approximately unknown class feature vectors and their attribute feature vectors, together construct the expanded semantic feature vector set. .

[0032] Step 3: Calculate the cosine similarity between all test samples and known categories, for each known category. Filter out the most similar A set of samples Based on sets Calculate the distribution prototype of this class. The calculation formula is as follows:

[0033] Here, softmax represents the normalization exponential function, which maps the elements of the input vector to a probability distribution that sums to 1. Representative set The first in The visual feature vectors of each sample. The distribution prototype of the category represents the global probability distribution pattern of that category.

[0034] Step 4: After obtaining the distribution prototypes of all known classes, for the test samples... The feature vector is obtained through a visual encoder. Then, the probability distribution of the sample in the extended semantic space is calculated. The calculation formula is as follows:

[0035] Step 5: Use the Bach coefficient to measure the similarity between the sample distribution and the prototype distribution of each category.

[0036] Calculate samples Category Matching score The formula is as follows:

[0037] in It is the dimension of the vector. and Let the nth element in the test sample probability distribution vector and the class prototype probability distribution vector be respectively represented. There are several components. The range of values ​​for the Bach coefficient is... .

[0038] Calculate samples The maximum matching score among all known categories is calculated using the following formula: .

[0039] like If the value exceeds a preset threshold, this invention sets the preset threshold to 0.8, then the sample is classified into the corresponding known class. Otherwise, reject the sample and mark it as an unknown class.

[0040] The effects of this invention can be further illustrated by the following experiments.

[0041] 1. Experimental conditions This invention was simulated using a Gen Intel® Core™ i9-10920X CPU, an NVIDIA RTX 3080 GPU, 32GB of RAM, and a Linux operating system, with a model built using the PyTorch deep learning framework. To comprehensively evaluate the performance of this invention, the performance of the probability distribution matching network was verified on the large-scale object classification dataset (ImageNet) and the general object classification dataset (Caltech101). ImageNet contains image data of 1000 categories; to simulate zero-shot open set scenarios, this invention adopted a 50% / 50% category splitting strategy, where 500 categories were designated as known classes for constructing the semantic space and distribution prototype, and the remaining 500 categories were designated as unknown classes to test the model's ability to reject unseen categories. Caltech101 includes 101 object categories; similarly, a 50 / 50 split was used, selecting 51 categories as known classes and the remaining 50 categories as unknown classes. The experiment strictly followed the zero-sample setting, using no images for training and relying solely on the semantic information of known classes in the text for inference testing.

[0042] 2. Experiment Content Considering the popularity of zero-shot open set algorithms, seven deep learning-based zero-shot open set methods were selected from ImageNet and Caltech 101 for performance comparison to verify the performance of the probability distribution matching mechanism of this invention. This invention uses overall accuracy to quantify the performance of each method. The overall accuracy of the method of this invention and the comparison algorithms are shown in Tables 1 and 2 below: Table 1 Overall accuracy of various methods in ImageNet

[0043] Table 2 Overall accuracy of each method in Caltech 101

[0044] As shown in Tables 1 and 2, the zero-shot open set identification method based on probability distribution matching of this invention exhibits the best classification performance. This is because the proposed probability distribution matching framework first utilizes a large language model to introduce fine-grained attributes and synthesize approximate unknown classes, constructing a dense and complete semantic space, effectively overcoming the problem of sparse semantic information in traditional methods. Simultaneously, it abandons the unstable maximum similarity matching paradigm and instead uses a class distribution prototype for global probability distribution matching, enabling the model to capture the overall statistical characteristics of the classes. Therefore, the method of this invention can more accurately identify known classes and robustly reject unknown classes, significantly outperforming the comparative methods. Comparative experiments on public datasets further demonstrate the advanced effectiveness of the method of this invention.

[0045] The comparison algorithms are derived from the following documents: [1] "Y.Ming, Z.Cai, J.Gu, Y.Sun, W.Li, and Y.Li, "Delving into out-of-distribution detection with vision-language representations," Advances in Neural Information Processing Systems, vol.35, pp.35087-35102, 2022." [2] "H.Wang, Y.Li, H.Yao, and X.Li, "Clipn for zero-shot ooddetection: Teaching clip to say no," in Proceedings of the IEEE / CVFInternational Conference on Computer Vision, pp.1802-1812, 2023." [3] "X.Liu and C.Zach, "Tag:Text prompt augmentation for zero-shotout-of-distribution detection," in Proceedings of the European Conference on Computer Vision, pp.364-380, 2024." [4]“C.Cao, Z.Zhong, Z.Zhou, Y.Liu, T.Liu, and B.Han, “Envisioningoutlier exposure by large language models for out-of-distribution detec-tion,” in Proceedings of the 41th International Conference on MachineLearning, pp.5629-5659, 2024.” [5]“X.Jiang, F.Liu, Z.Fang, H.Chen, T.Liu, F.Zheng, and B.Han,“Negative label guided OOD detection with pretrained vision-language models,”in Proceedings of the International Conference on Learning Representations,2024.” [6]“M.Chen, J.Gao, and C.Xu, “Conjugated semantic pool improves ooddetection with pre-trained vision-language models,” Advances in NeuralInformation Processing Systems, vol.37, pp.82560-82593, 2024.” [7]“A.Miyai, Q.Yu, G.Irie, and K.Aizawa, “Gl-mcm:Global and localmaximum concept matching for zero-shot out-of-distribution detection,”International Journal of Computer Vision, vol.133, no.6, pp.3586- 3596,2025.” On the other hand, embodiments of the present invention also provide a computer-readable storage medium, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements the zero-sample open set identification method based on probability distribution matching as described in the above technical solution.

[0046] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A zero-sample open set identification method based on probability distribution matching, characterized in that, Includes the following steps: Step 1: Use a text encoder to extract the text feature vector for each category in the known category set, and use a visual encoder to extract the image feature vector for all unlabeled samples to be tested; Step 2: Use the large language model for each known class ,generate Each attribute is described in fine detail, and its textual features are extracted. For each known class The corresponding approximate unknown class features are generated through hybrid interpolation; Step 3: Calculate the cosine similarity between all test samples and known categories, for each known category. Filter out the most similar A set of samples And calculate the distribution prototype; Step 4, for the test sample The feature vector is obtained through a visual encoder. Then, the probability distribution of the test samples in the extended semantic space is calculated; Step 5, calculate the sample Category The matching score is calculated, and the maximum matching score is obtained. If the maximum matching score is higher than a preset threshold, the sample is classified into the corresponding known class. Otherwise, reject the sample and mark it as an unknown class.

2. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: In step 1, the text encoder of the pre-trained visual-language model CLIP is used to extract text feature vectors for each category.

3. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: Approximate unknown features The calculation formula is as follows: in, For interpolation hyperparameters, Given the number of known classes, This indicates other known categories besides the current category.

4. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: Distribution prototype The calculation formula is as follows: Here, softmax represents the normalization exponential function, which maps the elements of the input vector to a probability distribution that sums to 1. Representative set The first in The visual feature vectors of each sample, the known class feature vectors and their attribute feature vectors, and the approximately unknown class feature vectors and their attribute feature vectors together construct the expanded semantic feature vector set. The distribution prototype of a category represents the global probability distribution pattern of that category.

5. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: probability distribution The calculation formula is as follows: Among them, the known class feature vectors and their attribute feature vectors, and the approximately unknown class feature vectors and their attribute feature vectors together construct the expanded semantic feature vector set. .

6. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: Calculate samples Category Matching score The formula is as follows: in It is the dimension of the vector. and Let the nth element in the test sample probability distribution vector and the class prototype probability distribution vector be respectively represented. Each component.

7. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: The maximum matching score is calculated using the following formula: 。 8. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: The known set of categories comes from a general object dataset in the field of image processing.

9. The zero-sample open set identification method based on probability distribution matching as described in claim 1, characterized in that: The matching score ranges from 0 to 1.

10. A computer-readable storage medium, characterized in that, It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the zero-sample open set identification method based on probability distribution matching as described in any one of claims 1-8.

Images