A distribution-out detection method and device based on a CLIP model and a word tree

Abstract

The application provides a distribution-out detection method and device based on a CLIP model and a word tree, and the method comprises the following steps: identifying the position of each in-distribution word in a in-distribution data set in WordNet, and selecting k out-of-distribution words most similar to each in-distribution word in WordNet; constructing a word tree by using the in-distribution words and the k out-of-distribution words most similar to each in-distribution word; obtaining a to-be-detected image; performing concept matching on the to-be-detected image and the words on the word tree by using a CLIP model to obtain a concept matching score; calculating a distribution-out score based on the concept matching score, and performing distribution-out detection according to the distribution-out score to obtain a distribution-out detection result. By searching for the k out-of-distribution words closest to the in-distribution words in WordNet, the word tree is constructed, and the concept matching is performed in combination with the CLIP model to calculate the distribution-out score for the distribution-out detection, so that the connection between the words in WordNet is fully utilized, and more accurate distribution-out detection effect is realized.

Description

Technical Field

[0001] This invention belongs to the field of out-of-distribution detection, and specifically relates to an out-of-distribution detection method and apparatus based on CLIP model and word tree. Background Technology

[0002] In recent years, with the development of technology, deep neural networks have demonstrated superior performance in fields such as computer vision and text processing. In closed systems where training and testing data are sampled from the same distribution, neural networks can exhibit excellent performance. However, when deploying neural networks in real-world applications, they often receive a significant amount of data outside the categories found in the training dataset, leading to unexpected outputs. Recent research indicates that even for completely unrecognizable or irrelevant inputs, neural networks tend to make high-confidence predictions. In fields like autonomous driving and medicine, if neural networks provide incorrect outputs with high confidence for never-before-seen inputs, there are significant risks. Ideally, neural networks should proactively recognize inputs they cannot classify or process, assigning significant uncertainty to these out-of-distribution examples and delegating the task to human intervention.

[0003] It is crucial for classifiers to recognize uncertainty when encountering new types of input, i.e., out-of-distribution examples. Therefore, the ability to accurately detect out-of-distribution examples is an important part of visual recognition tasks.

[0004] The existing technical solutions mainly include:

[0005] 1. Distance-based methods: These methods utilize feature embeddings extracted from the model to calculate the distance between the test data and the training data, thereby enabling out-of-distribution detection.

[0006] 2. Density-based methods: Detecting out-of-distribution data by modeling data density.

[0007] 3. Classification-based methods: One main approach relies on the classification scores provided by the classifier for out-of-distribution detection; another main approach incorporates generated out-of-distribution data during training, allowing the neural network to generate low-confidence scores on the out-of-distribution data, thereby improving the robustness of the neural network; there is also an out-of-distribution detection method based on the CLIP model, which uses the CLIP model to perform concept matching between the image and the in-distribution labels or the out-of-distribution labels generated by the text generator based on the image, and then uses the concept matching score to calculate the out-of-distribution detection score, thereby performing out-of-distribution detection.

[0008] Disadvantages of existing technology:

[0009] Distance-based methods make a strong assumption that the underlying feature space is a conditionally Gaussian distribution. Density-based methods require modeling the distribution of the in-distribution dataset. Some classification-based methods rely on classification scores from the classifier for out-of-distribution detection. Therefore, the out-of-distribution detection performance of these methods is largely limited by the performance and capacity of the classification model. Furthermore, methods that incorporate out-of-distribution data during training to improve the robustness of the neural network are time-consuming. Current out-of-distribution detection methods based on the CLIP model rely only on in-distribution labels and a small number of out-of-distribution labels; a wealth of valuable knowledge within the CLIP model remains untapped. Summary of the Invention

[0010] The purpose of this invention is to provide an out-of-distribution detection method and apparatus based on CLIP model and word tree, which can effectively improve the accuracy and versatility of detecting out-of-distribution samples.

[0011] To achieve the above objectives, this invention provides an out-of-distribution detection method based on the CLIP model and word tree, comprising:

[0012] Identify the position of each in-distribution word in the in-distribution dataset in WordNet, and select the k out-of-distribution words in WordNet that are most similar to each of the in-distribution words;

[0013] Construct a word tree using the in-distribution words and the k out-of-distribution words that are most similar to each of the in-distribution words;

[0014] Acquire the image to be detected;

[0015] The CLIP model is used to perform concept matching between the image to be detected and the words in the word tree to obtain a concept matching score.

[0016] The out-of-distribution score is calculated based on the concept matching score, and out-of-distribution detection is performed based on the out-of-distribution score to obtain the out-of-distribution detection result.

[0017] Preferably, the step of constructing a word tree using in-distribution words and the selected k out-of-distribution words most similar to each in-distribution word includes:

[0018] Based on the edges presented in WordNet, connections are established between the in-distribution words and the k out-of-distribution words to construct a word tree;

[0019] If multiple subtrees exist, identify the lowest common ancestor word of the multiple subtrees and merge them until a single word tree structure is obtained.

[0020] Preferably, the similarity calculation formula between the in-distribution words and the k out-of-distribution words is as follows:

[0021]

[0022] Where LCS(n) i ,n j ) indicates that it is used to determine n i and n j The function of depth(n) is the lowest common ancestor of two words. i ) is used to calculate the word n i A function of the distance to the root word.

[0023] Preferably, the word tree is represented as T =<W,R> Where W represents the set of words {w1, w2, ..., w} that constitute the word tree. n R encapsulates the relationships between these words, including parent-child relationships. p Relationship with siblings R p .

[0024] Preferably, the step of using the CLIP model to perform concept matching between the image to be detected and the words in the word tree includes:

[0025] For the image x to be detected, a large-scale pre-trained model CLIP is used to compare it with the words W = {w1, w2, ..., w} in the word tree. n} to perform matching;

[0026] For each word in the word tree, construct a concept set "This is a photo of..." <wi>", where i = 1, ..., n, and the CLIP text encoder is used. Y () Extract the corresponding text feature embeddings;

[0027] For the image to be detected, x, the CLIP image encoder using the CLIP model is used. I () Extract image feature embeddings from the image to be detected x;

[0028] The concept matching score {s(w1),s(w2),...,s(w...} is obtained by calculating the cosine similarity between the text feature embedding and the image feature embedding respectively. n )}, where s(wi)∈[-1,1], the formula is as follows:

[0029] s(w i ) = CM(x,w i )

[0030] CM(x,w i =cosine(CLIP) I (x), CLIP Y (w i ))

[0031] CLIP I (·) denotes a function used to extract image feature embeddings, CLIP Y (·) denotes the function used to extract text feature embeddings. cosine(·,·) denotes the function used to calculate the cosine similarity between two vectors.

[0032] Preferably, the step of calculating out-of-distribution scores based on concept matching scores and performing out-of-distribution detection based on the out-of-distribution scores to obtain out-of-distribution detection results includes:

[0033] The softmax operation is used to normalize all concept matching scores and calculate the summation score of out-of-distribution words.

[0034] The summarized score is used as an indicator of the probability that the image to be detected x falls into an out-of-distribution category, as shown in the following formula:

[0035]

[0036] τ is the threshold. The set of words outside the distribution in the representative word tree, s(w i ) = CM(x,w i ) is the cosine similarity between the text feature embedding and the image feature embedding of the image to be detected x; This refers to calculating the cosine similarity s(w) of all words in the word tree. i To calculate an e of s(w) i Raise it to the power of 1, then sum it up; It is a word w i The cosine similarity score after softmax operation represents the proportion of a word's cosine similarity to the total cosine similarity of all words in the word tree; This represents the proportion of the sum of the cosine similarities of all out-of-distribution words in the word tree to the sum of the cosine similarities of all words in the word tree.

[0037] Preferably, the threshold setting varies depending on the different training datasets.

[0038] Preferably, the dataset CIFAR-10 is used as the dataset within the distribution.

[0039] The present invention also provides an out-of-distribution detection device based on the CLIP model and word tree, comprising:

[0040] The recognition module is used to identify the position of each in-distribution word in the in-distribution dataset in WordNet, and select the k out-of-distribution words in WordNet that are most similar to each in-distribution word;

[0041] A construction module is used to construct a word tree using in-distribution words and the selected k out-of-distribution words that are most similar to each in-distribution word;

[0042] The acquisition module is used to acquire the image to be detected;

[0043] The matching module is used to perform concept matching between the image to be detected and the words in the word tree using the CLIP model to obtain a concept matching score;

[0044] The detection module is used to calculate the out-of-distribution score based on the concept matching score, and to perform out-of-distribution detection based on the out-of-distribution score to obtain the out-of-distribution detection result.

[0045] Compared with existing technologies, this invention provides an out-of-distribution detection method and apparatus based on the CLIP model and word tree, which can effectively improve the accuracy and versatility of out-of-distribution sample detection. First, this technical solution constructs a word tree by selecting the k most similar out-of-distribution words to the in-distribution words in WordNet, and adding them to the in-distribution words. This word tree is not merely a set of words but also considers the semantic relationships between words. The CLIP large model is used to perform concept matching between the image to be detected and the words in the word tree, deeply mining the semantic information in the word tree to determine the degree of concept association between the image to be detected and the words. Finally, the out-of-distribution score is calculated using the concept matching score, thereby performing out-of-distribution detection. This technical solution fully utilizes the connections between words in WordNet to achieve more accurate matching. Second, this technical solution uses Wu & Palmer Similarity as the similarity metric between in-distribution and out-of-distribution words. This metric comprehensively considers the relationship between in-distribution and out-of-distribution words in WordNet. The found out-of-distribution words are more likely to confuse the neural network. Combined with the concept matching function of the CLIP large model, better out-of-distribution detection accuracy can be obtained. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating an out-of-distribution detection method based on the CLIP model and word tree, provided in a preferred embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram illustrating the operation of an out-of-distribution detection method based on the CLIP model and word tree, according to a preferred embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram of the structure of an out-of-distribution detection device based on the CLIP model and word tree, provided in a preferred embodiment of the present invention. Detailed Implementation

[0049] The embodiments of the present invention will be described in detail below. The embodiments described below are implemented based on the technical solution of the present invention, and detailed implementation methods and specific operation processes are given. However, the protection scope of the present invention is not limited to the embodiments described below.

[0050] like Figure 1 , Figure 2 As shown, this invention provides an out-of-distribution detection method based on the CLIP model and word tree, the method comprising the following steps:

[0051] S1 identifies the position of each in-distribution word (i.e., the label of the in-distribution dataset) in WordNet and selects the k out-of-distribution words in WordNet that are most similar to each in-distribution word, where k is an adjustable parameter, and k = 2000 can generally be selected.

[0052] S2, construct a word tree using the in-distribution words and the selected k out-of-distribution words that are most similar to each in-distribution word;

[0053] S3, acquire the image to be detected;

[0054] S4. Use the CLIP model to perform concept matching between the image to be detected and the words in the word tree to obtain a concept matching score.

[0055] S5 calculates out-of-distribution scores based on concept matching scores, and performs out-of-distribution detection based on the out-of-distribution scores to obtain out-of-distribution detection results.

[0056] This technical solution constructs a word tree by selecting the k most similar out-of-distribution words from WordNet. This word tree is not merely a set of words but also considers the semantic relationships between them. It then utilizes the CLIP large model to perform concept matching between the image to be detected and the words in the word tree, deeply mining the semantic information within the word tree to determine the degree of concept association between the image and the words. Finally, it uses the concept matching score to calculate the out-of-distribution score, thereby performing out-of-distribution detection. This technical solution fully leverages the connections between words in WordNet, achieving more accurate out-of-distribution detection results.

[0057] Unlike traditional classifiers, the CLIP model integrates visual and semantic information, achieving a higher level of conceptual understanding. Therefore, it is less susceptible to performance and capacity limitations in out-of-distribution detection tasks. To fully leverage the rich knowledge encapsulated within the CLIP model, we utilize the powerful features of WordNet to fully utilize the abundant knowledge embedded within CLIP.

[0058] This invention uses the large-scale pre-trained CLIP model. Since these models have been exposed to different open-world samples, their out-of-distribution detection performance is not easily affected by model performance and capacity. Therefore, they can effectively identify and distinguish between in-distribution and out-of-distribution samples. We extract rich structured knowledge from the CLIP model to detect out-of-distribution samples more effectively.

[0059] The process involves constructing a word tree using the k most similar out-of-distribution words selected from each in-distribution word. This includes: establishing connections between in-distribution words and the k most similar out-of-distribution words based on the edges presented in WordNet, and merging word trees; if multiple subtrees exist, identifying the lowest common ancestor word of the multiple subtrees and merging them accordingly, until a single word tree structure is obtained.

[0060] This is done primarily to facilitate the distinction between out-of-distribution words and in-distribution words in the word tree. Apart from in-distribution words and their ancestor and descendant nodes, all other nodes are out-of-distribution words; secondly, it also facilitates storage and computation.

[0061] The similarity measure between in-distribution words and k out-of-distribution words is Wu & Palmer Similarity, calculated using the following formula:

[0062]

[0063] Where LCS(n) i ,n j ) indicates that it is used to determine n i and n j The function of depth(n) is the lowest common ancestor of two words. i ) is used to calculate the word n i A function of the distance to the root word.

[0064] In this embodiment of the invention, Wu & Palmer Similarity is used as the metric. The unique aspect of this choice is that this metric comprehensively considers the relationship between the two words in the word tree. Therefore, the selected out-of-distribution words are semantically very similar to the in-distribution words, making it easier for the neural network to confuse them. Furthermore, this method combines concept matching with the CLIP large model, which greatly improves the accuracy of out-of-distribution sample detection.

[0065] The word tree is represented as T =<W,R> Where W represents the set of words {w1, w2, ..., w} that constitute the word tree. n R encapsulates the relationships between these words, including parent-child relationships. p Relationship with siblings R p .

[0066] Using the CLIP model to perform concept matching between the image to be detected and words in the word tree includes:

[0067] For the image x to be detected, a large-scale pre-trained model CLIP is used to compare it with the words W = {w1, w2, ..., w} in the word tree. n } to perform matching;

[0068] For each word in the word tree, construct a concept set "This is a photo of..." <wi>", where i = 1, ..., n, and the CLIP text encoder is used. Y () Extract the corresponding text feature embeddings of these concept sets;

[0069] For the image to be detected, x, the CLIP image encoder using the CLIP model is used. I () Extract image feature embeddings from the image to be detected x;

[0070] The cosine similarity of the text feature embeddings and image feature embeddings is calculated separately to obtain the concept matching score {s(w1),s(w2),...,s(w... n )}, where s(wi)∈[-1,1], the formula is as follows:

[0071] s(w i ) = CM(x,w i )

[0072] CM(x,w i =cosione(CLIP) I (x), CLIP Y (w i ))

[0073] CLIP I (·) denotes a function used to extract image feature embeddings, CLIP Y (·) denotes the function used to extract text feature embeddings. cosine(·,·) denotes the function used to calculate the cosine similarity between two vectors.

[0074] Out-of-distribution scores are calculated based on concept matching scores, and out-of-distribution detection is performed based on these scores. The out-of-distribution detection results include:

[0075] The softmax operation is used to normalize all concept matching scores and calculate the summation score of out-of-distribution words.

[0076] The summation score is used as an indicator of the probability that the image to be detected, x, falls into an out-of-distribution class, as shown in the following formula:

[0077]

[0078] τ is the threshold. The set of words outside the distribution in the representative word tree, s(w i ) = CM(x,w i () is the cosine similarity between the feature embeddings of words in the word tree and the image feature embeddings of the image to be detected x; This refers to the cosine similarity s(w) of all out-of-distribution words in the word tree. i To calculate an e of s(w) i Raise it to the power of 1, then sum it up; It is a word w i The cosine similarity score after softmax operation represents the proportion of a word's cosine similarity to the total cosine similarity of all words in the tree. This represents the proportion of the sum of the cosine similarities of all out-of-distribution words in the word tree to the sum of the cosine similarities of all words in the word tree.

[0079] The threshold setting varies depending on the training dataset, and it is necessary to ensure that 95% of the in-distribution data is correctly classified. In practice, we use the commonly used dataset CIFAR-10 as the in-distribution dataset and test the out-of-distribution detection performance on other out-of-distribution datasets (e.g., iSUN, Texture, Places datasets).

[0080] First, search WordNet for the k most similar out-of-distribution words to the in-distribution words in CIFAR-10. Then, construct a word tree using these k out-of-distribution words, the in-distribution words, and their relationships in WordNet. Next, use the CLIP model to perform concept matching between the image to be detected and the words in the word tree. Finally, calculate an out-of-distribution score using the concept matching scores of the words in the word tree, and perform out-of-distribution detection.

[0081] See Figure 3 Another embodiment of the present invention provides an out-of-distribution detection device based on the CLIP model and word tree, comprising:

[0082] Recognition module 1 is used to identify the position of each in-distribution word in the in-distribution dataset in WordNet, and select the k out-of-distribution words in WordNet that are most similar to each in-distribution word. Here, k is an adjustable parameter, and k = 2000 can generally be selected.

[0083] Module 2 is used to construct a word tree using in-distribution words and the selected k out-of-distribution words that are most similar to each in-distribution word;

[0084] Module 3 is used to acquire the image to be detected;

[0085] Matching module 4 is used to perform concept matching between the image to be detected and words on the word tree using the CLIP model, and obtain a concept matching score;

[0086] The detection module 5 is used to calculate the out-of-distribution score based on the concept matching score, and to perform out-of-distribution detection based on the out-of-distribution score to obtain the out-of-distribution detection result.

[0087] This invention constructs a word tree by searching for the k closest out-of-distribution words in WordNet that are the closest to the in-distribution words, and combines this with the CLIP model for concept matching to calculate out-of-distribution scores for out-of-distribution detection. Compared to existing technologies, this key difference lies in its full utilization of the relationships between words in WordNet, achieving more accurate matching. Traditional methods may rely solely on surface textual information, while this invention fully explores the semantic connections between in-distribution and out-of-distribution words, resulting in more accurate matching and improved detection performance. Furthermore, the out-of-distribution detection performance of traditional methods is largely affected by the capacity of the neural network, while this invention uses the large-scale pre-trained CLIP model, which itself possesses rich knowledge. Combined with the generated word tree, the potential of the large CLIP model can be fully exploited.

[0088] Secondly, the similarity metric between in-distribution and out-of-distribution words is Wu&Palmer Similarity. This metric takes into account the relationship between in-distribution and out-of-distribution words in WordNet. The out-of-distribution words found are more likely to confuse the neural network. Combined with the concept matching function of the CLIP large model, better out-of-distribution detection accuracy can be obtained.

[0089] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.< / wi> < / wi>

Claims

1. An out-of-distribution detection method based on the CLIP model and word tree, characterized in that, include: Identify the position of each in-distribution word in the in-distribution dataset in WordNet, and select the k out-of-distribution words in WordNet that are most similar to each of the in-distribution words; A word tree is constructed using the in-distribution words and the selected k out-of-distribution words that are most similar to each of the in-distribution words; Acquire the image to be detected; The CLIP model is used to perform concept matching between the image to be detected and the words in the word tree to obtain a concept matching score. The out-of-distribution score is calculated based on the concept matching score, and out-of-distribution detection is performed based on the out-of-distribution score to obtain the out-of-distribution detection result; The step of constructing a word tree using the in-distribution words and the selected k out-of-distribution words most similar to each of the in-distribution words includes: Based on the edges presented in WordNet, establish connections between the in-distribution words and the selected k out-of-distribution words most similar to each in-distribution word, and construct a word tree; If multiple subtrees exist, identify the lowest common ancestor word of the multiple subtrees and merge them until a single word tree structure is obtained; The step of using the CLIP model to perform concept matching between the image to be detected and the words in the word tree includes: For the image x to be detected, a large-scale pre-trained model CLIP is used to compare it with the words in the word tree. Perform a match; For each word in the word tree, construct a concept set "This is a photo of ⟨w". i >”, where and using the text encoder of the CLIP model. Extract and embed the corresponding text features; For the image to be detected x, the image encoder using the CLIP model is... Extract and embed image features from the image to be detected, x. The concept matching score is obtained by calculating the cosine similarity between the text feature embedding and the image feature embedding, respectively. ,in The formula is as follows: in, This represents a function used to extract text feature embeddings. This represents a function used to extract image feature embeddings. This represents the function used to calculate the cosine similarity between two vectors.

2. The method according to claim 1, characterized in that, The similarity calculation formula between the in-distribution words and the k out-of-distribution words is as follows: Indicates used to determine and A function of the lowest common ancestor of two words. It is used to calculate subordinate terms A function of the distance to the root word.

3. The method according to claim 1, characterized in that, The word tree is represented as ,in This represents the set of words that make up the word tree. ,and It encapsulates the relationships between these words, including parent-child relationships and sibling relationships.

4. The method according to claim 1, characterized in that, The calculation of out-of-distribution scores based on the concept matching scores, and the out-of-distribution detection based on the out-of-distribution scores to obtain out-of-distribution detection results include: The softmax operation is used to normalize all concept matching scores and calculate the summation score of out-of-distribution words. The summarized score is used as an indicator of the probability that the image to be detected x falls into an out-of-distribution category, as shown in the following formula: It is a threshold. This represents the set of words distributed outside the word tree. It is the cosine similarity between the feature embeddings of words in the word tree and the image feature embeddings of the input image; This refers to the cosine similarity of all words in the word tree. To calculate an e Raise to the power of 1, then sum; It is a word The cosine similarity score after softmax operation represents the proportion of a word's cosine similarity to the total cosine similarity of all words in the word tree; This represents the proportion of the sum of the cosine similarities of all out-of-distribution words in the word tree to the sum of the cosine similarities of all words in the word tree; This indicates that the image to be detected, x, is classified as an out-of-distribution category. This indicates that the image x to be detected is classified into an in-distribution category.

5. The method according to claim 4, characterized in that, The threshold setting varies depending on the different training datasets.

6. The method according to claim 1, characterized in that, The dataset CIFAR-10 was used as the dataset within the distribution.

7. An out-of-distribution detection device based on the CLIP model and word tree, characterized in that, include: The recognition module is used to identify the position of each in-distribution word in the in-distribution dataset in WordNet, and select the k out-of-distribution words in WordNet that are most similar to each in-distribution word; The construction module is used to construct a word tree using the in-distribution words and the selected k out-of-distribution words that are most similar to each of the in-distribution words; The acquisition module is used to acquire the image to be detected; The matching module is used to perform concept matching between the image to be detected and the words in the word tree using the CLIP model to obtain a concept matching score; The detection module is used to calculate the out-of-distribution score based on the concept matching score, and to perform out-of-distribution detection based on the out-of-distribution score to obtain the out-of-distribution detection result; Using the in-distribution words and the k out-of-distribution words most similar to each in-distribution word, construct a word tree, including: Based on the edges presented in WordNet, connections are established between in-distribution words and the selected k out-of-distribution words that are most similar to each in-distribution word, and a word tree is constructed. If multiple subtrees exist, identify the lowest common ancestor word of the multiple subtrees and merge them until a single word tree structure is obtained; Using the CLIP model to perform concept matching between the image to be detected and words in the word tree includes: For the image x to be detected, a large-scale pre-trained model CLIP is used to compare it with words in the word tree. Perform a match; For each word in the word tree, construct a concept set "This is a photo of ⟨w". i >”, where and a text encoder using the CLIP model. Extract and embed the corresponding text features; For the image to be detected, x, an image encoder using the CLIP model is used. Extract and embed image features from the image to be detected, x. The concept matching score is obtained by calculating the cosine similarity between the text feature embedding and the image feature embedding, respectively. ,in The formula is as follows: in, This represents a function used to extract text feature embeddings. This represents a function used to extract image feature embeddings. This represents the function used to calculate the cosine similarity between two vectors.

Images