Anomaly detection methods based on few-shot cue learning
By employing a few-shot cue learning method, the training data is cropped and segmented using a visual language model to construct in-distribution and out-of-distribution data. This solves the problem that the model has difficulty identifying out-of-distribution samples, achieving powerful anomaly detection capabilities with limited data and improving the reliability of safety-critical applications.
Patent Information
- Application Number
- CN202311372850.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-23
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-23
AI Technical Summary
Existing anomaly detection models struggle to identify out-of-distribution samples in open worlds, potentially leading to erroneous decisions, especially in safety-critical applications. Furthermore, few-shot learning faces the challenge of insufficient sample size.
We employ a few-shot cue learning method, which constructs out-of-distribution and in-distribution data by randomly cropping and partitioning the training data. We then use a pre-trained visual language model to learn cueing for out-of-distribution data and identify out-of-distribution samples.
It achieves powerful anomaly detection capabilities with limited in-distribution data, improves the model's ability to identify anomalies on out-of-distribution data, and enhances the reliability of safety-critical applications.
Smart Images

Figure CN117611870B_ABST
Abstract
Description
Technical Field
[0001] This invention discloses an anomaly detection technique based on few-shot cue learning, relating to the fields of few-shot learning, cue learning, and pre-trained visual language models. Specifically, it relates to an anomaly detection method based on few-shot cue learning. Background Technology
[0002] In recent years, anomaly detection has become a core challenge in the secure deployment of machine learning models in open worlds, where models may encounter out-of-distribution samples that were not visible during training. These samples can lead to erroneous decisions and serious safety issues, particularly in safety-critical applications such as autonomous driving and medical diagnostics. Therefore, models should not only perform well on in-distribution samples but also be able to identify out-of-distribution samples.
[0003] In anomaly detection tasks, common algorithms explore how to train with large amounts of auxiliary anomaly data to regularize the model, enabling it to produce lower confidence and higher energy on out-of-distribution data. Recent state-of-the-art algorithms, such as the MSP and Energy models, have achieved advanced performance based on these techniques.
[0004] Few-shot learning is an emerging technique that aims to learn from a small number of supervised samples. Typically, model training requires a large amount of data, but acquiring large amounts of labeled data is very costly. To address this, few-shot learning was proposed. Few-shot learning usually revolves around data augmentation or how to constrain the model. The difficulty of few-shot learning lies in the small number of samples, which makes empirical risk minimization unreliable. The simplest solution is data augmentation, which, as the name suggests, aims to increase the amount of data available. Data augmentation methods include manual rule-based preprocessing of images (such as inversion, rotation, and deformation) or using generative adversarial networks (GANs) to generate data similar to real data. For constraining the model, a common approach is to use a large-scale pre-trained model, with its parameters as prior hypotheses, and then fine-tune the parameters to achieve better generalization performance.
[0005] Cue learning originates from natural language processing and is summarized as a fill-in-the-blank-like task. Its basic idea is to induce a pre-trained language model to generate answers based on fill-in-the-blank style prompts, which benefits many downstream tasks, such as sentiment analysis. Although cue learning has been widely applied in natural language processing, its exploration in the visual domain remains insufficient. Recently, CoOp implemented cue learning based on a pre-trained visual language model. This method utilizes a pre-trained visual language model to construct a learnable tensor from the input of the text encoder and updates the corresponding prompts through the loss between the image and the cue, thus achieving cue learning in the visual domain. Summary of the Invention
[0006] To overcome the shortcomings of existing technologies, and addressing the difficulty of identifying challenging anomaly samples with limited training data, this invention aims to propose an anomaly detection technique based on few-shot cue learning. The technical solution adopted in this invention is an anomaly detection method based on few-shot cue learning. This method constructs in-distribution and out-of-distribution data by randomly pruning and dividing the samples. The out-of-distribution data is treated as anomaly data, and a pre-trained visual language model is used to learn cue messages from the anomaly data, thereby helping the visual language model to identify challenging out-of-distribution samples.
[0007] The specific steps are as follows:
[0008] 101: For the data used for training The data is processed, where x represents a set of image data and y represents the category name corresponding to each image. The images are cropped multiple times and divided into in-distribution data. and out-of-distribution data "in" indicates in-distribution (within the distribution), and "out" indicates out-of-distribution (outside the distribution). x in and x out Each of these represents a subset constructed from x, and both are sets of images. in and y out These are the corresponding tag names. The specific steps are as follows:
[0009] 1011: Obtain a set of images {x1, x2, ... x} by performing multiple random cropping operations on a sample x. m};
[0010] 1012: Use an image encoder to encode the cropped samples to obtain a set of image features. img represents the features of an image;
[0011] 1013: Convert the class name corresponding to the sample into the corresponding text description "a photo of [class name]";
[0012] 1014: Input the text description into the text encoder to obtain the corresponding text features z txt txt represents the characteristics of text;
[0013] 1015: Calculate z txt and The cosine similarity between them is given by the formula: <z img ,z txt >=z img ·z txt / ‖z img |||z txt ‖, obtain a set of similarity scores {s1,s2,…s m};
[0014] 1016: Based on the similarity score, select the M with the highest score. in Among the samples in the distribution, M has the lowest score. out 1. 2. 3. 4. 5. 6. 7. 8. 8. 9. 10 ...
[0015] 1017: Perform the above operation on each sample to obtain in-distribution data. and out-of-distribution data
[0016] 102: Set different numbers of in-distribution and out-of-distribution cuees respectively, and calculate the loss using the in-distribution and out-of-distribution data constructed in step 101. λ is a hyperparameter, representing the weights of the loss function;
[0017] The process for calculating the loss in step 102 is as follows:
[0018] 1021: Input in-distribution and out-of-distribution cues into the text encoder to obtain two sets of text features. txt,in represents the text features of the in-distribution prompts, txt,out represents the text features of the out-of-distribution prompts, c represents the number of in-distribution categories, and k represents the number of out-of-distribution prompts.
[0019] 1022: Input the constructed in-distribution and out-of-distribution images into the image encoder to obtain two sets of image features. img,in represents the image features of in-distribution data, and img,out represents the image features of out-of-distribution data;
[0020] 1023: Obtain the cosine similarity between text features and image features, and record the similarity for each image. With all text The similarity between them was calculated, resulting in a set of similarity values as follows:
[0021] 1024: For those belonging to Data, execution loss
[0022]
[0023] exp represents exponential operation, and log represents logarithmic operation;
[0024] 1025: For those belonging to The sample, execution loss
[0025]
[0026] 1026: Regarding Execution loss:
[0027]
[0028] 103: Update model parameters using gradient descent;
[0029] 104: During the test, follow the steps 1021 to 1023 in section 102 to obtain the similarity score of the test image.
[0030] 105: Used for classification tasks The label corresponding to the maximum value in the middle is used for classification prediction, and the predicted probability is...
[0031] 106: For anomaly detection tasks, the percentage of out-of-distribution cues in the sum of scores is used as the score, S = ∑exp(s in ) / (∑exp(s in )+∑exp(s out The higher the score, the greater the probability that the data belongs to the in-distribution category; conversely, the lower the score, the greater the probability that the data belongs to the out-of-distribution category. The score is used to determine whether the sample is abnormal data.
[0032] Specifically The dataset consists of mammograms, where x represents a set of mammogram images containing both normal and known breast cancer cases, and y represents the category name for each image, such as normal or abnormal. The similarity score of the test image relative to different prompts is obtained through the aforementioned steps. in This represents the similarity score between the test results and known different types of breast images, used for classification; This indicates the score of whether the test image is likely an anomalous image, through... Determine whether the test sample is a previously unseen case.
[0033] Training data for autonomous driving systems The corresponding x represents a set of road condition image data, and y represents the category name corresponding to each image, including various driving scenarios of urban, highway and rural roads, as well as various weather conditions. The model is trained using the training method mentioned above, and a similarity score of a test scene relative to different prompts is obtained during testing. in This represents the similarity score between the test sample and known road conditions, and classification is performed based on the similarity score. This represents the score when the test sample is an unknown road condition.
[0034] The features and beneficial effects of this invention are:
[0035] 1. This invention is an anomaly detection technique based on few-sample cue learning, which requires only a small amount of in-distribution data to achieve powerful anomaly detection capabilities.
[0036] 2. This invention utilizes a pre-trained visual language model to construct a large amount of difficult-to-identify abnormal data from a small number of in-distribution samples, thereby aiding in model training.
[0037] 3. This invention has achieved excellent results on multiple standard datasets, such as ImageNet1k (iNaturalist, SUN, PLACES, TEXTURE), and even under unfair comparisons, it still surpasses most current methods. Attached image description:
[0038] Figure 1 This is a flowchart illustrating the training of the model and the use of the model for prediction in this invention.
[0039] Figure 2 This is a flowchart of the anomaly detection method of the present invention. Detailed Implementation
[0040] This invention proposes an anomaly detection technique based on few-shot cue learning. This method achieves good anomaly detection performance by mining information from a small number of in-distribution samples. Specifically, this invention uses a pre-trained visual language model to crop and identify a small number of in-distribution samples, thereby obtaining better in-distribution and out-of-distribution samples. The out-of-distribution samples are then used as difficult anomaly samples to help the model train and learn their corresponding cueing, thus obtaining better anomaly detection capabilities.
[0041] The specific steps of this invention are as follows:
[0042] An anomaly detection method based on few-shot cue learning, comprising the following steps:
[0043] 101: For the data used for training The image is processed by cropping it multiple times and dividing it into in-distribution data. and out-of-distribution data The specific steps are as follows:
[0044] 1011: Obtain a set of images {x1, x2, ... x} by performing multiple random cropping operations on a sample x. m}
[0045] 1012: Use an image encoder to encode the cropped samples to obtain a set of image features.
[0046] 1013: Convert the class name corresponding to the sample into the corresponding text description "a photo of [class name]".
[0047] 1014: Input the text description into the text encoder to obtain the corresponding text features z txt .
[0048] 1015: Calculate z txt and The cosine similarity between them is given by the formula: <z img ,z txt =z img ·z txt ||z img |||z txt ‖, obtain a set of similarity scores {s1,s2,…s m}
[0049] 1016: Based on the similarity score, select the M with the highest score. in Among the samples in the distribution, M has the lowest score. out One sample is taken as an out-of-distribution sample.
[0050] 1017: Perform the above operation on each sample to obtain in-distribution data. and out-of-distribution data
[0051] 102: Set different numbers of in-distribution and out-of-distribution cuees respectively, and calculate the loss using the in-distribution and out-of-distribution data constructed in step 101.
[0052] The process for calculating the loss in step 102 is as follows:
[0053] 1021: Input in-distribution and out-of-distribution cues into the text encoder to obtain two sets of text features.
[0054] 1022: Input the constructed in-distribution and out-of-distribution images into the image encoder to obtain two sets of image features.
[0055] 1023: Obtain the cosine similarity between text features and image features, and record the similarity between each image and all texts.
[0056] 1024: For those belonging to Data, execution loss
[0057] 1025: For those belonging to The sample, execution loss
[0058] 1026: Regarding Execution loss
[0059] 103: Update model parameters using gradient descent.
[0060] 104: During the test, follow the steps 1021 to 1023 in section 102 to obtain the similarity score of the test image.
[0061] 105: Used for classification tasks The label corresponding to the maximum value in the middle is used as the classification prediction.
[0062] 106: For anomaly detection tasks, the percentage of out-of-distribution cues in the sum of scores is used as the score, S = ∑exp(s in ) / ∑exp(s in )+∑exp(s out ).
[0063] 107: The experiment details for ImageNet1k (iNaturalist, SUN, PLACES, TEXTURE) are as follows:
[0064] 1071: Experimental Environment. Hardware configuration: 3×RTX 3090; programming language and version: Python 3.8; open source environment: Anaconda3; deep learning framework: Pythorch; torch and torchvision versions: 1.4 and 0.5, respectively.
[0065] 1072: Data sources: The data used in the laboratory, ImageNet1k, iNaturalist, SUN, PLACES, and TEXTURE, are all publicly available RGB datasets. The pre-trained model CLIP used is also a publicly available model.
[0066] 1073: Experimental Procedure. First, hardware computing resources and a software development environment were acquired, including a GPU and Anaconda environment. Then, code development was performed using the PyTorch deep learning framework, including building the network structure, reading training data, preprocessing the data, and setting training parameters. Finally, the trained model was evaluated using a test set and compared with comparative methods.
[0067] 1074: Comparison Method. Currently advanced anomaly detection algorithms are selected as benchmark methods for comparison.
[0068] 1075: The experimental results are shown in the table below:
[0069]
[0070] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.
[0071] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.
[0072] Example 1
[0073] When performing medical image recognition tasks, the training data provided consists of mammograms from multiple hospitals. x represents a set of X-ray images of the breast, including normal cases and known cases of breast cancer, and y represents the category name for each image, such as normal or abnormal. However, in actual testing, variants of breast cancer or new breast diseases may appear, and these cases may not be in the training data. In this case, the model is trained using the training method mentioned above, and during testing, a similarity score of the test image relative to different prompts can be obtained. in This represents the similarity score between the test results and known different types of breast images, used for classification. This indicates the score of whether the test image is likely an anomalous image. (Through...) We can determine whether a test sample is a previously unseen case, thereby reducing misdiagnosis.
[0074] Example 2
[0075] Training data for autonomous driving systems is used to identify different road conditions in autonomous driving tasks. x represents a set of road condition image data, and y represents the category name corresponding to each image, including various driving scenarios such as urban, highway, and rural roads, as well as various weather conditions (sunny, rainy, snowy, etc.). However, during testing, autonomous vehicles may encounter unknown road conditions, such as unfamiliar geographical environments, road types (e.g., forest trails or construction sites), or extreme weather conditions (e.g., sandstorms). In this case, by training the model using the training method mentioned above, a similarity score of a test scenario relative to different prompts can be obtained during testing. in This represents the similarity score between the test sample and known road conditions, and classification is performed based on the similarity score. This represents the score for test samples that may involve unknown road conditions. When the unknown score exceeds a threshold, driving strategies can be adjusted, such as slowing down, pausing, requesting manual intervention, or following general safety rules. This reduces traffic accidents caused by misjudgments and improves driving safety and reliability.
[0076] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.
Claims
1. An anomaly detection method based on few-shot cue learning, characterized in that, By randomly cropping and dividing the samples, in-distribution and out-of-distribution data are constructed. The out-of-distribution data is regarded as outliers, and a pre-trained visual language model is used to learn the clues of the outliers, thereby helping the visual language model to identify difficult out-of-distribution samples. The specific steps are as follows: 101: For the data used for training Processing is carried out, among which Represented as a set of image data, This represents the category name corresponding to each image. The images are cropped multiple times and divided into in-distribution data. and out-of-distribution data This indicates out-of-distribution, meaning outside the distribution. and They are from A subset constructed from [the existing set], which is also a collection of images. and These are the corresponding tag names. The specific steps are as follows: 1011: Take a sample A set of images is obtained by performing multiple random cropping operations. 1012: Use an image encoder to encode the cropped samples to obtain a set of image features. img represents image features 1013: Convert the class name corresponding to this sample into the corresponding text description "a photo of [class name]"; 1014: Input the text description into the text encoder to obtain the corresponding text features. ; 1015: Calculation and The cosine similarity between them is given by the formula: Obtain a set of similarity scores 1016: Based on similarity scores, select the highest-scoring... The lowest score among the samples within the distribution.
1.
2.
3.
4.
5.
6.
7.
8.
8.
9. 10 ... 1017: Perform the above operation on each sample to obtain in-distribution data. and out-of-distribution data 102: Set different numbers of in-distribution and out-of-distribution cuees respectively, and calculate the loss using the in-distribution and out-of-distribution data constructed in step 101. The process for calculating the loss in step 102 is as follows: 1021: Input in-distribution and out-of-distribution cues into the text encoder to obtain two sets of text features. txt,in represents the text features of in-distribution prompts, txt,out represents the text features of out-of-distribution prompts, c represents the number of in-distribution categories, and k represents the number of out-of-distribution prompts. 1022: Input the constructed in-distribution and out-of-distribution images into the image encoder to obtain two sets of image features. img,in represents the image features of in-distribution data, and img,out represents the image features of out-of-distribution data. 1023: Obtain the cosine similarity between text features and image features, and record the similarity for each image. With all text The similarity between them was calculated, resulting in a set of similarity values as follows: ; 1024: For those belonging to Data, execution loss exp represents exponential operation, and log represents logarithmic operation; 1025: For those belonging to The sample, execution loss 1026: Regarding Execution loss: 103: Update model parameters using gradient descent; 104: During the test, follow the steps 1021 to 1023 in section 102 to obtain the similarity score of the test image. ; 105: Used for classification tasks The label corresponding to the maximum value in the middle is used for classification prediction, and the predicted probability is... 106: For anomaly detection tasks, the percentage of out-of-distribution cues in the sum of scores is used as the score. The higher the score, the greater the probability that the data belongs to the in-distribution category; conversely, the lower the score, the greater the probability that the data belongs to the out-of-distribution category. The score is used to determine whether the sample is abnormal data.
2. The anomaly detection method based on few-shot cue learning as described in claim 1, characterized in that, Specifically For mammogram datasets, the corresponding This is represented as a set of X-ray images of the breast, including normal cases and known cases of breast cancer. This represents the category name corresponding to each image, including normal or abnormal. The similarity score of the test image relative to different prompts is obtained through the aforementioned steps. ,in This represents the similarity score between the test results and known different types of breast images, used for classification; This indicates the score of whether the test image is likely an anomalous image, through... To determine whether the test sample is a previously unseen case.
3. The anomaly detection method based on few-shot cue learning as described in claim 1, characterized in that, Training data for autonomous driving systems Correspondingly This is represented as a set of image data of road conditions. This represents the category name corresponding to each image, including various driving scenarios such as urban, highway, and rural roads, as well as various weather conditions. The model is trained using the training method mentioned above, and during testing, a similarity score of a test scene relative to different prompts is obtained. ,in This represents the similarity score between the test sample and known road conditions. Classification is performed based on this similarity score. , which represents the score when the test sample is an unknown road condition.
