Zero sample anomaly detection method and device and storage medium
By combining text and image features in the anomaly detection model, the problem of image features being ignored in zero-shot anomaly detection is solved, achieving higher detection accuracy and generalization ability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-07
- Publication Date
- 2026-03-31
AI Technical Summary
Existing zero-shot anomaly detection methods focus only on text prompts and ignore image features, resulting in poor detection accuracy.
An anomaly detection model is adopted, including a text processing module, an image processing module, and a matching module. By extracting image features and text features from the image, cosine similarity is used for matching to generate a matching feature map to determine whether there are anomalies in the image.
It improves the accuracy of anomaly detection, reduces false positives and false negatives, and can be effectively applied in different scenarios.
Smart Images
Figure CN121767698A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of defect detection technology, and in particular to a zero-sample anomaly detection method, device and storage medium. Background Technology
[0002] Anomaly detection focuses on identifying deviations from normal patterns to ensure product quality in industrial production processes, playing a crucial role in various industrial scenarios. Traditional anomaly detection methods rely on manual inspection or simple image processing techniques, which are not only inefficient but also have accuracy limited by operator experience and fatigue. In recent years, deep learning-based methods have significantly improved the speed and accuracy of anomaly detection. However, the performance of these methods largely depends on large amounts of diverse and precisely labeled training data. Moreover, in most cases, a dedicated model needs to be trained for each workpiece, consuming significant computational resources. In real-world industrial scenarios, especially for specific types of anomalies, it is difficult to obtain enough anomaly samples, and labeling this data is both difficult and expensive.
[0003] Due to data limitations, many unsupervised, semi-supervised, and weakly supervised methods have been proposed and applied to anomaly detection. Unsupervised methods train the model using only normal samples, which can address the difficulty of obtaining abnormal samples to some extent. Semi-supervised methods utilize a large amount of unlabeled data and a small amount of labeled data, achieving good detection results while reducing labeling costs. Weakly supervised methods, compared to unsupervised methods, use coarse-grained labels to improve defect identification. However, these methods still require training a specific model for each scenario, resulting in low generalization. Moreover, the methods fail when training samples in the target scenario are unavailable.
[0004] To address the issue of low model generalization, zero-shot anomaly detection methods have been proposed. These methods rely on the strong generalization ability of the base model, allowing it to be directly applied to other scenarios after training in an auxiliary scenario. However, existing zero-shot anomaly detection methods focus only on text prompts, neglecting the characteristics of image features. This results in poor accuracy in anomaly detection. Summary of the Invention
[0005] The main technical problem addressed in this application is to provide a zero-shot anomaly detection method that solves the problem of poor accuracy in sample detection results caused by focusing only on text prompts and ignoring image features.
[0006] To address the aforementioned technical problems, this application provides a zero-shot anomaly detection method, comprising determining whether an anomaly exists in an image to be detected using an anomaly detection model. The anomaly detection model includes a text processing module, an image processing module, and a matching module. The text processing module extracts the semantics of target text in the image to be detected, generating personalized text prompts for each image as segmentation text features. The image processing module extracts image features from the image to be detected, generating local visual features. The matching module matches the segmentation text features with the local visual features based on the cosine similarity between the segmentation text features and the local visual features, generating a matching feature map, and determining whether an anomaly exists in the image to be detected based on the matching feature map.
[0007] In some embodiments, the text processing module includes a text prompting unit and a text encoding unit. The text prompting unit includes an abnormal text prompting block and a normal text prompting block. The abnormal text prompting block is used to detect abnormal text in the image to be detected and output abnormal text features. The normal text prompting block is used to detect normal text in the image to be detected and output normal text features. The text encoding unit is used to encode the abnormal text features and the normal text features.
[0008] In some embodiments, the text prompting unit further includes a transformation prompting block, which is used to perform a non-linear transformation on the image prompting features extracted by the image processing module to generate an offset embedding. The offset embedding is superimposed on the target text to obtain transformed text features. The transformed text features, together with the abnormal text features and the normal text features, are encoded by the text encoding unit to generate segmented text features and classified text features.
[0009] In some embodiments, the image processing module includes an image prompting unit, which is used to extract features from the image to be detected and output image prompting features. The image prompting features are input to the transformation prompting block, and the transformation prompting block performs a nonlinear transformation on the image prompting features.
[0010] In some embodiments, the image processing module further includes an image encoding unit, which includes multiple pairs of image attention blocks and image convolution blocks. The image attention blocks are used to extract information of samples in the image to be detected. The image attention blocks include multiple attention layers. The output of the last attention layer is input into the image convolution block, which outputs local visual features. The local visual features are input into the next image attention block and matching module.
[0011] In some embodiments, the matching module receives the local visual features and the segmented text features, calculates the cosine similarity between the local visual features and the segmented text features, matches the segmented text features with the local visual features, generates a matching feature map, and determines whether there is an anomaly in the image to be detected based on the matching feature map.
[0012] The matching module is represented as follows:
[0013]
[0014]
[0015]
[0016] in, This represents a correlation matching graph between the normal text features and the local visual features. This refers to the image to be detected. This represents the vertical pixel count of the image to be detected. represents the number of horizontal pixels in the image to be detected; i represents an image patch in the image to be detected; CosSim represents similarity calculation. This represents the normal text features. This represents the local visual features. This represents a correlation matching graph between the abnormal text features and the local visual features. This indicates the abnormal text features. Represents text and visual feature maps.
[0017] In some embodiments, the matching module includes an adaptation unit with multiple dilated convolutional layers. The adaptation unit is used to optimize the local visual features and generate a visual and visual feature map. The text and visual feature map and the visual and visual feature map are superimposed to form a matching feature map.
[0018] In some embodiments, the total loss function for training the anomaly detection model includes a classification loss function and a segmentation loss function, expressed as follows:
[0019]
[0020] In the formula, Represents the total loss function, Represents the classification loss function. Represents the segmentation loss function;
[0021] The classification function is obtained by the cross-entropy loss between the classification result and the true label, and is expressed as:
[0022]
[0023] in, Represents cross-entropy loss, Indicates the classification result. Indicates the true label;
[0024] The classification result is expressed as follows:
[0025]
[0026] in, This represents the text features of the classification. Represents the segmented text features;
[0027] The segmentation loss function is expressed as:
[0028]
[0029] Where GT represents the true segmentation label. and These represent the Focal loss function and the Dice loss function, respectively. This indicates the prediction result.
[0030] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.
[0031] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described.
[0032] The beneficial effects of this application are as follows: The text processing module can accurately reflect the semantics of the target text in the image. The segmented text features generated after enhancement can accurately reflect the information in the image to be detected, thereby more accurately judging whether the samples in the image to be detected meet the standards and effectively reducing the false detection and false negative rates. The image processing module generates local visual features, and the matching module then matches the segmented text features with the local visual features based on cosine similarity to generate a matching feature map. This comprehensively considers both text semantics and image visual information, enabling the capture of abnormal features in the image to be detected from different angles. This ensures the accuracy of anomaly detection in the image to be detected. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the framework of an anomaly detection model according to an embodiment of this application;
[0034] Figure 2This is a schematic diagram of the framework of a matching module according to an embodiment of this application;
[0035] Figure 3 This is a schematic diagram comparing the test results according to an embodiment of this application. Detailed Implementation
[0036] To facilitate understanding of this application, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of this application are shown in the drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application.
[0037] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.
[0038] Figures 1-3 An embodiment of the zero-sample anomaly detection method of this application is shown, including:
[0039] An anomaly detection model is used to determine whether there are anomalies in the image to be detected. The anomaly detection model includes a text processing module 1, an image processing module 2, and a matching module 3.
[0040] The text processing module 1 is used to extract the semantics of the target text in the image to be detected and generate personalized text prompts for each image to be detected as text segmentation features;
[0041] Image processing module 2 is used to extract image features from the image to be detected and generate local visual features;
[0042] The matching module 3 matches the segmented text features with the local visual features based on the cosine similarity between the segmented text features and the local visual features to generate a matching feature map, and determines whether there are any abnormalities in the image to be detected based on the matching feature map.
[0043] This application considers the extreme case where no information about the detection sample is available, i.e., no relevant text or image information about the workpiece is available. Furthermore, to address complex anomaly detection scenarios, the text processing module 1 accurately reflects the semantics of the target text in the image. The enhanced segmented text features accurately reflect the information in the image to be detected, thus more accurately determining whether the sample in the image meets the standard and effectively reducing false positives and false negatives. The image processing module 2 generates local visual features, and the matching module 3 then matches the segmented text features with the local visual features based on cosine similarity to generate a matching feature map. This comprehensive consideration of text semantics and image visual information enables the capture of anomaly features in the image to be detected from different angles, thereby ensuring the accuracy of anomaly detection in the image.
[0044] In some embodiments, the text processing module 1 includes a text prompting unit 11 and a text encoding unit 12. The text prompting unit 11 includes an abnormal text prompting block 111 and a normal text prompting block 112. The abnormal text prompting block 111 is used to detect abnormal text in the image to be detected and output abnormal text features. The normal text prompting block 112 is used to detect normal text in the image to be detected and output normal text features. The text encoding unit 12 is used to encode the abnormal text features and normal text features to generate segmentation text features and classification text features.
[0045] In this application, the text processing module 1 extracts the target text from the image to be detected and generates personalized text prompts for each image to be detected. The segmentation text features and classification text features can be set with different semantic fine granularities to be efficiently applied to different application scenarios. The segmentation text features and classification text features can also be effectively fused with local visual features to improve the accuracy of zero-shot detection.
[0046] In some embodiments, the exception text prompt block 111 is represented as follows:
[0047]
[0048] in, Indicates abnormal text features, Let E represent the learnable embedding vectors of the abnormal context, where E is the total number of learnable embedding vectors of the context. It is learnable; the model learns the contextual information of the text prompts while training on the training dataset. This represents the embedding of corrupted text. Corrupted text is used to distinguish between normal text features and anomalous text features. This indicates the embedding of the target text, which represents all the workpiece names, thus enabling zero-sample anomaly detection when the text information is unknown.
[0049] In some embodiments, the normal text prompt block 112 is represented as follows:
[0050]
[0051] in, Indicates normal text features. This indicates that the embedding vector can be learned from the normal context. It is learnable; the model learns the contextual information of the text prompts while training on the training dataset.
[0052] The semantic meaning of words in the target text is not strictly consistent with the semantic meaning of the specific target product, and it is not combined with the image. Using general terms may lead to the loss of some semantic information, thus limiting the performance of the model.
[0053] To address this issue, the text prompting unit 11 further includes a transformation prompting block 113. The transformation prompting block 113 performs a non-linear transformation on the image prompting features extracted by the image processing module 2 to generate an offset embedding. This offset embedding is then superimposed on the target text to obtain transformed text features. These transformed text features, along with abnormal and normal text features, are encoded together by the text encoding unit 12 to generate segmented and categorized text features. This allows for the combination of image prompting features to enhance the target text, thereby further improving the accuracy of anomaly detection.
[0054] In some embodiments, the transformation prompt block 113 may employ a multilayer perceptron network (MLP), a recurrent neural network, or an extreme learning machine, etc. The transformation prompt block 113 is used to perform a nonlinear transformation on the image prompt features extracted by the image prompt unit 21 to obtain the offset embedding of each workpiece. The offset embedding is then superimposed with the embedding of the target text to obtain the transformed text features. This is represented as:
[0055]
[0056] in, Indicates transformed text features. Indicates offset embedding,
[0057] After adding the transformation hint block, the abnormal text features are represented as follows:
[0058]
[0059] Normal text features are represented as follows:
[0060]
[0061] Abnormal text features, normal text features, and transformed text features are input into the text encoding unit; the text encoding unit processes the input abnormal text features, normal text features, and transformed text features, and outputs segmented text features and classified text features. The text encoding unit can be a deep neural network with a Transformer architecture, such as an autoregressive decoder architecture, a bidirectional Transformer encoder, etc.
[0062] In some embodiments, the image processing module 2 includes an image prompting unit 21 and an image encoding unit 22. The image prompting unit 21 is used to extract features from the image to be detected and output image prompting features. The image prompting features are input into the transformation prompting block 113. The transformation prompting block 113 performs a nonlinear transformation on the image prompting features.
[0063] The image prompting unit 21 can be a deep neural network with a Vision Transformer or Vision Encoder architecture.
[0064] The image encoding unit 22 includes multiple pairs of image attention blocks 221 and image convolution blocks 222. The input of the image attention block 221 is the image to be detected. The image attention block 221 is used to extract information of samples in the image to be detected. The image attention block 221 includes multiple attention layers. The output of the last attention layer is input into the image convolution block 222. The image convolution block 222 outputs local visual features. The local visual features are input into the next image attention block 221 and the matching module 3.
[0065] The multi-layer attention layer adopts a V (value)-V (value)-V (value) attention mechanism and freezes parameters, instead of the traditional Q (query)-K (key)-V (value) attention mechanism. The VVV attention mechanism can focus on more fine-grained local visual semantics and is more suitable for anomaly segmentation.
[0066] In some embodiments, the matching module 3 receives local visual features and segmented text features, calculates the cosine similarity between them, matches the segmented text features with the local visual features to generate a matching feature map, and determines whether there is an anomaly in the image to be detected based on the matching feature map.
[0067] The matching module obtains text and visual feature maps. Represented as follows:
[0068]
[0069]
[0070]
[0071] in, This represents a matching graph showing the correlation between normal text features and local visual features. This represents the image to be detected. This represents the vertical pixel count of the image to be detected. This represents the number of horizontal pixels in the image to be detected; i represents the image patch in the image to be detected; and CosSim represents the similarity calculation. Indicates normal text features. Indicates local visual features, A matching graph representing the correlation between anomalous text features and local visual features. Indicates abnormal text features, Represents text and visual feature maps.
[0072] Text and visual feature maps can be used as matching feature maps, but they overlook an important characteristic of local visual features: visually similar image patches are often densely distributed in the feature space, which results in poor segmentation performance.
[0073] To solve this problem, an adaptation unit 31 is set in the matching module 3, such as... Figure 2 As shown, the adaptation unit 31 is used to optimize local visual features. The adaptation unit 31 generates visual and visual feature maps, and the text and visual feature maps and visual and visual feature maps together form a matching feature map, thereby enhancing the model's ability to perceive the context of abnormal regions and helping to obtain accurate matching feature maps.
[0074] The adaptation unit 31 distinguishes abnormal regions by utilizing the differences between abnormal and normal regions. Since abnormal regions are clearly different from other normal regions, the similarity between the features of abnormal image patches and even their most similar image patches is very low, while the cosine similarity between normal image patches is relatively high. This difference in similarity can be used to distinguish between normal and abnormal regions. The adaptation unit 31 has multiple dilated convolutional layers, with dilation rates that can be set to 1, 2, and 4. Through the adaptation unit 31, local visual features can be optimized, expanding the receptive field of view while enabling the module to capture multi-scale contextual information.
[0075] In some embodiments, the adapter unit 31 is represented as follows:
[0076]
[0077]
[0078] in, The local visual features adjusted by the adaptation unit are denoted as the adjusted visual features. Indicates the location The characteristics of the place, Indicates that, apart from the position External feature set, , This indicates the operation of retrieving the maximum value. Represents visual perception and visual feature maps.
[0079] The matching feature map is obtained by superimposing the text and visual feature maps and the visual and visual feature maps, and is represented as follows:
[0080]
[0081] in, Represents the matching feature map. It is a parameter value used to balance the weight of text and visual feature maps and visual features.
[0082] In some embodiments, the anomaly detection model can be trained using a training dataset before use. The training dataset can contain images with different samples. After training, it can be directly used for detecting zero-sample images. During training, the total loss function of the anomaly detection model includes a classification loss function and a segmentation loss function, expressed as:
[0083]
[0084] In the formula, Represents the total loss function. Represents the classification loss function. This represents the segmentation loss function.
[0085] The classification function is obtained by the cross-entropy loss between the classification result and the true label, and is expressed as:
[0086]
[0087] in, Represents cross-entropy loss, This indicates the classification results, which include normal classification scores and abnormal classification scores. This indicates the actual label.
[0088] The classification results are represented as follows:
[0089]
[0090] in, Representing the features of categorized text, This represents the features of the segmented text.
[0091] In some embodiments, the segmentation task first superimposes the mask prediction results obtained from different layers, then upsamples the superimposed result to obtain a final prediction result of the same size as the image, and then combines the Focal loss function and the Dice loss function to achieve a balance between detail and comprehensiveness in the segmentation result.
[0092] The segmentation loss function is expressed as:
[0093]
[0094] Here, GT represents the true segmentation label. and These represent the Focal loss function and the Dice loss function, respectively. This indicates the prediction result.
[0095] Figure 3 The first row shows the image to be detected, containing items such as pills, screws, cables (cross-section), zippers, capsules, chewing gum, mesh, tiles, wood, and hazelnuts. The second row shows the detection results derived from ground truth. The third row shows the detection results obtained using an existing zero-shot detection framework. The fourth row shows the detection results obtained using this application. Figure 3 As can be seen from the results, the abnormal areas obtained by this application are closer to the true annotations, and the abnormality is more significant, indicating that it performs better in terms of accuracy and visual significance in anomaly detection.
[0096] The text processing module in this application accurately reflects the semantics of the target text in the image. The enhanced segmented text features accurately reflect the information in the image to be detected, thus more accurately determining whether the samples in the image meet the standards and effectively reducing the false detection and false negative rates. The image processing module generates local visual features, and the matching module then matches the segmented text features with the local visual features based on cosine similarity to generate a matching feature map. This comprehensively considers both text semantics and image visual information, enabling the capture of abnormal features in the image to be detected from different angles, thereby ensuring the accuracy of anomaly detection in the image.
[0097] The above are merely embodiments of this application and do not limit the scope of this patent application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.
Claims
1. A zero-shot anomaly detection method, characterized in that, The method comprises determining whether an anomaly exists in the to-be-detected image by an anomaly detection model, the anomaly detection model comprising a text processing module, an image processing module, and a matching module; The text processing module is configured to extract semantics of target text in the to-be-detected image, and generate a text prompt personalized for each to-be-detected image as a segmented text feature; The image processing module is configured to extract image features in the to-be-detected image, and generate local visual features; The matching module is configured to match the segmented text feature and the local visual feature based on cosine similarity between the segmented text feature and the local visual feature, generate a matching feature map, and determine whether an anomaly exists in the to-be-detected image based on the matching feature map.
2. The zero-sample anomaly detection method of claim 1, wherein, The text processing module comprises a text prompt unit and a text encoding unit, the text prompt unit comprises an abnormal text prompt block and a normal text prompt block, the abnormal text prompt block is configured to detect abnormal text in the to-be-detected image and output abnormal text features, the normal text prompt block is configured to detect normal text in the to-be-detected image and output normal text features, and the text encoding unit is configured to encode the abnormal text features and the normal text features.
3. The zero-sample anomaly detection method of claim 2, wherein, The text prompt unit further comprises a transformation prompt block, the transformation prompt block is configured to perform nonlinear transformation on image prompt features extracted by the image processing module, generate offset embeddings, superimpose the offset embeddings on the target text to obtain transformed text features, and encode the transformed text features, the abnormal text features, and the normal text features together by the text encoding unit to generate segmented text features and classified text features.
4. The zero-sample anomaly detection method of claim 3, wherein, The image processing module comprises an image prompt unit, the image prompt unit is configured to extract features in the to-be-detected image and output image prompt features, and the image prompt features are input to the transformation prompt block, and the transformation prompt block performs nonlinear transformation on the image prompt features.
5. The zero-sample anomaly detection method of claim 4, wherein, The image processing module further comprises an image encoding unit, the image encoding unit comprises multiple pairs of image attention blocks and image convolution blocks, the image attention blocks are configured to extract information of samples in the to-be-detected image, the image attention blocks comprise multiple attention layers, an output result of a last attention layer is input to the image convolution blocks, and local visual features are output by the image convolution blocks, the local visual features are input to a next image attention block and a matching module.
6. The zero-shot anomaly detection method of claim 5, wherein The matching module receives the local visual features and the segmented text features, calculates cosine similarity between the local visual features and the segmented text features, matches the segmented text features and the local visual features to generate a matching feature map, and determines whether an anomaly exists in the to-be-detected image based on the matching feature map; The matching module is represented as follows: wherein, represents a relevance matching graph of the normal text feature and the local visual feature, represents the image to be detected, represents the number of vertical pixels of the image to be detected, represents the number of horizontal pixels of the image to be detected; i represents an image block in the image to be detected, and CosSim represents similarity calculation, represents the normal text feature, represents the local visual feature, represents a relevance matching graph of the abnormal text feature and the local visual feature, represents the abnormal text feature, represents a text and visual feature graph.
7. The zero-sample anomaly detection method of claim 6, wherein, The matching module comprises a setting adaptation unit, and the adaptation unit comprises multiple dilated convolution layers.
8. The zero-sample anomaly detection method of claim 1, wherein, The total loss function for training the anomaly detection model comprises a classification loss function and a segmentation loss function, denoted as: wherein denotes the total loss function, denotes the classification loss function, denotes the segmentation loss function; The classification function is obtained by cross-entropy loss of the classification result and the true label, denoted as: wherein, denotes the cross-entropy loss, denotes the classification result, denotes the true label; The classification result is denoted as: wherein, represents the classification text feature, represents the segmentation text feature; The segmentation loss function is denoted as: wherein GT represents a true segmentation label, and respectively represent a Focal loss function and a Dice loss function, represents a prediction result.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method in any one of claims 1 to 8.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method in any one of claims 1 to 8.