Zero sample anomaly detection method based on bimodal prompt enhancement
Through the collaborative optimization of the PIM and PMB modules, the problems of modal semantic bias and insufficient fine-grained representation in zero-sample anomaly detection are solved, achieving higher detection accuracy and generalization ability, which is suitable for industrial quality inspection and medical imaging diagnosis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-17
- Publication Date
- 2026-04-03
AI Technical Summary
Existing technologies suffer from modal semantic bias and insufficient fine-grained representation capabilities in zero-shot anomaly detection, resulting in poor generalization ability of the model in unknown scenarios and the easy oversight of minor defects.
A bimodal cue enhancement approach is adopted, which uses the PIM module to realize dynamic cross-modal interaction and the PMB module to build a hierarchical anomaly prototype library. Combined with uncertainty perception scoring, the fine-grained capture capability of visual features is improved.
It significantly improves the model's generalization ability and fine-grained detection accuracy in unknown scenarios, with significant improvements in both image-level AUROC and pixel-level AUPRO metrics, making it suitable for industrial quality inspection and medical image diagnosis.
Smart Images
Figure CN121788901A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a zero-shot anomaly detection method based on bimodal cue enhancement, belonging to the field of zero-shot anomaly detection (ZSAD). Background Technology
[0002] CLIP models construct a shared semantic space through massive image-text comparison learning, mapping images and text to the same embedding space and achieving zero-shot classification with the help of natural language prompts. Representative works include CLIP (Radford et al., 2021), WinCLIP (Jeong et al., 2023), and AnomalyCLIP (Zhou et al., 2023). These methods achieve anomaly detection by optimizing text prompts to align visual and textual features.
[0003] However, existing technologies have the following key problems:
[0004] 1. Modal semantic bias: The global semantic alignment mechanism during CLIP pre-training struggles to capture fine-grained features of local anomalies (such as microscopic defects like scratches and cracks). Existing methods often rely on single text prompts for optimization, lacking dynamic cross-modal interaction, resulting in insufficient matching between text prompts and visual targets.
[0005] 2. Weak fine-grained representation capability: Feature comparison relies heavily on single-level comparison mechanisms (such as global image-text matching or local window similarity), failing to establish a hierarchical relationship between micro-texture and macro-structure, making it difficult to adaptively capture multi-scale defects. For example, AnomalyCLIP simplifies design through object-independent cues, but still cannot effectively solve the problem of missing small defects; WinCLIP's window-based multi-scale patch method improves local perception, but fails to achieve text-visual collaborative optimization. These limitations severely restrict ZSAD's generalization ability and accuracy in fields such as industrial inspection and medical imaging.
[0006] In summary, the main drawbacks of existing technologies are as follows:
[0007] 1. Lack of cross-modal interaction mechanism: There is a semantic discrepancy between text prompts and visual features, resulting in poor generalization ability of the model in unknown scenarios.
[0008] 2. Insufficient fine-grained anomaly capture: Single-level feature matching cannot simultaneously handle micro-texture variations and macro-component missingness, and minute defects are easily overwhelmed by normal features. Summary of the Invention
[0009] To address the aforementioned shortcomings, the primary technical challenge of this invention is how to design a dynamic cross-modal interaction mechanism to transform abstract semantics from the text into directional constraints for visual cue generation, thereby mitigating modal semantic bias and improving generalization ability for unknown anomaly categories. A secondary technical challenge is how to construct a hierarchical anomaly representation library and enhance the model's ability to accurately capture subtle defects through multi-granularity prototype matching and uncertainty-aware scoring. These technical challenges stem directly from the limitations of existing technologies; therefore, this invention provides a zero-shot anomaly detection method based on bimodal cue enhancement to address each challenge individually.
[0010] The specific plan is as follows:
[0011] A zero-shot anomaly detection method based on bimodal cue enhancement includes the following steps:
[0012] S1. Input Encoding and Multimodal Feature Extraction: The input image and text prompts are encoded into feature vectors to achieve initial alignment of image and text features, providing a foundation for subsequent cross-modal interaction;
[0013] S2. Dynamic Visual-Text Prompt Interaction: The PIM module is adopted to achieve dynamic guidance of visual features by text prompts through a learnable projection matrix, which solves the problem of modal semantic bias. Among them, the PIM module enables text semantics to penetrate into visual features layer by layer, guides the model to focus on anomaly-related areas, and improves the generalization ability to unknown anomalies.
[0014] S3. Construction of Hierarchical Anomaly Prototype Memory: Using the PMB module, a multi-granularity anomaly prototype library is constructed. Fine-grained representation is enhanced through micro-textures and macro-component prototypes, providing a foundation for subsequent hierarchical matching.
[0015] S4. Uncertainty Perception Anomaly Scoring Mechanism: Combining multi-granularity prototype matching and uncertainty weights, anomaly scores are calculated; through adaptive weighted multi-granularity features, subtle defects are accurately captured, solving the prototype ambiguity problem.
[0016] S5. Anomaly Detection and Output: Integrates text-visual similarity and PMB score to complete anomaly detection. Achieves end-to-end zero-sample anomaly detection, suitable for industrial quality inspection or medical imaging diagnosis.
[0017] Further, in step S1, the extraction of image features includes: input image Multi-level visual features are extracted through visual encoder (such as ViT):
[0018] Global Visual Embedding: CLS Token Features ,in For example, 512 is the feature dimension;
[0019] Local visual embedding: Patch tokens features ,in and This refers to the spatial dimensions.
[0020] Further, in step S1, the text feature extraction includes: text prompts (such as "A photo of a normal object" and "A photo of a damaged object") are encoded into embedding vectors by a text encoder; normal text embeddings are then used. ∈RD, Abnormal text embedding ∈RD.
[0021] Further, step S2 includes:
[0022] S21. Text-side multi-layered prompt generation: The global text embedding is mapped to prompt vectors for each ViT layer. Let the total number of ViT layers be L, then the text prompt for the l-th layer... The generating formula is:
[0023] , =1,2,...,L
[0024] Where Fproj(l) is a learnable projection network;
[0025] S22, Cross-modal projection: Projecting the visual features of the l-th layer ∈R(N+1)×D and the text prompt p(l) are projected into a shared space, where the l-th layer visual features contain N patch tokens and 1 CLS token:
[0026]
[0027] in , ∈RD×D is the learnable parameter matrix;
[0028] S23. Dynamic Fusion: Fusing projection features through weighted averaging or attention mechanisms to generate new visual cue features. :
[0029]
[0030] Where λ(l)∈[0,1] are the learnable weights. Attention function; fused features Input the subsequent network layers.
[0031] Further, step S3 includes:
[0032] S31, Construction of Microtexture Prototype Library
[0033] From the ViT intermediate layer (such as the first) Extract patch tokens features (8 layers) Use clustering algorithms (such as K-Means) to generate normal microscopic prototypes. and abnormal micro prototypes ,in =50;
[0034] S32, Construction of Macro-component Prototype Library
[0035] From the top level of ViT (such as the first) Extracting CLS token features (12 layers) Aggregation generates normal macroscopic prototypes. and abnormal macroscopic prototypes ,in =10.
[0036] Further, step S4 includes:
[0037] S41. Multi-granularity similarity calculation
[0038] Microscopic layer similarity:
[0039]
[0040] Macro-level similarity:
[0041]
[0042] in Cosine similarity;
[0043] S42, Calculation of Uncertainty Weights
[0044] Through weighted network (e.g., a two-layer MLP) Estimate the credibility weights of each layer:
[0045]
[0046] After normalization:
[0047]
[0048] S43, Hierarchical Scoring Integration
[0049] Combining micro and macro scores:
[0050]
[0051] Further, step S5 includes:
[0052] S51, Text-Visual Similarity Score
[0053]
[0054] S52, Final Anomaly Detection
[0055] Combination and The system outputs image-level or pixel-level abnormal results by thresholding or weighted fusion.
[0056]
[0057] The beneficial effects of this invention are:
[0058] 1. Addressing Modal Semantic Bias: The PIM module transforms abstract textual attributes into visual cue constraints through dynamic cross-modal projection, enabling visual features to focus on anomalous regions. Experiments show that on industrial datasets (such as MVTec AD), image-level AUROC is improved to 92.5% (compared to 86.1% of existing methods such as AnomalyCLIP), significantly enhancing cross-class generalization ability.
[0059] 2. Improved fine-grained detection accuracy: The PMB module accurately captures microscopic defects through multi-granularity prototype matching and uncertainty weighting. The pixel-level localization metric AUPRO achieves 77.0% on the VisA dataset (compared to the benchmark CLIP's 14.8%), effectively reducing missed detections.
[0060] 3. High data efficiency: In scenarios with few samples (such as training with 2 samples per class), ACLIP still achieves 89.5% AUROC on MVTEC, avoiding the inadequate generalization problem of existing methods (such as WinCLIP's 68.2%). Attached Figure Description
[0061] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation
[0062] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.
[0063] This invention provides a zero-sample anomaly detection method based on bimodal cue enhancement, comprising the following steps:
[0064] S1. Input Encoding and Multimodal Feature Extraction: The input image and text prompts are encoded into feature vectors to achieve initial alignment of image and text features, providing a foundation for subsequent cross-modal interaction;
[0065] S2. Dynamic Visual-Text Prompt Interaction: The PIM module is adopted to achieve dynamic guidance of visual features by text prompts through a learnable projection matrix, which solves the problem of modal semantic bias. Among them, the PIM module enables text semantics to penetrate into visual features layer by layer, guides the model to focus on anomaly-related areas, and improves the generalization ability to unknown anomalies.
[0066] S3. Construction of Hierarchical Anomaly Prototype Memory: Using the PMB module, a multi-granularity anomaly prototype library is constructed. Fine-grained representation is enhanced through micro-textures and macro-component prototypes, providing a foundation for subsequent hierarchical matching.
[0067] S4. Uncertainty Perception Anomaly Scoring Mechanism: Combining multi-granularity prototype matching and uncertainty weights, anomaly scores are calculated; through adaptive weighted multi-granularity features, subtle defects are accurately captured, solving the prototype ambiguity problem.
[0068] S5. Anomaly Detection and Output: Integrates text-visual similarity and PMB score to complete anomaly detection. Achieves end-to-end zero-sample anomaly detection, suitable for industrial quality inspection or medical imaging diagnosis.
[0069] Further, in step S1, the extraction of image features includes: input image Multi-level visual features are extracted through visual encoder (such as ViT):
[0070] Global Visual Embedding: CLS Token Features ,in For example, 512 is the feature dimension;
[0071] Local visual embedding: Patch tokens features ,in and This refers to the spatial dimensions.
[0072] Further, in step S1, the text feature extraction includes: text prompts (such as "A photo of a normal object" and "A photo of a damaged object") are encoded into embedding vectors by a text encoder; normal text embeddings are then used. ∈RD, Abnormal text embedding ∈RD.
[0073] Further, step S2 includes:
[0074] S21. Text-side multi-layered prompt generation: The global text embedding is mapped to prompt vectors for each ViT layer. Let the total number of ViT layers be L, then the text prompt for the l-th layer... The generating formula is:
[0075] , =1,2,...,L
[0076] Where Fproj(l) is a learnable projection network;
[0077] S22, Cross-modal projection: Projecting the visual features of the l-th layer ∈R(N+1)×D and the text prompt p(l) are projected into a shared space, where the l-th layer visual features contain N patch tokens and 1 CLS token:
[0078]
[0079] in , ∈RD×D is a learnable parameter matrix; it can be initialized using different strategies or sparse constraints to optimize computational efficiency.
[0080] S23. Dynamic Fusion: Fusing projection features through weighted averaging or attention mechanisms to generate new visual cue features. :
[0081]
[0082] Where λ(l)∈[0,1] are the learnable weights. Attention function; fused features The input is then fed into subsequent network layers. Furthermore, the attention mechanism (Attn) can be replaced with a gated recurrent unit (GRU) or a Transformer encoder to enhance long-range dependency modeling.
[0083] Further, step S3 includes:
[0084] S31, Construction of Microtexture Prototype Library
[0085] From the ViT intermediate layer (such as the first) Extract patch tokens features (8 layers) Use clustering algorithms (such as K-Means, or alternatively DBSCAN or Gaussian mixture models) to generate normal microscopic prototypes. and abnormal micro prototypes ,in =50;
[0086] S32, Construction of Macro-component Prototype Library
[0087] From the top level of ViT (such as the first) Extracting CLS token features (12 layers) Aggregation generates normal macroscopic prototypes. and abnormal macroscopic prototypes ,in =10. Furthermore, macroscopic prototypes can be generated based on attention weights, rather than simple aggregation.
[0088] Further, step S4 includes:
[0089] S41. Multi-granularity similarity calculation
[0090] Microscopic layer similarity:
[0091]
[0092] Macro-level similarity:
[0093]
[0094] in Cosine similarity;
[0095] S42, Calculation of Uncertainty Weights
[0096] Through weighted network (e.g., a two-layer MLP) Estimate the credibility weights of each layer:
[0097]
[0098] After normalization:
[0099]
[0100] S43, Hierarchical Scoring Integration
[0101] Combining micro and macro scores:
[0102]
[0103] Further, step S5 includes:
[0104] S51, Text-Visual Similarity Score
[0105]
[0106] S52, Final Anomaly Detection
[0107] Combination and The system outputs image-level or pixel-level abnormal results by thresholding or weighted fusion.
[0108]
[0109] Table 1: Comparison of Zero-Sample Anomaly Detection Performance in Industrial Applications
[0110] Table 1 Zero-Shot Anomaly Detection Performance Comparison in the Industrial Domain
[0111]
[0112] In summary, this invention not only designs a dynamic visual-text adapter (PIM) and, for the first time, a learnable cross-modal projection matrix to achieve layer-by-layer dynamic guidance of visual features through text prompts, breaking through single-modal dependence, but also employs a hierarchical anomaly prototype memory (PMB). By constructing a multi-granularity prototype library (micro-texture + macro-component) and combining it with uncertainty perception scoring, adaptive multi-scale defect capture is achieved. Furthermore, the collaborative optimization framework of this invention allows the PIM and PMB modules to work together, simultaneously solving modal bias and fine-grained perception problems, and improving zero-shot generalization capability.
[0113] The technical means disclosed in this invention are not limited to those disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications are also considered within the scope of protection of this invention.
Claims
1. A zero-shot anomaly detection method based on bimodal cue enhancement, characterized in that: Includes the following steps, S1. Input Encoding and Multimodal Feature Extraction: The input image and text prompts are encoded into feature vectors to achieve initial alignment of image and text features; S2, Dynamic Visual-Text Prompt Interaction: The PIM module is used to achieve dynamic, layer-by-layer guidance of visual features through a learnable projection matrix; the PIM module enables text semantics to gradually penetrate into visual features, guiding the model to focus on abnormally relevant areas; S3. Construction of Hierarchical Anomaly Prototype Memory: Using the PMB module, a multi-granularity anomaly prototype library is constructed, which enhances fine-grained representation through micro-texture and macro-component prototypes. S4. Uncertainty perception anomaly scoring mechanism: Combining multi-granularity prototype matching and uncertainty weights, anomaly scores are calculated. S5. Anomaly Detection and Output: Integrate text-visual similarity and PMB score to complete anomaly detection.
2. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 1, characterized in that: In step S1, the extraction of image features includes: input image After processing by a visual encoder, multi-level visual features are extracted: Global Visual Embedding: CLS Token Features ,in For feature dimensions; Local visual embedding: Patch tokens features ,in and This refers to the spatial dimensions.
3. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 2, characterized in that: In step S1, the feature extraction of the text includes: text prompts generating embedding vectors via a text encoder; and normal text embedding. ∈RD, Abnormal text embedding ∈RD.
4. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 3, characterized in that: Step S2 includes: S21. Text-side multi-layered prompt generation: The global text embedding is mapped to prompt vectors for each ViT layer. Let the total number of ViT layers be L, then the text prompt for the l-th layer... The generating formula is: , =1,2,...,L； Where Fproj(l) is a learnable projection network; S22, Cross-modal projection: Projecting the visual features of the l-th layer ∈R(N+1)×D and the text prompt p(l) are projected into a shared space, where the l-th layer visual features contain N patch tokens and 1 CLS token: ； in , ∈RD×D is the learnable parameter matrix; S23. Dynamic Fusion: Fusing projection features through weighted averaging or attention mechanisms to generate new visual cue features. : ； Where λ(l)∈[0,1] are the learnable weights. Attention function; fused features Input the subsequent network layers.
5. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 4, characterized in that: Step S3 includes: S31, Construction of Microtexture Prototype Library Extract patch tokens features from the ViT middle layer Generate normal microscopic prototypes using clustering algorithms. and abnormal micro prototypes ,in =50; S32, Construction of Macro-component Prototype Library Extract CLS token features from the top layer of ViT Aggregation generates normal macroscopic prototypes. and abnormal macroscopic prototypes ,in =10.
6. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 5, characterized in that: Step S4 includes: S41. Multi-granularity similarity calculation Microscopic layer similarity: ； Macro-level similarity: ； in Cosine similarity; S42, Calculation of Uncertainty Weights Through weighted network Estimate the credibility weights at each level: ； After normalization: ； S43, Hierarchical Scoring Integration Combining micro and macro scores: 。 7. The zero-sample anomaly detection method based on bimodal cue enhancement as described in claim 6, characterized in that: Step S5 includes: S51, Text-Visual Similarity Score ； S52, Final Anomaly Detection Combination and The system outputs image-level or pixel-level anomaly results through thresholding or weighted fusion. 。