A zero-shot anomaly detection method based on graphic-text symmetry prompt cooperation

Abstract

The application discloses a zero-shot anomaly detection method based on graphic-text symmetry prompt cooperation, and the method comprises the following steps: acquiring a text input sequence and an input image sample for zero-shot anomaly detection; constructing a zero-shot anomaly detection model based on graphic-text symmetry prompt cooperation; training the constructed zero-shot anomaly detection model according to the text input sequence and the input image sample to minimize the loss value of a set joint training target function, and acquiring an optimal anomaly detection model; and realizing zero-shot anomaly detection based on graphic-text symmetry prompt cooperation according to the optimal anomaly detection model. The application solves the problems of single anomaly semantic on the text side, lack of sample-level dynamic self-adaptation and visual symmetry representation, one-way cross-modal information flow, mixed visual features and lack of multi-scale spatial enhancement in the prior art.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and industrial anomaly detection technology, and in particular to a zero-sample anomaly detection method based on image-text symmetric cue collaboration. Background Technology

[0002] Visual anomaly detection plays a central role in industrial quality inspection and medical image analysis. In industrial settings, product surface defects (such as scratches, dents, contamination, and misalignment) are diverse, varied in form, and difficult to predict; in medical settings, the morphology and texture of lesion areas also exhibit great diversity. These applications require models to simultaneously perform two tasks: first, to determine whether the entire image contains anomalies (image-level anomaly classification); and second, to accurately delineate the contours of anomalous regions (pixel-level anomaly localization).

[0003] Most current mainstream anomaly detection methods assume that sufficient normal training samples in the target domain are available before deployment, and that deviations are identified by learning the feature distribution of normal samples. However, this assumption is often violated in practice: newly launched products may have no training data before deployment, and medical data is often difficult to obtain due to privacy restrictions. Furthermore, training a dedicated model for each new product or scenario is unacceptable in terms of annotation costs, training time, and maintenance overhead. To address these challenges, zero-shot anomaly detection methods based on CLIP (Contrastive Language-Image Pre-training) have gradually emerged in recent years. These methods are trained on an auxiliary source domain using image-level labels and pixel-level masks, and then directly deployed to an unseen target domain without requiring any target domain images, labels, or test-time parameter updates. By aligning textual descriptions of normal / abnormal states with multi-layered visual features, these methods unify image-level scoring and pixel-level localization in a single pipeline, and retain CLIP's cross-domain transfer prior through parameter-efficient learnable prompts.

[0004] However, existing CLIP-based zero-shot anomaly detection methods still face four core issues that have not been fully explored when performing cross-domain transfer: (i) Limited semantic representation of text-based anomalies: Existing methods typically learn only a single general anomaly textual cue (e.g., "damaged object"), treating "anomaly" as a single semantic category. However, in reality, anomaly patterns within the same normal semantic category are extremely rich—for example, "metal surface" may simultaneously exhibit various defects such as scratches (texture anomalies), dents (geometric deformations), color differences (color anomalies), and burrs (edge ​​anomalies). A single anomaly cue cannot cover this multi-granularity anomaly semantic space, resulting in significantly insufficient sensitivity for detecting different types of anomalies. Some methods attempt to expand anomaly descriptions by manually designing multiple cue templates or heuristic splicing rules, but this non-learning-based rule generation makes it difficult to guarantee the completeness and complementarity of coverage, and the cuees may be highly redundant rather than truly complementary.

[0005] (ii) Lack of Sample-Level Dynamic Adaptation and Cross-Modal Bidirectional Interaction: Existing methods provide fixed anomaly cues after training, failing to personalize them based on the visual characteristics of different test samples and making it difficult to adapt to significant variations in anomaly patterns between test samples. More importantly, the information flow of existing methods is unidirectional—calculating similarity between visual and text features only (visual → text), lacking a reverse guidance channel from text to vision, and completely ignoring the learnable anomaly representations that also exist in the visual space. In fact, learning anomaly cues on the text side (text space) and learning anomaly representations on the visual side (visual space) should be two symmetrical and complementary processes. A bidirectional information interaction should be established between text features and visual features to fully utilize cross-modal complementary information to enhance anomaly detection.

[0006] (iii) Mixing of general and anomalous features in visual encoders: Existing methods maintain a standard self-attention mechanism in visual encoders, with all patch tokens interacting through a unified QKV attention mechanism. This results in a mixture of general semantic features and anomalous-specific features in the same feature space, where anomalous-sensitive features are overwhelmed by a large number of normal region features, which is detrimental to subsequent anomaly detection. At the same time, the patch features output by standard ViT lack explicit local spatial inductive bias, which is unfavorable for pixel-level anomaly detection that requires precise spatial localization—the receptive field of a single patch is limited by a fixed grid division, making it difficult to capture anomalous patterns at different spatial scales.

[0007] In summary, existing technologies have significant shortcomings in the following aspects: (a) the semantic dimension of text-side anomalies is singular, lacking a learnable mechanism to decompose anomaly semantics into multiple complementary fine-grained sub-modalities; (b) they lack sample-level dynamic adaptation capabilities and visual-side symmetrical anomaly representations, and the cross-modal information flow is unidirectional; (c) general and anomaly features are mixed in the visual encoder, lacking effective feature separation and multi-scale spatial enhancement mechanisms; and (d) no method has been found to organically integrate text anomaly cue decomposition, visual anomaly cue generation, cross-modal bidirectional adaptation, and dual-stream feature decoupling. Summary of the Invention

[0008] This invention provides a zero-sample anomaly detection method based on text-image symmetry cue collaboration to overcome the problems of existing technologies, such as the semantic simplicity of text-side anomalies, lack of sample-level dynamic adaptation and visual symmetry representation, unidirectional cross-modal information flow, and mixed visual features and lack of multi-scale spatial enhancement.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows: A zero-shot anomaly detection method based on image-text symmetric cue collaboration includes the following steps: S1: Obtain the text input sequence and input image samples for zero-shot anomaly detection; Furthermore, the text input sequence includes a normal semantic prompt and several abnormal semantic prompts; the input image samples include, but are not limited to, images of surface defects in industrial products or medical images; S2: Construct a zero-sample anomaly detection model based on the collaboration of image and text symmetric prompts; Furthermore, the zero-shot anomaly detection model includes an image encoder module, a text encoder module, a visual anomaly prompt generation module, an image-driven text adaptation module, a text-driven visual adaptation module, and a dual-modal fusion and anomaly determination module. The image encoder module is used to obtain spatially enhanced patch features and global image features based on each input image sample through a multi-head self-attention mechanism and a two-stream attention mechanism using the CLIP visual encoder; the text encoder module is used to obtain normal text features and several abnormal text features based on the text input sequence using the CLIP text encoder; obtain aggregated abnormal text features based on the several abnormal text features; and obtain each coarsely detected text anomaly map based on the aggregated abnormal text features and patch features. The visual anomaly prompt generation module is used to obtain visual normal tokens and visual anomaly tokens based on global image features; the image-driven text adaptation module is used to obtain fine-detection text anomaly maps based on coarse-detection text anomaly maps; the text-driven visual adaptation module is used to obtain adapted visual anomaly tokens based on aggregated abnormal text features, visual anomaly tokens, and normal text features; and to obtain a visual anomaly map based on the visual normal tokens and adapted visual anomaly tokens; the dual-modal fusion and anomaly determination module is used to fuse the coarse-detection text anomaly map, fine-detection text anomaly map, and visual anomaly map according to set visual fusion weights to obtain a fused anomaly map, and after applying Gaussian filtering to smooth the fused anomaly map, output the final anomaly heatmap and the corresponding image-level anomaly classification result; S3: Based on the text input sequence and input image samples, train the zero-shot anomaly detection model to minimize the loss value of the set joint training objective function, and obtain the optimal anomaly detection model; implement zero-shot anomaly detection based on image-text symmetric prompt collaboration based on the optimal anomaly detection model.

[0010] Furthermore, the methods for obtaining spatially enhanced patch features and global image features in S2 are as follows: The input image sample is input to a preset CLIP visual encoder; the CLIP visual encoder comprises 24 coding layers with different kernel parameters connected sequentially; layers 1 to 19 are defined as shallow encoder layers, and layers 20 to 24 are defined as deep encoder layers; the shallow encoder layers are used to extract general visual features from the input image sample based on standard multi-head self-attention, and the general visual features include, but are not limited to, texture, edge, and color features; the deep encoder layers are used to obtain general semantic features and anomaly-specific features based on the output of the shallow encoder layers using a two-stream attention mechanism, with the corresponding formulas as follows:

[0011]

[0012] In the formula: Represents general semantic features; Indicates abnormal and specific characteristics; Represents the query vector, key vector, and value vector; Indicates the feature dimension; Indicates transpose; Based on the set of layer indexes Multi-layer patch features are extracted from the output of each coding layer in the CLIP visual encoder. Channel-level weighted aggregation is performed on the spatial neighborhood of each patch feature using Gaussian kernels of different sizes to obtain spatially enhanced patch features and global image features. The channel-level weighted aggregation method is as follows: the patch features are first reshaped into a two-dimensional spatial grid, then the neighborhood window of the two-dimensional spatial grid is extracted by the unfold operation, and the neighborhood window is summed element-wise by preset normalized Gaussian weights.

[0013] Furthermore, the method for obtaining the coarse-detection text anomaly graph described in S2 is as follows: The text input sequence is fed into a preset CLIP text encoder to extract the text semantic features of the normal semantic prompts and several abnormal semantic prompts, thereby obtaining the normal text features. With several abnormal text features ; Based on several abnormal text features, the aggregated abnormal text features are obtained as follows:

[0014] In the formula: Indicates aggregated abnormal text features; The total number of anomalous text features; Indicates the first One abnormal text feature; A coarse-detected text anomaly map is obtained based on aggregated anomalous text features and patch features. This is achieved by calculating the cosine similarity between the patch features and the aggregated anomalous text features, and then... The function outputs a coarsely detected text anomaly graph.

[0015] Furthermore, the visual anomaly cue generation module in S2 includes an input layer and an image conditional generation network; the input layer is used to process the global features of the image. The image is transmitted to an image conditional generation network, which includes a shared backbone network, a normal head, and an abnormal head. The shared backbone network includes a batch normalization layer, a linear layer, and a sequentially connected batch normalization layer. Activation function layer; the batch normalization layer is used for global image features. Perform normalization operation; the linear layer is used to perform a linear operation on the output of the batch normalization layer; the The activation function layer is used to perform activation operations on the output of the linear layer; the normal header is used to pass through a preset first fully connected layer to activate the output of the linear layer. The activation function layer performs a fully connected mapping operation to obtain a normal token increment. To obtain a visually normal token; the abnormal header is used to pass through a pre-defined second fully connected layer to the... The activation function layer performs a fully connected mapping operation to obtain... An abnormal token increment To obtain A visual anomaly token, the expression of which is:

[0016]

[0017] In the formula: Indicates L2 regularization; Indicates a visually normal token; , Represents learnable basis vectors; Indicates the learnable gating factor; Indicates the first A visual anomaly token.

[0018] Furthermore, the method for obtaining the fine-grained text anomaly map described in S2 is as follows: Obtain the K patch positions with the highest anomaly scores in each coarsely detected text anomaly image; and the anomaly score is determined by the cosine similarity between the patch features and the aggregated anomaly text features, with a higher cosine similarity resulting in a higher anomaly score. The bias vector is obtained from the K patch positions using the feature-bias mapping network:

[0019] In the formula: Represents the bias vector; This represents a vector concatenation operation; This represents a multilayer perceptron; This represents the patch space index with the k-th highest anomaly score in the coarse anomaly detection graph, and... ; bias vector The abnormal semantic token parameter of each abnormal semantic prompt is added additively, and the text encoder module is used to re-obtain the fine-detection text abnormal graph.

[0020] Furthermore, the method for obtaining the visual anomaly map described in S2 is as follows: Based on the aggregated abnormal text features and the visual abnormal token, the adapted visual abnormal token is obtained as follows:

[0021]

[0022] In the formula: This indicates a visual anomaly token after adaptation. This represents the refined increment of the abnormal token; Indicates the first The refined increment of an abnormal token and ; This represents a pre-defined prior mapping network; Indicates aggregated abnormal text features; Indicates the difference between anomalous and normal text features; This represents the learnable scaling factor; Indicates the first A visual anomaly token; Based on the visually normal token and the adapted visually abnormal token, the visually abnormal image is obtained as follows:

[0023] In the formula: Represents a visual anomaly diagram; Indicates the first Each patch feature; Indicates the temperature coefficient; This indicates a visually normal token.

[0024] Furthermore, in S2, the method for outputting the final anomaly heatmap and corresponding image-level anomaly classification results through the dual-modal fusion and anomaly detection module is as follows: Based on the set visual fusion weights, the coarse-detected text anomaly map, the fine-detected text anomaly map, and the visual anomaly map are fused to obtain the fused anomaly map:

[0025]

[0026] In the formula: Represents the fusion anomaly graph; Indicates visual fusion weights; This represents a rough anomaly detection graph for text. This represents a graph showing text anomalies detected through precise analysis. Indicates intermediate parameters; express The abbreviated form; Based on the fused anomaly map and the text classification anomaly probability, the image-level anomaly score is obtained as follows:

[0027]

[0028] In the formula: Represents image-level anomaly score; Indicates the probability of text classification anomalies; This represents the set of indices of the K highest-scoring patch locations in the fusion anomaly graph; express The abbreviated form; After applying Gaussian filtering to smooth the fused anomaly map, the final anomaly heatmap and corresponding image-level anomaly classification results are output by combining the image-level anomaly score.

[0029] Furthermore, the joint training objective function described in S3 is the sum of several loss terms obtained based on the set weight coefficients; and the loss terms include text anomaly map loss, visual anomaly map loss, image-level classification loss, orthogonal constraint loss, bias regularization loss, visual-text alignment loss, visual token diversity loss, and prior residual regularization loss for text-driven visual adaptation.

[0030] This invention provides a zero-sample anomaly detection method based on image-text symmetric cue collaboration, with the following beneficial effects: (1) Decoupling architecture of text-image symmetry: The text encoder module decomposes the abnormal semantic prompts into several fine-grained prompts with orthogonal constraints, i.e., abnormal text features, in the text space through the text encoder module based on the CLIP text encoder; the visual abnormal prompt generation module decomposes the abnormal representation of the input image sample into several visual abnormal tokens for each image condition in the visual space through the visual space; the two modules form a cross-modal prompt decoupling pair, realizing symmetrical abnormal semantic decomposition on both the text and visual sides. Compared with the existing technology that only provides single prompts on the text side, multi-granular decoupling enables the model to respond to multiple abnormal sub-modes such as texture, structure, color, and edge at the same time, and the detection coverage is significantly improved. (2) Bidirectional cross-modal prior adaptation: By forming a cross-modal prior adaptation through the image-driven text adaptation module and the text-driven visual adaptation module, the limitation of the existing technology of only one-way information flow is broken, and true bidirectional image and text collaboration is realized. The gain of bidirectional adaptation is greater than that of any one-way adaptation. (3) Dual-stream feature separation and multi-scale enhancement: Through the dual-stream attention mechanism, namely the multi-head self-attention mechanism and the dual-stream attention mechanism, the general semantic features and the anomaly-specific features are effectively separated in the deep layer of the visual encoder. The anomaly-sensitive features are no longer submerged by a large number of normal region features. Combined with parameter-independent multi-scale Gaussian neighborhood aggregation, the effective receptive field of dense prediction is expanded with zero additional parameters, which significantly improves the accuracy and spatial consistency of anomaly localization.

[0031] (4) Lightweight and efficient deployment: The trained model can be directly deployed to any unseen target domain without any adaptation operation or data access on the target domain side, which has extremely high practical value in cross-domain scenarios such as industrial defect detection and medical image screening. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart of the zero-sample anomaly detection method based on image-text symmetric prompting collaboration of the present invention; Figure 2 This is a schematic diagram of the zero-sample anomaly detection model based on image-text symmetric prompting collaboration in this embodiment; Figure 3 This is a schematic diagram comparing the paradigms of the method of the present invention with existing CLIP-based zero-sample anomaly detection methods; Figure 4 This is a schematic diagram of the visual anomaly prompt generation module in this embodiment; Figure 5 This is a schematic diagram of the structure and cross-modal information flow of the text-driven visual adaptation module in this embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This embodiment provides a zero-shot anomaly detection method based on image-text symmetric cue collaboration. It can be applied to multiple scenarios for zero-shot anomaly detection and pixel-level anomaly localization tasks, including industrial product surface defect detection (such as scratches on metal parts, PCB board soldering defects, and abnormal fabric textures) and medical imaging anomaly screening (such as brain MRI tumor detection, colonoscopy polyp identification, and dermoscopy melanoma screening). Figure 1 As shown, the specific steps include: S1: Obtain the text input sequence and input image samples for zero-shot anomaly detection; Furthermore, the text input sequence includes one normal semantic prompt and several abnormal semantic prompts; the input image samples include, but are not limited to, images of surface defects in industrial products or medical images; in this embodiment, the template for the normal semantic prompt is... Where [N] represents the number of learnable normal semantic tokens. [object] represents a category name; the template for the anomaly semantic prompt in this embodiment is... ,in This represents the unique learnable exception semantic token for the p-th exception message and its quantity. That is, 10 exception prompts, each with its own independent parameters, share the normal semantic token prefix; S2: Construct a zero-sample anomaly detection model based on the collaboration of image and text symmetric prompts; like Figures 2 to 3 As shown, the zero-sample anomaly detection model includes an image encoder module, a text encoder module, a visual anomaly prompt generation module, an image-driven text adaptation module, a text-driven visual adaptation module, and a dual-modal fusion and anomaly determination module. The image encoder module is used to obtain spatially enhanced patch features and global image features based on each input image sample using a CLIP visual encoder through a multi-head self-attention mechanism and a two-stream attention mechanism. Specifically, it includes: The input image sample is input into a preset CLIP visual encoder with frozen parameters. The CLIP visual encoder comprises 24 coding layers with different kernel parameters connected sequentially. Coding layers 1 to 19 are defined as the encoder shallow layer, and coding layers 20 to 24 are defined as the encoder deep layer. The encoder shallow layer is used to extract general visual features from input image samples based on standard multi-head self-attention, and these general visual features include, but are not limited to, texture, edge, and color features. The encoder deep layer is used to obtain general semantic features and anomaly-specific features based on a two-stream attention mechanism (i.e., simultaneously calculating the original self-attention stream: QKV path, outputting general semantic features and an anomaly-sensitive stream: Q=K=V path, outputting anomaly-specific features). The corresponding formulas are as follows:

[0036]

[0037] In the formula: The general semantic features are the output features of the original self-attention stream, which are used to preserve general semantic information; The output characteristics of anomaly-specific flows, i.e., anomaly-sensitive flows, are used to highlight local anomaly-specific information. The query vector, key vector, and value vector represent the visual token sequence of the corresponding input image obtained based on empirical values ​​before entering this layer, which is obtained through linear projection. Indicates the feature dimension; This indicates transposition; in this embodiment, the two feature streams are processed by residual connections and feedforward networks (FFN) respectively to form parallel dual-stream feature representations, which are maintained separately in subsequent layers without being mixed. Based on the set of layer indexes Extract multi-layer patch features from the outputs of the corresponding coding layers in the CLIP visual encoder. The first row contains class tokens, and the remaining N rows contain patch tokens. The feature dimension d=768. Gaussian kernels of different sizes (a set of kernel sizes) are used. Standard deviation The spatial neighborhood of each patch feature is aggregated using channel-level weighted aggregation to obtain spatially enhanced patch features and global image features. The channel-level weighted aggregation method is as follows: In this embodiment, when the Gaussian kernel size is 1, it degenerates into an identity mapping; when the kernel size is greater than 1, the patch features are first reshaped into a two-dimensional spatial grid, then the neighborhood window of the two-dimensional spatial grid is extracted through an unfold operation, and the neighborhood window is summed element-wise using preset normalized Gaussian weights. In this embodiment, the preset normalized Gaussian kernel weights remain fixed throughout the training and inference process, without introducing any learnable parameters. The aggregation results of different layers and scales are fused across layers according to the mean rule, that is, the aggregation results of all selected layers and all kernel scales are averaged item by item to generate spatially enhanced patch features. With global features of the image That is, it is obtained from the visual projection layer of the CLIP visual encoder through which the category lexical units are frozen; The text encoder module is used for CLIP text encoder-based applications. The process involves: obtaining normal text features and several abnormal text features based on the text input sequence; obtaining aggregated abnormal text features based on the several abnormal text features; and obtaining coarse-detected text anomaly maps based on the aggregated abnormal text features and patch features. Specifically, the text input sequence is input into a preset CLIP text encoder, and the text semantic features of the normal semantic prompts and several abnormal semantic prompts are extracted respectively to obtain the normal text features. and P abnormal text features Furthermore, this embodiment also includes applying orthogonal constraint loss (the sum of the absolute values ​​of pairwise cosine similarity) to the P abnormal text features:

[0038] In this embodiment, the loss function forces the text features of different anomaly prompts to approach orthogonality pairwise, causing each prompt to automatically converge to a semantically orthogonal anomaly sub-pattern (such as responding to texture anomalies, structural anomalies, color anomalies, edge anomalies, etc.), achieving multi-granularity decoupling of anomaly semantics. Simultaneously, a set of learnable composite semantic tokens (4 tokens per layer, 768 dimensions) is injected into each of the D intermediate layers (D=9) of the text encoder, and then passed through independent linear projection layers (…). After mapping, the corresponding learnable token position in the current layer token sequence is replaced, which can achieve semantic enhancement at different abstraction levels of text encoding; Based on several abnormal text features, the aggregated abnormal text features are obtained as follows:

[0039] In the formula: Indicates aggregated abnormal text features; The total number of anomalous text features; Indicates the first One abnormal text feature; A coarse-detected text anomaly map is obtained based on aggregated anomalous text features and patch features. This is achieved by calculating the cosine similarity between the patch features and the aggregated anomalous text features, and then... Function (temperature coefficient) The output is a coarsely detected text anomaly map; The visual anomaly alert generation module is used to obtain visually normal tokens and visually abnormal tokens based on global image features; specifically, such as... Figure 4 The visual anomaly prompt generation module shown includes an input layer and an image conditional generation network; the input layer is used to generate global features of the image. The image is transmitted to the image conditional generation network, which includes a shared backbone network, a normal head, and an abnormal head. The shared backbone network includes a batch normalization layer, a linear layer, and a sequentially connected batch normalization layer. Activation function layer The batch normalization layer is used to normalize global features of the image. Perform normalization operation; the linear layer is used to perform a linear operation on the output of the batch normalization layer; the The activation function layer is used to perform activation operations on the output of the linear layer; the normal header is... For use through a pre-set first fully connected layer, to the The activation function layer performs a fully connected mapping operation to obtain a normal token increment. To obtain a visually normal token; the abnormal header is... For use with a pre-set second fully connected layer, to the The activation function layer performs a fully connected mapping operation to obtain... An abnormal token increment To obtain A visual anomaly token; in this embodiment, the visual token adopts a ternary construction of "learnable basis + learnable gating factor × image conditional increment", the expression of which is:

[0040]

[0041] In the formula: Indicates L2 regularization; Indicates a visually normal token; , Represents the learnable basis vectors (randomly initialized with a standard deviation of 0.02). This represents the learnable gating factor (after gradient clamping) (To prevent an explosion in incremental training later on) Indicates the first A visual anomaly token. In this embodiment, the VAPG (Visual Anomaly Cue Generation Module) functions in the visual space symmetrically to the TAPD (Textual Anomaly Detection Module) in the text space—TAPD decomposes the semantic representation of text anomalies into P=10 orthogonal cues, and VAPG decomposes the visual anomaly representation into... The visual tokens for each image condition constitute a cross-modal cue decoupling pair; The image-driven text adaptation module is used to obtain a fine-detection text anomaly map based on the coarse-detection text anomaly map. Specifically, in this embodiment, a personalized text anomaly prior bias is generated for each sample using the visual features of the preset test samples to achieve cross-modal adaptation from vision to text: the K patch positions with the highest anomaly scores in each coarse-detection text anomaly map are obtained; and the anomaly score is determined by the cosine similarity between the patch features and the aggregated anomaly text features, with a higher cosine similarity resulting in a higher anomaly score. The bias vector is obtained from the K patch positions using the feature-bias mapping network:

[0042] In the formula: Represents the bias vector; This represents a vector concatenation operation; This represents a multilayer perceptron; This represents the patch space index with the k-th highest anomaly score in the coarse anomaly detection graph, and... For each sample, the K patch positions with the highest anomaly scores are independently selected (K=10). Their visual feature vectors, each with a dimension d=768, are concatenated end-to-end according to spatial index order. 3D vector; via feature-bias mapping network Mapped to a bias vector of d=768 dimensions. ; the bias vector Added additively to the exception semantic token parameter of each exception semantic prompt (the operation is equivalent to...) The text anomaly map is then re-obtained through the text encoder module; in this embodiment, a bias regularization loss is applied to the normal samples during training as follows:

[0043] In this embodiment, the loss is applied only when the training sample label is normal, constraining the bias vector of normal samples to approach the zero vector, preventing the model from generating false abnormal biases for normal samples, and realizing cross-modal adaptation from vision to text: the visual abnormal patch features of the test sample drive the personalized bias adjustment of the text prompt. The text-driven visual adaptation module, TDVA, is used to obtain an adapted visual anomaly token based on aggregated abnormal text features and a visual anomaly token; and to obtain a visual anomaly map based on the visually normal token and the adapted visual anomaly token, such as... Figure 5 As shown, it specifically includes: Based on the aggregated abnormal text features, visual anomaly token, and normal text features, the adapted visual anomaly token is obtained as follows:

[0044]

[0045] In the formula: This indicates a visual anomaly token after adaptation. This represents the refined increment of the abnormal token; Indicates the first The refined increment of an abnormal token and ; This represents a pre-defined prior mapping network; Indicates aggregated abnormal text features; Indicates the difference between anomalous and normal text features; This represents the learnable scaling factor; Indicates the first A visual anomaly token; In this embodiment, aggregating abnormal text features will be used. Abnormal-normal difference characteristics The concatenation yields a prior input vector of dimension 2d=1536, which is then passed through a prior mapping network, resulting in the following structure: The output layer weights are initialized to zero to ensure that the refinement increment approaches zero in the initial training phase. The refinement increment for generating abnormal tokens is used in this embodiment. A learnable scaling factor is used in this embodiment. Control the refining intensity (initial value 0.001, gradient-controlled to) Then, the visual anomaly token is updated using residuals, resulting in the adapted visual anomaly token. In this embodiment, during training, probability is used... Randomly skipping TDVA updates (TDAP dropout strategy) enhances the model's robustness to text prior quality and prevents overfitting to specific text feature patterns. In this embodiment, cross-modal adaptation from text to vision is achieved. TDVA is text-to-visual (text difference-driven visual token refinement), and the two constitute cross-modal prior adaptation. Based on the visually normal token and the adapted visually abnormal token, the visually abnormal image is obtained as follows:

[0046] In the formula: Represents a visual anomaly diagram; Indicates the first Each patch feature; Indicates the temperature coefficient; This represents a visually normal token. In this embodiment, the visually normal token and the adapted visually abnormal token are respectively matched with patch features. Calculating cosine similarity — with The maximum similarity among visually anomalous tokens is taken. (temperature coefficient) ) to obtain a visual anomaly map; The dual-modal fusion and anomaly detection module is used to fuse coarse-detected text anomaly maps, fine-detected text anomaly maps, and visual anomaly maps according to set visual fusion weights to obtain a fused anomaly map. After applying Gaussian filtering to smooth the fused anomaly map, it outputs the final anomaly heatmap and the corresponding image-level anomaly classification result; specifically including: Based on the set visual fusion weights, the coarse-detected text anomaly map, the fine-detected text anomaly map, and the visual anomaly map are fused to obtain the fused anomaly map:

[0047]

[0048] In the formula: Represents the fusion anomaly graph; Indicates visual fusion weights; This represents a rough anomaly detection graph for text. This represents a graph showing text anomalies detected through precise analysis. Indicates intermediate parameters; express The abbreviation of [the document]; in this embodiment, the final text anomaly map is weighted by the first round of coarse detection and the second round of fine detection in a 1:2 ratio. This weighting reflects that fine detection (the result after sample adaptive bias adjustment) should dominate, while preserving the global stability of coarse detection; the fused anomaly map is composed of the text anomaly map and the visual anomaly map according to the visual fusion weight. Weighted; Based on the fused anomaly map and the text classification anomaly probability, the image-level anomaly score is obtained as follows:

[0049]

[0050] In the formula: Represents image-level anomaly score; Indicates the probability of text classification anomalies; This represents the set of indices of the K highest-scoring patch locations in the fusion anomaly graph; express The abbreviation form; in this embodiment, the image-level anomaly score is calculated by weighting the text classification anomaly probability with the mean of the maximum values ​​of the Top-K (K=10) values ​​in the fusion anomaly map. After applying Gaussian filtering to smooth the fused anomaly map, the final anomaly heatmap (pixel-level anomaly localization result) and the corresponding image-level anomaly classification result are output by combining the image-level anomaly score. S3: Based on the text input sequence and input image samples, train a zero-shot anomaly detection model to minimize the loss value of the set joint training objective function, and obtain the optimal anomaly detection model; implement zero-shot anomaly detection based on image-text symmetric cue collaboration based on the optimal anomaly detection model. Specifically, on a known source domain anomaly detection dataset, freeze all parameters of the CLIP visual encoder and CLIP text encoder, and optimize only the following lightweight component parameters: learnable normal cue token ( (768 dimensions), learnable exception message token ( (Number of tokens, 768 dimensions per token), deep composite semantic tokens (D=9 layers × 4 tokens per layer × 768 dimensions) and its linear projection layer (each layer) Learnable basis vectors and (A total of 1+4=5 768-dimensional vectors), shared MLP backbone and dual-head output layer parameters, learnable gating factor and Feature-bias mapping network patch_meta_net ( Prior mapping network (); ) and learnable refinement scaling factor The total number of trainable parameters is approximately 0.8M, which is only a fraction of CLIP. The base model (approximately 427M parameters) accounts for approximately 0.19%; the joint training objective function described in this embodiment is the sum of several loss terms obtained based on the set weight coefficients; and the loss terms include text anomaly map loss, visual anomaly map loss, image-level classification loss, orthogonality constraint loss, bias regularization loss, visual-text alignment loss, visual token diversity loss, and prior residual regularization loss for text-driven visual adaptation: The text anomaly graph loss: Used to simultaneously apply to coarse detection of text anomalies. With precise detection of text anomalies The loss of the visual anomaly map is 0.5. Used to act on visual anomaly maps The image-level classification loss, namely cross-entropy loss, is used for image-level classification prediction. The orthogonal constraint loss is... The weighting coefficient is set to 0.1 to force orthogonal decoupling of the 10 anomaly alerts; the bias regularization loss is... : Only normal samples are subject to (constraint bias approaches zero); The visual-text alignment loss: the weight coefficient is 0.01, i.e., visual normal tokens With normal text features Cosine distance + visual anomaly token mean and aggregated anomaly text features The cosine distance; the visual token diversity loss: the weight coefficient is 0.01, i.e., a pairwise cosine penalty based on the hinge margin, margin = 0.3, forced The visual anomaly tokens maintain diversity to prevent degradation into a single representation; the prior residual regularization loss for text-driven visual adaptation has a weight coefficient of 0.0001, i.e. The incremental range is constrained and refined; in this embodiment, the Adam optimizer (learning rate 0.001) is used during model training. The batch size was 8, and the training lasted for 10 epochs, all on a single NVIDIA RTX 4090 GPU.

[0051] In this embodiment, after the model training is completed, all learnable parameters are fixed (not updated with test samples), and the following inference process is executed for any unseen target domain image: (1) Dual-stream feature aggregation: Encoding → Shallow standard attention (layers 1-19) → Deep dual-stream attention (layers 20-24) → Multi-scale Gaussian neighborhood aggregation → Output space-enhanced patch features and global image features; (2) First round on the text side: Construct normal prompts + P=10 abnormal prompts (orthogonal constraints + D=9 layers of composite semantic token injection) → Text encoder encoding → Aggregate abnormal text features → Calculate the first round of coarse detection text abnormal map (3) Visual adaptation update: In the coarse detection of text anomaly graph Select Top-K anomaly patch → Concatenate features (7680 dimensions) → patch_meta_net → Bias vector →Inject TAPD exception token; (4) TAPD second round: re-encode text after bias injection → fine detection of text exception graph +Text Features and (5) VAPG+ TDVA: Global Image Features → Shared Backbone → Generation and indivual → splicing →TDVA Prior MLP→ → →Visual anomaly diagram (6) Dual-modal fusion: In this embodiment Image-level scores Gaussian filtering ( The entire process requires no target domain data access, tag utilization, or online parameter adjustment. The method described in this embodiment constructs a two-layer symmetric collaborative architecture: the first layer is a cross-modal cue decoupling pair: TAPD decomposes the anomaly semantics into P orthogonal fine-grained cues in the text space, and VAPG symmetrically decomposes the anomaly representation into... The first layer consists of visual tokens for each image condition; the second layer is cross-modal prior adaptation: visual patch features are mapped to adaptation biases for text prompts (visual → text), and TDVA symmetrically maps text feature differences to refined increments of visual tokens (text → visual); the two pairs of modules form an organic synergy through the following data flow: (a) first-round coarse detection results of text → visual bias generation → second-round fine detection of text; (b) second-round text features → TDVA prior → VAPG visual token refinement; (c) text anomaly map + VAPG visual anomaly map → dual-modal fusion; the entire data flow forms a complete feedback loop: text → visual → text → fusion. Compared with the prior art, the method described in this embodiment has the following advantages: (1) A symmetrical decoupling architecture for visual and textual prompts: In the text space, the anomaly semantics are decomposed into fine-grained prompts with P=10 orthogonal constraints, and the VAPG symmetrically decomposes the anomaly representation into... The visual tokens for each image condition; the two modules form a cross-modal cue decoupling pair, realizing symmetrical anomaly semantic decomposition on both the text and visual sides; compared with the existing technology that only provides single text-side cueing, the multi-granularity decoupling enables the model to respond to multiple anomaly sub-patterns such as texture, structure, color, and edge simultaneously, significantly improving detection coverage; (2) Bidirectional cross-modal prior adaptation: It realizes the adaptation of vision → text (the abnormal patch features of the test sample drive the personalized adjustment of text prompts), and TDVA symmetrically realizes the adaptation of text → vision (the text feature difference is used as the prior to refine the visual token). The two modules constitute cross-modal prior adaptation, which breaks through the limitation of the existing technology of only one-way information flow and realizes true bidirectional collaboration between images and text; ablation experiments have confirmed that the gain of bidirectional adaptation is greater than that of any one-way adaptation. (3) Dual-stream feature separation and multi-scale enhancement: Through a dual-stream attention mechanism (QKV general stream + Q=K=V anomaly stream), general semantic features and anomaly-specific features are effectively separated in the deep layer of the visual encoder, so that anomaly-sensitive features are no longer overwhelmed by a large number of normal region features; combined with parameter-independent multi-scale Gaussian neighborhood aggregation (kernel size) This method expands the effective receptive field of dense prediction with zero additional parameters, significantly improving the accuracy and spatial consistency of anomaly localization. (4) Extremely lightweight and efficient deployment: Completely freeze CLIP With 427M pre-trained parameters, only about 0.8M new parameters (accounting for 0.19%) are trained; model training can be completed in about 2 hours on a single consumer-grade GPU (such as NVIDIA RTX 4090). After training, the model can be directly deployed to any unseen target domain without any adaptation operations or data access on the target domain side, which has extremely high practical value in cross-domain scenarios such as industrial defect detection and medical image screening. (5) Modular symmetrical design is flexible and interpretable: The decomposition design of text and VAPG cues allows for independent adjustment of the number of text-side cues P and the number of visual-side tokens. The adaptation mechanisms for vision and TDVA are independent of the cue decomposition module; the positive contribution of each module and the synergistic gain between modules can be independently verified through ablation experiments. The gain generated by the superposition of modules exceeds the sum of the independent contributions of each module, verifying the effectiveness of the two-layer symmetric collaborative architecture.

[0052] This embodiment also includes the following application examples: Example 1: Zero-Sample Detection of Surface Defects in Industrial Products This example applies the concept of zero-sample detection of surface defects in industrial products. The quality inspection process of an electronics production line involves more than ten different types of products, including printed circuit boards (PCBs), metal nuts, screws, capsules, transistors, and cables. The production line needs to achieve automatic detection and pixel-level segmentation of surface defects in any product without collecting and labeling defect data separately for each product.

[0053] Model configuration and module principles utilize pre-trained CLIPs @336px serves as the foundational visual language model, freezing all 427M original parameters to retain its powerful general representation capabilities. Based on this, the following core lightweight components are built and integrated: Dual-stream feature aggregation module: Replaces the starting layer of the CLIP backbone network with a dual-stream attention mechanism (using... Taking a 24-layer model as an example, the first 19 layers are standard attention layers, and layers 20-24 are replaced with dual-stream attention layers. The dual-stream interaction starting layer is set to d=20. The module is configured with multi-scale Gaussian kernel size, and the cross-layer fusion rule adopts the mean strategy, aiming to establish a deep interaction channel between text context and multi-scale visual features. The text-side prompt generation module: configures the number of normal prompt tokens, sets the total number of abnormal prompts to P=10, and assigns a unique number of tokens to each abnormal prompt. A deep composite semantic token injection mechanism is introduced, with the number of injection layers set to D=9 (i.e., each layer contains 4 learnable tokens, the core dimension is 768, and after being projected to 896 dimensions through a linear layer, the representation at the corresponding position is dynamically replaced), in order to adaptively model multi-level complex defect text semantics. Visual Anomaly Prompt Generation Module (VAPG): Sets the number of visual anomaly tokens, shares the hidden layer dimension of the backbone network hidden_dim = 192, and sets the gating clamping constant, which is used to map local visual anomaly patches in the image to a unified language prompt space; Visual text update module: Set the dynamic interactive filtering threshold Top-K = 10, where the key feature mapping network patch_meta_net topology configuration, by establishing a dynamic feedback link between visual defect patches and text features, realizes real-time fine-tuning of cross-modal features; Text-Driven Visual Attention Refinement Module (TDVA): Constructs a priori mapping network, sets the hidden layer dimension hidden_dim = 192, sets the initial value of the refinement scaling factor, sets the numerical clamping interval boundary, configures the dropout probability during dynamic adjustment, and is used to suppress environmental noise and improve the contrast of pixel-level anomaly scores.

[0054] Data Preparation and Coverage Categories: This embodiment uses two mainstream publicly available anomaly detection datasets for algorithm validation. During the training phase, a strict unsupervised / zero-sample setting of "relying only on normal samples" is implemented, with no defect images, defect annotations, or anomaly masks input. Images are uniformly scaled to 336×336 resolution before input. The industrial objects and complex scenes covered specifically include: Dataset: Contains 15 industrial objects, subdivided into texture objects (carpet, grid, leather, tile, wood) and structure objects (bottle, capsule, pill, transistor, zipper, cable, hazelnut, metal nut, screw, toothbrush). Dataset: Contains 12 more complex multi-object scenarios and challenging scenes, including: candles, capsules, cashews, and chewing gum. ,French fries Macaroni (Class I), Macaroni (Class II), Printed Circuit Boards (Class I, PCB), Printed Circuit Boards (Class II, PCB), Printed Circuit Boards (Class III, PCB), Printed Circuit Boards (Class IV, PCB), Pipes (for French fries) .

[0055] During the training phase, the algorithm maintains all parameters of the CLIP basic visual language model frozen, and only updates the gradients of approximately 0.8M lightweight parameters contained in the five newly added modules. The Adam optimizer is used, with an initial learning rate of 0.001 and a batch size of 8, for a total of 15 epochs of training. The full training process is executed on a single NVIDIA RTX 4090 GPU, taking only about 2 hours in total. This extremely low parameter count and time overhead make the method of this invention highly valuable for rapid deployment and machine switching in the industry.

[0056] In the actual online inference phase, the model performs zero-shot evaluation on completely unseen test set samples (including normal and various defect samples). For each input product image to be tested, a complete end-to-end inference pipeline is executed: including basic visual feature extraction, dual-stream interactive feature aggregation, dynamic adjustment of dynamic text side prompts, local patch score calculation, and score refinement based on TDVA, ultimately outputting image-level macroscopic anomaly scores and pixel-level microscopic defect segmentation masks; the average inference time for a single 336×336 resolution image is consistently around 50ms, perfectly meeting the high-speed, high-frequency real-time automated optical inspection (AOI) performance requirements of industrial production lines; Technical Performance and Refined Data Analysis: Through comprehensive evaluation on two large-scale public datasets, this invention demonstrates extremely strong generalized detection performance and high-precision pixel-level defect localization capability without requiring target domain defect data annotation. Specific experimental test results are as follows: exist Performance on the dataset: Macroscopic average performance: The system achieves extremely high detection accuracy, with an image-level AUROC average of 93.6% and an image-level AP average of 97.1%; in pixel-level defect segmentation and localization, the pixel-level AUROC average reaches 92.4% and the pixel-level AUPRO average reaches 83.0%.

[0057] Detailed Analysis of Typical Categories: In the categories of "carpet" and "leather," which have complex surface textures, the method described in this embodiment achieved a perfect recognition rate of 100% for both image-level AUROC and image-level AP. Meanwhile, in categories such as "grid," "tile," "wood," "zipper," and "toothbrush," the image-level AP exceeded the 99% threshold. For "transistors," a notoriously difficult micro-component to handle in industry, the method still achieved an image-level AUROC of 86.4% and an image-level AP of 83.3% under the premise of training only with normal samples, strongly demonstrating the ability of the method described in this embodiment to capture irregular structures and complex edge defects.

[0058] exist Performance on the dataset: Macro-level average performance: When faced with more complex scenarios involving multiple object arrangements and various types of foreign objects, the method described in this embodiment achieves an average AUROC of 85.1% and an average AP of 87.2% at the image level; while its advantages are even more significant in fine-grained abnormal area localization, with an average AUROC of 96.0% at the pixel level and an average AUPRO of 88.3% at the pixel level.

[0059] Detailed analysis of typical categories: For "chewing gum" with dense structure and easily confused arrangement The pixel-level AUROC reached 99.4%, and the image-level AP reached 99.2%; for "tube-shaped French fries" with special aspect ratios and textures... The image quality index (AUROC) is among the highest in the industry (98.5% for pixel-level AUROC, 96.6% for pixel-level AUROC, 97.1% for image-level AUROC, and 98.6% for image-level AP). In tests on various types of printed circuit boards (PCB1 to PCB4) with highly irregular shapes and dense component arrangements, especially PCB4, the image-level AUROC reached 97.9%, and the pixel-level AUROC reached 97.2%, fully verifying the robustness and high practical value of the method described in this embodiment on complex electronic manufacturing production lines.

[0060] Example 2: Cross-domain zero-sample anomaly screening in medical imaging Verify the generalization ability of the method described in this embodiment in cross-domain zero-shot scenarios—the model is trained only on industrial anomaly detection data and directly applied to medical image anomaly detection.

[0061] Scenario Description: In medical imaging-assisted diagnosis, it is necessary to screen for abnormalities in medical images of different organs (brain, head, thyroid, colon, skin, etc.). The image characteristics of each organ are significantly different and labeled data is scarce. Traditional methods require collecting labeled data separately for each examination site and training a dedicated model.

[0062] Implementation process: Use the exact same trained model parameters as in Example 1 (training was completed only on industrial anomaly detection data); directly apply the model to medical image datasets containing multiple organs (e.g., for colonoscopy polyp detection). The dataset (such as the br35h dataset for brain tumor detection) is used without any domain adaptation or fine-tuning; the inference process is completely consistent with Example 1.

[0063] Technical Results: In cross-domain migration scenarios (industrial → medical), the method described in this embodiment achieves excellent anomaly detection and localization performance without requiring any fine-tuning with medical data. Specific evaluation results show that in brain tumor screening (br35h dataset), the model exhibits extremely high image-level detection accuracy, with an image-level AUROC ( Up to 97.9%, image-level AP ( The success rate reached 97.8%. In the localization of polyp lesions during colonoscopy (…). In the dataset task, the model in this embodiment can not only identify anomalies, but also perform effective pixel-level lesion segmentation, and its pixel-level AUROC ( The pixel-level AUPRO (AuPRO) reached 84.8%. The percentage reached 75.9%.

[0064] Cross-domain generalization mechanism analysis: The orthogonal constraints of the TAPD module enable P=10 anomaly prompts to automatically learn general anomaly sub-patterns that can be transferred to medical scenarios on industrial data (such as texture anomalies corresponding to changes in tissue texture, shape anomalies corresponding to organ contour deformation, density anomalies corresponding to changes in lesion grayscale, etc.), rather than the defect semantics specific to industry; the VAPG module generates personalized visual anomaly tokens for each medical image, making anomaly representation independent of the appearance of the source domain; the bidirectional adaptation mechanism further bridges the domain differences between industry and medicine through sample-level bias and text-guided refinement; the synergy of the three enables the model to maintain high-precision anomaly detection and lesion localization capabilities even on completely unseen medical images.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-sample anomaly detection method based on image-text symmetric cue collaboration, characterized in that, Specifically, the following steps are included: S1: Obtain the text input sequence and input image samples for zero-shot anomaly detection; Furthermore, the text input sequence includes a normal semantic prompt and several abnormal semantic prompts; the input image samples include, but are not limited to, images of surface defects in industrial products or medical images; S2: Construct a zero-sample anomaly detection model based on the collaboration of image and text symmetric prompts; Furthermore, the zero-shot anomaly detection model includes an image encoder module, a text encoder module, a visual anomaly prompt generation module, an image-driven text adaptation module, a text-driven visual adaptation module, and a dual-modal fusion and anomaly determination module. The image encoder module is used to obtain spatially enhanced patch features and global image features based on each input image sample using a multi-head self-attention mechanism and a two-stream attention mechanism, based on a preset CLIP visual encoder. The text encoder module is used to obtain normal text features and several abnormal text features based on the text input sequence, based on a preset CLIP text encoder; to obtain aggregated abnormal text features based on the several abnormal text features; and to obtain coarse-detected text anomaly maps based on the aggregated abnormal text features and patch features. The visual anomaly prompt generation module is used to obtain visual normal tokens and visual anomaly tokens based on global image features. The image-driven text adaptation module is used to obtain fine-detected text anomaly maps based on the coarse-detected text anomaly maps. The text-driven visual adaptation module is used to obtain adapted visual anomaly tokens based on the aggregated abnormal text features, visual anomaly tokens, and normal text features; and to obtain visual anomaly maps based on the visual normal tokens and adapted visual anomaly tokens. The dual-modal fusion and anomaly determination module is used to fuse the coarse-detected text anomaly map, the fine-detected text anomaly map, and the visual anomaly map according to the set visual fusion weights to obtain a fused anomaly map. After applying Gaussian filtering to smooth the fused anomaly map, it outputs the final anomaly heatmap and the corresponding image-level anomaly classification result. S3: Based on the text input sequence and input image samples, train the zero-shot anomaly detection model to minimize the loss value of the set joint training objective function, and obtain the optimal anomaly detection model; implement zero-shot anomaly detection based on image-text symmetric prompt collaboration based on the optimal anomaly detection model.

2. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 1, characterized in that, The methods for obtaining spatially enhanced patch features and global image features in S2 are as follows: The input image sample is input to a preset CLIP visual encoder; the CLIP visual encoder comprises 24 coding layers with different kernel parameters connected sequentially; layers 1 to 19 are defined as shallow encoder layers, and layers 20 to 24 are defined as deep encoder layers; the shallow encoder layers are used to extract general visual features from the input image sample based on standard multi-head self-attention, and the general visual features include, but are not limited to, texture, edge, and color features; the deep encoder layers are used to obtain general semantic features and anomaly-specific features based on the output of the shallow encoder layers using a two-stream attention mechanism, with the corresponding formulas as follows: In the formula: Represents general semantic features; Indicates abnormal and specific characteristics; Represents the query vector, key vector, and value vector; Indicates the feature dimension; Indicates transpose; Based on the set of layer indexes Extract multi-layer patch features from the output of each corresponding coding layer in the CLIP visual encoder; Channel-level weighted aggregation is performed on the spatial neighborhood of each patch feature using Gaussian kernels of different sizes to obtain spatially enhanced patch features and global image features. The channel-level weighted aggregation method is as follows: first, the patch features are reshaped into a two-dimensional spatial grid, then the neighborhood window of the two-dimensional spatial grid is extracted through the unfold operation, and the neighborhood window is summed element-wise using preset normalized Gaussian weights.

3. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 2, characterized in that, The method for obtaining the coarse-detection text anomaly graph described in S2 is as follows: The text input sequence is fed into a preset CLIP text encoder to extract the text semantic features of the normal semantic prompts and several abnormal semantic prompts, thereby obtaining the normal text features. With several abnormal text features ; Based on several abnormal text features, the aggregated abnormal text features are obtained as follows: In the formula: Indicates aggregated abnormal text features; The total number of anomalous text features; Indicates the first One abnormal text feature; A coarse-detected text anomaly map is obtained based on aggregated anomalous text features and patch features. This is achieved by calculating the cosine similarity between the patch features and the aggregated anomalous text features, and then... The function outputs a coarsely detected text anomaly graph.

4. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 3, characterized in that, The visual anomaly prompt generation module in S2 includes an input layer and an image conditional generation network; the input layer is used to process the global features of the image. The image is transmitted to an image conditional generation network, which includes a shared backbone network, a normal head, and an abnormal head. The shared backbone network includes a batch normalization layer, a linear layer, and a sequentially connected batch normalization layer. Activation function layer; the batch normalization layer is used for global image features. Perform normalization operation; the linear layer is used to perform a linear operation on the output of the batch normalization layer; the Activation function layers are used to perform activation operations on the output of linear layers; The normal head is used to pass through a pre-installed first fully connected layer to the... The activation function layer performs a fully connected mapping operation to obtain a normal token increment. To obtain a visually normal token; the abnormal header is used to pass through a pre-defined second fully connected layer to the... The activation function layer performs a fully connected mapping operation to obtain... An abnormal token increment To obtain A visual anomaly token, the expression of which is: In the formula: Indicates L2 regularization; Indicates a visually normal token; , Represents learnable basis vectors; Indicates the learnable gating factor; Indicates the first A visual anomaly token.

5. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 4, characterized in that, The method for obtaining the fine-grained text anomaly map described in S2 is as follows: Obtain the K patch positions with the highest anomaly scores in each coarsely detected text anomaly image; and the anomaly score is determined by the cosine similarity between the patch features and the aggregated anomaly text features, with a higher cosine similarity resulting in a higher anomaly score. The bias vector is obtained from the K patch positions using the feature-bias mapping network: In the formula: Represents the bias vector; This represents a vector concatenation operation; This represents a multilayer perceptron; This represents the patch space index with the k-th highest anomaly score in the coarse anomaly detection graph, and... ; bias vector The abnormal semantic token parameter of each abnormal semantic prompt is added additively, and the text encoder module is used to re-obtain the fine-detection text abnormal graph.

6. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 5, characterized in that, The method for obtaining the visual anomaly map described in S2 is as follows: Based on the aggregated abnormal text features and the visual abnormal token, the adapted visual abnormal token is obtained as follows: In the formula: This indicates a visual anomaly token after adaptation. This represents the refined increment of the abnormal token; Indicates the first The refined increment of an abnormal token and ; This represents a pre-defined prior mapping network; Indicates aggregated abnormal text features; Indicates the difference between anomalous and normal text features; This represents the learnable scaling factor; Indicates the first A visual anomaly token; Based on the visually normal token and the adapted visually abnormal token, the visually abnormal image is obtained as follows: In the formula: Represents a visual anomaly diagram; Indicates the first Each patch feature; Indicates the temperature coefficient; This indicates a visually normal token.

7. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 6, characterized in that, The method used in S2 to output the final anomaly heatmap and corresponding image-level anomaly classification results through the dual-modal fusion and anomaly detection module is as follows: Based on the set visual fusion weights, the coarse-detected text anomaly map, the fine-detected text anomaly map, and the visual anomaly map are fused to obtain the fused anomaly map: In the formula: Represents the fusion anomaly graph; Indicates visual fusion weights; This represents a rough anomaly detection graph for text. This represents a graph showing text anomalies detected through precise analysis. Indicates intermediate parameters; express The abbreviated form; Based on the fused anomaly map and the text classification anomaly probability, the image-level anomaly score is obtained as follows: In the formula: Represents image-level anomaly score; Indicates the probability of text classification anomalies; This represents the set of indices of the K highest-scoring patch locations in the fusion anomaly graph; express The abbreviated form; After applying Gaussian filtering to smooth the fused anomaly map, the final anomaly heatmap and corresponding image-level anomaly classification results are output by combining the image-level anomaly score.

8. The zero-sample anomaly detection method based on image-text symmetric cue collaboration according to claim 7, characterized in that, The joint training objective function described in S3 is the sum of several loss terms obtained based on the set weight coefficients; and the loss terms include text anomaly map loss, visual anomaly map loss, image-level classification loss, orthogonal constraint loss, bias regularization loss, visual-text alignment loss, visual token diversity loss, and prior residual regularization loss for text-driven visual adaptation.

Images