Zero-shot anomaly detection method and apparatus based on visually guided prototype library

Abstract

This invention discloses a zero-shot anomaly detection method and apparatus based on a visually guided prototype library, relating to the field of image processing. The method includes: inputting the image to be detected into a trained zero-shot anomaly detection model; first, generating global image features and image patch feature sets through an image encoder; then, aggregating the global image features into anchor points in the visually guided prototype library, and inputting them into corresponding adapters for adjustment to obtain context label embeddings, normal state label embeddings, and abnormal state label embeddings; concatenating the context label embeddings and normal state label embeddings, and then inputting the concatenated context label embeddings and abnormal state label embeddings into a text encoder to obtain normal prompt word embeddings and abnormal prompt word embeddings; finally, inputting the global image features, image patch feature sets, normal prompt word embeddings, and abnormal prompt word embeddings into an anomaly scoring module to obtain an anomaly map and a classification score. This invention addresses the problems of poor generalization ability and instability.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more specifically to a zero-sample anomaly detection method and apparatus based on a visually guided prototype library. Background Technology

[0002] Anomaly detection and segmentation play a crucial role in security-sensitive applications such as industrial inspection and medical image analysis. In many real-world scenarios, the collection and annotation of various anomalous samples are extremely costly, while obtaining normal data is relatively easy. This asymmetry in data distribution has driven the development of Zero-Shot Anomaly Detection (ZSAD) technology—the core of which is to train the model on an auxiliary dataset and conduct evaluations without accessing anomalous data in the target domain.

[0003] Zero-shot anomaly detection (ZSAD) leverages a base model to perform anomaly identification tasks without training data in the target domain. Recent mainstream methods generally construct ZSAD as an image-text alignment task based on CLIP. Early methods (such as WinCLIP and APRIL-GAN) employed static ensemble strategies using manually designed templates to improve prediction stability, but these methods lack adaptability to diverse anomaly morphologies. To address this issue, learnable frameworks such as AnomalyCLIP and CLIP-AD use optimized vectors learned from auxiliary data to replace manually designed tokens. Subsequent research further improves performance by incorporating visual contextual information or introducing probabilistic modeling methods, and extends this to complex routing mechanisms such as Hybrid Experts (MoE).

[0004] Despite the increasing architectural complexity of these methods, a fundamental limitation remains: they primarily expand model capacity within an abstract text parameter space. Whether using hand-crafted templates or probabilistic flow-based cue words, the generated content lacks a clear connection to the geometric space of the visual manifold, resulting in weak coupling between text embedding generation and visual geometric features. This weak coupling leads to two key drawbacks: first, insufficient robustness of the model in cross-domain scenarios; and second, oversensitivity of normal cue words to benign changes in data, while failing to accurately capture the abnormal prototypes of specific objects. Summary of the Invention

[0005] The purpose of this application is to propose a zero-sample anomaly detection method and device based on a visually guided prototype library to address the aforementioned technical problems.

[0006] In a first aspect, the present invention provides a zero-shot anomaly detection method based on a visually guided prototype library, comprising the following steps:

[0007] A zero-shot anomaly detection model based on a visually guided prototype library and an auxiliary training memory are constructed. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, the visually guided prototype library and its corresponding adapter. The auxiliary training memory is used to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, except for the image encoder and the text encoder, to obtain the trained zero-shot anomaly detection model.

[0008] The image to be detected is acquired and input into a trained zero-shot anomaly detection model. The image is then input into the prompt word embedding generation module, where it first passes through an image encoder to generate global image features and image patch feature sets. The global image features are then aggregated at anchor points in a visually guided prototype library and input into the corresponding adapters for adjustment, resulting in context label embeddings, normal state label embeddings, and abnormal state label embeddings. The context label embeddings and normal state label embeddings are concatenated and input into a text encoder to obtain normal prompt word embeddings. The context label embeddings and abnormal state label embeddings are also concatenated and input into a text encoder to obtain abnormal prompt word embeddings. The global image features, image patch feature sets, normal prompt word embeddings, and abnormal prompt word embeddings are then input into an anomaly scoring module to obtain an anomaly map and classification score. Based on the classification score, the anomaly detection result of the image is determined, and the anomaly detection result is either abnormal or normal.

[0009] Preferably, the visually guided prototype library includes a context prototype library for storing context prototypes, a normal prototype library for storing normal prototypes, and an abnormal prototype library for storing abnormal prototypes. The adapter includes a context adapter corresponding to the context prototype library, a normal state adapter corresponding to the normal prototype library, and an abnormal state adapter corresponding to the abnormal prototype library. The adapter structure includes a multilayer perceptron.

[0010] Preferably, the auxiliary training memory adopts a first-in-first-out (FIFO) mode to store memory prototypes and their corresponding anomaly detection labels. The memory prototypes are the global image features extracted after each image in the training dataset is input into the image encoder during the training process. The auxiliary training memory uses only memory prototypes in the context branch and uses memory prototypes and their corresponding anomaly detection labels in the state branch. The context branch includes a context prototype library and its corresponding context adapter, and the state branch includes a normal prototype library, an anomaly prototype library, and their respective normal state adapters and anomaly state adapters.

[0011] Preferably, the loss functions used in the memory-guided training process include context loss, state loss, classification loss, segmentation loss and cue orthogonality loss, wherein the context loss is constructed based on context branches and auxiliary training memory, and the state loss is constructed based on state branches and auxiliary training memory.

[0012] The context loss is constructed as follows:

[0013] The similarity between the global image features of the image and each memory prototype in the auxiliary training memory is calculated, and Top-K aggregation is performed to obtain the index set of the Top-K nearest neighbors. The K memory prototypes in the dataset; the index set for the Top-K nearest neighbors is... Given K memory prototypes, calculate the weight of each memory prototype as shown in the following formula:

[0014] ;

[0015] in, Represents an image. The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index j in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index j in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between the memory prototypes with index j' in the middle;

[0016] Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the model is weighted with the corresponding memory prototype to obtain the context summary prototype, as shown in the following formula:

[0017] ;

[0018] in, This represents a context summary prototype. The set of Top-K nearest neighbor indices used in the context branch is: The memory prototype with index j in the middle;

[0019] The global image features of each image in the training dataset are aggregated at anchor points in the context prototype library to obtain the corresponding context fusion prototype. The context fusion prototype is then used as the context parameterized prototype and fused with the context summary prototype to obtain the context fusion vector, as shown in the following formula:

[0020] ;

[0021] ;

[0022] in, This represents a context-mixed prototype. This indicates a context-parameterized prototype. Indicates context fusion weights, Represents the context fusion vector;

[0023] The context parameterized prototype and the context fusion vector are respectively processed through a context adapter to obtain the context-adjusted features and the context fusion features, as shown in the following equation:

[0024] ;

[0025] ;

[0026] in, This indicates the function corresponding to the context adapter. and These represent context-adjusted features and context-fused features, respectively.

[0027] L2 normalization is applied to the global image features, context adjustment features, and context fusion features of the image, respectively, to obtain the L2-normalized global image features, L2-normalized context adjustment features, and L2-normalized context fusion features. The context loss is then constructed based on these features, as shown in the following equation:

[0028] ;

[0029] in, , , These are, respectively, the global image features after L2 normalization, the context-adjusted features after L2 normalization, and the context-fused features after L2 normalization. Indicates contextual loss. Expressing expectations, Represents the L2 norm;

[0030] The process of constructing the state loss is as follows:

[0031] The global image features of the image are compared with the similarity scores of each memory prototype labeled as a normal state and each memory prototype labeled as an abnormal state in the auxiliary training memory. Top-K aggregation is then performed to obtain the index set of the Top-K nearest neighbors. The K memory prototypes and the index set of the Top-K nearest neighbors are: The K memory prototypes in the middle;

[0032] The index set for Top-K nearest neighbors is The K memory prototypes and the index set of the Top-K nearest neighbors are: For each of the K memory prototypes, calculate the index set of the Top-K nearest neighbors. The weights of each memory prototype and the set of indices of the Top-K nearest neighbors are: The weight of each memory prototype in the equation is as follows:

[0033] ;

[0034] ;

[0035] in, The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index q. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index q. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index q'; The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index p in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index p. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index p' in the middle;

[0036] Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the dataset is weighted with its corresponding memory prototype to obtain the normal state summary prototype. The index set of the Top-K nearest neighbors is then calculated. The weight of each memory prototype in the equation is weighted with the corresponding memory prototype to obtain the abnormal state summary prototype, as shown in the following formula:

[0037] ;

[0038] ;

[0039] in, The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype is indexed as q. The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype with index p in the middle. This represents the prototype summarizing the normal state. This represents a summary prototype of abnormal states;

[0040] The global image features of each image in the training dataset are aggregated at anchor points in the normal prototype library to obtain the corresponding normal state hybrid prototype. The normal state hybrid prototype is then used as the normal state parameterized prototype and fused with the normal state summary prototype to obtain the normal state fusion vector, as shown in the following formula:

[0041] ;

[0042] ;

[0043] in, This represents a hybrid prototype in its normal state. This represents the parameterized prototype for the normal state. Represents the state fusion weights. Represents the normal state fusion vector;

[0044] The global image features of each image in the training dataset are aggregated at anchor points in the anomaly prototype library to obtain the corresponding anomaly state hybrid prototype. The anomaly state hybrid prototype is used as the anomaly state parameterized prototype and fused with the anomaly state summary prototype to obtain the anomaly state fusion vector, as shown in the following formula:

[0045] ;

[0046] ;

[0047] in, This represents a mixed prototype of abnormal states. This represents the parameterized prototype of the abnormal state. Represents the state fusion weights. Represents the abnormal state fusion vector;

[0048] The normal state fusion vector and the abnormal state fusion vector are processed through a normal state adapter and an abnormal state adapter, respectively, to obtain the normal state adjustment feature and the abnormal state adjustment feature, as shown in the following formula:

[0049] ;

[0050] ;

[0051] in, and These represent the functions corresponding to the normal state adapter and the abnormal state adapter, respectively. and These represent adjustment characteristics under normal and abnormal states, respectively.

[0052] L2 normalization is applied to both the normal state adjustment features and the abnormal state adjustment features to obtain L2-normalized normal state adjustment features and L2-normalized abnormal state adjustment features. The state loss is then constructed based on the L2-normalized global image features, the L2-normalized normal state adjustment features, and the L2-normalized abnormal state adjustment features, as shown in the following equation:

[0053] ;

[0054] in, Indicates state loss. and These represent the normal state adjustment features and the abnormal state adjustment features after L2 normalization, respectively. Represents the training dataset. Represents a subset of normal images. Represents a subset of abnormal images. Indicates the abnormal state label corresponding to the image. This indicates that the image is labeled as being in a normal state. This indicates that the image is labeled as an abnormal state. This represents the total number of samples in the set; Represents cosine similarity. For interval parameters, This indicates taking the maximum value; and These represent the first weight and the second weight, respectively.

[0055] The total loss function used in the memory-guided training process is a weighted sum of context loss, state loss, classification loss, segmentation loss, and cue orthogonality loss, as shown in the following formula:

[0056] ;

[0057] in, Represents the total loss function. Represents classification loss. Indicates the partition loss. This indicates an orthogonal loss. , , and These represent the segmentation weight, cue orthogonality weight, context weight, and state weight, respectively.

[0058] The classification loss uses the cross-entropy loss function, whose expression is:

[0059] ;

[0060] in, Represents the cross-entropy loss function. This represents the classification score corresponding to the image;

[0061] The segmentation loss uses the Focal loss function and the Dice loss function, and its expression is as follows:

[0062] ;

[0063] in, This represents the actual mask corresponding to the image. This represents the anomaly graph corresponding to the image. Represents the Focal loss function. Represents the Dice loss function;

[0064] The process of constructing the orthogonality loss is as follows:

[0065] For each image in the training dataset, the context mixing prototype, normal state mixing prototype, and abnormal state mixing prototype are input into the corresponding adapter and adjusted to obtain the corresponding context-labeled embedding, normal state-labeled embedding, and abnormal state-labeled embedding. The context-labeled embedding and the normal state-labeled embedding are concatenated and input into the text encoder to obtain the normal cue word embedding. The context-labeled embedding and the abnormal state-labeled embedding are concatenated and input into the text encoder to obtain the abnormal cue word embedding. The normal cue word embeddings and abnormal cue word embeddings of all images in the training dataset are used as text embeddings and all are put into the text embedding set. The similarity between any two text embeddings in the text embedding set is calculated, and the cue orthogonality loss is calculated as shown in the following formula:

[0066]

[0067] in, This represents the total number of text embeddings in the text embedding set. and These represent the o-th and w-th text embeddings in the text embedding set, respectively. This represents the cosine similarity.

[0068] Preferably, the global image features are aggregated into anchor points in the visually guided prototype library, specifically including:

[0069] The similarity between global image features and each visual prototype in one of the prototype libraries in the visually guided prototype library is calculated using the following formula:

[0070] ;

[0071] in, This represents the index of one of the prototype libraries in the visually guided prototype library. Corresponding context prototype library, Corresponding to the normal prototype library, Corresponding exception prototype library; Represents the L2 norm. Represents cosine similarity. Represents global image features. This represents the k-th visual prototype in one of the prototype libraries within the visual guidance prototype library. , This represents the total number of visual prototypes in one of the prototype libraries within the visually guided prototype library. This represents the similarity between global image features and the k-th visual prototype in one of the prototype libraries in the visually guided prototype library;

[0072] The blending weights for each visual prototype in one of the prototype libraries in the visual-guided prototype library are calculated using the following formula:

[0073] ;

[0074] in, This represents the blending weight of the k-th visual prototype in one of the prototype libraries in the visual guidance prototype library. For temperature parameters, This represents the similarity between global image features and the i-th visual prototype in one of the prototype libraries in the visual guidance prototype library;

[0075] The hybrid prototype obtained through anchor point aggregation is defined as follows:

[0076] ;

[0077] in, This represents a hybrid prototype corresponding to one of the prototype libraries in the visually guided prototype library.

[0078] Preferably, the anomaly scoring module includes a first linear layer, an average pooling layer, and a second linear layer. The image patch feature set is first mapped to the joint embedding space through the first linear layer, and then the cosine similarity is calculated with the normal prompt word embedding and the abnormal prompt word embedding, respectively. Finally, the anomaly map is calculated, as shown in the following formula:

[0079]

[0080] in, Represents a set of image patch features. For the sigmoid function, Represents cosine similarity. This represents the anomaly graph corresponding to the image. This represents the function corresponding to the first linear layer; This indicates the embedding of error message words. This indicates that normal prompt words are embedded;

[0081] The image score is obtained by taking the maximum value from the anomaly graph, as shown in the following formula:

[0082] ;

[0083] in, This indicates taking the maximum value. Indicates the image score;

[0084] The image patch features in the image patch feature set are concatenated along the channel dimension, and then input into the average pooling layer for global average pooling along the spatial dimension. Finally, they are mapped to a space with the same dimension as the global image features through a second linear layer to obtain multi-scale features. The multi-scale features are added to the global image features to obtain the enhanced global features, as shown in the following formula:

[0085] ;

[0086] in, Represents global image features. Representing multi-scale features, This indicates enhanced global features;

[0087] The enhanced global features are compared with the normal and abnormal prompt word embeddings respectively to calculate the cosine similarity, and then the text score is calculated, as shown in the following formula:

[0088] ;

[0089] in, Indicates the text score;

[0090] The image score and the text score are summed to obtain the classification score, as shown in the following formula:

[0091] ;

[0092] in, This indicates the category score.

[0093] Secondly, the present invention provides a zero-sample anomaly detection device based on a visually guided prototype library, comprising:

[0094] The model building module is configured to build a zero-shot anomaly detection model based on a visually guided prototype library and an auxiliary training memory. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, the visually guided prototype library, and its corresponding adapter. The auxiliary training memory is used to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, except for the image encoder and the text encoder, to obtain the trained zero-shot anomaly detection model.

[0095] The detection module is configured to acquire the image to be detected and input it into a trained zero-shot anomaly detection model. The image is input into the prompt word embedding generation module, where it first passes through an image encoder to generate global image features and image patch feature sets. The global image features are then aggregated into anchor points in a visually guided prototype library and input into the corresponding adapters for adjustment, resulting in context label embeddings, normal state label embeddings, and abnormal state label embeddings. The context label embeddings and normal state label embeddings are concatenated and input into a text encoder to obtain normal prompt word embeddings. The context label embeddings and abnormal state label embeddings are concatenated and input into a text encoder to obtain abnormal prompt word embeddings. The global image features, image patch feature sets, normal prompt word embeddings, and abnormal prompt word embeddings are input into an anomaly scoring module to obtain an anomaly map and a classification score. Based on the classification score, the anomaly detection result of the image is determined, and the anomaly detection result is either abnormal or normal.

[0096] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0097] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0098] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0099] Compared with the prior art, the present invention has the following beneficial effects:

[0100] (1) The zero-shot anomaly detection method based on a visually guided prototype library proposed in this invention proposes a visually guided prototype library and adopts a retrieval-based framework to directly anchor and aggregate in the image feature space to retrieve the corresponding hybrid prototypes. The visually guided prototype library includes learnable context, normal and abnormal prototype libraries, and generates deterministic prompt word embeddings through visual retrieval to ensure strict alignment with the basic visual data manifold.

[0101] (2) The zero-sample anomaly detection method based on visually guided prototype library proposed in this invention guides the training target through the auxiliary training memory library, effectively avoids the prototype collapse problem, forces the differentiation between normal and abnormal prototypes, and does not increase the computational burden of the inference stage.

[0102] (3) The zero-sample anomaly detection method based on the visually guided prototype library proposed in this invention uses a deterministic prototype mixing strategy to strongly bind the generation process of the prompt word embedding with the visual manifold. While avoiding the instability of sampling methods, it achieves the robust generalization ability of the model and does not introduce any abnormal data from the target domain during the training process. Attached Figure Description

[0103] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0104] Figure 1 This is a flowchart illustrating a zero-shot anomaly detection method based on a visually guided prototyping library, as an embodiment of this application.

[0105] Figure 2 This is a schematic diagram illustrating the training and inference processes of the zero-shot anomaly detection model in the zero-shot anomaly detection method based on a visually guided prototype library, as described in an embodiment of this application.

[0106] Figure 3 A qualitative comparison of anomaly segmentation results between different zero-shot anomaly detection methods and the zero-shot anomaly detection method based on a visually guided prototype library of the embodiments of this application;

[0107] Figure 4 The zero-shot generalization stability of the zero-shot anomaly detection method based on the visually guided prototype library, as an embodiment of this application, is compared during the training process under "no memory guidance" and "with memory guidance".

[0108] Figure 5This is a comparison chart of the inference efficiency (latency and memory) of the zero-shot anomaly detection method based on the visually guided prototype library of this application and the baseline method on the MVTec-AD dataset.

[0109] Figure 6 A t-SNE visualization of the visual-language alignment of the zero-shot anomaly detection method based on a visually guided prototyping library, as an embodiment of this application;

[0110] Figure 7 This is a schematic diagram of a zero-sample anomaly detection device based on a visually guided prototyping library, as an embodiment of this application.

[0111] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0112] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0113] Figure 1 The illustration shows an embodiment of this application providing a zero-shot anomaly detection method based on a visually guided prototyping library, comprising the following steps:

[0114] S1. Construct a zero-shot anomaly detection model and an auxiliary training memory based on a visually guided prototype library. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, a visually guided prototype library, and its corresponding adapter. Use the auxiliary training memory to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, except for the image encoder and the text encoder, to obtain the trained zero-shot anomaly detection model.

[0115] In a specific embodiment, the visual guidance prototype library includes a context prototype library for storing context prototypes, a normal prototype library for storing normal prototypes, and an abnormal prototype library for storing abnormal prototypes. The adapter includes a context adapter corresponding to the context prototype library, a normal state adapter corresponding to the normal prototype library, and an abnormal state adapter corresponding to the abnormal prototype library. The adapter structure includes a multilayer perceptron.

[0116] In a specific embodiment, the auxiliary training memory adopts a first-in-first-out (FIFO) mode to store memory prototypes and their corresponding anomaly detection labels. The memory prototypes are the global image features extracted after each image in the training dataset is input into the image encoder during the training process. The auxiliary training memory uses only memory prototypes in the context branch and uses memory prototypes and their corresponding anomaly detection labels in the state branch. The context branch includes a context prototype library and its corresponding context adapter, and the state branch includes a normal prototype library, an anomaly prototype library, and their respective normal state adapters and anomaly state adapters.

[0117] Specifically, the zero-shot anomaly detection model based on a visually guided prototype library proposed in this application introduces a visually guided prototype library and its corresponding adapter on top of the standard zero-shot anomaly detection (ZSAD) model, and incorporates an auxiliary training memory library during training. The standard zero-shot anomaly detection model uses a vision-language model (VLM); let the input image be... , Represents the set of real numbers. and These represent the height and width of the image, respectively. Visual-language models, exemplified by Contrastive Language Image Pre-training (CLIP), consist of two core modules: an image encoder and a text encoder. The global image features output by the image encoder (corresponding to CLS tags) are denoted as... , Let be the number of channels, and denote the set of image patch features output by the multi-layer network as . ,in, For the features of the l-th image patch, This represents the total number of image patches.

[0118] The embodiments of this application transform the cue word design problem in the ZSAD task into a process of selecting and combining a small number of visual prototypes for each input image. Unlike existing methods that model cue words as a single learnable module or sample cue words from implicit probability distributions, the embodiments of this application explicitly specify that: for a given input image The CLIP model needs to focus on which contextual prototypes and which normal / abnormal state prototypes. In the embodiments of this application, a visually guided prototype library is constructed to store learnable visual prototypes. Given an input image, the corresponding global image features are used as queries to retrieve contextual and state prototypes from the visually guided prototype library. These visual prototypes are aggregated and mapped via an adapter, thereby adjusting the CLIP's cue word embeddings.

[0119] The visual guidance prototype library constructed in the embodiments of this application includes a context prototype library, a normal prototype library, and an abnormal prototype library, which are used to capture context prototypes, normal prototypes, and abnormal prototypes, respectively; wherein, the context prototype library is represented as: A normal prototype library is represented as: The exception prototype library is represented as: Each visual prototype in the Visual Guidance Prototype Library can be viewed as an anchor point, providing an explicit parameterized representation of the cue word space and the corresponding contextual prototype, normal prototype, or abnormal prototype in CLIP's image feature space.

[0120] During training, embodiments of this application maintain a first-in-first-out (FIFO) auxiliary training memory. This is used to store the global image features and anomaly detection labels of recent images, and uses the global image features as a memory prototype. In the first... Auxiliary training memory bank at each time step It consists of several sample pairs, among which This represents the memory prototype in the j-th sample pair. This represents the anomaly detection label in the j-th sample pair, used to mark the image as normal (0) or abnormal (1). This represents the total number of sample pairs. Limited by a fixed capacity, new global image features will emerge as training progresses. It will be added and become a memory prototype. When the capacity reaches the limit, the earliest stored sample pair will be removed.

[0121] For simplicity, when only the memory prototype is used, this auxiliary training memory is considered as an unlabeled context memory. When anomaly detection labels are required, the auxiliary training memory is treated as a labeled state memory. In practical applications, and It is a copy of the same global image features, used only for calculations in different branches. and These represent the total number of samples in the unlabeled context memory and the total number of sample pairs in the labeled state memory, respectively.

[0122] In a specific embodiment, the loss function used in the memory-guided training process includes context loss, state loss, classification loss, segmentation loss and cue orthogonality loss, wherein the context loss is constructed based on context branches and auxiliary training memory, and the state loss is constructed based on state branches and auxiliary training memory.

[0123] The context loss is constructed as follows:

[0124] The similarity between the global image features of the image and each memory prototype in the auxiliary training memory is calculated, and Top-K aggregation is performed to obtain the index set of the Top-K nearest neighbors. The K memory prototypes in the dataset; the index set for the Top-K nearest neighbors is... Given K memory prototypes, calculate the weight of each memory prototype as shown in the following formula:

[0125] ;

[0126] in, Represents an image. The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index j in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index j in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between the memory prototypes with index j' in the middle;

[0127] Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the model is weighted with the corresponding memory prototype to obtain the context summary prototype, as shown in the following formula:

[0128] ;

[0129] in, This represents a context summary prototype. The set of Top-K nearest neighbor indices used in the context branch is: The memory prototype with index j in the middle;

[0130] The global image features of each image in the training dataset are aggregated at anchor points in the context prototype library to obtain the corresponding context fusion prototype. The context fusion prototype is then used as the context parameterized prototype and fused with the context summary prototype to obtain the context fusion vector, as shown in the following formula:

[0131] ;

[0132] ;

[0133] in, This represents a context-mixed prototype. This indicates a context-parameterized prototype. Indicates context fusion weights, Represents the context fusion vector;

[0134] The context parameterized prototype and the context fusion vector are respectively processed through a context adapter to obtain the context-adjusted features and the context fusion features, as shown in the following equation:

[0135] ;

[0136] ;

[0137] in, This indicates the function corresponding to the context adapter. and These represent context-adjusted features and context-fused features, respectively.

[0138] L2 normalization is applied to the global image features, context adjustment features, and context fusion features of the image, respectively, to obtain the L2-normalized global image features, L2-normalized context adjustment features, and L2-normalized context fusion features. The context loss is then constructed based on these features, as shown in the following equation:

[0139] ;

[0140] in, , , These are, respectively, the global image features after L2 normalization, the context-adjusted features after L2 normalization, and the context-fused features after L2 normalization. Indicates contextual loss. Expressing expectations, Represents the L2 norm;

[0141] The process of constructing the state loss is as follows:

[0142] The global image features of the image are compared with the similarity scores of each memory prototype labeled as a normal state and each memory prototype labeled as an abnormal state in the auxiliary training memory. Top-K aggregation is then performed to obtain the index set of the Top-K nearest neighbors. The K memory prototypes and the index set of the Top-K nearest neighbors are: The K memory prototypes in the middle;

[0143] The index set for Top-K nearest neighbors is The K memory prototypes and the index set of the Top-K nearest neighbors are: For each of the K memory prototypes, calculate the index set of the Top-K nearest neighbors. The weights of each memory prototype and the set of indices of the Top-K nearest neighbors are: The weight of each memory prototype in the equation is as follows:

[0144] ;

[0145] ;

[0146] in, The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index q. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index q. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index q'; The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index p in the middle. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index p. The set of indices representing the global image features and Top-K nearest neighbors of an image is: The similarity between memory prototypes with index p' in the middle;

[0147] Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the dataset is weighted with its corresponding memory prototype to obtain the normal state summary prototype. The index set of the Top-K nearest neighbors is then calculated. The weight of each memory prototype in the equation is weighted with the corresponding memory prototype to obtain the abnormal state summary prototype, as shown in the following formula:

[0148] ;

[0149] ;

[0150] in, The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype is indexed as q. The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype with index p in the middle. This represents the prototype summarizing the normal state. This represents a summary prototype of abnormal states;

[0151] The global image features of each image in the training dataset are aggregated at anchor points in the normal prototype library to obtain the corresponding normal state hybrid prototype. The normal state hybrid prototype is then used as the normal state parameterized prototype and fused with the normal state summary prototype to obtain the normal state fusion vector, as shown in the following formula:

[0152] ;

[0153] ;

[0154] in, This represents a hybrid prototype in its normal state. This represents the parameterized prototype for the normal state. Represents the state fusion weights. Represents the normal state fusion vector;

[0155] The global image features of each image in the training dataset are aggregated at anchor points in the anomaly prototype library to obtain the corresponding anomaly state hybrid prototype. The anomaly state hybrid prototype is used as the anomaly state parameterized prototype and fused with the anomaly state summary prototype to obtain the anomaly state fusion vector, as shown in the following formula:

[0156] ;

[0157] ;

[0158] in, This represents a mixed prototype of abnormal states. This represents the parameterized prototype of the abnormal state. Represents the state fusion weights. Represents the abnormal state fusion vector;

[0159] The normal state fusion vector and the abnormal state fusion vector are processed through a normal state adapter and an abnormal state adapter, respectively, to obtain the normal state adjustment feature and the abnormal state adjustment feature, as shown in the following formula:

[0160] ;

[0161] ;

[0162] in, and These represent the functions corresponding to the normal state adapter and the abnormal state adapter, respectively. and These represent adjustment characteristics under normal and abnormal states, respectively.

[0163] L2 normalization is applied to both the normal state adjustment features and the abnormal state adjustment features to obtain L2-normalized normal state adjustment features and L2-normalized abnormal state adjustment features. The state loss is then constructed based on the L2-normalized global image features, the L2-normalized normal state adjustment features, and the L2-normalized abnormal state adjustment features, as shown in the following equation:

[0164] ;

[0165] in, Indicates state loss. and These represent the normal state adjustment features and the abnormal state adjustment features after L2 normalization, respectively. Represents the training dataset. Represents a subset of normal images. Represents a subset of abnormal images. Indicates the abnormal state label corresponding to the image. This indicates that the image is labeled as being in a normal state. This indicates that the image is labeled as an abnormal state. This represents the total number of samples in the set; Represents cosine similarity. For interval parameters, This indicates taking the maximum value; and These represent the first weight and the second weight, respectively.

[0166] The total loss function used in the memory-guided training process is a weighted sum of context loss, state loss, classification loss, segmentation loss, and cue orthogonality loss, as shown in the following formula:

[0167] ;

[0168] in, Represents the total loss function. Represents classification loss. Indicates the partition loss. This indicates an orthogonal loss. , , and These represent the segmentation weight, cue orthogonality weight, context weight, and state weight, respectively.

[0169] The classification loss uses the cross-entropy loss function, whose expression is:

[0170] ;

[0171] in, Represents the cross-entropy loss function. This represents the classification score corresponding to the image;

[0172] The segmentation loss uses the Focal loss function and the Dice loss function, and its expression is as follows:

[0173] ;

[0174] in, This represents the actual mask corresponding to the image. This represents the anomaly graph corresponding to the image. Represents the Focal loss function. Represents the Dice loss function;

[0175] The process of constructing the orthogonality loss is as follows:

[0176] For each image in the training dataset, the context mixing prototype, normal state mixing prototype, and abnormal state mixing prototype are input into the corresponding adapter and adjusted to obtain the corresponding context-labeled embedding, normal state-labeled embedding, and abnormal state-labeled embedding. The context-labeled embedding and the normal state-labeled embedding are concatenated and input into the text encoder to obtain the normal cue word embedding. The context-labeled embedding and the abnormal state-labeled embedding are concatenated and input into the text encoder to obtain the abnormal cue word embedding. The normal cue word embeddings and abnormal cue word embeddings of all images in the training dataset are used as text embeddings and all are put into the text embedding set. The similarity between any two text embeddings in the text embedding set is calculated, and the cue orthogonality loss is calculated as shown in the following formula:

[0177]

[0178] in, This represents the total number of text embeddings in the text embedding set. and These represent the o-th and w-th text embeddings in the text embedding set, respectively. This represents the cosine similarity.

[0179] Specifically, given global image features As a query, memory prototypes are retrieved for the context branch by using cosine similarity and Top-K aggregation. Softmax weighted combination of the memory prototypes of the Top-K nearest neighbors is applied to summarize the memory prototypes similar to the current image in the unlabeled context memory, resulting in a context summary prototype, which provides a reference for the context branch. For the state branch, the same auxiliary training memory is reused, but the search scope is limited to the labeled state memory with anomaly detection labels. The same Softmax weighted combination method is used, but calculation is only performed within the nearest neighbor set specific to the anomaly detection label, resulting in normal state summary prototypes and abnormal state summary prototypes. The context branch consists of a context prototype library and its corresponding context adapter, while the state branch consists of a normal prototype library, an abnormal prototype library, and their respective normal state adapters and abnormal state adapters. Specifically, the normal prototype library and its corresponding normal state adapter constitute the normal state branch, and the abnormal prototype library and its corresponding abnormal state adapter constitute the abnormal state branch.

[0180] In the subsequent alignment loss, these memory prototypes are always combined with the hybrid prototypes that the current image depends on. This fusion means that the supervisory signal simultaneously relies on the current global image features and its most similar nearest neighbors in the auxiliary training memory. This joint constraint improves both stability and generalization: using only the current CLS label of a single sample is prone to overfitting and producing updates with high variance, while relying solely on historical nearest neighbors binds the prototype to coarse data patterns and ignores specific cues; the combination of both stabilizes the update process, generates a smoother prototype library with stronger state consistency, and thus transfers better to unseen categories.

[0181] Context loss and state loss are constructed for context branches and state branches respectively. First, context alignment and distillation are performed on the context branches. The abnormal state hybrid prototype is used as the abnormal state parameterized prototype, which can be further fused with the abnormal state summary prototype to obtain the abnormal state fusion vector. The abnormal state parameterized prototype and the abnormal state fusion vector are then passed through the context adapter. The context adjustment features and context fusion features are obtained. To align the context aggregation prototype with the global image features and to distill the effect of the memory prototype into the context prototype library, the global image features, context adjustment features, and context fusion features are L2 normalized respectively. Then, the context loss is constructed by combining the L2 normalized context adjustment features and L2 normalized context fusion features with the L2 normalized global image features. . minimize This allows the memory prototype to pull the context prototype library toward the current image and the data formed by its neighbors, while simultaneously training the context prototype library to align with global image features without relying on memory. This implicit prototype learning signal prompts the context prototypes to organize into meaningful patterns, rather than collapsing into a few overused samples.

[0182] Secondly, state branches are aligned and separated. A similar fusion strategy is used for both normal and abnormal state hybrid prototypes, so each state branch is guided by the current image and its nearest labeled neighbor. To ensure that the normal and abnormal prototype libraries encode different behaviors, embodiments of this application design a state loss, the function of which is as follows:

[0183] (i) Optimize the normal state adjustment features and abnormal state adjustment features that match the state branch corresponding to the anomaly detection label (normal or abnormal) of the image and pull them towards the global image features;

[0184] (ii) Push away the normal state adjustment features and abnormal state adjustment features of the opposite state branches.

[0185] (iii) Force an interval between normal state adjustment features and abnormal state adjustment features.

[0186] In one example, the state loss The state loss causes the normal state branches and abnormal state branches to converge to different regions in the feature space. If the visual guidance prototype library... When a crash occurs, the separation term and the error branch penalty term in the state loss will result in a significant loss, thus effectively suppressing the crash and ensuring the generation of diverse and state-consistent visual prototypes. The context loss and state loss are combined to form the memory-guided prototype loss.

[0187] To supervise image-level anomaly classification and pixel-level segmentation tasks on known categories, embodiments of this application also introduce the standard ZSAD loss. Given an image-level classification score... Anomaly detection tags and pixel-level anomaly images Real mask Define the standard ZSAD loss, including classification loss based on binary cross-entropy. And segmentation loss based on Focal loss and Dice loss .

[0188] To avoid redundancy and encourage the generation of diverse cue representations, embodiments of this application combine normal cue word embeddings and abnormal cue word embeddings into a text embedding set. Orthogonality regularization is applied to the text embeddings in this set to construct a cue orthogonality loss. In the embodiments of this application, the total loss function used during the training of the zero-shot anomaly detection model not only combines the memory-guided prototype loss with the standard ZSAD loss, but also further combines it with the cue orthogonality loss.

[0189] To comprehensively evaluate the zero-shot anomaly detection (ZSAD) method based on a visually guided prototype library proposed in this application, experiments were conducted on 15 real-world datasets covering industrial and medical fields. The industrial datasets include standard benchmarks such as MVTec-AD, BTAD, KSDD2, RSDD, and DAGM; the medical datasets include DTD-Synthetic, HeadCT, BrainMRI, CVC-ClinicDB, CVC-ColonDB, and ISIC, covering tasks such as brain tumor classification, polyp segmentation, and skin cancer detection.

[0190] In the ZSAD setting, the embodiments of this application employ a disjoint protocol: the zero-shot anomaly detection model is primarily trained on the MVTec-AD dataset, and then directly infers on other unseen datasets without additional adaptation. This application uses standard metrics to evaluate the performance of anomaly classification and segmentation tasks:

[0191] (1) Image-level classification, reporting receiver operating feature area under curve (AUROC) and average precision (AP);

[0192] (2) Pixel-level segmentation, using AUROC and region overlap rate (PRO) to evaluate positioning accuracy.

[0193] The embodiments of this application use the CLIP (ViT-L / 14-336) model as the backbone network, with an input image size of 518×518. Features from layers 6, 12, 18, and 24 are extracted as image patch features for prompt generation. Setting the prototype library size: Context Prototype Library Normal / abnormal state prototype library ; Memory capacity: in context branches In the state branch Search for Top-5 nearest neighbors. And train using the Adam optimizer for 40 epochs, with the learning rate set to... Other parameters, such as the learning rate, are set to... .

[0194] The zero-shot anomaly detection method based on a visually guided prototype library (denoted as VGPB) proposed in the embodiments of this application is compared with current mainstream ZSAD methods (such as WinCLIP, APRIL-GAN, CLIP-AD, AnomalyCLIP, AdaCLIP, Bayes-PFL, etc.). To ensure a fair comparison, all methods follow a unified disjoint protocol.

[0195] Table 1: Comparison results with the current mainstream ZSAD method on 15 datasets

[0196]

[0197] Table 1 compares the VGPB proposed in this application with the current mainstream ZSAD method. In industrial benchmark tests, VGPB consistently outperforms methods that rely on globally learnable shared vectors (such as AnomalyCLIP). These methods struggle to capture instance-specific changes due to rigid representations, while VGPB achieves dynamic adaptation through image conditional retrieval. This application also observes that Bayes-PFL, an equal-probability model, is affected by random sampling, often exhibiting variance fluctuations and inaccurate boundary localization on complex textures. In contrast, VGPB employs deterministic prototype retrieval to stably anchor features, demonstrating superior stability on challenging datasets such as DTD-Synthetic. In the medical field, despite a slight decrease in image-level performance due to global semantic differences between industrial training data and medical targets, VGPB still maintains top-tier pixel-level accuracy. This confirms that the visual prototypes learned by the embodiments of this application can robustly transfer to unseen categories even when global semantic alignment is challenged.

[0198] refer to Figure 3 Baseline methods such as WinCLIP and April-GAN often generate diffuse anomaly maps with significant background noise, failing to accurately delineate anomaly boundaries. In contrast, VGPB generates highly concentrated anomaly maps with extremely high fidelity to real labels. This accuracy is evident in both fine-grained industrial anomalies (such as printed circuit boards) and complex medical lesions (such as skin and colon lesions), demonstrating that the visual guidance prototype library can effectively anchor semantic prompts to visual prototypes, achieving robust localization.

[0199] The embodiments of this application also conducted ablation experiments on the MVTec-AD dataset to systematically explore the role of each component in VGPB, and the results are summarized in Table 2.

[0200] Table 2: Ablation study results of different components on the MVTec-AD dataset

[0201]

[0202] To verify the core assumptions of the VGPB framework, embodiments of this application remove the retrieval mechanism and replace the prototype library with statically learnable vectors directly optimized through backpropagation (denoted as "no retrieval"). Referring to Table 2, this leads to a significant performance decrease (image-level AUROC decrease of 1.4%, pixel-level PRO decrease of 3.6%), demonstrating that anchoring cues to the visual feature space is superior to learning unconstrained textual parameters. Compared to visual guidance methods, statically learnable vectors struggle to capture diverse and anomalous prototypes.

[0203] One of VGPB's core contributions is preventing prototype crashes through a memory-guided training mechanism. When the auxiliary training memory is disabled (denoted as "memory-free guidance"), the pixel-level PRO drops significantly from 88.1% to 84.2%, indicating that the prototype degenerates into a generic prototype in the absence of geometric constraints. To further verify this, in Figure 4 The zero-shot generalization curves are visualized in the diagram. Notably, both settings use the same maximum epoch configuration, but the baseline model (blue curve) triggers early stopping prematurely—its performance on unseen classes drops rapidly after a few training epochs, indicating overfitting to the auxiliary data. In contrast, VGPB (red curve) maintains robust generalization capabilities and continues to improve, demonstrating that the auxiliary training memory effectively regularizes the prototype space. Finally, removing the cue orthogonality loss results in a slight decrease in metrics, validating its role in ensuring representation diversity. In summary, VGPB achieves optimal performance only when all components work synergistically.

[0204] like Figure 5 As shown, VGPB benefits from deterministic retrieval, avoiding the high sampling costs of streaming model-like methods. Its inference latency is only 102.3 milliseconds, significantly faster than WinCLIP and Bayes-PFL models, while comparable to lightweight baseline models such as CLIP-AD. Combined with a compact memory footprint of 2.3GB, VGPB demonstrates high practicality in industrial deployments.

[0205] To verify the coupling between the visual and textual modalities, this application... Figure 6 The distribution of t-SNE features is visualized in the image. Unlike static cue word embeddings (blue), which form rigid clusters isolated from the visual manifold, the cue word embeddings (red) in VGPB are dynamically dispersed and closely aligned with image features (gray). Crucially, the connecting lines highlight the instance-specific nature of the VGPB framework, indicating that each cue word embedding is explicitly anchored to its corresponding visual input. This confirms that the retrieval mechanism effectively mitigates the visual-semantic misalignment problem by generating representations that fit the underlying visual geometry.

[0206] S2: Acquire the image to be detected and input it into the trained zero-shot anomaly detection model. The image is input into the prompt word embedding generation module. First, it passes through the image encoder to generate global image features and image patch feature sets. The global image features are then aggregated into anchor points in the visual guidance prototype library and input into the corresponding adapters for adjustment to obtain context label embedding, normal state label embedding, and abnormal state label embedding. The context label embedding and normal state label embedding are concatenated and input into the text encoder to obtain normal prompt word embedding. The context label embedding and abnormal state label embedding are concatenated and input into the text encoder to obtain abnormal prompt word embedding. The global image features, image patch feature sets, normal prompt word embedding, and abnormal prompt word embedding are input into the anomaly scoring module to obtain an anomaly map and classification score. Based on the classification score, the anomaly detection result of the image is determined, and the anomaly detection result is either abnormal or normal.

[0207] In a specific embodiment, the global image features are aggregated into anchor points in the visually guided prototype library, specifically including:

[0208] The similarity between global image features and each visual prototype in one of the prototype libraries in the visually guided prototype library is calculated using the following formula:

[0209] ;

[0210] in, This represents the index of one of the prototype libraries in the visually guided prototype library. Corresponding context prototype library, Corresponding to the normal prototype library, Corresponding exception prototype library; Represents the L2 norm. Represents cosine similarity. Represents global image features. This represents the k-th visual prototype in one of the prototype libraries within the visual guidance prototype library. , This represents the total number of visual prototypes in one of the prototype libraries within the visually guided prototype library. This represents the similarity between global image features and the k-th visual prototype in one of the prototype libraries in the visually guided prototype library;

[0211] The blending weights for each visual prototype in one of the prototype libraries in the visual-guided prototype library are calculated using the following formula:

[0212] ;

[0213] in, This represents the blending weight of the k-th visual prototype in one of the prototype libraries in the visual guidance prototype library. For temperature parameters, This represents the similarity between global image features and the i-th visual prototype in one of the prototype libraries in the visual guidance prototype library;

[0214] The hybrid prototype obtained through anchor point aggregation is defined as follows:

[0215] ;

[0216] in, This represents a hybrid prototype corresponding to one of the prototype libraries in the visually guided prototype library.

[0217] In a specific embodiment, the anomaly scoring module includes a first linear layer, an average pooling layer, and a second linear layer. The image patch feature set is first mapped to the joint embedding space through the first linear layer, and then cosine similarity is calculated with the normal prompt word embedding and the abnormal prompt word embedding, respectively. Finally, the anomaly map is calculated, as shown in the following formula:

[0218]

[0219] in, Represents a set of image patch features. For the sigmoid function, Represents cosine similarity. This represents the anomaly graph corresponding to the image. This represents the function corresponding to the first linear layer; This indicates the embedding of error message words. This indicates that normal prompt words are embedded;

[0220] The image score is obtained by taking the maximum value from the anomaly graph, as shown in the following formula:

[0221] ;

[0222] in, This indicates taking the maximum value. Indicates the image score;

[0223] The image patch features in the image patch feature set are concatenated along the channel dimension, and then input into the average pooling layer for global average pooling along the spatial dimension. Finally, they are mapped to a space with the same dimension as the global image features through a second linear layer to obtain multi-scale features. The multi-scale features are added to the global image features to obtain the enhanced global features, as shown in the following formula:

[0224] ;

[0225] in, Represents global image features. Representing multi-scale features, This indicates enhanced global features;

[0226] The enhanced global features are compared with the normal and abnormal prompt word embeddings respectively to calculate the cosine similarity, and then the text score is calculated, as shown in the following formula:

[0227] ;

[0228] in, Indicates the text score;

[0229] The image score and the text score are summed to obtain the classification score, as shown in the following formula:

[0230] ;

[0231] in, This indicates the category score.

[0232] Specifically, during the inference process, the auxiliary training memory is no longer used; only the frozen CLIP backbone network and the learned visual guidance prototype library are used. For images from unseen categories, they are first input into the cue word embedding generation module, and then calculated through deterministic retrieval. By generating image-conditional normal and abnormal prompt word embeddings, these can be further combined with global image features and image patch feature sets and input into the anomaly scoring module to participate in the generation of anomaly maps and the calculation of classification scores.

[0233] In the prompt word embedding generation module, the image to be detected first passes through an image encoder to generate corresponding global image features and image patch feature sets. The global image features are then aggregated at anchor points in a visually guided prototype library to retrieve the corresponding contextual hybrid prototype. Normal state hybrid prototype Hybrid Prototype with Abnormal States The retrieved hybrid prototype is used to adjust the learnable context and is processed before the input text encoder; the contextual hybrid prototype is then used to adjust the learnable context. Normal state hybrid prototype Hybrid Prototype with Abnormal States The inputs are fed into the corresponding adapters and adjusted to obtain context label embeddings, normal state label embeddings, and abnormal state label embeddings. This adjustment is achieved by interpolating the adjusted features output by the adapters with the corresponding basic context label embeddings. Basic normal state tag embedding Or embedding basic abnormal state markers The addition is implemented to obtain the corresponding context tag embedding. Normal state marker embedding and abnormal state tag embedding Among them, the basic context tag embedding Basic normal state tag embedding and basic abnormal state marker embedding These are all learnable parameter vectors that are iteratively updated through backpropagation during model training, used to provide a general semantic representation basis.

[0234] Embed context tags on this basis Embedded with normal state markers The final normal prototype is obtained by piecing together the parts. Embed context tags Embedded with abnormal state markers The final abnormal prototype is obtained by splicing them together. The final normal prototype and the final abnormal prototype are input into the text encoder respectively to obtain the corresponding normal prompt word embeddings. and embedding of error message words Through the first linear layer The image patch feature set is mapped to the joint embedding space, and then its embedding with the normal prompt word is calculated separately. and embedding of error message words The cosine similarity between the images is used to obtain the anomaly map. The classification score integrates the overall semantics and fine-grained semantics from both the text and image branches. For the text branch, image patch features from different layers are first concatenated along the channel dimension, then globally averaged pooled along the spatial dimension, and finally mapped to the image using a learnable second linear layer. Space that matches the dimensions, and with The scores are added together to obtain the enhanced global features, and the final text score is calculated. For the image branch, the image score is the maximum value of the anomaly map. The final image-level classification score is the sum of the text score and the image score.

[0235] Unlike methods that rely on random sampling or complex routing, VGPB uses a single forward propagation, with computational complexity comparable to the standard CLIP baseline, and is faster in practice than recent cue-based baseline methods.

[0236] Further reference Figure 7 As an implementation of the methods shown in the above figures, this application provides an embodiment of a zero-sample anomaly detection device based on a visually guided prototype library. This device embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0237] This application provides a zero-shot anomaly detection device based on a visually guided prototype library, comprising:

[0238] Model building module 1 is configured to build a zero-shot anomaly detection model based on a visually guided prototype library and an auxiliary training memory. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, a visually guided prototype library and its corresponding adapter. The auxiliary training memory is used to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, except for the image encoder and the text encoder, to obtain the trained zero-shot anomaly detection model.

[0239] Detection module 2 is configured to acquire the image to be detected and input it into a trained zero-shot anomaly detection model. The image is input into the prompt word embedding generation module, where it first passes through an image encoder to generate global image features and image patch feature sets. The global image features are then aggregated into anchor points in a visually guided prototype library and input into the corresponding adapters for adjustment, resulting in context label embedding, normal state label embedding, and abnormal state label embedding. The context label embedding and normal state label embedding are concatenated and input into a text encoder to obtain normal prompt word embedding. The context label embedding and abnormal state label embedding are concatenated and input into a text encoder to obtain abnormal prompt word embedding. The global image features, image patch feature sets, normal prompt word embedding, and abnormal prompt word embedding are input into an anomaly scoring module to obtain an anomaly map and classification score. Based on the classification score, the anomaly detection result of the image is determined, and the anomaly detection result is either abnormal or normal.

[0240] Figure 8 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 8 As shown, the electronic device of this embodiment includes a processor 801 and a memory 802; wherein the memory 802 is used to store computer execution instructions; and the processor 801 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0241] Alternatively, the memory 802 can be either standalone or integrated with the processor 801.

[0242] When the memory 802 is set up independently, the electronic device also includes a bus 803 for connecting the memory 802 and the processor 801.

[0243] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by processor 801, implement the above method.

[0244] This invention also provides a computer program product, including a computer program that, when executed by a processor 801, implements the above-described method.

[0245] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0246] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0247] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit formed by the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0248] The integrated modules implemented as software functional modules described above can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor 801 to execute some steps of the methods of the various embodiments of this application.

[0249] It should be understood that the processor 801 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor, or the processor 801 can be any conventional processor 801. The steps of the method disclosed in this invention can be directly manifested as the hardware processor 801 executing the steps, or as a combination of hardware and software modules within the processor 801 executing the steps.

[0250] The memory 802 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0251] Bus 803 can be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Bus 803 can be divided into address bus, data bus, control bus, etc. For ease of illustration, the bus 803 in the accompanying drawings of this application is not limited to only one bus 803 or one type of bus 803.

[0252] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0253] An exemplary storage medium is coupled to a processor 801, enabling the processor 801 to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor 801. The processor 801 and the storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor 801 and the storage medium can exist as discrete components in an electronic device or a host device.

[0254] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-shot anomaly detection method based on a visually guided prototype library, characterized in that, Includes the following steps: A zero-shot anomaly detection model based on a visually guided prototype library and an auxiliary training memory are constructed. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, the visually guided prototype library and its corresponding adapter. The auxiliary training memory is used to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, except for the image encoder and the text encoder, to obtain the trained zero-shot anomaly detection model. An image to be detected is acquired and input into the trained zero-shot anomaly detection model. The image is then input into the prompt word embedding generation module, where it first passes through the image encoder to generate a global image feature set and an image patch feature set. The global image features are then aggregated at anchor points in the visual guidance prototype library, and then input into the corresponding adapters for adjustment to obtain context label embedding, normal state label embedding, and anomaly state label embedding. The context label embedding and normal state label embedding are concatenated and input into the text encoder to obtain the normal prompt word embedding. The context marker embedding and the abnormal state marker embedding are concatenated and input into the text encoder to obtain the abnormal prompt word embedding; the global image features, image block feature set, normal prompt word embedding and abnormal prompt word embedding are input into the abnormal scoring module to obtain the abnormal image and classification score; based on the classification score, the abnormal detection result of the image is determined, and the abnormal detection result is abnormal or normal.

2. The zero-shot anomaly detection method based on a visually guided prototype library according to claim 1, characterized in that, The visual guidance prototype library includes a context prototype library for storing context prototypes, a normal prototype library for storing normal prototypes, and an abnormal prototype library for storing abnormal prototypes. The adapter includes a context adapter corresponding to the context prototype library, a normal state adapter corresponding to the normal prototype library, and an abnormal state adapter corresponding to the abnormal prototype library. The adapter structure includes a multilayer perceptron.

3. The zero-shot anomaly detection method based on a visually guided prototype library according to claim 2, characterized in that, The auxiliary training memory adopts a first-in-first-out mode to store memory prototypes and their corresponding anomaly detection labels. The memory prototypes are the global image features extracted after each image in the training dataset is input into the image encoder during the training process. The auxiliary training memory library uses only memory prototypes in the context branch and uses memory prototypes and their corresponding anomaly detection labels in the state branch. The context branch includes a context prototype library and its corresponding context adapter, and the state branch includes a normal prototype library, an abnormal prototype library and their respective normal state adapters and abnormal state adapters.

4. The zero-shot anomaly detection method based on a visually guided prototype library according to claim 3, characterized in that, The loss functions used in the memory-guided training process include context loss, state loss, classification loss, segmentation loss, and cue orthogonality loss, wherein the context loss is constructed based on the context branches and the auxiliary training memory bank, and the state loss is constructed based on the state branches and the auxiliary training memory bank; The context loss is constructed as follows: The similarity between the global image features of the image and each memory prototype in the auxiliary training memory is calculated, and Top-K aggregation is performed to obtain the Top-K nearest neighbor index set. The K memory prototypes in the dataset; the index set for the Top-K nearest neighbors is... Given K memory prototypes, calculate the weight of each memory prototype as shown in the following formula: ； in, Represents an image. The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index j in the middle. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between memory prototypes with index j in the middle. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between the memory prototypes with index j' in the middle; Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the model is weighted with the corresponding memory prototype to obtain the context summary prototype, as shown in the following formula: ； in, This represents a context summary prototype. The set of Top-K nearest neighbor indices used in the context branch is: The memory prototype with index j in the middle; The global image features of each image in the training dataset are aggregated at anchor points in the context prototype library to obtain the corresponding context fusion prototype. The context fusion prototype is then used as the context parameterized prototype and fused with the context summary prototype to obtain the context fusion vector, as shown in the following formula: ； ； in, This represents a context-mixed prototype. This indicates a context-parameterized prototype. Indicates context fusion weights, Represents the context fusion vector; The context parameterized prototype and the context fusion vector are respectively processed through the context adapter to obtain the context adjustment features and the context fusion features, as shown in the following formula: ； ； in, This indicates the function corresponding to the context adapter. and These represent context-adjusted features and context-fused features, respectively. The global image features, context adjustment features, and context fusion features of the image are all L2 normalized to obtain L2-normalized global image features, L2-normalized context adjustment features, and L2-normalized context fusion features. A context loss is then constructed based on these L2-normalized global image features, L2-normalized context adjustment features, and L2-normalized context fusion features, as shown in the following equation: ； in, , , These are, respectively, the global image features after L2 normalization, the context-adjusted features after L2 normalization, and the context-fused features after L2 normalization. Indicates contextual loss. Expressing expectations, Represents the L2 norm; The process of constructing the state loss is as follows: The global image features of the image are compared with the similarity of each memory prototype labeled as a normal state and each memory prototype labeled as an abnormal state in the auxiliary training memory bank. Top-K aggregation is then performed to obtain the index set of the Top-K nearest neighbors. The K memory prototypes and the index set of the Top-K nearest neighbors are: The K memory prototypes in the middle; The index set for Top-K nearest neighbors is The K memory prototypes and the index set of the Top-K nearest neighbors are: For each of the K memory prototypes, calculate the index set of the Top-K nearest neighbors. The weights of each memory prototype and the set of indices of the Top-K nearest neighbors are: The weight of each memory prototype in the equation is as follows: ； ； in, The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index q. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between memory prototypes with index q. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between memory prototypes with index q'; The set of indices representing the Top-K nearest neighbors is: The weight of the memory prototype with index p in the middle. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between memory prototypes with index p. The set of indices representing the global image features of the image and their Top-K nearest neighbors is: The similarity between memory prototypes with index p' in the middle; Set the index set of Top-K nearest neighbors as The weight of each memory prototype in the dataset is weighted with its corresponding memory prototype to obtain the normal state summary prototype. The index set of the Top-K nearest neighbors is then calculated. The weight of each memory prototype in the equation is weighted with the corresponding memory prototype to obtain the abnormal state summary prototype, as shown in the following formula: ； ； in, The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype is indexed as q. The set of Top-K nearest neighbor indices used in the state branch is: The memory prototype with index p in the middle. This represents the prototype summarizing the normal state. This represents a summary prototype of abnormal states; The global image features of each image in the training dataset are aggregated at anchor points in the normal prototype library to obtain the corresponding normal state hybrid prototype. The normal state hybrid prototype is then used as the normal state parameterized prototype and fused with the normal state summary prototype to obtain the normal state fusion vector, as shown in the following formula: ； ； in, This represents a hybrid prototype in its normal state. This represents the parameterized prototype for the normal state. Represents the state fusion weights. Represents the normal state fusion vector; The global image features of each image in the training dataset are aggregated at anchor points in the anomaly prototype library to obtain the corresponding anomaly state hybrid prototype. The anomaly state hybrid prototype is then used as the anomaly state parameterized prototype and fused with the anomaly state summary prototype to obtain the anomaly state fusion vector, as shown in the following formula: ； ； in, This represents a mixed prototype of abnormal states. This represents the parameterized prototype of the abnormal state. Represents the state fusion weights. Represents the abnormal state fusion vector; The normal state fusion vector and the abnormal state fusion vector are respectively processed by the normal state adapter and the abnormal state adapter to obtain the normal state adjustment feature and the abnormal state adjustment feature, as shown in the following formula: ； ； in, and These represent the functions corresponding to the normal state adapter and the abnormal state adapter, respectively. and These represent adjustment characteristics under normal and abnormal states, respectively. The normal state adjustment features and abnormal state adjustment features are respectively subjected to L2 normalization to obtain L2 normal state adjustment features and L2 normalized abnormal state adjustment features. Based on the L2 normalized global image features, the L2 normalized normal state adjustment features, and the L2 normalized abnormal state adjustment features, a state loss is constructed as shown in the following equation: ； in, Indicates state loss. and These represent the normal state adjustment features and the abnormal state adjustment features after L2 normalization, respectively. Represents the training dataset. Represents a subset of normal images. Represents a subset of abnormal images. Indicates the abnormal state label corresponding to the image. This indicates that the image is labeled as being in a normal state. This indicates that the image is labeled as an abnormal state. This represents the total number of samples in the set; Represents cosine similarity. For interval parameters, This indicates taking the maximum value; and These represent the first weight and the second weight, respectively. The total loss function used in the memory-guided training process is a weighted sum of context loss, state loss, classification loss, segmentation loss, and cue orthogonality loss, as shown in the following formula: ； in, Represents the total loss function. Represents classification loss. Indicates the partition loss. This indicates an orthogonal loss. , , and These represent the segmentation weight, cue orthogonality weight, context weight, and state weight, respectively. The classification loss uses the cross-entropy loss function, whose expression is: ； in, Represents the cross-entropy loss function. This represents the classification score corresponding to the image; The segmentation loss employs the Focal loss function and the Dice loss function, and its expression is as follows: ； in, This represents the actual mask corresponding to the image. This represents the anomaly graph corresponding to the image. Represents the Focal loss function. Represents the Dice loss function; The process of constructing the orthogonality loss is as follows: The context mixing prototype, normal state mixing prototype, and abnormal state mixing prototype of each image in the training dataset are input into the corresponding adapter and adjusted to obtain the corresponding context label embedding, normal state label embedding, and abnormal state label embedding. The context label embedding and normal state label embedding are concatenated and input into the text encoder to obtain the normal prompt word embedding. The context label embedding and abnormal state label embedding are concatenated and input into the text encoder to obtain the abnormal prompt word embedding. The normal prompt word embedding and abnormal prompt word embedding of all images in the training dataset are used as text embeddings and all are put into the text embedding set. The similarity between any two text embeddings in the text embedding set is calculated, and the prompt orthogonality loss is calculated as shown in the following formula: in, This represents the total number of text embeddings in the text embedding set. and These represent the o-th and w-th text embeddings in the text embedding set, respectively. This represents the cosine similarity.

5. The zero-shot anomaly detection method based on a visually guided prototype library according to claim 1, characterized in that, The global image features are aggregated into anchor points in the visual guidance prototype library, specifically including: The similarity between the global image features and each visual prototype in one of the prototype libraries in the visual guidance prototype library is calculated using the following formula: ； in, This represents the index of one of the prototype libraries in the visually guided prototype library. Corresponding context prototype library, Corresponding to the normal prototype library, Corresponding exception prototype library; Represents the L2 norm. Represents cosine similarity. Represents global image features. This represents the k-th visual prototype in one of the prototype libraries in the visual guidance prototype library. , This represents the total number of visual prototypes in one of the prototype libraries within the visually guided prototype library. This represents the similarity between the global image features and the k-th visual prototype in one of the prototype libraries in the visual guidance prototype library; The blending weights for each visual prototype in one of the prototype libraries in the visually guided prototype library are calculated using the following formula: ； in, This represents the blending weight of the k-th visual prototype in one of the prototype libraries in the visual guidance prototype library. For temperature parameters, This represents the similarity between the global image features and the i-th visual prototype in one of the prototype libraries in the visual guidance prototype library; The hybrid prototype obtained through anchor point aggregation is defined as follows: ； in, This indicates a hybrid prototype corresponding to one of the prototype libraries in the visually guided prototype library.

6. The zero-shot anomaly detection method based on a visually guided prototype library according to claim 1, characterized in that, The anomaly scoring module includes a first linear layer, an average pooling layer, and a second linear layer. The image patch feature set is first mapped to the joint embedding space through the first linear layer, and then cosine similarity is calculated with the normal prompt word embedding and the abnormal prompt word embedding, respectively. Finally, the anomaly map is calculated, as shown in the following formula: in, Represents a set of image patch features. For the sigmoid function, Represents cosine similarity. This represents the anomaly graph corresponding to the image. This represents the function corresponding to the first linear layer; This indicates the embedding of error message words. This indicates that normal prompt words are embedded; The image score is obtained by taking the maximum value from the anomaly graph, as shown in the following formula: ； in, This indicates taking the maximum value. Indicates the image score; The image patch features in the image patch feature set are concatenated along the channel dimension, and then input into the average pooling layer for global average pooling along the spatial dimension. Finally, they are mapped to a space with the same dimension as the global image features through the second linear layer to obtain multi-scale features. The multi-scale features are added to the global image features to obtain the enhanced global features, as shown in the following formula: ； in, Represents global image features. Representing multi-scale features, This indicates enhanced global features; The enhanced global features are compared with the normal prompt word embeddings and the abnormal prompt word embeddings respectively to calculate the cosine similarity, and then the text score is calculated, as shown in the following formula: ； in, Indicates the text score; The image score and the text score are summed to obtain the classification score, as shown in the following formula: ； in, This indicates the category score.

7. A zero-sample anomaly detection device based on a visually guided prototype library, characterized in that, include: The model building module is configured to build a zero-shot anomaly detection model based on a visually guided prototype library and an auxiliary training memory. The zero-shot anomaly detection model based on the visually guided prototype library includes a cue word embedding generation module and an anomaly scoring module based on a vision-language model. The cue word embedding generation module includes an image encoder, a text encoder, the visually guided prototype library, and its corresponding adapter. The auxiliary training memory is used to train the structures in the zero-shot anomaly detection model based on the visually guided prototype library, excluding the image encoder and the text encoder, to obtain a trained zero-shot anomaly detection model. The detection module is configured to acquire the image to be detected and input it into the trained zero-shot anomaly detection model. The image is input into the prompt word embedding generation module, where it first passes through the image encoder to generate a global image feature set and an image patch feature set. The global image features are then aggregated at anchor points in the visual guidance prototype library, and then input into the corresponding adapters for adjustment to obtain context label embedding, normal state label embedding, and anomaly state label embedding. The context label embedding and normal state label embedding are concatenated and input into the text encoder to obtain the normal prompt word embedding. The context marker embedding and the abnormal state marker embedding are concatenated and input into the text encoder to obtain the abnormal prompt word embedding; the global image features, image block feature set, normal prompt word embedding and abnormal prompt word embedding are input into the abnormal scoring module to obtain the abnormal image and classification score; based on the classification score, the abnormal detection result of the image is determined, and the abnormal detection result is abnormal or normal.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images