Multi-class anomaly detection method and system based on pre-training visual language model in training data scarcity scene

Abstract

The invention provides a multi-class anomaly detection method and system based on a pre-training visual language model in a training data scarcity scene, and relates to the technical field of anomaly detection. A first pre-training visual language model is used for obtaining feature representation of a small number of normal sample images in a text space; using a second pre-training visual language model to obtain global features and block features of a small number of normal sample images, constructing and training an adaptive prompt vector generator, in the training process, updating parameters of the adaptive prompt vector generator to obtain a trained adaptive prompt vector generator, and obtaining a training result of the adaptive prompt vector generator; according to the method, the ability of a pre-training visual language model is effectively combined, a training prompt vector generator is guided, a self-adaptive prompt vector for an anomaly detection task is generated, a one-to-many visual memory warehouse and a one-to-many prompt vector warehouse are constructed, a one-to-many training normal form is adapted, and the detection performance of anomaly detection in a training data scarcity scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and more specifically, to a multi-category anomaly detection method and system based on a pre-trained visual language model in scenarios where training data is scarce. Background Art

[0002] Anomaly detection aims to identify abnormal data that deviates from the main body of normal data. Detecting abnormal data in a timely manner can prevent further impact from abnormal data. Therefore, the anomaly detection task is of great significance and has a wide range of application scenarios, such as diagnosing diseases in the medical field, detecting defects in the industrial field, detecting abnormal transactions in the financial field, and combating cyber attacks in the network field.

[0003] Due to the diversity and scarcity of anomalies, most current mainstream anomaly detection methods assume that a dataset with a large amount of normal data can be used for unsupervised learning to characterize normal data. Unsupervised anomaly detection methods can be divided into two categories: one is reconstruction-based anomaly detection methods, which reconstruct the input image into a normal image by training a generative model, using the reconstruction error as an anomaly score to detect or locate anomalies in the image. The other method is based on feature matching, which generally extracts features through a pre-trained model, trains the model to characterize the distribution or pattern of normal features, and uses the similarity of feature matching as the anomaly score. Both of these methods are based on the assumption that a large amount of normal data can be used to train the model, but in actual application scenarios, it is often difficult to obtain a large amount of normal data. Therefore, researchers have begun to propose anomaly detection methods based on small-sample learning, proposing that the training dataset can be increased through data augmentation or the introduction of auxiliary datasets.

[0004] With the development of visual language models, the generalization of the learning ability of the Contrastive Language-Image Pre-training (CLIP) model on zero-shot / small-shot tasks has been verified. Researchers have also begun to explore the potential of CLIP in anomaly detection tasks and proposed high-performance small-shot anomaly detection methods.

[0005] While both unsupervised and small-sample anomaly detection can achieve good anomaly detection performance, these methods follow a "one-to-one" training paradigm, where a separate model is trained for each data category in the dataset, resulting in high training and storage costs. In contrast, the "one-to-many" training paradigm trains a unified model to detect anomalies across all data categories. However, all current "one-to-many" anomaly detection methods still rely on large amounts of normal data. Summary of the Invention

[0006] In order to solve the problem of how to achieve multi-category anomaly detection in scenarios where training data is scarce, the present invention proposes a multi-category anomaly detection method and system based on a pre-trained visual language model in scenarios where training data is scarce. The knowledge of the pre-trained visual language model is introduced into the anomaly detection task, adapted to the "one-to-many" training paradigm, and improved the detection performance of anomaly detection in scenarios where training data is scarce.

[0007] In order to achieve the above technical effects, the technical solutions of the present invention are as follows:

[0008] In the first aspect, this application proposes a multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce, comprising the following steps:

[0009] S1: Using the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and using the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images;

[0010] S2: constructing an adaptive prompt vector generator and training it. During the training process, updating the parameters of the adaptive prompt vector generator to obtain a trained adaptive prompt vector generator;

[0011] S3: Based on the feature representation of block features and a small number of normal sample images in the text space, a "one-to-many" visual memory warehouse is constructed;

[0012] S4: Input the global features and the feature representations of a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. Using the second pre-trained visual language model, extract the normal prompt vector word embedding and the abnormal prompt vector word embedding for the normal description prompt vector and the abnormal description prompt vector respectively;

[0013] S5: Based on the normal prompt vector word embedding, the abnormal prompt vector word embedding and the feature representation of a small number of normal sample images in the text space, a "one-to-many" prompt vector warehouse is constructed;

[0014] S6: Based on the visual memory warehouse and the prompt vector warehouse, the test image is respectively subjected to image-level anomaly score calculation based on visual features and anomaly score map calculation based on prompt vectors to obtain the final anomaly detection result.

[0015] Preferably, the first pre-trained visual language model is Q-Former, and the second pre-trained visual language model is CLIP, CLIP includes a text encoder and a visual encoder, and the first pre-trained visual language model Q-Former is used to obtain a normal image x i Feature representation in text space, get the image description Z i ;

[0016] The visual encoder of the L layer in the second pre-trained visual language model CLIP extracts global features and block features to form a feature set F i Represents the features output by the i-th layer of the visual encoder, including a global feature and several block features.

[0017] Preferably, the adaptive cue vector generator includes a visual feature fusion network and a normal-abnormal category representation mapping module; wherein the visual feature fusion network includes a normal image x i The global features extracted by the visual encoder and the block features are mapped to the projection network f in the text space p (·):R d →R T×d , R represents the real space, T represents the number of initialized prompt vector token units, d represents the dimension of the space, and also includes a cross-attention network CA(·,·,·) for fusing the global features and block features extracted from the visual encoder with the image description from the first pre-trained visual language model Q-Former, and a multi-layer perceptron f after the cross-attention network a (·):R d →R T×d The normal-abnormal category representation mapping module compresses the image description mapping from the Q-former into an object description suitable for the anomaly detection task: c (·):R M×d →R m×d , M represents the number of token units of the subject description extracted by Q-former, m is the number of token units of the subject description extracted by Q-former after compression through the mapping network, and d represents the dimension of the space.

[0018] By adopting the above technical means, a visual feature fusion network is used to integrate visual features into the basic prompt vectors (universal learnable normal prompt vector and universal learnable abnormal prompt vector), and the normal-abnormal category representation mapping module is used to compress the image description mapping from the Q-former into an object description that is more suitable for anomaly detection tasks.

[0019] Preferably, before training the adaptive hint vector generator, the method further includes:

[0020] Use block features to construct synthetic features;

[0021] Design normal text prompt words and abnormal text prompt words, and use the text encoder of the second pre-trained visual language model CLIP to obtain the normal text feature word embedding of the normal text prompt words and the abnormal text feature word embedding of the abnormal text prompt words respectively;

[0022] The global features and the feature representations of a small number of normal sample images in the text space are input into the adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector of the image. The normal description prompt vector and the abnormal description prompt vector are then input into the text encoder of the second pre-trained visual language model CLIP to obtain the normal prompt vector word embedding and the abnormal prompt vector word embedding respectively.

[0023] Preferably, the process of constructing synthetic features using block features is: select A subset of Used to train the adaptive prompt vector generator, the subset The features between adjacent layers are added together, and additional Gaussian noise is added to construct synthetic features:

[0024]

[0025] in, represents the synthetic features of the k-th layer, represents the normal sample features of the kth layer, and ∈ represents Gaussian noise.

[0026] By adopting the above technical means, the synthetic features have both normal patterns and unnatural abnormal areas. The features between adjacent layers are added and the features between different layers are fused, which not only retains the normal pattern components of the layer but also introduces low-level semantics to construct unnatural phenomena. The addition of Gaussian noise prevents the synthetic features from falling into the high-density area of ​​normal features.

[0027] Preferably, a set of universal learnable normal prompt vectors is represented as: [P1][P2]…[P T ], a set of universal learnable anomaly prompt vectors are represented as: [N1][N2]…[N T ], where [P i ],[N i ] represents the i-th learnable normal prompt unit representation and the learnable abnormal prompt unit representation, which is a d-dimensional vector; all different types of data can generate adaptive prompt vectors based on the universal learnable normal prompt vector or the universal learnable abnormal prompt vector, and obtain the image x from the Q-former through the normal-abnormal category representation mapping module. i , the description map is compressed into an object description suitable for anomaly detection tasks: And spliced ​​at the end of the universal learnable normal prompt vector and the universal learnable abnormal prompt vector; the instance-related conditional prompt obtained by the visual feature fusion network Get The method is formalized as:

[0028]

[0029] Among them, E v (x i ) is the feature extracted by CLIP’s visual encoder, They are the normal representation and abnormal representation of the Q-former image representation after mapping through the multi-layer perceptron; the normal description prompt vector Hint vector with exception description They are formalized as:

[0030]

[0031] Extract normal prompt vector word embedding for normal description prompt vector and abnormal description prompt vector respectively and abnormal prompt vector word embedding

[0032] Preferably, the loss function used to train the adaptive prompt vector generator includes: from the image semantic level alignment level: through contrast learning, maximizing the similarity between the normal prompt vector word embedding and the normal block features and normal text features, and minimizing the similarity between the normal prompt vector word embedding and the synthetic features and abnormal text features, satisfying the expression:

[0033]

[0034] Represents image x i The normal block features at (h, w) are output by the jth layer of the CLIP visual encoder, where H represents the number of blocks in the height direction of the image, W represents the number of blocks in the width direction of the image, and l represents the number of visual encoder output layers selected for training. Minimize the similarity between the abnormal prompt vector word embedding and the normal block features and normal text features, and maximize the similarity between the abnormal prompt vector word embedding and the synthetic features and abnormal text features, satisfying the expression:

[0035]

[0036] Represents image x i The synthetic feature at (h,w) obtained by the CLIP visual encoder at layer j;

[0037] From the image unit level alignment level: for normal description hint vector Hint vector with exception description Through CLIP's text encoder, in addition to obtaining normal prompt vector word embedding and abnormal prompt vector word embedding You will also get the hint vector for each unit token for word embedding encoding and

[0038] Calculate the normal description prompt vector separately Exception description hint vector Similarity matrix M with image block features i ∈R (T+m)×(H×W) , T represents the number of token units of the initialized prompt vector, m represents the number of output token units obtained after the Q-former extracts the image theme representation token unit through the mapping network, and the element M in the matrix i,jk Represents the similarity between the jth unit word embedding of the prompt vector and the kth block feature of the image; i Perform min-max normalization in the image direction, and then select the threshold to retain the elements greater than the threshold and set the elements less than the threshold to 0;

[0039] Normalize in the column direction to obtain the relevance weight of each block for each hint vector unit; finally, obtain the hint vector v by combining the block features based on the weighted summation of the relevance weights. i,j , by contrastive learning to bring v closer i,j With p ij , push away p ij With {v i,k} {k≠j} distance; normal description hint vector The image unit level alignment, the loss function is:

[0040]

[0041] Exception description hint vector The image unit level alignment, the loss function is:

[0042]

[0043] Among them, T'=T+m, which represents the total length of the prompt vector token unit obtained by the adaptive prompt vector generator for the current input image, E x represents the expectation of the loss calculated for the entire training dataset x.

[0044] From the perspective of expert text alignment: through comparative learning, the distance between the normal prompt vector word embedding and the normal text word embedding is shortened, the distance between the abnormal prompt vector word embedding and the abnormal text word embedding is shortened, and the distance between the abnormal prompt vector word embedding and the normal text word embedding is pushed further away. The loss function is:

[0045]

[0046] The loss function for training the adaptive hint vector generator is:

[0047]

[0048] Among them, α and β are weight factors to balance different alignment losses.

[0049] By employing the aforementioned techniques, the loss function used by the adaptive cue vector generator is divided into three parts. At the image semantic-level alignment level, contrastive learning is used to guide the adaptive cue generator to align the overall semantics of the generated cue with the image's block features. For normal cues, the adaptive cue vector generator is guided to generate normal cues that are spatially closer to the image's normal block features and further away from the synthesized features. For abnormal cues, the adaptive cue vector generator is guided to generate abnormal cues that are spatially further away from the image's normal block features and closer to the synthesized features. At the image unit-level alignment level, considering that certain parts of an image in natural language are often only relevant to individual words in a sentence, a proposal is made to adaptively discover the correlation between the generated cue and the image, and to bring the relevant block features and cue vector closer together through contrastive learning. This guides each unit of the cue vector generated by the adaptive generator to focus on a specific region, which is more conducive to locating abnormal regions in anomaly detection. At the expert text alignment level, due to the lack of real abnormal samples, the generation of abnormal prompts is guided by synthetic features, which may cause the adaptive prompt vector generator to learn to generate biased abnormal prompt vectors. It uses domain expert knowledge to describe known abnormalities in natural language, and then generates normal text descriptions and abnormal text descriptions through templates.

[0050] Preferably, for image x i , stores the block features extracted by CLIP's visual encoder, and stores Q-former for x i The obtained image description Z i , formalized as:

[0051] B v ={[Z1;F 1j ],[Z2;F 2j ],…,[Z N ; F Nj ]}

[0052] Among them, B v Represents a "one-to-many" visual memory warehouse, F ij Represents image x i The block features extracted by the jth layer of the visual encoder;

[0053] After obtaining the trained adaptive hint vector generator, for the image x i , first obtain the normal description hint vector through the adaptive hint vector generator Hint vector with exception description Then, store the normal description hint vector at the same time Hint vector with exception description Normal prompt vector word embedding extracted by CLIP text encoder and abnormal prompt vector word embedding And simultaneously store Q-former for x i The obtained image description Z i , formalized as:

[0054]

[0055] Among them, B p Represents a one-to-many hint vector repository.

[0056] Preferably, in step S6, the image-level anomaly score calculation process based on visual features is:

[0057] Extract the test image x through the first pre-trained visual language model Q-former t Image description of Z t ;

[0058] Obtain the "one-to-many" visual memory warehouse B in descending order of similarity v Between the current Z t The first k Z with similarity i The corresponding visual block feature v;

[0059] With the test image x t The extracted block features are used for feature matching, and the image-level anomaly score calculation expression based on visual features is:

[0060]

[0061] in, represents the block features with coordinates (h, w) output from the j1th layer of the visual encoder, represents the block feature with coordinates (h, w) output from the j2th layer of the visual encoder, j1 and j2 represent the corresponding layer numbers of the visual encoder used to construct the visual memory warehouse;

[0062] The image-level anomaly score calculation process based on visual features is:

[0063] Extract the test image x through the first pre-trained visual language model Q-former tImage description of Z t ;

[0064] Get the "one-to-many" hint vector warehouse B p Normal prompt vector word embedding Word embedding with abnormal prompt vector With picture description Z t Similarity, sort by similarity from large to small, get the first k Z i Corresponding normal prompt vector word embedding Word embedding with abnormal prompt vector

[0065] The test image x t Generate normal description hint vectors through adaptive hint vector generator Hint vector with exception description Normal description hint vector Hint vector with exception description Input to CLIP's text encoder to get Get the hint vector word embedding used to calculate the anomaly map:

[0066]

[0067] Then the calculation expression of the abnormal score map based on the prompt vector is:

[0068]

[0069] in, The jth layer of CLIP's visual encoder is the block feature F at the (h, w) position j,hw Considering multiple layers of features and taking the average, we get the average anomaly score: represents the complete anomaly score map of the test image calculated based on the output of the jth layer of the visual encoder; the final anomaly detection result expression is: Among them, A v A represents the complete anomaly score map calculated based on the visual memory warehouse, which is a matrix of size H×W; img Indicates the abnormal classification score of the entire image, which is the probability that the entire image contains an abnormality.

[0070] In a second aspect, the present application also proposes a multi-category anomaly detection system based on a pre-trained visual language model in scenarios where training data is scarce. The system is used to implement the multi-category anomaly detection method based on a pre-trained visual language model in scenarios where training data is scarce, comprising:

[0071] A feature acquisition module, which uses the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and uses the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images;

[0072] The generator constructs a training module for constructing and training an adaptive prompt vector generator, iteratively updating the parameters of the adaptive prompt vector generator, and obtaining a trained adaptive prompt vector generator;

[0073] The visual memory warehouse construction module constructs a "one-to-many" visual memory warehouse based on the block features and the feature representation of a small number of normal sample images in the text space;

[0074] The generation module inputs the feature representation of the global features and a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. The normal prompt vector and the abnormal prompt vector word embedding are extracted from the normal description prompt vector and the abnormal description prompt vector respectively using the second pre-trained visual language model;

[0075] The prompt vector warehouse construction module constructs a "one-to-many" prompt vector warehouse based on the normal prompt vector word embedding, the abnormal prompt vector word embedding, and the feature representation of a small number of normal sample images in the text space;

[0076] The anomaly detection module, based on the visual memory warehouse and the prompt vector warehouse, calculates the image-level anomaly score based on visual features and the anomaly score map based on the prompt vector for the test image to obtain the final anomaly detection result.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] The present invention proposes a multi-category anomaly detection method and system based on a pre-trained visual language model in a scenario where training data is scarce. The method uses a first pre-trained visual language model to obtain feature representations of a small number of normal sample images in a text space, and uses a second pre-trained visual language model to obtain global features and block features of a small number of normal sample images. An adaptive prompt vector generator is constructed and trained. During the training process, the parameters of the adaptive prompt vector generator are updated to obtain a trained adaptive prompt vector generator. The capabilities of the pre-trained visual language model are effectively combined to guide the training of the prompt vector generator, generate prompt vectors adaptive to anomaly detection tasks, construct a "one-to-many" visual memory warehouse and a "one-to-many" prompt vector warehouse, adapt to the "one-to-many" training paradigm, and improve the detection performance of anomaly detection in scenarios where training data is scarce. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1A schematic diagram showing a flow chart of a multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce, as proposed in Example 1 of the present invention;

[0080] Figure 2 A diagram showing a process of processing an image sample based on a pre-trained visual language model and an adaptive hint vector generator proposed in Example 2 of the present invention;

[0081] Figure 3 A structural diagram of a multi-category anomaly detection system based on a pre-trained visual language model in a scenario where training data is scarce, as proposed in Example 3 of the present invention. DETAILED DESCRIPTION

[0082] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0083] In order to better illustrate this embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the actual size;

[0084] It is understandable to those skilled in the art that descriptions of certain well-known contents may be omitted in the drawings.

[0085] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0086] The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent;

[0087] Example 1

[0088] This embodiment proposes a multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce. The flowchart of this method is shown in Figure 1 , including the following steps:

[0089] S1: Using the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and using the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images;

[0090] S2: constructing an adaptive prompt vector generator and training it. During the training process, updating the parameters of the adaptive prompt vector generator to obtain a trained adaptive prompt vector generator;

[0091] S3: Based on the feature representation of block features and a small number of normal sample images in the text space, a "one-to-many" visual memory warehouse is constructed;

[0092] S4: Input the global features and the feature representations of a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. Using the second pre-trained visual language model, extract the normal prompt vector word embedding and the abnormal prompt vector word embedding for the normal description prompt vector and the abnormal description prompt vector respectively;

[0093] S5: Based on the normal prompt vector word embedding, the abnormal prompt vector word embedding and the feature representation of a small number of normal sample images in the text space, a "one-to-many" prompt vector warehouse is constructed;

[0094] S6: Based on the visual memory warehouse and the prompt vector warehouse, the test image is respectively subjected to image-level anomaly score calculation based on visual features and anomaly score map calculation based on prompt vectors to obtain the final anomaly detection result.

[0095] In this embodiment, step S1 as a whole processes a small number of normal samples in the data set and extracts features. Specifically, the first pre-trained visual language model is Q-Former, and the second pre-trained visual language model is CLIP. CLIP includes a text encoder and a visual encoder. The pre-trained Q-former used in this example comes from the open source BLIP-Diffusion project, and the pre-trained CLIP comes from the open source OpenCLIP, with an architecture of VIT-L / 14. Q-Former is the core module in the BLIP-2 framework for connecting pre-trained visual models with large language models (LLMs). Its core goal is to achieve cross-modal semantic alignment through lightweight design. Q-Former consists of two sub-modules that share a self-attention layer and interact with the frozen visual encoder through learnable query vectors (LearnableQueries): Among them, Image Transformer (left): inputs 32 learnable query vectors (each 768 dimensions), models the dependencies between vectors through self-attention, and interacts with the image features output by the frozen visual encoder (such as ViT-L / 14) through cross-attention; Text Transformer (right): acts as an encoder and decoder, controlling the interaction between the query vector and the text through different attention mask strategies (ITC, ITM, ITG). Achieve normal image x through two-stage pre-training i Efficient conversion to text: The first stage of multi-task learning selects visual features that are strongly related to the text, and the second stage converts them into natural language descriptions through the generation ability of LLM, that is, using the first pre-trained visual language model Q-Former to obtain the normal image x i Feature representation in text space, get the image description Z i; The visual encoder of the L layer in the second pre-trained visual language model CLIP extracts global features and block features to form a feature set F i Represents the features of the output of the i-th layer of the visual encoder, including a global feature and several block features. Assume that the visual encoder converts the image x i Divided into 16×16 blocks, then F i The dimension size is 257×768, including 1 global feature and 256 block features. In this embodiment, let image x i The feature set is extracted by a 24-layer visual encoder in CLIP F i Represents the features of the encoder layer i output.

[0096] Example 2

[0097] In this embodiment, the adaptive cue vector generator includes a visual feature fusion network and a normal-abnormal category representation mapping module; wherein the visual feature fusion network includes a normal image x i The global features extracted by the visual encoder and the block features are mapped to the projection network f in the text space p (·):R 768 →R 24×768 , also includes a cross-attention network CA(·,·,·) for fusing the global features and block features extracted from the visual encoder with the image description from the first pre-trained visual language model Q-Former, and a multi-layer perceptron f after the cross-attention network a (·):R 768 →R 24×768 The normal-abnormal category representation mapping module compresses the image description mapping from the Q-former into an object description suitable for the anomaly detection task: c (·):R 16×768 →R 4×768 .

[0098] By adopting the above approach, a visual feature fusion network is used to incorporate visual features into the basic cue vectors (universal learnable normal cue vector and universal learnable abnormal cue vector), and a normal-abnormal category representation mapping module is used to compress the image description mapping from the Q-former into an object description that is more suitable for the anomaly detection task.

[0099] Before training the adaptive hint vector generator, also include:

[0100] A. Use block features to construct synthetic features; the process is: select A subset of For training the adaptive hint vector generator, this embodiment selects the {6, 12, 18, 24}th layer. In order to make the synthetic features have both normal patterns and unnatural abnormal areas, the subset The features between adjacent layers are added, and additional Gaussian noise is added to construct the synthetic features of the input image:

[0101]

[0102] in represents the synthetic features of the k-th layer, Denotes the normal sample features of the kth layer, and ∈ denotes Gaussian noise. By fusing features from different layers, this method preserves the normal pattern components of the layer while introducing low-level semantics to create unnatural phenomena. Adding Gaussian noise prevents the synthesized features from falling into the high-density areas of normal features.

[0103] B. Design normal text prompts and abnormal text prompts. Use the text encoder of the second pre-trained visual language model, CLIP, to obtain normal text feature word embeddings for normal text prompts and abnormal text feature word embeddings for abnormal text prompts. In this step, expert domain knowledge is leveraged to manually design normal text prompts and abnormal text prompts. Specifically, the normal text prompt is designed as "This is a normal / damaged [category name] photo," and the abnormal text prompt is designed as "This is a scratched / damaged [category name] photo."

[0104] C. Input the feature representation of global features and a small number of normal sample images in the text space into the adaptive prompt vector generator to obtain the normal description prompt vector of the image Hint vector with exception description Then the normal description prompt vector Hint vector with exception description Input to the text encoder of the second pre-trained visual language model CLIP to obtain normal prompt vector word embedding and abnormal prompt vector word embedding

[0105] In this embodiment, a set of universal learnable normal prompt vectors is represented as: [P1][P2]…[P T ], a set of universal learnable anomaly prompt vectors are represented as: [N1][N2]…[N T ], where [P i ],[N i] represents the i-th learnable normal prompt unit representation and the learnable abnormal prompt unit representation, which is a d-dimensional vector; all different types of data can generate adaptive prompt vectors based on the universal learnable normal prompt vector or the universal learnable abnormal prompt vector, and obtain the image x from the Q-former through the normal-abnormal category representation mapping module. i , the description map is compressed into an object description suitable for anomaly detection tasks: And spliced ​​at the end of the universal learnable normal prompt vector and the universal learnable abnormal prompt vector; the instance-related conditional prompt obtained by the visual feature fusion network Get is formalized as:

[0106]

[0107] Among them, E v (x i ) is the feature extracted by CLIP’s visual encoder, They are the normal representation and abnormal representation of the Q-former image representation after being mapped by the multi-layer perceptron;

[0108] Normal description hint vector Hint vector with exception description They are formalized as:

[0109]

[0110] Extract normal prompt vector word embedding for normal description prompt vector and abnormal description prompt vector respectively and abnormal prompt vector word embedding

[0111] In this embodiment, three different levels of alignment loss functions are used to guide the training of the adaptive prompt vector generator. At the image semantic level alignment level, contrastive learning is used to guide the adaptive prompt generator to align the overall semantics of the generated prompt with the image's block features. For normal prompts, the normal prompts generated by the adaptive prompt vector generator are guided to be close to the normal block features of the image in space and push away from the synthetic features. For abnormal prompts, the abnormal prompts generated by the adaptive prompt vector generator are guided to be pushed away from the normal block features of the image in space and close to the synthetic features. At the image unit level alignment level, considering that some parts of the image in natural language are often only related to individual words in a sentence, it is proposed to adaptively discover the correlation between the generated prompt and the image, bring the related block features and the prompt vector closer through contrastive learning, and guide each unit of the prompt vector generated by the adaptive generator to focus on a specific area, which is more conducive to locating abnormal areas in abnormality detection. At the expert text alignment level, due to the lack of real abnormal samples, the generation of abnormal prompts is guided by synthetic features, which may cause the adaptive prompt vector generator to learn to generate biased abnormal prompt vectors. It uses domain expert knowledge to describe known abnormalities in natural language, and then generates normal text descriptions and abnormal text descriptions through templates.

[0112] Specifically, the loss functions used to train the adaptive hint vector generator include:

[0113] From the perspective of image semantic alignment: through contrastive learning, maximize the similarity between normal prompt vector word embedding and normal block features and normal text features, and minimize the similarity between normal prompt vector word embedding and synthetic features and abnormal text features, satisfying the expression:

[0114]

[0115] Represents image x i The jth layer of the CLIP visual encoder outputs a normalized block feature at (h, w). H represents the number of blocks in the image's height, W represents the number of blocks in the image's width, and l represents the number of visual encoder output layers selected for training. For example, if an image is divided into 16×16 blocks, then H and W are both 16. (h, w) represents the coordinates of the block feature currently being used to calculate the loss, which can be understood as the block feature with h as the vertical coordinate and w as the horizontal coordinate. l refers to the number of visual encoder output layers selected for training. Assuming a 24-layer visual encoder with {6, 12, 18, 24} layers selected for training, then l = 4.

[0116] Minimize the similarity between the abnormal prompt vector word embedding and the normal block features and normal text features, and maximize the similarity between the abnormal prompt vector word embedding and the synthetic features and abnormal text features, satisfying the expression:

[0117]

[0118] Represents image x i The synthetic feature at (h,w) obtained by the CLIP visual encoder at layer j;

[0119] From the image unit level alignment level: for normal description hint vector Hint vector with exception description Through CLIP's text encoder, in addition to obtaining normal prompt vector word embedding and abnormal prompt vector word embedding You will also get the hint vector for each unit token for word embedding encoding and

[0120] Calculate the normal description prompt vector separately Exception description hint vector Similarity matrix M with image block features i ∈R (T+m)×(H×W) , T represents the number of token units of the initialized prompt vector, m represents the number of output token units obtained after the Q-former extracts the image theme representation token unit through the mapping network, and the element M in the matrix i,jk Represents the similarity between the jth unit word embedding of the prompt vector and the kth block feature of the image; i Perform min-max normalization in the image direction, and then select the threshold to retain elements greater than the threshold and set elements less than the threshold to 0, so that only the relationship between unit-block pairs with high correlation is retained.

[0121] Normalize in the column direction to obtain the relevance weight of each block for each hint vector unit; finally, obtain the hint vector v by combining the block features based on the weighted summation of the relevance weights. i,j , by contrastive learning to bring v closer i,j With p ij , push away p ij With {v i,k} {k≠j} distance; normal description hint vector The image unit level alignment, the loss function is:

[0122]

[0123] Exception description hint vector The image unit level alignment, the loss function is:

[0124]

[0125] Among them, T'=T+m, which represents the total length of the prompt vector token unit obtained by the adaptive prompt vector generator for the current input image, E x represents the expectation of the loss calculated for the entire training dataset x.

[0126] From the perspective of expert text alignment: through comparative learning, the distance between the normal prompt vector word embedding and the normal text word embedding is shortened, the distance between the abnormal prompt vector word embedding and the abnormal text word embedding is shortened, and the distance between the abnormal prompt vector word embedding and the normal text word embedding is pushed further away. The loss function is:

[0127]

[0128] The loss function for training the adaptive hint vector generator is:

[0129]

[0130] Among them, α and β are weight factors for balancing different alignment losses. In this embodiment, they are set to 10. The loss function is calculated and the model parameters of the adaptive prompt generator are updated by the gradient descent method. The model parameters include: trainable neural network layer weights, cross-modal alignment parameters, etc.

[0131] In this embodiment, for the image x i , stores the block features extracted by CLIP's visual encoder, and stores Q-former for x i The obtained image description Z i , formalized as:

[0132] B v ={[Z1;F 1j ],[Z2;F 2j ],…,[Z N ; F Nj ]}

[0133] Among them, B v Represents a "one-to-many" visual memory warehouse, F ij Represents image x i The block features extracted by the jth layer of the visual encoder; we generally design the visual encoder to use the output of the 6th and 18th layers to build a visual memory warehouse, denoted as B v1 , B v2 .

[0134] After obtaining the trained adaptive hint vector generator, such as Figure 2 As shown, for image xi First, after feature extraction by the first pre-trained visual language model and the second pre-trained visual language model, the normal description prompt vector is obtained through the adaptive prompt vector generator. Hint vector with exception description Then, store the normal description hint vector at the same time Hint vector with exception description Normal prompt vector word embedding extracted by CLIP text encoder and abnormal prompt vector word embedding And simultaneously store Q-former for x i The obtained image description Z i , formalized as:

[0135]

[0136] Among them, B p Represents a "one-to-many" prompt vector warehouse,

[0137] In step S6, the image to be tested is input into the prompt word generator to generate a prompt vector corresponding to the current test image. At the same time, the feature set with the highest similarity is obtained from the visual memory warehouse and the prompt vector warehouse. The prompt vector and the retrieved feature set are used to efficiently detect whether there are abnormal areas in the test image by matching the visual block features extracted by the CLIP visual encoder of the test image.

[0138] The image-level anomaly score calculation process based on visual features is:

[0139] Extract the test image x through the first pre-trained visual language model Q-former t Image description of Z t ;

[0140] Obtain the "one-to-many" visual memory warehouse B in descending order of similarity v Between the current Z t The first k Z with similarity i The corresponding visual block feature v;

[0141] With the test image x t The extracted block features are used for feature matching, and the image-level anomaly score calculation expression based on visual features is:

[0142]

[0143] in, represents the block features with coordinates (h, w) output from the j1th layer of the visual encoder, Represents the block features with coordinates (h, w) output from the j2th layer of the visual encoder. j1 and j2 respectively represent the corresponding layer numbers of the visual encoder used to construct the visual memory warehouse. For example, if the output features of the 6th and 18th layers of the visual encoder are used to construct the visual memory warehouse, then j1 is 6 and j2 is 18. The visual memory warehouse does not contain the structure of the middle layer of the visual encoder, but only stores the block features of the training samples output at the j1th and j2th layers of the visual encoder.

[0144] The image-level anomaly score calculation process based on visual features is:

[0145] Extract the test image x through the first pre-trained visual language model Q-former t Image description of Z t ;

[0146] Get the "one-to-many" hint vector warehouse B p Normal prompt vector word embedding Word embedding with abnormal prompt vector With picture description Z t Similarity, sort by similarity from large to small, get the first k Z i Corresponding normal prompt vector word embedding Word embedding with abnormal prompt vector

[0147] The test image x t Generate normal description hint vectors through adaptive hint vector generator Hint vector with exception description Normal description hint vector Hint vector with exception description Input to CLIP's text encoder to get Get the hint vector word embedding used to calculate the anomaly map:

[0148]

[0149] Then the calculation expression of the abnormal score map based on the prompt vector is:

[0150]

[0151] in, The jth layer of CLIP's visual encoder is the block feature F at the (h, w) position j,hw Considering multiple layers of features and taking the average, we get the average anomaly score: represents the complete anomaly score map of the test image calculated based on the output of the jth layer of the visual encoder; the final anomaly detection result expression is: Among them, Av A represents the complete anomaly score map calculated based on the visual memory warehouse, which is a matrix of size H×W; img Indicates the abnormal classification score of the entire image, which is the probability that the entire image contains an abnormality.

[0152] The effectiveness of the method proposed in the embodiment of the present invention is verified below with reference to a specific data set.

[0153] The datasets used in this embodiment are MVTec and VisA datasets. The MVTec training dataset has 3629 images, all of which are defect-free images; the test dataset has 1725 images, which are defective and defect-free images, totaling 15 categories of data; the VisA dataset is a SPot-the-Difference self-supervised pre-training dataset for anomaly detection and segmentation. It contains 12 subsets, corresponding to 12 different objects, as shown in the figure. There are 10,821 images, including 9,621 normal samples and 1,200 abnormal samples. In this example, 4 normal images are taken from different categories of each dataset to train a unified adaptive prompt vector generator. Different abnormal images are detected in all categories of the dataset.

[0154] like Figure 2 As shown in the figure, the training image data is passed through CLIP's visual encoder and Q-former to extract the image description vector and image features respectively, obtain the corresponding generation prompt vector, calculate the similarity with the block feature, use the calculation result of the objective function to calculate the gradient of the model parameters, update the model parameters of the adaptive vector generator, and after multiple gradient backpropagations, obtain the final adaptive prompt vector generator. Then, a "one-to-many" visual memory warehouse and a "one-to-many" prompt vector warehouse are constructed to calculate the anomaly score of the image to be tested.

[0155] Example 3

[0156] like Figure 3 As shown, this embodiment proposes a multi-category anomaly detection system based on a pre-trained visual language model in a scenario where training data is scarce. The system is used to implement the multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce, including:

[0157] A feature acquisition module, which uses the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and uses the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images;

[0158] The generator constructs a training module for constructing and training an adaptive prompt vector generator, iteratively updating the parameters of the adaptive prompt vector generator, and obtaining a trained adaptive prompt vector generator;

[0159] The visual memory warehouse construction module constructs a "one-to-many" visual memory warehouse based on the block features and the feature representation of a small number of normal sample images in the text space;

[0160] The generation module inputs the feature representation of the global features and a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. The normal prompt vector and the abnormal prompt vector word embedding are extracted from the normal description prompt vector and the abnormal description prompt vector respectively using the second pre-trained visual language model;

[0161] The prompt vector warehouse construction module constructs a "one-to-many" prompt vector warehouse based on the normal prompt vector word embedding, the abnormal prompt vector word embedding, and the feature representation of a small number of normal sample images in the text space;

[0162] The anomaly detection module, based on the visual memory warehouse and the prompt vector warehouse, calculates the image-level anomaly score based on visual features and the anomaly score map based on the prompt vector for the test image to obtain the final anomaly detection result.

[0163] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-category anomaly detection method based on a pre-trained visual language model in scenarios where training data is scarce, characterized by: The following steps are involved: S1: Using the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and using the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images; S2: constructing an adaptive prompt vector generator and training it. During the training process, updating the parameters of the adaptive prompt vector generator to obtain a trained adaptive prompt vector generator; S3: Based on the feature representation of block features and a small number of normal sample images in the text space, a "one-to-many" visual memory warehouse is constructed; S4: Input the global features and the feature representations of a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. Using the second pre-trained visual language model, extract the normal prompt vector word embedding and the abnormal prompt vector word embedding for the normal description prompt vector and the abnormal description prompt vector respectively; S5: Based on the normal prompt vector word embedding, the abnormal prompt vector word embedding and the feature representation of a small number of normal sample images in the text space, a "one-to-many" prompt vector warehouse is constructed; S6: Based on the visual memory warehouse and the prompt vector warehouse, the test image is respectively subjected to image-level anomaly score calculation based on visual features and anomaly score map calculation based on prompt vectors to obtain the final anomaly detection result.

2. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 1 is characterized in that: The first pre-trained visual language model is Q-Former, and the second pre-trained visual language model is CLIP. CLIP includes a text encoder and a visual encoder. The first pre-trained visual language model Q-Former is used to obtain a normal image x i Feature representation in text space, get the image description Z i ; The visual encoder of the L layer in the second pre-trained visual language model CLIP extracts global features and block features to form a feature set F i Represents the features output by the i-th layer of the visual encoder, including a global feature and several block features.

3. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 2, characterized in that: The adaptive cue vector generator includes a visual feature fusion network and a normal-abnormal category representation mapping module; wherein the visual feature fusion network includes a normal image x i The global features extracted by the visual encoder and the block features are mapped to the projection network f in the text space p (·):R d →R T×d , R represents the real space, T represents the number of initialized prompt vector token units, and d represents the dimension of the space; it also includes a cross-attention network CA(·,·,·) for fusing the global features and block features extracted from the visual encoder with the image description from the first pre-trained visual language model Q-Former, and a multi-layer perceptron f after the cross-attention network a (·):R d →R T×d The normal-abnormal category representation mapping module compresses the image description mapping from the Q-former into an object description suitable for the anomaly detection task: c (·):R M×d →R m×d , M represents the number of token units of the subject description extracted by Q-former, m is the number of token units of the subject description extracted by Q-former after compression through the mapping network, and d represents the dimension of the space.

4. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 3, characterized in that: Before training the adaptive hint vector generator, also include: Use block features to construct synthetic features; Design normal text prompt words and abnormal text prompt words, and use the text encoder of the second pre-trained visual language model CLIP to obtain the normal text feature word embedding of the normal text prompt words and the abnormal text feature word embedding of the abnormal text prompt words respectively; The global features and the feature representations of a small number of normal sample images in the text space are input into the adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector of the image. The normal description prompt vector and the abnormal description prompt vector are then input into the text encoder of the second pre-trained visual language model CLIP to obtain the normal prompt vector word embedding and the abnormal prompt vector word embedding respectively.

5. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 4, characterized in that: The process of constructing synthetic features using block features is as follows: A subset of Used to train the adaptive prompt vector generator, the subset The features between adjacent layers are added together, and additional Gaussian noise is added to construct synthetic features: in, represents the synthetic features of the k-th layer, represents the normal sample features of the kth layer, and ∈ represents Gaussian noise.

6. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 5, characterized in that: Assume that a set of universal learnable normal prompt vectors is represented as: [P1][P2]…[P T ], a set of universal learnable anomaly prompt vectors are represented as: [N1][N2]…[N T ], where [P i ],[N i ] represents the i-th learnable normal prompt unit representation and the learnable abnormal prompt unit representation, which is a d-dimensional vector; all different types of data can generate adaptive prompt vectors based on the universal learnable normal prompt vector or the universal learnable abnormal prompt vector, and obtain the image x from the Q-former through the normal-abnormal category representation mapping module. i , the description map is compressed into an object description suitable for anomaly detection tasks: And spliced ​​at the end of the universal learnable normal prompt vector and the universal learnable abnormal prompt vector; the instance-related conditional prompt obtained by the visual feature fusion network Get The method is formalized as: Among them, E v (x i ) is the feature extracted by CLIP’s visual encoder, They are the normal representation and abnormal representation of the Q-former image representation after mapping through the multi-layer perceptron; the normal description prompt vector Hint vector with exception description They are formalized as: Extract normal prompt vector word embedding for normal description prompt vector and abnormal description prompt vector respectively and abnormal prompt vector word embedding 7. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 2, characterized in that: The loss function used to train the adaptive prompt vector generator includes: from the image semantic level alignment level: through contrast learning, maximize the similarity between the normal prompt vector word embedding and the normal block features and normal text features, and minimize the similarity between the normal prompt vector word embedding and the synthetic features and abnormal text features, satisfying the expression: Represents image x i The normal block features at (h, w) are output by the jth layer of the CLIP visual encoder, where H represents the number of blocks in the height direction of the image, W represents the number of blocks in the width direction of the image, and l represents the number of visual encoder output layers selected for training. Minimize the similarity between the abnormal prompt vector word embedding and the normal block features and normal text features, and maximize the similarity between the abnormal prompt vector word embedding and the synthetic features and abnormal text features, satisfying the expression: Represents image x i The synthetic feature at (h,w) obtained by the CLIP visual encoder at layer j; From the image unit level alignment level: for normal description hint vector Hint vector with exception description Through CLIP's text encoder, in addition to obtaining normal prompt vector word embedding and abnormal prompt vector word embedding You will also get the hint vector for each unit token for word embedding encoding and Calculate the normal description prompt vector separately Exception description hint vector Similarity matrix M with image block features i ∈R (T+m)×(H×W) , T represents the number of token units of the initialized prompt vector, m represents the number of output token units obtained after the Q-former extracts the image theme representation token unit through the mapping network, and the element M in the matrix i,jk Represents the similarity between the jth unit word embedding of the prompt vector and the kth block feature of the image; i Perform min-max normalization in the image direction, and then select the threshold to retain the elements greater than the threshold and set the elements less than the threshold to 0; Normalize in the column direction to obtain the relevance weight of each block for each hint vector unit; finally, obtain the hint vector v by combining the block features based on the weighted summation of the relevance weights. i,j , by contrastive learning to bring v closer i,j With p ij , push away p ij With {v i,k } {k≠j} distance; normal description hint vector The image unit level alignment, the loss function is: Exception description hint vector The image unit level alignment, the loss function is: Among them, T'=T+m, which represents the total length of the prompt vector token unit obtained by the adaptive prompt vector generator for the current input image, E x represents the expectation of the loss calculated for the entire training dataset x; From the perspective of expert text alignment: through comparative learning, the distance between the normal prompt vector word embedding and the normal text word embedding is shortened, the distance between the abnormal prompt vector word embedding and the abnormal text word embedding is shortened, and the distance between the abnormal prompt vector word embedding and the normal text word embedding is pushed further away. The loss function is: The loss function for training the adaptive hint vector generator is: Among them, α and β are weight factors that balance different alignment losses.

8. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 7, characterized in that: For image x i , stores the block features extracted by CLIP's visual encoder, and stores Q-former for x i The obtained image description Z i , formalized as: B v ={[Z1;F 1j ],[Z2;F 2j ],…,[Z N ;F Nj ]} Among them, B v represents a "one-to-many" visual memory warehouse, F ij Represents image x i The block features extracted by the jth layer of the visual encoder; after obtaining the trained adaptive prompt vector generator, for the image x i , first obtain the normal description hint vector through the adaptive hint vector generator Hint vector with exception description Then, store the normal description hint vector at the same time Hint vector with exception description Normal prompt vector word embedding extracted by CLIP text encoder and abnormal prompt vector word embedding And simultaneously store Q-former for x i The obtained image description Z i , formalized as: Among them, B p Represents a one-to-many hint vector repository.

9. The multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce according to claim 8, characterized in that: In step S6, the image-level anomaly score calculation process based on visual features is as follows: the test image x is extracted by the first pre-trained visual language model Q-former t Image description of Z t ; Obtain the "one-to-many" visual memory warehouse B in descending order of similarity v Between the current Z t The first k Z with similarity i The corresponding visual block feature v; With the test image x t The extracted block features are used for feature matching, and the image-level anomaly score calculation expression based on visual features is: in, represents the block features with coordinates (h, w) output from the j1th layer of the visual encoder, represents the block feature with coordinates (h, w) output from the j2th layer of the visual encoder, j1 and j2 represent the corresponding layer numbers of the visual encoder used to construct the visual memory warehouse; The image-level anomaly score calculation process based on visual features is: Extract the test image x through the first pre-trained visual language model Q-former t Image description of Z t ; Get the "one-to-many" prompt vector warehouse B p Normal prompt vector word embedding Word embedding with abnormal prompt vector With picture description Z t Similarity, sort by similarity from large to small, get the first k Z i Corresponding normal prompt vector word embedding Word embedding with abnormal prompt vector The test image x t Generate normal description hint vectors through adaptive hint vector generator Hint vector with exception description Normal description hint vector Hint vector with exception description Input to CLIP's text encoder to get Get the hint vector word embedding used to calculate the anomaly map: Then the calculation expression of the abnormal score map based on the prompt vector is: in, The jth layer of CLIP's visual encoder is the block feature F at the (h, w) position j,hw Anomaly score; consider multiple layers of features and take the average to get the average anomaly score represents the complete anomaly score map of the image to be tested calculated based on the output of the j-th layer of the visual encoder; The final anomaly detection result expression is: Among them, A v A represents the complete anomaly score map calculated based on the visual memory warehouse, which is a matrix of size H×W; img Indicates the abnormal classification score of the entire image, which is the probability that the entire image contains an abnormality.

10. A multi-category anomaly detection system based on a pre-trained visual language model in scenarios where training data is scarce, characterized by: The system is used to implement the multi-category anomaly detection method based on a pre-trained visual language model in a scenario where training data is scarce as described in any one of claims 1 to 9, comprising: A feature acquisition module, which uses the first pre-trained visual language model to obtain feature representations of a small number of normal sample images in the text space, and uses the second pre-trained visual language model to obtain global features and block features of a small number of normal sample images; The generator constructs a training module for constructing and training an adaptive prompt vector generator, iteratively updating the parameters of the adaptive prompt vector generator, and obtaining a trained adaptive prompt vector generator; The visual memory warehouse construction module constructs a "one-to-many" visual memory warehouse based on the block features and the feature representation of a small number of normal sample images in the text space; The generation module inputs the feature representation of the global features and a small number of normal sample images in the text space into the trained adaptive prompt vector generator to obtain the normal description prompt vector and the abnormal description prompt vector. The normal prompt vector and the abnormal prompt vector word embedding are extracted from the normal description prompt vector and the abnormal description prompt vector respectively using the second pre-trained visual language model; The prompt vector warehouse construction module constructs a "one-to-many" prompt vector warehouse based on the normal prompt vector word embedding, the abnormal prompt vector word embedding, and the feature representation of a small number of normal sample images in the text space; The anomaly detection module, based on the visual memory warehouse and the prompt vector warehouse, calculates the image-level anomaly score based on visual features and the anomaly score map based on the prompt vector for the test image to obtain the final anomaly detection result.

Images