Single-branch prompt zero sample anomaly detection method and system based on prototype guidance

Abstract

The invention provides a prototype guidance-based single-branch prompt zero sample anomaly detection method and system. The method comprises the steps of extracting image features by using a visual encoder; generating a single-branch text prompt matched with the image features by using a prototype-guided self-adaptive prompt generation strategy; extracting text features by using a text encoder; predicting an image-level abnormal score and a pixel-level abnormal score according to the similarity between the image features and the text features; and according to the image-level anomaly score and the pixel-level anomaly, judging whether the input image contains anomaly and the specific position of the anomaly. According to the method, the pre-trained visual language model is introduced into a zero sample anomaly detection task, and the accuracy and efficiency of anomaly detection are remarkably improved through a proposed prototype-guided adaptive prompt module and a single-branch text architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and in particular to a prototype-guided single-branch prompt zero-sample anomaly detection method and system. Background Art

[0002] With the increasing demand for anomaly detection technology in scenarios such as industrial inspection, medical imaging, and autonomous driving, most existing anomaly detection technologies focus on supervised learning or multi-sample learning models. However, supervised anomaly detection requires a large amount of labeled data for training, which is not realistic in many practical applications. Therefore, zero-sample anomaly detection has gradually become a research hotspot.

[0003] Existing zero-shot anomaly detection methods usually use a complex dual-branch structure and combine it with text prompt templates to improve detection accuracy. However, this method has problems such as strong dependence on prior knowledge and high computational overhead, which seriously limits its application in dynamic and diverse environments. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a prototype-guided single-branch prompt zero-sample anomaly detection method and system, which effectively solves the problems of high computational overhead, strong reliance on prior knowledge and insufficient cross-modal consistency in the prior art.

[0005] The technical solution of the present invention is: a prototype-guided single-branch prompt zero-sample anomaly detection method, comprising the following steps:

[0006] S1), receiving at least one input training image;

[0007] S2), using the pre-trained visual language model to extract features from the image to obtain image features;

[0008] S3), based on image features, using a prototype-guided adaptive prompt generation strategy to generate a single-branch text prompt that matches the image features;

[0009] S4), converting the text prompt input into the pre-trained visual language model into text features;

[0010] S5), mapping image features and text features to a shared multimodal feature space, calculating the similarity between features, and using a single-branch text architecture to calculate image-level anomaly scores and pixel-level anomalies;

[0011] S6) According to the image-level anomaly score and the pixel-level anomaly, determine whether the input image contains an anomaly and the specific location of the anomaly.

[0012] Preferably, in step S2), the visual language model includes a visual encoder and a text encoder; and the image is feature extracted by the pre-trained visual encoder; and the text prompt is converted into text features by the pre-trained text encoder;

[0013] The visual language model is jointly pre-trained using images and text matching the image content, and the visual language model is optimized to reduce the distance between image features and corresponding text features, and increase the distance between them and irrelevant text features.

[0014] Preferably, in step S2), the visual encoder simultaneously outputs a category password and a block password, the category password summarizes image information for image-level anomaly detection, and the block password represents local information of the image for pixel-level anomaly detection.

[0015] Preferably, in step S3), a prototype-guided adaptive prompt generation strategy is used to generate a single-branch text prompt that matches the image feature, which specifically includes the following steps:

[0016] S31), randomly initialize a number of learnable prototype vectors;

[0017] S32), during the training process, a prototype password is generated through the prototype vector and the corresponding image block, and multiple prototype passwords are spliced ​​into a prototype prompt;

[0018] S33), combining the prototype prompt with the learnable bias matrix to generate a text prompt.

[0019] Preferably, in step S32), the calculation expression of the prototype password p corresponding to each image block password is calculated by the prototype vector as follows:

[0020]

[0021] Where σ(·) is the function used for scale transformation; v m represents the learnable prototype vector; h p (·) is the image feature f of the dth layer d The function that projects to the same space as the text dimension; θ m and is a learnable parameter; To balance the values ​​and prevent gradient explosion, C T represents the dimension of the prototype vector, Represents the image feature f d The block password.

[0022] Preferably, in step S33), the expression for generating the text prompt W is:

[0023] W=α·P+β·Q

[0024] Where P is a matrix composed of prototype passwords p, and Q is a shared bias vector [q 1 ;q 2 ;…;q M ]; α and β are weights used to balance the contributions of P and Q.

[0025] Preferably, in step S5), mapping the image features and the text features to a shared multimodal feature space specifically includes the following steps:

[0026] S511), using a single layer of learnable linear projection to adjust the dimensions of the image category password and the block password to match the dimensions of the text features;

[0027] S512), using a single layer of learnable linear projection to adjust the representation of text features without changing their dimension.

[0028] Preferably, in step S5), a single-branch text architecture is used to calculate the image-level anomaly score and the pixel-level anomaly, which specifically includes the following steps:

[0029] S521), calculating the inner product of the category password and the average text feature to obtain an image-level anomaly score;

[0030] S522), calculating the inner product of the block password and the text feature to generate a pixel-level anomaly score;

[0031] S523) Determine whether there is an abnormality in the image based on the size of the inner product, and determine it as an abnormality if it exceeds a predetermined similarity threshold.

[0032] Preferably, the present invention also provides a prototype-guided single-branch prompt zero-sample anomaly detection system, comprising:

[0033] The visual encoding module is used to extract features from the input image, divide the image into multiple image blocks, and extract features from these image blocks through multiple attention layers to obtain the image category password and block password;

[0034] An adaptive prompt module, used to generate a corresponding text prompt according to the extracted image block password;

[0035] A text encoding module that receives text prompts generated by the prototype-guided adaptive prompt module, converts them into text embeddings corresponding to image features, and maps them to a visual-text shared feature space;

[0036] The anomaly detection module is used to calculate the similarity between image features and text prompt features and output anomaly detection results.

[0037] Preferably, the adaptive prompt module includes:

[0038] A prototype generation unit, used for generating a prototype vector based on image features, and constructing a prototype matrix through the prototype vector to represent potential abnormal patterns in the image;

[0039] A deviation generation unit, used to generate a global deviation matrix, which is combined with the prototype matrix to generate a text prompt, wherein the deviation matrix is ​​used to adjust the generation of the prototype vector to make it more adaptable and robust;

[0040] The weighted synthesis unit is used to weight the prototype matrix and the deviation matrix according to a predefined weight coefficient to form a final text prompt vector.

[0041] The beneficial effects of the present invention are:

[0042] 1. This paper introduces a pre-trained visual language model into the zero-shot anomaly detection task, and significantly improves the accuracy and efficiency of anomaly detection through the proposed prototype-guided adaptive prompt module and single-branch text architecture;

[0043] 2. The present invention adopts a single-branch architecture, which greatly simplifies the network structure and reduces the consumption of computing resources; the single-branch architecture makes the entire detection process more efficient and can be executed quickly under limited computing resources, and is particularly suitable for real-time application scenarios;

[0044] 3. The present invention generates text prompts consistent with image features through a prototype-guided adaptive prompt generation strategy, avoiding dependence on specific prior knowledge, so that the present invention can perform anomaly detection without relying on any prior labels or templates, has stronger universality, and can be widely used in different fields and dynamically changing environments;

[0045] 4. The prototype-guided adaptive prompt module of the present invention has strong adaptability and flexibility, and can dynamically generate text prompts according to the characteristics of the input image without the need for fixed templates or label support; and adjusts the length and content of the generated text prompts according to actual needs, making the model more flexible and accurate in handling different types of anomalies; this adaptive capability enables the present invention to operate stably in various dynamic environments and handle a variety of complex anomaly detection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the process of the present invention;

[0047] Figure 2 Schematic diagram of the adaptive prompt module guided by the prototype of the present invention. DETAILED DESCRIPTION

[0048] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings:

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a single-branch prompt zero-sample anomaly detection method based on prototype guidance, including the following steps:

[0051] S1), input training image X i and its corresponding exception mask S i , where i∈{0,1,…,I N}, I N Represents the number of input images; and for image X i Perform the following preprocessing:

[0052] The mean value on the RGB channel is [0.485, 0.456, 0.406] and the variance is

[0053] The input image is normalized to [0.229, 0.224, 0.225] and the input size is adjusted to 512 × 512 to obtain the preprocessed image X i ;

[0054] S2), use the visual encoder g(·) to extract the preprocessed image X i The image features, including the category password f c and block password Right now:

[0055]

[0056] Where n∈{0,1,…,N}, N represents the number of image blocks;

[0057] In this embodiment, in the process of extracting image features by the visual encoder, the preprocessed image X is first i The network is divided into N fixed-size blocks and projected into the feature space through a convolutional layer to generate N block passwords. At the same time, a learnable category password is concatenated with these block passwords. After calculations by multiple attention modules based on the Transformer architecture and a linear projection layer, the output category password f c and block password

[0058] S3), generate a single-branch text prompt W through a prototype-guided adaptive prompt generation strategy; Learn the corresponding prompts in p m , where m∈{0,1,…,M}; specifically includes the following steps:

[0059] S31), D blocks of image features extracted by the visual encoder Used to generate text prompts;

[0060] S32), randomly initialize M learnable prototype vectors v m , whose dimension C V Same dimensions as block password;

[0061] In this embodiment, step S31) and step S32) can be interchanged; that is, first randomly initialize M prototype vectors v m Then input the block password to the adaptive prompt module of the prototype boot.

[0062] S33), according to M prototype vectors v m and the corresponding layer d block password Calculate the prototype password p m ,Right now:

[0063]

[0064] Among them, σ(·) is the softmax function used for scale transformation, h p (·) is the image feature f of the dth layer d A linear projection layer that projects to the same space as the text dimension, with parameter θ m and Belongs to the linear projection layer, the dividend Used to balance values ​​and prevent gradient explosion, C T Represents the dimension of the prototype vector;

[0065] Each block password Calculate M prototype passwords, concatenate these prototype passwords into a prototype matrix P, and thus generate D prototype matrices P for D block passwords;

[0066] S34), randomly initialize D shared bias matrices Q, Q is composed of the bias vector [q 1 ;q 2 ;…;q M ], the dimension of q is the same as p; calculate D text prompts W:

[0067] W = α·P + β·Q;

[0068] Among them, α and β are weights used to balance the contributions of P and Q.

[0069] S4), use the text encoder t(·) to extract the text features of the text prompt W The calculation formula is:

[0070]

[0071] Where i∈{0,1,…,D}, D represents the number of image blocks, and function h t (·) is the d-th layer text feature f i t Linear projection layer to multimodal feature space, θ d are the parameters belonging to the linear projection layer;

[0072] S5), calculate the category password f of the image feature c and block password and text features f i t Similarity, predicting image-level anomaly scores and pixel-level anomaly scores The calculation formula is:

[0073]

[0074] Among them, h c 、h v is a linear projection layer that projects the password into the multimodal feature space, μ d , d represents the learnable weight, c(·) represents the reduction function;

[0075] In this embodiment, due to The size of the input image X i The sizes do not match, and the dot product yields a vector, so it needs to be resized and bilinearly interpolated to expand to the image X. i size.

[0076] S6), calculate the image level loss L i and pixel-level loss L p ;

[0077]

[0078] Among them, δ represents the true outlier value, δ∈{0,1}, 0 represents normal, 1 represents abnormal; S represents the abnormal mask.

[0079] Determine whether the current training round e meets the set value E. If so, save the model and stop training. Otherwise, update the model parameters through gradient propagation and continue to repeat steps S3)-S6).

[0080] In practical applications, the performance of this embodiment on the data set is better than other methods, showing a significant performance improvement. This embodiment and three existing image anomaly detection methods, the method SAA based on the segmentation everything model, the method WinCLIP based on the sliding window, and the ACLIP based on the dual text branch, are used to calculate the image-level and pixel-level anomaly score maps for all test images in the MVTec, VisA and BTAD test data sets, respectively, and the area enclosed by the characteristic curves of various methods and the horizontal axis, i.e., AUC, is calculated. The results are shown in Table 1.

[0081] Table 1 Comparison of pixel-level AUC evaluation results (%) of four methods

[0082]

[0083] The pixel-level AUC is the area AUC enclosed by the pixel-level test characteristic curve and the horizontal axis, and is used to measure the effect of the anomaly positioning method. Its value is between 0% and 100%. The larger the value, the better the positioning effect.

[0084] As can be seen from Table 1, the average value of the pixel-level AUC index of this embodiment on the test set is higher than that of the other three methods, which proves that this embodiment has a better zero-sample anomaly detection effect.

[0085] Example 2

[0086] This embodiment provides a prototype-guided single-branch prompt adaptive zero-sample anomaly detection system, which is characterized by comprising:

[0087] The visual encoding module is used to transform the input image X i Extract image features, divide the image into multiple image blocks, and extract features from the image blocks through multiple attention layers to obtain the image category password and block password;

[0088] The prototype-guided adaptive prompt module is used to generate corresponding text prompts based on the extracted image block password; the details are as follows:

[0089] A) Extract image features based on the visual encoder and obtain D block passwords for generating text prompts

[0090] B) Randomly initialize M learnable prototype vectors v m , whose dimension C V Same dimensions as block password;

[0091] C), according to M prototype vectors v m and the corresponding layer d block password Calculate the prototype password p m , the calculation formula is:

[0092]

[0093] Among them, σ(·) is the softmax function used for scale transformation, h p (·) is the image feature f of the dth layer d A linear projection layer that projects to the same space as the text dimension, with parameter θ m and Belongs to the linear projection layer, the dividend Used to balance values ​​and prevent gradient explosion, C T Represents the dimension of the prototype vector;

[0094] Each block password Calculate M prototype passwords, concatenate these prototype passwords into a prototype matrix P, and thus generate D prototype matrices P for D block passwords;

[0095] D), randomly initialize D shared bias matrices Q, Q is composed of the bias vector [q 1 ;q 2 ;…;q M ], the dimension of q is the same as that of p;

[0096] E) Calculate D text prompts W:

[0097] W = α·P + β·Q;

[0098] Among them, α and β are weights used to balance the contributions of P and Q.

[0099] The text encoding module is used to receive the text prompts generated by the prototype-guided adaptive prompt module, convert them into text embeddings corresponding to image features, and map them to the visual-text shared feature space.

[0100] The anomaly detection module is used to calculate the similarity between image features and text prompt features, and output anomaly detection results based on the similarity prediction.

[0101] In this embodiment, the anomaly detection result includes the predicted image-level anomaly score and pixel-level anomaly scores Right now:

[0102]

[0103] Among them, h c ,h v is a linear projection layer that projects the password into the multimodal feature space, μ d ,λ d represents the learnable weights, and c(·) represents the reduction function.

[0104] In this embodiment, the prototype-guided adaptive prompt module includes:

[0105] A prototype generation unit, used for generating a prototype vector based on image features, and constructing a prototype matrix through the prototype vector to represent potential abnormal patterns in the image;

[0106] Deviation generation unit, used to generate a global deviation matrix, which is combined with the prototype matrix to generate text prompts. The deviation matrix is ​​used to adjust the generation of prototype vectors to make them more adaptable and robust;

[0107] The weighted synthesis unit is used to weight the prototype matrix and the deviation matrix according to a predefined weight coefficient to form a final text prompt vector.

[0108] The above embodiments and descriptions are only for illustrating the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, all of which fall within the scope of the present invention to be protected.

Claims

1. A prototype-guided single-branch prompt zero-sample anomaly detection method, characterized in that: The following steps are involved: S1), receiving at least one input training image; S2), using the pre-trained visual language model to extract features from the image to obtain image features; S3), based on image features, using a prototype-guided adaptive prompt generation strategy to generate a single-branch text prompt that matches the image features; S4), converting the text prompt input into the pre-trained visual language model into text features; S5), mapping image features and text features to a shared multimodal feature space, calculating the similarity between features, and using a single-branch text architecture to calculate image-level anomaly scores and pixel-level anomalies; S6) According to the image-level anomaly score and the pixel-level anomaly, determine whether the input image contains an anomaly and the specific location of the anomaly.

2. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 1, characterized in that: In step S2), the visual language model includes a visual encoder and a text encoder; the image features are extracted from the image by the pre-trained visual encoder; the text prompts are converted into text features by the pre-trained text encoder; the visual language model is jointly pre-trained using the image features and the text features matching the image content, and the visual language model is optimized to reduce the distance between the image features and the corresponding text features, and increase the distance between the image features and the irrelevant text features.

3. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 2, characterized in that: In step S2), the visual encoder simultaneously outputs a category password and a block password, wherein the category password summarizes image information for image-level anomaly detection, and the block password represents local information of the image for pixel-level anomaly detection.

4. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 1, characterized in that: In step S3), a prototype-guided adaptive prompt generation strategy is used to generate a single-branch text prompt that matches the image features, which specifically includes the following steps: S31), randomly initialize a number of learnable prototype vectors; S32), during the training process, a prototype password is generated through the prototype vector and the corresponding image block, and multiple prototype passwords are spliced ​​into a prototype prompt; S33), combining the prototype prompt with the learnable bias matrix to generate text prompts.

5. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 4, characterized in that: In step S32), the calculation expression of the prototype password p corresponding to each image block password is calculated by the prototype vector: Where σ(·) is the function used for scale transformation; v m represents the learnable prototype vector; h p (·) is the image feature f of the dth layer d The function that projects to the same space as the text dimension; θ m and is a learnable parameter; To balance the values ​​and prevent gradient explosion, C T represents the dimension of the prototype vector, Represents the image feature f d The block password.

6. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 4, characterized in that: In step S33), the expression for generating the text prompt W is: W=α·P+β·Q Where P is a matrix composed of prototype passwords p, and Q is a shared bias vector [q1; q2; ...; q M ]; α and β are weights used to balance the contributions of P and Q.

7. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 1, characterized in that: In step S5), the image features and text features are mapped to a shared multimodal feature space, which specifically includes the following steps: S511), using a single layer of learnable linear projection to adjust the dimensions of the image category password and the block password to match the dimensions of the text features; S512), using a single layer of learnable linear projection to adjust the representation of text features without changing their dimension.

8. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 7, characterized in that: In step S5), a single-branch text architecture is used to calculate image-level anomaly scores and pixel-level anomalies, which specifically includes the following steps: S521), calculate the inner product of the category password and the average text feature to obtain the image-level anomaly score Right now: In the formula, h c is a linear projection layer that projects the password into the multimodal feature space; f c is the category password; μ d is the learnable weight; is the block password; D is the number of block passwords; S522), calculate the inner product of the block password and the text feature, and generate a pixel-level anomaly score Right now: In the formula, represents pixel-level anomaly; c(·) represents restoration function; λ d represents the learnable weight; h v is a linear projection layer that projects the password into the multimodal feature space; is the block password; D is the number of block passwords; S523) Determine whether there is an abnormality in the image based on the size of the inner product, and determine it as an abnormality if it exceeds a predetermined similarity threshold.

9. The prototype-guided single-branch prompt zero-sample anomaly detection method according to claim 8, characterized in that: In step S6), according to the image-level anomaly score and pixel-level anomaly scores Calculate the image-level loss L i and pixel-level loss L p ,Right now: Where δ represents the true outlier value, δ∈{0,1}, 0 represents normal, 1 represents abnormal; S represents the abnormal mask.

10. A prototype-guided single-branch prompt zero-sample anomaly detection system, characterized in that: The system uses the method described in any one of claims 1 to 9 to perform predictive detection, and the system comprises: The visual encoding module is used to extract image features from the input image, divide the image into multiple image blocks, and extract features from these image blocks through multiple attention layers to obtain the image category password and block password; An adaptive prompt module, used to generate a corresponding text prompt according to the extracted image block password; A text encoding module that receives text prompts generated by the prototype-guided adaptive prompt module, converts them into text embeddings corresponding to image features, and maps them to a visual-text shared feature space; The anomaly detection module is used to calculate the similarity between image features and text prompt features and output anomaly detection results.

Images