A Prototype-Guided Single-Branch Hint-Based Zero-Shot Anomaly Detection Method and System

Abstract

This invention provides a prototype-guided single-branch prompt zero-shot anomaly detection method and system, comprising: extracting image features using a visual encoder; generating single-branch text prompts matching the image features using a prototype-guided adaptive prompt generation strategy; extracting text features using a text encoder; predicting image-level and pixel-level anomaly scores based on the similarity between image and text features; and determining whether the input image contains an anomaly and its specific location based on the image-level and pixel-level anomaly scores. This invention introduces a pre-trained visual language model into the zero-shot anomaly detection task, and significantly improves the accuracy and efficiency of anomaly detection through the proposed prototype-guided adaptive prompt module and single-branch text architecture.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, and in particular to a prototype-guided single-branch prompting zero-sample anomaly detection method and system. Background Technology

[0002] With the increasing demand for anomaly detection technology in scenarios such as industrial inspection, medical imaging, and autonomous driving, most existing anomaly detection technologies focus on supervised learning or multi-shot learning models. However, supervised anomaly detection requires a large amount of labeled data for training, which is impractical in many real-world applications. Therefore, zero-shot anomaly detection has gradually become a research hotspot.

[0003] Existing zero-shot anomaly detection methods typically employ complex two-branch structures and combine them with text prompt templates to improve detection accuracy. However, these methods suffer from problems such as strong reliance on prior knowledge and high computational overhead, which severely limit their application in dynamic and diverse environments. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a prototype-guided single-branch prompting zero-sample anomaly detection method and system. This invention effectively solves the problems of high computational overhead, strong reliance on prior knowledge, and insufficient cross-modal consistency in existing technologies.

[0005] The technical solution of this invention is: a prototype-guided single-branch hint zero-sample anomaly detection method, comprising the following steps:

[0006] S1) Receive at least one input training image;

[0007] S2) Use a pre-trained visual language model to extract features from the image to obtain image features;

[0008] S3) Based on image features, a prototype-guided adaptive prompt generation strategy is used to generate single-branch text prompts that match the image features;

[0009] S4) Convert the pre-trained visual language model of the text prompt input into text features;

[0010] S5) Map image features and text features to a shared multimodal feature space, calculate the similarity between features, and use a single-branch text architecture to calculate image-level anomaly scores and pixel-level anomalies;

[0011] S6) Based on the image-level anomaly score and pixel-level anomaly, determine whether the input image contains anomalies and the specific location of the anomalies.

[0012] Preferably, in step S2), the visual language model includes a visual encoder and a text encoder; and the image is feature extracted by the pre-trained visual encoder; and the text prompts are converted into text features by the pre-trained text encoder.

[0013] The visual language model is jointly pre-trained using images and text matching the image content to optimize the model, thereby reducing the distance between image features and corresponding text features and increasing the distance with irrelevant text features.

[0014] Preferably, in step S2), the visual encoder simultaneously outputs a category password and a block password. The category password summarizes image information for image-level anomaly detection, and the block password represents local information of the image for pixel-level anomaly detection.

[0015] Preferably, in step S3), a prototype-guided adaptive prompt generation strategy is used to generate a single-branch text prompt that matches the image features, specifically including the following steps:

[0016] S31) Randomly initialize several learnable prototype vectors;

[0017] S32) During training, prototype passwords are generated by using prototype vectors and corresponding image blocks, and multiple prototype passwords are concatenated into prototype prompts.

[0018] S33) Combine the prototype prompt with a learnable bias matrix to generate a text prompt.

[0019] Preferably, in step S32), the calculation expression for the prototype password p corresponding to each image block password using the prototype vector is as follows:

[0020]

[0021] In the formula, σ(·) is the function used for scaling; v m h represents a learnable prototype vector. p (·) represents the image features f of the d-th layer. d A function projected onto the same space as the text dimension; θ m and These are learnable parameters; To balance the values ​​and prevent gradient explosion, C T The dimension of the prototype vector. Representing image features f d The block password.

[0022] Preferably, in step S33), the expression for generating the text prompt W is:

[0023] W = α·P + β·Q

[0024] In the formula, P is a matrix composed of prototype passwords p, and Q is a matrix composed of shared deviation vectors [q1; q2; ...; q M The bias matrix is ​​composed of α and β, which are weights used to balance the contributions of P and Q.

[0025] Preferably, in step S5), the image features and text features are mapped to a shared multimodal feature space, specifically including the following steps:

[0026] S511) Use a single-layer learnable linear projection to adjust the dimensions of image category passwords and block passwords to match the dimensions of text features;

[0027] S512) Adjust the representation of text features using a single-layer learnable linear projection without changing its dimensionality.

[0028] Preferably, in step S5), a single-branch text architecture is used to calculate image-level anomaly scores and pixel-level anomalies, specifically including the following steps:

[0029] S521) Calculate the inner product of the category password and the average text feature to obtain the image-level anomaly score;

[0030] S522), calculate the inner product of the block password and text features to generate pixel-level anomaly scores;

[0031] S523) Determine whether there is an anomaly in the image based on the size of the inner product. If the similarity exceeds a predetermined threshold, it is determined to be an anomaly.

[0032] Preferably, the present invention also provides a prototype-guided single-branch prompting zero-sample anomaly detection system, comprising:

[0033] The visual encoding module is used to extract features from the input image, segment the image into multiple image patches, and extract features from these image patches through multiple attention layers to obtain the image's category password and patch password.

[0034] The adaptive prompting module is used to generate corresponding text prompts based on the extracted image block passwords.

[0035] The text encoding module receives text prompts generated by the prototype-guided adaptive prompting module, converts them into text embeddings corresponding to image features, and maps them to a visual-text shared feature space.

[0036] The anomaly detection module is used to calculate the similarity between image features and text prompt features, and output the anomaly detection results.

[0037] Preferably, the adaptive prompting module includes:

[0038] The prototype generation unit is used to generate prototype vectors based on image features and construct a prototype matrix using the prototype vectors to represent potential abnormal patterns in the image.

[0039] The deviation generation unit is used to generate a global deviation matrix, which is combined with the prototype matrix to generate a text prompt. The deviation matrix is ​​used to adjust the generation of the prototype vector to make it more adaptable and robust.

[0040] The weighted synthesis unit is used to weight and synthesize the prototype matrix and the deviation matrix according to predefined weighting coefficients to form the final text prompt vector.

[0041] The beneficial effects of this invention are as follows:

[0042] 1. This invention introduces a pre-trained visual language model into the zero-shot anomaly detection task. Through the proposed prototype-guided adaptive prompting module and single-branch text architecture, the accuracy and efficiency of anomaly detection are significantly improved.

[0043] 2. This invention adopts a single-branch architecture, which greatly simplifies the network structure and reduces the consumption of computing resources. The single-branch architecture makes the entire detection process more efficient and can be executed quickly with limited computing resources, making it especially suitable for real-time application scenarios.

[0044] 3. This invention generates text prompts consistent with image features through a prototype-guided adaptive prompt generation strategy, avoiding reliance on specific prior knowledge. This enables the invention to perform anomaly detection without relying on any prior labels or templates, making it more universal and widely applicable to different fields and dynamically changing environments.

[0045] 4. The prototype-guided adaptive prompting module of this invention has strong adaptability and flexibility. It can dynamically generate text prompts based on the features of the input image without the need for fixed templates or labels. Moreover, it can adjust the length and content of the generated text prompts according to actual needs, making the model more flexible and accurate in handling different types of anomalies. This adaptive capability enables this invention to run stably in various dynamic environments and handle a variety of complex anomaly detection tasks. Attached Figure Description

[0046] Figure 1 This is a schematic flowchart of the method of the present invention;

[0047] Figure 2 This is a schematic diagram of the adaptive prompting module guided by the prototype of the present invention. Detailed Implementation

[0048] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a prototype-guided single-branch hint zero-sample anomaly detection method, including the following steps:

[0051] S1), Input training image X i and its corresponding anomaly mask S i where i∈{0,1,…,I} N}, I N Indicates the number of input images; and for image X i Perform the following preprocessing:

[0052] The mean is [0.485, 0.456, 0.406] and the variance is...

[0053] The image X is obtained by normalizing the input to [0.229, 0.224, 0.225] and adjusting the input size to 512×512. i ;

[0054] S2) Extract the preprocessed image X using the visual encoder g(·). i Image features, including category password f c and block password Right now:

[0055]

[0056] Where n∈{0,1,…,N}, N represents the number of image patches;

[0057] In this embodiment, during the process of visual encoder extracting image features, the preprocessed image X is first... i The code is divided into N fixed-size blocks, which are then projected onto the feature space through convolutional layers to generate N block passwords. Simultaneously, a learnable class password is concatenated with these block passwords. After computation through multiple attention modules based on the Transformer architecture and a linear projection layer, the class password f is output. c and block password

[0058] S3) Generate a single-branch text hint W using a prototype-guided adaptive hint generation strategy; generate hints from block passwords. Learn the corresponding prompts and commands in the middle. m Where m∈{0,1,…,M}; specifically, the steps are as follows:

[0059] S31) The D blocks of image features extracted by the visual encoder Used to generate text prompts;

[0060] S32) Randomly initialize M learnable prototype vectors v m Its dimension C V Same dimension as block password;

[0061] In this embodiment, steps S31) and S32) can be interchanged; that is, M prototype vectors v are first randomly initialized. m Enter the block password into the prototype-guided adaptive prompt module.

[0062] S33), based on M prototype vectors v m and the corresponding d-th layer block password Calculate the prototype password p m ,Right now:

[0063]

[0064] Where σ(·) is the softmax function used for scaling, h p (·) represents the image features f of the d-th layer. d Projected onto a linear projection layer in the same space as the text dimension, with parameter θ m and Belongs to the linear projection layer, dividend Used to balance values ​​and prevent gradient explosion, C T The dimension of the prototype vector;

[0065] Each block password M prototype passwords are calculated and concatenated into a prototype matrix P, thereby generating D prototype matrices P for D block passwords;

[0066] S34) Randomly initialize D shared deviation matrices Q, where Q is formed by the deviation vector [q1; q2; ...; q...]. M Composed of [a set of text prompts], Q has the same dimension as P; calculate D text prompts W:

[0067] W = α·P + β·Q;

[0068] Here, α and β are weights used to balance the contributions of P and Q.

[0069] S4) Use the text encoder t(·) to extract the text features of the text cue W. The calculation formula is:

[0070]

[0071] Where i∈{0,1,…,D}, D represents the number of image patches, and the function h t(·) represents the text features f of the d-th layer. i t A linear projection layer projected onto a multimodal feature space, θ d These are parameters belonging to the linear projection layer;

[0072] S5) Calculate the category password for image features. c and block password and text features f i t Similarity to predict image-level anomaly scores and pixel-level anomaly scores The calculation formula is:

[0073]

[0074] Among them, h c h v It is a linear projection layer that projects the password onto the multimodal feature space, μ d , λ d c represents the learnable weights, and c(·) represents the restoration function;

[0075] In this embodiment, because Size and input image X i The dimensions are mismatched, and the result after dot product is a vector, so it needs to be resized and bilinearly interpolated to enlarge it to the image X. i The size.

[0076] S6) Calculate the image-level loss L i and pixel-level loss L p ;

[0077]

[0078] Where δ represents the true outlier, δ∈{0,1}, 0 represents normal, and 1 represents outlier; S represents the outlier mask.

[0079] Determine whether the current training round e satisfies the set value E. If it does, save the model and stop training. Otherwise, update the model parameters through gradient propagation and continue repeating steps S3)-S6).

[0080] In practical applications, this embodiment outperforms other methods on the dataset, demonstrating a significant performance improvement. Using this embodiment and three existing image anomaly detection methods—the SAA method based on a segmentation model, the WinCLIP method based on a sliding window, and the ACLIP method based on dual text branches—image-level and pixel-level anomaly score maps were calculated for all test images in the MVTec, VisA, and BTAD test datasets. The area under the feature curves of each method, i.e., the AUC, was calculated, and the results are shown in Table 1.

[0081] Table 1. Comparison of pixel-level AUC evaluation results (%) for four methods

[0082]

[0083] The pixel-level AUC is the area AUC enclosed by the pixel-level test feature curve and the horizontal axis. It is used to measure the effectiveness of the anomaly localization method. Its value is between 0% and 100%, and the larger the value, the better the localization effect.

[0084] As can be seen from Table 1, the average pixel-level AUC of this embodiment on the test set is higher than that of the other three methods, proving that this embodiment has better zero-sample anomaly detection performance.

[0085] Example 2

[0086] This embodiment provides a prototype-guided, single-branch cue-based adaptive zero-sample anomaly detection system, characterized by comprising:

[0087] The visual encoding module is used to extract data from the input image X. i The image features are extracted, the image is segmented into multiple image blocks, and the image blocks are processed through multiple attention layers to extract features, thereby obtaining the image category password and block password;

[0088] The prototype-guided adaptive prompting module is used to generate corresponding text prompts based on the extracted image patch passwords; specifically as follows:

[0089] A) Based on the visual encoder, extract image features to obtain D block passwords for generating text prompts.

[0090] B) Randomly initialize M learnable prototype vectors v m Its dimension C V Same dimension as block password;

[0091] C) Based on M prototype vectors v m and the corresponding d-th layer block password Calculate the prototype password p m The calculation formula is:

[0092]

[0093] Where σ(·) is the softmax function used for scaling, h p (·) represents the image features f of the d-th layer. d Projected onto a linear projection layer in the same space as the text dimension, with parameter θ m and Belongs to the linear projection layer, dividend Used to balance values ​​and prevent gradient explosion, C T The dimension of the prototype vector;

[0094] Each block password M prototype passwords are calculated and concatenated into a prototype matrix P, thereby generating D prototype matrices P for D block passwords;

[0095] D) Randomly initialize D shared deviation matrices Q, where Q is formed by the deviation vector [q1; q2; ...; q...]. M The composition of Q is the same as that of P;

[0096] E) Calculate D text prompts W:

[0097] W = α·P + β·Q;

[0098] Here, α and β are weights used to balance the contributions of P and Q.

[0099] The text encoding module receives text prompts generated by the prototype-guided adaptive prompting module, converts them into text embeddings corresponding to image features, and maps them to a visual-text shared feature space.

[0100] The anomaly detection module is used to calculate the similarity between image features and text prompt features, and predict and output anomaly detection results based on the similarity.

[0101] In this embodiment, the anomaly detection result includes a predicted image-level anomaly score. and pixel-level anomaly scores Right now:

[0102]

[0103] Among them, h c ,h v It is a linear projection layer that projects the password onto the multimodal feature space, μ d ,λ d represents the learnable weights, and c(·) represents the restoration function.

[0104] In this embodiment, the prototype-guided adaptive prompting module includes:

[0105] The prototype generation unit is used to generate prototype vectors based on image features and construct a prototype matrix using the prototype vectors to represent potential abnormal patterns in the image.

[0106] The deviation generation unit generates a global deviation matrix, which, when combined with the prototype matrix, produces the text prompt. This deviation matrix is ​​used to adjust the generation of the prototype vector, making it more adaptable and robust.

[0107] The weighted synthesis unit is used to weight and synthesize the prototype matrix and the deviation matrix according to predefined weighting coefficients to form the final text prompt vector.

[0108] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A prototype-guided single-branch hint zero-shot anomaly detection method, characterized in that, Includes the following steps: S1) Receive at least one input training image; S2) Use a pre-trained visual language model to extract features from the image to obtain image features; S3) Based on image features, a prototype-guided adaptive prompt generation strategy is used to generate single-branch text prompts that match the image features; specifically, the following steps are included: S31) Randomly initialize several learnable prototype vectors; S32) During training, prototype passwords are generated by using prototype vectors and corresponding image blocks, and multiple prototype passwords are concatenated into prototype prompts. The expression for calculating the prototype password p is: In the formula, σ(·) is the function used for scaling; v m h represents a learnable prototype vector. p (·) represents the image features f of the d-th layer. d A function projected onto the same space as the text dimension; θ m and These are learnable parameters; To balance the values ​​and prevent gradient explosion, C T The dimension of the prototype vector. Representing image features f d The block password; S33) Combine prototype prompts with a learnable bias matrix to generate text prompts; S4) Convert the pre-trained visual language model of the text prompt input into text features; S5) Map image features and text features to a shared multimodal feature space, calculate the similarity between features, and use a single-branch text architecture to calculate image-level anomaly scores and pixel-level anomalies; S6) Based on the image-level anomaly score and pixel-level anomaly, determine whether the input image contains anomalies and the specific location of the anomalies.

2. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 1, characterized in that: In step S2), the visual language model includes a visual encoder and a text encoder; and the image features are extracted from the image by the pre-trained visual encoder; the text prompts are converted into text features by the pre-trained text encoder; the visual language model is jointly pre-trained using the image features and the text features that match the image content, and the visual language model is optimized to reduce the distance between the image features and the corresponding text features, and increase the distance with the unrelated text features.

3. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 2, characterized in that: In step S2), the visual encoder simultaneously outputs a category password and a block password. The category password summarizes image information for image-level anomaly detection, and the block password represents local information of the image for pixel-level anomaly detection.

4. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 1, characterized in that: In step S33), the expression for generating the text prompt W is: W = α·P + β·Q In the formula, P is a matrix composed of prototype passwords p, and Q is a matrix composed of shared deviation vectors [q1; q2; ...; q M The bias matrix is ​​composed of α and β, which are weights used to balance the contributions of P and Q.

5. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 1, characterized in that: In step S5), the image features and text features are mapped to a shared multimodal feature space, specifically including the following steps: S511) Use a single-layer learnable linear projection to adjust the dimensions of image category passwords and block passwords to match the dimensions of text features; S512) Adjust the representation of text features using a single-layer learnable linear projection without changing its dimensionality.

6. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 5, characterized in that: In step S5), an image-level anomaly score and pixel-level anomaly score are calculated using a single-branch text architecture, specifically including the following steps: S521) Calculate the inner product of the category password and the average text feature to obtain the image-level anomaly score. Right now: In the formula, h c A linear projection layer to project the password onto the multimodal feature space; f c For category password; μ d These are learnable weights; D is the block password; D is the number of block passwords. S522) Calculate the inner product of the block password and text features to generate pixel-level anomaly scores. Right now: In the formula, Indicates pixel-level anomalies; c(·) represents the restoration function; λ d h represents the learnable weights. v A linear projection layer to project the password into a multimodal feature space; D is the block password; D is the number of block passwords. S523) Determine whether there is an anomaly in the image based on the size of the inner product. If the similarity exceeds a predetermined threshold, it is determined to be an anomaly.

7. The prototype-guided single-branch hint zero-sample anomaly detection method according to claim 6, characterized in that: In step S6), based on the image-level anomaly score... and pixel-level anomaly scores Calculate image-level loss L i and pixel-level loss L p ,Right now: In the formula; δ represents the true outlier, δ∈{0,1}, where 0 represents normal and 1 represents outlier; S represents the anomaly mask.

8. A prototype-guided single-branch zero-shot anomaly detection system, characterized in that, The system described herein performs prediction detection using the method described in any one of claims 1-7, and the system comprises: The visual encoding module is used to extract image features from the input image, segment the image into multiple image blocks, and extract features from these image blocks through multiple attention layers to obtain the image's category password and block password. The adaptive prompting module is used to generate corresponding text prompts based on the extracted image block passwords. The text encoding module receives text prompts generated by the prototype-guided adaptive prompting module, converts them into text embeddings corresponding to image features, and maps them to a visual-text shared feature space. The anomaly detection module is used to calculate the similarity between image features and text prompt features, and output the anomaly detection results.

Images