A zero-shot anomaly image detection method based on learnable prompts

Abstract

The application discloses a zero-shot abnormal image detection method based on a learnable prompt. A learnable prompt generation module based on context optimization is designed, which contains a learnable prompt and an image abnormal state prompt that can be optimized. A multi-level visual coding feature of a to-be-detected image is obtained by using an image coding network of a visual language large model, and a text feature of a learnable prompt embedding is obtained by using a text coding network. A multi-level cosine similarity between the visual coding feature and the text feature is calculated to construct an image abnormal area calculation module, so that an abnormal area of the to-be-detected image is obtained. The learnable prompt avoids the complexity and instability of manually designed prompts, improves the accuracy of image abnormal detection, guarantees the effectiveness and efficiency of zero-shot learning, and greatly reduces the cost of pre-training of a visual language large model to a downstream task.

Description

Technical Field

[0001] This invention relates to an image anomaly detection method in the fields of computer vision and anomaly image detection, and particularly to a zero-shot anomaly image detection method based on learnable cues. Background Technology

[0002] Anomaly detection primarily aims to identify situations that differ from normal patterns, and it is widely used in video surveillance, product quality control, and medical diagnosis. Most current popular anomaly detection methods train anomaly detection models using anomaly-free images to locate abnormal regions. However, these methods construct corresponding anomaly detection models for each specific object category and rely on a large number of normal images of the corresponding category, failing to meet the real-world anomaly detection needs that include a wide range of object categories. Zero-shot learning based on cue engineering has opened up new possibilities for the real-world application of anomaly detection.

[0003] However, the design of suggestion engineering relies heavily on the experience of domain experts and requires a significant amount of time to manually optimize suggestions.

[0004] In addition, even minor text changes in the hand-designed prompts can have a significant impact on detection performance, making them highly unstable. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by proposing a zero-shot anomaly image detection method based on learnable cues. This method enhances the generalization ability of the cue engineering by constructing learnable cues, reducing the reliance on domain-specific knowledge for manually designed cues. Furthermore, the organic combination of learnable cues and status cues improves the accuracy of zero-shot anomaly detection while ensuring the stability and reliability of the cue engineering.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] 1) Establish a zero-shot anomaly image detection network;

[0008] 2) Utilizing anomaly image detection networks;

[0009] 3) Input the image to be tested into the trained abnormal image detection network to obtain an abnormal region heatmap, and obtain the abnormal region based on the abnormal region heatmap.

[0010] The aforementioned abnormal image detection network includes a visual encoding network, a text encoding network, a multilayer perceptron (MLP), a learnable cue generation module, and an abnormal region calculation module. The output of the visual encoding network is fed into the abnormal region calculation module after passing through multiple MLPs. Simultaneously, the learnable cue generation module generates learnable cue embeddings, which are then processed by the text encoding network and fed into the abnormal region calculation module. The output of the abnormal region calculation module is then subjected to a feature dimension change operation and an element-wise multiplication operation to obtain the abnormal region heatmap.

[0011] The visual coding network comprises four stages performed sequentially. Each stage contains multiple layers of residual attention modules and outputs visual features. The four stages output a first visual feature T1, a second visual feature T2, a third visual feature T3, and a fourth visual feature T4 as the output of the visual coding network, respectively. The first stage receives the original input image, and each of the other stages receives the visual features output by the previous stage.

[0012] The four visual features output by the visual coding network are processed by their respective multilayer perceptrons (MLPs) to obtain four visual coding features, which are then input into the abnormal region calculation module.

[0013] The learnable hint generation module includes learnable hints and state hints [state];

[0014] The learnable hint embedding t is obtained from the learnable hint and the state hint [state] according to the following formula, and is used as the output of the learnable hint generation module:

[0015] t = [V]1[V]2…[V] N [state]

[0016] The learnable hints contain multiple vectors V N Each vector has the same dimension as the state cue, is obtained through random initialization, and is continuously updated and optimized with the network gradient during training;

[0017] The aforementioned state prompt [state] is obtained by encoding the state through the Visual Language Contrast Model (CLIP model). The state is a string indicating whether the state is normal or not. It is represented by general text with weak domain knowledge and is divided into strings belonging to normal states and strings belonging to abnormal states.

[0018] The string "state" for the normal state can typically be: 'normal class', 'flawless class', 'perfect class', 'class without defect', representing the normal class, the flawless class, the perfect class, and the class without defects, respectively. 'class' indicates the object category in the image to be detected.

[0019] The string "state" for abnormal states can typically be: 'abnormal class', 'damaged class', 'broken class', 'class with defect', representing abnormal, damaged, broken, and defective categories, respectively.

[0020] The learnable cue generation module outputs learnable cue embeddings, which are then input into a text encoding network to obtain text features. The text encoding network employs a visual-language contrastive model (CLIP model).

[0021] The abnormal region calculation module obtains the region based on the cosine similarity between the input visual encoding features and text features in the following manner:

[0022] G m =softmax(F m @W T ), m=1,2,3,4

[0023] Where the symbol @ represents matrix multiplication, softmax represents the activation function, and W T F represents the transpose of text features. m G represents the visual coding feature obtained at stage m in the visual coding network. m Represents visual encoding features F m The corresponding preliminary abnormal area map.

[0024] The four preliminary anomaly region maps output by the anomaly region calculation module were reshaped by changing the feature dimensions to obtain four anomaly region maps M. m Then map the four abnormal regions M m Multiply the corresponding elements to obtain the final heatmap of the abnormal area.

[0025] In the abnormal region heatmap, the pixel regions larger than the set abnormal threshold are the final abnormal regions.

[0026] Step 2) employs zero-shot transfer learning during training. The network parameters of the visual encoding network and text encoding network in the anomaly image detection network are not updated; only the parameters of the learnable prompt generation module and the multilayer perceptron (MLP) are optimized. The training process uses the Adam optimizer with a fixed learning rate of 0.001, a batch size of 8, and one epoch.

[0027] In step 3), specifically, in the abnormal region heatmap, image regions with pixel values ​​greater than a preset abnormal threshold are considered as abnormal regions.

[0028] This invention designs a context-optimized learnable cue generation module, which includes learnable cuees with optimizable parameters and image anomaly state cuees. It utilizes an image coding network of a large visual language model to obtain multi-level visual coding features of the image to be inspected, and simultaneously utilizes a text coding network to obtain text features embedded in the learnable cuees. By calculating the multi-level cosine similarity between the visual coding features and the text features, an image anomaly region calculation module is constructed to obtain the anomaly regions of the image to be inspected.

[0029] The beneficial effects of this invention are:

[0030] The method of this invention can construct prompting engineering by utilizing general textual expressions of weak domain knowledge, which greatly reduces the reliance on expert knowledge.

[0031] The learnable hints proposed in this invention avoid the complexity and instability of manually designed hints, while improving the accuracy of image anomaly detection. Furthermore, the designed learnable hints can continuously optimize with network gradients, achieving end-to-end learning and training, ensuring the effectiveness and efficiency of zero-shot learning, and greatly reducing the cost of transferring pre-trained large visual language models to downstream tasks; it also maintains consistency across different detection categories, improving the generalization performance and stability of the learnable hints. Attached Figure Description

[0032] Figure 1 It is an abnormal image detection network based on learnable prompts;

[0033] Figure 2 This is a schematic diagram of a zero-shot anomaly image detection method based on learnable prompts. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] It should be understood that the embodiments described in this invention are exemplary, and the specific parameters used in the embodiment description are only for the purpose of describing this invention and are not intended to limit this invention.

[0036] like Figure 2 As shown, an embodiment of the present invention based on zero-shot anomaly image detection with learnable prompts is as follows:

[0037] Step 1: Build a learnable hint generation module.

[0038] The constructed learnable hint generation module is used to build a learnable hint embedding t, which consists of two parts: learnable hints and state hints. The learnable hints are obtained through a random initialization strategy, while the state hints are constructed using general domain knowledge. The learnable hint embedding is expressed as follows:

[0039] t = [V]1[V]2…[V] N [state]

[0040] Here, [state] represents the embedding of the state cue, and its encoding method is consistent with the CLIP model. [V] N It is a randomly initialized vector with the same dimensions as [state]. N represents the number of vectors included in the learnable hints.

[0041] The state indicator uses a generic text representation with weak domain knowledge, where:

[0042] The state cue for the normal state contains four categories: 'normal class', 'flawless class', 'perfect class', and 'class without defect'. Based on the cue vectors, a learnable cue embedding is constructed for each state cue, resulting in a total of four learnable cue embeddings.

[0043] Similarly, the state prompt for abnormal states contains four categories: 'abnormal class', 'damaged class', 'broken class', and 'class with defect', and a total of four learnable prompt embeddings are constructed.

[0044] Each state cue is encoded using the CLIP model to obtain a state cue embedding [state]. The state cue embedding [state] and the cue vector [V] are then compared. N The concatenation yields the complete learnable cue embedding t. The state cue embedding maintains the same dimension as the cue vector.

[0045] For different status cues for normal and abnormal samples, such as 'normal class' and 'abnormal class', the same cue vector [V] is used. N .

[0046] In this embodiment, the cue vector [V] N The number of vectors is set to 16, i.e., N = 16. Each vector is obtained through random initialization. The vector dimension is 16 × 768.

[0047] Step 2: Construct the abnormal region calculation module.

[0048] The probability of a location being an anomalous region is determined based on the cosine similarity between visual encoding features and text features. Anomalous regions are obtained by calculating the cosine similarity between visual encoding features and text features. This invention extracts visual encoding features F from four stages. m m = 1, 2, 3, 4, and text features W based on learnable cue embeddings were extracted. Visual encoding features F m Preliminary anomaly map G calculated between text feature W and text feature W m The expression is as follows:

[0049] G m =softmax(F m @W T m = 1, 2, 3, 4

[0050] Where the symbol @ represents matrix multiplication, softmax represents the activation function, and W T This represents the transpose of text features.

[0051] An abnormal region calculation module is constructed based on the above formula. The calculation result of the formula represents the probability value that the region is an abnormal region.

[0052] Step 3: Construct an anomaly detection network.

[0053] The constructed anomaly image detection network based on learnable prompts is as follows: Figure 1 As shown in the diagram, both the visual encoding network and the text encoding network used in this network employ pre-trained models provided by the CLIP method. The visual encoding network contains 24 layers of residual attention modules. This invention utilizes visual features at different levels, dividing the CLIP visual encoding network into four stages, labeled as stage one, stage two, stage three, and stage four, each containing six layers of residual attention modules. The specific structure of the residual attention modules can be found in the CLIP model and will not be elaborated upon here.

[0054] The symbols in the network are as follows: B represents the number of images input to the network in a batch; H and W represent the height and width of the image; L, C, and d all represent the dimension of a certain feature; T1, T2, T3, and T4 represent the visual features of the first, second, third, and fourth stages, respectively, with each feature having a dimension of B×L×C; mlp represents a multilayer perceptron, and the number in parentheses represents the number of nodes in the multilayer perceptron; the reshape operation represents changing the dimension and shape of the feature.

[0055] The anomaly detection network takes a two-dimensional image to be inspected, I, and a learnable cue embedding, t, as input to predict anomalous regions in the image. The image to be inspected undergoes first, second, third, and fourth stages of visual processing to obtain visual features T1, T2, T3, and T4, respectively. These four visual features are then processed by mlp(768) to obtain visual encoding features F1, F2, F3, and F4. Simultaneously, the learnable cue embedding constructed by the learnable cue generation module is processed by a text encoding network to obtain text encoding features W.

[0056] Text features W are used in conjunction with visual encoding features F1, F2, F3, and F4, respectively. After passing through the anomaly region calculation module and undergoing a reshape operation, four anomaly region maps M1, M2, M3, and M4 are obtained. The dimensions of all four anomaly region maps are B×H×W, consistent with the spatial dimensions of the input image. The four anomaly region maps are multiplied element-wise to obtain the final anomaly region heatmap of the image to be inspected.

[0057] In this embodiment, the input image size is H×W = 518×518, and the batch size is B = 8. Dimension L is 1370, dimension C is 1024, and dimension d is 768. Both the visual encoding network and the text encoding network use pre-trained models from the CLIP open-source platform.

[0058] The text encoding network needs to process four learnable cue embeddings from normal samples simultaneously to obtain four corresponding text encoding features. The average of these four text encoding features is taken as the text encoding feature for the normal sample, with a dimension of B×1×d. Similarly, for four learnable cue embeddings from anomalous samples, the resulting anomalous sample text encoding feature has a dimension of B×1×d. The text encoding features of the normal and anomalous samples are concatenated to obtain the text encoding feature W, with a dimension of B×2×d.

[0059] Step 4: Training the constructed anomaly detection network. This invention employs a zero-shot anomaly detection method, which cannot use relevant data from the dataset to be evaluated. To verify the effectiveness of the proposed method, this invention trains the constructed anomaly detection network using only the test set portion of the MVTec AD dataset and evaluates the network on the test set portion of the VisA dataset. During network training, only the MLP(768) network portion and the learnable cue vector portion of the multilayer perceptron are optimized and updated; other network parameters are not updated (the visual encoding network and text encoding network in the network use the parameters provided by the CLIP model and are not updated or optimized). The training process uses the Adam optimizer with a fixed learning rate of 0.001, a batch size of 8, and 1 epoch.

[0060] Step 5: After training, the anomaly detection network directly predicts the anomaly region heatmap of the input object image. In practical applications, an anomaly threshold can be set; regions in the anomaly region heatmap predicted by the network that exceed the set threshold are the final anomaly regions.

[0061] Compared with existing technologies, this invention constructs a learnable cue generation module, integrating cue engineering design into the network parameter optimization process, thereby improving the efficiency of cue engineering construction and overcoming the complexity and instability of manually designed cue engineering. Simultaneously, the organic combination of learnable cue and status cue ensures the generalization performance of text features and significantly reduces the cost of transferring large visual language models to downstream anomaly detection tasks, demonstrating strong engineering application value.

[0062] Testing on the VisA dataset showed that this invention achieved a pixel-level anomaly detection accuracy of 95.2%. This anomaly detection performance significantly outperforms the classic zero-shot detection method, WinCLIP.

[0063] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited to the above embodiment. Any equivalent substitutions or changes made by those skilled in the art within the technical scope described in the present invention, based on the technical solution and inventive concept of the present invention, should be included within the scope of protection of the present invention.

Claims

1. A zero-shot anomaly image detection method based on learnable prompts, characterized in that: 1) Establish an abnormal image detection network; 2) Utilizing anomaly image detection networks; 3) Input the image to be tested into the trained abnormal image detection network to obtain an abnormal region heatmap, and obtain the abnormal region based on the abnormal region heatmap; The aforementioned abnormal image detection network includes a visual encoding network, a text encoding network, a multilayer perceptron (MLP), a learnable cue generation module, and an abnormal region calculation module. The output of the visual encoding network is fed into the abnormal region calculation module after passing through multiple MLPs. Simultaneously, the learnable cue generation module generates learnable cue embeddings, which are then processed by the text encoding network and fed into the abnormal region calculation module. The output of the abnormal region calculation module is then subjected to a feature dimension change operation and an element-wise multiplication operation to obtain the abnormal region heatmap. The visual coding network comprises four stages performed sequentially. Each stage contains multiple layers of residual attention modules and outputs visual features. The four stages output the first visual feature T1, the second visual feature T2, the third visual feature T3, and the fourth visual feature T4 as the output of the visual coding network, respectively. The abnormal region calculation module obtains the region based on the cosine similarity between the input visual encoding features and text features in the following manner: Gm=softmax(Fm @ WT), m=1,2,3,4 Where @ represents matrix multiplication, softmax represents the activation function, WT represents the transpose of the text feature, Fm represents the visual coding feature obtained in the m-th stage of the visual coding network, and Gm represents the preliminary anomaly region map obtained corresponding to the visual coding feature Fm.

2. The zero-shot anomaly image detection method based on learnable prompts according to claim 1, characterized in that: The four visual features output by the visual coding network are processed by their respective multilayer perceptrons (MLPs) to obtain four visual coding features, which are then input into the abnormal region calculation module.

3. The zero-shot anomaly image detection method based on learnable prompts according to claim 1, characterized in that: The learnable hint generation module includes learnable hints and state hints [state]; The learnable hint embedding t is obtained from the learnable hint and the state hint [state] according to the following formula, and is used as the output of the learnable hint generation module: t = [V]1[V]2…[V] N [state] The learnable hints contain multiple vectors V N Each vector has the same dimension as the state cue [state], and is obtained through random initialization; The aforementioned state prompt [state] is obtained by encoding the state through a visual language contrast model. The state is a string indicating whether the state is normal or not, and is divided into strings belonging to normal states and strings belonging to abnormal states.

4. The zero-shot anomaly image detection method based on learnable prompts according to claim 3, characterized in that: The learnable hint generation module outputs learnable hints, which are then embedded and input into a text encoding network to obtain text features.

5. The zero-shot anomaly image detection method based on learnable prompts according to claim 1, characterized in that: The four preliminary anomaly region maps output by the anomaly region calculation module were reshaped by changing the feature dimensions to obtain four anomaly region maps M. m Then map the four abnormal regions M m Multiply the corresponding elements to obtain the final heatmap of the abnormal area.

6. The zero-shot anomaly image detection method based on learnable prompts according to claim 1, characterized in that: In step 2), zero-shot transfer learning is used during training. The network parameters of the visual encoding network and text encoding network in the abnormal image detection network are not updated. Only the parameters of the learnable prompt generation module and the parameters of the multilayer perceptron (MLP) are optimized and trained.

7. The zero-shot anomaly image detection method based on learnable prompts according to claim 1, characterized in that: In step 3), specifically, in the abnormal region heatmap, image regions with pixel values ​​greater than a preset abnormal threshold are considered as abnormal regions.

Images