A zero-shot anomaly detection method based on multimodal learnable cues

Abstract

The present invention discloses a zero-sample anomaly detection method based on multimodal learnable prompts. The method includes constructing a highly versatile learnable text prompt embedding feature for normal sample and abnormal sample states, designing a learnable visual prompt vector for a visual coding network, and inserting it into the input space of a specified visual coding network, extracting visual features of an input image from multiple stages using a visual coding network of a large visual language model, calculating cosine similarity between the obtained multiple visual features and the constructed learnable text prompt embedding feature, and predicting the abnormal area of ​​the input image to be detected through a constructed abnormal area calculation module. The learnable text prompt embedding feature proposed by the present invention effectively reduces the design difficulty of the prompt engineering, the learnable visual prompt vector reduces the difficulty of fine-tuning the large model, the anomaly detection network does not rely on the text coding network, greatly reduces the complexity of the network, and improves the accuracy of anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and anomaly detection, and in particular relates to a zero-sample anomaly detection method based on multimodal learnable prompts. Background Art

[0002] The main task of anomaly detection is to identify situations that are different from the normal pattern. It is widely used in industrial defect detection, medical image analysis, video surveillance and other fields. Supervised anomaly detection methods rely on carefully organized anomaly samples and cannot effectively detect anomaly types that do not exist in the training data. Unsupervised anomaly detection methods are only trained on normal samples and regard feature data outside the normal sample distribution as anomalies. They have become an important research direction for anomaly detection. However, unsupervised methods need to learn anomaly boundaries on a large number of normal samples, which brings a heavy data collection task. Zero-shot anomaly detection methods do not rely on any data in the target domain, providing a new solution for new anomaly types and scenarios where data is difficult to collect. Most of the existing zero-shot anomaly detection methods are based on large visual language models and rely on expert experience to design prompt templates, which increases the difficulty of network design and reduces the generalization performance of the model. In addition, most of the existing methods rely on the text encoding network of the large model to extract text description features, which greatly increases the complexity of the anomaly detection model and is not conducive to improving the efficiency of anomaly detection. Summary of the invention

[0003] In order to solve the problems existing in the background technology, the purpose of the present invention is to provide a zero-sample anomaly detection method based on multimodal learnable prompts to solve the problems of complex structure and low detection efficiency of existing anomaly detection models. The present invention alleviates the dependence of prompt engineering on expert domain knowledge by designing a highly versatile learnable prompt embedding to represent abnormal and normal states. In addition, an anomaly detection framework that does not rely on text encoding networks is constructed to effectively reduce the computational complexity of the model.

[0004] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:

[0005] Step S1) constructing an abnormal image detection network based on multimodal learnable cues;

[0006] Step S2) training an abnormal image detection network;

[0007] Step S3) The image to be tested is input into the trained abnormal image detection network to obtain an abnormal region heat map, and the abnormal region of the image to be tested is obtained according to the abnormal region heat map.

[0008] The abnormal image detection network includes a visual coding network, a multi-layer perceptron MLP, a multimodal learnable prompt module and an abnormal area calculation module; the visual coding network is loaded with a learnable visual prompt vector in the multimodal learnable prompt module, the output of the visual coding network is input into the abnormal area calculation module after passing through multiple multi-layer perceptrons MLP, and at the same time, a learnable text prompt embedding feature is obtained through the multimodal learnable prompt module, and the learnable text prompt embedding feature is input into the abnormal area calculation module, and the output of the abnormal area calculation module is successively subjected to a feature dimension change operation and an element addition operation to obtain the result of the abnormal area heat map.

[0009] The visual coding network includes four stages that are performed in sequence, each stage includes multiple layers of residual attention layers connected in sequence, each stage of the visual coding network outputs visual coding features, and the four stages respectively output the first visual coding feature T1, the second visual coding feature T2, the third visual coding feature T3 and the fourth visual coding feature T4 as the output of the visual coding network.

[0010] The four coded visual features output by the visual coding network are processed by respective multi-layer perceptrons (MLP) to obtain four visual features, and the four visual features are input into the abnormal area calculation module together.

[0011] The multimodal learnable prompt module includes a learnable text prompt embedding feature E. The steps of constructing the learnable text prompt embedding feature E are as follows:

[0012] 1) Initialize state text description: First, construct the state text description of the corresponding sample according to the state of the sample;

[0013] 2) Initialize state encoding features: Use the pre-trained word segmentation embedding module of the visual language model CLIP to encode the state text descriptions of normal samples and abnormal samples respectively to obtain the state encoding features FN of normal samples and the state encoding features FA of abnormal samples;

[0014] 3) Construct a learnable text prompt vector: average the state encoding feature FN of normal samples and the state encoding feature FA of abnormal samples respectively to obtain the learnable text prompt vector VN of normal samples and the learnable text prompt vector VA of abnormal samples;

[0015] 4) Construct the learnable text prompt embedding feature E: Concatenate the learnable text prompt vector VN of the normal sample and the learnable text prompt vector VA of the abnormal sample in the 0th dimension to obtain the learnable text prompt embedding feature E, and use it as the output of the multimodal learnable prompt module and input it into the abnormal area calculation module.

[0016] The multimodal learnable cue module further includes a learnable visual cue vector P, which is inserted into the residual attention layer of the visual encoding network. The steps of constructing the learnable visual cue vector P are as follows:

[0017] 1) First, determine the number of tokens n in the learnable visual cue vector P to be inserted;

[0018] 2) Then, the dimension of the learnable visual cue vector P is set to n×D according to the number of tokens n, where D represents a preset constant;

[0019] 3) Insert the learnable visual cue vector P of set dimension into the first and second stages of the visual encoding network i The input of the residual attention layer, i Indicates a preset constant.

[0020] In the abnormal area calculation module, the abnormal area is obtained by calculating the cosine similarity of the visual features output by the visual encoding network and the learnable text prompt embedding features output by the multimodal learnable prompt module in the following manner:

[0021] G m =softmax(F m @E T ), m=1,2,3,4

[0022] The symbol @ represents matrix multiplication, softmax() represents the activation function, and E T represents the transpose of the learnable textual hint embedding feature, F m represents the visual features obtained in the mth stage of the visual encoding network, G m Represents the visual feature F m Corresponding to the preliminary abnormal area map obtained.

[0023] The four preliminary abnormal area graphs G output by the abnormal area calculation module m After reshaping the feature dimension, four target abnormal area maps M are obtained. m , and then the four target abnormal area maps M m Add the corresponding elements to obtain the final abnormal area heat map.

[0024] In the step S2), a zero-sample transfer learning method is used during training, and the network parameters of the visual encoding network in the abnormal image detection network are not updated. Only the parameters of the learnable text prompt embedding feature E, the learnable visual prompt vector P and the parameters of the multi-layer perceptron mlp in the multimodal learnable prompt module are optimized and trained.

[0025] In the step S3), the image area whose pixel value is greater than a preset abnormal threshold value is taken as the abnormal area in the abnormal area heat map.

[0026] In the abnormal image detection network training phase, the image to be detected I passes through the first stage, the second stage, the third stage and the fourth stage of the visual coding network in sequence, and obtains the first visual coding feature T1, the second visual coding feature T2, the third visual coding feature T3 and the fourth visual coding feature T4. The four visual coding features are respectively passed through their respective multi-layer perceptrons mlp (768) to obtain the first visual feature F1, the second visual feature F2, the third visual feature F3 and the fourth visual feature F4. The four visual features are respectively calculated with the constructed learnable text prompt embedding feature E through the abnormal area calculation module for cosine similarity, and the first target abnormal area map M1, the second target abnormal area map M2, the third target abnormal area map M3 and the fourth target abnormal area map M4 are obtained through the reshape operation. The four target abnormal area maps calculate the distance loss with the annotated real abnormal area, and iteratively optimize the learnable text prompt embedding feature and the learnable visual prompt vector. After the training is completed, the learnable text prompt embedding feature directly participates in the abnormal detection task in the test phase, and does not require any specific information of the test image, so as to achieve the purpose of zero-sample abnormal detection.

[0027] In the model testing phase, the image to be inspected and the optimized learnable text prompt embedding features are used as inputs. The visual coding network extracts the first visual coding feature T1, the second visual coding feature T2, the third visual coding feature T3 and the fourth visual coding feature T4, and the first target abnormal area map M1, the second target abnormal area map M2, the third target abnormal area map M3 and the fourth target abnormal area map M4 are obtained by the abnormal area calculation module with the learnable text prompt embedding features. The dimensions of the four target abnormal area maps are B×H×W, which is consistent with the input image space size. Then, the four target abnormal area maps are summed according to the corresponding elements to obtain the final abnormal area heat map of the image to be inspected.

[0028] The present invention constructs a highly versatile learnable text prompt embedding feature for normal samples and abnormal sample states. For the visual coding network, a learnable visual prompt vector is designed and inserted into the input space of the specified visual coding network. The visual features of the input image are extracted from multiple stages using the visual coding network of the visual language large model. The cosine similarity of the obtained multiple visual features and the constructed learnable text prompt embedding feature is calculated, and the abnormal area of ​​the input image to be tested is predicted by the constructed abnormal area calculation module. The learnable text prompt embedding feature proposed by the present invention avoids the dependence of conventional methods on prompt templates, and effectively reduces the design difficulty of prompt engineering; the designed learnable visual prompt vector avoids the full parameter optimization of the large model, and reduces the difficulty of fine-tuning the large model. In addition, the designed anomaly detection network does not rely on the text coding network, which greatly reduces the complexity of the network and improves the accuracy of anomaly detection.

[0029] The beneficial effects of the present invention are:

[0030] The present invention uses highly versatile text descriptions to initialize the embedding features of learnable text prompts, effectively avoiding the problem of strong dependence of prompt engineering on expert domain knowledge. In addition, the optimization of learnable prompts is completely data-driven and is not affected by any data characteristics of the target domain, which greatly improves the zero-sample generalization performance of the model. At the same time, the constructed anomaly detection network completely discards the text encoding network, greatly reducing the computational complexity of the model and improving the accuracy of anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a topological diagram of the anomaly detection network structure based on multimodal learnable cues;

[0032] Figure 2 It is a flow chart of the zero-shot anomaly detection method based on multimodal learnable cues;

[0033] Figure 3 is a schematic diagram of the network stage for inserting learnable visual cue vectors;

[0034] Figure 4 is a schematic diagram of the network stage without inserting the learnable visual cue vector;

[0035] Figure 5 Schematic diagram of how to insert a learnable visual cue vector. DETAILED DESCRIPTION

[0036] The present invention is described in detail below in conjunction with specific implementation cases. The following implementation cases will help those skilled in the art to further understand the present invention, but will not limit the present invention in any form.

[0037] The abnormal image detection network of the present invention uses the visual coding network in the pre-trained visual language model CLIP to extract the visual features of the image to be detected. The visual coding network includes twenty-four residual attention layers. The present invention divides the visual coding network into six residual attention layers per stage, and obtains four visual feature extraction stages, which are marked as the first stage, the second stage, the third stage and the fourth stage respectively. The specific structure of the residual attention layer is obtained according to the existing visual language model CLIP, and the present invention will not repeat it.

[0038] Abnormal image detection network based on multimodal learnable cues Figure 1 As shown. Among them, B represents the number of images input to the abnormal image detection network in a batch; H and W represent the height and width of the image respectively; L, C, d all represent the dimension of a certain feature; T1, T2, T3, T4 represent the output visual encoding features of the first stage, the second stage, the third stage and the fourth stage respectively, and the dimensions of the visual encoding features are all B×L×C; mlp represents a multi-layer perceptron, and the numbers in brackets represent the number of nodes of the multi-layer perceptron; the reshape operation represents changing the dimensional shape of the feature. Among them, the learnable visual cue vector is inserted in the first and second stages of the visual encoding network, as shown in Figure 3 As shown in Figure 2, the insertion position of the learnable visual cue vector is indicated by the dashed box. The third and fourth stages of the visual encoding network are constructed by the original CLIP residual attention layer, as shown in Figure 2. Figure 4 shown.

[0039] The specific steps of the embodiment of the present invention are as follows: Figure 2 As shown:

[0040] Step 1: Construct a multimodal learnable prompt module, wherein the construction of the multimodal learnable prompt module includes the construction of a learnable text prompt embedding feature E and the construction of a learnable visual prompt vector P. The construction of the learnable text prompt embedding feature E includes the following steps:

[0041] 1) Initialize state text description: First, construct the state text description of the corresponding sample according to the state of the sample;

[0042] The state of the sample includes normal state and abnormal state. The sample in normal state belongs to normal sample, and the sample in abnormal state belongs to abnormal sample. The state text of normal sample is described as 'normal photo'; the state text of abnormal sample is described as 'abnormal photo'.

[0043] 2) Initialize state encoding features: Use the pre-trained word segmentation embedding module of the visual language large model CLIP to encode the state text descriptions of normal samples and abnormal samples respectively to obtain the state encoding features FN of normal samples and the state encoding features FA of abnormal samples. The dimensions of the two state encoding features are both k×d, where k represents the number of word segmentation tokens, which is determined by the word segmentation embedding module according to the length of the word segmentation in the state text description, and d is the encoding dimension of each word segmentation token;

[0044] 3) Construct a learnable text prompt vector: calculate the average value of the state encoding feature FN of the normal sample and the state encoding feature FA of the abnormal sample in the dimension where the word segmentation token number k is located, so as to obtain the learnable text prompt vector VN of the normal sample and the learnable text prompt vector VA of the abnormal sample;

[0045] Taking the state encoding feature FN of normal samples as an example, on the 0th dimension, the k word segmentation tokens are averaged to obtain the learnable text prompt vector VN of the normal sample, with a dimension of 1×d; similarly, on the 0th dimension, for the state encoding feature FA of abnormal samples, the k word segmentation tokens are averaged to obtain the learnable text prompt vector VA of the abnormal sample, with a dimension of 1×d.

[0046] 4) Construct the learnable text prompt embedding feature E: Concatenate the learnable text prompt vector VN of the normal sample and the learnable text prompt vector VA of the abnormal sample in the 0th dimension to obtain the learnable text prompt embedding feature E. The dimension of the learnable text prompt embedding feature E is 2×d, and d is 768. It is used as the output of the multimodal learnable prompt module and input into the abnormal area calculation module.

[0047] The dimension of the learnable text prompt vector is 1×d, and the 0th dimension of the learnable text prompt vector is the dimension where the dimension 1 in the learnable text prompt vector is located. The learnable text prompt vector includes the learnable text prompt vector VN of the normal sample and the learnable text prompt vector VA of the abnormal sample.

[0048] The construction of the learnable visual cue vector P includes the following steps:

[0049] 1) First, determine the number of tokens n in the learnable visual cue vector P to be inserted, where n is 4;

[0050] 2) Then, according to the number of tokens n, the dimension of the learnable visual cue vector P is set to n×D, where D is 1024;

[0051] 3) Insert the learnable visual cue vector P of set dimension into the first and second stages of the visual encoding network iThe input end of the residual attention layer, in the specific implementation, i The value is 3, which can learn the insertion of the visual cue vector. The specific insertion method of the residual attention layer is as follows Figure 5 shown.

[0052] Step 2: Construct the abnormal area calculation module:

[0053] In the abnormal region calculation module, the abnormal region is obtained by calculating the cosine similarity between the visual features output by the visual encoding network and the learnable text prompt embedding features output by the multimodal learnable prompt module in the following way:

[0054] G m =softmax(F m @E T ), m=1,2,3,4

[0055] The symbol @ represents matrix multiplication, softmax() represents the activation function, and E T represents the transpose of the learnable textual hint embedding feature, F m represents the visual features obtained in the mth stage of the visual encoding network, G m Represents the visual feature F m Corresponding to the preliminary abnormal area map obtained.

[0056] The abnormal area calculation module is constructed according to the above formula, and the calculation result of the formula represents the probability value of the area being an abnormal area.

[0057] Step 3: Build a zero-shot anomaly detection network:

[0058] The constructed abnormal image detection network based on multimodal learnable cues is as follows Figure 2 As shown. The abnormal image detection network includes a visual encoding network, a multi-layer perceptron mlp, a multimodal learnable prompt module and an abnormal area calculation module; the visual encoding network is loaded with a learnable visual prompt vector in the multimodal learnable prompt module, and the output of the visual encoding network is input into the abnormal area calculation module after passing through multiple multi-layer perceptron mlp. At the same time, the learnable text prompt embedding feature is obtained through the multimodal learnable prompt module, and the learnable text prompt embedding feature is input into the abnormal area calculation module. The output of the abnormal area calculation module is successively changed by the feature dimension operation and the element addition operation to obtain the abnormal area heat map result. The training batch B of the abnormal image detection network is set to 8, the input image size H×W is 518×518, the dimension L is 1370, the dimension C is 1024, and the dimension d is 768.

[0059] Step 4: Train the constructed abnormal image detection network:

[0060] The present invention only uses the test set of the MVTec AD dataset to train the constructed zero-sample anomaly detection network, and evaluates the network performance on the test set of the VisA dataset. During the training process of the abnormal image detection network, the abnormal image detection network only optimizes and updates the parameters of the multi-layer perceptron mlp (768) network, the learnable visual prompt vector, and the learnable text prompt embedding feature, and other parameters of the network are not updated (the visual encoding network in the network uses the parameters provided by the CLIP model and is not updated and optimized). The training process uses the Adam optimizer, with a fixed learning rate of 0.001, a training batch size of 8, and an iteration number of 130 epochs.

[0061] Step 5: Calculate the abnormal area of ​​the image to be detected:

[0062] The image to be inspected and the learnable text hint embedding features are input into the optimized anomaly detection network. The abnormal area heat map is judged according to the cosine similarity between the extracted visual features and the learnable text hint embedding features. The area in the abnormal area heat map that is greater than the set threshold is the final abnormal area.

[0063] The present invention constructs a text description initialization text embedding with strong semantic universality, and constructs a learnable text prompt embedding feature without using any specific information of the source domain and the target domain, which effectively improves the generalization performance of the model in different anomaly detection scenarios. In addition, the text modality information is only expressed by the learnable text prompt embedding feature, which does not rely on any text encoding network, greatly reducing the computational complexity of the model and facilitating the improvement of anomaly detection efficiency. Experiments on the VisA dataset show that the present invention achieves a pixel-level anomaly detection accuracy of 94.3%. The anomaly detection performance significantly surpasses the classic zero-sample detection method WinCLIP.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A zero-shot anomaly detection method based on multimodal learnable cues, It is characterized in that The following steps are involved: Step S1) constructing an abnormal image detection network based on multimodal learnable cues; The multimodal learnable prompts include learnable visual prompt vectors and learnable text prompt embedding features; Step S2) training an abnormal image detection network; Step S3) inputting the image to be tested into the trained abnormal image detection network to obtain an abnormal region heat map, and obtaining the abnormal region of the image to be tested according to the abnormal region heat map; The abnormal image detection network includes a visual coding network, a multi-layer perceptron MLP, a multi-modal learnable prompt module and an abnormal area calculation module; the visual coding network is loaded with a learnable visual prompt vector in the multi-modal learnable prompt module, the output of the visual coding network is input into the abnormal area calculation module after passing through multiple multi-layer perceptrons MLP, and at the same time, a learnable text prompt embedding feature is obtained through the multi-modal learnable prompt module, and the learnable text prompt embedding feature is input into the abnormal area calculation module, and the output of the abnormal area calculation module is successively subjected to a feature dimension change operation and an element addition operation to obtain the result of the abnormal area heat map; The multimodal learnable prompt module includes a learnable text prompt embedding feature E. The steps of constructing the learnable text prompt embedding feature E are as follows: 1) Initialize state text description: First, construct the state text description of the corresponding sample according to the state of the sample; 2) Initialize state encoding features: Use the word segmentation embedding module pre-trained by the visual language large model to encode the state text descriptions of normal samples and abnormal samples respectively to obtain the state encoding features FN of normal samples and the state encoding features FA of abnormal samples; 3) Construct a learnable text prompt vector: average the state encoding feature FN of normal samples and the state encoding feature FA of abnormal samples respectively to obtain the learnable text prompt vector VN of normal samples and the learnable text prompt vector VA of abnormal samples; 4) Construct the learnable text prompt embedding feature E: Concatenate the learnable text prompt vector VN of the normal sample and the learnable text prompt vector VA of the abnormal sample in the 0th dimension to obtain the learnable text prompt embedding feature E, and use it as the output of the multimodal learnable prompt module and input it into the abnormal area calculation module; The multimodal learnable cue module further includes a learnable visual cue vector P, which is inserted into the visual encoding network. The steps of constructing the learnable visual cue vector P are as follows: 1) First, determine the number of tokens n in the learnable visual cue vector P to be inserted; 2) Then, the dimension of the learnable visual cue vector P is set to n×D according to the number of tokens n, where D represents a preset constant; 3) Insert the learnable visual cue vector P of set dimension into the first and second stages of the visual encoding network i The input of the residual attention layer.

2. A zero-shot anomaly detection method based on multimodal learnable cues according to claim 1, Features: The visual coding network includes four stages that are performed in sequence, each stage includes multiple layers of residual attention layers connected in sequence, each stage of the visual coding network outputs visual coding features, and the four stages respectively output the first visual coding feature T1, the second visual coding feature T2, the third visual coding feature T3 and the fourth visual coding feature T4 as the output of the visual coding network.

3. The zero-shot anomaly detection method based on multimodal learnable cues according to claim 2, Features: The four coded visual features output by the visual coding network are processed by respective multi-layer perceptrons (MLP) to obtain four visual features, and the four visual features are input into the abnormal area calculation module together.

4. The zero-shot anomaly detection method based on multimodal learnable cues according to claim 1, Features: In the abnormal area calculation module, the abnormal area is obtained by calculating the cosine similarity of the visual features output by the visual encoding network and the learnable text prompt embedding features output by the multimodal learnable prompt module in the following manner: G m =softmax(F m @ E T ),m=1,2,3,4 The symbol @ represents matrix multiplication, softmax() represents the activation function, and E T represents the transpose of the learnable textual hint embedding feature, F m represents the visual features obtained in the mth stage of the visual encoding network, G m Represents the visual feature F m Corresponding to the preliminary abnormal area map obtained.

5. The zero-shot anomaly detection method based on multimodal learnable cues according to claim 4, Features: The four preliminary abnormal area graphs G output by the abnormal area calculation module m After changing the feature dimension, we obtain four target abnormal area maps M. m , and then the four target abnormal area maps M m Add the corresponding elements to obtain the final abnormal area heat map.

6. A zero-shot anomaly detection method based on multimodal learnable cues according to claim 4, Features: In the step S2), a zero-sample transfer learning method is used during training, and the network parameters of the visual encoding network in the abnormal image detection network are not updated. Only the parameters of the learnable text prompt embedding feature E, the learnable visual prompt vector P and the parameters of the multi-layer perceptron mlp in the multimodal learnable prompt module are optimized and trained.

7. The zero-shot anomaly detection method based on multimodal learnable cues according to claim 4, Features: In the step S3), the image area whose pixel value is greater than a preset abnormal threshold value is taken as the abnormal area in the abnormal area heat map.

Images