A zero-shot anomaly image detection method based on dynamic learning cues

Abstract

The application discloses a zero-shot abnormal image detection method based on dynamic learning prompts. A zero-shot abnormal detection network is constructed based on a visual language large model, visual coding features of an image and text features of dynamic prompts are extracted by using a visual coding network and a text coding network, and the pre-training visual language large model is zero-shot migrated to a downstream abnormal image detection task by using language supervision of the text features. In multiple feature extraction stages of the visual coding network, cosine similarity between the visual coding features and the text features is calculated, an image abnormal area calculation module is constructed, and an abnormal area of the image to be detected is obtained. The dynamic learning prompts proposed by the application can be continuously optimized along with network gradients, the complex artificial prompt construction process is avoided, and the design efficiency of the prompt engineering is improved. In addition, the dynamic learning prompts generate corresponding prompt features for each image, and the zero-shot migration performance of the network to new objects and new environments is enhanced.

Description

Technical Field

[0001] This invention relates to an image detection method in the fields of computer vision and anomaly image detection, and particularly to a zero-shot anomaly image detection method based on dynamic learning prompts. Background Technology

[0002] Anomaly detection primarily aims to identify situations that differ from conventional patterns, and it is widely used in fields such as industrial product defect detection, video security monitoring, and medical disease diagnosis. Most current anomaly detection models rely on anomaly-free images, determining the location of abnormal regions by comparing the differences between normal and the image to be detected in a feature space. However, this contrast-based approach is closely related to object categories and detection scenarios, typically requiring the construction of specific anomaly detection models for each object category and the collection of a large number of normal samples. This makes it unsuitable for real-world production environments with diverse object types and complex, ever-changing scenarios. Zero-shot transfer learning based on cueing engineering provides a new and efficient solution for anomaly detection tasks in practical production.

[0003] However, effective prompting engineering requires extensive domain knowledge and repeated fine-tuning by relevant experts, which limits the application scope and design efficiency of prompting engineering.

[0004] In addition, zero-shot transfer tasks are highly sensitive to cue engineering. Even minor changes in cue engineering can cause huge fluctuations in network performance, further increasing the difficulty and instability of manually designing cue engineering. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by proposing a zero-shot anomaly image detection method based on dynamic learning prompts. By constructing dynamic learning prompts, the design flexibility and versatility of the prompt engineering are enhanced, alleviating the reliance on domain expert experience and reducing the complexity of manually designed prompts. Furthermore, the organic integration of dynamic, static, and state-based prompts enhances the generalization performance and robustness of the prompt engineering, and improves the accuracy of zero-shot transfer learning tasks.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] 1) Establish a zero-sample anomaly detection network;

[0008] 2) Utilize a zero-shot anomaly detection network;

[0009] 3) Input the image to be tested into the trained zero-shot anomaly detection network to obtain an anomaly region heatmap, and obtain the anomaly region based on the anomaly region heatmap.

[0010] The zero-shot anomaly detection network includes a visual encoding network, a text encoding network, a multilayer perceptron (MLP), a dynamic learning prompt generation module, and an anomaly region calculation module.

[0011] The output of the visual encoding network is fed into the anomaly region calculation module after passing through multiple multilayer perceptrons (MLPs). Simultaneously, the output of the visual encoding network is fed into the anomaly region calculation module after passing through the dynamic learning prompt generation module and the text encoding network. The output of the anomaly region calculation module is then processed by changing the feature dimension and element-wise multiplication to obtain the anomaly region heatmap.

[0012] The visual coding network comprises four stages performed sequentially. Each stage contains multiple layers of residual attention modules and outputs visual features. The four stages output a first visual feature T1, a second visual feature T2, a third visual feature T3, and a fourth visual feature T4 as the output of the visual coding network, respectively. The first stage receives the original input image, and each of the other stages receives the visual features output by the previous stage.

[0013] The four visual features output by the visual coding network are processed by their respective multilayer perceptrons (MLPs) to obtain four visual coding features, which are then input into the anomaly region calculation module.

[0014] The fourth visual feature output from the fourth stage of the visual coding network is directly input into the dynamic learning prompt generation module.

[0015] The dynamic learning prompt generation module includes a lightweight network, a summation operation, a concatenation operation, static prompts, and a state prompt. The lightweight network includes a global average pooling operation and a multilayer perceptron (MLP) performed sequentially. The fourth visual feature T4 is input into the lightweight network for processing to obtain the dynamic prompt FD. The dynamic prompt FD and each vector V in the pre-obtained static prompt are then compared. N Perform summation operations separately, and then concatenate the summation results with the pre-obtained state cues [state] to obtain the dynamic learning cues DT, which is the output of the dynamic learning cues generation module;

[0016] The static prompt contains multiple vectors V N It is obtained through random initialization and is continuously updated and optimized with the network gradient during training;

[0017] The aforementioned state prompt [state] is obtained by encoding the state through the Visual Language Contrast Model (CLIP model). The state is a string indicating whether the state is normal or not. It is represented by general text with weak domain knowledge and is divided into strings belonging to normal states and strings belonging to abnormal states.

[0018] The string "state" representing the normal state can typically be: 'flawless class', 'perfect class', or 'class without defect', which represent the flawless class, the perfect class, and the class without defects, respectively. 'class' indicates the object category in the image to be detected.

[0019] The string "state" for abnormal states can typically be: 'damaged class', 'broken class', or 'class with defect', representing the categories of damage, brokenness, and defects, respectively.

[0020] The dynamic learning prompt generation module outputs dynamic learning prompts (DT), which are then input into a text encoding network to obtain text features. The text encoding network employs a visual-language contrastive model (CLIP model).

[0021] The abnormal region calculation module obtains the region based on the cosine similarity between the input visual encoding features and text features in the following manner:

[0022] G m =softmax(F m @W T ), m=1,2,3,4

[0023] Where the symbol @ represents matrix multiplication, softmax represents the activation function, and W T F represents the transpose of text features. m G represents the visual coding feature obtained at stage m in the visual coding network. m Represents visual encoding features F m The corresponding preliminary abnormal area map.

[0024] The four preliminary anomaly region maps output by the anomaly region calculation module were reshaped by changing the feature dimensions to obtain four anomaly region maps M. m Then map the four abnormal regions M m Multiply the corresponding elements to obtain the final heatmap of the abnormal area.

[0025] In the abnormal region heatmap, the pixel regions larger than the set abnormal threshold are the final abnormal regions.

[0026] Step 2) employs zero-shot transfer learning during training. The network parameters of the visual encoding and text encoding networks in the zero-shot anomaly detection network are not updated; only the parameters of the dynamic learning prompt generation module and the multilayer perceptron (MLP) are optimized. The training process uses the Adam optimizer with a fixed learning rate of 0.001, a batch size of 8, and one epoch.

[0027] In step 3), specifically, in the abnormal region heatmap, image regions with pixel values ​​greater than a preset abnormal threshold are considered as abnormal regions.

[0028] This invention designs a dynamic learning prompt generation module that can be updated by following network gradient optimization, including dynamic prompts, static prompts, and status prompts. A zero-shot anomaly detection network is constructed based on a large visual-language model, utilizing visual encoding and text encoding networks to extract visual encoding features from images and text features from dynamic prompts. Language supervision using text features is applied to the pre-trained large visual-language model for zero-shot transfer to downstream anomaly image detection tasks. In multiple feature extraction stages of the visual encoding network, the cosine similarity between visual encoding features and text features is calculated, and an image anomaly region calculation module is constructed to obtain the anomaly regions of the image to be detected.

[0029] The beneficial effects of this invention are:

[0030] The method of this invention can automatically construct prompting projects through learning. The proposed dynamic learning prompts can be continuously optimized with the network gradient, avoiding over-reliance on expert domain knowledge and improving the efficiency and accuracy of prompting project construction.

[0031] In addition, dynamic learning prompts generate corresponding prompt features for each image, which enhances the network's zero-shot transfer performance for new objects and new environments.

[0032] Meanwhile, the designed dynamic learning prompting module can be seamlessly integrated into existing visual language models, achieving end-to-end training without human intervention, and ensuring the generalization performance and robustness of the prompting project to unknown environments. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating a zero-shot anomaly image detection method based on dynamic learning prompts according to the present invention.

[0034] Figure 2 This is a schematic diagram of the zero-shot anomaly detection network based on dynamic learning prompts of the present invention;

[0035] Figure 3 This is a schematic diagram of the dynamic learning prompt generation module of the present invention. Detailed Implementation

[0036] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0037] It should be understood that the embodiments described in this invention are exemplary, and the specific parameters used in the embodiment description are only for the purpose of describing this invention and are not intended to limit this invention.

[0038] like Figure 1 As shown, an embodiment of the zero-shot anomaly image detection based on dynamic learning prompts of the present invention is as follows:

[0039] Step 1: Build a dynamic learning prompt generation module.

[0040] Build as Figure 3 The diagram shows the dynamic learning cue generation module. Here, T4 represents visual features; FD represents dynamic cues with a dimension of B×d; DT represents dynamic learning cuees with a dimension of B×6×K×d; V N [N] represents the Nth vector of the static cue; [state] represents the state cue. mlp(768) represents a multilayer perceptron, where the number in parentheses indicates the number of nodes in that layer. The construction process of dynamic learning cues involves the following steps:

[0041] (1) Constructing dynamic cues. Dynamic cues FD are obtained through a lightweight network. This network takes visual features T4 as input, passes them through global average pooling and mlp(768) network to obtain dynamic cues FD.

[0042] (2) Construct static suggestions. Static suggestion V N The results are obtained through random initialization and continuously updated and optimized based on network gradients. During the testing phase, static hints remain constant.

[0043] (3) Constructing State Hints. State hints [state] are obtained by encoding the state using the CLIP model. The state is represented by general text with weak domain knowledge, and is divided into strings for normal states and strings for abnormal states. Among them,

[0044] The normal state contains three categories: 'flawless class', 'perfect class', and 'class without defect'. Each normal sample state cue is concatenated with the static cue and the dynamic cue to form a dynamic learning cue, resulting in a total of three dynamic learning cuees for the normal sample state cue.

[0045] Similarly, the state prompt for abnormal states includes three categories: 'damaged class', 'broken class', and 'class with defect', which are three dynamic learning prompts for abnormal sample state prompts;

[0046] (4) Construct dynamic learning prompts. This involves dynamic prompts (FD) and static prompts (V). N Based on the state cues, dynamic learning cues are constructed. First, the dynamic cues FD are compared with each vector V in the static cues. N Perform a summation operation; then concatenate the summation result with the state cues to obtain the dynamic learning cues (DT). The dimensions of DT are B×6×K×d, where 6 corresponds to 6 state cues and K corresponds to the text length in the CLIP model, which is 77.

[0047] In this embodiment, B is set to 8, d to 768, and K to 77. The number of vectors N for static cue V is set to 8. The dimensions of dynamic learning cue DT are 8×6×77×768.

[0048] Step 2: Construct the abnormal region calculation module.

[0049] Anomaly regions are obtained using the cosine similarity between visual encoding features and text features. This invention extracts visual encoding features F from four stages of the visual encoding network. m The formula m = 1, 2, 3, 4 allows for multi-level representation of abnormal regions. Simultaneously, text features W are obtained based on dynamic learning prompts and text encoding processing.

[0050] The cosine similarity between the four visual encoding features and the text features is calculated, with the visual encoding feature F... m For example, the similarity G between it and the text feature W. m The calculation formula is as follows:

[0051] G m =softmax(F m @W T ), m=1,2,3,4

[0052] Where the symbol @ represents matrix multiplication, softmax represents the activation function, and W T This represents the transpose of text features.

[0053] An abnormal region calculation module is constructed based on the above formula. The abnormal region is judged based on the cosine similarity between the visual coding feature F and the text coding feature W. The calculation result of the formula represents the probability value that the region is an abnormal region.

[0054] Step 3: Construct a zero-sample anomaly detection network.

[0055] Build as Figure 2 The zero-shot anomaly detection network is shown. A visual encoding network and a text encoding network are constructed using the visual language pre-trained model CLIP, where the visual encoding network contains 24 layers of residual attention modules. The visual encoding network is divided into four stages, labeled as stage 1, stage 2, stage 3, and stage 4, each containing 6 layers of residual attention modules. The specific structure of the residual attention modules can be found in the CLIP model and will not be elaborated further in this invention.

[0056] The symbols in the network are represented as follows: B represents the number of images input into the network in a batch; H and W represent the height and width of the image; T1, T2, T3, and T4 represent the visual features of the first, second, third, and fourth stages, respectively, with each feature having a dimension of B×L×C; mlp represents a multilayer perceptron, and the number in parentheses represents the number of nodes in the multilayer perceptron; F1, F2, F3, and F4 represent the visual encoding features after the mlp operation, with each feature having a dimension of B×L×d; M1, M2, M3, and M4 represent four abnormal region maps, each with a dimension of B×H×W, corresponding to the visual encoding features F1, F2, F3, and F4, respectively; L, C, and d all represent the dimension of a feature; the reshape operation represents changing the dimension and shape of the feature.

[0057] The anomaly detection network takes the image to be detected I as input and obtains visual features T1, T2, T3, and T4 after the first, second, third, and fourth stages of visual encoding, respectively. The four visual features are then processed by mlp(768) to obtain encoded features F1, F2, F3, and F4.

[0058] Simultaneously, visual feature T4 is processed by the dynamic learning cue generation module to obtain dynamic learning cue DT. Dynamic learning cue DT is then processed by a text encoding network to obtain text encoding feature W.

[0059] Text-encoded features W and visual-encoded features F1, F2, F3, and F4 are processed by the anomaly region calculation module to calculate cosine similarity. After a reshape operation, four anomaly region maps M1, M2, M3, and M4 are obtained. The dimensions of all four anomaly region maps are B×H×W, consistent with the spatial dimensions of the input image. The four anomaly region maps are multiplied element-wise to obtain the final anomaly region heatmap of the image to be inspected.

[0060] In this embodiment, the input image size is H×W = 518×518, and the batch size is B = 8. Dimension L is 1370, dimension C is 1024, and dimension d is 768. Both the visual encoding network and the text encoding network use pre-trained models from the CLIP open-source platform.

[0061] The text encoding network takes dynamic learning prompts (DT) as input and outputs text features W with dimensions B×2×d. This text feature W consists of two parts: the first part is the text encoding feature for normal sample states; the second part is the text encoding feature for abnormal sample states, both with dimensions B×1×d. The text encoding features for normal and abnormal sample states are concatenated to obtain the text feature W with dimensions B×2×d.

[0062] Step 4: Training the constructed anomaly detection network. To verify the performance of the proposed method, this invention uses both the MVTec AD dataset and the VisA dataset. The constructed zero-shot anomaly detection network is trained using the test set portion of the MVTec AD dataset, and the proposed network is evaluated using the test set portion of the VisA dataset. This invention employs a zero-shot transfer learning method. During the training phase, the network parameters of the visual encoding network and the text encoding network are fixed (both the visual encoding network and the text encoding network use the parameters provided by the CLIP model and do not need to be updated), and only the relevant parameters of the dynamic learning prompt generation module and the parameters of the mlp(768) network are optimized and updated. The training process uses the Adam optimizer, with a fixed learning rate of 0.001, a training batch size of 8, and 1 epoch.

[0063] Step 5: After training, the anomaly detection network directly predicts the anomaly region heatmap of the input image. In practical applications, an anomaly threshold is set, and the regions in the anomaly region heatmap predicted by the network that exceed the set threshold are the final anomaly regions.

[0064] Compared with existing technologies, this invention constructs a dynamic learning prompt generation module that automatically learns dynamic prompts using a lightweight neural network and seamlessly integrates with existing large-scale visual language models. This improves the design efficiency of prompt engineering in zero-shot learning tasks and avoids complex manual design and tuning. Furthermore, the dynamic learning prompts are updated and optimized based on input image features, exhibiting strong adaptability to unknown objects and environments, enhancing the generalization performance and robustness of prompt engineering, and further improving the accuracy of zero-shot transfer learning.

[0065] Testing on the VisA dataset showed that this invention achieved a pixel-level anomaly detection accuracy of 95.3%. This anomaly detection performance significantly outperforms the classic zero-shot detection method, WinCLIP.

[0066] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited to the above embodiment. Any equivalent substitutions or changes made by those skilled in the art within the technical scope described in the present invention, based on the technical solution and inventive concept of the present invention, should be included within the scope of protection of the present invention.

Claims

1. A zero-shot anomaly image detection method based on dynamic learning prompts, characterized in that: 1) Establish a zero-sample anomaly detection network; 2) Utilize a zero-shot anomaly detection network; 3) Input the image to be tested into the trained zero-shot anomaly detection network to obtain an anomaly region heatmap, and obtain the anomaly region based on the anomaly region heatmap; The zero-shot anomaly detection network includes a visual encoding network, a text encoding network, a multilayer perceptron (MLP), a dynamic learning prompt generation module, and an anomaly region calculation module. The output of the visual encoding network is fed into the anomaly region calculation module after passing through multiple multilayer perceptrons (MLPs). Simultaneously, the output of the visual encoding network is fed into the anomaly region calculation module after passing through the dynamic learning prompt generation module and the text encoding network. The output of the anomaly region calculation module is then processed by the feature dimension change operation and the element-wise multiplication operation to obtain the anomaly region heatmap. The visual coding network comprises four stages performed sequentially. Each stage contains multiple layers of residual attention modules and outputs visual features. The four stages output the first visual feature T1, the second visual feature T2, the third visual feature T3, and the fourth visual feature T4 as the output of the visual coding network, respectively. The dynamic learning prompt generation module includes a lightweight network, a summation operation, a concatenation operation, static prompts, and a state prompt. The lightweight network includes a global average pooling operation and a multilayer perceptron (MLP) performed sequentially. The fourth visual feature T4 is input into the lightweight network for processing to obtain the dynamic prompt FD. The dynamic prompt FD and each vector V in the static prompt are then processed. N Perform the summation operation separately, and then concatenate the summation results with the state prompt [state] to obtain the dynamic learning prompt DT, which is the output of the dynamic learning prompt generation module; The static prompt contains multiple vectors V N It is obtained through random initialization; The aforementioned state prompt [state] is obtained by encoding the state through a visual language contrast model. The state is a string indicating whether the state is normal or not, and is divided into strings belonging to normal states and strings belonging to abnormal states.

2. The zero-shot anomaly image detection method based on dynamic learning prompts according to claim 1, characterized in that: The four visual features output by the visual coding network are processed by their respective multilayer perceptrons (MLPs) to obtain four visual coding features, which are then input into the anomaly region calculation module. The fourth visual feature output by the fourth stage of the visual coding network is directly input into the dynamic learning prompt generation module.

3. The zero-shot anomaly image detection method based on dynamic learning prompts according to claim 1, characterized in that: The dynamic learning prompt generation module outputs dynamic learning prompts (DT), which are then input into a text encoding network to obtain text features.

4. A zero-shot anomaly image detection method based on dynamic learning prompts according to claim 1 or 3, characterized in that: The abnormal region calculation module obtains the region based on the cosine similarity between the input visual encoding features and text features in the following manner: G m =softmax(F m @ W T ), m=1,2,3,4 Where the symbol @ represents matrix multiplication, softmax represents the activation function, and W T F represents the transpose of text features. m G represents the visual coding feature obtained at stage m in the visual coding network. m Represents visual encoding features F m The corresponding preliminary abnormal area map.

5. The zero-shot anomaly image detection method based on dynamic learning prompts according to claim 4, characterized in that: The four preliminary anomaly region maps output by the anomaly region calculation module were reshaped by changing the feature dimensions to obtain four anomaly region maps M. m Then map the four abnormal regions M m Multiply the corresponding elements to obtain the final heatmap of the abnormal area.

6. The zero-shot anomaly image detection method based on dynamic learning prompts according to claim 1, characterized in that: In step 2), zero-shot transfer learning is used during training. The network parameters of the visual encoding network and the text encoding network in the zero-shot anomaly detection network are not updated. Only the parameters of the dynamic learning prompt generation module and the parameters of the multilayer perceptron (MLP) are optimized and trained.

7. The zero-shot anomaly image detection method based on dynamic learning prompts according to claim 1, characterized in that: In step 3), specifically, in the abnormal region heatmap, image regions with pixel values ​​greater than a preset abnormal threshold are considered as abnormal regions.

Images