Adaptive zero-shot anomaly detection method based on visual language model

Abstract

This invention relates to the field of artificial intelligence, and in particular provides an adaptive zero-shot anomaly detection method based on a visual language model. The method includes: acquiring an image to be detected; extracting visual features from the image to be detected; adaptively constructing a unique text prompt for the image to be detected based on the visual features and guided by visual information; inputting the visual features and the unique text prompt into a pre-trained visual language model for cross-modal interaction and anomaly perception; and obtaining the anomaly detection result of the image to be detected based on the output of the visual language model. This invention, by calculating the similarity between local visual features and text prompts, guides more visual features from regions rich in anomaly information to flow into the prompt embedding, reducing the interference of semantically irrelevant visual information on prompt generation, and effectively improving the model's ability to focus on fine-grained anomaly regions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to an adaptive zero-shot anomaly detection method based on a visual language model. Background Technology

[0002] Image anomaly detection aims to identify samples with defects in a target set and accurately locate the defective regions. This technology is of paramount importance in fields such as industrial product quality control and medical diagnosis. Traditionally, anomaly detection methods rely primarily on unsupervised learning paradigms or few-shot learning, both of which require training with normal samples or as references. While these traditional methods can achieve good results in certain specific scenarios, unsupervised learning methods often fall short when dealing with privacy-protected data or novel datasets. To overcome this limitation, Zero-Shot Anomaly Detection (ZSAD) has emerged. The core objective of ZSAD is to detect anomalous patterns in unseen classes without having encountered any samples of the target class.

[0003] In recent years, Vision-Language Models (VLMs) have achieved remarkable results in downstream tasks in the field of vision, attracting more and more researchers to try to use VLMs like CLIP to solve the zero-shot anomaly detection problem. The basic working principle of VLMs is to determine whether an image contains an anomaly by calculating the similarity between text cues and image features. Specifically, text cues are used to represent normal or abnormal states, while image features are used to describe the visual content of the image to be detected. By comparing the similarity between these two, the model can assess whether an anomaly exists in the image.

[0004] Currently, some research works, such as WinCLIP, AnVoL, APRIL-GAN, and CLIP-AD, have attempted to use hand-designed static general text as cue words, achieving significant performance improvements in zero-shot anomaly detection tasks. However, these methods based on static cue words have two main drawbacks. First, static cue words lack flexibility and adaptability, making it difficult to dynamically adjust according to different image content. Consequently, they perform poorly when faced with data samples containing more complex anomaly shapes, hindering more refined and accurate anomaly detection. Second, most of these methods lack dynamic inflow of visual information, failing to fully utilize the rich visual details in the input image to guide the generation of text cue words. Since the information content of text cue words is limited, while the input image contains much richer visual information, this lack of visual information interaction limits the model's attention to fine-grained visual features in the image, thus affecting the model's ability to detect complex scenes.

[0005] In summary, current research on zero-shot anomaly detection (ZSAD) tasks based on visual language models (VLMs) faces the following main challenges: Current ZSAD methods largely rely on manually designed static text prompts. While these prompts can be effective in specific scenarios, their design process is often time-consuming and labor-intensive. More importantly, this static approach lacks flexibility and struggles to adapt to detection tasks in other scenarios, thus limiting the model's generalization performance. Therefore, how to adaptively and dynamically generate appropriate text prompts based on different testing scenarios has become a pressing issue.

[0006] Existing methods process visual and textual features relatively independently, lacking effective cross-modal information interaction. In fact, test images themselves contain rich visual anomaly information that has not yet been fully explored and utilized. If visual information in images can be used to guide the generation of textual prompts, it will help the model better focus on anomaly-related regions, thereby improving detection performance.

[0007] To address the shortcomings of existing technologies, this invention provides an adaptive zero-shot anomaly detection method based on a visual language model. Summary of the Invention

[0008] To achieve the above objectives, the present invention adopts the following technical solution: One aspect of the present invention provides an adaptive zero-shot anomaly detection method based on a visual language model, comprising the following steps: Acquire the image to be detected; Extract the visual features of the image to be detected; Based on the aforementioned visual features, a unique text prompt is constructed for the image to be detected using visual information guidance. The visual features and the specific text prompts are input into a pre-trained visual language model for cross-modal interaction and anomaly detection. Based on the output of the visual language model, the anomaly detection result of the image to be detected is obtained.

[0009] In one optional implementation, a zero-shot anomaly detection model is constructed based on the extracted visual features of the image to be detected; the model includes a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module, and a region context prompt learning module.

[0010] In one optional implementation, the general text prompt learner is trained using the zero-shot anomaly detection model to obtain a general text prompt template containing generalized anomaly pattern knowledge; specifically including: Given an input image instance in the auxiliary dataset, it is input into the pre-trained CLIP image encoder to obtain the global image features and local patch features of the input image instance; Initialize the text prompt template and combine it with the status template words to obtain the abnormal text prompt template and the normal text prompt template; input the abnormal text prompt template and the normal text prompt template into the CLIP text encoder respectively to obtain the abnormal text features and the normal text features; Calculate the cosine similarity between the global image features, local patch features, and the abnormal text features and normal text features respectively to obtain the anomaly score and pixel-level anomaly region mask; By optimizing global and local properties through cross-entropy loss, Focal loss, and Dice loss functions, the learnable text prompter is continuously optimized, ultimately resulting in a set of general prompt templates and general text prompts for abnormal patterns.

[0011] In one optional implementation, based on the component context prompting learning module and the region context prompting learning module, visual information is embedded into text prompts to generate visually guided dynamic text prompts; specifically including: Given an input image instance, the pre-trained CLIP image encoder is used to obtain global image features, which are then fed into the composition context cue learning module. The composition context cue learning module optimizes the lightweight weight prediction network and the common bias basis vector, uses the global image features to predict the weights of the common bias basis vector, and linearly combines the common bias basis vector to obtain the final global visual bias. For an input image instance, multi-stage local patch features obtained using VV attention from the CLIP image encoder are extracted. The features from each stage are fused to make shallow visual information and deep visual information complementary, resulting in local visual features. The local visual features are then input into the region context cue learning module.

[0012] In one optional implementation, the local visual features are remapped into conditional representations representing abnormal patterns and conditional representations representing normal patterns using a lightweight mapping network; the semantic affinity between the abnormal conditional representation P and the general abnormal text prompt, and between the normal conditional representation and the general normal text prompt, are calculated to obtain a semantic affinity matrix. The semantic affinity matrix is ​​summed along the dimension represented by L to obtain the semantic similarity of each patch feature to the overall general text prompt; the summed semantic affinity matrix is ​​multiplied by the corresponding conditional representation to obtain the text prompt bias containing local visual information. The global visual bias and the local visual bias are fused to obtain the final normal mode visual bias and the abnormal mode visual bias. The final normal mode visual bias and the abnormal mode visual bias are then embedded into the obtained general text prompts to obtain dynamic text prompts guided by visual information.

[0013] In one optional implementation, the cosine similarity between the global image features, local patch features, and the text features corresponding to the dynamic text prompt is calculated to obtain an anomaly score and a pixel-level anomaly region mask; and the global and local properties are optimized through cross-entropy loss, Focal loss, and Dice loss functions to finally obtain the visual guidance adaptive text prompt for the input instance.

[0014] In one optional implementation, the auxiliary dataset is a dataset related to the object detection task but without object dataset annotation information, used to train the general text prompt learner to learn generalized normal and abnormal patterns; The state template words are descriptive terms used to distinguish between normal and abnormal states; The general text prompt continuously adjusts the embedded representation of its internal token through an optimization process; The general text prompts V and W are learnable, and their internal token embedding representations are continuously adjusted through an optimization process.

[0015] Another aspect of the present invention provides an adaptive zero-shot anomaly detection system based on a visual language model, comprising: The model building unit is used to build an adaptive zero-shot anomaly detection model based on a visual language model. The model includes a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module, and a region context prompt learning module. A general text prompt learning unit is used to execute the general text prompt learner to obtain a general text prompt template; The context prompt learning and visual information embedding unit is used to train the constituent context prompt learning module and the region context prompt learning module to embed visual information into text prompts and generate dynamic text prompts. The anomaly detection unit is used to input the dataset to be tested into the trained model to perform anomaly detection and output anomaly scores and pixel-level anomaly region masks. The storage unit is used to store model parameters, intermediate results, the trained model, and detection results during the training process.

[0016] In another aspect, the present invention provides an electronic device comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are executed by the processor to implement the aforementioned adaptive zero-shot anomaly detection method based on a visual language model.

[0017] In another aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by at least one processor, implement the aforementioned adaptive zero-shot anomaly detection method based on a visual language model.

[0018] Effects of the invention: Current zero-shot anomaly detection methods based on visual language models mostly rely on manually designed text templates as prompts, which consumes a lot of manpower and time. Furthermore, such static text prompts are difficult to adapt to the diverse anomaly patterns in the real world, resulting in poor model generalization. This invention utilizes the visual information of the input instance to guide the generation of text prompts, adaptively customizing prompts for each instance to be detected based on the visual features of the input image, achieving interaction between the visual and text modalities, and enhancing the model's generalization ability. By embedding global visual features into the cue text space, this method significantly enhances the model's ability to perceive global anomalies in images. Simultaneously, it innovatively learns a set of common basis vectors and predicts the weights of each basis vector based on the global visual features of different instances, then linearly combines these basis vectors. This process effectively reduces bias caused by differences in image semantics and avoids the problem of over-focusing on image semantics that may occur when directly using global visual features to learn cue vectors. By calculating the similarity between local visual features and text prompts, more visual features from regions rich in anomalous information are guided into the prompt embedding, reducing the interference of semantically irrelevant visual information on prompt generation and effectively improving the model's ability to focus on fine-grained anomalous regions. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an adaptive zero-shot anomaly detection method based on a visual language model provided in Embodiment 1 of the present invention; Figure 2 This is a framework diagram of an adaptive zero-shot anomaly detection system based on a visual language model provided in Embodiment 3 of the present invention; Figure 3This is a block diagram of the electronic device provided in Embodiment 4 of the present invention; Figure 4 This is a block diagram of a computer-readable storage medium provided in Embodiment 4 of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0021] Hereinafter, the terms "first," "second," etc., are used for descriptive convenience only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0022] In this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed mechanical connection, a detachable mechanical connection, or an integral part; or, "connection" can be a direct connection or an indirect connection through an intermediate medium. Furthermore, unless otherwise explicitly specified and limited, the term "coupling" should be interpreted broadly. For example, "coupling" can be a direct electrical connection, such as physical contact and electrical conduction between two components; it can also be understood as an electrical connection between different components in a circuit structure through physical lines capable of transmitting electrical signals, such as copper foil or wires on a printed circuit board (PCB), to transmit electrical signals; or, "coupling" can be an indirect electrical connection between two components through an intermediate medium; or, "coupling" can be an electrical connection between two components in a non-contact manner, such as an electrical connection between two components using capacitive coupling to transmit electrical signals.

[0023] In this embodiment of the invention, directional terms such as "up," "down," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and can change accordingly depending on the orientation of the components in the accompanying drawings.

[0024] Example 1: like Figure 1 As shown, this embodiment of the invention provides an adaptive zero-shot anomaly detection method based on a visual language model, comprising the following steps: Step S100: Acquire the image to be detected; Step S200: Extract the visual features of the image to be detected; Step S300: Based on the visual features, and guided by visual information, adaptively construct a unique text prompt for the image to be detected; Step S400: Input the visual features and the specific text prompts into a pre-trained visual language model to perform cross-modal interaction and anomaly perception; Step S500: Based on the output of the visual language model, obtain the anomaly detection result of the image to be detected.

[0025] In the above embodiments, the model is trained using an auxiliary dataset without needing to access the target dataset, enabling efficient detection even in cold-start scenarios such as data privacy concerns. This approach can be widely applied to various scenarios, including industrial product quality control and assisted medical testing. Existing zero-shot anomaly detection methods are typically based on static text prompts, which struggle to handle complex and varied anomaly scenarios. Furthermore, due to the lack of visual modal guidance in text prompts, the model often focuses more on the semantics of foreground objects in the image than on the anomalies themselves. This method, through several innovative modules, embeds rich visual information into the text space, adaptively constructing unique text prompts for each input instance. This achieves interaction between the image and text modalities, improving the model's ability to perceive anomaly regions.

[0026] Example 2: like Figure 1 As shown, based on Example 1, the steps provided in this embodiment of the invention construct an adaptive zero-shot anomaly detection model based on a visual language model; the model includes a pre-trained visual language model CLIP, a general text prompt learner, a composition context prompt learning module, and a region context prompt learning module.

[0027] Specifically, the general text prompt learner is trained to obtain a general text prompt template containing knowledge of generalized anomaly patterns; this includes: Given an input image instance x in the auxiliary dataset, it is input into the pre-trained CLIP image encoder to obtain the global image features FI and local patch features FiP of the input image instance x, where i represents the i-th stage of the CLIP image encoder. Initialize the text prompt template and combine it with the status template words to obtain the abnormal text prompt template and the normal text prompt template; input the abnormal text prompt template and the normal text prompt template into the CLIP text encoder respectively to obtain the abnormal text feature FA and the normal text feature FN; Calculate the cosine similarity between the global image feature FI, the local patch feature FiP, the abnormal text feature FA, and the normal text feature FN, respectively, to obtain the anomaly score and the pixel-level anomaly region mask; By optimizing global and local properties through cross-entropy loss, Focal loss, and Dice loss functions, the learnable text prompter is continuously optimized, ultimately resulting in a set of general prompt templates V and W; where V represents the general text prompt for normal mode and W represents the general text prompt for abnormal mode.

[0028] Specifically, the component context cue learning module and the region context cue learning module are trained to embed visual information into text cuees, thereby generating visually guided dynamic text cuees; this includes: Given an input image instance x, the global image feature FI is obtained using the pre-trained CLIP image encoder and fed as input into the composition context cue learning module. The composition context cue learning module optimizes the lightweight weight prediction network and the common bias basis vector, uses the global image feature FI to predict the weights of the common bias basis vector, and linearly combines the common bias basis vector to obtain the final global visual bias BNI and BAI. For an input image instance x, multi-stage local patch features FiP obtained using VV attention of the CLIP image encoder are extracted. The features of each stage are fused to make shallow visual information and deep visual information complementary, resulting in local visual features FP. The local visual features FP are then input into the region context cue learning module.

[0029] Specifically, the local visual features FP are remapped into conditional representations PA' representing the abnormal mode and PN' representing the normal mode through a lightweight mapping network; the semantic affinity between the abnormal conditional representation PA' and the general abnormal text prompt W, and between the normal conditional representation PN' and the general normal text prompt V are calculated respectively, to obtain the semantic affinity matrix AN / A∈RL×P, where L represents the length of the general text prompt and P represents the number of patch regions in the local visual features; The semantic affinity matrix is ​​summed along the dimension represented by L to obtain the semantic similarity of each patch feature to the overall general text prompt; the summed semantic affinity matrix is ​​multiplied by the corresponding conditional representations PA' and PN' to obtain the text prompt biases BAP and BNP containing local visual information. The global visual biases BNI and BAI are fused with the local visual biases BNP and BAP respectively to obtain the final normal mode visual bias and abnormal mode visual bias. The final normal mode visual bias and abnormal mode visual bias are then embedded into the general text prompts V and W obtained in step two to obtain the visual information-guided dynamic text prompts Vx and Wx.

[0030] Specifically, the cosine similarity between the global image features FI, the local patch features FiP, and the text features corresponding to the dynamic text prompts Vx and Wx is calculated to obtain the anomaly score and pixel-level anomaly region mask; and the global and local properties are optimized through cross-entropy loss, Focal loss, and Dice loss functions, and finally, for the input instance x, its visual guidance adaptive text prompt is obtained.

[0031] Specifically, the auxiliary dataset is a dataset related to the object detection task but without object dataset annotation information, used to train the general text prompt learner to learn generalized normal and abnormal patterns; The state template words are descriptive terms used to distinguish between normal and abnormal states; The general text prompts V and W are learnable, and their internal token embedding representations are continuously adjusted through an optimization process.

[0032] In the above embodiments, zero-shot learning is a learning paradigm that does not directly use target category labeled data during training. Unlike traditional supervised learning, which relies on a large amount of precisely labeled data for the target category, the goal of zero-shot learning is to enable the model to identify and classify new categories that have never been seen during the training phase, i.e., "zero-shot" categories. The core of this learning approach lies in utilizing existing knowledge (such as auxiliary information, semantic descriptions, etc.) to infer and identify unseen categories, thereby solving the problems of insufficient labeled data or the high cost of obtaining labeled data. Zero-shot learning is particularly important in the field of anomaly detection. Typically, labeled data for anomalous images is very scarce because anomalous events themselves are rare, and obtaining labeled data may require specialized knowledge or be very costly. Zero-shot learning enables the model to identify anomalous images without direct labels. This method not only reduces the dependence on labeled data but also improves the model's generalization ability, enabling it to adapt to various unseen anomalous situations, thus demonstrating strong application potential in anomaly detection tasks.

[0033] Visual language models are artificial intelligence models that combine visual and linguistic modalities. They process image and text data to achieve semantic understanding and description of visual information. Through cross-modal learning, they align and fuse global visual features of an image with linguistic descriptions, thereby understanding the semantic information of the image content. In zero-shot anomaly detection tasks, the core role of visual language models is to utilize the visual features of an image and its associated linguistic descriptions to identify and locate anomalous regions within the image, even if these anomaly categories were not directly labeled during training.

[0034] Visual information guidance: The text modality and visual modality of visual language models are generally independent. For downstream tasks such as anomaly detection, the common practice is to manually design a set of task-related prompts. This approach neglects the utilization of visual information. Visual information guidance enhances the model's ability to understand images by embedding rich visual features into the text space, guiding text prompts to align with visual features.

[0035] The detection method includes the following steps: Step 1: Construct an adaptive zero-shot anomaly detection model based on a visual language model; the model includes, in sequence, a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module that embeds global image features, and a regional context prompt learning module that guides the embedding of visual information of anomaly regions.

[0036] Step Two: Train a general text prompt learner to obtain a naive prompt template containing knowledge of generalized anomaly patterns. The specific process includes: 1. Given an input image instance x from an auxiliary dataset, input it into a pre-trained CLIP image encoder to obtain the global image features for each image. , and local patch features , where i represents the i-th stage of the image encoder from which the local patch feature comes.

[0037] 2. Initialize the text prompt template and combine it with the status template words to obtain the abnormal / normal text prompt templates. Then, input the text prompts into the CLIP text encoder to obtain the abnormal text features. Features of normal text .

[0038] 3. Calculate global image features separately. , and local patch features The cosine similarity between the feature and the text feature yields the anomaly score and the pixel-level anomaly region mask.

[0039] 4. Global and local properties are optimized using cross-entropy loss, Focal loss, and Dice loss functions. The learnable text prompter is continuously optimized, ultimately resulting in a set of general prompt templates V and W. V represents the general text prompt for normal modes, and W represents the general text prompt for abnormal modes.

[0040]

[0041]

[0042] Step 3: Train the component context cue learning module and the region context cue learning module to embed visual information into text cue. The specific process includes: 1. Given an input image instance x, obtain global image features using a pre-trained CLIP image encoder. This information is then fed as input into the combined contextual cue learning module. In this module, the lightweight weight prediction network and common bias basis vectors are optimized to avoid excessive penetration of foreground object semantic information into the text space when utilizing global image features. This module utilizes... Predict the weights of the common basis vectors and perform a linear combination of the basis vectors to obtain the final global visual bias. and .

[0043] 2. For input image instance x, extract multi-stage local patch features obtained using VV attention from the CLIP image encoder. By fusing features from each stage, shallow visual information and deep visual information complement each other, resulting in local visual features. Local visual features are input into the region contextual cue learning module to guide the generation of text cuees using fine-grained visual features.

[0044] 3. Remap local visual features into conditional representations of abnormal and normal patterns using a lightweight mapping network. and Calculate the abnormal conditions separately. Compared to general exception text prompts and normal condition representations The semantic affinity matrix is ​​obtained by comparing the semantic affinity between the text and the general normal text prompt V. , where L represents the length of the general text prompt and P represents the number of patch regions in the local visual features. The {i,j}th element in the affinity matrix represents the semantic similarity between the token of the i-th general text prompt and the j-th local patch feature.

[0045] 4. Summing the semantic affinity matrix along the dimension represented by L yields the semantic similarity of each patch feature to the overall general text prompt. Higher similarity indicates a stronger correlation between the visual features of the patch region and the current abnormal / normal general text prompt. Subsequently, the summed semantic affinity matrix is ​​compared with the conditional representation. and Corresponding multiplication yields the text cue bias containing local visual information. and This allows more local visual information from the anomaly-related area to flow into the anomaly text prompt, while suppressing the influence of visual information from the anomaly-irrelevant area on the anomaly text prompt.

[0046] 5. Fuse the global visual bias with the local visual bias, taking into account both global and local visual information, to obtain the final visual bias. Embed the visual bias into the general text prompts V and W obtained in step two to obtain dynamic text prompts guided by visual information.

[0047] 6. Calculate global image features , and local patch features The cosine similarity between the input and dynamic text features yields anomaly scores and pixel-level anomaly region masks. Global and local optimizations are performed using cross-entropy loss, Focal loss, and Dice loss functions. Finally, for the input instance x, its visually guided adaptive text prompt can be represented as:

[0048]

[0049] Step 4: Input the dataset to be tested into the trained adaptive zero-shot anomaly detection model based on a visual language model to perform anomaly detection, and obtain anomaly scores and pixel-level anomaly region masks. The specific process is as follows: 1. For each input image instance x, the model calculates based on its global image features. , and local patch features The text cues V and W are adaptively optimized to obtain customized text cues V(x) and W(x) containing visual information, which are then fed into the CLIP text encoder to obtain dynamic text features. and .

[0050] 2. Input the manually designed static text prompts into the CLIP text encoder to obtain static text features. and The dynamic and static text features are fused and cosine similarity is calculated with the global image features and local patch features of the input instance to finally obtain the anomaly score and pixel-level anomaly region mask.

[0051] The Combined Contextual Cue Learning Module (CCP) and the Region Contextual Cue Learning Module (RCP) are the two core modules of this invention. 1) The Combined Contextual Cue Learning Module maps global image features to contextual biases, preventing overfitting to the entire image through a set of basis vectors shared across images. 2) The Region Contextual Cue Learning Module learns contextual biases based on local visual features to guide the model to focus on regions relevant to the cues.

[0052] The effectiveness and superiority of the adaptive zero-shot anomaly detection method based on a visual language model, as presented in this invention, are verified through extensive experiments. Experiments were conducted on popular industrial anomaly detection datasets (MVTec-AD, VisA, MPDD, BTAD) and medical anomaly detection datasets (HeadCT, BrainMRI, Br35H, ISIC, ColonDB, ClinicDB, Kvasir, Endo). Specifically, the model was first trained using MVTec-AD as an auxiliary dataset and then directly tested with zero shots on other industrial datasets. For the MVTec-AD dataset, the VisA dataset was used as an auxiliary dataset during training. Furthermore, to verify the cross-domain zero-shot generalization capability of this invention, the model trained on the industrial dataset MVTec-AD was directly tested on medical datasets.

[0053] This invention evaluates model performance from both image-level and pixel-level metrics. Image-level metrics are evaluated using the Area Under the Receiver Operating Feature Curve (AUROC) and Average Precision (AP), while pixel-level metrics are evaluated using the Area Under the Receiver Operating Feature Curve and Region Overlap Score (PRO). The invention is compared with mainstream zero-shot anomaly detection methods in the field: WinCLIP, APRIL-GAN, AnomalyCLIP, and AdaCLIP. Specific experimental results are shown in Tables 1 and 2. As can be seen from the tables, the performance of this invention surpasses current methods in both image-level and pixel-level metrics, demonstrating the superiority of this invention.

[0054] Table 1. Experimental results of various methods on the industrial anomaly detection dataset.

[0055] Table 2. Experimental results of various methods on the medical anomaly detection dataset.

[0056] Furthermore, to more intuitively compare the anomaly localization results of this invention with other methods, visualization experiments were conducted on representative industrial / medical datasets MVTec-AD, VisA, ClinicDB, ColonDB, and ISIC. It can be seen that this invention has more accurate anomaly segmentation capabilities, detecting local anomalies while reducing false positive predictions for regions unrelated to the anomalies.

[0057] Example 3: like Figure 2 As shown in the embodiments, based on the present invention, an adaptive zero-shot anomaly detection system based on a visual language model is provided, comprising: The model building unit is used to build an adaptive zero-shot anomaly detection model based on a visual language model. The model includes a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module, and a region context prompt learning module. A general text prompt learning unit is used to execute the general text prompt learner to obtain general text prompt templates V and W; The context cue learning and visual information embedding unit is used to train the constituent context cue learning module and the region context cue learning module, embedding visual information into text cue to generate dynamic text cue Vx and Wx; The anomaly detection unit is used to input the dataset to be tested into the trained model to perform anomaly detection and output anomaly scores and pixel-level anomaly region masks. The storage unit is used to store model parameters, intermediate results, the trained model, and detection results during the training process.

[0058] Example 4: Figure 3 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present invention is shown.

[0059] The electronic device may include a central processing unit / microprocessor / main control chip, etc. 4; and a storage medium 5, coupled to the central processing unit / microprocessor / main control chip, etc. 4, and storing computer-executable instructions therein for performing the steps of various methods of embodiments of the present invention when executed by the processor.

[0060] The central processing unit / microprocessor / main control chip, etc., can include, but are not limited to, one or more processors or microprocessors.

[0061] Storage medium 5 may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, computer storage media (e.g., hard disk, floppy disk, solid-state drive, removable disk, CD-ROM, DVD-ROM, Blu-ray disc, etc.).

[0062] In addition, the electronic device may also include (but is not limited to) a data bus 6, an input / output bus / external bus / device bus 7, a display 8, and input / output devices 9 (e.g., keyboard, mouse, speaker, etc.).

[0063] The central processing unit / microprocessor / main control chip, etc. 4 can communicate with external devices (8, 9, etc.) via I / O bus 7 through wired or wireless network (not shown).

[0064] The storage medium 5 may also store at least one computer-executable instruction for performing the steps of various functions and / or methods in the embodiments described herein when the central processing unit / microprocessor / main control chip, etc., 4 is running.

[0065] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0066] Figure 4 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown.

[0067] like Figure 4 As shown, the non-transitory computer-readable storage medium 11 stores instructions, such as computer-readable instructions 10. When the computer-readable instructions 10 are executed by a processor, the various methods described above can be performed. The non-transitory computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-transitory non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, the non-transitory computer-readable storage medium 11 can be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions 10 stored on the computer-readable storage medium 11, the various methods described above can be performed.

[0068] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0069] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0071] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods of the various embodiments of this invention through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive zero-shot anomaly detection method based on a visual language model, characterized in that, Includes the following steps: Acquire the image to be detected; Extract the visual features of the image to be detected; Based on the aforementioned visual features, a unique text prompt is constructed for the image to be detected using visual information guidance. The visual features and the specific text prompts are input into a pre-trained visual language model for cross-modal interaction and anomaly detection. Based on the output of the visual language model, the anomaly detection result of the image to be detected is obtained.

2. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 1, characterized in that, A zero-shot anomaly detection model is constructed based on the extracted visual features of the image to be detected; the model includes a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module, and a region context prompt learning module.

3. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 2, characterized in that, The general text prompt learner is trained using the zero-shot anomaly detection model to obtain a general text prompt template containing knowledge of generalized anomaly patterns; specifically including: Given an input image instance in the auxiliary dataset, it is input into the pre-trained CLIP image encoder to obtain the global image features and local patch features of the input image instance; Initialize the text prompt template and combine it with the status template words to obtain the abnormal text prompt template and the normal text prompt template; input the abnormal text prompt template and the normal text prompt template into the CLIP text encoder respectively to obtain the abnormal text features and the normal text features; Calculate the cosine similarity between the global image features, local patch features, and the abnormal text features and normal text features respectively to obtain the anomaly score and pixel-level anomaly region mask; By optimizing global and local properties through cross-entropy loss, Focal loss, and Dice loss functions, the learnable text prompter is continuously optimized, ultimately resulting in a set of general prompt templates and general text prompts for abnormal patterns.

4. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 3, characterized in that, Based on the aforementioned component context prompting learning module and region context prompting learning module, visual information is embedded into text prompts to generate visually guided dynamic text prompts. Specifically, it includes: Given an input image instance, the pre-trained CLIP image encoder is used to obtain global image features, which are then fed into the composition context cue learning module. The composition context cue learning module optimizes the lightweight weight prediction network and the common bias basis vector, uses the global image features to predict the weights of the common bias basis vector, and linearly combines the common bias basis vector to obtain the final global visual bias. For an input image instance, multi-stage local patch features obtained using VV attention from the CLIP image encoder are extracted. The features from each stage are fused to make shallow visual information and deep visual information complementary, resulting in local visual features. The local visual features are then input into the region context cue learning module.

5. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 4, characterized in that, Local visual features are remapped into conditional representations of abnormal patterns and normal patterns using a lightweight mapping network; the semantic affinity between the abnormal conditional representation P and the general abnormal text prompt, and between the normal conditional representation and the general normal text prompt are calculated to obtain the semantic affinity matrix. The semantic affinity matrix is ​​summed along the dimension represented by L to obtain the semantic similarity of each patch feature to the overall general text prompt; the summed semantic affinity matrix is ​​multiplied by the corresponding conditional representation to obtain the text prompt bias containing local visual information. The global visual bias and the local visual bias are fused separately to obtain the final normal mode visual bias and abnormal mode visual bias. By embedding the final normal mode visual bias and abnormal mode visual bias into the obtained general text prompts, a dynamic text prompt guided by visual information is obtained.

6. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 5, characterized in that, The cosine similarity between the global image features, local patch features, and the text features corresponding to the dynamic text prompt is calculated to obtain the anomaly score and pixel-level anomaly region mask. The global and local properties are optimized by cross-entropy loss, Focal loss, and Dice loss functions. Finally, for the input instance, its visual guidance adaptive text prompt is obtained.

7. The adaptive zero-shot anomaly detection method based on a visual language model as described in claim 6, characterized in that, The auxiliary dataset is a dataset related to the object detection task but without object dataset annotation information, used to train the general text prompt learner to learn generalized normal and abnormal patterns; The state template words are descriptive terms used to distinguish between normal and abnormal states; The general text prompt continuously adjusts the embedded representation of its internal token through an optimization process.

8. An adaptive zero-shot anomaly detection system based on a visual language model, as described in any one of claims 1 to 7, characterized in that, include: The model building unit is used to build an adaptive zero-shot anomaly detection model based on a visual language model. The model includes a pre-trained visual language model CLIP, a general text prompt learner, a compositional context prompt learning module, and a region context prompt learning module. A general text prompt learning unit is used to execute the general text prompt learner to obtain a general text prompt template; The context prompt learning and visual information embedding unit is used to train the constituent context prompt learning module and the region context prompt learning module to embed visual information into text prompts and generate dynamic text prompts. The anomaly detection unit is used to input the dataset to be tested into the trained model to perform anomaly detection and output anomaly scores and pixel-level anomaly region masks. The storage unit is used to store model parameters, intermediate results, the trained model, and detection results during the training process.

9. An electronic device, comprising: At least one memory stores computer-executable instructions non-transiently; At least one processor, configured to run the computer-executable instructions, The computer-executable instructions are implemented by the processor at runtime according to any one of claims 1-7, which describes the adaptive zero-shot anomaly detection method based on a visual language model.

10. A computer-readable storage medium, wherein, The computer-readable storage medium stores computer-executable instructions that, when executed by at least one processor, implement the adaptive zero-shot anomaly detection method based on a visual language model according to any one of claims 1-7.

Images