Industrial anomaly detection method, system and equipment and storage medium

Abstract

The invention discloses an industrial anomaly detection method, system and device and a storage medium, which are corresponding schemes, and in the scheme, based on an optional reference image and natural language instruction drive, unified modeling of semantic comprehension and pixel-level segmentation is realized by introducing a learnable segmentation token and expanding the generation capability of a VLM; by designing a structured prompt text, a normal sample is used as a selectable visual reference input, and the VLM is guided to perceive an abnormal difference in comparative learning; and meanwhile, mapping a segmentation token output by the VLM into a segmentation prompt, and injecting the segmentation prompt into a universal visual segmentation model to realize explicit control on image segmentation. According to the method, direct participation of the VLM in visual result generation is realized through a segmentation mechanism in which the token directly participates, a module barrier of semantic understanding and space segmentation is broken, and an industrial anomaly detection scheme which is accurate in positioning and higher in generalization ability in zero-sample and few-sample scenes is constructed.

Description

Technical Field

[0001] This invention relates to the field of industrial anomaly detection technology, and in particular to an industrial anomaly detection method, system, equipment and storage medium. Background Technology

[0002] Industrial anomaly detection is a method that utilizes advanced technologies such as deep learning and computer vision to automatically identify defects and other anomalies in industrial production processes. Its main goal is to analyze data from the production line to promptly detect product defects, thereby ensuring product quality, optimizing production processes, and reducing losses. Industrial anomaly detection is a challenging task. It requires accurate identification of product defects, which involves processing large amounts of data and ensuring the adaptability and robustness of the algorithm under different operating conditions. Specifically, this task requires considering the definitions and distinctions between normal and abnormal states to accurately determine abnormal conditions, while also considering the model's generalization ability to handle unknown types of anomalies. Furthermore, due to the diversity and subtle differences in anomalies in industrial environments, the system needs to possess high sensitivity to detect minute changes. With the rapid development of deep learning technology, most existing methods achieve anomaly detection by constructing deep learning model architectures and training them with data, thereby improving the accuracy and reliability of detection.

[0003] Based on the different modal types of the input model information, industrial anomaly detection methods can be divided into methods based on single-modal information and methods based on multi-modal information. Methods based on single-modal information can be further classified into supervised methods and unsupervised methods.

[0004] Early research employed supervised learning strategies. These methods assumed high contrast in the target region, utilizing the significant differences between anomalous and normal regions for saliency detection, suppressing unwanted areas, and highlighting the target region. For example, DevNet, proposed by Pang et al. in their paper "Deep anomaly detection with deviation networks. Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining 2019," and Dynamic Weighted Principal Component Analysis (DWPCA), proposed by Qiu et al. in their paper "An effective framework of automated visual surface defect detection for metal parts. IEEE Sensors journal 2021," achieved preliminary anomaly detection by extracting and analyzing salient features in images. However, these methods often fall short when dealing with low-contrast anomalous regions, complex shapes, or the presence of interference.

[0005] In the field of industrial anomaly detection, unsupervised learning methods have attracted widespread attention because they do not rely on large amounts of labeled data. Many studies attempt to identify anomalies by establishing a data distribution model of normal samples. These methods typically assume that the training data contains only normal samples, and the model learns its latent distribution characteristics, identifying anomalies in samples that deviate from this distribution during the testing phase. For example, the Cflow-ad method proposed by Gudovskiy et al. in their paper "Cflow-ad: Real-time unsupervised anomaly detection with localization via conditional normalizingflows. Proceedings of the IEEE / CVF winter conference on applications of computer vision 2022" constructs a distribution model of normal samples by estimating the probability density of multi-scale feature vectors and calculates the anomaly score of test samples based on the distribution model of normal data. The Realnet method proposed by Zhang et al. in their paper "Realnet: A feature selection network with realistic synthetic anomaly for anomaly detection. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition 2024" learns the distribution of normal samples and attempts to reconstruct the original normal image from images containing anomaly information. It then compares the differences between the reconstructed image and the original image to identify the anomaly region. Unsupervised methods typically exhibit good detection performance within the scope of the training data (i.e., within the domain), especially in industrial scenarios with relatively stable structures and minimal changes, where they can achieve high detection accuracy. However, unsupervised learning methods often exhibit weak generalization ability when faced with new scenarios, new product types, or environmental changes. The main reason for this problem is that the model's modeling of normal samples depends on the distribution characteristics of the training data. Once the test data and the training data differ significantly in appearance, structure, or environmental conditions, the model is prone to misjudging normal changes as abnormalities or failing to identify truly new types of abnormalities, resulting in a significant decrease in detection performance.

[0006] Some studies have begun to explore the use of multimodal base models to enhance the perception and reasoning capabilities of systems. Most of these studies utilize auxiliary information from multiple modalities, combine features of reference images, and compare them with features of query images to help models better understand complex product anomaly scenarios. For example, WinCLIP in Jeong et al.'s paper "Winclip: Zero- / few-shot anomaly classification and segmentation. Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023," and AnomalyCLIP proposed by Zhou et al. in their paper "Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310.18961, 2023," are anomaly detection models developed from the CLIP (Contrastive Language-Image Pretraining) model proposed by Radford et al. in their paper "Learning transferable visual models from natural language supervision. International conference on machine learning. PmLR, 2021." CLIP, through large-scale contrastive learning tasks, can establish rich correlations between text and images, providing a strong foundation for cross-modal understanding. CLIP-based anomaly detection models such as WinCLIP and AnomalyCLIP primarily perform image-level classification and pixel-level segmentation by calculating the cosine similarity between query image features and text features, as well as reference image features. However, these CLIP-based models still face certain challenges when handling anomaly detection tasks.Since CLIP is primarily pre-trained on natural language and general image datasets, its ability to perceive anomalous semantics is limited. This results in WinCLIP and AnomalyCLIP still performing poorly in identifying anomalous products in specific industrial environments. Another type of model utilizes the pre-trained weights of SAM (Segment Anything Model) proposed by Kirillov et al. in "Segment anything. Proceedings of the IEEE / CVF international conference on computer vision. 2023" and Grounding DINO proposed by Liu et al. in "Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European conference on computer vision. 2024" to enhance the understanding of anomalous scenes. Cao et al., in their paper "Personalizing vision-language models with hybrid prompts for zero-shot anomaly detection. IEEE Transactions on Cybernetics. 2025," proposed AnomalyVLM, which first coarsely searches for anomalous regions using Grounding DINO, and then refines these coarse anomalous regions into pixel-level segmentation masks using SAM. However, because the visual data annotation granularity used by SAM during its pre-training process differs significantly from the anomaly types commonly found in real-world industrial environments, AnomalyVLM cannot effectively detect subtle product defects or irregular shapes when directly applied to industrial anomaly detection tasks using SAM.

[0007] Although these methods perform well in certain specific scenarios, due to limitations in the modeling capabilities of the existing methods, they have not yet been able to fully utilize the potential advantages of more advanced technologies (such as multimodal large models), resulting in their generalization ability and adaptability still needing further improvement.

[0008] Multimodal large models (e.g., visual language large models, VLMs) have more powerful perception and contextual understanding capabilities in processing multimodal information such as vision and text. For example, the ALFA (Adaptive LLM-empowered model for zero-shot visual anomaly detection with Fine-grained Alignment) method proposed by Zhu et al. in the article "Do LLMs Understand Visual Anomalies? Uncovering LLM's Capabilities in Zero-shot Anomaly Detection. Proceedings of the 32nd ACM International Conference on Multimedia. 2024" utilizes LLM to generate and optimize cue text about query images; AnomalyGPT proposed by Gu et al. in the article "Anomalygpt: Detecting industrial anomalies using large vision-language models. Proceedings of the AAAIconference on artificial intelligence. 2024" and AnomalyGPT proposed by Li et al. in the article "Myriad: Large multimodal model by applying vision experts for industrial anomaly detection. arXiv preprint arXiv:2310.19070". The Myriad model proposed in "2023" generates segmentation results and then inputs them into a VLM to obtain classification results from the text responses. While their approach can utilize the powerful generalization and semantic understanding capabilities of VLM to some extent, the role of VLM is relatively indirect. It fails to directly establish a connection between VLM and the output segmentation results, using it only as an output classification result, thus failing to fully realize the potential of VLM.

[0009] In the realm of general multimodal large models, most current VLMs use the overall image and cue text as input to generate plain text responses based on the input image. While these VLM models are complex, they can only output text and still cannot generate their fine-grained segmentation masks within a visual context. Fine-grained segmentation masks are crucial for applications such as visual understanding of details. For example, VisionLLM, proposed by Wang et al. in their paper "VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems. 2023," attempts to directly correlate the results generated after fine-grained visual understanding with VLMs by parsing the segmentation mask into a sequence of polygons, making the segmentation mask a plain text representation, and allowing end-to-end training within the framework of existing VLMs. However, unless significant computational resources are used, end-to-end training using polygon sequences introduces substantial training overhead and may compromise the original capabilities of VLMs.

[0010] In view of this, the present invention is hereby proposed. Summary of the Invention

[0011] The purpose of this invention is to provide an industrial anomaly detection method, system, device, and storage medium. Through a segmentation mechanism involving direct token participation, it enables the large visual language model to directly participate in the generation of visual results, breaking down the module barriers between semantic understanding and spatial segmentation in traditional methods. This constructs an industrial anomaly detection solution with accurate localization and stronger generalization ability in zero-sample and few-sample scenarios.

[0012] The objective of this invention is achieved through the following technical solution: An industrial anomaly detection method includes: Learnable segmentation tokens are introduced into the vocabulary of the visual language big model. Based on the cue text, or the reference image and cue text, the visual language big model is guided to reason about the input image to be detected and output a text response. When there is an anomaly in the image to be detected, the text response contains the segmentation token. The implicit vector corresponding to the segmentation token is mapped to a segmentation cue vector that matches the input space of the visual feature extractor through a projection layer. Visual feature vectors are extracted from the image to be detected using the visual feature extractor in the general visual segmentation model. The visual feature vectors and segmentation cue vectors are then input into the visual feature decoder in the general visual segmentation model to obtain the segmentation mask.

[0013] An industrial anomaly detection system for implementing the aforementioned method includes: A large visual language model incorporates learnable segmentation tokens into its vocabulary for reasoning guided by cue text or a reference image and cue text, outputting a text response. When an anomaly is detected in the image to be detected, the text response contains the segmentation token. The projection layer is used to map the implicit vector corresponding to the segmentation token to a segmentation cue vector that matches the input space of the visual feature extractor. A general visual segmentation model includes a visual feature extractor and a visual feature decoder; wherein, the visual feature extractor is used to extract visual feature vectors from the image to be detected, and the visual feature decoder is used to output a segmentation mask using the visual feature vectors and segmentation cue vectors.

[0014] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0015] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0016] As can be seen from the technical solution provided by the present invention, based on optional reference images and natural language instructions, by introducing learnable segmentation tokens and expanding the generation capabilities of the visual language large model, unified modeling of semantic understanding and pixel-level segmentation is achieved; by designing structured prompt text, normal samples are used as optional visual reference inputs to guide the visual language large model to perceive abnormal differences in contrastive learning; at the same time, the segmentation tokens output by the visual language large model are mapped to visual prompts (segmentation prompts) and injected into a general visual segmentation model (such as SAM) to achieve explicit control over image segmentation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an industrial anomaly detection method provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the overall architecture of an industrial anomaly detection method provided in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram illustrating an example of the visualization results from an experiment provided in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of an industrial anomaly detection system provided in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0024] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0025] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0026] Unless otherwise explicitly specified or limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this document according to the specific circumstances.

[0027] The following is a detailed description of an industrial anomaly detection method, system, device, and storage medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, conventional conditions in the art or conditions recommended by the manufacturer shall apply. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0028] Example 1 This invention provides an industrial anomaly detection method, such as... Figure 1 As shown, the main steps include the following: Step 1: Output segmentation cue vectors based on the visual language large model.

[0029] In this embodiment of the invention, a learnable segmentation token is introduced into the vocabulary of the visual language large model. Based on the cue text, or the reference image and cue text, the visual language large model is guided to reason about the input image to be detected and output a text response. When there is an anomaly in the image to be detected, the text response contains the segmentation token. The implicit vector corresponding to the segmentation token is mapped to a segmentation cue vector that matches the input space of the visual feature extractor through a projection layer.

[0030] In this embodiment of the invention, when the visual language model is guided by prompt text to infer the input image to be detected, the application scenario is zero-sample industrial anomaly detection; when the visual language model is guided by reference image and prompt text to infer the input image to be detected, the application scenario is few-sample industrial anomaly detection.

[0031] In this embodiment of the invention, the prompt text adopts a structured question-and-answer text instruction format, which is used to combine the input image to be detected with the task description to instruct the visual language big model to perform abnormal region recognition; when the visual language big model is guided to reason about the input image to be detected based on the reference image and the prompt text, the prompt text also includes a relevant description of the reference image.

[0032] Step 2: Pixel-level segmentation guided by segmentation cue vectors.

[0033] In this embodiment of the invention, a visual feature vector is extracted from the image to be detected based on the visual feature extractor in the general visual segmentation model. The visual feature vector and the segmentation cue vector are then input into the visual feature decoder in the general visual segmentation model to obtain a segmentation mask.

[0034] Those skilled in the art will understand that the segmentation mask output by the visual feature decoder is the segmentation result of the image to be detected, which locates the abnormal regions of the image to be detected.

[0035] In this embodiment of the invention, the large visual language model, the projection layer, and the general visual segmentation model constitute an industrial anomaly detection model. The text response output by the large visual language model and the segmentation mask output by the visual segmentation model are used to construct corresponding losses, which are then combined into the total loss of the industrial anomaly detection model. The LoRA strategy is used for fine-tuning; wherein, the LoRA strategy is a low-rank adaptive strategy.

[0036] In this embodiment of the invention, abnormal data is synthesized using the NSA method to form a training dataset for fine-tuning an industrial anomaly detection model. The synthesis of abnormal data using the NSA method includes: given a normal image to be synthesized, randomly selecting another normal image from the same dataset, and using the NSA method to synthesize an abnormal data; wherein, the NSA method is a natural anomaly synthesis method.

[0037] In this embodiment of the invention, the total loss of the industrial anomaly detection model is expressed as: ; Where L represents the total loss of the industrial anomaly detection model. For text generation loss, For segmentation mask loss, The weights for the two losses; Text generation loss The calculation method is as follows: ;in, The text response output by the large visual language model. The labels are real text labels, and CE(.) is the autoregressive cross-entropy loss function. Segmentation mask loss The calculation method is as follows: ;in, Here, M is the segmentation mask output by the visual segmentation model, BCE(.) is the binary cross-entropy loss function, and DICE(.) is the Dice loss function. These are two adjustable hyperparameters.

[0038] In this embodiment of the invention, the fine-tuning using the LoRA strategy includes: for the Transformer structure in the large visual language model, introducing a LoRA module after each Transformer block to fine-tune it in the form of a low-rank matrix; fine-tuning the word vector layer and the large language model head of the LLM in the large visual language model; and fine-tuning the projection layer and the visual feature decoder.

[0039] The solution provided in this invention addresses the issues of insufficient localization accuracy and generalization in industrial anomaly detection. Based on selectable reference images and natural language commands, this solution introduces learnable segmentation tokens and expands the generation capabilities of a large visual language model to achieve unified modeling of semantic understanding and pixel-level segmentation. By designing structured contextual cue templates, normal samples are used as optional visual reference inputs, guiding the model to perceive anomaly differences through contrastive learning. Simultaneously, the segmentation tokens output by the large visual language model are mapped to visual cues and injected into the visual feature decoder, enabling explicit control over image segmentation. Finally, a textual response and anomaly mask are output within an end-to-end framework. Based on the above description, the solution provided by the present invention mainly has the following core technical points: (1) A context learning mechanism based on reference image (optional) and text instruction is proposed. By constructing "normal-abnormal" comparison input, the visual language big model's perception of abnormalities is enhanced, and zero-sample and few-sample industrial anomaly detection is achieved; (2) Learnable segmentation tokens are added and structured prompt templates are introduced, so that the visual language big model can trigger segmentation operations while generating text responses, and realize coherent reasoning from language understanding to visual localization; (3) A fusion architecture of visual language big model and general visual segmentation model is constructed, and the segmentation tokens in the language space are mapped to visual prompts to drive pixel-level anomaly mask generation.

[0040] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0041] I. Overall Introduction to the Plan.

[0042] This invention aims to improve the performance of product defect detection by constructing an industrial anomaly detection model that integrates a large-scale Vision Language Model (VLM) and a general visual segmentation model, achieving end-to-end industrial anomaly detection from semantic understanding to spatial localization. The model designed in this invention fully leverages the advantages of the large-scale Vision Language Model in semantic understanding and contextual reasoning, combined with the pixel-level segmentation capability of the general visual segmentation model. It can perform zero-shot anomaly detection based on natural language cues, or few-shot anomaly detection using reference images, automatically identifying and accurately locating abnormal regions in images, while simultaneously generating corresponding segmentation masks.

[0043] This invention employs a self-supervised synthesis strategy based on normal samples to construct training data and provides corresponding prompt text templates for a large-scale visual language model to assist in segmentation. To enhance the adaptability of the large-scale visual language model in low-sample scenarios, this invention incorporates a context learning mechanism by combining reference images and updates the prompt text templates accordingly. This guides the large-scale visual language model to identify anomalies through comparative analysis, demonstrating good zero-sample and low-sample adaptability.

[0044] First, this invention introduces learnable segmentation tokens (such as...) into the vocabulary of a large visual language model. <seg>This allows the segmentation token to be explicitly output during text generation, rather than conveying fine-grained visual information through text. As a semantic instruction requiring segmentation, the segmentation token is mapped to a feature vector (segmentation cue vector) and passed as a visual cue to downstream visual feature decoders (such as the mask decoder of SAM). In this way, the large-scale visual language model no longer merely generates text but actively participates in the segmentation process, thereby enabling direct guidance of pixel-level output by language instructions.

[0045] Furthermore, this invention inputs the segmentation cue vector and the visual features extracted by the visual feature extractor into the visual feature decoder to generate the corresponding segmentation mask. The entire process is optimized end-to-end during the training phase. The large visual-language model determines whether to output a segmentation token based on the input image and cue text, while the visual feature decoder generates an accurate mask for abnormal regions accordingly. This mechanism enables collaborative training of language generation and visual generation.

[0046] During training, the backbone parameters of the visual encoder and the visual language model Transformer are frozen. In the visual language model, a parameter-efficient fine-tuning method (such as LoRA) is used to update the low-rank adaptation parameters inside the Transformer, and the word vector layer and head of the language model are also updated. At the same time, the projection layer and the visual feature decoder are fine-tuned to both preserve pre-trained knowledge and reduce computational overhead.

[0047] Ultimately, this invention achieves direct participation of the visual language large model in the generation of visual results through a segmentation mechanism involving tokens. This breaks down the module barriers between semantic understanding and spatial segmentation in traditional methods, and constructs an industrial anomaly detection method with accurate localization and stronger generalization ability in zero-sample and few-sample scenarios.

[0048] II. Detailed introduction of the plan.

[0049] The solution provided in this embodiment of the invention is applicable to two scenarios: zero sample and few sample. The following is a description of these two scenarios respectively.

[0050] 1. Zero-sample industrial anomaly detection.

[0051] (1.1) Data synthesis strategy.

[0052] For training data, this invention adopts the NSA (Natural Synthetic Anomalies) method proposed by Schlüter et al. in the article "Natural synthetic anomalies for self-supervised anomaly detection and localization. European Conference on Computer Vision. 2022" to generate high-quality simulated anomaly samples, thereby enhancing the model's generalization ability and detection accuracy in industrial anomaly detection tasks.

[0053] NSA is a synthetic anomaly generation strategy that has been widely used in the field of anomaly detection in recent years. It uses image editing techniques to naturally paste local regions from normal samples into regions from other images, thereby simulating relatively more realistic anomaly images.

[0054] The NSA method is primarily based on the CutPaste method proposed by Li et al. in their paper "Cutpaste: Self-supervised learning for anomaly detection and localization. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021". CutPaste is a commonly used image enhancement method in industrial anomaly detection. The basic process involves randomly cropping a region from an image and pasting it into a random location within the same or another image to simulate anomaly structures or textures. Anomaly samples generated in this way exhibit a certain degree of diversity, effectively improving the model's ability to recognize unknown anomaly types during training. However, traditional CutPaste methods often produce noticeable boundary discontinuities or color inconsistencies when pasting image patches, resulting in unnatural stitching marks. This discontinuity not only reduces the realism of the synthesized image but may also lead the model to learn incorrect feature representations during training.

[0055] To address this issue, the NSA method uses the Poisson image editing method proposed by Pérez et al. in their paper "Poisson image editing. Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 2023" as a post-processing step. Poisson image editing is an image fusion method based on partial differential equations. By solving the Poisson equation, the source image is fused into the target image, effectively eliminating color differences and edge breaks between the pasted area and the background, thus synthesizing a visually more natural and realistic image.

[0056] This invention uses the NSA (Non-Standard Image Synthesis) method. Given a normal image to be synthesized, another normal image is randomly selected from the same dataset to synthesize the anomalous training data. By combining the randomness of CutPaste with the smoothness of Poisson image editing, diverse anomalous samples are generated while maintaining the overall consistency and visual naturalness of the images. This synthesis strategy not only improves the richness of the training data but also provides stronger robustness for the model when facing complex and varied anomaly types in real-world industrial scenarios.

[0057] (1.2) Prompt text template.

[0058] In terms of data format, in order to guide the visual-language large model to perform effective visual-language reasoning in industrial anomaly detection tasks and stimulate its ability to generate segmentation masks, a structured question-and-answer text instruction format was designed. Combining image input and task description, it clearly instructs the model to perform anomaly region recognition.

[0059] For example, the following text input template can be used: USER: This is a photo of a {class Name} for anomalydetection. A normal {class Name} should be without any damage, flaw, defect,scratch, hole or broken part.Can you segment the anomaly area in the photo? ASSISTANT: Sure, it is <seg>. in: It is a placeholder token for image patch features, representing the vector representation of the input image after being extracted by the visual encoder of the VLM; {class Name} represents the class name to which the current sample belongs (such as "bottle", "screw", "capsule", etc.), which is used to guide the model to understand the image content by combining semantic information; <seg>It is a special segmentation request token that is introduced to trigger the model to generate a segmentation mask corresponding to the abnormal region.

[0060] The design of the text template, by explicitly describing the characteristics that normal samples should possess (such as "no damage, no defects, no scratches"), provides the model with clear contextual information, helping it to more accurately determine whether there are anomalies in the current image and effectively guiding the model to insert [the text template] into the output sequence. <seg>The token activates the subsequent segmentation and decoding process. The template structure is simple and uniform, suitable for different types of industrial product inspection tasks.

[0061] (1.3) Visual Language Model.

[0062] In terms of model architecture, the first step is to introduce an additional semantic segmentation token into the output vocabulary of the large visual language model. <seg>, is used to indicate a request for image segmentation mask generation. <seg>The token shares the same vector space as other text output tokens in VLM and is optimized along with the overall model parameters during training.

[0063] Given a text instruction Query image The input image is fed into the visual language model F, enabling the model to identify abnormal regions in the query image that do not conform to the normal pattern. Based on the instructions, the model will output the corresponding text response. When the visual language model determines that the current query image contains an anomaly and requires the generation of a segmentation mask, the text response output by the visual language model will include... <seg>Token.

[0064] Then, extract the corresponding language from the output of the last layer of the large visual language model. <seg>The implicit vector of the token position is denoted as It encodes the overall semantic understanding and contextual reasoning results of a large visual language model for abnormal regions in an image. To convert it into a feature representation that can be processed by a visual model, it is further processed through a lightweight MLP projection layer. ,Will Vector mapping to visual feature extractor Feature vectors that match the input space .

[0065] (1.4) General visual segmentation model.

[0066] Meanwhile, the visual feature extractor in the general visual segmentation model... From query image Extracting visual feature vectors , recorded as .

[0067] Will and Input to the visual feature decoder (Mask Decoder) middle, Combining visual feature vectors Vectors generated by large visual language models Decode to generate the final segmentation mask. In this way, the industrial anomaly detection model can not only identify whether there are anomalies in an image, but also locate the specific position of the anomaly area, achieving fine-grained anomaly detection.

[0068] (1.5) Loss function.

[0069] To achieve end-to-end training of a multimodal large model in anomaly detection tasks, a loss function consisting of multiple tasks was used, primarily including text generation loss. and segmentation mask loss The overall optimization objective of the industrial anomaly detection model is a weighted sum of these two losses, with the weights determined by the hyperparameters. Control, in specific forms, is as follows: .

[0070] Text generation loss This measure evaluates the model's performance on text generation tasks, employing autoregressive cross-entropy loss to calculate the predicted text output by the model. With real text labels The differences between them are specifically defined as follows: ; This loss enables the model to generate a response in the desired manner based on the input image and instructions.

[0071] Segmentation mask loss This loss is used to guide the model in generating high-quality anomaly region segmentation masks. It is designed as a weighted combination of two losses: Binary Cross-Entropy Loss (BCE) and Dice Loss. The specific form is as follows: ; in, Represents the true binary segmentation mask; The segmentation mask predicted by the model is represented; the BCE loss (Binary Cross-Entropy Loss) is used to optimize the classification accuracy of each pixel; the Dice loss focuses on the overlap between the segmentation and the ground truth, and has better robustness to the class imbalance problem. These are two adjustable hyperparameters used to balance the respective contributions of the two terms to the total loss.

[0072] Through the design of the above-mentioned composite loss function, this invention can not only generate specified text responses, but also accurately locate abnormal regions in images, thereby achieving the goal of text-visual collaborative understanding and reasoning.

[0073] (1.6) Training strategies.

[0074] In terms of training strategy, to achieve efficient fine-tuning of the model on specific industrial anomaly detection tasks while retaining the original capabilities of the pre-trained visual language model, this embodiment of the invention employs a parameter-efficient fine-tuning strategy—Low-Rank Adaptation (LoRA). This significantly reduces training costs and the number of parameters while maintaining model performance, facilitating rapid training and iteration of the model under limited computing resources.

[0075] Specifically, for the Transformer structure in F, a LoRA module is introduced after each Transformer block to fine-tune the original weights in the form of a low-rank matrix. Simultaneously, the following parts are trained: the LLM word vector layer (embed_tokens) to adapt to task-related lexical representations; and the large language model head (lm_head) for generating text responses and segmentation tokens. <seg>Projection layer This is used to map the semantic vectors output by a large visual language model to the visual feature space, so that they can be used by a subsequent visual feature decoder. use.

[0076] Furthermore, visual feature decoder The visual feature decoder is fully fine-tuned to fuse the semantic vectors generated by the large visual language model with visual features and decode them into the final segmentation mask. To better adapt it to anomaly detection tasks, we fully fine-tune all its parameters to maximize its expressive power in anomaly segmentation.

[0077] The above strategy achieves a multimodal large model fine-tuning method that is parameter-efficient, knowledge-preserving, and task-adaptable. It not only makes full use of the powerful generalization ability of the pre-trained visual language large model, but also significantly improves the model's performance in industrial anomaly detection and segmentation tasks through techniques such as LoRA.

[0078] 2. Small sample industrial anomaly detection.

[0079] Based on the zero-sample industrial anomaly detection method introduced in Part 1 above, context learning is introduced to perform few-sample industrial anomaly detection by combining reference images.

[0080] Those skilled in the art will understand that "few samples" is a common technical term in the field, mainly referring to a small number of reference images available for learning (not exceeding a set value). This set value can be set by the user based on actual circumstances or experience. For example, a set value of 10 may be used to represent 1, 5, or 10 reference images.

[0081] Improvements have been made to the visual language model and prompt text templates, primarily including: (2.1) Prompt text template.

[0082] In this embodiment of the invention, a structured question-and-answer text instruction format is designed to explicitly instruct the model to identify abnormal regions. For example, the following prompt text template can be used: USER: This is a photo of {class Name} for anomaly detection.The left one is a normal {class Name} for reference, which should be withoutany damage, flaw, defect, scratch, hole or broken part. And the right one isfor anomaly detection Can you segment the anomaly area on the right one? ASSISTANT: Sure, it is <seg>. The aforementioned prompt text template, by clearly indicating that the left image is a "normal sample without any damage, defects, scratches, holes, or breaks," establishes a semantic benchmark for "normal" in the visual language large-scale model. Subsequently, the right image is defined as the detection object, and an instructional question is posed in natural language: "Can you segment the abnormal regions in the right image?" This guides the model to focus on potential anomalies in the right image, automatically inserting prompts into the text response when necessary. <seg>The marker activates the subsequent segmentation and decoding process. The prompt template is simple in design, highly versatile, and suitable for various industrial product categories.

[0083] (2.2) Visual Language Large Model.

[0084] Based on the prompt text template described in (2.1) above, the reference image (normal sample) will be used. With query image (The image to be detected) is stitched together to obtain The data is then input into the visual language model F. Through this context-guided approach, the model can compare the visual differences between the two images, identify anomalous regions in the query image that deviate from the normal pattern under the guidance of semantic instructions, and generate corresponding natural language responses. When the visual language model determines that there is an anomaly in the query image, its output text will contain specific segmentation tokens. <seg>This instructs the subsequent segmentation module to generate the corresponding abnormal region mask. The relevant process is similar to Part 1 above, so it will not be described in detail here.

[0085] Furthermore, based on the above explanation, for industrial anomaly detection with a small sample size, it is also necessary to train a corresponding industrial anomaly detection model. The training scheme can also be found in the previous introduction, and will not be repeated here.

[0086] Based on the above introduction, the following is an example of the overall solution architecture for a few-sample industrial anomaly detection solution: Figure 2 As shown. It should be noted that, Figure 2 The reference image, the image to be detected, and the specific content of the prompt text template shown are only examples. In actual applications, users can adjust them according to the actual situation.

[0087] III. Effect Verification.

[0088] To intuitively illustrate the effectiveness of the above-mentioned scheme provided in the embodiments of the present invention, experiments were conducted on the MVTec AD dataset proposed in Bergmann et al.'s article "MVTec AD--A comprehensive real-world dataset for unsupervised anomaly detection. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2019", and the VisA dataset proposed in Zou et al.'s article "Spot-the-difference self-supervised pre-training for anomaly detection and segmentation. European conference on computer vision. 2022" to analyze the advantages and positive effects of the model. Following the experimental setup of AnomalyGPT, training was performed on the MVTec AD dataset, and generalization experiments for zero-shot and few-shot industrial anomaly detection were conducted on the VisA dataset.

[0089] The evaluation metrics used are as follows: (1) Area Under the Receiver Operating Characteristic Curve (AUROC), the higher the better; (2) Per-Region-Overlap (PRO), the higher the better; (3) F1 score, the higher the better.

[0090] 1. Experimental details.

[0091] In the experiment, this invention is implemented based on the PyTorch open-source framework, using LLaVA-7B-v1-1 as the large visual language model F, and employing ViT-H SAM as the general visual segmentation model, i.e. and Projection layer It is an MLP (Multilayer Perceptron) with channels [256, 4096, 4096]. The training code is based on the Deepspeed engine proposed by Rasley et al. in the paper "Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020". The optimizer uses AdamW proposed by Loshchilov et al. in the paper "Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101. 2017", with the learning rate and weight decay set to 0.0003 and 0, respectively. Lora_r is set to 8, lora_alpha to 16, lora_dropout to 0.05, and lora_target_modules to "q_proj,v_proj". WarmupDecayLR is used as the learning rate scheduler, with the number of warm-up iterations set to 100. Weights of text generation loss and the weights of the segmentation mask loss The weights are set to 1.0 and 1.0 respectively, and the weights of the BCE loss are... Weights of DICE loss Set them to 2.0 and 0.5 respectively. Set the batch size to 2, the gradient accumulation step size to 10, and the iterations to 1000.

[0092] 2. Experimental data and visualization results.

[0093] The model in this invention is compared with existing algorithms such as WinCLIP and AnomalyGPT in terms of both evaluation metrics and intuitive visualization quality.

[0094] (1) Comparison of evaluation indicators.

[0095] Table 1 shows the comparison results of evaluation metrics for zero-sample industrial anomaly detection on the VisA dataset.

[0096] Table 1: Comparison of evaluation metrics for zero-sample industrial anomaly detection on the VisA dataset

[0097] From the AUROC metric, this invention performed best, reaching 88.9, significantly outperforming the other two methods. This indicates that in pixel-level segmentation tasks, this invention has a stronger ability to distinguish normal and abnormal samples. AnomalyGPT came in second with an AUROC of 86.1, while WinCLIP performed relatively weaker, with only 79.6. In the PRO metric, this invention also performed excellently, achieving a score of 64.4, significantly higher than AnomalyGPT's 35.7. This shows that this invention has higher accuracy and completeness in pixel-level anomaly localization, and can more accurately identify abnormal regions. Although WinCLIP also performed well in PRO (56.8), it still lags behind this invention. In the F1 score, the performance of the three methods was relatively close, with this invention reaching 15.3, exceeding WinCLIP's 14.8 and higher than AnomalyGPT's 14.0. This indicates that this invention is equally excellent in balancing precision and recall.

[0098] Table 2 shows a comparison of evaluation metrics for zero-sample and few-sample industrial anomaly detection on the VisA dataset.

[0099] Table 2: Comparison of evaluation metrics in experiments on the VisA dataset

[0100] The few-shot industrial anomaly detection method outperforms the zero-shot industrial anomaly detection method across all three metrics, particularly in the key metrics of AUROC and PRO, achieving scores of 89.6 and 66.9 respectively, significantly surpassing zero-shot industrial anomaly detection. The F1 score is also improved by 1.0 percentage point compared to zero-shot industrial anomaly detection, demonstrating stronger pixel-level anomaly localization capabilities. Compared to zero-shot industrial anomaly detection, few-shot industrial anomaly detection effectively improves the model's generalization ability and detection accuracy by utilizing reference images.

[0101] (2) Visualization quality.

[0102] Examples of visualization results are as follows Figure 3 As shown in the figure, the three columns from left to right are the query image, the Ground Truth (labeled image), and the model prediction result. The three rows from top to bottom are a capsule, a bottle, and a cable. The capsule has a damaged surface, the bottle contains foreign objects, and the cable has a gap in the lower left corner, showing different types of anomalies. From these three examples, it can be seen that the industrial anomaly detection model provided by this invention exhibits good anomaly detection capabilities on different types of industrial products. Whether it is a defect on the surface of a capsule, a thin foreign object in a bottled liquid, or an abnormal conductor inside a cable, the industrial anomaly detection model can accurately identify and segment the abnormal area.

[0103] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0104] Example 2 This invention also provides an industrial anomaly detection system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 4 As shown, the system mainly includes: A large visual language model incorporates learnable segmentation tokens into its vocabulary for reasoning guided by cue text or a reference image and cue text, outputting a text response. When an anomaly is detected in the image to be detected, the text response contains the segmentation token. The projection layer is used to map the implicit vector corresponding to the segmentation token to a segmentation cue vector that matches the input space of the visual feature extractor. A general visual segmentation model includes a visual feature extractor and a visual feature decoder; wherein, the visual feature extractor is used to extract visual feature vectors from the image to be detected, and the visual feature decoder is used to output a segmentation mask using the visual feature vectors and segmentation cue vectors.

[0105] Since the main technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0106] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0107] Example 3 The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0108] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0109] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0110] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0111] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0112] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.< / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg> < / seg>

Claims

1. An industrial anomaly detection method, characterized in that, include: Learnable segmentation tokens are introduced into the vocabulary of the visual language big model. Based on the cue text, or the reference image and cue text, the visual language big model is guided to reason about the input image to be detected and output a text response. When there is an anomaly in the image to be detected, the text response contains the segmentation token. The implicit vector corresponding to the segmentation token is mapped to a segmentation cue vector that matches the input space of the visual feature extractor through a projection layer. Visual feature vectors are extracted from the image to be detected using the visual feature extractor in the general visual segmentation model. The visual feature vectors and segmentation cue vectors are then input into the visual feature decoder in the general visual segmentation model to obtain the segmentation mask.

2. The industrial anomaly detection method according to claim 1, characterized in that, When the visual language model is guided to infer the input image to be detected based on the prompt text, the application scenario is zero-shot industrial anomaly detection; when the visual language model is guided to infer the input image to be detected based on the reference image and the prompt text, the application scenario is few-shot industrial anomaly detection.

3. An industrial anomaly detection method according to claim 1 or 2, characterized in that, The prompt text adopts a structured question-and-answer text instruction format, which is used to combine the input image to be detected with the task description to instruct the visual language big model to perform the recognition of abnormal regions; When the visual language model is guided to perform inference on the input image to be detected based on the reference image and the prompt text, the prompt text also includes a relevant description of the reference image.

4. The industrial anomaly detection method according to claim 1, characterized in that, The large visual language model, projection layer, and general visual segmentation model constitute the industrial anomaly detection model. The text response output by the large visual language model and the segmentation mask output by the visual segmentation model are used to construct the corresponding losses, which are then combined into the total loss of the industrial anomaly detection model. The LoRA strategy is used for fine-tuning; where LoRA strategy is a low-rank adaptive strategy.

5. The industrial anomaly detection method according to claim 4, characterized in that, Also includes: The NSA method is used to synthesize anomalous data to form a training dataset for fine-tuning the industrial anomaly detection model. The method of synthesizing anomalous data using the NSA method includes: given a normal image to be synthesized, randomly selecting another normal image from the same dataset, and using the NSA method to synthesize an anomalous data; wherein, the NSA method is a natural anomalous synthesis method.

6. The industrial anomaly detection method according to claim 4, characterized in that, The total loss of the industrial anomaly detection model is expressed as: ； Where L represents the total loss of the industrial anomaly detection model. For text generation loss, For segmentation mask loss, The weights for the two losses; Text generation loss The calculation method is as follows: ;in, The text response output by the large visual language model. The labels are real text labels, and CE(.) is the autoregressive cross-entropy loss function. Segmentation mask loss The calculation method is as follows: ;in, Here, M is the segmentation mask output by the visual segmentation model, BCE(.) is the binary cross-entropy loss function, and DICE(.) is the Dice loss function. These are two adjustable hyperparameters.

7. An industrial anomaly detection method according to any one of claims 4 to 6, characterized in that, The fine-tuning using the LoRA strategy includes: For the Transformer structure in the large visual language model, a LoRA module is introduced after each Transformer block to fine-tune it in the form of a low-rank matrix. In addition, the word vector layer and the large language model head of the LLM in the large visual language model are fine-tuned respectively, and the projection layer and the visual feature decoder are fine-tuned respectively.

8. An industrial anomaly detection system, characterized in that, The method for implementing any one of claims 1 to 7 includes: a large visual language model, wherein a learnable segmentation token is introduced into the vocabulary of the large visual language model for reasoning under the guidance of cue text, or a reference image and cue text, and outputting a text response, wherein the text response contains the segmentation token when there is an anomaly in the image to be detected. The projection layer is used to map the implicit vector corresponding to the segmentation token to a segmentation cue vector that matches the input space of the visual feature extractor. A general visual segmentation model includes a visual feature extractor and a visual feature decoder; wherein, the visual feature extractor is used to extract visual feature vectors from the image to be detected, and the visual feature decoder is used to output a segmentation mask using the visual feature vectors and segmentation cue vectors.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images