Zero-shot industrial anomaly detection system and method based on vision-language models
By aligning image and text data in the same semantic space using a visual-language model, a defect saliency heatmap is generated and a detection report is automatically generated. This solves the problems of annotation dependence and insufficient generalization ability of traditional methods, and realizes a closed-loop system for zero-shot detection and report generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HUICUI INTELLIGENT TECH CO LTD
- Filing Date
- 2026-02-26
- Publication Date
- 2026-08-04
AI Technical Summary
Existing industrial testing methods rely on a large amount of labeled data, lack zero-sample generalization ability, lack semantic interpretation of test results, make it difficult to form a closed-loop system, and cannot automatically generate structured reports.
A zero-shot industrial anomaly detection system based on a vision-language model is adopted. By aligning image and text data in the same semantic space through an image encoder and a text encoder, a defect saliency heatmap is generated and defect parameters are extracted, and an inspection report is automatically generated.
It achieves zero-sample testing, has strong generalization ability, can interpret test results and automatically generate structured reports, improves quality management efficiency, and reduces manual intervention.
Smart Images
Figure CN121724992B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of quality inspection technology, and more specifically, to a zero-sample industrial anomaly detection system and method based on a vision-language model. Background Technology
[0002] In modern industrial production, quality inspection is a crucial step in ensuring product consistency, reliability, and safety.
[0003] With the development of industrial automation and intelligent manufacturing, traditional manual visual inspection is gradually being replaced by machine vision systems. Machine vision uses cameras and image processing algorithms to automatically identify defects, measure dimensions, and detect surface flaws in production lines or workshop environments.
[0004] However, current mainstream detection solutions still have significant limitations. Traditional machine vision methods rely on rule-based image processing (such as edge detection and threshold segmentation), are sensitive to lighting and background, and have poor generalization ability. Deep learning methods require a large amount of labeled data, have high training costs, and cannot identify defects outside the training set. Supervised / self-supervised methods can detect anomalies but cannot explain "what kind of anomaly" it is. Although multimodal pre-trained models have image-text alignment capabilities, they are still in the exploratory stage in industrial detection and have not yet formed a complete system. Summary of the Invention
[0005] The purpose of this invention is to provide a zero-sample industrial anomaly detection system and method based on a vision-language model, which solves the technical problems existing in the prior art.
[0006] The first aspect of this invention provides a zero-shot industrial anomaly detection system based on a vision-language model, comprising:
[0007] The system consists of an input layer, a visual-linguistic feature extraction layer, a zero-shot anomaly detection layer, a report generation layer, and an output layer.
[0008] The input layer includes an industrial image acquisition module and a natural language input module, used to acquire image data and text data;
[0009] The visual-language feature extraction layer includes an image encoder and a text encoder, used to represent image data and text data into vectors;
[0010] The zero-sample anomaly detection layer includes a similarity calculation module and a region localization module, which are used to calculate the matching degree between image vectors and text vectors and generate a defect saliency heatmap.
[0011] The report generation layer is used to extract defect parameters based on the defect saliency heatmap and generate an industrial inspection report;
[0012] The output layer is used to visualize and output industrial testing reports.
[0013] In this solution, the industrial image acquisition module is used to acquire image data of industrial surfaces and / or interiors, and the natural language input module is used to acquire text data describing defects by the user.
[0014] In this scheme, the image encoder is used to map the input image data to a semantic space to obtain an image vector; the text encoder is used to map the input text data to a text vector, wherein the image vector and the text vector are aligned in the same semantic space.
[0015] In this scheme, the similarity calculation module is used to calculate the matching degree between image vectors and text vectors through cosine similarity, and the region localization module is used to divide image data into different sub-regions using a sampling block attention mechanism, and calculate the similarity between the image and text based on the sub-region images one by one to generate a defect saliency heatmap.
[0016] In this scheme, semantic alignment is achieved through contrastive learning, where the loss function is as follows:
[0017] ;
[0018] in, For loss function, The number of training samples, For the first The image vectors corresponding to each sub-region In order to be with the first Each sub-region corresponds to a text vector that matches the image. This is the temperature coefficient.
[0019] In this scheme, the input image data is divided into multiple sub-regions. The feature vector of each sub-region is compared with the text vector to generate a salient heatmap, as shown in the following formula:
[0020] ;
[0021] in, For saliency heatmaps at pixel coordinates The value at that location, For normalization function, For attention weights, Represents pixel coordinates Does it belong to a sub-region? , For the first The similarity calculation results between the image vectors of the corresponding images of each sub-region and their matching text vectors. The number of sub-regions.
[0022] In this scheme, defect parameters are extracted based on a saliency heatmap. Threshold segmentation is used to extract defect regions, and the results are as follows:
[0023] ;
[0024] in, This is a defective area. The segmentation threshold is... For image vectors and text defects, area Defect center of gravity The number of defects was obtained through connected component analysis.
[0025] In this solution, the defect parameters include at least the defect type, location, quantity, and area. After obtaining the detection results, an industrial inspection report is generated using a fill-in template or a language model.
[0026] A second aspect of the present invention also provides a zero-shot industrial anomaly detection method based on a vision-language model, applied to a zero-shot industrial anomaly detection system based on a vision-language model as described in any of the preceding claims, wherein the method includes the following steps:
[0027] Acquire image and text data;
[0028] Represent image and text data using vectors;
[0029] Calculate the matching degree between image vectors and text vectors, and generate a heatmap of defect saliency;
[0030] Defect parameters will be extracted and an industrial inspection report will be generated.
[0031] Visualize and output industrial testing reports.
[0032] A third aspect of the present invention provides a computer-readable storage medium comprising a machine program for a zero-sample industrial anomaly detection method based on a vision-language model, wherein when the zero-sample industrial anomaly detection method program based on a vision-language model is executed by a processor, it implements the steps of a zero-sample industrial anomaly detection method based on a vision-language model as described in any of the preceding claims.
[0033] This invention discloses a zero-shot industrial anomaly detection system and method based on a vision-language model. It achieves zero-shot industrial anomaly detection and report generation through a vision-language model, solving the pain points of traditional methods in terms of annotation dependence, generalization ability, interpretability, and automation. It possesses strong innovation and practicality, making it suitable for widespread application in fields such as intelligent manufacturing and quality inspection. Specific beneficial effects are as follows:
[0034] 1. Eliminate the dependence on large-scale annotation. By introducing a large-scale pre-trained vision-language model, zero-shot detection based on text description is achieved. Users only need to input natural language prompts to detect novel defects without the need for additional labeled data.
[0035] 2. It has zero-shot generalization capability. Since the system's detection is driven by natural language, when a new defect category appears, only a new text prompt needs to be input, without retraining, which effectively solves the problem that existing methods "can only identify known defects".
[0036] 3. It provides semantic detection results and human-computer interaction. Through visual-language alignment, this invention can not only give the location of defects, but also interpret the detection results. For example, the report can automatically generate "Two tiny scratches with a length of about 3mm were found in the lower left corner of the workpiece surface".
[0037] 4. Achieving a closed loop between inspection and report generation: This invention automatically generates a structured report after inspection, containing key information such as defect type, location, quantity, size, and severity. This will greatly improve quality management efficiency and reduce manual intervention;
[0038] 5. This invention promotes the application of visual-language models in industry. It innovatively combines natural language interaction, zero-shot detection, and report generation to form a complete closed-loop system for industrial applications, breaking through the limitations of existing methods. Attached Figure Description
[0039] Figure 1 A block diagram of a zero-shot industrial anomaly detection system based on a vision-language model according to the present invention is shown.
[0040] Figure 2 A flowchart illustrating the similarity calculation process of a zero-shot industrial anomaly detection system based on a vision-language model according to the present invention is shown.
[0041] Figure 3 A schematic diagram of defect segmentation for a zero-shot industrial anomaly detection system based on a vision-language model according to the present invention is shown.
[0042] Figure 4 The diagram illustrates the steps of a zero-shot industrial anomaly detection method based on a vision-language model according to the present invention. Detailed Implementation
[0043] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.
[0044] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.
[0045] Traditional machine vision relies on regularized features and supervised training using deep learning. Regularized methods analyze targets using classic image processing techniques such as edge detection, thresholding, and morphological operations. For example, scratch detection can be achieved by calculating edge responses using gradient operators (such as the Sobel operator). However, these methods are extremely sensitive to complex backgrounds and lighting variations. Deep learning methods train large-scale labeled datasets using Convolutional Neural Networks (CNNs) to learn anomalous features. For example, the U-Net structure can be used for pixel-level defect segmentation. However, these methods heavily rely on manual annotation and lack generalization ability for new types of defects.
[0046] In existing technologies, researchers typically utilize object detection frameworks such as CNN, Faster R-CNN, and YOLO to locate surface defects. For example, for steel plate scratch detection, it is usually necessary to collect tens of thousands of scratch samples and obtain a detection model through supervised learning. However, its limitations are: it is difficult to obtain a robust model for small sample defects; the model has poor transferability, and data needs to be re-labeled when changing production lines or defect types.
[0047] Meanwhile, another type of method attempts to bypass annotation by using self-supervised or unsupervised anomaly detection. The core idea is that the network is trained to learn only the distribution of "normal samples". During inference, if the input sample Deviations from the distribution are considered defects. Typical implementations include: based on autoencoders... ,in, For encoder, For the decoder, if the reconstruction error If the value is too large, it is considered an anomaly, and generative adversarial networks are used to detect whether the input deviates from the normal distribution through a discriminator. However, the results of such methods are usually "whether it is an anomaly" and lack the ability to interpret the semantics of anomalies.
[0048] With the emergence of visual-language models such as CLIP (Contrastive Language–Image Pretraining), multimodal alignment has become a research hotspot. A few scholars have attempted to apply CLIP to industrial scenarios, such as "using text descriptions to guide defect classification." For example, given the input text prompt "scratches on metal surface," the model can retrieve corresponding features from the image. However, current research is still in the exploratory stage and has not yet formed a complete closed-loop system for detection-interpretation-report generation.
[0049] Therefore, the shortcomings of the existing technologies mentioned above include: (1) They rely on a large amount of labeled data, which is costly. Existing deep learning detection methods require the establishment of large-scale labeled datasets for each type of defect. For complex and varied defects in industrial scenarios (scratches, cracks, bubbles, foreign objects, uneven surfaces, etc.), the labeling cost is extremely high and requires professional knowledge; (2) They lack zero-shot capability and have insufficient generalization. Existing methods can usually only detect defects that have appeared in the training set. If a new type of defect appears on the production line (such as a previously unseen surface corrosion morphology), the model cannot identify it; (3) The detection results lack semantic interpretation and human-computer interaction capabilities. Although unsupervised methods can detect anomalies, they often only give an "anomaly score." However, such results only indicate "there is a problem" but cannot answer "what is the problem," "where does it occur," or "how severe is it." This makes it difficult for the system to provide useful decision-making basis for engineers; (4) Report generation is lacking, making it difficult to form a closed loop. Existing detection systems often only output binary results or defect masks and cannot automatically generate detection reports. Reports need to be manually compiled, which is time-consuming and labor-intensive, and it is also easy to miss key information. This means that there is a gap between the detection and quality management process; (5) The application of multimodal models is not yet mature. Although visual-language models such as CLIP have shown strong cross-modal understanding capabilities, their application in the industrial field still has the following shortcomings: most of the work only stays at classification or retrieval, lacking pixel-level precise positioning; text interaction capabilities have not been fully utilized, and it is impossible to realize the dynamic detection needs based on natural language; it has not yet been combined with industrial report generation, making it difficult to implement in actual enterprise scenarios.
[0050] To address the aforementioned shortcomings of existing technologies, this application proposes a zero-shot industrial anomaly detection system and method based on a vision-language model, wherein... Figure 1 The diagram shows a block diagram of a zero-sample industrial anomaly detection system based on a vision-language model according to the present invention.
[0051] like Figure 1 As shown, this invention discloses a zero-shot industrial anomaly detection system based on a vision-language model, comprising:
[0052] The system consists of an input layer, a visual-linguistic feature extraction layer, a zero-shot anomaly detection layer, a report generation layer, and an output layer.
[0053] The input layer includes an industrial image acquisition module and a natural language input module, used to acquire image data and text data;
[0054] The visual-language feature extraction layer includes an image encoder and a text encoder, used to represent image data and text data into vectors;
[0055] The zero-sample anomaly detection layer includes a similarity calculation module and a region localization module, which are used to calculate the matching degree between image vectors and text vectors and generate a defect saliency heatmap.
[0056] The report generation layer is used to extract defect parameters based on the defect saliency heatmap and generate an industrial inspection report;
[0057] The output layer is used to visualize and output industrial testing reports.
[0058] It should be noted that, in this embodiment, the industrial image acquisition module is used to acquire image data of industrial surfaces and / or interiors. , Indicates altitude, Indicates width, Indicating the number of channels, the natural language input module is used to acquire text data describing the defects from the user. .
[0059] Furthermore, in this embodiment, the image encoder Used to map input image data to a semantic space to obtain image vectors. The text encoder Used to map input text data into text vectors The image vector and the text vector are aligned in the same semantic space.
[0060] It should be noted that, in this embodiment, semantic alignment is achieved through contrastive learning, wherein the loss function is as follows:
[0061] ;
[0062] in, For loss function, The number of training samples, For the first The image vectors corresponding to each sub-region In order to be with the first Each sub-region corresponds to a text vector that matches the image. Using temperature coefficients, this mechanism enables the model to learn a one-to-one correspondence between image features and text semantics, thus achieving zero-shot detection capability.
[0063] Furthermore, in this embodiment, the similarity calculation module is used to calculate the matching degree between the image vector and the text vector through cosine similarity, and the region localization module is used to divide the image data into different sub-regions using a sampling block attention mechanism, and calculate the similarity between the image and the text one by one based on the sub-region to generate a defect saliency heatmap.
[0064] It should be noted that, in this embodiment, as Figure 2 The diagram shows a flowchart for similarity calculation. The input image data is divided into multiple sub-regions. The feature vector of each sub-region is compared with the text vector to generate a saliency heatmap. The formula is as follows:
[0065] ;
[0066] in, For saliency heatmaps at pixel coordinates The value at that location, For normalization function, For attention weights, Represents pixel coordinates Does it belong to a sub-region? , For the first The similarity calculation results between the image vectors of the corresponding images of each sub-region and their matching text vectors. The number of sub-regions.
[0067] Furthermore, in this embodiment, as Figure 3 The diagram shown illustrates defect segmentation. Defect parameters are extracted based on a saliency heatmap, with threshold segmentation used to extract defect regions. The results are as follows:
[0068] ;
[0069] in, This is a defective area. The segmentation threshold is... For image vectors and text defects, area Defect center of gravity The number of defects was obtained through connected component analysis.
[0070] Furthermore, in one embodiment of the invention, the defect parameters include at least defect type, location, quantity, and area, and an industrial inspection report is generated using a fill-in template or language model after the inspection results are obtained.
[0071] It should be noted that, in this embodiment, based on the detection results, information such as defect type, location, quantity, and area is extracted. The text generation module uses a language model to convert the detection results into a highly readable industrial inspection report. Specifically, the report generation module uses a fill-in template or a language model for generation. For example, the detection results are... One defect, the report template is categorized as: User-input text prompt. Specify; Quantity: ;area: ;Location: The final output is a natural language description, such as: "Two tiny scratches were detected on the surface of the workpiece, located at the lower left corner and the right edge, with lengths of approximately 3mm and 4mm respectively." Furthermore, through visualization output, the defect location results can be intuitively visualized (e.g., mask or heat map). At the same time, the visualization of the inspection report also makes it easier for quality inspectors to understand and archive.
[0072] Furthermore, this invention is not only applicable to metal surface inspection but can also be extended to other fields, such as PCB circuit board defect detection; medical image-assisted diagnosis; automotive parts quality inspection; and aerospace composite material surface inspection. Only the input text prompts need to be changed; no modification to the model itself is required. The goal of this invention is to achieve zero-sample industrial anomaly detection and automatic report generation. Its core idea is to utilize the semantic alignment capability of a visual-language model. In other embodiments, different visual encoders (ViT, CNN, Swin Transformer…); text encoders (CLIP-Text, BERT, GPT…); similarity measurement methods (cosine distance, Euclidean distance, learned metric networks…); region localization methods (window scanning, RPN, saliency networks…); and report generation methods (template filling, language model generation…) can all achieve the same inventive objective.
[0073] Furthermore, such as Figure 4 As shown, this invention discloses a zero-shot industrial anomaly detection method based on a vision-language model, comprising the following steps:
[0074] S402, acquire image data and text data;
[0075] S404 represents image data and text data as vectors;
[0076] S406, calculate the matching degree between image vectors and text vectors, and generate a heatmap of defect salience;
[0077] S408 will extract defect parameters and generate an industrial inspection report;
[0078] S410 provides a visual output of industrial testing reports.
[0079] It should be noted that, in this embodiment, the zero-shot industrial anomaly detection method based on a visual-language model disclosed in this application is specifically applied to the zero-shot industrial anomaly detection system based on a visual-language model described in the above embodiment. The specific implementation details are consistent with the content of the zero-shot industrial anomaly detection system based on a visual-language model described above, so no further details will be provided in this embodiment.
[0080] A third aspect of the present invention provides a computer-readable storage medium comprising a program for a zero-sample industrial anomaly detection method based on a vision-language model, wherein when the program is executed by a processor, it implements the steps of a zero-sample industrial anomaly detection method based on a vision-language model as described in any of the preceding claims.
[0081] This invention discloses a zero-shot industrial anomaly detection system and method based on a vision-language model. It achieves zero-shot industrial anomaly detection and report generation through a vision-language model, solving the pain points of traditional methods in terms of annotation dependence, generalization ability, interpretability and automation. It has strong innovation and practicality and is suitable for promotion and application in fields such as intelligent manufacturing and quality inspection.
[0082] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0083] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0084] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0085] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0086] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
Claims
1. A zero-shot industrial anomaly detection system based on a vision-language model, characterized in that, include: The system consists of an input layer, a visual-linguistic feature extraction layer, a zero-shot anomaly detection layer, a report generation layer, and an output layer. The input layer includes an industrial image acquisition module and a natural language input module, used to acquire image data and text data; The visual-language feature extraction layer includes an image encoder and a text encoder, used to represent image data and text data into vectors; The zero-shot anomaly detection layer includes a similarity calculation module and a region localization module, which are used to calculate the matching degree between image vectors and text vectors and generate a defect saliency heatmap. The report generation layer is used to extract defect parameters based on the defect saliency heatmap and generate an industrial inspection report; The output layer is used to visualize and output industrial testing reports; The industrial image acquisition module is used to acquire image data of industrial surfaces and / or interiors, and the natural language input module is used to acquire text data describing defects by users. The similarity calculation module is used to calculate the matching degree between image vectors and text vectors through cosine similarity. The region localization module is used to divide image data into different sub-regions using a sampling block attention mechanism, and calculate the similarity between the image and text based on the sub-region images one by one to generate a defect saliency heatmap. Semantic alignment is achieved through contrastive learning, where the loss function is as follows: ； in, For loss function, The number of training samples, For the first The image vectors corresponding to each sub-region In order to be with the first Each sub-region corresponds to a text vector that matches the image. Temperature coefficient; The input image data is divided into multiple sub-regions. The similarity between the feature vector of each sub-region and the text vector is calculated to generate a salient heatmap, as shown in the following formula: ； in, For saliency heatmaps at pixel coordinates The value at that location, For normalization function, For attention weights, Represents pixel coordinates Does it belong to a sub-region? , For the first The similarity calculation results between the image vectors of the corresponding images of each sub-region and their matching text vectors. The number of sub-regions.
2. The zero-shot industrial anomaly detection system based on a vision-language model according to claim 1, characterized in that, The image encoder is used to map the input image data to a semantic space to obtain an image vector; the text encoder is used to map the input text data to a text vector, wherein the image vector and the text vector are aligned in the same semantic space.
3. The zero-shot industrial anomaly detection system based on a vision-language model according to claim 1, characterized in that, Defect parameters were extracted based on a saliency heatmap, and the defect region was extracted using threshold segmentation. The results are as follows: ； in, This is a defective area. The segmentation threshold is... For image vectors and text defects, area Defect center of gravity The number of defects was obtained through connected component analysis.
4. The zero-shot industrial anomaly detection system based on a vision-language model according to claim 1, characterized in that, The defect parameters include at least the defect type, location, quantity, and area. After obtaining the test results, an industrial test report is generated using a fill-in template or language model.
5. A zero-shot industrial anomaly detection method based on a vision-language model, characterized in that, The method is applied to a zero-shot industrial anomaly detection system based on a vision-language model as described in any one of claims 1-4, wherein the method comprises the following steps: Acquire image and text data; Represent image and text data using vectors; Calculate the matching degree between image vectors and text vectors, and generate a heatmap of defect salience; Defect parameters will be extracted and an industrial inspection report will be generated. Visualize and output industrial testing reports.
6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a zero-shot industrial anomaly detection method program based on a vision-language model. When the zero-shot industrial anomaly detection method program based on a vision-language model is executed by a processor, it implements the steps of the zero-shot industrial anomaly detection method based on a vision-language model as described in claim 5.