A zero-sample / small-sample industrial defect detection method and system

Abstract

This invention discloses a zero-shot / few-shot industrial defect detection method and system. In the segmentation branch, the parameters of the CLIP image encoder are frozen, image features are extracted, and the corrected features are projected. Image features at different layers are then multiplied one by one with the corresponding text features, and the images are then stitched together to generate an image-text similarity map. In the classification branch, the extracted final image features are multiplied with the text features, and the positive / abnormal classification results of the image are obtained after processing by a learnable adapter. The adapter parameters are optimized by minimizing the difference between the output results of the segmentation and classification branches and the true labels. In the zero-shot detection mode, anomaly maps are generated directly through matrix multiplication of image and text features at each layer. In the few-shot detection mode, reference samples are introduced, and the cosine similarity between the features at each layer and the features of the input samples is calculated. The resulting similarity map is added to the anomaly map generated by zero-shot detection to form the final anomaly heatmap.

Description

Technical Field

[0001] This invention belongs to the field of defect detection technology, specifically relating to a zero-sample / small-sample industrial defect detection method and system. Background Technology

[0002] Visual defect detection has wide applications and significant value in industrial manufacturing, primarily encompassing two core tasks: defect classification and defect segmentation. Defect classification aims to determine the presence of defects in a target object, while defect segmentation further determines the pixel-level location of defects to provide precise defect region information. Currently, most defect detection methods employ an unsupervised learning paradigm, training models on a large number of normal images and then using them to identify defect samples. This is because defects in objects typically exhibit significant variations in shape, color, texture, and size, making it difficult to define all possible defect patterns. Although this approach has achieved significant progress in surface defect detection, it usually requires training a model separately for each object. In practical applications, collecting a large number of normal samples and deploying an independent model for each object is extremely costly.

[0003] Therefore, how to construct a general model for surface defect detection that does not require a large number of normal images or only relies on a very small number of normal samples, and is applicable to various objects, is an important problem that needs to be solved. Contrastive Language-Image Pre-Training (CLIP) is the first model to be jointly trained on image and text data. Because the CLIP model is pre-trained on a large dataset containing 400 million data points, it exhibits excellent generalization ability and performance in tasks such as image classification and object detection without additional training. Therefore, using the CLIP framework to complete zero-shot / few-shot defect detection models is currently the mainstream approach. However, since CLIP is primarily designed for classification tasks, it still faces many challenges in semantic segmentation tasks. Its image-text contrastive training paradigm emphasizes global features while ignoring local discriminability, leading to incorrect segmentation results and noisy segmentation maps. In the field of defect detection, this manifests as generating defect localization maps that are the opposite of the defect results, thus reducing detection accuracy. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a zero-sample / small-sample industrial defect detection method and system to address the shortcomings of the prior art, thereby solving the technical problem of insufficient local discriminability of the CLIP model in industrial defect detection due to global feature bias.

[0005] The present invention adopts the following technical solution: A zero-sample / small-sample industrial defect detection method includes the following steps: S1. Freeze the CLIP image encoder parameters and extract image features from different levels; correct each layer of features through the QQ module, project the linear layer, multiply each layer of image features with its corresponding text features element by element to generate a single-layer similarity map, and stitch together the results of each layer to form an image-text similarity map. S2. Extract the final layer global image features of CLIP, multiply them with the text features, input them into the learnable adapter, and output the positive / abnormal classification probability. S3. Calculate the segmentation loss based on the image-text similarity map obtained in step S1, and calculate the classification loss based on the positive / abnormal classification probabilities obtained in step S2; sum the segmentation loss and classification loss with weights, and optimize only the adapter parameters through backpropagation. S4. In zero-sample mode, the image-text similarity map generated in step S1 is upsampled and its resolution is adjusted before being superimposed to generate an initial anomaly heatmap. In few-sample mode, reference samples are introduced, and the cosine similarity map of the reference samples and test samples at each layer of features is calculated. This cosine similarity map is added to the zero-sample anomaly map to form the final heatmap. At the same time, the classification score from step S2 is integrated to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0006] Preferably, in step S1, the feature fine-tuning method in ClearClip is used to insert it into the CLIP visual encoder as a QQ module to achieve preliminary fine-tuning of multi-level features, as detailed below: Remove residual connections from the Transformer layers that extract features; replace the original query-key attention mechanism with a query-query attention mechanism; remove the feedforward network and process the image features obtained from the nth Transformer layer. The corrected image features are obtained by making corrections. .

[0007] Preferably, the image features corrected by the nth Transformer layer for:

[0008] in, For query vector, For value vectors, For projection layer, is the dimension of the key vector.

[0009] Preferably, in step S2, the learnable adapter specifically comprises: For each correction obtained Introducing an image block adapter It contains only one linear connection layer; after adapter processing, the image patch features are obtained after fine-tuning the nth Transformer layer. ; For the category embedding vector output by the visual encoder ViT, the adapter structure is a two-layer fully connected network with a bottleneck structure, which is used to optimize the alignment between image category features and text features to obtain the final fine-tuned category embedding vector.

[0010] Preferably, the final fine-tuned category embedding vector for:

[0011] in, The class embedding vectors output by the CLIP model. This is a trainable adapter used to fine-tune the class embedding vector.

[0012] Preferably, for objects of the same category, the text prompts for both normal and abnormal states under all templates are input into the text encoder of the CLIP model, and their outputs are averaged. The text embedding vector for the normal state is represented as follows: The text embedding vector in an abnormal state is represented as Text embedding vector C represents the number of channels for the text features. This indicates that the vector elements are real values.

[0013] Preferably, in step S3, the classification loss and segmentation loss They are respectively:

[0014]

[0015] in, For binary focus loss, To assign a defect classification score to the image under test, The actual category of the image to be tested. The tags are pixel-level. To extract the probability map corresponding to the abnormal state map as the defect map of the image. To obtain the probability map corresponding to the normal state, For the focus of loss, This is a loss for Dice.

[0016] Preferably, in step S4, the anomaly graph in the small sample mode for:

[0017] in, The anomaly map obtained in zero-sample mode. This is an exception diagram for the reference branch.

[0018] Preferably, refer to the anomaly graph under the branch. for:

[0019] in, The image patch features of the reference image obtained by the Transformer layer in the nth stage. The image patch features of the test image obtained from the nth stage Transformer layer. For image coordinates, Flattening of image patch features of the reference image obtained from the nth Transformer layer. This is the nth stage.

[0020] Secondly, embodiments of the present invention provide a zero-sample / small-sample industrial defect detection system, comprising: The QQ module freezes the CLIP image encoder parameters and extracts image features from different levels. It corrects the features of each layer, projects them through a linear layer, and multiplies each layer's image features element-wise with its corresponding text features to generate a single-layer similarity map. The results of each layer are then stitched together to form an image-text similarity map. The adapter module extracts the final layer global image features of CLIP, multiplies them with the text features, and inputs them into the learnable adapter to output the positive / abnormal classification probability. The loss module calculates the segmentation loss based on the obtained image-text similarity map and the classification loss based on the obtained positive / abnormal classification probabilities; the segmentation loss and classification loss are weighted and summed, and the adapter parameters are optimized only through backpropagation. The output module, in zero-sample mode, overlays the generated image-text similarity map after upsampling and adjusting the resolution to generate an initial anomaly heatmap; in few-sample mode, reference samples are introduced, and the cosine similarity map of the reference sample and the test sample at each layer of features is calculated. This is then added to the zero-sample anomaly map to form the final heatmap. At the same time, the classification score is fused to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0021] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the zero-sample / small-sample industrial defect detection method described above.

[0022] Fourthly, embodiments of the present invention provide a computer-readable storage medium including a computer program, which, when executed by a processor, implements the steps of the above-described zero-sample / small-sample industrial defect detection method.

[0023] Fifthly, a chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the zero-sample / small-sample industrial defect detection method described above.

[0024] In a sixth aspect, embodiments of the present invention provide an electronic device including a computer program, which, when executed by the electronic device, implements the steps of the above-described zero-sample / small-sample industrial defect detection method.

[0025] Compared with the prior art, the present invention has at least the following beneficial effects: A zero-sample / small-sample industrial defect detection method Heatmaps are generated by extracting and correcting multi-layer local features through a segmentation branch, while a classification branch processes global features and outputs classification probabilities. During training, the dual-branch loss is jointly optimized, and during testing, zero-sample and few-sample information are dynamically fused. Heatmaps are generated directly in zero-sample mode without training. For few-sample mode, a cosine similarity map of reference samples is introduced, which can improve localization accuracy with just one sample. Dual-branch collaboration reduces task conflicts.

[0026] Furthermore, in each layer of CLIP: residual connections are removed, "query-key" attention is replaced with "query-query" self-attention, and feedforward networks are deleted; local feature discriminativeness is enhanced, noise is significantly reduced; computational cost is reduced by 30%, making it suitable for real-time industrial detection.

[0027] Furthermore, layer normalization is first performed on each layer of features, and then the corrected features are output through query vector self-attention calculation; layer normalization improves training stability; self-attention sharpens defect edges.

[0028] Furthermore, the segmentation branch uses a single linear layer adapter to optimize local features; the classification branch uses a two-layer fully connected network (including a bottleneck structure) to optimize global features; the linear layer preserves spatial information and improves segmentation accuracy; the bottleneck structure compresses classification noise.

[0029] Furthermore, for the category features of CLIP output: first projected through a linear layer, then activated by a Gaussian error linear unit, and finally output by a quadratic linear projection; the activation function enhances the abnormal response; the bottleneck structure accelerates convergence and improves training efficiency by 2 times.

[0030] Furthermore, a template "A photo of {object state}{object category}" is constructed, and underscores are removed from the category name; the average vector of normal / abnormal states is calculated for the text encoding of all templates for the same object; a unified template covers multiple categories; and mean fusion is used to eliminate prompt bias.

[0031] Furthermore, the classification loss adopts binary focal loss, which focuses on difficult-to-classify samples; the segmentation loss combines focal loss (to solve pixel imbalance) and Dice loss (to optimize contour integrity), and the two losses are weighted and balanced; focal loss improves the detection rate of small defects; Dice loss ensures contour continuity.

[0032] Furthermore, the zero-sample heatmap is added to the hierarchical cosine similarity map of the reference samples to form the final heatmap; cosine similarity provides prior information for normal samples and corrects false detections; it supports dynamic expansion from 1 to 4 samples without retraining.

[0033] Furthermore, for each test sample and each reference sample, the feature cosine similarity is calculated at four levels, and the layers are superimposed to generate a reference map; the multiple levels cover features of different granularities; the similarity between reference samples is averaged to resist illumination interference.

[0034] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0035] In summary, this invention enhances the discriminative power of local features through the QQ module, achieves segmentation-classification collaborative optimization through the dual-branch adapter, and implements a dynamic small sample fusion mechanism, enabling over 90% positioning accuracy with just one reference sample. It significantly reduces the training sample requirement by 90%, supports rapid deployment of multi-category defects in industrial applications, and provides a new paradigm for generalized detection.

[0036] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the CLIP model of the present invention; Figure 2 This is a schematic diagram of the residual attention module under the qk attention mechanism of the present invention; Figure 3 This is a schematic diagram of the QQ module of the present invention; Figure 4 This is a flowchart of the method of the present invention; Figure 5 A schematic diagram illustrating defect localization results with different numbers of reference images; Figure 6 This is a schematic diagram of a strip imaging system; Figure 7 This is a schematic diagram of the machine's operating status; Figure 8 A schematic diagram illustrating the defect detection results of different methods on the strip steel dataset; Figure 9 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 10 This is a block diagram of a chip provided according to an embodiment of the present invention.

[0038] Among them, 60. Computer equipment; 61. Processor; 62. Memory; 63. Computer program; 600. Electronic device; 610. Processing unit; 620. Storage unit; 6201. Random access memory unit; 6202. Cache memory unit; 6203. Read-only memory unit; 6204. Program / utility; 6205. Program module; 630. Bus; 640. Display unit; 650. Input / output interface; 660. Network adapter; 700. External device. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0041] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0042] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0043] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0044] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0045] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0046] This invention provides a zero-shot / few-shot industrial defect detection method. It introduces a multi-level QQ module for initial correction of image patch features and introduces learnable parameters for aligning image patch features with text embeddings, thereby enabling the model to better adapt to defect detection tasks. The method can run in both zero-shot and few-shot modes and has been validated on multiple datasets. Experimental results show that this method is highly competitive in both defect classification and defect segmentation tasks, effectively improving the accuracy and robustness of defect detection.

[0047] Please see Figure 1 The CLIP model is a multimodal model for both image and language processing. It comprises two independent encoders, one for images and one for text. The encoded features contain the core information of both the image and text, facilitating subsequent matching and comparison. Within a contrastive learning framework, CLIP maps visual and textual information to the same space and trains by constructing similarities between images and text, enabling the model to understand the semantic relationships between them. Because the CLIP model is pre-trained on a large dataset containing 400 million data points, it demonstrates excellent generalization ability and performance in tasks such as image classification and object detection without requiring additional training.

[0048] The visual encoder in the CLIP model is based on the Vision Transformer (ViT) architecture and mainly consists of L consecutive residual attention blocks. Given an input image... The image is divided Non-overlapping image patches ( , , P (Representing the resolution of the image patch), in each module, the input is a visual embedding vector. ,in, This represents a category embedding vector that can reflect global information. Let the image patch embedding vector be denoted by . For the first The output of the residual attention module, based on the Transformer's structural representation, is the... l The output of each residual attention module Represented as: (1) (2) (3) (4) in, LN Representation layer normalization, Proj Indicates the projection layer. FFN Represents a feedforward network. represent k Dimensions.

[0049] Because the CLIP model is trained within a visual and linguistic contrastive loss framework, it suffers from insufficient alignment accuracy between local image regions and text representations. For example, defective image regions may score lower than their ground truth labels, while normal regions and the background score higher, resulting in the inclusion of many noise points in the results, which score significantly higher than the surrounding areas. Although introducing an adapter can improve defect detection accuracy, further improvements are needed to enhance the alignment between local image and text features.

[0050] This invention employs the feature fine-tuning method from ClearClip, inserting it as a QQ module into the CLIP visual encoder to achieve initial fine-tuning of multi-level features. Specifically: the parameters of the visual encoder ViT are kept frozen, and the target is set to the image features obtained from the nth Transformer layer. After correction .

[0051] First, remove residual connections in the Transformer layer that extracts features; Then, the original query-key (qk) attention mechanism was replaced with a query-query (qq) attention mechanism; Finally, the feedforward network (FFN) is removed, and the image features corrected for the nth Transformer layer are obtained after this adjustment. .

[0052] The above process is represented as follows: (5) (6) in, For query vector, For value vectors, The projection layer corresponding to the query vector. The projection layer corresponding to the value vector, is the dimension of the key vector.

[0053] Adapter module (1) Image patch embedding vector adapter For each correction obtained An image tile adapter was introduced, denoted as The parameters of this adapter are trainable, and its purpose is to further optimize the alignment between image patch features and text features. This adapter contains only one linear connection layer; after processing by the adapter, the image patch features fine-tuned by the nth Transformer layer... Represented as: (7) (2) Category embedding vector adapter For the category embedding vector ultimately output by the visual encoder ViT, the adapter structure introduced in this invention is a two-layer fully connected network with a bottleneck structure. The parameters of this adapter are trainable, and its purpose is to further optimize the alignment between image category features and text features. This process can be represented as: (8) in, This refers to the final, fine-tuned class embedding vector. This refers to the class embedding vector output by the CLIP model.

[0054] In designing the text prompts, this invention draws inspiration from the APRIL-GAN method. For example, for the template "Aphoto of the {state}{obj}", "state" represents the object's state, such as normal (perfect, flawless, etc.) or abnormal (damaged, broken, etc.); "obj" represents the object's category. The difference is that for categories containing the underscore ("_"), this invention removes the underscore. For objects of the same category, the text prompts for both normal and abnormal states under all templates are input into CLIP's text encoder, and their outputs are averaged. The text embedding vector for the normal state is represented as follows: The text embedding vector in an abnormal state is represented as Therefore, the text embedding vector is represented as Where C represents the number of channels for the text features.

[0055] Please see Figure 4 This invention provides a zero-sample / small-sample industrial defect detection method, comprising the following steps: S1, Zero-sample defect detection S101, Defect Classification In zero-shot detection mode, the class embedding vector, after adapter fine-tuning, is multiplied by the text embedding vector, and then a softmax operation is performed on the result to obtain the probability that the image belongs to a normal or abnormal state. The probability corresponding to the abnormal state is then used as the defect classification score of the image. .

[0056] This process can be represented as: (9) Since a learnable adapter is introduced in the classification branch, the classification loss for this branch during model training is expressed as follows: (10) in, This represents the binary focal loss.

[0057] S102, Defect Segmentation Based on experience, all layers of ViT are divided into four stages to generate anomaly maps for each stage. For the image features output from each stage, the QQ module is used for correction, and a linear layer maps the corrected features to the image-text feature joint space. Then, the image features are multiplied by the text features, resulting in the final anomaly map in zero-shot mode. This is the sum of the anomaly graphs for the four stages.

[0058] (11) in, This means that the obtained anomaly image will be upsampled to the same resolution as the input image.

[0059] Segmentation loss The definition is as follows: (12) Where G represents the pixel-level label, This indicates that the probability map corresponding to the abnormal state map is taken as the defect map of the image. This indicates that the probability map corresponding to the normal state is taken. This represents focal loss. This represents Dice's loss.

[0060] During model training, the total loss is defined as follows: (13) In this invention, .

[0061] S2, Small Sample Defect Detection S201, Defect Segmentation For segmentation tasks in small-sample defect detection, this invention adopts the mainstream detection mode that introduces small samples. For example... Figure 2 As shown, in the small-sample defect detection setup (e.g., 1, 2, or 4 reference samples), for each reference image, a CLIP image encoder is used, parameters are frozen, and image features corresponding to the four stages are extracted. In the testing phase, the cosine similarity between the test image features and the reference image features is calculated at each of the four stages, and these anomaly maps are summed to obtain the anomaly map under the reference branch. Finally, these anomaly maps are added to the anomaly maps from the zero-shot mode to obtain the anomaly maps from the few-shot mode. .

[0062] Refer to the anomaly diagram under the branch for:

[0063] in, The image patch features of the reference image obtained by the Transformer layer in the nth stage. The image patch features of the test image obtained from the nth stage Transformer layer. For image coordinates, Flattening of image patch features of the reference image obtained from the nth Transformer layer. This is the nth stage.

[0064] Anomaly graphs in small sample mode for:

[0065] in, The anomaly map obtained in zero-sample mode. This is an exception diagram for the reference branch.

[0066] S202, Defect Classification The anomaly score of an image consists of two parts: the first part is the anomaly score obtained through text prompts in zero-shot mode; the second part takes into account the maximum value in the anomaly map; the final defect classification score is the sum of these two parts, and the classification score is combined to supplement the anomaly score to form the final heatmap.

[0067] In another embodiment of the present invention, a zero-sample / small-sample industrial defect detection system is provided. This system can be used to implement the above-mentioned zero-sample / small-sample industrial defect detection method. Specifically, the zero-sample / small-sample industrial defect detection system includes a QQ module, an adapter module, a loss module, and an output module.

[0068] The QQ module freezes the CLIP image encoder parameters and extracts image features from different levels. It corrects each layer of features, projects them through a linear layer, and multiplies each layer of image features with its corresponding text features element by element to generate a single-layer similarity map. The results of each layer are then stitched together to form an image-text similarity map. The adapter module extracts the final layer global image features of CLIP, multiplies them with the text features, and inputs them into the learnable adapter to output the positive / abnormal classification probability. The loss module calculates the segmentation loss based on the obtained image-text similarity map and the classification loss based on the obtained positive / abnormal classification probabilities; the segmentation loss and classification loss are weighted and summed, and the adapter parameters are optimized only through backpropagation. The output module, in zero-sample mode, overlays the generated image-text similarity map after upsampling and adjusting the resolution to generate an initial anomaly heatmap; in few-sample mode, reference samples are introduced, and the cosine similarity map of the reference sample and the test sample at each layer of features is calculated. This is then added to the zero-sample anomaly map to form the final heatmap. At the same time, the classification score is fused to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0069] This invention provides a terminal device comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or function. The processor described in this embodiment can be used for the operation of zero-sample / small-sample industrial defect detection methods, including: The CLIP image encoder parameters are frozen, and image features are extracted from different levels. Each layer's features are corrected using the QQ module. After linear layer projection, each layer's image features are multiplied element-wise with its corresponding text features to generate a single-layer similarity map. The results from each layer are then stitched together to form an image-text similarity map. The final layer's global image features of the CLIP are extracted, multiplied with text features, and input into a learnable adapter to output positive / abnormal classification probabilities. Segmentation loss is calculated based on the obtained image-text similarity map, and classification loss is calculated based on the obtained positive / abnormal classification probabilities. The segmentation loss and classification loss are weighted and summed, and backpropagation is used to optimize only the adapter parameters. In zero-shot mode, the generated image-text similarity maps are upsampled and their resolution adjusted before being superimposed to generate an initial abnormality heatmap. In few-shot mode, reference samples are introduced, and the cosine similarity map between the reference sample and the test sample at each layer's features is calculated. This cosine similarity map is added to the zero-shot abnormality map to form the final heatmap. Simultaneously, classification scores are fused to supplement the abnormality score. The final defect classification score is the sum of the abnormality score generated by the text prompt and the maximum value of the abnormality map.

[0070] Please see Figure 9The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the method for estimating the concentration of radioactive iodine species in the containment vessel after an accident, as described in this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the zero-sample / small-sample industrial defect detection system of this embodiment. To avoid repetition, these details are not elaborated here.

[0071] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 9 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0072] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0073] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 60.

[0074] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0075] Please see Figure 10 The terminal device is an electronic device 600, which is manifested in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0076] The storage unit stores program code, which can be executed by the processing unit 610 to perform the steps described in the method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1 The steps are shown in the figure.

[0077] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0078] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0079] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.

[0080] Electronic device 600 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem). This communication can be performed via input / output interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network, wide area network, and / or public network, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0081] Example 4 This invention also provides a storage medium, specifically a computer-readable storage medium, which is a memory device in a terminal device for storing programs and data. It is understood that the computer-readable storage medium here can include both built-in storage media in the terminal device and extended storage media supported by the terminal device; it can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor, which can be one or more computer programs (including program code). More specific examples of the computer-readable storage medium include: an electrical connection with one or more wires, a portable disk, a hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical fiber, portable compact disk read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0082] Computer-readable storage media also include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium can also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium can be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0083] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0084] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the zero-sample / small-sample industrial defect detection method in the above embodiments; one or more instructions in the computer-readable storage medium are loaded by the processor and executed as follows: The CLIP image encoder parameters are frozen, and image features are extracted from different levels. Each layer's features are corrected using the QQ module. After linear layer projection, each layer's image features are multiplied element-wise with its corresponding text features to generate a single-layer similarity map. The results from each layer are then stitched together to form an image-text similarity map. The final layer's global image features of the CLIP are extracted, multiplied with text features, and input into a learnable adapter to output positive / abnormal classification probabilities. Segmentation loss is calculated based on the obtained image-text similarity map, and classification loss is calculated based on the obtained positive / abnormal classification probabilities. The segmentation loss and classification loss are weighted and summed, and backpropagation is used to optimize only the adapter parameters. In zero-shot mode, the generated image-text similarity maps are upsampled and their resolution adjusted before being superimposed to generate an initial abnormality heatmap. In few-shot mode, reference samples are introduced, and the cosine similarity map between the reference sample and the test sample at each layer's features is calculated. This cosine similarity map is added to the zero-shot abnormality map to form the final heatmap. Simultaneously, classification scores are fused to supplement the abnormality score. The final defect classification score is the sum of the abnormality score generated by the text prompt and the maximum value of the abnormality map.

[0085] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0086] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0087] Experimental setup (1) Dataset The MVTec AD and VisA datasets are standard datasets widely used in industrial defect detection, and their sample sets do not overlap. Both datasets are divided into training and testing sets. The training set contains normal samples, while the testing set includes normal samples as well as samples containing various defects and their corresponding labels. The MVTec AD dataset contains 15 categories and a total of 3629 images, while the VisA defect detection dataset contains 12 categories and a total of 10821 images. This invention focuses on building a zero-shot defect detection model. Therefore, to verify the generalization ability of this method, when testing with the MVTec AD dataset, this invention is trained on the VisA test set; conversely, when testing with VisA, this invention is trained on the MVTec AD test set.

[0088] (2) Measurement standards Based on previous research, this invention uses the following evaluation metrics for reporting. In the defect classification task, this invention uses AUROC, AUPR, and F1-max (F1 score at the optimal threshold). In the defect segmentation task, this invention uses pixel-level AUROC, PRO, and pixel-level 1-max.

[0089] (3) Parameter settings The CLIP model used in this invention is the ViT-L / 14@336px version, which contains 24 layers, with each stage consisting of 6 layers. During training, the size of all images is set to... This invention uses the Adam optimizer with a learning rate of 1e-3, a batch size of 16 during training, and utilizes only one GPU (NVIDIA GeForce RTX 3090). To avoid overfitting and improve generalization ability, this invention was trained for 9 epochs on the MVTec AD dataset and 3 epochs on the VisA dataset.

[0090] This invention compares the performance of WinCLIP, APRIL-GAN, PaDiM, and PatchCore in defect classification and segmentation tasks under zero-shot and few-shot modes.

[0091] Table 1. Performance comparison of defect detection on the MVTec AD public dataset.

[0092] Table 2. Performance comparison of defect detection on the VisA public dataset.

[0093] Table 1 shows the comparison results of the proposed method and other methods on MVTec AD. The results show that the proposed method significantly improves performance on defect segmentation tasks. Table 2 shows the comparison results of the proposed method and other methods on the VisA dataset.

[0094] In the zero-sample setting, the method proposed in this invention significantly outperforms other methods in AUROC and PRO for the index segmentation task, and also significantly outperforms other methods in the three indices AUROC, F1-max, and AP for the classification task. In the small-sample setting, the method proposed in this invention achieves better performance in the defect segmentation task, proving the effectiveness of the method.

[0095] Please see Figure 5 The visualization of the detection results is shown. As can be seen from the figure, in zero-shot mode, although the model has never seen the test sample, it can still roughly locate the defect based on the pre-trained detection model's understanding of the defect. However, some defects still cannot be detected. After introducing reference samples, the defect localization becomes more accurate because the model has learned the normal pattern. Furthermore, since the test model on the VisA dataset is trained on the MVTec dataset, and defects in the MVTec AD dataset are typically large, the defect regions detected on VisA are often slightly larger than the true values.

[0096] The QQ module adjusts the original image patch features to enhance the alignment between local regions and text features. This is because the CLIP model was originally designed for classification tasks, and when directly used for defect detection, it may suffer from insufficient perception of local regions.

[0097] This invention extracts image patch features at four levels and adjusts them to better match text features. The adjusted image patch features are input into an adapter and further optimized in a fine-tuning stage to enhance the alignment between image and text features.

[0098] Table 3. Impact of the QQ module on defect detection tasks

[0099] To verify the effectiveness of the QQ module, the present invention evaluated its impact on defect detection performance in Table 3. The experimental results show that the introduction of this module helps to improve detection accuracy, especially in tasks that require precise location of defect areas.

[0100] Application of this invention in practical engineering A steel strip imaging system was built in a Jiangsu enterprise using a CameraLink linear CMOS camera, a high-brightness line light source DH-4LINS700-W, and an image acquisition card of model OR-Y4C0-XMX00. The device design diagram is shown below. Figure 6 As shown, the image when the steel strip is inserted during machine operation is as follows: Figure 7 As shown.

[0101] The acquired images are processed to create a test dataset called strip steel. The dataset is divided into a training set and a test set. The training set consists entirely of normal samples, while the test set contains both normal samples and defective samples. For each defective sample, the defective region is labeled.

[0102] Table 4 Comparison of defect detection performance on the strip steel dataset

[0103] The comparative experimental results of this invention with current mainstream detection algorithms on the strip steel dataset are shown in Table 4, and the schematic diagram of the detection results is shown below. Figure 8 As shown, the model used for testing on this dataset was trained on the MVTec AD dataset.

[0104] from Figure 8 The detection results show that the method proposed in this invention can detect a more complete defect profile compared to the other two methods. Particularly in the second example, this method demonstrates excellent capture capability even for extremely small defects. The quantitative analysis results in Table 4 further demonstrate that this invention outperforms the comparative methods in all detection indicators, fully validating the effectiveness of the method presented in this invention.

[0105] In summary, this invention presents a zero-shot / few-shot industrial defect detection method and system. Based on the CLIP model, it proposes a novel defect detection method to address the challenges of zero-shot and few-shot industrial visual defect detection. This invention proposes a multi-stage QQ module to optimize image patch features and initially reduce defect segmentation noise. Furthermore, this invention introduces an adapter module to map image features to an image-text joint embedding space, enabling comparison with text features and thus achieving more accurate defect detection. In few-shot mode, a reference image is further incorporated to improve defect localization accuracy. Extensive experimental results demonstrate that the method of this invention is highly competitive in zero-shot and few-shot defect detection tasks, effectively improving segmentation accuracy while reducing reliance on large-scale normal samples. The visual-language model has broad application prospects in the field of industrial defect detection and provides new ideas for future research on general and efficient defect detection frameworks.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0109] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random-access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A zero-shot / small-shot industrial defect detection method, characterized in that, Includes the following steps: S1. Freeze the CLIP image encoder parameters and extract image features from different levels; correct each layer of features through the QQ module, project the linear layer, multiply each layer of image features with its corresponding text features element by element to generate a single-layer similarity map, and stitch together the results of each layer to form an image-text similarity map. S2. Extract the final layer global image features of CLIP, multiply them with the text features, input them into the learnable adapter, and output the positive / abnormal classification probability. S3. Calculate the segmentation loss based on the image-text similarity map obtained in step S1, and calculate the classification loss based on the positive / abnormal classification probabilities obtained in step S2; sum the segmentation loss and classification loss with weights, and optimize only the adapter parameters through backpropagation. S4. Zero-sample mode: The image-text similarity map generated in step S1 is upsampled and its resolution is adjusted before being superimposed to generate an initial abnormal heat map. The small sample mode introduces reference samples, calculates the cosine similarity map of the reference samples and test samples at each layer of features, adds it to the zero sample anomaly map to form the final heatmap, and integrates the classification score from step S2 to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

2. The zero-shot / few-shot industrial defect detection method of claim 1, wherein, In step S1, the feature fine-tuning method in ClearClip is used to insert it into the CLIP visual encoder as a QQ module, thereby achieving preliminary fine-tuning of multi-level features, as detailed below: Remove residual connections from the Transformer layers that extract features; replace the original query-key attention mechanism with a query-query attention mechanism; remove the feedforward network and process the image features obtained from the nth Transformer layer. The corrected image features are obtained by making corrections. .

3. The zero-sample / small-sample industrial defect detection method according to claim 2, characterized in that, Image features corrected by the nth Transformer layer for: in, For query vector, For value vectors, For projection layer, is the dimension of the key vector.

4. The zero-sample / small-sample industrial defect detection method according to claim 1, characterized in that, In step S2, the learnable adapter specifically refers to: For each correction obtained Introducing an image block adapter It contains only one linear connection layer; after adapter processing, the image patch features are obtained after fine-tuning the nth Transformer layer. ; For the category embedding vector output by the visual encoder ViT, the adapter structure is a two-layer fully connected network with a bottleneck structure, which is used to optimize the alignment between image category features and text features to obtain the final fine-tuned category embedding vector.

5. The zero-sample / small-sample industrial defect detection method according to claim 4, characterized in that, Final fine-tuned class embedding vector for: in, The class embedding vectors output by the CLIP model. This is a trainable adapter used to fine-tune the class embedding vector.

6. The zero-sample / small-sample industrial defect detection method according to claim 4, characterized in that, For objects of the same category, the text prompts for both normal and abnormal states under all templates are input into the text encoder of the CLIP model, and their outputs are averaged. The text embedding vector for the normal state is represented as follows: The text embedding vector in an abnormal state is represented as Text embedding vector C represents the number of channels for the text features. This indicates that the vector elements are real values.

7. The zero-sample / small-sample industrial defect detection method according to claim 1, characterized in that, In step S3, classification loss and segmentation loss They are respectively: in, For binary focus loss, To assign a defect classification score to the image under test, The actual category of the image to be tested. The tags are pixel-level. To extract the probability map corresponding to the abnormal state map as the defect map of the image. To obtain the probability map corresponding to the normal state, For the focus of loss, This is a loss for Dice.

8. The zero-sample / small-sample industrial defect detection method according to claim 1, characterized in that, In step S4, the anomaly graph in the few-sample mode for: in, The anomaly map obtained in zero-sample mode. The anomaly graph obtained under the reference branch.

9. The zero-sample / small-sample industrial defect detection method according to claim 8, characterized in that, Refer to the anomaly diagram under the branch for: in, The image patch features of the reference image obtained by the Transformer layer in the nth stage. The image patch features of the test image obtained from the nth stage Transformer layer. For image coordinates, Flattening of image patch features of the reference image obtained from the nth Transformer layer. This is the nth stage.

10. A zero-sample / small-sample industrial defect detection system, characterized in that, include: The QQ module freezes CLIP image encoder parameters and extracts image features from different levels. Each layer of features is corrected, and after linear layer projection, each layer of image features is multiplied element by element with its corresponding text features to generate a single-layer similarity map. The results of each layer are then stitched together to form an image-text similarity map. The adapter module extracts the final layer global image features of CLIP, multiplies them with the text features, and inputs them into the learnable adapter to output the positive / abnormal classification probability. The loss module calculates the segmentation loss based on the obtained image-text similarity map and the classification loss based on the obtained positive / abnormal classification probabilities; the segmentation loss and classification loss are weighted and summed, and the adapter parameters are optimized only through backpropagation. The output module, in zero-sample mode, overlays the generated image-text similarity maps after upsampling and resolution adjustment to generate an initial anomaly heatmap. The small sample mode introduces reference samples, calculates the cosine similarity map of the reference samples and test samples at each layer of features, adds it to the zero sample anomaly map to form the final heatmap, and integrates the classification score to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

Images