Zero sample / small sample industrial defect detection method and system

Abstract

The invention discloses a zero sample / small sample industrial defect detection method and system, and the method comprises the steps: freezing the parameters of a CLIP image encoder in a segmentation branch, extracting image features, projecting the corrected features, carrying out the one-by-one product operation of the image features of different layers and corresponding text features, and splicing to generate an image-text similarity graph; in the classification branch, the extracted final image features are multiplied by the text features, and a positive / abnormal classification result of the image is obtained after processing through a learnable adapter; the parameters of the adapter are optimized through the difference between the output result of the minimization segmentation branch and the classification branch and the real label, and the zero sample detection mode directly generates an abnormal graph through the matrix multiplication of the image and the text features among the layers; a reference sample is introduced in a small sample detection mode, the cosine similarity between each layer of features and input sample features is calculated, and an obtained similarity graph and an anomaly graph generated by a zero sample are added to form a final anomaly thermodynamic diagram.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of defect detection, and particularly relates to a zero-shot / small-shot industrial defect detection method and system. BACKGROUND

[0002] Visual defect detection has wide application and important value in industrial manufacturing, mainly including two core tasks of defect classification and defect segmentation. Defect classification aims to judge whether the target object has defects, while defect segmentation further determines the pixel-level position of the defects to provide accurate defect region information. At present, most defect detection methods adopt an unsupervised learning paradigm, that is, a model is trained on a large number of normal images, and then the model is used to identify defect samples. This is because the defects of objects usually have great variation in shape, color, texture and size, and it is difficult to define all possible defect patterns. Although this method has made significant progress in surface defect detection, it usually needs to train a model for each object, and in practical applications, the cost of collecting a large number of normal samples and deploying an independent model for each object is extremely high.

[0003] Therefore, how to construct a general model for surface defect detection of multiple objects without a large number of normal images or relying on only a small number of normal samples is an important problem to be solved. The contrastive language-image pre-training (CLIP) model is the first model jointly trained on image and text data. Since the CLIP model is pre-trained on a large dataset containing 400 million data, it can exhibit excellent generalization ability and performance in image classification, object detection and other tasks without additional training. Therefore, using the CLIP framework to complete the zero-shot / small-shot defect detection model is the current mainstream method. Since CLIP is mainly designed for classification tasks, it still faces many challenges in semantic segmentation tasks. The image-text contrast training paradigm adopted by CLIP emphasizes global features, but ignores local discriminability, resulting in false segmentation results and noisy segmentation maps. In the field of defect detection, it will generate defect localization maps opposite to the defect results, thereby reducing the detection accuracy. SUMMARY

[0004] The technical problem to be solved by the application is to provide a zero-shot / small-shot industrial defect detection method and system to solve the technical problem of insufficient local discriminability caused by global feature preference of the CLIP model in industrial defect detection, in view of the deficiencies in the prior art.

[0005] The application adopts the following technical solutions: A zero-shot / small-shot industrial defect detection method, comprising the following steps: S1, freeze the CLIP image encoder parameters, extract image features from different levels; correct each layer of features through the QQ module, multiply each layer of image features with its corresponding text features after linear layer projection, generate a single layer of similar images, and splice the results of each layer to form a text-image similarity graph; S2, extract the global image features of the final layer of CLIP, multiply them with the text features, and input them into the learnable adapter to output the normal / abnormal classification probability; S3, calculate the segmentation loss based on the text-image similarity graph obtained in step S1, and calculate the classification loss based on the normal / abnormal classification probability obtained in step S2; weight the segmentation loss and the classification loss and sum them up, and only optimize the adapter parameters through back propagation; S4, in the zero-shot mode, the text-image similarity graph generated in step S1 is adjusted in resolution after upsampling and superimposed to generate an initial anomaly heat map; in the small sample mode, a reference sample is introduced, and the cosine similarity graph of the reference sample and the test sample in each layer of features is calculated, and the final heat map is formed by adding the zero-shot anomaly graph, and the classification score in step S2 is fused to supplement the anomaly score, and the final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly graph.

[0006] Preferably, in step S1, the feature fine-tuning method in clearclip is used to insert it into the visual encoder of CLIP in the form of a QQ module to realize preliminary fine-tuning of multi-level features, which is as follows: Remove the residual connection in the Transformer layer that extracts the feature; replace the original query-key attention mechanism with a query-query attention mechanism; remove the feedforward network, and modify the image feature obtained by the nth Transformer layer to obtain the modified image feature .

[0007] Preferably, the modified image feature of the nth Transformer layer is .

[0008] wherein, is the query vector, is the value vector, is the projection layer, is the dimension of the key vector.

[0009] Preferably, in step S2, the learnable adapter is specifically: For each modified , an image block adapter is introduced, which only contains a linear connection layer; after processing by the adapter, the image block feature of the nth Transformer layer after fine-tuning is obtained ; For the category embedding vector output by the visual encoder ViT, the adapter structure is a two-layer fully connected network with a bottleneck structure, which is used to optimize the alignment between image category features and text features to obtain the final fine-tuned category embedding vector.

[0010] Preferably, the final fine-tuned category embedding vector is:

[0011] wherein, is the category embedding vector output by the CLIP model, is a trainable adapter used to fine-tune the category embedding vector.

[0012] Preferably, for objects of the same category, the text prompt words of all templates under normal and abnormal states are respectively input into the text encoder of the CLIP model, and the outputs are respectively averaged, the text embedding vector under the normal state is represented as , and the text embedding vector under the abnormal state is represented as , the text embedding vector , C represents the number of channels of the text features, represents that the vector element is a real value.

[0013] Preferably, in step S3, the classification loss and the segmentation loss are respectively:

[0014]

[0015] wherein, is the binary focal loss, is the defect classification score of the image to be tested, is the actual category of the image to be tested, is the pixel-level label, is to take the abnormal state image corresponding probability map as the defect map of the image, is to take the normal state corresponding probability map, is the focal loss, is the Dice loss.

[0016] Preferably, in step S4, the abnormal image under the small sample mode is:

[0017] wherein, is the abnormal image obtained under the zero sample mode, For reference under the branch of the anomaly map.

[0018] Preferably, the anomaly map under the reference branch is For:

[0019] Wherein, is the image block feature of the reference image obtained by the nth stage of the Transformer layer, is the image block feature of the test image obtained by the nth stage of the Transformer layer, is the image coordinate, is the flattening of the image block feature of the reference image obtained by the nth Transformer layer, is the nth stage.

[0020] In a second aspect, an embodiment of the present application provides an industrial defect detection system with zero-shot / small-shot, comprising: A QQ module, which freezes the CLIP image encoder parameters, extracts image features from different levels, corrects each layer of features, and after projection by a linear layer, multiplies each layer of image features with its corresponding text features element by element to generate a single-layer similarity map, and splices the results of each layer to form a graph-text similarity map; An adapter module, which extracts the global image features of the final layer of CLIP, multiplies them with the text features, and inputs them into a learnable adapter to output positive / abnormal classification probabilities; A loss module, which calculates a segmentation loss based on the obtained graph-text similarity map, and calculates a classification loss based on the obtained positive / abnormal classification probabilities; and adds the segmentation loss and the classification loss to obtain a weighted sum, and only optimizes the adapter parameters through backpropagation; An output module, which, in zero-shot mode, superimposes the generated graph-text similarity map after adjusting the resolution through upsampling to generate an initial anomaly heat map; and in small-shot mode, introduces a reference sample, calculates the cosine similarity map of the reference sample and the test sample at each layer of features, adds it to the zero-shot anomaly map to form a final heat map, and simultaneously fuses the classification score to supplement the anomaly score, so that the final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0021] In a third aspect, a computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned zero-shot / small-shot industrial defect detection method when executing the computer program.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned zero-shot / small-shot industrial defect detection method.

[0023] In a fifth aspect, a chip is provided, which includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the zero-shot / small-shot industrial defect detection method when the computer program is executed.

[0024] In a sixth aspect, an electronic device is provided, which includes a computer program, and the computer program implements the steps of the zero-shot / small-shot industrial defect detection method when executed by the electronic device.

[0025] Compared with the prior art, the present application has at least the following beneficial effects: A zero-shot / small-shot industrial defect detection method, The local features of multiple layers are extracted and corrected by the segmentation branch to generate a heat map, the global features are processed by the classification branch to output a classification probability, the double-branch loss is jointly optimized during training, and the zero-shot and small-shot information is dynamically fused during testing; in the zero-shot mode, the heat map is directly generated without training; in the small-shot mode, a reference sample cosine similarity graph is introduced, and the positioning accuracy is improved by one sample; and the task conflict is reduced by the cooperation of the double branches.

[0026] Further, in each layer of the CLIP: the residual connection is removed, the "query-key" attention is replaced by "query-query" self-attention, and the feedforward network is deleted; the discriminability of the local features is enhanced, and the noise is significantly reduced; the computational complexity is reduced by 30%, and the industrial real-time detection is adapted.

[0027] Further, the features of each layer are first normalized, and then the corrected features are output by calculating the query vector self-attention; the layer normalization improves the training stability; and the self-attention sharpens the defect edges.

[0028] Further, the segmentation branch uses a single linear layer adapter to optimize the local features; the classification branch uses a two-layer fully connected network (including a bottleneck structure) to optimize the global features; the linear layer retains spatial information and improves the segmentation accuracy; and the bottleneck structure compresses the classification noise.

[0029] Further, for the category features output by the CLIP: the features are first projected by a linear layer, then activated by a Gaussian error linear unit, and finally output by a second linear projection; the activation function strengthens the abnormal response; the bottleneck structure accelerates the convergence, and the training efficiency is improved by 2 times.

[0030] Further, a template "A photo of {object state} {object category}" is constructed, and the underscores in the category name are removed; the average vectors of the normal / abnormal states are calculated respectively for the text encoding of all templates of the same object; the templates are unified to cover multiple categories; and the mean fusion eliminates the prompt bias.

[0031] Further, the classification loss adopts a binary focal loss, focusing on difficult-to-classify samples; the segmentation loss combines a focal loss (to solve pixel imbalance) and a Dice loss (to optimize contour integrity), and the two losses are weighted and balanced; the focal loss improves the detection rate of small defects; and the Dice loss ensures contour continuity.

[0032] Further, the zero sample heat map is added to the hierarchical cosine similarity graph of the reference sample to form a final heat map; the cosine similarity provides a normal sample prior, corrects false positives; supports 1→4 sample dynamic expansion without retraining.

[0033] Further, the cosine similarity of the features is calculated for each reference sample and the test sample at four levels, respectively, and the reference graph is generated by hierarchical superposition; multiple levels cover different granularity features; the similarity between reference samples is averaged to resist light interference.

[0034] It can be understood that the beneficial effects of the above-mentioned second aspect to the sixth aspect can be referred to the related description in the first aspect, which will not be repeated here.

[0035] In summary, the application enhances the discriminability of local features by the QQ module, realizes segmentation-classification collaborative optimization by the double-branch adapter, and realizes a dynamic small sample fusion mechanism, so that 1 reference sample can achieve a positioning accuracy of more than 90%; greatly reduces the requirement of 90% training samples, supports rapid deployment of industrial multi-class defects, and provides a new paradigm for general detection.

[0036] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 It is a CLIP model schematic diagram of the application; Figure 2 It is a residual attention module schematic diagram under the q-k attention mechanism of the application; Figure 3 It is a QQ module schematic diagram of the application; Figure 4 It is a method flowchart of the application; Figure 5 It is a defect positioning result display schematic diagram under different numbers of reference images; Figure 6 It is a strip steel imaging system schematic diagram; Figure 7 It is a machine running state schematic diagram; Figure 8 It is a defect detection result display schematic diagram of different methods on the strip steel data set; Figure 9 It is a schematic diagram of a computer device provided by an embodiment of the application; Figure 10 Figure 1 is a block diagram of a chip according to an embodiment of the present application.

[0038] In the figure, 60 is a computer device; 61 is a processor; 62 is a memory; 63 is a computer program; 600 is an electronic device; 610 is a processing unit; 620 is a storage unit; 6201 is a random access storage unit; 6202 is a cache storage unit; 6203 is a read-only storage unit; 6204 is a program / utility; 6205 is a program module; 630 is a bus; 640 is a display unit; 650 is an input / output interface; 660 is a network adapter; and 700 is an external device. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0040] In the description of the present application, it should be understood that the terms “comprise” and “include” indicate the presence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0041] It should also be understood that the terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clearly indicated by the context, the singular forms “a”, “an” and “the” are intended to include the plural forms as well.

[0042] It should be further understood that the term “and / or” used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character “ / ” in the present application generally represents an “or” relationship between the front and rear associated objects.

[0043] It should be understood that, although the terms first, second, third, etc. can be employed in describing the preset ranges, etc. in the embodiments of the present application, the preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range, without departing from the scope of the embodiments of the present application.

[0044] Depending on the context, the word "if" as used herein can be interpreted to mean "when" or "while" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be interpreted to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0045] Various structural diagrams according to the disclosed embodiments of the present application are shown in the accompanying drawings. These diagrams are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and precision, and certain details can be omitted. The shapes of various regions, layers, and their relative sizes and positional relationships shown in the diagrams are only exemplary, and in actuality, they can deviate due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0046] The present application provides a zero-shot / small-shot industrial defect detection method, which introduces a multi-level QQ module to realize preliminary correction of image block features, and introduces learnable parameters for alignment of image block features and text embedding, so that the model better adapts to the defect detection task; it can run in zero-shot and small-shot modes, and is verified on multiple datasets. Experimental results show that this method has strong competitiveness in defect classification and defect segmentation tasks, and can effectively improve the precision and robustness of defect detection.

[0047] Please refer to Figure 1 , the CLIP model is a multimodal model of image and language, which contains two independent encoders, one for image and one for text. The encoded features contain the core information of image and text, which is convenient for subsequent matching and comparison. Under the contrast learning framework, CLIP maps visual and textual information to the same space and trains by constructing the similarity between image and text, so that the model can understand the semantic association between image and text. Since the CLIP model is pre-trained on a large dataset containing 400 million data, it does not need additional training and can exhibit excellent generalization ability and performance in image classification, object detection and other tasks.

[0048] The visual encoder in the CLIP model is based on the Vision Transformer (ViT) architecture, mainly composed of L consecutive residual attention blocks. The image is divided non-overlapping image blocks , , P represent the resolution of the image block), in each module, the input is a visual embedding vector , where represents a category embedding vector that can reflect global information, represents an image block embedding vector, and let be the output of the th residual attention block, according to the structure representation of the Transformer, the output of the l th residual attention block is represented as: (1) (2) (3) (4) where LN denotes layer normalization, Proj denotes a projection layer, FFN denotes a feedforward network, denotes k dimension.

[0049] Since the CLIP model is trained in the contrastive loss framework of vision and language, this leads to insufficient alignment accuracy between local image regions and text representations. For example, defective image regions may be opposite to the true value label, and the scores of defective regions are often low, while the scores of normal regions and backgrounds are high, resulting in many noise points in the results, and the scores of these noise points are significantly higher than those of the surrounding regions. Although the introduction of an adapter can improve the defect detection accuracy, in order to further improve the alignment between local images and text features.

[0050] The present application adopts the feature fine-tuning method in clearclip, which is inserted into the visual encoder of CLIP in the form of a QQ module to realize preliminary fine-tuning of multi-level features. Specifically: keep the parameters of the visual encoder ViT frozen, and set the target to correct the image features obtained by the th Transformer layer .

[0051] Firstly, remove the residual connection in the Transformer layer of the extracted feature; Then replace the original query-key (q-k) attention mechanism with a query-query (q-q) attention mechanism; Finally, remove the feed-forward network (FFN), and after this adjustment, the modified image feature of the nth Transformer layer is obtained .

[0052] The above process is represented as follows: (5) (6) wherein, is the query vector, is the value vector, is the projection layer corresponding to the query vector, is the projection layer corresponding to the value vector, is the dimension of the key vector.

[0053] Adapter module (1) Image block embedding vector adapter For each modified , an image block adapter is introduced, denoted as The parameters of the adapter are trainable, and the purpose is to further optimize the alignment between the image block feature and the text feature. The adapter only contains a linear connection layer; after the adapter processing, the image block feature of the nth Transformer layer after fine-tuning is represented as: (7) (2) Category embedding vector adapter For the category embedding vector output by the visual encoder ViT, the adapter structure introduced by the present application is a two-layer full-link network containing a bottleneck structure. The parameters of the adapter are trainable, and the purpose is to further optimize the alignment between the image category feature and the text feature. This process is represented as: (8) wherein, refers to the final fine-tuned category embedding vector, refers to the category embedding vector output by the CLIP model.

[0054] In the design of text prompt words, the application learns from the method in APRIL-GAN. For example, for the template "A photo of the {state}{obj}", where "state" represents the state of the object, such as normal (perfect, flawless, etc.) or abnormal (damaged, broken, etc.); "obj" represents the category of the object, except that for categories containing "_" symbols, the application removes the "_". For objects of the same category, input the text prompt words of all templates under normal and abnormal states into the text encoder of CLIP, and average the outputs respectively. The text embedding vector under the normal state is represented as , and the text embedding vector under the abnormal state is represented as , so the text embedding vector is represented as , where C represents the number of channels of the text feature.

[0055] Referring to Figure 4 , the application is a zero-shot / small-shot industrial defect detection method, comprising the following steps: S1, zero-shot defect detection S101, defect classification In the zero-shot detection mode, the class embedding vector after fine-tuning by the adapter is multiplied by the text embedding vector, and then a Softmax operation is performed on the result to obtain the probability that the image belongs to the normal or abnormal state, and then the probability corresponding to the abnormal state is used as the defect classification score of the image .

[0056] The process is represented as: (9) Since the learnable adapter is introduced in the classification branch, the classification loss of this branch during model training is represented as follows: (10) where represents the binary focal loss (Binary Focal Loss).

[0057] S102, defect segmentation According to experience, all layers of ViT are divided into four stages in order to generate abnormal images of different stages. For the image features output by each stage, the QQ module is used for correction, and a linear layer is used to map the corrected features to the image-text feature joint space; then the image features are multiplied by the text features; and finally the abnormal image under the zero-shot mode is the sum of the four-stage abnormal images.

[0058] (11) in, It means that the obtained anomaly map is adjusted to the same resolution as the input image through upsampling operation.

[0059] Segmentation loss The definition is as follows: (12) Among them, G represents the pixel-level label, Indicates that the probability map corresponding to the abnormal state map is taken as the defect map of the image, It means taking the probability map corresponding to the normal state, stands for Focal Loss, Represents Dice loss.

[0060] During model training, the total loss is defined as follows: (13) In the present invention, .

[0061] S2. Small sample defect detection S201, Defect Segmentation For the segmentation task in small sample defect detection, the present invention adopts the mainstream introduction of small sample detection mode. Figure 2 As shown in the figure, in the setting of small sample defect detection (for example, the number of reference samples is 1, 2 or 4), for each reference image, the CLIP image encoder is used, the parameters are kept frozen, and then the image features corresponding to the four stages are extracted. In the test phase, the cosine similarity between the test image features and the reference image features in the four stages is calculated respectively, and the obtained anomaly maps are summed to obtain the anomaly map under the reference branch. Finally, these anomaly maps are added to the anomaly map in zero-sample mode to obtain the anomaly map in small-sample mode. .

[0062] Anomaly graph under reference branch for:

[0063] in, is the image block feature of the reference image obtained by the Transformer layer of the nth stage, is the image block feature of the test image obtained by the Transformer layer of the nth stage, are image coordinates, is the flattened image block feature of the reference image obtained by the nth Transformer layer, is the nth stage.

[0064] Anomaly map under small sample mode is:

[0065] wherein, is the anomaly map obtained under the zero sample mode, is the anomaly map under the reference branch.

[0066] S202, defect classification The anomaly score of the image is composed of two parts: the first part is the anomaly score obtained by the text prompt guidance under the zero sample mode; the second part considers the maximum value in the anomaly map; and the final defect classification score is the sum of the two parts, and the classification score is fused to supplement the anomaly score to form the final heat map.

[0067] In another embodiment of the present application, a zero sample / small sample industrial defect detection system is provided, which can be used to implement the above-mentioned zero sample / small sample industrial defect detection method. Specifically, the zero sample / small sample industrial defect detection system comprises a QQ module, an adapter module, a loss module, and an output module.

[0068] The QQ module freezes the CLIP image encoder parameters, extracts image features from different levels; corrects each layer of features, and after projection by a linear layer, multiplies each layer of image features with its corresponding text features element by element to generate a single-layer similarity map, and splices the results of each layer to form a text-image similarity map; The adapter module extracts the final layer global image feature of CLIP, multiplies it with the text feature, and inputs it into the learnable adapter to output the positive / abnormal classification probability; The loss module calculates the segmentation loss based on the obtained text-image similarity map, and calculates the classification loss based on the obtained positive / abnormal classification probability; the segmentation loss and the classification loss are weighted and summed, and only the adapter parameters are optimized through back propagation; The output module superimposes the generated text-image similarity map after adjusting the resolution by upsampling to generate an initial anomaly heat map under the zero sample mode; the reference sample is introduced under the small sample mode, the cosine similarity map of the reference sample and the test sample at each layer of features is calculated, and the final heat map is formed by adding the zero sample anomaly map, and the classification score is fused to supplement the anomaly score, and the final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0069] The application provides a terminal device, which comprises a processor and a memory for storing a computer program comprising program instructions, and the processor is used for executing the program instructions stored in the computer storage medium. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and the like, which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or a corresponding function; the processor in the embodiments of the application can be used for the operation of the zero-sample / small-sample industrial defect detection method, comprising: Freeze the CLIP image encoder parameters, extract the image features from different levels; correct each layer of features through the QQ module, multiply each layer of image features and its corresponding text features element by element after projection through the linear layer to generate a single-layer similarity map, splice the results of each layer to form a picture-text similarity map; extract the global image features of the final layer of CLIP, multiply them with the text features and input them into the learnable adapter after multiplication, and output the normal / abnormal classification probability; calculate the segmentation loss based on the obtained picture-text similarity map, and calculate the classification loss based on the obtained normal / abnormal classification probability; weight the segmentation loss and the classification loss and sum them, and only optimize the adapter parameters through back propagation; in the zero-sample mode, the generated picture-text similarity map is superimposed after adjusting the resolution through upsampling to generate an initial abnormal heat map; in the small-sample mode, a reference sample is introduced, the cosine similarity map of the reference sample and the test sample in each layer of features is calculated, and the final heat map is formed by adding the zero-sample abnormal map, and the classification score is fused to supplement the abnormal score, and the final defect classification score is the sum of the abnormal score generated by the text prompt and the maximum value of the abnormal map.

[0070] Please refer to Figure 9The terminal device is a computer device. The computer device 60 in this embodiment includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and capable of running on the processor 61. When the computer program 63 is executed by the processor 61, the method for estimating the concentration of radioactive iodine species in the post-accident containment in the embodiment is implemented. To avoid repetition, details are not described here. Alternatively, when the computer program 63 is executed by the processor 61, the functions of each model / unit in the zero-sample / small-sample industrial defect detection system in the embodiment are implemented. To avoid repetition, details are not described here.

[0071] The computer device 60 can be a desktop computer, a notebook computer, a palm computer, a cloud server, or the like. The computer device 60 can include, but is not limited to, the processor 61 and the memory 62. Those skilled in the art can understand that the computer device 60 can include more or fewer components, or some components can be combined, or different components can be included, for example, the computer device can also include an input / output device, a network access device, a bus, and the like. Figure 9 The computer device 60 is only an example and does not constitute a limitation on the computer device 60, and can include more or fewer components than those shown, or some components can be combined, or different components can be included, for example, the computer device can also include an input / output device, a network access device, a bus, and the like.

[0072] The processor 61 can be a central processing unit (CPU), and can also be other general-purpose processors, graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0073] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or a memory of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like.

[0074] Further, the memory 62 can include both a volatile memory unit and a nonvolatile memory unit. The memory 62 can also include a removable memory unit. The memory 62 is utilized by the computer device 60 to store data and programs that are utilized by the computer device 60. The memory 62 can also be used to temporarily store data that is being outputted or to be outputted.

[0075] Referring to Figure 10 , the terminal device is an electronic device 600 in the form of a general purpose computing device. Components of the electronic device can include, but are not limited to, at least one processing unit 610, at least one memory unit 620, a bus 630 that connects the various platform components including the memory unit 620 and the processing unit 610, a display unit 640, and the like.

[0076] The memory unit stores program code that can be executed by the processing unit 610 such that the processing unit 610 performs the steps in accordance with the various example embodiments of the present application as described in the method portion of the specification. For example, the processing unit 610 can perform the steps as shown in Figure 1 .

[0077] The memory unit 620 can include a read-only memory (ROM) 6203 in the form of a nonvolatile memory unit, as well as a random access memory (RAM) 6201 and / or a cache memory unit 6202 in the form of a volatile memory unit that can be supplemented by and / or supplemented as a nonvolatile memory unit.

[0078] The memory unit 620 can further include a program / utility 6204 having a set (at least one) of program modules 6205 that include, but are not limited to, an operating system, one or more applications, other program modules, and program data, and each or a combination thereof can include implementation of a network environment.

[0079] The bus 630 can be representative of one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or local bus using any of a variety of bus structures, and the like.

[0080] The electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, or a database, and / or one or more devices that enable a user to interact with the electronic device 600 and / or one or more devices (e.g., routers, modems) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication can occur via an input / output interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or the Internet, through a network adapter 660. The network adapter 660 can be any of a plurality of different types of adapters suitable for interfacing the electronic device 600 to various networking schemes. It should be appreciated that, while not shown explicitly, other hardware and / or software modules could be used in connection with the electronic device 600 including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0081] Embodiment 4 The present application also provides a storage medium, specifically a computer readable storage medium, which is a memory device in the terminal device, and is used to store programs and data. It should be understood that the computer readable storage medium herein can include the built-in storage medium in the terminal device, and of course can also include the expansion storage medium supported by the terminal device, and can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or apparatus. The computer readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that more specific examples of the computer readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0082] Computer readable storage media further includes data signals transported through a carrier wave and a communications medium, including or excluding wired storage media.

[0083] The program code can be executed by using any combination of one or more programming languages including an object oriented programming language such as Java, C++ or the like, and a conventional procedural programming language such as a "C" language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network ("LAN"), or a wide area network ("WAN"), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider ("ISP").

[0084] The one or more instructions stored in the computer readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the zero-shot / small-shot industrial defect detection method in the above embodiments; the one or more instructions stored in the computer readable storage medium are loaded and executed by the processor to implement the following steps: The CLIP image encoder parameters are frozen and image features are extracted from different layers. The features of each layer are corrected through the QQ module. After linear layer projection, the image features of each layer are multiplied element-by-element with their corresponding text features to generate a single-layer similarity map. The results of each layer are spliced ​​to form an image-text similarity map. The global image features of the final layer of CLIP are extracted, multiplied with the text features, and then input into the learnable adapter to output the positive / abnormal classification probability. The segmentation loss is calculated based on the obtained image-text similarity map, and the classification loss is calculated based on the obtained positive / abnormal classification probabilities. The segmentation loss and classification loss are weighted and summed, and only the adapter parameters are optimized through backpropagation. The zero-shot mode superimposes the generated image-text similarity map after upsampling and adjusting the resolution to generate an initial anomaly heat map. The small-shot mode introduces a reference sample, calculates the cosine similarity map of the reference sample and the test sample at each layer, and adds it to the zero-shot anomaly map to form the final heat map. At the same time, the classification score is fused to supplement the anomaly score. The final defect classification score is the sum of the anomaly score generated by the text prompt and the maximum value of the anomaly map.

[0085] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0086] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0087] Experimental setup (1) Dataset MVTec AD dataset and VisA dataset are standard datasets widely used in industrial defect detection, and the sample sets of the two are mutually exclusive. Both datasets are divided into training set and test set, wherein the training set contains normal samples, and the test set contains samples containing various defects and their corresponding labels in addition to normal samples. MVTec AD dataset contains 15 categories, a total of 3629 pictures, while VisA defect detection dataset contains 12 categories, a total of 10821 pictures. The present application mainly focuses on the construction of zero-shot defect detection model, therefore in order to verify the generalization ability of the present method, when using MVTec AD dataset for testing, the present application is trained on VisA test set; conversely, when VisA is tested, the present application is trained on MVTec AD test set.

[0088] (2) Measurement standards According to previous studies, the following evaluation indicators are used for reporting in the present application. In the defect classification task, the present application uses AUROC, AUPR and F1-max (F1 score under the best threshold). In the defect segmentation task, the present application uses pixel-level AUROC, PRO and pixel-level 1-max.

[0089] (3) Parameter settings The CLIP model used in the present application is the ViT-L / 14@336px version, which contains a total of 24 layers, wherein every 6 layers constitute a stage. In the training process, the size of all images is set to The present application uses Adam optimizer, the learning rate is set to 1e-3, the batch size during training is 16, and only 1 GPU (NVIDIA GeForce RTX 3090) is used. In order to avoid model overfitting and improve generalization ability, when training on MVTec AD dataset, the present application trains for 9 epochs, while training on VisA dataset for 3 epochs.

[0090] The present application compares the performance of defect classification and segmentation tasks under zero-shot and small sample mode with WinCLIP, APRIL-GAN, PaDiM, and PatchCore.

[0091] Table 1 Comparison of defect detection performance on MVTec AD public dataset

[0092] Table 2 Comparison of defect detection performance on VisA public dataset

[0093] Table 1 shows the comparison results of the proposed method and other methods on MVTec AD. From the results, it can be seen that the proposed method has improved performance in the defect segmentation task; Table 2 shows the comparison results of the proposed method and other methods on the VisA dataset.

[0094] In the zero-shot setting, the proposed method significantly outperforms other methods in the AUROC and PRO of the index segmentation task, and also significantly outperforms other methods in the three indexes AUROC, F1-max, and AP of the classification task; in the small sample setting, the proposed method achieves better performance in the defect segmentation task, proving the effectiveness of the method.

[0095] Please refer to Figure 5 , which shows the visualization results of the detection. From the figure, it can be seen that in the zero-shot mode, although the model has never seen the test sample, it can still roughly locate the defect position relying on the understanding of the pre-trained detection model for defects. However, some defects cannot be detected. After introducing the reference sample, since the model has learned the normal mode, the defect positioning is more accurate. In addition, since the test model on the VisA dataset is trained based on the MVTec dataset, and the defects in the MVTec AD dataset are usually larger, the detected defect area on VisA is usually slightly larger than the true value.

[0096] The QQ module adjusts the original image block features, aiming to enhance the alignment ability of local regions with text features. This is because the CLIP model is originally designed for classification tasks, and when directly used for defect detection, there may be a problem of insufficient perception of local regions.

[0097] The present application extracts four levels of image block features and adjusts them to make them more suitable for matching with text features. The adjusted image block features are input into the adapter and further optimized in the fine-tuning stage to enhance the alignment ability between image and text features.

[0098] Table 3 Effect of QQ module on defect detection task

[0099] To verify the effectiveness of the QQ module, the present application evaluates its impact on defect detection performance in Table 3, and the experimental results show that the introduction of this module helps to improve detection accuracy, especially in tasks that require precise positioning of defect areas.

[0100] Application of the present application in practical engineering A strip steel imaging system was built by a Jiangsu enterprise using a CameraLink linear array CMOS camera, a high-brightness line light source DH-4LINS700-W, and an image acquisition card with model OR-Y4C0-XMX00, and a device design diagram is shown in Figure 6 When the machine is running, the image when the steel strip is penetrated is shown in Figure 7 .

[0101] The acquired image is processed to establish a to-be-tested data set strip steel, and the data set is divided into a training set and a test set, wherein the training set is all normal samples, and the test set contains normal samples and defect samples, and for each defect sample, the annotation of the defect area is provided.

[0102] Table 4 Comparison of defect detection performance on strip steel data set

[0103] The comparison experimental results of the present application on the strip steel data set with the current mainstream detection algorithm are shown in Table 4, and a detection result schematic diagram is shown in Figure 8 The model used for testing the data set is trained on the MVTec AD data set.

[0104] From the detection results of Figure 8 It can be seen that the method proposed in the present application can detect more complete defect contours compared with the other two methods. In particular, in the second example, the method also shows good capture ability for extremely small defects. The quantitative analysis results in Table 4 further show that the present application is superior to the comparison methods in all detection indicators, fully verifying the effectiveness of the method of the present application.

[0105] In summary, the zero sample / small sample industrial defect detection method and system of the present application proposes a new defect detection method based on the CLIP model to cope with the challenges of zero sample and small sample industrial visual defect detection; the present application proposes a multi-stage QQ module to optimize the image block features to preliminarily reduce the segmentation noise of defects; in addition, the present application introduces an adapter module to map the image features to an image-text joint embedding space, so that it can be compared with the text features, thereby realizing more accurate defect detection. In the small sample mode, further combined with the reference image to improve the defect positioning accuracy; a large number of experimental results show that the method of the present application has strong competitiveness in the zero sample and small sample defect detection task, not only effectively improves the segmentation accuracy, but also reduces the dependence on large-scale normal samples. The visual-linguistic model has broad application prospects in the field of industrial defect detection, and provides a new idea for future research on general and efficient defect detection framework.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0107] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0108] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0109] In the embodiments provided by the present application, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the above-described device / terminal embodiments are only schematic. For example, the division of the modules or units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed units can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0110] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0111] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0112] The integrated module / unit, if implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the computer readable medium can include or exclude contents according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0113] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices, and computer program products of embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure One The device that implements the functions specified in one flow or multiple flows and / or blocks Figure One The device that implements the functions specified in one flow or multiple flows and / or blocks

[0114] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure Oneone or more processes and / or blocks Figure One the function specified in the one or more blocks.

[0115] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate a computer-implemented process, so that the instructions executed on the computer or other programmable data processing devices provide a process for implementing the flow Figure One one or more processes and / or blocks Figure One the function specified in the one or more blocks.

[0116] The above is only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the claims of the present application.

Claims

1. A zero-shot / small-shot industrial defect detection method, characterized in that, The method comprises the following steps: S1, freeze the CLIP image encoder parameters, extract image features from different levels; correct each layer of features through the QQ module, multiply each layer of image features with the corresponding text features element by element after linear layer projection, generate a single layer of similarity map, and splice the results of each layer to form a text-image similarity map; S2, extract the global image features of the final layer of CLIP, multiply them with the text features, and input them into a learnable adapter to output positive / abnormal classification probability; S3, calculate the segmentation loss based on the text-image similarity map obtained in step S1, and calculate the classification loss based on the positive / abnormal classification probability obtained in step S2; weight and sum the segmentation loss and the classification loss, and only optimize the adapter parameters through back propagation; S4, in the zero-shot mode, superimpose the text-image similarity map generated in step S1 after adjusting the resolution through upsampling to generate an initial abnormal heat map; In the few-shot mode, a reference sample is introduced, the cosine similarity map of the reference sample and the test sample in each layer of features is calculated, and the final heat map is formed by adding the zero-shot abnormal map, and the classification score is fused to supplement the abnormal score, and the final defect classification score is the sum of the abnormal score generated by the text prompt and the maximum value of the abnormal map.

2. The zero-shot / few-shot industrial defect detection method of claim 1, wherein, In step S1, the feature fine-tuning method in clearclip is used to insert the QQ module into the visual encoder of CLIP to realize the preliminary fine-tuning of multi-level features, which is as follows: remove the residual connection in the Transformer layer extracting the feature; replace the original query-key attention mechanism with a query-query attention mechanism; remove the feedforward network, and obtain the image feature from the nth Transformer layer correct the image feature to obtain a corrected image feature .

3. The zero-shot / few-shot industrial defect detection method of claim 2, wherein, the image feature modified by the nthtransformer layer is: wherein, is a query vector, is a value vector, is a projection layer, is a dimension of the key vector.

4. The zero / few-shot industrial defect detection method of claim 1, wherein, In step S2, the learnable adapter is as follows: For each corrected , introducing the image block adapter , contains only one linear connection layer; after the adapter processing, the image block features after fine-tuning of the nth Transformer layer are obtained ; For the category embedding vector output by the visual encoder ViT, the adapter structure is a two-layer fully connected network with a bottleneck structure, which is used to optimize the alignment between the image category feature and the text feature to obtain the final fine-tuned category embedding vector.

5. The zero-shot / few-shot industrial defect detection method of claim 4, wherein, Final fine-tuned class embedding vectors For: wherein, is a class embedding vector output by the CLIP model, is a trainable adapter used to fine-tune the class embedding vector.

6. The zero-shot / few-shot industrial defect detection method of claim 4, wherein, For the same category of objects, the text prompt words of all templates under normal and abnormal states are respectively input into the text encoder of the CLIP model, and the outputs thereof are averaged, the text embedding vector under the normal state is represented as , the text embedding vector under the abnormal state is represented as , the text embedding vector , C represents the number of channels of the text features, , and the vector elements are real values.

7. The zero / few-shot industrial defect detection method of claim 1, wherein, In step S3, the classification loss Lclass and the segmentation loss Lseg are respectively: Lclass = -log(p) and Lseg = -log(p)​ wherein, is a binary focal loss, is a defect classification score of the image to be tested, is an actual class of the image to be tested, is a pixel-level label, is to take the probability map corresponding to the abnormal state graph as the defect graph of the image, is to take the probability map corresponding to the normal state, is a focal loss, is a Dice loss.

8. The zero / few-shot industrial defect detection method of claim 1, wherein, In step S4, the anomaly map in the small sample mode is: wherein is the anomaly map obtained for the zero-sample mode, is the anomaly map obtained for the reference branch.

9. The zero-shot / few-shot industrial defect detection method of claim 8, wherein, Anomaly graph under reference branch for: wherein, is the image block feature of the reference image obtained by the n-th stage of Transformer layer, is the image block feature of the test image obtained by the n-th stage of Transformer layer, is the image coordinate, is the flattening of the image block feature of the reference image obtained by the n-th Transformer layer, is the n-th stage.

10. A zero-shot / small-shot industrial defect detection system, comprising: It comprises: The QQ module freezes the CLIP image encoder parameters, extracts image features from different levels; correct each layer of features, multiply each layer of image features with the corresponding text features element by element after linear layer projection, generate a single layer of similarity map, and splice the results of each layer to form a text-image similarity map; The adapter module extracts the global image features of the final layer of CLIP, inputs them into a learnable adapter after multiplying them with the text features, and outputs positive / abnormal classification probability; The loss module calculates the segmentation loss based on the obtained text-image similarity map, and calculates the classification loss based on the obtained positive / abnormal classification probability; weight and sum the segmentation loss and the classification loss, and only optimize the adapter parameters through back propagation; The output module superimposes the generated text-image similarity map after adjusting the resolution through upsampling to generate an initial abnormal heat map in the zero-shot mode; In the few-shot mode, a reference sample is introduced, the cosine similarity map of the reference sample and the test sample in each layer of features is calculated, and the final heat map is formed by adding the zero-shot abnormal map, and the classification score is fused to supplement the abnormal score, and the final defect classification score is the sum of the abnormal score generated by the text prompt and the maximum value of the abnormal map.

Images