Zero sample defect detection method and defect detection equipment based on prompt learning

Abstract

The invention relates to a zero sample defect detection method and defect detection equipment based on prompt learning. The method comprises the following steps: S1, a data acquisition step: acquiring sample data of a general industrial defect data set and acquiring to-be-detected new image data of a product through an image acquisition device; s2, a prompt fine tuning step: constructing a prompt pool containing a plurality of prompts, performing parameter adjustment on the prompt pool by using the image data of the sample data and the annotation text corresponding to the image data based on the defect detection model, and determining parameters used by the prompt pool; and S3, a product detection step: performing defect detection on the new image data to be detected by using the defect detection model after the prompt pool parameter determination, determining whether the new image data is a defective product image or not, and marking the defect position of the defective product image. According to the method, zero-sample defect detection can be carried out, the anomaly detection model can be timely adjusted according to a to-be-detected sample image, the capacity of a large model is migrated to the field of defect detection by using a public defect data set, and the high-performance detection capacity can be kept.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a zero-sample defect detection method and defect detection equipment based on prompt learning. Background Art

[0002] Industrial anomaly detection plays an important role in actual production. When performing anomaly detection on industrial products, the distribution characteristics of normal products are generally established first, and then the similarity between the characteristics of the products to be detected and the distribution characteristics is determined to determine whether the products are normal. Before establishing the distribution characteristics of normal products, a large number of normal products need to be collected, that is, the abnormal products need to be detected and removed from the large number of products.

[0003] In the field of anomaly detection (AD), current existing technologies are usually based on deep learning models, which use a fully supervised training mode. Since they are usually trained for specific types of anomalies, the detection performance of these models will drop significantly when faced with anomalies of different appearances and backgrounds. A model trained for a defect on a certain product may not be able to effectively identify defects on another product. This limitation requires collecting and annotating a large number of anomaly samples for each new application scenario and retraining the model, which is not only time-consuming but also costly. Commonly used deep learning methods include autoencoders, convolutional neural networks, generative adversarial networks, and graph neural networks. The emergence of deep learning has greatly promoted the rapid development of anomaly detection, but there are still many difficulties to be solved in anomaly detection algorithms based on deep learning.

[0004] Zero-Shot anomaly detection (ZSAD) is an emerging task of anomaly detection, which requires detecting anomalies without any target dataset training samples. In this case, the above-mentioned anomaly detection methods are not feasible. Since anomalies usually have large variations in their visual appearance from different application scenarios, foreground objects and background features, such as defects on the surface of one product versus defects on other products, lesions / tumors on different organs, or industrial defects versus tumors / lesions in medical images, a detection model with strong generalization ability for such variations is required. Existing large models have demonstrated excellent capabilities in image understanding, but lack specific domain knowledge and have a weak understanding of local details in images, which results in these large models not being directly used for industrial anomaly detection tasks. This invention is the result. Summary of the invention

[0005] The embodiments of the present application provide a zero-sample defect detection method and defect detection device based on prompt learning. The method timely adjusts the anomaly detection model according to the sample image to be detected, and migrates the capabilities of the large model to the defect detection field by using a public defect data set, thereby maintaining high-performance detection capabilities.

[0006] In a first aspect, an embodiment of the present application provides a zero-sample defect detection method based on prompt learning, characterized in that it includes:

[0007] S1, data acquisition step: acquiring sample data of a general industrial defect data set and acquiring new image data of a product to be tested through an image acquisition device;

[0008] S2, prompt fine-tuning step: construct a prompt pool containing multiple prompts, adjust parameters of the prompt pool based on the defect detection model using the image data of the sample data and the annotation text corresponding to the image data, and determine the parameters used by the prompt pool;

[0009] S3, product detection step: use the defect detection model determined by the prompt pool parameters to perform defect detection on the new image data to be tested, determine whether the new image data is a defective product image, and mark the defect position of the defective product image.

[0010] In some possible embodiments, the sample data of method S1 includes normal samples and abnormal samples; wherein the normal samples and abnormal samples both include image data and annotation data of the corresponding images.

[0011] In some possible embodiments, in the prompt fine-tuning step S2 of the method, the number of prompts in the prompt pool is more than 2, and the prompt pool includes normal prompt words and defect prompt words; the normal prompt words are used as prompt templates for normal samples; the defect prompt words are used as prompt templates for abnormal samples;

[0012] A connection is established between the text feature encoding information and the image feature encoding information of each prompt in the prompt pool; and the text-image alignment method is based on a text-image pre-training model using contrastive learning.

[0013] In some possible embodiments, in the prompt fine-tuning step S2 of the method, the normal prompt word is represented as:

[0014] The encoding vector representing the positive prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02;

[0015] Preferably, the defect prompt word is expressed as:

[0016] The encoding vector representing the defect prompt word, the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.02.

[0017] In some possible embodiments, in the prompt fine-tuning step S2 of the method, each prompt in the prompt word is selected according to the following steps:

[0018] S21, text prompt word initialization steps:

[0019] Initialize a set of text prompt codes Among them, N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding;

[0020] S22, text feature encoding steps:

[0021] The initialized text prompt word enters the text feature encoder E T , get the text feature encoding vector F T :

[0022] F T =E T (T),

[0023] in Its size is consistent with the input text encoding vector;

[0024] S23, image feature encoding step

[0025] Perform image feature encoding on the corresponding image data I to obtain an image-level feature vector and a pixel-level feature vector;

[0026] F I ,F P =E I (I);

[0027] in HW represents the number of patches into which the image data I is divided when the VIT (Vision Transformer) model is used;

[0028] S24, text feature selection steps:

[0029] By aligning the text feature encoding vector and the image-level feature vector, K text features that are aligned with the image feature encoding information are selected.

[0030]

[0031] Where <·> represents the cosine similarity calculation, Where K is a natural number greater than 1; D represents the dimension.

[0032] In some possible embodiments, the prompt fine-tuning step of method S2 further includes the step of fusing all selected prompts in the prompt pool into one prompt in the following manner:

[0033]

[0034] in Where K is a natural number greater than 1; D represents the dimension.

[0035] In some possible embodiments, in the prompt fine-tuning step of method S2, the prompt fine-tuning step uses the loss function L to adjust the parameters of the prompt pool of the defect detection model through the prompt pool:

[0036] L=L CLS +λL SEG ;

[0037] in, is the defect classification loss function;

[0038] For image

[0039] Cut loss function;

[0040] Y I = [0,1] indicates whether there is a defect, 0 indicates no defect, 1 indicates a defect; represents the mask image of the defect area, and λ is the loss coefficient.

[0041] In a second aspect, an embodiment of the present invention provides a defect detection device, characterized in that the device includes an image acquisition device, a processor and a memory,

[0042] The image acquisition device is used to capture the image of the product to be tested;

[0043] The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method described above are implemented.

[0044] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that it includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the method described above.

[0045] In a fourth aspect, an embodiment of the present invention provides a computer program product, characterized in that the computer program product comprises software codes, and the software codes are used to execute the steps of the method described above.

[0046] This application provides a zero-sample defect detection method and defect detection device based on prompt learning. It uses multiple prompts to fine-tune and merge them into a prompt pool. It can adjust a small number of trainable parameters to adapt the large model to downstream tasks such as defect detection and scenario classification, ensuring the efficiency of training. Since only the comparison and learning adaptation of multiple prompts in the prompt pool and image features are required, the migration ability of the defect detection model is expanded. After the model is trained, it can be directly migrated to other types of defect data outside the training set, and high-performance detection can still be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A method flow chart of a zero-sample defect detection method based on prompt learning provided in an embodiment of the present application;

[0049] Figure 2 A model architecture diagram of a zero-sample defect detection method based on prompt learning provided in an embodiment of the present application;

[0050] Figure 3 A method flow chart of a prompt adjustment method in zero-sample defect detection based on prompt learning provided in an embodiment of the present application;

[0051] Figure 4 A method flow chart of a prompt adjustment method in zero-sample defect detection based on prompt learning provided by another embodiment of the present application;

[0052] Figure 5 A method flow chart of a prompt adjustment method in zero-sample defect detection based on prompt learning provided by another embodiment of the present application;

[0053] Figure 6 A comparison chart of the detection effects provided by the existing method and the embodiments of the present application;

[0054] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for a specific embodiment of the present application;

[0055] Figure 8 A schematic diagram of the structure of a defect detection device provided in a specific embodiment of the present application;

[0056] Fig. 9 Another structural block diagram of the defect detection device provided in a specific embodiment of the present application. DETAILED DESCRIPTION

[0057] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0058] It should be understood that in the description of the specification and claims of this application, if there are terms such as "first", "second", etc., they are only used to distinguish the objects described, and do not have any order or technical meaning. Therefore, the objects defined as "first", "second", etc. may explicitly or implicitly include one or more of the objects. In addition, "one" or "a" and similar words do not indicate a quantitative limitation, but indicate the existence of at least one, and "multiple" means no less than two.

[0059] In view of the remarkable progress of deep learning in the fields of computer vision and natural language processing in the prior art, there are currently some technologies that attempt to connect computer vision and natural language processing. Large visual language models (Vision-Language Models, VLMs) are a modeling technology that attempts to connect computer vision and natural language processing. In typical large visual language models, contrastive training is a common strategy, using normal samples and abnormal sample pairs for training. VLM is trained by predicting similar representations of normal sample pairs and different representations of abnormal sample pairs. Based on the Contrastive Language-Image Pre-Training (CLIP) model, text data and image data are combined, and language-image is pre-trained using contrastive learning methods. This is an efficient and scalable natural language supervised learning method.

[0060] However, existing technologies directly apply models such as CLIP to anomaly detection, which are usually optimized for specific types of anomalies. When transferred to new domains or anomaly types, the performance of the model will drop significantly because they fail to capture common anomaly patterns across domains. In order to adapt to different anomaly detection tasks, existing methods need to design and optimize a large number of manual text prompts, which is not only labor-intensive but also requires professional knowledge (needs to be provided by professionals). The performance of existing models depends on the alignment of the semantics of foreground objects, which is a significant disadvantage in the absence of training samples and severely limits the versatility and flexibility of the model.

[0061] The present invention designs a learnable prompt pool, designs more prompts in the prompt pool, aligns multiple prompts with the image features of normal samples and abnormal samples, and decomposes different prompts into the language space. In this way, when a large model performs anomaly detection, it can adaptively correspond to the input image to be tested through the prompt pool, thereby improving the zero-sample anomaly detection performance and expanding the application field of anomaly detection. The present invention can be applied to the field of artificial intelligence visual technology, especially to industries such as industrial defect detection or computer vision.

[0062] The present invention relates to computer vision technology in the field of artificial intelligence, and is particularly suitable for industries such as industrial quality control and product defect detection and analysis. It can improve the anomaly detection capability of models such as CLIP in the absence of training samples.

[0063] The zero-sample defect detection method based on prompt learning of the present invention is described in detail below.

[0064] like Figure 1 As shown, an embodiment of the present invention provides a zero-sample defect detection method based on prompt learning, comprising the following steps:

[0065] S1, data acquisition step: acquiring sample data of a general industrial defect data set and acquiring new image data of a product to be tested through an image acquisition device;

[0066] S2, prompt fine-tuning step: construct a prompt pool containing multiple prompts, adjust parameters of the prompt pool based on the defect detection model using the image data of the sample data and the annotation text corresponding to the image data, and determine the parameters used by the prompt pool;

[0067] S3, product detection step: use the defect detection model determined by the prompt pool parameters to perform defect detection on the new image data to be tested, determine whether the new image data is a defective product image, and mark the defect position of the defective product image.

[0068] The zero-sample defect detection method based on prompt learning of the present invention only requires a general industrial defect data set and new image data to be tested, and does not require the training of product image data and text data that are the same as the new image data to be tested. In the S1 data acquisition step, data collection only requires a general industrial defect data set, including normal samples and abnormal samples. This method only requires a part of the defect detection data set, which is used to migrate the performance of the large model to the defect detection field. Based on the convenience of obtaining data sources, open source industrial detection data sets are usually used as examples.

[0069] The general industrial defect dataset is used as sample data, and only some simple image preprocessing is needed. For example, all images and annotation data are uniformly converted to the same size by interpolation and cropping, and no other image augmentation transformation is used. The specific data augmentation operation can be to crop all the real defect images after the above cleaning to a uniform size, such as a resolution of 1024×1024, and center cropping can be performed to remove excess background.

[0070] The acquisition of new image data to be tested can be obtained by an image acquisition device. Product samples are collected from the product inspection production line, and an image acquisition device such as a camera (high-resolution camera, industrial camera, etc.) is used to calibrate the camera. Then, the product sample is placed in the center of the photographing area and photographed to obtain a real sample image as the new image data to be tested.

[0071] The acquisition of new image data to be tested can be achieved through standard modules of image acquisition devices. For example, a digital camera or video camera can be used to capture images by scanning line by line or frame by frame. This is the most common way to acquire images and is suitable for most scenarios. In special scenarios, infrared sensors or thermal infrared cameras can also be used to capture infrared or thermal infrared images that are invisible to the human eye. Alternatively, the three-dimensional shape and texture information of an object can be obtained by using technologies such as structured light, time of flight (TOF) or stereo cameras.

[0072] like Figure 3 As shown, in the S2 prompt fine-tuning step, a prompt pool containing multiple prompts is constructed, and the prompt pool is parameterized based on the defect detection model using the image data of the sample data and the annotation text corresponding to the image data to determine the parameters used by the prompt pool; the specific prompt fine-tuning steps are as follows Figure 3 shown.

[0073] S21, text prompt initialization step (prompt initing):

[0074] Initialize and construct a set of text prompt encodings, including positive prompt words and defect prompt words, as the input of the defect detection model.

[0075] Typically, you can initialize a set of text prompt codes Where 2 represents two groups of prompt words (positive prompt words and defect prompt words), N n ,N a They represent the number of positive prompt words (n) and defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding, which is consistent with the input dimension of the CLIP model used. The same identifiers have the same meaning and will not be repeated here. For example, positive prompt words can be expressed as follows:

[0076] The encoding vector representing the positive prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02;

[0077] Defect prompt words are expressed as:

[0078] The encoding vector representing the defect prompt word, the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.02.

[0079] S22, text feature encoding steps:

[0080] The defect detection model is used to encode the text features of the initialized text prompt words using a text feature encoder to obtain a prompt text feature encoding vector.

[0081] Specifically, the initialized text prompt word enters the text feature encoder E T , get the text feature encoding vector F T :

[0082] F T =E T (T),

[0083] in Its size is consistent with the input text encoding vector;

[0084] S23, image feature encoding step

[0085] The image data of the sample data is input into the defect detection model, and after image feature encoding using an image feature encoder, an image-level feature vector and a pixel-level feature vector are obtained.

[0086] For example, the corresponding image data I is subjected to image feature encoding to obtain an image-level feature vector and a pixel-level feature vector;

[0087] F I ,F P =E I (I);

[0088] in HW represents the number of patches into which the image data I is divided when the VIT (Vision Transformer) model is used;

[0089] S24, text feature selection steps:

[0090] The defect detection model is used to compare and learn the alignment of text feature encoding vectors and image-level feature vectors, and multiple text features that are aligned with the image feature encoding information are selected.

[0091] By aligning the text feature encoding vector and the image-level feature vector, K text features that are aligned with the image feature encoding information are selected as the key prompt text features.

[0092]

[0093] Where <·> represents the cosine similarity calculation, Where K is a natural number greater than 1; D represents the dimension. By comparing F T and F I , select the K text features closest to

[0094] The key prompt text feature is the prompt text feature with the closest vector distance. Obtaining the key prompt text feature can obtain a more accurate and customized description of different sample data. This is different from conventional prompt tuning. In conventional prompt tuning, there is only one prompt, while in the prompt tuning of the present invention, there are multiple prompts in the prompt pool, and the multiple prompts are various, aiming to provide better generalization ability for more accurate and customized description of different sample data.

[0095] The purpose of prompt adjustment is to change the input prompt to obtain better model modeling results, so that it can adapt to new downstream tasks without updating model parameters, and it is easier to apply large pre-trained models to actual downstream tasks. The input of the defect detection model of the present invention can be expanded by prompts, and the prompts can be manually created natural language instructions, automatically generated natural language instructions, or automatically generated vector representations. The template of the prompt is a natural language instruction, and such a prompt is a hard prompt; and the prompt is executed in the embedding space of the model, which is an automatically generated vector representation called a soft prompt. In the present invention, each prompt preferably uses a soft prompt, which eliminates the parameterization limitations of the defect detection model pre-training. Moreover, the prompt pool has its own parameters, and there is room for further optimization.

[0096] In an embodiment of the present invention, there are multiple prompts in the prompt pool, and the multiple prompts can be selected according to different downstream characters. For example, when detecting defects, a prompt that matches the image to be detected can be selected from the prompt pool for use. The association between the prompt and the image to be detected can be determined by decomposing the abnormal visual features into the text space, and then adaptively assigning the prompt according to the input sample. Since a single prompt is used in the prior art, features of different image data distributions may be confused. Aligning various visual image features with a single text prompt may result in insufficient matching granularity. This simplification may fail to capture subtle differences between visually similar normal and abnormal patches, resulting in increased false detections, especially in pixel-level detection.

[0097] Using multiple prompts and adaptively assigning prompts according to data distribution characteristics can provide a more accurate and customized description of different samples. The optimal transport (OT) method can be used to evaluate the structural distance between visual features and text prompt embeddings.

[0098] A more preferred embodiment of the present invention is as follows Figure 4 As shown, the S2 prompt fine-tuning step also includes:

[0099] S25, prompt pool fusion step: a step of merging all selected prompts in the prompt pool into one prompt in the following manner:

[0100]

[0101] in Where K is a natural number greater than 1; D represents the dimension. Perform an average pooling operation AvgPool(·) on the selected text features to obtain the final text features

[0102] In the preferred technical solution of the present invention, all prompts in the prompt pool are fused. The text features of each prompt in the prompt pool are fused into one prompt, so that the defect detection model is pre-trained, and the calculation is still the calculation amount of one prompt. In this way, the large model can be adapted to downstream tasks with a small number of trainable parameters, ensuring the efficiency of training. After the defect detection model is trained, it can be directly migrated to other types of defect data outside the training set, and high-performance detection can still be maintained.

[0103] like Figure 5 As shown, the S2 prompt fine-tuning step also includes:

[0104] S26, prompt pool parameter update step:

[0105] The loss function is used to adjust the parameters of the prompt pool of the defect detection model through the prompt pool. The difference between the prediction model prediction result and the actual result is measured through the loss function. The different prompts used in the prompt pool are screened and updated according to the results of the loss function.

[0106] The cross-entropy loss function, Focal loss function and Dice loss function are used to define the overall loss function L, and the parameters are updated according to the overall loss function L.

[0107] Among them, the overall loss function L is expressed as:

[0108] L=L CLS +λL SEG ;

[0109] in, is the defect classification loss function;

[0110] For image

[0111] Cut loss function;

[0112] Y I = [0,1] indicates whether there is a defect, 0 indicates no defect, 1 indicates a defect; represents the mask image of the defect area, and λ is the loss coefficient.

[0113] The present invention uses the cross entropy loss function, the Focal loss function and the Dice loss function to define the overall loss function. The cross entropy loss function is defined as:

[0114] CE(p,y)=-ylog p;

[0115] Where y is the probability distribution of the true result, i.e. the label, and p is the probability distribution of the predicted result. The cross-entropy loss function evaluates the performance of the model by minimizing the difference between the predicted probability distribution and the true probability distribution.

[0116] The Focal loss function is defined as:

[0117] Among them, γ is the focus parameter. The Focal loss function increases the weight of samples that are difficult to classify and reduces the weight of samples that are easy to classify to avoid the problem of class imbalance.

[0118] The Dice loss function is defined as:

[0119] The Dice loss function is used to calculate the similarity between two samples, and Smooth is a constant added.

[0120] The overall loss function can be used to evaluate the difference between the model's predicted results and the actual results, and the prompts used in the prompt pool can be adjusted based on these differences.

[0121] S3, product inspection step: Use the defect detection model determined by the prompt pool parameters to perform defect detection on the new image data to be tested, determine whether the new image data is a defective product image, and mark the defect position of the defective product image. Use the trained prompt word parameters and the new image data to be tested to input into the defect detection model such as the CLIP model for inference. Use the text encoder and image encoder of CLIP to extract features of the input prompt words and image data, calculate the similarity between the prompt word feature vector and the image feature, and obtain the anomaly score of defect detection; determine whether the new image data to be tested is a defective product image based on the anomaly score of defect detection. If it is a defective product image, perform a binarization operation, use bilinear interpolation to restore the anomaly score used for defect location to the size of the image to be tested, and then mark the defect position of the image to be tested to obtain the final defect position marked image.

[0122] The results are as follows Figure 6 As shown, Figure 6 The first row in the figure shows the benchmark data (Ground Truth, GT), reference standard, and label annotation results. Figure 6 The third row in Fig. 5 uses a learnable cue, which leads to many incorrect recognitions when performing defect detection, especially cross-class defect detection. Figure 6 The second row uses a prompt pool of multiple prompts. When performing defect detection, multiple prompts can effectively reduce feature confusion. The prompt pool significantly improves model performance at both the pixel level and the image level. The figure shows the zero-sample detection results of 11 different types of products. From the results, it can be seen that this solution can complete the detection of different defect categories of different products and effectively avoid problems such as missed detection and over-killing.

[0123] Compared with traditional defect detection methods, this method fully considers the problem of defect data shortage and transfers the capabilities of large models to the field of defect detection by using public defect datasets. This method adapts large models to downstream tasks with a small number of trainable parameters by fine-tuning prompt words, ensuring the efficiency of training. The model has strong migration capabilities. After the model is trained, it can be directly migrated to other types of defect data outside the training set and still maintain high performance detection.

[0124] Reference Figure 7 , Figure 7It is a schematic diagram of the framework of the computer-readable storage medium of the specific implementation method of the present application. The computer-readable storage medium 1 of the embodiment of the present application stores a computer program instruction, and the method provided by the present application is implemented when the program instruction is executed. Among them, the computer program instruction can form a program file and be stored in the above-mentioned computer-readable storage medium in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the various implementation methods of the present application. The aforementioned computer-readable storage medium 1 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0125] More specifically, Figure 8 The principle block diagram of a defect detection device provided by an embodiment of the present invention is shown. The image generation model training device 2 includes a processor and a memory, the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method provided by the present application are implemented. Specifically, it can be as follows Figure 8 As shown, in this embodiment, the image generation model training device 2 includes a memory, a processor and a communication link. The processor is coupled to the memory and the communication circuit respectively. The memory stores program data. The processor implements the image generation method in this application by executing the program data in the memory.

[0126] The processor may also be referred to as a processor (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0127] like Fig. 9 As shown, an embodiment of the present invention provides another specific implementation of a defect detection device, the device includes an image acquisition device, a processor and a memory, the image acquisition device is used to capture the image of the product to be tested; the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the defect detection method of the present application are implemented.

[0128] More specifically, the processor includes a neural network chip, which can be used as a device for some data processing classification reasoning and data processing functions, such as Fig. 9 As shown, in addition to the neural network chip, other common components (for example, central processing unit (CPU), graphics processing unit (GPU), application processor (AP), sensors, communication modules, etc.) and so on.

[0129] Such an electronic device may be included in various types of electronic devices such as personal computers (PCs), server devices, mobile devices, and embedded devices. Such a device may correspond to a hardware component included in, for example, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, a medical device, etc., which performs, for example, speech recognition, image recognition, and image classification using a neural network. Therefore, the device may correspond to a dedicated hardware accelerator installed on the above-mentioned electronic device, or to a hardware accelerator such as a neural processor (NPU), a tensor processor (TPU), or a neural engine (NeuralEngine) (a neural engine is a dedicated module for driving a neural network), etc.).

[0130] The memory may be hardware that stores various neuromorphic data processed in the neural network chip. The memory may be DRAM, etc. The memory may include at least one of a volatile memory or a non-volatile memory. Non-volatile memories include ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, phase change random access memory (PRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memories include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In one embodiment, the memory may include at least one of HDD, SSD, CF, SD, micro SD, mini SD, xD, and a memory stick.

[0131] The device can analyze input data in real time based on a neural network to extract valid information, based on the extracted information. For example, the device can be applied to robotic devices (such as drones and advanced driver assistance systems (ADAS)), smart TVs, smart phones, medical devices, mobile devices, image display devices, measuring devices, and IoT devices, etc. In addition, the device can be installed on at least one of various other types of electronic devices. The device may include a processor, RAM, a neural network chip, a memory, various sensors including an image capture device, and a communication module. It may also include an input / output module, a security module, a power control device, and the like. The processor can control the overall operation of the electronic device. The processor may include one processor core (single-core) or multiple processor cores (multi-core). The processor may process or execute programs and / or data stored in a memory. In some example embodiments, the processor may control the function of the neural network chip by executing a program stored in a memory. The processor may be implemented as a CPU, a GPU, an AP, and the like.

[0132] RAM can temporarily store programs, data or instructions. For example, programs and / or data stored in the memory can be temporarily stored in RAM according to the control or boot code of the processor. RAM can be implemented as dynamic RAM (DRAM), static RAM (SRAM), etc.

[0133] The neural network chip can perform calculations of the neural network based on the received input data, and generate an information signal based on the calculation results. The neural network may include, but is not limited to, CNN, RNN, FNN, deep belief network, restricted Boltzmann machine, etc. According to an example embodiment, the neural network chip is a hardware accelerator dedicated to the neural network or a device including a hardware accelerator, and may include a neuromorphic processor according to the above-mentioned example embodiment.

[0134] The information signal may include one of various types of recognition signals (such as a voice recognition signal, an object recognition signal, an image recognition signal, and a biometric information recognition signal). For example, the neural network chip may receive frame data included in a video stream as input data, and generate a recognition signal of an object included in an image represented by the frame data from the frame data. In other example embodiments, the neural network chip may receive various types of input data according to the type or function of the electronic device on which the electronic device is installed, and may generate a recognition signal according to the input data.

[0135] The memory is a storage location for storing data, and can store an operating system (OS), various programs, and various data. In one embodiment, the memory can store intermediate results generated during the computational processing of the neural network chip.

[0136] The sensor may collect peripheral information of the electronic device on which the electronic device is mounted. The sensor may sense or receive a signal (e.g., a video signal, an audio signal, a magnetic signal, a bio-signal, a touch signal, etc.) from the outside of the electronic device and convert the sensed or received signal into data. For example, the sensor may include at least one of various types of sensing devices (such as a microphone, an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, an infrared sensor, a biosensor, and a touch sensor).

[0137] The sensor may provide the converted data as input data to the neural network chip. For example, the sensor may include an image sensor that generates a video stream by photographing the external environment of the electronic device, and sequentially provides continuous data frames of the video stream to the neural network chip as input data. In other example embodiments, the sensor may provide various types of data to the neural network chip.

[0138] The communication module may be provided with various wired or wireless interfaces capable of communicating with an external device. For example, the communication module may include a communication interface capable of accessing a wired local area network (LAN), a wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi)), a wireless personal area network (WPAN) (such as Bluetooth), a wireless universal serial bus (wireless USB), Zigbee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC) or a mobile cellular network (such as third generation (3G), fourth generation (4G) and long term evolution (LTE)).

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A zero-shot defect detection method based on prompt learning, characterized in that: include: S1, data acquisition step: acquiring sample data of a general industrial defect data set and acquiring an image of a product to be tested through an image acquisition device; S2, prompt fine-tuning step: construct a prompt pool containing multiple prompts, adjust parameters of the prompt pool using image data and annotation text corresponding to the image data in the sample data based on the defect detection model, and determine the parameters used by the prompt pool; S3, product detection step: use the defect detection model determined by the prompt pool parameters to perform defect detection on the new image data to be tested, determine whether the new image data is a defective product image, and mark the defect position of the defective product image.

2. The defect detection method according to claim 1, characterized in that: The method S1 sample data includes normal samples and abnormal samples; wherein the normal samples and abnormal samples both include image data and annotation data of the corresponding images.

3. The detection method according to claim 1, characterized in that: In the method S2 prompt fine-tuning step, the number of prompts in the prompt pool is more than 2, and the prompt pool includes normal prompt words and defect prompt words; the normal prompt words are used as prompt templates for normal samples; The defect prompt word is used as a prompt template for abnormal samples; The text feature encoding information and the image feature encoding information of each prompt in the prompt pool are aligned with each other; the text image alignment method is based on a text-image pre-training model using contrastive learning to perform alignment.

4. The detection method according to claim 3, characterized in that: In the method S2 prompt fine-tuning step, the normal prompt word is represented as: The encoding vector representing the positive prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02; Preferably, the defect prompt word is expressed as: The encoding vector representing the defect prompt word, the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.

02.

5. The detection method according to claim 4, characterized in that: In the method S2, in the prompt fine-tuning step, each prompt in the prompt word is selected according to the following steps: S21, text prompt word initialization steps: Initialize a set of text prompt codes Among them, N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding; S22, text feature encoding steps: The initialized text prompt word enters the text feature encoder E T , get the text feature encoding vector F T : F T =E T (T), in Its size is consistent with the input text encoding vector; S23, image feature encoding step Perform image feature encoding on the corresponding image data I to obtain an image-level feature vector and a pixel-level feature vector; F I ,F P =E I (I); in HW represents the number of patches into which the image data I is divided when the VIT (Vision Transformer) model is used; S24, text feature selection steps: By aligning the text feature encoding vector and the image-level feature vector, K text features that are aligned with the image feature encoding information are selected. Where <.> indicates cosine similarity calculation, Where K is a natural number greater than 1; D represents the dimension.

6. The detection method according to claim 5, characterized in that: The method S2 prompt fine-tuning step also includes the step of fusing all selected prompts in the prompt pool into one prompt in the following manner: in Where K is a natural number greater than 1; D represents the dimension.

7. The detection method according to claim 6, characterized in that: In the method S2, in the prompt fine-tuning step, the prompt pool is used to adjust the parameters of the defect detection model using the loss function L through the prompt pool: L=L CLS +λL SEG ; in, is the defect classification loss function; For image Cut loss function; Y I = [0,1] indicates whether there is a defect, 0 indicates no defect, 1 indicates a defect; represents the mask image of the defect area, and λ is the loss coefficient.

8. A defect detection device, characterized in that: The device comprises an image acquisition device, a processor and a memory, The image acquisition device is used to capture the image of the product to be tested; The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a processor, the processor performs the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises software code for executing the steps of the method according to any one of claims 1-7.

Images