Multi-scale mask-based training-free defect detection method and defect detection equipment

Abstract

The invention relates to a multi-scale mask-based training-free defect detection method and defect detection equipment. The method comprises the following steps: S1, a data acquisition step: acquiring a to-be-detected image of a current detection product through an image acquisition device; s2, a text image alignment step: respectively carrying out multi-scale mask masking processing on the to-be-detected image, entering an image feature encoder according to a text-image pair pre-training model to carry out image feature encoding to obtain multi-scale image feature encoding information and text feature encoding information corresponding to the to-be-detected image, aligning the multi-scale image feature coding information with text feature coding information corresponding to the to-be-detected image; and S3, a product detection step: determining whether the to-be-detected image is a defective product image or not and marking the defect position of the to-be-detected image according to a coding information alignment result. According to the method, by means of the pre-trained model weight, large-scale model training starting from the beginning is avoided, so that the training time is shortened, the use of computing resources is optimized, and the high-performance detection capability can be kept.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a training-free defect detection method and defect detection equipment based on multi-scale masks. Background Art

[0002] Industrial anomaly detection plays an important role in actual production. When performing anomaly detection on industrial products, the distribution characteristics of normal products are generally established first, and then the similarity between the characteristics of the products to be detected and the distribution characteristics is determined to determine whether the products are normal. Before establishing the distribution characteristics of normal products, a large number of normal products need to be collected, that is, the abnormal products need to be detected and removed from the large number of products.

[0003] In the field of anomaly detection (AD), current existing technologies are usually based on deep learning models. Most existing methods use deep learning models based on a single visual modality and mainly rely on a large amount of high-quality annotated image data for training. Common deep learning methods include autoencoders, convolutional neural networks, generative adversarial networks, and graph neural networks. The emergence of deep learning has greatly promoted the rapid development of anomaly detection, but there are still many difficulties to be solved in deep learning-based anomaly detection algorithms. However, the cost of obtaining high-quality image data is extremely high, and the collection and annotation of defect samples is not only time-consuming but also expensive. In actual scenarios, defects vary in size and shape, and visual information of a single scale is not enough to capture complex defect features. Feature extraction under a single visual modality may not accurately capture defect features due to sample bias, resulting in insufficient generalization ability of the model.

[0004] Zero-Shot anomaly detection (ZSAD) is an emerging task of anomaly detection, which requires detecting anomalies without any target dataset training samples. In this case, the above-mentioned anomaly detection methods are not feasible. Since anomalies usually have large variations in their visual appearance from different application scenarios, foreground objects and background features, such as defects on the surface of one product versus defects on other products, lesions / tumors on different organs, or industrial defects versus tumors / lesions in medical images, a detection model with strong generalization ability for such variations is required. Existing large models have demonstrated excellent capabilities in image understanding, but lack domain-specific knowledge and have a weak understanding of local details in images, which results in these large models not being directly used for industrial anomaly detection tasks. This invention is the result. Summary of the invention

[0005] The embodiment of the present application provides a training-free defect detection method and defect detection device based on multi-scale masks. The method combines a small amount of image data and text descriptions, significantly reducing the dependence on large-scale high-quality annotated image data. The method uses pre-trained model weights, and the method avoids large-scale model training from scratch, which not only reduces training time, but also optimizes the use of computing resources.

[0006] In a first aspect, an embodiment of the present application provides a training-free defect detection method based on a multi-scale mask, characterized in that it includes:

[0007] S1, data acquisition step: acquiring the image of the product to be tested by the image acquisition device;

[0008] S2, text image alignment step: performing multi-scale masking processing on the images to be tested respectively, entering the image feature encoder to perform image feature encoding according to the text-image pair pre-training model to obtain multi-scale image feature encoding information and obtain text feature encoding information corresponding to the image to be tested, and aligning the multi-scale image feature encoding information with the text feature encoding information corresponding to the image to be tested;

[0009] S3, product inspection step: according to the result of the coding information alignment, determine whether the image to be tested is a defective product image and mark the defect position of the image to be tested.

[0010] In some possible embodiments, in the text-image alignment step S2 of the method, the initial weight parameters of the text-image pair pre-trained model use the weight parameters of the model trained with the existing LAION-2B dataset.

[0011] In some possible embodiments, in the text image alignment step S2 of the method, the text information corresponding to the image to be tested is encoded with text features using a text encoder;

[0012] Preferably, the text feature encoding information corresponding to the image to be tested is a normal text description S of the current detection object corresponding to the image to be tested. n and the exception text description S a , both enter the text feature encoder g at the same time, and obtain the text feature vector g of the normal text description respectively. n and the text feature vector g describing the abnormal text a :

[0013] g n =g(S n );

[0014] g a =g(S a ).

[0015] In some possible embodiments, the image feature encoding used in the text image alignment step S2 of the method adopts an image encoder to perform image feature encoding;

[0016] Preferably, the image feature encoding comprises the following steps:

[0017] (S211) Image gridding step: gridding the image to be tested to obtain sub-patches of the same size, splicing the sub-patches in order into a sub-patch sequence, and adding a class token representing the overall information of the sub-patches to the head of the sub-patch sequence;

[0018] (S212) Mask construction step: constructing mask matrices W of k scales according to the size of the sub-blocks;

[0019] (S213) Image feature encoding step: After masking the sub-image blocks using mask matrices of different scales, the masked sub-image blocks are input into the image encoder for image feature encoding to obtain multi-scale image feature information, including global image feature information F cls And the local image feature information of the sub-block

[0020] F cls =lobal(f(x));

[0021]

[0022] Where x is the preprocessed image to be tested; W i The mask matrix representing the i-th scale is a Boolean matrix whose element values ​​are 0 or 1, 1 indicates that the area corresponding to the image to be tested is covered by the mask, and 0 indicates that the area corresponding to the image to be tested is visible; i is a natural number from 1 to k; k is the scale range, which is a natural number greater than or equal to 2;

[0023] f is the image feature encoder; ⊙ represents element-by-element multiplication;

[0024] Represents the features of the current image after being processed by the mask matrix of the i-th scale;

[0025] Global(·) means extracting the class token mentioned above, which is used to judge whether the overall image is abnormal or not;

[0026] Local(·) means extracting the local image information of all sub-blocks except the class token.

[0027] In some possible embodiments, the encoding information alignment in the text image alignment step S2 of the method is to align the multi-scale image feature information with the text feature vector g described by the normal text. n and the text feature vector g describing the abnormal text a Perform cosine similarity calculation to obtain the similarity score between the new image to be tested and the normal text description or abnormal text description:

[0028] M n =sim(F cls , g n );

[0029] M a =sim(F cls , g a );

[0030]

[0031]

[0032] Among them, sim() represents the cosine similarity calculation, M n Represents the similarity score between the current overall image and the normal text description, that is, the probability of a normal sample; M a Represents the similarity score between the current overall image and the abnormal text description, that is, the probability of the abnormal sample;

[0033] is the anomaly score used for defect localization, which is the average of the similarity scores between each sub-block and the normal and abnormal text descriptions at k scales.

[0034] In some possible embodiments, in the method, the normal text description includes a normal prompt word template; the abnormal text description includes a defect prompt word template;

[0035] The normal prompt words are used as prompt templates for normal samples; the defect prompt words are used as prompt templates for abnormal samples;

[0036] The normal prompt words are expressed as:

[0037] Represents the encoding vector of the positive prompt word, and the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.02; N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding;

[0038] Preferably, the defect prompt word is expressed as:

[0039] The encoding vector representing the defect prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02; N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the encoding dimension of each prompt word.

[0040] In some possible embodiments, in the product detection step S3 of the method, whether to classify as a normal sample or a defective sample is determined based on whether the similarity score between the entire image to be tested and the normal text description is greater than the similarity score between the entire image to be tested and the abnormal text description; when the similarity score between the entire image to be tested and the normal text description is greater than the similarity score between the entire image to be tested and the abnormal text description, it is classified as a normal sample; otherwise, it is classified as a defective sample;

[0041] Preferably, bilinear interpolation is used to restore the anomaly score used for defect location to the size of the image to be tested, and the defect position of the image to be tested is marked.

[0042] In a second aspect, an embodiment of the present invention provides a defect detection device, characterized in that the device includes an image acquisition device, a processor and a memory,

[0043] The image acquisition device is used to capture the image of the product to be tested;

[0044] The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method described above are implemented.

[0045] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, characterized in that it includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the method described above.

[0046] In a fourth aspect, an embodiment of the present invention provides a computer program product, characterized in that the computer program product comprises software codes, and the software codes are used to execute the steps of the method described above.

[0047] The present application provides a training-free defect detection method and defect detection device based on multi-scale masks. The multi-scale mask masking process can fully extract the local and global information of the image, so that the model can not only capture the overall abnormal features, but also identify the local subtle differences in detail. This is crucial for accurately identifying and locating defects of various sizes and types. This method directly uses the trained model without large-scale training. With the help of pre-trained model weights, large-scale model training from scratch is avoided, which not only reduces training time, but also optimizes the use of computing resources. In addition, this method is particularly effective for data scarcity in specific industrial fields. This method expands the migration capability of the defect detection model. After the model is trained, it can be directly migrated to other types of defect data outside the training set, and high-performance detection can still be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] Figure 1 A method flow chart of a training-free defect detection method based on multi-scale masks provided in an embodiment of the present application;

[0050] Figure 2 A model architecture diagram of a training-free defect detection method based on multi-scale masks provided in an embodiment of the present application;

[0051] Figure 3 A method flow chart of the text image alignment step in the training-free defect detection method provided in an embodiment of the present application;

[0052] Figure 4 A method flow chart of an image feature encoding step in a training-free defect detection method provided by another embodiment of the present application;

[0053] Figure 5 A flowchart of a method for text feature encoding in a training-free defect detection method provided by another embodiment of the present application;

[0054] Figure 6 A comparison chart of the detection effects provided by the existing method and the embodiments of the present application;

[0055] Figure 7 A schematic diagram of the structure of a computer-readable storage medium provided for a specific embodiment of the present application;

[0056] Figure 8 A schematic diagram of the structure of a defect detection device provided in a specific embodiment of the present application;

[0057] Fig. 9 Another structural block diagram of the defect detection device provided in a specific embodiment of the present application. DETAILED DESCRIPTION

[0058] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0059] It should be understood that in the description of the specification and claims of this application, if there are terms such as "first", "second", etc., they are only used to distinguish the objects described, and do not have any order or technical meaning. Therefore, the objects defined as "first", "second", etc. may explicitly or implicitly include one or more of the objects. In addition, "one" or "a" and similar words do not indicate a quantitative limitation, but indicate the existence of at least one, and "multiple" means no less than two.

[0060] In view of the remarkable progress of deep learning in the fields of computer vision and natural language processing in the prior art, there are currently some technologies that attempt to connect computer vision and natural language processing. Large visual language models (Vision-Language Models, VLMs) are a modeling technology that attempts to connect computer vision and natural language processing. In typical large visual language models, contrastive training is a common strategy, using normal samples and abnormal sample pairs for training. VLMs are trained by predicting similar representations of normal sample pairs and different representations of abnormal sample pairs. Based on the Contrastive Language-Image Pre-Training (CLIP) model, text data and image data are combined, and language-image pre-training is performed using contrastive learning methods. This is an efficient and scalable natural language supervised learning method.

[0061] The CLIP model includes a text encoder and an image encoder. The training uses text-image pairs and aligns the text feature encoding information extracted by the text encoder with the image feature encoding information extracted by the image encoder through comparative learning. The text encoder is used to extract text features and can use the text transformer model commonly used in NLP; while the image encoder is used to extract image features and can use the commonly used CNN model or vision transformer model.

[0062] CLIP's training data is text-image pairs: an image and its corresponding text description. The CLIP model compares the text features and image features extracted by the encoder and calculates the similarity between each text feature and each image feature. The training goal of CLIP is to maximize the similarity between images and their corresponding texts (positive samples) while minimizing the similarity of other samples (negative samples). After the CLIP model is trained, it can achieve classification and text description generation on a specific downstream task without any training data.

[0063] However, existing technologies directly apply models such as CLIP to anomaly detection, which are usually optimized for specific types of anomalies. However, the cost of acquiring high-quality image data in the field of anomaly detection is extremely high, and the collection and annotation of defect samples is not only time-consuming but also expensive. In actual scenarios, defects vary in size and shape, and visual information of a single scale is not enough to capture complex defect features. Feature extraction under a single visual modality may not be able to accurately capture defect features due to sample bias, resulting in insufficient generalization of the model.

[0064] The present invention performs mask processing on the image by designing a multi-scale mask technology. In this way, when the language-image comparison pre-training model is used for training, the image encoder only processes the part not covered by the mask when extracting image features. On the one hand, the amount of calculation of the image encoder is greatly reduced, and the pre-training time is reduced; the use of multi-scale mask technology also improves the generalization performance of the model, and can process models with larger data volumes. On the other hand, due to the multi-scale mask technology, local and global information of the image can be extracted, so that the model can not only capture the overall abnormal features, but also identify the subtle local differences in detail. This is crucial for accurately identifying and locating defects of various sizes and types.

[0065] The present invention relates to computer vision technology in the field of artificial intelligence, and is particularly suitable for industries such as industrial quality control and product defect detection and analysis. It can improve the anomaly detection capability of models such as CLIP in the absence of training samples, and can more accurately respond to new or variant defects that may occur in the production process.

[0066] The following specifically describes the training-free defect detection method based on multi-scale masks of the present invention.

[0067] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a training-free defect detection method based on a multi-scale mask, comprising the following steps:

[0068] S1, data acquisition step: acquiring the image of the product to be tested by the image acquisition device;

[0069] S2, text image alignment step: performing multi-scale masking processing on the images to be tested respectively, entering the image feature encoder to perform image feature encoding according to the text-image pair pre-training model to obtain multi-scale image feature encoding information and obtain text feature encoding information corresponding to the image to be tested, so that the multi-scale image feature encoding information is aligned with the text feature encoding information corresponding to the image to be tested;

[0070] S3, product inspection step: according to the result of the coding information alignment, determine whether the image to be tested is a defective product image and mark the defect position of the image to be tested.

[0071] The training-free defect detection method based on multi-scale mask of the present invention is training-free and only needs to use the image to be tested of the current detection product. However, it uses the text-image pair pre-trained model as the basic model, and the parameters used require the initialization weight parameters of the basic model. Based on this, the embodiment of the present invention uses the LAION-2B (2 billion image-text pairs) data set and the trained model weights as the initialization weights of the image and text encoder parameters of the text-image pair pre-trained model of the present invention.

[0072] The LAION-2B (2 billion image-text pairs) dataset used may include normal samples and abnormal samples. The images to be tested of the current inspection products are used as sample images, and only some simple image preprocessing is required. For example, all images and annotation data are uniformly converted to the same size using interpolation and cropping, and no other image augmentation transformations are used. The specific data preprocessing operation may be to crop all real defect images to a uniform size, such as a resolution of 1024×1024, and center cropping may be performed to remove excess background. The prior art relies heavily on high-quality, large-scale annotated image data, which is both costly and difficult to obtain in practical applications. The present invention can effectively utilize easily accessible non-visual data (such as text descriptions) to supplement or replace part of the image data requirements, thereby reducing dependence on expensive image data and solving the data dependency problem.

[0073] The image to be tested can be acquired through an image acquisition device. Collect product samples from the product inspection production line, use an image acquisition device such as a camera (high-resolution camera, industrial camera, etc.) to calibrate the camera, and then place the product sample in the center of the photo area to take a photo, and obtain the real sample image as the image to be tested. The acquisition of the image to be tested can be achieved through the standard module of the image acquisition device. For example, a digital camera or a video camera is used to capture images by scanning line by line or frame by frame. This is the most common image acquisition method and is suitable for most scenarios. In special scenarios, infrared sensors or thermal infrared cameras can also be used to capture infrared or thermal infrared images that are invisible to the human eye. Alternatively, the three-dimensional shape and texture information of the object can be obtained by using technologies such as structured light, time of flight (TOF) or stereo cameras.

[0074] In the embodiment of the present invention, a small amount of defective image data for testing model performance can be selected for model verification of the present invention. These defective image data can be specially collected based on the known potential defect types in the manufacturing process and captured using photographic equipment under a controlled environment. Since the cost of collating and labeling such data is relatively high, a smaller amount of defect-free image data can be appropriately collected according to actual conditions.

[0075] For these defective image data, it is necessary to collect text descriptions related to the defects, which are divided into two categories: normal text descriptions and abnormal text descriptions, for training the CLIP model so that it can understand the visual content and semantic information of the corresponding text in these defective image data. Professional image annotation tools can be used to accurately annotate these defective images with masks to determine the exact location and category of the defective area. These defective images can be used for subsequent model performance evaluation. With the help of pre-trained model weights, this method avoids large-scale model training from scratch and does not require large-scale training, which not only reduces training time, but also optimizes the use of computing resources. In addition, this method is particularly effective for data scarcity in specific industrial fields.

[0076] like Figure 3 As shown, the S2 text-image alignment step includes:

[0077] (S21) image feature encoding step: performing multi-scale masking processing on the image to be tested, entering the image feature encoder according to the text-image pair pre-training model to perform image feature encoding to obtain multi-scale image feature encoding information;

[0078] (S22) text feature encoding step: entering a text feature encoder according to a pre-trained model based on text-image pairs to perform image feature encoding to obtain text feature encoding information;

[0079] (S23) Text-image alignment step: aligning the multi-scale image feature coding information with the text feature coding information corresponding to the image to be tested according to the similarity.

[0080] like Figure 4 As shown, the image feature encoding includes the following steps:

[0081] (S211) Image gridding step: gridding the image to be tested to obtain sub-patches of the same size, splicing the sub-patches in order into a sub-patch sequence, and adding a class token representing the overall information of the sub-patches to the head of the sub-patch sequence;

[0082] (S212) Mask construction step: constructing mask matrices W of k scales according to the size of the sub-blocks;

[0083] (S213) Image feature encoding step: After masking the sub-image blocks using mask matrices of different scales, the masked sub-image blocks are input into the image encoder for image feature encoding to obtain multi-scale image feature information, including global image feature information F cls And the local image feature information of the sub-block

[0084] F cls =lobal(f(x));

[0085]

[0086] Where x is the preprocessed image to be tested; W i The mask matrix representing the i-th scale is a Boolean matrix whose element values ​​are 0 or 1, 1 indicates that the area corresponding to the image to be tested is covered by the mask, and 0 indicates that the area corresponding to the image to be tested is visible; i is a natural number from 1 to k; k is the scale range, which is a natural number greater than or equal to 2;

[0087] f is the image feature encoder; ⊙ represents element-by-element multiplication;

[0088] Represents the features of the current image after being processed by the mask matrix of the i-th scale;

[0089] Global(·) means extracting the class token mentioned above, which is used to judge whether the overall image is abnormal or not;

[0090] Local(·) means extracting the local image information of all sub-blocks except the class token.

[0091] First, the test image x enters the image feature encoder f. Then, the Vit model in the CLIP model divides the image into grids to obtain a batch of sub-image blocks of the same size. For example, if an image with a resolution of 224*224 is divided into 16 pixels (pixel) units, 14*14 sub-image blocks of size 16*16 will be obtained, and these 196 sub-image blocks are sequentially spliced ​​into a sequence, and finally the class token (Class Token) representing the overall information of the sub-image block is spliced ​​to the head of the sequence.

[0092] Second, use masks W of varying sizes i (The mask matrix of the i-th scale) constructs feature information of different scales on the sub-blocks. For example, a large-area mask will make most of the remaining features invisible, which enhances the model's perception of small local areas. In this way, a multi-scale feature of an image including global information and local information is obtained.

[0093] When the mask matrix W i , where i∈{1,2,3} represents mask matrices of small, medium, and large scales, which can indicate that the mask matrix has three different masking areas. For example, the masking area of ​​the small-scale mask matrix (the masked image features are invisible) is equal to 25%; the masking area of ​​the medium-scale mask matrix is ​​equal to 50%; and the masking area of ​​the small-scale mask matrix is ​​equal to 75%. The specific relationship between the scale size and the masking area can be defined according to the actual situation. When i∈{1,2,3,4,5}, it represents a mask matrix of five scales. The masking areas of the corresponding mask matrices of the five scales can be 20%, 30%, 40%, 50%, and 60%. Generally speaking, the larger the masking area, the more conducive it is for the image feature encoder to obtain local detail feature information. This enhances the feature extraction capability of the CLIP model, and uses a multi-scale masking strategy to extract local and global information of the image, so that the model can not only capture the overall abnormal features, but also identify the subtle local differences in detail. This is crucial for accurately identifying and locating defects of various sizes and types.

[0094] (S22) The normal text description S corresponding to the currently detected object in the text feature encoding step n and the exception text description S a , both enter the text feature encoder g at the same time, and obtain the text feature vector g of the normal text description respectively. n and the text feature vector g describing the abnormal text a :

[0095] g n =g(S n );

[0096] g a =g(Sa ).

[0097] The text information corresponding to the image to be tested is encoded with the text feature using the Text Transformer text encoder, denoted as g, and the corresponding text description templates (prompt template) are constructed for positive and negative samples respectively. For example, for the category of [glass bottle], there are two text descriptions, positive and negative, "This is a picture of [defective] [glass bottle]", and "This is a picture of [normal] [glass bottle]". Through such text descriptions, a text pool (prompt pool) is constructed for use in subsequent anomaly detection.

[0098] Specifically, Figure 5 As shown, (S22) the text feature encoding step can be performed according to the following steps:

[0099] S221, text prompt initialization step (prompt initing):

[0100] Initialize and construct a set of text prompt encodings, including positive prompt words and defect prompt words, as the input of the CLIP model text feature encoder.

[0101] Typically, you can initialize a set of text prompt codes Where 2 represents two groups of prompt words (positive prompt words and defect prompt words), N n ,N a They represent the number of positive prompt words (n) and defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding, which is consistent with the input dimension of the CLIP model used. The same identifiers have the same meaning and will not be repeated here. For example, positive prompt words can be expressed as follows:

[0102] The encoding vector representing the positive prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02;

[0103] Defect prompt words are expressed as:

[0104] The encoding vector representing the defect prompt word, the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.02.

[0105] The CLIP model of the present invention is a text-guided multi-scale training-free defect detection based on pre-trained multimodal CLIP, and its text input can be expanded by prompts, which can be manually created natural language instructions, automatically generated natural language instructions, or automatically generated vector representations. The template of the prompt is a natural language instruction, and such a prompt is a hard prompt; and the prompt is executed in the embedding space of the model, and the automatically generated vector representation is a soft prompt. The prompt of the present invention preferably adopts a soft prompt, which eliminates the parameterization limitation of the pre-training of the defect detection model, and there is room for further optimization, and the prompt adjustment can be performed later. The purpose of the prompt adjustment is to change the input prompt to obtain a better model modeling result technology, so that it can adapt to new downstream tasks without updating the model parameters, and it is easier to apply large pre-trained models to actual downstream tasks. By combining image and text data, text descriptions are used to enhance the model's understanding and recognition of defect features in images, effectively improving the generalization ability of the model. This multimodal learning method enables the model to better adapt to defect detection tasks in various complex and changeable actual environments. This method can dynamically adjust the model's understanding of the image based on actual flexible text descriptions. This flexibility enables the model to more accurately respond to new or variant defects that may appear in the production process, and achieve dynamic and flexible defect detection.

[0106] S222, text feature encoding step:

[0107] The CLIP model is used to encode the text features of the initialized text prompt words using a text feature encoder to obtain a prompt text feature encoding vector.

[0108] Specifically, the initialized text prompt word enters the text feature encoder g to obtain the text feature vector g of the normal text description n and the text feature vector g describing the abnormal text a :

[0109] g n =g(S n );

[0110] g a =g(S a );

[0111] where g n , g a Its size is consistent with the input text encoding vector.

[0112] (S23) In the text-image alignment step, after obtaining the image feature encoding information and the text feature encoding information, let F cls And local features of images at each scale Respectively with gn and g a Calculate the cosine similarity and get M n and M a ,

[0113] M n =sim(F cls ,g n );

[0114] M a =sim(F cls ,g a );

[0115]

[0116]

[0117] Among them: sim() represents the cosine similarity calculation, the formula is

[0118]

[0119] M n Represents the similarity score between the current overall image and the normal text description, that is, the probability of being a positive sample. The higher the score, the closer the current image is to the normal sample. a Represents the similarity score between the current overall image and the abnormal text description, that is, the probability of an abnormal sample. The higher the score, the more likely the current image is to have defects. n and M a Classify the image based on whether it is abnormal as a whole.

[0120] It is the anomaly score used for the final defect location. First, the similarity score (probability) between each sub-block and the normal and abnormal text descriptions at each scale is calculated, and then an average value is taken for these k scales, and the information of the three scales is fused to obtain the final result.

[0121] When k=3, the mask adopts three scales, such as the masking area of ​​the mask matrix is ​​equal to 25%, 50%, and 75% respectively. The result is:

[0122]

[0123]

[0124] yes

[0125] In this way, we first calculate the similarity scores (probabilities) between each sub-block and the normal and abnormal text descriptions at the scales of 25%, 50%, and 75% of the mask area of ​​the mask matrix, and then take an average of the three scales, fuse the information of the three scales, and get the final result. Since the defect detection method of a single modality (image only) may not be able to fully capture all types of defect features. Therefore, by using multimodality as a bridge, combining image features of different scales and related text descriptions during image-text alignment, the model's understanding and recognition of complex defect features is enhanced, solving the problem of modal bias.

[0126] When performing the product inspection step S3, it is determined whether the image to be tested is a defective product image and the defect position of the image to be tested is marked according to the result of the alignment of the coding information. n and M a The size of M is used to judge whether the current sample as a whole has defects. n >M a , then the sample is classified as a normal sample. If M n <M a , the sample is classified as a defective sample; then bilinear interpolation is used to The image is restored to its original size and the anomalies are visually marked. Through visualization technology, the defect results identified by the model are intuitively displayed on the original image, providing operators with direct and easy-to-understand defect inspection feedback, thereby improving the monitoring and quality control efficiency of the production line.

[0127] The results are as follows Figure 6 As shown, the first line is the image to be tested of the product; the second line is the rendering of the baseline model (Ground Truth, GT), the third line is the rendering after the present invention adopts multi-scale masking, and the fourth line is the defect position (corresponding label) located by the current image after multi-scale masking. The red highlight indicates the area that the model pays attention to. The darker the color, the higher the probability that the current area is the target abnormal area, and vice versa. The lower the probability. Given an image to be tested, if the CLIP model is used directly for detection, such as the visualization rendering on the left, it will cause a large number of erroneously activated areas or fail to accurately focus on the target abnormal area. After using the mask, not only can the model identify the target abnormal area more accurately, but also reduce the attention to non-target abnormal areas.

[0128] Compared with traditional defect detection methods, this method fully considers the problem of defect data shortage. By using a pre-trained multimodal CLIP model combined with a small amount of image data and text descriptions, it significantly reduces the dependence and cost of large-scale high-quality annotated image data. This greatly reduces the cost of data acquisition and annotation in actual industrial applications, especially when dealing with difficult-to-obtain or expensive defect images. By initializing with the weight parameters of the existing CLIP model, zero normal sample reasoning can be performed without using any additional normal image samples as reference, and the capabilities of the large model can be transferred to the field of defect detection.

[0129] The method of the present invention solves how to improve the training efficiency and generalization ability of the model under limited data resources through innovative learning strategies, model structures, and existing large-scale and trained model weights. The text-guided multi-scale training-free defect detection method adapts the large model to downstream tasks based on the pre-trained multimodal CLIP model. The model has strong migration ability. After the model is trained, it can be directly migrated to other types of defect data outside the training set, and can still maintain high-performance detection and have visual defect recognition results.

[0130] Reference Figure 7 , Figure 7 It is a schematic diagram of the framework of the computer-readable storage medium of the specific implementation method of the present application. The computer-readable storage medium 1 of the embodiment of the present application stores a computer program instruction, and the method provided by the present application is implemented when the program instruction is executed. Among them, the computer program instruction can form a program file and be stored in the above-mentioned computer-readable storage medium in the form of a software product, so that a computer device (which can be a personal computer, a server, or a network device, etc.) executes all or part of the steps of the various implementation methods of the present application. The aforementioned computer-readable storage medium 1 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk, or a terminal device such as a computer, a server, a mobile phone, and a tablet.

[0131] More specifically, Figure 8 The principle block diagram of a defect detection device provided by an embodiment of the present invention is shown. The image generation model training device 2 includes a processor and a memory, the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method provided by the present application are implemented. Specifically, it can be as follows Figure 8As shown, in this embodiment, the image generation model training device 2 includes a memory, a processor and a communication link. The processor is coupled to the memory and the communication circuit respectively. The memory stores program data. The processor implements the image generation method in this application by executing the program data in the memory.

[0132] The processor may also be referred to as a processor (Central Processing Unit). The processor may be an integrated circuit chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0133] like Fig. 9 As shown, an embodiment of the present invention provides another specific implementation of a defect detection device, the device includes an image acquisition device, a processor and a memory, the image acquisition device is used to capture the real product image after the integrated circuit transistor is produced; the processor and the memory are coupled, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the defect detection method of the present application are implemented.

[0134] More specifically, the processor includes a neural network chip, which can be used as a device for some data processing classification reasoning and data processing functions, such as Fig. 9 As shown, in addition to the neural network chip, other common components (for example, central processing unit (CPU), graphics processing unit (GPU), application processor (AP), sensors, communication modules, etc.) and so on.

[0135] Such an electronic device may be included in various types of electronic devices such as personal computers (PCs), server devices, mobile devices, and embedded devices. Such a device may correspond to a hardware component included in, for example, a smart phone, a tablet device, an augmented reality (AR) device, an Internet of Things (IoT) device, an autonomous vehicle, a robot, a medical device, etc., which performs, for example, speech recognition, image recognition, and image classification using a neural network. Therefore, the device may correspond to a dedicated hardware accelerator installed on the above-mentioned electronic device, or to a hardware accelerator such as a neural processor (NPU), a tensor processor (TPU), or a neural engine (NeuralEngine) (a neural engine is a dedicated module for driving a neural network), etc.).

[0136] The memory may be hardware that stores various neuromorphic data processed in the neural network chip. The memory may be DRAM, etc. The memory may include at least one of a volatile memory or a non-volatile memory. Non-volatile memories include ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, phase change random access memory (PRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memories include DRAM, SRAM, SDRAM, PRAM, MRAM, RRAM, FeRAM, etc. In one embodiment, the memory may include at least one of HDD, SSD, CF, SD, micro SD, mini SD, xD, and a memory stick.

[0137] The device can analyze input data in real time based on a neural network to extract valid information, based on the extracted information. For example, the device can be applied to robotic devices (such as drones and advanced driver assistance systems (ADAS)), smart TVs, smart phones, medical devices, mobile devices, image display devices, measuring devices, and IoT devices, etc. In addition, the device can be installed on at least one of various other types of electronic devices. The device may include a processor, RAM, a neural network chip, a memory, various sensors including an image capture device, and a communication module. It may also include an input / output module, a security module, a power control device, and the like. The processor can control the overall operation of the electronic device. The processor may include one processor core (single-core) or multiple processor cores (multi-core). The processor may process or execute programs and / or data stored in a memory. In some example embodiments, the processor may control the function of the neural network chip by executing a program stored in a memory. The processor may be implemented as a CPU, a GPU, an AP, and the like.

[0138] RAM can temporarily store programs, data or instructions. For example, programs and / or data stored in the memory can be temporarily stored in RAM according to the control or boot code of the processor. RAM can be implemented as dynamic RAM (DRAM), static RAM (SRAM), etc.

[0139] The neural network chip can perform calculations of the neural network based on the received input data, and generate an information signal based on the calculation results. The neural network may include, but is not limited to, CNN, RNN, FNN, deep belief network, restricted Boltzmann machine, etc. According to an example embodiment, the neural network chip is a hardware accelerator dedicated to the neural network or a device including a hardware accelerator, and may include a neuromorphic processor according to the above-mentioned example embodiment.

[0140] The information signal may include one of various types of recognition signals (such as a voice recognition signal, an object recognition signal, an image recognition signal, and a biometric information recognition signal). For example, the neural network chip may receive frame data included in a video stream as input data, and generate a recognition signal of an object included in an image represented by the frame data from the frame data. In other example embodiments, the neural network chip may receive various types of input data according to the type or function of the electronic device on which the electronic device is installed, and may generate a recognition signal according to the input data.

[0141] The memory is a storage location for storing data, and can store an operating system (OS), various programs, and various data. In one embodiment, the memory can store intermediate results generated during the computational processing of the neural network chip.

[0142] The sensor may collect peripheral information of the electronic device on which the electronic device is mounted. The sensor may sense or receive a signal (e.g., a video signal, an audio signal, a magnetic signal, a bio-signal, a touch signal, etc.) from the outside of the electronic device and convert the sensed or received signal into data. For example, the sensor may include at least one of various types of sensing devices (such as a microphone, an imaging device, an image sensor, a light detection and ranging (LIDAR) sensor, an ultrasonic sensor, an infrared sensor, a biosensor, and a touch sensor).

[0143] The sensor may provide the converted data as input data to the neural network chip. For example, the sensor may include an image sensor that generates a video stream by photographing the external environment of the electronic device, and sequentially provides continuous data frames of the video stream to the neural network chip as input data. In other example embodiments, the sensor may provide various types of data to the neural network chip.

[0144] The communication module may be provided with various wired or wireless interfaces capable of communicating with an external device. For example, the communication module may include a communication interface capable of accessing a wired local area network (LAN), a wireless local area network (WLAN) (such as wireless fidelity (Wi-Fi)), a wireless personal area network (WPAN) (such as Bluetooth), a wireless universal serial bus (wireless USB), Zigbee, near field communication (NFC), radio frequency identification (RFID), power line communication (PLC) or a mobile cellular network (such as third generation (3G), fourth generation (4G) and long term evolution (LTE)).

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A training-free defect detection method based on multi-scale masks, characterized in that: include: S1, data acquisition step: acquiring the image of the product to be tested by the image acquisition device; S2, text image alignment step: performing multi-scale masking processing on the images to be tested respectively, performing image feature encoding based on the text-image pair pre-training model to obtain multi-scale image feature encoding information and performing text feature encoding to obtain text feature encoding information corresponding to the image to be tested, and aligning the multi-scale image feature encoding information with the text feature encoding information corresponding to the image to be tested; S3, product inspection step: according to the result of the coding information alignment, determine whether the image to be tested is a defective product image and mark the defect position of the image to be tested.

2. The defect detection method according to claim 1, characterized in that: In the text-image alignment step S2 of the method, the initial weight parameters of the text-image pair pre-trained model use the weight parameters of the model trained with the existing LAION dataset; preferably, the weight parameters of the model trained with the existing LAION-2B dataset are used.

3. The detection method according to claim 1, characterized in that: In the text image alignment step S2 of the method, the text information corresponding to the image to be tested is encoded with text features using a text encoder; Preferably, the text feature encoding information corresponding to the image to be tested is a normal text description S of the current detection object corresponding to the image to be tested. n and the exception text description S a , both enter the text feature encoder g at the same time, and obtain the text feature vector g of the normal text description respectively. n and the text feature vector g describing the abnormal text a : g n =g(S n ); g a =g(S a )。 4. The detection method according to claim 3, characterized in that: The image feature encoding used in the text image alignment step S2 of the method adopts an image encoder to perform image feature encoding; Preferably, the image feature encoding comprises the following steps: (S211) Image gridding step: gridding the image to be tested to obtain sub-patches of the same size, splicing the sub-patches in order into a sub-patch sequence, and adding a class token representing the overall information of the sub-patches to the head of the sub-patch sequence; (S212) Mask construction step: constructing mask matrices W of k scales according to the size of the sub-blocks; (S213) Image feature encoding step: After masking the sub-image blocks using mask matrices of different scales, the masked sub-image blocks are input into the image encoder for image feature encoding to obtain multi-scale image feature information, including global image feature information F cls And the local image feature information of the sub-block F cls =lobal(f(x)); Where x is the preprocessed image to be tested; W i The mask matrix representing the i-th scale is a Boolean matrix whose element values ​​are 0 or 1, 1 indicates that the area corresponding to the image to be tested is covered by the mask, and 0 indicates that the area corresponding to the image to be tested is visible; i is a natural number from 1 to k; k is the scale range, which is a natural number greater than or equal to 2; f is the image feature encoder; ⊙ represents element-by-element multiplication; Represents the features of the current image after being processed by the mask matrix of the i-th scale; Global(·) represents the class token of the extracted image, which is used to determine whether the overall image is abnormal or not; Local(·) means extracting the local image information of all sub-blocks except the class token.

5. The detection method according to claim 4, characterized in that: In the text image alignment step S2 of the method, the encoded information alignment is to align the multi-scale image feature information with the text feature vector g described by the normal text n and the text feature vector g describing the abnormal text a Perform cosine similarity calculation to obtain the similarity score between the new image to be tested and the normal text description or abnormal text description: M n =sim(F cls ,g n ); M a =sim(F cls ,g a ); Among them, sim() represents the cosine similarity calculation, M n Represents the similarity score between the current overall image and the normal text description, that is, the probability of a normal sample; M a Represents the similarity score between the current overall image and the abnormal text description, that is, the probability of the abnormal sample; is the anomaly score used for defect localization, which is the average of the similarity scores between each sub-block and the normal text description and the abnormal text description at k scales.

6. The detection method according to claim 5, characterized in that: In the method, the normal text description includes a normal prompt word template; the abnormal text description includes a defect prompt word template; The normal prompt words are used as prompt templates for normal samples; The defect prompt word is used as a prompt template for abnormal samples; The normal prompt words are expressed as: Represents the encoding vector of the positive prompt word, and the initialized value conforms to the normal distribution with a mean of 0 and a variance of 0.02; N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding; Preferably, the defect prompt word is expressed as: The encoding vector representing the defect prompt word, the initialization value conforms to the normal distribution with a mean of 0 and a variance of 0.02; N n Represents the number of positive prompt words (n), N a represents the number of defect prompt words (a), L represents the length of each prompt word, and D represents the dimension of each prompt word encoding.

7. The detection method according to claim 1, characterized in that: In the product detection step S3 of the method, whether to classify as a normal sample or a defective sample is determined based on whether the similarity score between the entire image to be tested and the normal text description is greater than the similarity score between the entire image to be tested and the abnormal text description; when the similarity score between the entire image to be tested and the normal text description is greater than the similarity score between the entire image to be tested and the abnormal text description, it is classified as a normal sample; Otherwise, it is classified as a defective sample; Preferably, bilinear interpolation is used to restore the anomaly score used for defect location to the size of the image to be tested, and the defect position of the image to be tested is marked.

8. A defect detection device, characterized in that: The device comprises an image acquisition device, a processor and a memory, The image acquisition device is used to capture the image of the product to be tested; The processor is coupled to a memory, the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a processor, the processor performs the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises software code for executing the steps of the method according to any one of claims 1-7.

Images