Generalizable deep learning defect detection method based on conditional tokens

Abstract

The present invention discloses a generalizable deep learning defect detection method based on conditional tokens, which relates to the fields of computer vision and deep learning defect detection technology. The method includes selecting #imgabs0# qualified sample images as reference images for the same batch of samples, dividing the sample images to be detected into small images of the area to be detected, and obtaining a reference small image at the same position on the reference image; amplifying the reference small image, using the amplified reference small image and the small image of the area to be detected as inputs of the Transformer network, obtaining the key feature information between the two through a multi-layer self-attention mechanism, performing feature fusion, and then using the decision network to perform defect detection to obtain the position mask and category of the defect. Therefore, the above-mentioned generalizable deep learning defect detection method based on conditional tokens can cope with various environmental noises and product process changes faced by the industrial detection field, and improve the accuracy and robustness of defect detection.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning defect detection technology, and in particular to a generalizable deep learning defect detection method based on conditional tokens. Background Art

[0002] As the manufacturing industry rapidly evolves toward intelligent and automated processes, product quality inspection is becoming increasingly important in industrial production. Traditional defect detection methods primarily rely on manual visual inspection or rule-based algorithms. These methods are labor-intensive, inefficient, inaccurate, and susceptible to human influence. To address these issues, deep learning-based defect detection technology has been widely used in the industrial sector, showing great potential in improving detection accuracy and reducing human intervention.

[0003] Convolutional neural networks (CNNs) in deep learning have achieved some success in defect detection, but they have limitations in capturing global image information and handling long-range dependencies. The Transformer, originally developed as a model for natural language processing tasks, leverages its powerful self-attention mechanism to effectively capture global contextual information. As the Transformer architecture continues to gain traction in computer vision, research has applied it to image processing, enhancing defect detection performance, particularly in complex structures and large-scale data.

[0004] However, existing deep learning defect detection methods have weak generalization capabilities when dealing with diverse and uncertain defects. In particular, when training data is insufficient or the sample distribution changes, the model's performance often degrades significantly. To address this issue, a defect detection method based on conditional Transformer tokens is proposed. By introducing conditional information, the model's generalization capabilities are enhanced, enabling efficient defect detection in different scenarios and maintaining high detection accuracy and robustness in a variety of industrial applications. Summary of the Invention

[0005] The purpose of the present invention is to provide a generalizable deep learning defect detection method based on conditional tokens, which can solve the generalization problem of defect detection, learn deeper image features, and improve the defect detection performance of the deep learning model.

[0006] To achieve the above objectives, the present invention provides a generalizable deep learning defect detection method based on conditional tokens, comprising the following steps:

[0007] S1. For the same batch of samples, select A qualified sample image is used as a reference image, and the sample image to be tested is divided into small images of the test area. At the same time, a reference small image at the same position on the reference image is obtained and amplified to increase the diversity of the sample;

[0008] S2: The amplified reference image and the image of the area to be detected are used as inputs to the Transformer network. A shared encoder is used to extract features from each reference image, extract the corresponding global semantic representation, and aggregate them into conditional tokens. At the same time, image block features are extracted from the image of the area to be detected. The conditional tokens and the image block features of the image of the area to be detected are combined as a joint input, and the first feature is obtained through the self-attention mechanism.

[0009] S3. Through the cross-image attention fusion module, the image block features of the small image of the inspection area and the reference small image are extracted respectively. The image block features of the small image of the inspection area are used to generate a query vector, and the image block features of the reference small image are used to generate a key vector and a value vector. The second feature is obtained through the cross-attention mechanism, and the first and second features are weightedly fused to obtain the fused feature;

[0010] S4. Based on the fused features, we construct a total loss function by combining embedding distance loss and cross entropy loss, and use the decision network to perform defect detection to obtain a trained deep learning defect detection model.

[0011] S5. For the same batch of samples to be tested, the trained deep learning defect detection model is used to extract the key features between the small image of the area to be tested and the reference small image. The softmax score map is then obtained through the decision network to obtain the category mask corresponding to the maximum probability, thereby obtaining the defect location and category information of the samples to be tested.

[0012] Preferably, step S1 includes selecting a reference image as a standard image, and performing correction and registration processing on the other reference images and the sample image to be tested with the standard image.

[0013] Preferably, in step S2, enlarging the reference thumbnail includes processing the image by random translation and noise.

[0014] Preferably, in step S2, using the decision network to perform defect detection includes updating and optimizing network parameters based on the total loss function through stochastic gradient descent and back propagation algorithms.

[0015] Preferably, the expression of the total loss function is as follows:

[0016] ;

[0017] in,

[0018] ;

[0019] ;

[0020] ;

[0021] Where, is the embedding distance loss, is the cross entropy loss, 、 is the corresponding weight coefficient, The thumbnail of the area to be detected image patch features, is a conditional token, Indicates the area to be detected. The matching of image blocks and conditional tokens, is the interval boundary, for The corresponding truth value, is the input of the softmax layer, is the output of the softmax layer, represents the number of categories, represents the Softmax cross entropy loss.

[0022] Therefore, the present invention adopts the above-mentioned generalizable deep learning defect detection method based on conditional tokens, which has the following technical effects:

[0023] (1) Based on the comparison learning method of Transformer conditional tokens, the same area of the qualified product is used as a reference image for comparative analysis to obtain deeper image features and improve the accuracy of defect detection.

[0024] (2) The pre-trained defect detection model of the present invention has stronger robustness and can cope with various environmental noises and product process changes faced in the industrial inspection field, and has a wider range of applications.

[0025] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of a small image of a test area and a reference small image in an embodiment of a generalizable deep learning defect detection method based on conditional tokens;

[0027] Figure 2 2. Schematic diagram of a deep learning defect detection model structure in an embodiment of a generalizable deep learning defect detection method based on conditional tokens. DETAILED DESCRIPTION

[0028] The present invention can be explained in more detail by the following examples. The purpose of disclosing the present invention is to protect all changes and improvements within the scope of the present invention. The present invention is not limited to the following examples.

[0029] Deep learning defect detection methods usually use image data recognition and analysis to detect defects on the surface of objects, such as cracks, scratches, stains, etc. in the manufacturing process. They are widely used in automated quality inspection and control of industrial manufacturing and production lines, helping to automatically detect product defects and improve production efficiency. In general, deep learning models usually need to use a large amount of training data to automatically learn the patterns and features of detecting specific defects. However, the biggest problem facing defect detection is the generalization problem. If a model trained on a batch of pictures is replaced with a batch of new pictures for testing, the performance will drop significantly. Based on this, the present invention provides a generalizable deep learning defect detection method based on conditional tokens, which uses a set of reference images to generate conditional tokens (Condition Token) representing the "normal sample distribution" to guide the model to focus on the difference area when comparing the reference image and the image to be tested, thereby realizing unsupervised or weakly supervised comparative learning, including the following steps:

[0030] S1. For samples from the same batch, select Qualified sample images are used as reference images, also known as reference OK sample images. To minimize the overall differences between reference OK sample images, the reference OK sample images need to be calibrated and registered before defect detection. Specifically, one reference OK sample image is used as a benchmark for calibration and registration of all other reference OK samples. Similarly, calibration and registration are performed on all samples to be tested.

[0031] See Figure 1 , given a sample image to be inspected, divide it into small images of the area to be inspected ,For example or Size, and cut out the area on the reference OK sample image that is at the same position as the small image of the area to be inspected as the reference small image .

[0032] In order to reduce the impact of various environmental noises and product process changes on the samples, this embodiment performs random amplification on the reference thumbnail, such as random translation and noise operations, to increase the diversity of the image.

[0033] S2, see Figure 2 , first, through the shared encoder of the Transformer model Extract features of each amplified reference image , and extract global semantic representations (such as CLS tokens) from each feature , and then aggregate the global semantic representation of the extracted reference thumbnail into a conditional token , which is used to represent the average features of the reference image and can be regarded as the semantic “prototype” of the qualified product.

[0034] Then, the same shared encoder of the Transformer model is used to extract Image patch features , and compare it with As the combined input, , extract the first feature through the self-attention mechanism , in order to achieve conditionally guided attention contrast learning.

[0035] S3. To better capture the differences between the reference image and the image to be tested, this embodiment also designs a cross-image attention fusion module to perform feature comparison between different images. Through a multi-layer attention mechanism, key information such as similar features and difference features between the two images is found to obtain fused features. The specific operations are as follows:

[0036] The image to be tested and the reference image are input into the shared encoder to extract the corresponding image block features. Used to generate query vectors , image block features of the reference small image Used to generate key vectors Sum vector , the second feature is obtained through the cross attention mechanism , and then and Perform weighted fusion to obtain the fused features ,as follows:

[0037] ;

[0038] Where, It can be a constant, and in this embodiment, it is 0.5. In another embodiment, It can be learned through the attention mechanism.

[0039] S4. Based on the fused features By defining the embedding distance loss between positive and negative sample pairs and combining it with the cross entropy loss of defect detection, a total loss function is constructed. The decision network is then used for defect detection, and the location mask and category of the X defect are output.

[0040] The expression of the total loss function is:

[0041] ;

[0042] Where, is the embedding distance loss, is the cross entropy loss, 、 is the corresponding weight coefficient.

[0043] Among them, the expression of embedding distance loss is:

[0044] ;

[0045] Where, Representation diagram matches the conditional token (normal), Indicates mismatch (abnormal), The interval boundary.

[0046] Cross entropy loss is used to measure the output results and defect mask annotation information The error between them is expressed as follows:

[0047] ;

[0048] ;

[0049] Where, for The corresponding truth value, is the input of the softmax layer, is the output of the softmax layer, represents the number of categories, represents the Softmax cross entropy loss.

[0050] Based on the above total loss function, the stochastic gradient descent and back propagation algorithms are used to update and optimize the network parameters to obtain the closest The defect category mask is used to implement the pre-training process of the deep learning defect detection model.

[0051] S5. For the same batch of samples to be tested, the trained deep learning defect detection model is used to extract the key features between the test image and the reference image. The softmax score map is then obtained through the decision network to obtain the category mask corresponding to the maximum probability, thereby obtaining the defect location and category information of the test sample, which can greatly improve the accuracy and robustness of defect detection.

[0052] Therefore, the present invention adopts the above-mentioned generalizable deep learning defect detection method based on conditional tokens, introduces qualified sample images as conditions, obtains deeper image features through comparative learning, and can cope with various environmental noises and product process changes faced in the industrial inspection field, and has stronger robustness.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A generalizable deep learning defect detection method based on conditional tokens, characterized by: The following steps are involved: S1. For the same batch of samples, select A qualified sample image is used as a reference image, and the sample image to be tested is divided into small images of the test area. At the same time, a reference small image at the same position on the reference image is obtained and amplified to increase the diversity of the sample; S2: The amplified reference image and the image of the area to be detected are used as inputs to the Transformer network. A shared encoder is used to extract features from each reference image, extract the corresponding global semantic representation, and aggregate them into conditional tokens. At the same time, image block features are extracted from the image of the area to be detected. The conditional tokens and the image block features of the image of the area to be detected are combined as a joint input, and the first feature is obtained through the self-attention mechanism. S3. Through the cross-image attention fusion module, the image block features of the small image of the inspection area and the reference small image are extracted respectively. The image block features of the small image of the inspection area are used to generate a query vector, and the image block features of the reference small image are used to generate a key vector and a value vector. The second feature is obtained through the cross-attention mechanism, and the first and second features are weightedly fused to obtain the fused feature; S4. Based on the fused features, we construct a total loss function by combining embedding distance loss and cross entropy loss, and use the decision network to perform defect detection to obtain a trained deep learning defect detection model. S5. For the same batch of samples to be tested, the trained deep learning defect detection model is used to extract the key features between the small image of the area to be tested and the reference small image. The softmax score map is then obtained through the decision network to obtain the category mask corresponding to the maximum probability, thereby obtaining the defect location and category information of the samples to be tested.

2. The generalizable deep learning defect detection method based on conditional tokens according to claim 1 is characterized in that: In step S1, a reference image is selected as a standard image, and other reference images and the sample image to be tested are calibrated and registered with the standard image.

3. The generalizable deep learning defect detection method based on conditional tokens according to claim 1, characterized in that: In step S2, enlarging the reference thumbnail includes processing the image by random translation and noise.

4. The generalizable deep learning defect detection method based on conditional tokens according to claim 1, characterized in that: In step S2, defect detection using the decision network includes updating and optimizing network parameters based on the total loss function through stochastic gradient descent and back propagation algorithms.

5. The generalizable deep learning defect detection method based on conditional tokens according to claim 1, characterized in that: The expression of the total loss function is as follows: ； in, ； ； ； Where, is the embedding distance loss, is the cross entropy loss, 、 is the corresponding weight coefficient, The thumbnail of the area to be detected image patch features, is a conditional token, Indicates the area to be detected. The matching of image blocks and conditional tokens, is the interval boundary, for The corresponding truth value, is the input of the softmax layer, is the output of the softmax layer, represents the number of categories, represents the Softmax cross entropy loss.

Images