Defect classification method based on template image

Abstract

The invention discloses a defect classification method based on a template image. The method comprises the following steps: preparing an OK template image; preparing a sample defect image, a sample OK image and a reference template drawing; calculating the gray difference between the sample defect image or the sample OK image and the reference template image one by one, correspondingly generating a difference image, and synthesizing a difference sample; classifying and marking all differential samples, inputting a convolutional neural network model, training until the model converges, and forming a new model; acquiring an image of a product to be detected, and generating a preliminary defect image and a new difference image; through the mode, the defect classification method based on the template image, provided by the invention, has the advantages that on the basis of solving the defect classification problem by adopting a general model, the problem that the model is not applicable due to relatively large difference of different image backgrounds is avoided through secondary iteration of the model and secondary generation of the difference image; therefore, repeated training is avoided, and the reliability of defect category detection is fully improved.

Description

Technical Field

[0001] The present invention relates to the field of automatic optical detection, and in particular to a defect classification method based on template images. Background Art

[0002] Traditional defect classification methods use defect features, such as grayscale, contrast, etc., which are difficult to implement. Since the grayscales of various defects are similar, the classification accuracy is low. With the increase in product types, different standards need to be set for each type of product, which greatly affects the use of operators; since the patterns of different products are different, deep learning needs to be trained for each type of product, and a network is deployed for classification for each type of product, which has extremely low training and use efficiency.

[0003] CN112884744A proposes to mark defects, use deep learning models for training, and use the trained models for detection, which can greatly reduce false detections. However, different product types will lead to different training models, that is, different networks need to be trained and deployed on site according to different products.

[0004] CN114913110A proposes to continue to add training samples on the basis of the original detection model, so as to correct the model, improve accuracy and realize training of small samples. When the product type is changed, there is no need to re-establish the learning model. The model can be iterated based on a small number of samples to update the learning model and improve the reusability of the model. However, in actual applications, there are large differences between product types. When a small number of samples of new products are used to iterate the model, the background differences between different types of products are large, resulting in the inability to achieve the expected effect.

[0005] Patent CN115965813A proposes to synthesize a 5-channel image consisting of the first template difference image, the second template difference image, the residual point image, the high template image, and the low template image and input it into a deep learning model to improve the accuracy. This solution has redundant inputs. The residual point image and the high and low template images contain the first and second template difference images. Since the images are too redundant, a single model can improve the accuracy of detecting a single product, but it cannot be used in multiple categories of products. The problem of using the same model for multiple categories of products proposed in this patent cannot be solved. Summary of the invention

[0006] The main technical problem solved by the present invention is to provide a defect classification method based on template images. On the basis of adopting a general model to solve the defect classification problem, the problem of model inapplicability caused by large differences in the backgrounds of different images is avoided through secondary iteration of the model and secondary generation of differential images, thereby avoiding multiple trainings and fully improving the reliability of defect category detection.

[0007] In order to solve the above technical problems, a technical solution adopted by the present invention is: to provide a defect classification method based on a template image, comprising the following steps:

[0008] Step 1: prepare a batch of OK product images, perform template matching and affine transformation alignment on each OK product image; then take the grayscale mean to obtain an OK template image;

[0009] Step 2: Prepare a batch of NG training samples and a batch of OK training samples.

[0010] Input NG training samples into the network, identify defects, add defect labels, and capture sample defect images;

[0011] Extract the coordinates of the sample defect image, and cut out the sample OK image at the corresponding coordinates of the OK training sample;

[0012] Step 3, extracting the coordinates of the sample defect image or the sample OK image, and cutting out the reference template image at the corresponding coordinates of the OK template image;

[0013] Step 4, calculating the grayscale difference between the sample defect image or the sample OK image and the reference template image one by one, generating a differential image accordingly, and synthesizing a differential sample;

[0014] Step 5: Classify and label all differential samples, input them into the convolutional neural network model for training until the model converges to form a new model;

[0015] Step 6: Collect images of the product to be tested, perform template matching and affine transformation alignment respectively; input the images into the network, identify preliminary defects, add preliminary defect marks, and capture preliminary defect images;

[0016] Extract the coordinates of the preliminary defect image, cut off the reference template image at the corresponding coordinates of the OK template image, calculate the grayscale difference between the preliminary defect image and the reference template image one by one, and generate a new differential image accordingly;

[0017] Step seven: deploy a new model, input the new differential image into the new model for defect classification, and obtain a new classification result.

[0018] Step 8: Use the new classification results to verify and correct the preliminary defect markings.

[0019] In a preferred embodiment of the present invention, the grayscale mean value is obtained by the following formula:

[0020]

[0021] Among them, I M (x, y) represents the grayscale of all pixels of the template image in the xy coordinate system, i(x, y). It represents the grayscale of all pixels of the i-th OK product image in the xy coordinate system.

[0022] In a preferred embodiment of the present invention, the method for identifying defects in step 2 includes but is not limited to a template comparison method or a threshold segmentation method.

[0023] In a preferred embodiment of the present invention, the steps 2 and 6 also include the following method:

[0024] The sample defect image, sample OK image, and preliminary defect image are scaled and the minimum enclosing rectangle is obtained respectively. If the length or width of the enclosing rectangle is less than the threshold r pixels, an image of r×r pixels is expanded and intercepted with the center of the minimum enclosing rectangle; if the length or width is greater than the threshold r pixels, a screenshot is taken according to the actual size, and then the image is scaled to the size of r×r pixels.

[0025] The beneficial effect of the present invention is that a defect classification method based on template images provided by the present invention, on the basis of adopting a general model to solve the defect classification problem, avoids the problem of model inapplicability caused by large differences in the backgrounds of different images through secondary iteration of the model and secondary generation of differential images, thereby avoiding multiple trainings and fully improving the reliability of defect category detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0027] Figure 1 A wafer image to be tested according to a preferred embodiment of a defect classification method based on a template image of the present invention;

[0028] Figure 2 These are examples of defect images to be classified and template images at the same location;

[0029] Figure 3 It is a schematic diagram of local calculation of differential image. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] like Figure 1-3As shown, the embodiment of the present invention includes:

[0032] A defect classification method based on a template image comprises the following steps:

[0033] Step 1: prepare a batch of OK product images, perform template matching and affine transformation alignment on each OK product image; then take the grayscale mean to obtain an OK template image;

[0034] The step 1 is the image modeling process, taking a wafer sample as an example, Figure 1 As shown in the figure, there are multiple products on each wafer, and multiple groups of product images on multiple wafers can be used for modeling. The modeling method adopted is average grayscale modeling. The products on the wafer are segmented, and the single product images are segmented; each single product image is template matched and affine transformed and aligned; for each single product image after affine transformation, the template image is obtained by summing and averaging.

[0035] Here, the template image is obtained by calculating the grayscale average value. The template image is obtained by calculating the average value of a single product image that is confirmed to be defect-free.

[0036] Step 2: Prepare a batch of NG training samples and a batch of OK training samples.

[0037] Input NG training samples into the network, identify defects, add defect labels, and capture sample defect images;

[0038] Extract the coordinates of the sample defect image, and cut out the sample OK image at the corresponding coordinates of the OK training sample;

[0039] The second step is the collection process of training set samples. Taking wafer samples as an example, defect samples on the wafer are used. 200-300 images of each type of defect sample are prepared. The samples contain all types of defects according to process requirements. Traditional algorithms (template matching, threshold segmentation, etc.) are used for detection to obtain defect images to be classified, and each defect image to be classified is intercepted on the detection image. The size of the defect image to be classified is set to 256*256. Figure 2 As shown in the figure, the red outline area is the detected defect area. The minimum enclosing rectangle is obtained. If the length or width of the enclosing rectangle is less than 256, the original enclosing rectangle center is automatically expanded to 256*256 size. If the length or width is greater than 256, it is intercepted according to the actual length. Finally, the image is scaled to 256*256 size. At the same time, some defect-free images are intercepted according to the same rules as OK image samples.

[0040] Step 3, extracting the coordinates of the sample defect image or the sample OK image, and cutting out the reference template image at the corresponding coordinates of the OK template image;

[0041] Specifically, for each defect image to be classified and the OK image, the image at the same position on the template image is also intercepted corresponding to the template image position. The interception rule is consistent with the previous steps. The intercepted image is as follows: Figure 2 shown.

[0042] Step 4, calculating the grayscale difference between the sample defect image or the sample OK image and the reference template image one by one, generating a differential image accordingly, and synthesizing a differential sample;

[0043] In order to remove background interference, the image captured by the original image is subtracted from the image captured by the template to obtain a differential image. The pixel value range of the differential image is [-255, 255]. Figure 3 As shown, after subtracting the pixels at a certain point, the pixel value is 0-191=-191.

[0044] Step 5: Classify and label all differential samples, input them into the convolutional neural network model for training until the model converges to form a new model;

[0045] In this step, the differential image samples are labeled using three types of defects, namely dirt, scratches and bubbles, and the samples include OK samples. The samples are input into the model for training until the model converges. The model selects a convolutional neural network (such as ResNet).

[0046] Step 6: Collect images of the product to be tested, perform template matching and affine transformation alignment respectively; input the images into the network, identify preliminary defects, add preliminary defect marks, and capture preliminary defect images;

[0047] Extract the coordinates of the preliminary defect image, cut off the reference template image at the corresponding coordinates of the OK template image, calculate the grayscale difference between the preliminary defect image and the reference template image one by one, and generate a new differential image accordingly;

[0048] Step seven: deploy a new model, input the new differential image into the new model for defect classification, and obtain a new classification result.

[0049] In this step, during actual inspection, the trained model is deployed to the inspection end. First, the wafer image is segmented to obtain the current single product image. The single product image is matched with the template image to obtain the single product image after affine transformation.

[0050] For the current single product image, the traditional algorithm is used for defect detection to obtain the defect area. The minimum bounding rectangle of the defect area is taken. If the length or width of the bounding rectangle is less than 256, the original bounding rectangle center is automatically expanded to capture the size of 256*256. If the length or width is greater than 256, it is captured according to the actual length. Finally, the defect image is scaled to 256*256. At the same time, the image at the same position in the template image is captured according to the same rules.

[0051] In order to remove background interference, the image captured by the original image is subtracted from the image captured by the template to obtain a differential image.

[0052] Step 8: Use the new classification results to verify and correct the preliminary defect markings.

[0053] In this step, the differential image is input into the trained model for defect classification. If the traditional algorithm misdetects, the model can mark the image as OK and filter the defective image, thereby reducing false detections and achieving accurate classification.

[0054] The grayscale mean is obtained by the following formula:

[0055]

[0056] Among them, I M (x, y) represents the grayscale of all pixels of the template image in the xy coordinate system, i (x, y). It represents the grayscale of all pixels of the i-th OK product image in the xy coordinate system.

[0057] The method for identifying defects in step 2 includes but is not limited to a template comparison method or a threshold segmentation method.

[0058] In addition, the steps 2 and 6 also include the following method:

[0059] The sample defect image, sample OK image, and preliminary defect image are scaled and the minimum enclosing rectangle is obtained respectively. If the length or width of the enclosing rectangle is less than the threshold r pixels, an image of r×r pixels is expanded and intercepted with the center of the minimum enclosing rectangle; if the length or width is greater than the threshold r pixels, a screenshot is taken according to the actual size, and then the image is scaled to the size of r×r pixels.

[0060] This method can effectively avoid false detection and over-detection, improve classification accuracy, and use differential images to remove background interference, which can be effectively applied to multiple types of products. When switching to new products, the original model can still be used for classification, or a small sample can be added for iterative training to obtain a new model, thereby improving classification accuracy and model applicability.

[0061] In summary, the present invention provides a defect classification method based on template images. On the basis of adopting a general model to solve the defect classification problem, the problem of model inapplicability caused by the large differences in the backgrounds of different images is avoided through secondary iteration of the model and secondary generation of differential images, thereby avoiding multiple trainings and fully improving the reliability of defect category detection.

[0062] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A defect classification method based on template image, characterized in that: The following steps are involved: Step 1: prepare a batch of OK product images, perform template matching and affine transformation alignment on each OK product image; then take the grayscale mean to obtain an OK template image; Step 2: Prepare a batch of NG training samples and a batch of OK training samples. Input NG training samples into the network, identify defects, add defect labels, and capture sample defect images; Extract the coordinates of the sample defect image, and cut out the sample OK image at the corresponding coordinates of the OK training sample; Step 3, extracting the coordinates of the sample defect image or the sample OK image, and cutting out the reference template image at the corresponding coordinates of the OK template image; Step 4, calculating the grayscale difference between the sample defect image or the sample OK image and the reference template image one by one, generating a differential image accordingly, and synthesizing a differential sample; Step 5: Classify and label all differential samples, input them into the convolutional neural network model for training until the model converges to form a new model; Step 6: Collect images of the product to be tested, perform template matching and affine transformation alignment respectively; input the images into the network, identify preliminary defects, add preliminary defect marks, and capture preliminary defect images; Extract the coordinates of the preliminary defect image, cut off the reference template image at the corresponding coordinates of the OK template image, calculate the grayscale difference between the preliminary defect image and the reference template image one by one, and generate a new differential image accordingly; Step seven: deploy a new model, input the new differential image into the new model for defect classification, and obtain a new classification result. Step 8: Use the new classification results to verify and correct the preliminary defect markings.

2. The defect classification method based on template image according to claim 1, characterized in that: The grayscale mean value is obtained by the following formula: Among them, I M (x, y) represents the grayscale of all pixels in the template image in the xy coordinate system, I i (x,y). It represents the grayscale of all pixels of the i-th OK product image in the xy coordinate system.

3. The defect classification method based on template image according to claim 1, characterized in that: The method for identifying defects in step 2 includes but is not limited to a template comparison method or a threshold segmentation method.

4. The defect classification method based on template image according to claim 1, characterized in that: The steps 2 and 6 also include the following method: The sample defect image, sample OK image, and preliminary defect image are scaled and the minimum enclosing rectangle is obtained respectively. If the length or width of the enclosing rectangle is less than the threshold r pixels, an image of r×r pixels is expanded and intercepted with the center of the minimum enclosing rectangle; if the length or width is greater than the threshold r pixels, a screenshot is taken according to the actual size, and then the image is scaled to the size of r×r pixels.

Images