Defect detection model training and defect detection method based on deep learning

Abstract

The invention relates to the technical field of image detection, and particularly provides a defect detection model training and defect detection method based on deep learning, and the method comprises the steps: 1, obtaining an original sample image; step 2, carrying out normalization processing on the original sample image; step 3, constructing a defect classification model; 4, judging whether defects exist or not and classifying the defects; 5, setting a judgment threshold value; step 6, obtaining defective block images in the step 5, and performing classification marking; step 7, performing marking and iterative training on the marked block image; 8, further training the classified and marked image; according to the method, two cutting steps are set for defect detection, down-sampling is performed on a sample image, a thumbnail corresponding to the sample image is obtained for training, the training time is shortened, and up-sampling is performed on a defective image, so that image features are clearer, smaller features are amplified, and the recognition accuracy is improved; according to the method, the false detection probability can be effectively reduced through multiple times of repeated training.

Description

Technical Field

[0001] The present invention relates to the technical field of image detection, and particularly to a method for training a defect detection model and defect detection based on deep learning. Background Art

[0002] In the process of semiconductor manufacturing, various defects may occur. Therefore, it is necessary to inspect semiconductor products for defects in actual production and eliminate the defective semiconductor products. In the process of defect detection of semiconductor products, the use of deep learning models is becoming increasingly mainstream.

[0003] Currently, machine learning technology is mostly used in production lines to train defect detection models, and based on the trained defect detection models, defect detection is performed on semiconductor product images to determine the defect types of unqualified products. Machine learning will deploy defect detection models based on existing defect data. However, in the field of industrial quality inspection, the deployed defect detection models can often only recognize the learned defect forms. However, in the actual production process, new defect forms may appear on the product surface as time goes by on the product production line. The defect detection models trained by machine learning technology cannot detect new defect forms, and the detection effect is not good. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a method for training a defect detection model and defect detection based on deep learning, which is used to solve the problem that the defect detection models trained by machine learning technology in the prior art cannot detect new defect forms.

[0005] To achieve the above purpose and other related purposes, the present invention provides a method for training a defect detection model and defect detection based on deep learning, including the following steps:

[0006] Step 1: Obtain original sample images;

[0007] Step 2: Perform normalization processing on the original sample images, and crop the original sample images into several block images of the same size;

[0008] Step 3: Construct a defect classification model. The defect classification model adopts a binary classification mechanism. By analyzing the characteristics of different defect types, n-dimensional features to be extracted are defined to form a set of feature vectors describing defects;

[0009] Step 4: Use the defect classification model to initially judge and classify the presence or absence of defects in each block image after normalization processing; the defect segmentation model trains the classified block images, and sets the first gradient confidence level, and centrally collects the block images that do not meet the prediction results as training samples;

[0010] Step 5: Set a judgment threshold in the defect classification model, and discard the judgment results of the block images with defects that are lower than the threshold after classification.

[0011] Step 6: Obtain the defective block images in Step 5, perform further cropping on them, and further classify and label the cropped block images according to the defect types.

[0012] Step 7: Mark the defect positions in the area with a confidence level lower than that of the block images that do not meet the prediction results in Step 4 through a heat map. At the same time, group the marked samples that do not meet the prediction results and the defects of the same type together for iterative training.

[0013] Step 8: Use the defect detection model to further train the classified and labeled images, and test the trained model. If misdetection is found, place the image in the correct classification for continued training until there is no misdetection.

[0014] In an embodiment of the present invention, the cropping in Step 2 is downsampling, which is used to obtain a thumbnail of the corresponding image; the cropping in Step 5 is upsampling, which is used to more clearly display the image details.

[0015] In an embodiment of the present invention, the downsampling includes average pooling and max pooling.

[0016] In an embodiment of the present invention, the upsampling includes interpolation and convolution.

[0017] In an embodiment of the present invention, in Step 3, the n-dimensional features include the length, width, contrast, texture features, entropy, and gradient of the defect.

[0018] In an embodiment of the present invention, in Step 4, the defect classification model uses a specific loss function. In the case of unbalanced positive and negative samples, the weights are adjusted through a balance factor and a hard sample adjustment factor to enable the model to focus on optimizing hard samples; the formula of the loss function is:

[0019] fl(p s )=-α(1-p s ) γ log(p s )

[0020] In the formula, p s is the prediction probability of the model for the correct class, fl(p s ) is the predicted value of the model, α is a constant,

[0021] (1-p s ) γ log(p s ) is the actual value.

[0022] In one embodiment of the present invention, in step seven, an attention mechanism is added to the iterative training, and the multi-scale input image and the attention model are jointly trained; the attention mechanism includes a first-layer filter and a second-layer filter, the size of the first-layer filter is 3*3*512, and the size of the second-layer filter is 1*1*M, where M is the number of scales used; the attention mechanism uses a weighted formula to learn multi-scale features, including the following formula:

[0023]

[0024] In the formula, is the score map of different scale outputs, represents the weights of different scales at each pixel position; After bilinear interpolation to the same resolution, it is weighted and fused into the final score map g i，b .

[0025] In one embodiment of the present invention, the method for describing the weights of different scales at each pixel position is as follows:

[0026]

[0027] In the formula, and are the outputs of the attention; the weights are normalized by softmax; represents that the weights of different scales at each pixel position are different, but are shared among all channels of the same scale and have no difference.

[0028] In one embodiment of the present invention, in step six, if there are multiple defects in the same piece of image, all of them need to be marked.

[0029] As described above, the defect detection model training and defect detection method based on deep learning of the present invention have the following beneficial effects:

[0030] The present invention constructs a defect detection model using a binary classification mechanism and uses a unique loss function to enable the model to focus on optimizing difficult samples, thereby improving the performance of the defect detection model. A first gradient confidence level is set in the defect detection model, and the block images that do not meet the prediction results are centrally collected as training samples. At the same time, the labeled samples that do not meet the prediction results and the defects of the same type are grouped together for iterative training, and the trained model is tested, which is conducive to quickly and accurately identifying defects in images and improving the detection efficiency. Compared with the prior art, the present invention can continuously identify new defects and perform iterative training, continuously adapt to new defect forms, and improve the defect detection effect. The present invention sets two cropping steps for defect detection. First, the sample image is downsampled to obtain a thumbnail of the corresponding sample image for training, which shortens the training time. Then, the defective image is upsampled to make the image features clearer, magnify the smaller features, and improve the recognition accuracy. Through repeated training multiple times, the present invention can effectively reduce the probability of false detection. Detailed implementation manners

[0031] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0032] The present invention provides a method for training a defect detection model and defect detection based on deep learning, including the following steps:

[0033] Step 1: Obtain an original sample image. Specifically, the original sample image is obtained by an industrial camera for image acquisition of semiconductor products.

[0034] Step 2: Normalize the original sample image and crop the original sample image into several block images of the same size. The cropping in this step is downsampling, including average pooling and max pooling. Specifically, average pooling is used to divide the image into non-overlapping regions, take the average of the pixel values in each region, and then replace the pixel values of the original region with the average value, which can effectively reduce the size and resolution of the image while retaining the main features of the image. Max pooling is used to divide the image into non-overlapping regions, take the maximum value of the pixel values in each region, and then replace the pixel values of the original region with the maximum value, which can reduce the size and resolution of the image while retaining the significant features of the image. Downsampling is used to obtain a thumbnail of the corresponding image for the purpose of shortening the training time.

[0035] Step 3: Construct a defect classification model. The defect classification model adopts a binary classification mechanism. Specifically, the binary classification mechanism refers to the process of dividing a dataset into two mutually exclusive categories. In machine learning, binary classification is a classification problem where the prediction results have only two categories, usually represented by 0 and 1, or by "yes" and "no", "positive" and "negative", etc. By analyzing the features of different defect types, define the n-dimensional features to be extracted, and form a set of feature vectors describing the defects. The n-dimensional features include the length, width, contrast, texture features, entropy, gradient, etc. of the defects.

[0036] Step 4: Use the defect classification model to initially judge and classify the presence or absence of defects in each normalized block image; the defect segmentation model trains the classified block images and sets the first gradient confidence level, and centrally collects the block images that do not meet the prediction results as training samples; the defect classification model uses a unique loss function. In the case of unbalanced positive and negative samples, the weights are adjusted through a balance factor and a hard sample adjustment factor to enable the model to focus on optimizing hard samples. The formula for the loss function is:

[0037] fl(p s )=-α(1-p s ) γ log(p s )

[0038] In the formula, p s is the prediction probability of the model for the correct category, fl(p s ) is the predicted value of the model, α is a constant, (1-p s ) γ log(p s ) is the actual value; the loss function is a non-negative real-valued function. The smaller its value, the closer the predicted result of the model is to the actual value, and the better the performance of the model. During the model training process, the loss function guides the optimization of the model by calculating the difference between the predicted value and the true value. The parameters of the model are gradually adjusted through methods such as gradient descent to minimize the loss function, thereby achieving the convergence of the model. The loss function is not only the core part of the empirical risk function but also includes the regularization term of the structural risk function to prevent overfitting.

[0039] Step 5: Set a judgment threshold in the defect classification model, and discard the judgment results of the block images with defects and lower than the threshold after classification; perform a large defect screening to avoid missed detections when there are fewer samples later.

[0040] Step 6: Obtain the defective block images in Step 5, perform further cropping on them, and further classify and label the cropped block images according to the defect types; if there are multiple defects in the same block image, all of them need to be labeled. The cropping in this step is upsampling, including interpolation methods and convolution methods; common interpolation methods include nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. According to the known pixel values around, the new pixel values are calculated to enlarge the image size; the convolution method performs convolution operations on the image by applying a specific convolution kernel (filter) to generate a larger-sized image; the upsampling is used to enlarge smaller features, making the image features clearer to capture more detailed information and improve the recognition accuracy.

[0041] Step 7: Mark the defect positions in the regions with confidence levels lower than the threshold for the block images that do not meet the prediction results in Step 4 through a heat map, and at the same time, group the marked samples that do not meet the prediction results and the defects of the same type together for iterative training; in this step, an attention mechanism is added to the iterative training, and the multi-scale input images and the attention model are jointly trained; the attention mechanism includes a first-layer filter and a second-layer filter. The size of the first-layer filter is 3*3*512, and the size of the second-layer filter is 1*1*M, where M is the number of scales used; the attention mechanism uses a weighted formula to learn multi-scale features, including the following formula:

[0042]

[0043] In the formula, is the score map output at different scales, represents the weights at different scales at each pixel position; After bilinear interpolation to the same resolution, they are weighted and fused into the final score map g i，b . The way to describe the weights at different scales at each pixel position is as follows:

[0044]

[0045] In the formula, and are the outputs of the attention, representing the attention output values at scale m / t and channel i; the weights are normalized by softmax; represents that the weights at different scales at each pixel position are different, but they are shared among all channels at the same scale and have no differences.

[0046] Step 8: Use the defect detection model to further train the classified and labeled images, and test the trained model. If misdetection is found, place the image in the correct classification for continued training until there is no misdetection.

[0047] The present invention constructs a defect detection model using a binary classification mechanism and uses a unique loss function to enable the model to focus on optimizing difficult samples, thereby improving the performance of the defect detection model. A first gradient confidence is set in the defect detection model, and the block images that do not meet the prediction results are centrally collected as training samples. At the same time, the labeled samples that do not meet the prediction results and the same type of defects are grouped together for iterative training, and the trained model is tested, which is conducive to quickly and accurately identifying defects in the image and improving the detection efficiency. Compared with the prior art, the present invention can continuously identify new defects and perform iterative training, continuously adapt to new defect forms, and improve the defect detection effect.

[0048] In summary, the present invention sets two cropping steps for defect detection. First, the sample image is downsampled to obtain a thumbnail of the corresponding sample image for training, which shortens the training time. Then, the defective image is upsampled to make the image features clearer and magnify the smaller features, improving the recognition accuracy. Through repeated training, the present invention can effectively reduce the false detection probability. Therefore, the present invention effectively overcomes various disadvantages in the prior art and has high industrial utilization value.

[0049] The above embodiments are only illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A defect detection model training and defect detection method based on deep learning, characterized in that, It includes the following steps: Step 1, obtain the original sample image; Step 2, perform normalization processing on the original sample image, and crop the original sample image into several block images with the same size; Step 3, construct a defect classification model. The defect classification model adopts a binary classification mechanism. By analyzing the features of different defect types, define the n-dimensional features to be extracted, and form a set of feature vectors describing the defects; Step 4, use the defect classification model to initially judge and classify the presence or absence of defects in each block image after normalization processing; The defect segmentation model trains the classified block images, and sets the first gradient confidence level, and centrally collects the block images that do not meet the prediction results as training samples; Step 5, set a judgment threshold in the defect classification model, and discard the judgment results of the block images with defects and lower than the threshold after classification; Step 6, obtain the defective block images in Step 5, perform further cropping processing on them, and further classify and label the cropped block images according to the defect types; Step 7, mark the defect positions in the area with a confidence level lower than that in the heat map for the block images that do not meet the prediction results in Step 4. At the same time, iteratively train the marked samples that do not meet the prediction results and the defects of the same type grouped together; Step 8, use the defect detection model to further train the classified and labeled images, and test the trained model. If misdetection is found, place the image in the correct classification and continue training until there is no misdetection.

2. The method for training a defect detection model and defect detection based on deep learning according to claim 1, characterized in that: The cropping in Step 2 is downsampling, which is used to obtain the thumbnail of the corresponding image; the cropping in Step 5 is upsampling, which is used to more clearly display the image details.

3. The defect detection model training and defect detection method based on deep learning according to claim 2, characterized in that: The downsampling includes average pooling and max pooling.

4. The defect detection model training and defect detection method based on deep learning according to claim 2, characterized in that: The upsampling includes interpolation method and convolution method.

5. The defect detection model training and defect detection method based on deep learning according to claim 1, characterized in that: In Step 3, the n-dimensional features include the length, width, contrast, texture feature, entropy, and gradient of the defect.

6. The defect detection model training and defect detection method based on deep learning according to claim 1, characterized in that In Step 4, the defect classification model uses a specific loss function. In the case of unbalanced positive and negative samples, adjust the weights through a balance factor and a hard sample adjustment factor to make the model focus on optimizing hard samples; the formula of the loss function is: pl(p s ) = -α(1 - p s ) γ log(p s ) where p s is the predicted probability of the model for the correct class, fl(p s ) is the predicted value of the model, α is a constant, (1 - p s ) γ log(p s ) is the actual value.

7. The defect detection model training and defect detection method based on deep learning according to claim 1, characterized in that: In Step 7, add an attention mechanism to the iterative training, and jointly train the multi-scale input image and the attention model; the attention mechanism includes a first-layer filter and a second-layer filter. The size of the first-layer filter is 3*3*512, and the size of the second-layer filter is 1*1*M, where M is the number of scales used; the attention mechanism uses a weighted formula to learn multi-scale features, including the following formula: In the formula, is the fractional map output at different scales, representing the weights at different scales at each pixel position; After bilinear interpolation to the same resolution, it is weighted and fused into the final score map g i，b .

8. The method for training a defect detection model and defect detection based on deep learning according to claim 7, characterized in that, Describe the weight method of different scales at each pixel position as follows: In the formula, and are the outputs of attention; The weights are normalized by softmax; It means that the weights at different scales are different at each pixel position, but are shared among all channels at the same scale and have no differences.

9. The defect detection model training and defect detection method based on deep learning according to claim 1, characterized in that: In Step 6, if there are multiple defects in the same block image, all need to be marked.

Images