Wafer appearance defect detection and classification method and system and computer storage medium
By using a combination of image acquisition and deep learning classification network in wafer appearance defect detection, the problems of long training time, large sample demand and low detection accuracy in the prior art are solved, and efficient and fast response defect recognition and Classification.
Patent Information
- Application Number
- CN202411492763.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-14
AI Technical Summary
The existing deep learning classification network has a long training time and a large sample demand in wafer appearance defect detection, and the emerging defect characteristics are quite different from existing sample characteristics, which affects the detection accuracy.
A method for detecting and classification of wafer appearance defects is proposed. The wafer is scanned by moving image acquisition components through the servo axis stepper motor, combined with image preprocessing, core particle positioning, segmentation and classification model feature extraction, using YOLOv8-cls classification network and The k-NN classification model performs deep feature extraction and outlier detection to achieve fast response defect identification and classification.
It realizes efficient detection of wafer appearance defects on small sample image datasets, reduces the time and labor costs of sample creation and maintenance, and improves detection accuracy and system fast response capabilities.
Smart Images
Figure CN119444695A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of wafer inspection technology, and in particular to a method, system and computer storage medium for detecting and classifying wafer appearance defects. Background Art
[0002] Nowadays, artificial intelligence is gradually empowering various industries, but there are still certain limitations in the application scenarios of some projects:
[0003] First, a large number of samples are required for training (fewer defect samples and more difficult to obtain). Generally speaking, positive samples (non-defective samples) are easier to obtain, while negative samples (defective samples) are more difficult to obtain, and the newly emerging defects are likely to be quite different from the characteristics of the existing defective samples.
[0004] Second, when the production process and appearance of the product change, or when the defect category needs to be modulated, the difference between the features of the training sample and the image to be detected becomes larger, which has a significant impact on the detection accuracy.
[0005] In summary, in order to reduce these differences or improve the robustness of the training model during reasoning, it is necessary to further maintain the original large-base training samples: add new samples, check the original samples or delete unnecessary interference samples, and retrain the model. It can only be used after being sent to the on-site equipment for testing. This is not friendly to production-oriented companies with tight production schedules and assembly lines.
[0006] In order to solve the above problems, we propose a method, system and computer storage medium for detecting and classifying wafer appearance defects. Summary of the invention
[0007] In view of this, the purpose of the invention is to propose a method, system and computer storage medium for detecting and classifying wafer appearance defects, so as to solve the problems of long training time and large sample requirements of existing deep learning classification networks.
[0008] Based on the above purpose, the invention provides a method for detecting and classifying wafer appearance defects, including reasoning process and model training, which specifically includes the following steps:
[0009] Step S1, moving the image acquisition component by the servo axis stepper motor to scan the RGB color image of the wafer area;
[0010] Step S2, while the image acquisition component scans across the wafer, the acquired image of a certain area of the wafer is assigned to different reasoning processes;
[0011] Step S3, the reasoning process includes image preprocessing, core particle location, segmentation, and classification model feature extraction;
[0012] Step S3.1, image preprocessing: normalize each RGB channel;
[0013] Step S3.2, core particle positioning: performing image smoothing processing on the image, and then determining the area of the wafer core particles through binarization processing;
[0014] Step S3.3, core grain segmentation: cut out each core grain area on the smoothed image in turn, and change the size of the core grain image to 244x 244x 3;
[0015] Step S3.4, classification model feature extraction: Use the YOLOv8-cls classification network pre-trained on ImageNet to obtain the feature output of the middle layer of the network as the deep feature information of the core grain image;
[0016] Step S4, model training includes sample collection, image augmentation and outlier detection model;
[0017] Step S4.1, sample collection: the cut core grain images are placed in the folders of the corresponding defect categories according to their appearance defects, among which the positive sample images without defects are placed in the established 'good' file, and generally 10-30 samples are placed in each category of defect samples;
[0018] Step S4.2, image augmentation: When training sample data, according to the defect category, the sample images of the corresponding category are augmented, and then the YOLOv8-cls classification network method pre-trained on ImageNet is used to extract the deep features of each image;
[0019] Then, kmeans is used to cluster each category separately, and the n center points obtained from each category clustering are used as the feature dictionary of the category;
[0020] Step S4.3, outlier detection model, includes the following sub-steps:
[0021] Step S4.31, the feature dictionary of the depth information of the positive sample image after the image augmentation in step S4.2 is used for training the outlier detection model LocalOutlierFactor, and the model is saved to the hard disk in the file form ml_model_LOF.pkl;
[0022] Step S4.32, using the feature dictionary of the depth information of the sample image of the defective core particle in step S4.2 and its corresponding category number as samples to train the k-NN classification model, and saving the model to the hard disk in the form of a file ml_model_knn.pkl;
[0023] Step S5, extracting deep feature information from the core grain image whose category needs to be predicted, and using the outlier detection model LocalOutlierFactor to pre-judge whether there is an anomaly;
[0024] Step S6, the predicted results will be saved in the log file: core particle image, outlier model: OK / NOK, classification: category name.
[0025] Preferably, in step S3.1, normalizing each RGB channel includes the following steps:
[0026] Calculate the target mean of each channel, and based on the target mean of each channel, calculate the coefficient v that needs to be adjusted for each channel, v = target mean / mean of the current channel of the original image, and then multiply the pixels of the corresponding channel by the corresponding coefficients v_r, v_g, v_b respectively.
[0027] Preferably, in step S3.3, during the segmentation process, the position of the core particle is converted to a global position by using the position of the core particle on the image and the position on the wafer corresponding to the predefined camera shooting position.
[0028] Preferably, in step S4.2, image augmentation is performed on the sample image including but not limited to based on hue, saturation, brightness, contrast change, image flipping, rotation, Gaussian noise, image displacement and center segmentation.
[0029] Preferably, in step S5, when using the outlier detection model LocalOutlierFactor to pre-judge whether there is an abnormality, if there is no defect, it is a normal core particle;
[0030] If there is an anomaly, the cosine similarity is calculated for each deep feature information in the defect feature dictionary in turn. The category number corresponding to the most similar feature after comparison with the feature information in the feature dictionary is the predicted defect category.
[0031] Preferably, in step S6, the inference result will be drawn on the original image for intuitive display and stored in the hard disk in the form of a log, so that when manual intervention is needed, manual intervention can be made at any time through the software interactive interface to adjust and optimize the training samples, and retraining can be carried out in spare time.
[0032] A wafer appearance defect detection and classification system, the system comprising:
[0033] Image processing: used to normalize each RGB channel;
[0034] Core positioning: used to determine the regional position of the wafer core;
[0035] Core grain segmentation: used to cut out each core grain area on the image and change the size of the core grain image;
[0036] Classification model feature extraction: used to extract deep feature information of mental images;
[0037] Sample collection: used to collect defective negative samples and non-defective positive samples, and place the positive and negative samples in corresponding folders;
[0038] Image augmentation: used to generate new versions of sample image data, thereby expanding the original sample training data set;
[0039] Outlier detection model: used to detect whether there are abnormalities in the core particles.
[0040] A computer storage medium stores a wafer appearance defect detection program, which implements the wafer appearance defect detection and classification method when executed by at least one processor.
[0041] Beneficial effects of the invention: The present invention realizes kmenas clustering based only on a small sample image data set, by extracting the features of the deep network from each category of images, and uses the n cluster centers of each category as a deep feature dictionary, and first performs outlier detection, and then uses the k-NN (nearest neighbor) classification model to realize the recognition and classification of wafer appearance defects. And by adding a human-computer interaction interface for sample management, manual intervention is performed on the classification results, reducing the time and labor costs of sample creation and maintenance, and when the features of the wafer defect image undergo large changes, the method can quickly respond to the changing detection requirements, so that the wafer defect detection system can be quickly implemented and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 It is a schematic diagram of the overall system flow of an embodiment of the invention;
[0044] Figure 2 A schematic diagram of the reasoning process flow of an embodiment of the invention;
[0045] Figure 3 A schematic diagram of a model training process of an embodiment of the invention;
[0046] Figure 4The confusion matrix diagram of the overall prediction results of the embodiment of the invention on the test set. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0048] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the invention should be understood by people with ordinary skills in the field to which the invention belongs. The "first", "second" and similar words used in the invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0049] like Figure 1 , Figure 2 , Figure 3 , Figure 4 As shown, a method for detecting and classifying wafer appearance defects, a method for detecting and classifying wafer appearance defects, including reasoning process and model training, specifically includes the following steps:
[0050] Step S1, moving the image acquisition component by the servo axis stepper motor to scan the RGB color image of the wafer area;
[0051] Step S2, while the image acquisition component scans across the wafer, the acquired image of a certain area of the wafer is assigned to different reasoning processes;
[0052] Step S3, the reasoning process includes image preprocessing, core particle location, segmentation, and classification model feature extraction;
[0053] Step S3.1, image preprocessing: normalize each channel (RGB) of the image; calculate the target mean of each channel, and based on the target mean of each channel, calculate the coefficient v that needs to be adjusted for each channel: v = target mean / mean of the current channel of the original image, and then multiply the pixels of the corresponding channel by the corresponding coefficient v (v_r, v_g, v_b).
[0054] Step S3.2, core particle positioning: perform image smoothing on the image, and then determine the area of the wafer core particles through binarization processing (if necessary, add morphological optimization binarization results);
[0055] Furthermore, the above-mentioned image normalization and image smoothing can be adjusted according to different image characteristics.
[0056] In addition, after image preprocessing, add a step: calculate the difference between the image and a predefined reference image (a defect-free core grain image that has undergone the same preprocessing). The difference image is used as input for subsequent processes to extract deep features (it is necessary to determine whether to use the difference image to extract features during inference and training).
[0057] Step S3.3, core grain segmentation: Cut out each core grain area on the smoothed image in turn, and change the size of the core grain image to 244x 244x 3; during the segmentation process, the core grain position is converted to the global position according to the position of the core grain on the image and the position on the wafer corresponding to the predefined camera shooting position.
[0058] Step S3.4, classification model feature extraction: Use the YOLOv8-cls classification network pre-trained on ImageNet to obtain the feature output of the middle layer of the network (after GlobalAvgPooling and before full layer connection in the Classify module) as the deep feature information of the core grain image;
[0059] The above-mentioned network for deep feature extraction can be replaced by other deep classification networks and deep feature extraction networks (such as yolov8-cls series, Resnet, GoogLeNet, etc.) according to sample characteristics, inference speed, and accuracy requirements.
[0060] Step S4, model training includes sample collection, image augmentation and outlier detection model;
[0061] Step S4.1, sample collection: The cut core grain images are placed in folders of corresponding defect categories according to their appearance defects, among which the defect-free positive sample images are placed in the established 'good' file, and generally 10-30 samples are placed in each category of defective samples; and the more easily obtained defect-free samples can be collected as much as possible (experiments show that the increase in the number of positive samples can reduce the proportion of over-detection during outlier model prediction).
[0062] Step S4.2, image augmentation: When training sample data, according to the defect category, the sample images of the corresponding category are augmented (based on hue, saturation, brightness, contrast changes; image flipping, rotation; Gaussian noise; image displacement; center segmentation, etc.), and then the YOLOv8-cls classification network method pre-trained on ImageNet is used to extract the deep features of each image;
[0063] Then, kmeans is used to cluster each category separately, and the n center points obtained from each category clustering are used as the feature dictionary of the category;
[0064] Step S4.3, outlier detection model, includes the following sub-steps:
[0065] Step S4.31, the feature dictionary of the depth information of the positive sample image after the image augmentation in step S4.2 is used for training the outlier detection model LocalOutlierFactor, and the model is saved to the hard disk in the file form ml_model_LOF.pkl;
[0066] Furthermore, the outlier detection algorithm can be replaced with another outlier detection algorithm based on positive samples according to different defect characteristics, which can be determined according to the specific usage scenario.
[0067] Step S4.32, using the feature dictionary of the depth information of the sample image of the defective core particle in step S4.2 and its corresponding category number as samples to train the k-NN (nearest neighbor) classification model, and saving the model to the hard disk in the form of a file ml_model_knn.pkl;
[0068] The above k-NN (nearest neighbor) classification algorithm can be replaced by other machine learning classification algorithms according to different business needs.
[0069] Step S5, extracting deep feature information from the core grain image whose category needs to be predicted, and using the outlier detection model LocalOutlierFactor to pre-judge whether there is an anomaly;
[0070] When using the outlier detection model LocalOutlierFactor to pre-judge whether there is an abnormality, if there is no defect, it is a normal core;
[0071] If there is an anomaly, the cosine similarity is calculated for each deep feature information in the defect feature dictionary in turn. The category number corresponding to the most similar feature after comparison with the feature information in the feature dictionary is the predicted defect category.
[0072] Step S6, the predicted results will be saved in the log file: core particle image, outlier model: OK / NOK, classification: category name.
[0073] In step S6, the inference result will be drawn on the original image for intuitive display and stored in the hard disk in the form of a log, so that when manual intervention is needed, manual intervention (manual re-inspection and secondary annotation) can be carried out at any time through the software interactive interface to adjust and optimize the training samples, and retrain in spare time.
[0074] This solution uses multiple GPUs for network reasoning. While the camera scans the wafer to acquire images, the acquired images are processed in parallel and assigned to different GPUs to extract deep features and predict defect categories, thereby improving detection efficiency.
[0075] When using the same training data and hardware, this solution is compared with directly using the corresponding deep network classification model: Taking YOLOv8m-cls as an example, when each achieves the best effect: this method reduces training time by 60%-70%, reduces inference time by 40%-55%, and improves accuracy by 9%-11%.
[0076] Overall accuracy of the algorithm: 96.13%, where the confusion matrix of the overall prediction results of the scheme on the test set is shown in Figure 4 shown.
[0077] The key parameters that affect the final detection effect in some algorithms in this scheme are shown in the following table;
[0078]
[0079] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the invention is limited to these examples; within the context of the invention, the technical features in the above embodiments or in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of simplicity.
[0080] The embodiments of the invention are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.
Claims
1. A method for detecting and classifying wafer appearance defects, characterized in that: Including the inference process and model training, specifically including the following steps: Step S1, moving the image acquisition component by the servo axis stepper motor to scan the RGB color image of the wafer area; Step S2, while the image acquisition component scans across the wafer, the acquired image of a certain area of the wafer is assigned to different reasoning processes; Step S3, the reasoning process includes image preprocessing, core particle location, segmentation, and classification model feature extraction; Step S3.1, image preprocessing: normalize each RGB channel; Step S3.2, core particle positioning: performing image smoothing processing on the image, and then determining the area of the wafer core particles through binarization processing; Step S3.3, core grain segmentation: cut out each core grain area on the smoothed image in turn, and change the size of the core grain image to 244x 244x 3; Step S3.4, classification model feature extraction: Use the YOLOv8-cls classification network pre-trained on ImageNet to obtain the feature output of the middle layer of the network as the deep feature information of the core grain image; Step S4, model training includes sample collection, image augmentation and outlier detection model; Step S4.1, sample collection: the cut core grain images are placed in the folders of the corresponding defect categories according to their appearance defects, among which the positive sample images without defects are placed in the established 'good' file, and generally 10-30 samples are placed in each category of defect samples; Step S4.2, image augmentation: When training sample data, perform image augmentation on sample images of corresponding categories according to defect categories, and then use the YOLOv8-cls classification network method pre-trained on ImageNet to extract deep features of each image; Then, kmeans is used to cluster each category separately, and the n center points obtained from each category clustering are used as the feature dictionary of the category; Step S4.3, outlier detection model, includes the following sub-steps: Step S4.31, the feature dictionary of the depth information of the positive sample image after the image augmentation in step S4.2 is used for training the outlier detection model LocalOutlierFactor, and the model is saved to the hard disk in the file form ml_model_LOF.pkl; Step S4.32, using the feature dictionary of the depth information of the sample image of the defective core particle in step S4.2 and its corresponding category number as samples to train the k-NN classification model, and saving the model to the hard disk in the form of a file ml_model_knn.pkl; Step S5, extracting deep feature information from the core grain image whose category needs to be predicted, and using the outlier detection model LocalOutlierFactor to pre-judge whether there is an anomaly; Step S6, the predicted results will be saved in the log file: core particle image, outlier model: OK / NOK, classification: category name.
2. The method for detecting and classifying wafer appearance defects according to claim 1, characterized in that: In step S3.1, normalizing each RGB channel includes the following steps: Calculate the target mean of each channel, and based on the target mean of each channel, calculate the coefficient v that needs to be adjusted for each channel, v = target mean / mean of the current channel of the original image, and then multiply the pixels of the corresponding channel by the corresponding coefficients v_r, v_g, v_b respectively.
3. The method for detecting and classifying wafer appearance defects according to claim 1, characterized in that: In step S3.3, during the segmentation process, the position of the core particle is converted to a global position by using the position of the core particle on the image and the position on the wafer corresponding to the predefined camera shooting position.
4. The method for detecting and classifying wafer appearance defects according to claim 1, characterized in that: In step S4.2, the sample image is augmented including but not limited to hue, saturation, brightness, contrast change, image flipping, rotation, Gaussian noise, image displacement and center segmentation.
5. The method for detecting and classifying wafer appearance defects according to claim 1, characterized in that: In step S5, when using the outlier detection model LocalOutlierFactor to pre-judge whether there is an abnormality, if there is no defect, it is a normal core particle; If there is an anomaly, the cosine similarity is calculated for each deep feature information in the defect feature dictionary in turn. The category number corresponding to the most similar feature after comparison with the feature information in the feature dictionary is the predicted defect category.
6. The method for detecting and classifying wafer appearance defects according to claim 1, characterized in that: In step S6, the inference result will be drawn on the original image for intuitive display and stored in the hard disk in the form of a log, so that when manual intervention is needed, manual intervention can be made at any time through the software interactive interface to adjust and optimize the training samples, and retraining can be carried out in spare time.
7. A wafer appearance defect detection and classification system, characterized in that: The system includes: Image processing: used to normalize each RGB channel; Core positioning: used to determine the regional position of the wafer core; Core grain segmentation: used to cut out each core grain area on the image and change the size of the core grain image; Classification model feature extraction: used to extract deep feature information of mental images; Sample collection: used to collect defective negative samples and non-defective positive samples, and place the positive and negative samples in corresponding folders; Image augmentation: used to generate new versions of sample image data, thereby expanding the original sample training data set; Outlier detection model: used to detect whether there are abnormalities in the core particles.
8. A computer storage medium storing a wafer appearance defect detection program, characterized in that: When the wafer appearance defect detection program is executed by at least one processor, the wafer appearance defect detection and classification method according to any one of claims 1 to 6 is implemented.