Semiconductor defect classification method using data enhancement and lightweight neural network
The data volume is expanded through data augmentation technology and the use of lightweight neural networks to classify semiconductor wafer defects, solving the problem of insufficient rapid and accurate detection efficiency in the existing technology, especially in the case of imbalance in defect categories, and realizing the Efficient and accurate defect identification.
Patent Information
- Application Number
- CN202411501819.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-11
AI Technical Summary
The prior art is difficult to achieve fast and accurate semiconductor wafer defect detection, especially when facing the problem of imbalance in defect categories, there are insufficient detection efficiency and accuracy.
Data augmentation technology is used to generate synthetic data through affine transformation, expand the number of a few data, and use lightweight neural networks to classify wafer defects. Specific steps include cropping images, defect segmentation, data augmentation, building a hybrid data set, and constructing and training a convolutional neural network.
Through the combination of data augmentation and lightweight neural networks, the rapid and accurate identification of semiconductor wafer defects is achieved, and the detection efficiency and overall detection quality are improved, especially in the case of imbalance in the treatment of defect categories.
Smart Images
Figure CN119417793A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of semiconductor defect detection, and in particular relates to a semiconductor defect classification method using data enhancement and a lightweight neural network. Background Art
[0002] With the development of Industry 4.0, automation and digitalization have become key means to improve the efficiency of manufacturing processes and maximize profits. The semiconductor manufacturing industry also needs to actively adopt these technologies to use the knowledge and data accumulated during the production process to improve process efficiency and reduce costs. Wafer defect detection is extremely important in the semiconductor manufacturing process. By detecting and correcting defects at an early stage, the defect rate of the final product can be significantly reduced, production efficiency can be improved, and product quality can be improved. Wafer defect detection is usually completed using image algorithms. Among the various image sources that can be captured in the semiconductor manufacturing chain, defect density images provide relevant information about the defect concentration in the sample being tested. When the laser scans the surface of the wafer, defects are detected by the scattering of light. The inspection reveals the coordinates of the various defects found, but does not directly classify them. When the number of defects detected exceeds a predefined threshold, automatic optical inspection is performed using an optical microscope or a scanning electron microscope (SEM). However, relying on experts to manually classify defects is a time-consuming and resource-inefficient process. In addition, the quality of inspection is unstable due to visual fatigue, work pressure, or simply their subjective judgment.
[0003] At present, many experts and scholars have studied wafer defect detection methods. Gómez-Sirvent et al. [JLGómez-Sirvent, FLde la Rosa, R.Sánchez-Reolid, A.Fernández-Caballero and R.Morales, (2022), Optimal Feature Selection for Defect Classification in Semiconductor Wafers, IEEE Transactions on Semiconductor Manufacturing, (pp.324-331).] proposed the OPTICS method and support vector machine (SVM) to deal with the multi-label classification problem of wafer maps. Saqlain et al. [M.Saqlain, B.Jargalsaikhan, JYLee. (2019). Voting ensemble classifier for wafer map defect patterns identification in semiconductor manufacturing. IEEE Trans.Semicond.Manuf., (pp.171-182).] proposed a voting integration method based on density, geometry and Radon feature classification to identify defect patterns in classified wafer maps. The classification results of such methods depend largely on the effectiveness of the extracted features. However, in most cases, it is impossible to determine which wafer image features are the most representative and effective. To address this problem, the study of deep learning methods applied to wafer image defect pattern recognition has emerged. Convolutional neural network (CNN) is a popular deep learning method [OM Parkhi, A. Vedaldi, A. Zisserman, Deep face recognition, Proceedings of the BMVC, (pp. 6).], which can automatically extract meaningful features from raw input without the need for manual feature extraction.
[0004] Nakazawa et al. [T. Nakazawa, DV Kulkarni. (2019). Anomaly detection and segmentation for wafer defect patterns using deep convolutional encoder–decoder neural network architectures in semiconductor manufacturing. IEEE Trans. Semicond. Manuf., (pp. 250-256).] used synthetic wafer images to build a CNN model, applied it to the classification of actual wafer images, and used a convolutional autoencoder to detect invisible wafer mapping patterns. Jin et al. [Chenghao Jin, Hyun-Jin Kim, etc. (2020). Wafer map defect pattern classification based on convolutional neural network features and error-correcting output codes. J. Intell. Manuf., (pp. 1861-1875)] proposed a wafer map defect pattern classification method based on image combined with CNN and ECOC-SVM, and obtained good classification results. Finally, residual networks have been used to classify semiconductor defects in SEM images [López de la Rosa F., Gómez-Sirvent JL, Morales R., Sánchez-Reolid R., Fernández-Caballero A. (2022). A deep residual neural network for semiconductor defect classification in imbalanced scanning electron microscope datasets. Applied Soft Computing, (pp. 109743).].
[0005] However, current methods for real-time, fast wafer defect detection are very immature. However, the use of lightweight models and a higher degree of automation introduced by CNN and computer vision systems contribute to more efficient and smarter manufacturing. In addition, in constructing wafer defect datasets, there is often a problem of sample imbalance, resulting in low detection rates for defect categories with a small number of samples.
[0006] In summary, the following problems remain to be solved in the field of semiconductor wafer defect detection: a fast and lightweight model is needed to achieve high-speed and accurate wafer defect detection, and the problem of imbalance in wafer defect categories needs to be solved. Summary of the invention
[0007] The purpose of the present invention is to provide a semiconductor defect classification method using data enhancement and lightweight neural network, which can accurately and quickly detect and identify defects on semiconductor wafers and provide a reliable detection method for improving wafer manufacturing quality. The technical solution adopted is:
[0008] Step S0: Use a scanning electron microscope (SEM) to scan and image defective wafer samples, and professionals classify them into different categories according to different types of defects.
[0009] The dataset contains 6 categories in total, defects include 5 categories, the features of each category are shown in Figure 4(b), and the dataset contains defect-free image categories.
[0010] Step S1: wafer defects often have class imbalance problems, resulting in reduced accuracy in minority class defect detection and classification. Step S2: synthetic data is generated through affine transformation to expand the number of minority data.
[0011] The specific process of step S1 is as follows: first, the image is cropped to ensure that each sub-image contains only one type of defect. Then, defect segmentation is performed, and the image threshold is calculated using the OTSU algorithm. The image is binarized to obtain a segmentation template MASK, which is a binary image composed of 0 and 1. The template and the original image are multiplied pixel by pixel to obtain the segmented defect image D1.
[0012] Step S2: Data enhancement. Random scaling, rotation, translation and flipping are used to perform data enhancement on a small number of defect images D1. The defect image D1 and its corresponding MASK must perform the same enhancement steps.
[0013] Among them, scaling includes changing the size of D1. Enlarging the defect can be done by performing bilinear interpolation on the image and then cropping the image to the original size. Shrinking the defect can be done by performing Gaussian filtering on the image and then deleting the even rows and columns of the image and then filling it with 0 to the original size. In the rotation technique, the image is in and the rotation center can be manually defined. However, it usually coincides with the center of the image. The translation operation only includes moving the image through the image X and Y axes. Flipping includes flipping the pixels of the image horizontally or vertically.
[0014] Step S3: construct a mixed data set.
[0015] In order to generate an image that is as close as possible to the original image, a uniform background is added to the generated synthetic image.
[0016] The specific steps are:
[0017] The enhanced defect map (output composite image) in step S2 is randomly overlaid onto a portion of the defect-free image set to obtain an enhanced data set.
[0018] Aggregate the enhanced data set and the data set that has not been data enhanced in step S2 to obtain a mixed data set;
[0019] After obtaining the mixed dataset, it is divided into two more subsets. The first subset (90% of the mixed dataset) is used for training, while the other 10% is reserved for validating the CNN model.
[0020] Here, coverage refers to multiplying the MASK with the defect-free image pixel by pixel and then adding it to D1.
[0021] Step S4: construct and train a convolutional neural network.
[0022] The neural network of the present invention is as follows Figures 2-3 As shown in the figure, the neural network includes a backbone network and four feature enhancement modules, which are connected in head-to-tail order. Module 1 consists of 3 dense layers and a transition layer; module 2 consists of 4 dense layers and a transition layer; module 3 consists of 8 dense layers and a transition layer; module 4 consists of 6 dense layers and a transition layer. Finally, a classification head is used to achieve wafer defect classification.
[0023] Among them, the backbone network contains two paths, each of which can extract different features, which can improve the feature expression ability of the network without bringing too much computation. The input image first passes through the 1*1 convolution module to obtain the feature map and then passes through the 1*1 convolution module, 3*3 convolution module, 1*1 convolution module subpath and 2*2 maximum pooling layer subpath. Then, the feature maps output by the two paths are spliced together and output after passing through the 1*1 convolution module. The dense layer also contains two paths. Path 1 includes a 1*1 convolution module and a 3*3 convolution module, and path 2 is the input feature map. Similarly, the output feature maps of the two paths are spliced according to the third dimension and output. The transition layer realizes feature dimension compression. It includes a 1*1 convolution module and a 2*2 maximum pooling layer. Finally, the network classification head performs a 7*7 global pooling operation on the feature map and performs a softmax operation after passing through the fully connected layer to output the predicted category probability.
[0024] The Adam optimizer is used to train the neural network, with a learning rate of 0.001 and a batch size of 10. For fast convergence, the learning rate is halved in the following rounds: {50, 100, 130, 160, 190, 220, 250, 280}. The loss function uses the cross entropy loss function.
[0025] Step S5: Using the trained neural network for actual wafer defect recognition and classification.
[0026] Compared with the prior art, the advantages of the present invention are:
[0027] The present invention uses a lightweight neural network to more efficiently realize defect detection and identification of semiconductor wafers. The data enhancement strategy used can effectively improve the defect identification capability with less data volume and improve the overall detection quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a flowchart of the method of the present invention;
[0029] Figure 2 is the neural network structure of the present invention;
[0030] Figure 3 It is the neural network submodule structure in the present invention;
[0031] FIG4( a ) is a wafer defect image taken by a scanning electron microscope;
[0032] Figure 4(b) is the feature description of each category;
[0033] Figure 5 is the defect image after segmentation.
[0034] Figure 6 The image after data augmentation is applied to the defect image. DETAILED DESCRIPTION
[0035] The following is a more detailed description of a semiconductor defect classification method using data enhancement and a lightweight neural network according to the present invention in conjunction with a schematic diagram. Figure 1 As shown, there are shown preferred embodiments of the present invention, it should be understood that those skilled in the art can modify the present invention described herein, and still achieve the beneficial effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.
[0036] Step S0: Use a scanning electron microscope (SEM) to scan and image defective wafer samples, and professionals classify them into different categories according to different types of defects.
[0037] Among them, according to the common defects of semiconductor wafers, the data set is divided into 6 categories, and the defects include 5 categories. The data set example is shown in Figure 4(a), and the characteristics of each category are shown in Figure 4(b). The data set contains defect-free image categories. The size of each image is 448*448 pixels.
[0038] Step S1: wafer defects often have class imbalance problems, resulting in reduced accuracy in minority class defect detection and classification. Step S2: synthetic data is generated through affine transformation to expand the number of minority data.
[0039] First, the image acquired by a scanning electron microscope (SEM) in step S0 is cropped to ensure that each sub-image contains only one type of defect.
[0040] Then, defect segmentation is performed. The image threshold is calculated using the OTSU algorithm. The image is binarized to obtain a segmentation template MASK. The template is a binary image composed of 0 and 1. The template and the original image (sub-image) are multiplied pixel by pixel to obtain the segmented defect image D1, as shown in Figure 5 As shown, defect diagram D1 refers to diagram B, diagram C, diagram D, diagram E, and diagram F.
[0041] The OTSU algorithm threshold calculation method is as follows:
[0042] First, calculate the normalized histogram p of the input image i , and then calculate the gray mean m of the image G For the normalized grayscale histogram, the calculation method is as follows:
[0043] First, we need to count the number of pixels at each gray level from 0 to 255 in the image. Assuming the image size is M*N, the original histogram h(u) represents the number of times gray level u appears. Next, we convert the original histogram into a normalized histogram pi. This step is done by dividing the number of occurrences of each gray level by the total number of pixels in the image (i.e., M*N).
[0044]
[0045] For an image of size M*N, its grayscale mean m G The calculation method is as follows:
[0046]
[0047] Among them, x and y are the horizontal and vertical coordinates of the image, M and N are the number of pixels in the height direction and the width direction of the image respectively, and I(x,y) represents the grayscale value of the image at the position (x,y).
[0048] For k = 0, 1, 2, ..., L-1, L is the total number of gray levels, calculate the between-class variance σ 2 (k) The smallest k value is used as the segmentation threshold.
[0049] The between-class variance is defined as:
[0050] σ 2 (k) = P 1(k)(m 1 -m G ) 2 +P 2 (k)(m 2 -m G ) 2
[0051] Among them, p 1 (k) is the probability of a pixel being classified as background, p 2 (k) is the probability of a pixel being classified as the foreground, which can be seen from the normalized grayscale distribution histogram;
[0052] m 1 is the mean pixel value of the foreground, m 2 is the background pixel mean. i is the pixel value.
[0053] The calculation formula is as follows:
[0054]
[0055] Step S2: data enhancement.
[0056] Random scaling, rotation, translation and flipping are used to perform data enhancement on the segmented defect image D1. The probability of scaling, rotating, translating and flipping each sample is the same, which is 25%. Data enhancement operations are continuously performed on samples of the minority class until the number of data in each category is the same. In addition, the defect image D1 and its corresponding MASK must perform the same enhancement steps.
[0057] Among them, scaling includes changing the size of the defect image D1. To enlarge the defect, D1 can be bilinearly interpolated and then the image can be cropped to the original size. To reduce the defect, D1 can be Gaussian filtered and then the even rows and columns of the image can be deleted and then filled with 0 to the original size.
[0058] The bilinear interpolation method is as follows:
[0059] First, find the four nearest pixels of the pixel to be interpolated P, which are the lower left corner Q 11 , upper left corner Q 12 , Q in the upper right corner 22 , Q in the lower right corner 21 Then perform linear transformation on the x-axis and y-axis to calculate the pixel value of point P.
[0060] The formula is as follows:
[0061]
[0062] x is the horizontal coordinate of P, and y is the vertical coordinate of P. 2 It's Q 22 and Q 21 The horizontal coordinate, x1 It's Q 12 and Q 11 The horizontal axis, y 1 It's Q 11 and Q 21 The vertical coordinate, y 2 It's Q 12 and Q 22 The vertical coordinate of . f() represents the pixel value.
[0063] The Gaussian filter kernel is defined as follows,
[0064]
[0065] Where σ is the standard deviation, which determines the width of the Gaussian kernel, which is 2 in this example. The Gaussian kernel size is 5*5 pixels. x and y are the xy coordinates of the kernel from the center pixel. Convolving the image with the Gaussian kernel gives the Gaussian filtered result.
[0066] In the rotation technique, the image is rotated and the center of rotation can be manually defined. However, it usually coincides with the center of the image. The rotation angle is randomly selected between 0-90°.
[0067] The translation operation only involves moving the image along the X and Y axes. The maximum translation of the X and Y axes is plus or minus 50 pixels, and a random number is used to generate the translation pixel amount each time. Rotation and translation need to ensure that the defect remains in the image after the operation. Flipping involves flipping the pixels of the image horizontally or vertically. The defect image after data enhancement is shown in the figure below. Figure 6 shown.
[0068] Step S3: construct a mixed data set.
[0069] In order to generate an image that is as close as possible to the original image, a uniform background is added to the synthesized image output in step S2 (the defect image enhanced in step S2).
[0070] The specific method is to randomly overlay the defect image enhanced in step S2 onto a portion of the defect-free image set to obtain an enhanced data set.
[0071] The enhanced dataset and the dataset that has not been enhanced in step S2 are aggregated to obtain a mixed dataset.
[0072] After obtaining the mixed dataset, it is divided into two more subsets.
[0073] The first subset (90% of the mixed dataset) is used for training, while the other 10% is reserved for validating the CNN model.
[0074] Here, coverage refers to multiplying the MASK with the defect-free image pixel by pixel and then adding it to D1.
[0075] Step S4: construct and train a convolutional neural network.
[0076] The neural network of the present invention is as follows Figures 2-3 shown.
[0077] Among them, the neural network includes a backbone network and four feature enhancement modules, which are connected in head-to-tail order.
[0078] Module 1 consists of 3 dense layers and one transition layer; module 2 consists of 4 dense layers and one transition layer; module 3 consists of 8 dense layers and one transition layer; module 4 consists of 6 dense layers and one transition layer.
[0079] Finally, wafer defect classification is achieved through a classification head.
[0080] Among them, the backbone network contains two paths, which can extract different features respectively, and can improve the feature expression ability of the network without incurring too much computational effort.
[0081] The input image first passes through the 1*1 convolution module to obtain the feature map and then passes through the 1*1 convolution module, 3*3 convolution module, 1*1 convolution module subpath and 2*2 maximum pooling layer subpath. Then, the feature maps output by the two paths are concatenated together and passed through the 1*1 convolution module before output.
[0082] The dense layer also contains two paths. Path 1 includes a 1*1 convolution module and a 3*3 convolution module, and path 2 is the input feature map. Similarly, the output feature maps of the two paths are concatenated and output according to the third dimension.
[0083] The transition layer achieves feature dimension compression. It includes a 1*1 convolution module and a 2*2 maximum pooling layer. Finally, the network classification head performs a 7*7 global pooling operation on the feature map and performs a softmax operation after passing through the fully connected layer to output the predicted probability of each category.
[0084] The size of the feature map output after the backbone network is 112*112 pixels, the size of the feature map output after the first feature enhancement module is 56*56 pixels, the size of the feature map output after the second feature enhancement module is 28*28 pixels, the size of the feature map output after the third feature enhancement module is 14*14 pixels, and the size of the feature map output after the fourth feature enhancement module is 7*7 pixels. Finally, after passing through the classification head, the network outputs the probability value of each class, and the value range is 0-1.
[0085] The Adam optimizer was used to train the neural network. The initial learning rate was set to 0.001 and the batch size was set to 10. In order to converge quickly, the learning rate was halved in the following rounds: {50, 100, 130, 160, 190, 220, 250, 280}.
[0086] The loss function uses the cross entropy loss function. It is defined as follows:
[0087]
[0088] Among them, y n is an indicative function, indicating which category the element belongs to. Only when the output of the neural network is indeed in this category, y n is 1 otherwise it is 0; p n is the predicted probability, the predicted probability that the observed sample belongs to category n; C is the total number of categories, which is 6 in this embodiment.
[0089] Step S5: Use the trained neural network for actual wafer defect recognition and classification, so as to quickly and accurately identify the defect type of the wafer.
[0090] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A semiconductor defect classification method using data enhancement and lightweight neural network, characterized in that: The following steps are involved: Step S0: Scanning and imaging defective wafer samples using a scanning electron microscope, and classifying them according to the defect types: Among them, the dataset contains a total of 6 categories, defects include 5 categories, and the dataset contains defect-free image categories; Step S1, performing defect segmentation on the data sets of 5 categories to obtain defect maps D1, and counting the number of defect maps corresponding to each defect category; Step S2: perform data enhancement on a small number of defect images D1: Step S3: construct a mixed data set: Adding a uniform background to the synthesized image output in step S2 to obtain an enhanced dataset; Aggregate the enhanced data set and the data set that has not been data enhanced in step S2 to obtain a mixed data set; Step S4, constructing a convolutional neural network, and training the convolutional neural network based on the mixed data set; Step S5: Using the trained neural network for actual wafer defect recognition and classification.
2. The semiconductor defect classification method using data enhancement and lightweight neural network according to claim 1, characterized in that: The defect segmentation in step S1 specifically includes the following steps: First, the image obtained in step S0 is cropped to ensure that each sub-image contains only one type of defect; Then, defect segmentation is performed, and the image threshold is calculated using the OTSU algorithm. The image is binarized to obtain the segmentation template MASK. The template and the original image are multiplied pixel by pixel to obtain the segmented defect image D1.
3. The semiconductor defect classification method using data enhancement and lightweight neural network according to claim 2, characterized in that: Step S3 specifically includes the following steps: Randomly overlay the composite image output in step S2 onto a portion of the defect-free image set; After obtaining the mixed dataset, the mixed dataset is divided into two other subsets; the first subset is used for training, while the other subset is used to validate the CNN model; Here, coverage refers to multiplying the MASK with the defect-free image pixel by pixel and then adding it to the synthetic image.
4. The semiconductor defect classification method using data enhancement and lightweight neural network according to claim 1, characterized in that: Step S2 specifically includes the following steps: Scaling includes changing the size of the defect image D1; In the rotation technique, the center of rotation can be defined manually; Translation operations involve moving the image along its X and Y axes; Flipping involves flipping the pixels of an image horizontally or vertically.
5. The semiconductor defect classification method using data enhancement and lightweight neural network according to claim 1, characterized in that: In step S4, the neural network includes a backbone network and four feature enhancement modules, which are connected in head-to-tail order; Module 1 consists of three dense layers and a transition layer; Module 2 consists of 4 dense layers and a transition layer; Module three consists of 8 dense layers and one transition layer; Module 4 consists of 6 dense layers and a transition layer; Finally, wafer defect classification is achieved through a classification head; The Adam optimizer is used to train the neural network, and the cross entropy loss function is used as the loss function.