A wafer defect detection method based on deep learning

Abstract

This invention belongs to the field of wafer inspection technology and proposes a deep learning-based wafer defect detection method. Based on the existing YOLOv7 target detection algorithm, a teacher network and a student network are obtained, which together form a dual-channel teacher-student network. The teacher network is trained on the input image to obtain a ubiquitous defect region. During training, the ubiquitous defect information obtained by the teacher network is transferred to the student network through knowledge distillation. The student network is then trained on the input image to obtain segmented defects within the ubiquitous defect region. In the final testing phase, only the trained student network is retained, and the input image is used to determine whether the wafer is normal or not, as well as the wafer's defect category. This invention ensures real-time defect segmentation, ensures learning effectiveness, and improves segmented defect detection performance.

Description

Technical Field

[0001] The present invention relates to the field of wafer detection technology, and in particular to a wafer defect detection method based on deep learning. Background Art

[0002] A wafer is a thin film used in the manufacture of integrated circuits. Common materials include silicon, germanium, gallium arsenide, zinc oxide, indium phosphide, gallium oxide, silicon carbide, and aluminum nitride. For example, high-purity polycrystalline silicon is dissolved, doped with silicon crystal seeds, and then slowly pulled out to form cylindrical single crystal silicon. Silicon ingots are then ground, polished, and sliced to form silicon wafers, also known as wafers. During wafer production, chemical vapor deposition, optical development, and chemical mechanical polishing can cause defects on the wafer surface during a series of processes, including single crystal pulling, slicing, grinding, polishing, layer addition, photolithography, doping, heat treatment, and scribing. Among the types of wafer defects, surface redundancy, crystal defects, and mechanical damage (scratch patterns) are more common. Crystal defects are often caused by uneven heating during crystal growth. Compared with other wafer surface defects, defects with their own characteristics have a greater impact on the wafer manufacturing process; mechanical damage generally occurs during the polishing, slicing and other steps of the wafer manufacturing process, caused by chemical mechanical grinding. It is a more serious wafer surface defect that can have a very serious impact on integrated circuit chips.

[0003] With the rapid advancement of integrated circuit manufacturing technology in my country, wafer integration continues to increase, and process stability and reliability are improving. However, defective wafers are still inevitable, placing higher demands on wafer inspection technology. High-precision detection of wafer surface defects has become a critical step in the manufacturing process. To prevent defective wafers from entering the next packaging process, optical inspection equipment is needed to identify, classify, and mark wafer surface defects, analyze the causes of the defects, and improve the manufacturing process. Early wafer optical inspection methods, such as CN 111982931 A, generally place the wafer in a bright environment and observe the wafer surface for defects such as machine damage, dust, or dirt through manual visual inspection or random sampling. However, as the characteristic size of the grains continues to decrease, the size of the defects has also shrunk accordingly. These simple early inspection methods are clearly no longer able to meet the industry's demand for high precision and high efficiency. If the wafer defect detection step fails, wafers with different defect types may enter the next packaging process, potentially resulting in product scrap. Therefore, to achieve rapid and accurate detection of multiple defects on the wafer surface, a detection method that combines real-time and high accuracy is needed. Summary of the Invention

[0004] The purpose of this invention is to propose a wafer defect detection method based on deep learning, introduce the target detection algorithm YOLOv7, and improve the algorithm according to the special needs of wafer defect detection, so as to realize real-time and accurate detection and segmentation of different types of wafer defects.

[0005] The technical solution of the present invention is:

[0006] A deep learning-based wafer defect detection method uses an improved teacher network and a student network based on the existing YOLOv7 object detection algorithm. The two networks form a dual-channel teacher-student network. The teacher network is trained on an input image to obtain ubiquitous defect regions, with the probability of a ubiquitous defect region being PA, and the probability of the image being normal being PN = 1-PA. During training, the ubiquitous defect information obtained by the teacher network is transferred to the student network through knowledge distillation. The student network is trained on the input image to obtain the subdivided defects within the ubiquitous defect region, namely the probability of a specific defect Pj, where j = 1, ..., M, where M represents the number of ubiquitous defect regions. In the final testing phase, only the trained student network is retained, and the input image is used to determine whether the wafer is normal and the wafer defect category.

[0007] The teacher network includes the backbone network, attention enhancement mechanism, multi-task learning mechanism and fusion module of the existing YOLOv7 target detection algorithm; the student network includes the backbone network, attention enhancement mechanism, multi-task learning mechanism and fusion module of the improved YOLOv7 target detection algorithm; the backbone network of the improved YOLOv7 target detection algorithm is streamlined based on the backbone network of the existing YOLOv7 target detection algorithm, and only two sets of CSP bottleneck layer modules are used; the multi-task learning mechanism includes the detection of ubiquitous defects and the detection of subdivided defects; when training the teacher network, only the detection task of ubiquitous defects is performed, and when training the student network, only the detection task of subdivided defects is performed; the fusion module is used to splice the output results of the multi-task learning mechanism and output defect probabilities and defect images.

[0008] The attention mechanism draws on the selective attention characteristics of human vision: humans quickly scan the entire image to identify target areas that require focus, then obtain detailed information about the target from these areas while suppressing useless information. Depending on the application scenario, attention can generally be categorized into temporal attention, spatial attention, and channel attention. Temporal attention assigns different weights to sequence features and automatically increases the model's focus on important features through probabilistic allocation, thereby improving the accuracy of sequence data processing without increasing computational and storage costs. Spatial attention transforms the spatial information in the original image into another space while retaining key information, thereby identifying areas of interest within the image and increasing the network's attention weight on these areas. Channel attention mines effective features from the feature dimension level and suppresses features irrelevant to the task, thereby improving network performance. The attention enhancement mechanism described in this invention includes spatial attention and channel attention. The spatial attention enhancement mechanism enhances the sensitivity of the algorithm to small defect areas, thereby improving its sensitivity to small defects. The channel attention enhancement mechanism then suppresses non-defect features, thereby highlighting defect features.

[0009] The feature map output by the backbone network is F, and the spatial attention weight As(F) and channel attention weight Ac(F) are as follows:

[0010]

[0011]

[0012] Among them, MLP() represents a multilayer perceptron with three layers, and the number of neurons is m, m / 4, and m respectively, where m represents the dimension of F channel after pooling, and Conv() represents a convolution operation with a kernel size of 7*7; represents the average value of the feature map F under the spatial attention weight; Represents the maximum value of the feature map F under the spatial attention weight; Represents the average value of the feature map F under the channel attention weight; Represents the maximum value of the feature map F under the channel attention weight;

[0013] The attention enhancement mechanism is expressed as follows:

[0014]

[0015]

[0016] The fusion module fuses and splices the probabilities of ubiquitous defects and specific defects. During the testing phase, the fusion module fuses and splices the probability PN that the image is normal and the probability Pj of a specific defect, and normalizes them through the Softmax activation function to obtain the probability that the wafer image belongs to a normal sample or a specific defect.

[0017] When the teacher network outputs the defect area results and defect area probabilities, considering that the defects on the wafer surface only account for a small part of the collected images, there may be a sample imbalance problem in target detection, that is, the number of positive samples and negative samples in the same image is unbalanced, the Focal loss function L is introduced F To alleviate the imbalance of training samples, the cross entropy loss function L is used. T Achieve ubiquitous flaw detection of ubiquitous flaws in teacher networks;

[0018] L F =-α(1-p i ) γ log(p i ) (5)

[0019]

[0020] Among them, the hyperparameters ɑ and γ are used to alleviate the imbalance problem of positive and negative samples and difficult and easy samples respectively; y i represents the training sample label, p i represents the output probability of ubiquitous defect detection, and K represents the number of training samples;

[0021] The student network relies on distilling information from the teacher network to mine multi-scale features, thereby reducing the spatiotemporal complexity of the student network. When the student network outputs the probability of segmentation defects and specific defects, the Softmax loss function Ls and the center loss function L are introduced. C ;

[0022]

[0023] Among them, M represents the number of subdivided defect types, Represents the one-hot encoding of the true label, Z i represents the probability that the sample belongs to the i-th defect;

[0024] Center loss function L C Constraints are applied to increase the inter-class clustering of classified samples and reduce the intra-class distance;

[0025]

[0026] Among them, x i represents the sample code, c yi Represents x iThe center of the corresponding specific defect category;

[0027] The dual-channel teacher-student network obtains a weighted combination loss function in a weighted manner, and the formula is as follows:

[0028] L=v1L T +v2L S +v3L F +v4L C (9)

[0029] Among them, v1, v2, v3, and v4 are the set weights.

[0030] Due to the excellent representation learning ability of convolutional neural networks, results far exceed those of traditional methods. Typical deep learning-based target detection algorithms include RCNN, Fast-RCNN, Faster-RCNN, YOLO, etc. However, the computational complexity of the above methods is large and it is difficult to meet the real-time operation requirements of the wafer surface defect detection system. Based on the requirements of balancing detection accuracy and speed, the present invention introduces the target detection network YOLOv7. The existing YOLOv7 target detection algorithm network consists of three parts: input, backbone network and detection head network, where the input part is used to initially input the image, the backbone network is used to extract features, and the detection head network is used for prediction. The operating steps of the existing YOLOv7 target detection algorithm are: first pre-process the input image, input an RGB image with an image size of 640*640*3 (height*width*number of channels), input it into the backbone network, and according to the three-layer output in the backbone network, the detection head network continues to output three layers of feature maps of different sizes, and after convolution reparameterization, predicts the image detection and outputs the final result.

[0031] Based on the existing YOLOv7 object detection algorithm, this paper proposes a dual-channel teacher-student architecture for defect detection. A deeper teacher network is pre-trained on a large-scale dataset to guide the student network to more effectively learn defect characteristics. The student network is lightweight by limiting its depth and adjusting the convolution kernel, ensuring generalization performance. During the testing phase, only the student network is used to segment wafer-specific defects, ensuring the algorithm's real-time performance.

[0032] Since the amount of data on ubiquitous defects is large, the detection of ubiquitous defects is implemented in a deeper teacher network. The amount of data on specific types of defects is small, so the segmentation of specific defects is implemented in a lightweight student network. The teacher network distills defect-related information into the student network to achieve knowledge transfer; then, the ubiquitous defects output by the teacher network and the segmented defects output by the student network are combined through a fusion module to finally output a wafer defect image.

[0033] There is complementarity between the detection of ubiquitous defects and the segmentation of specific defects. A multi-task learning mechanism is used to promote the complementarity between the general features and segmentation features of defects.

[0034] Beneficial effects of the present invention:

[0035] (1) Considering the small amount of wafer defect image data collected and the large number of deep neural network parameters, as well as the high real-time requirements of the detection system, this paper draws on the multi-task learning concept and proposes a dual-channel teacher-student architecture based on YOLOv7 to achieve defect detection: the deeper teacher network is pre-trained on a large-scale dataset to guide the student network to learn defect features more effectively. The student network is lightweight by limiting the depth and adjusting the convolution kernel, ensuring the generalization performance of the network. During the testing phase, only the student network is used to implement wafer-specific defect segmentation, ensuring the real-time performance of the algorithm.

[0036] (2) Considering the problem that there are many types of wafer defects and they are difficult to segment, and considering the dual-channel teacher-student architecture adopted by the present invention, the present invention designs different tasks based on the characteristics of different networks: ubiquitous defect detection and specific defect segmentation. The amount of ubiquitous defect data is large, so ubiquitous defect detection is achieved in a deeper teacher network; the amount of data for each specific type of defect is small, so specific defect segmentation is achieved in a lightweight student network. At the same time, a spatial attention mechanism is introduced into the network to improve the representation learning effect.

[0037] (3) Considering that the defect area in the captured wafer image accounts for a relatively small proportion of the entire image, sample imbalance is a common problem in defect detection. The present invention introduces the Focal loss function to alleviate sample imbalance. At the same time, the center loss function is introduced as a constraint to increase the inter-class clustering of classified samples, reduce the intra-class distance, and improve the performance of subdivided defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of a wafer defect detection method based on deep learning proposed in the present invention;

[0039] Figure 2 It is a structural flow chart of the teacher network proposed in the present invention;

[0040] Figure 3 It is a structural flow chart of the student network proposed in the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings, but the protection scope of the present invention is not limited thereto.

[0042] Figure 1A flowchart of a wafer defect detection method based on deep learning is given:

[0043] This paper proposes a deep learning-based wafer defect detection method. Deep learning technology is first introduced to automatically detect wafer defects by improving the YOLOv7 target detection algorithm. To address the limited amount of wafer defect image data, an attention-enhanced teacher-student architecture is designed based on the existing YOLOv7 network. The teacher network is used to detect and locate ubiquitous defects in the image, while the student network is used to segment specific defects. The teacher network distills defect-related information into the student network. Furthermore, a multi-task learning mechanism is introduced to simultaneously train the teacher network for detecting ubiquitous defects and the student network for detecting segmented defects. A focal loss function and center constraint are introduced to improve classification performance.

[0044] The specific operating steps of the present invention are:

[0045] 1) Introducing the YOLOv7 target detection algorithm and proposing a dual-channel teacher-student architecture based on it

[0046] ①Introducing YOLOv7 target detection algorithm

[0047] The existing YOLOv7 target detection algorithm network consists of three parts: input, backbone network and detection head network. The input part is used to input the image at the beginning, the backbone network is used to extract features, and the detection head network is used for prediction.

[0048] First, preprocess the input image. The input image size is 640*640*3 (height*width*number of channels) RGB image, which is input into the backbone network. According to the three-layer output of the backbone network, the detection head network continues to output three layers of feature maps of different sizes. After convolution reparameterization, the image detection is predicted and the final result is output.

[0049] ② Propose a dual-channel teacher-student architecture

[0050] like Figure 2 The figure shows the structure of the proposed teacher network. Based on the existing YOLOv7 backbone network, the teacher network of the present invention proposes an attention enhancement mechanism based on the characteristics of wafer defect detection. Since the area occupied by wafer defects accounts for a small proportion of the overall collected image, there will be a problem of insufficient feature extraction in small areas. In response to the above problems, the present invention introduces an attention enhancement mechanism. First, the spatial attention enhancement mechanism is used to enhance the sensitivity of the algorithm to small defect areas. Then, the channel attention enhancement mechanism is used to suppress non-defect features, thereby highlighting defect features. Assuming that the output feature map is F, the spatial attention weight As(F) and the channel attention weight Ac(F) are respectively as follows:

[0051]

[0052]

[0053] Where MLP() represents a shared multilayer perceptron (three layers with m, m / 4, and m neurons, respectively, where m represents the dimension of the F channel after pooling), and Conv() represents a convolution operation with a kernel size of 7*7.

[0054]

[0055]

[0056] like Figure 3 Figure 2 shows the structure of the proposed student network. Unlike the teacher network, the student network undergoes the following lightweight processing: First, the backbone is streamlined, retaining only two sets of bottleneck layer CSP modules. The student network primarily relies on information distilled from the teacher network to mine multi-scale features, thereby reducing the student network's spatiotemporal complexity. The rest of the student network, including attention enhancement, multi-task design, result fusion, and output, remains consistent with the teacher network.

[0057] 2) Simultaneously train to detect general defects and subdivided defects and feed them into the teacher network and student network respectively

[0058] ① Introducing a multi-task learning mechanism to simultaneously train a teacher network for detecting ubiquitous defects and a student network for detecting specific defects;

[0059] This paper proposes a multi-task learning mechanism for wafer defect detection: wafer defect detection tasks are generally divided into the detection of general defects and the segmentation of specific defects. These two tasks are complementary, so a multi-task learning approach is used to promote the complementarity between the general defect features and the segmented defect features. The specific ideas are as follows:

[0060] First, the probability of the defective suspicious region ROI output by the ubiquitous defect detection is calculated, and its probability is recorded as PA. The probability that the image is normal is PN = 1-PA. Then, the probability that each suspicious region ROI of the subdivided defect belongs to a specific defect is calculated, recorded as Pj (j = 1,...,M), where M is the number of suspicious region ROIs. Finally, PN is fused and spliced with all Pj and normalized using the Softmax activation function to obtain the probability that the wafer image belongs to a normal sample or a specific defect.

[0061] ② Detect ubiquitous defects and specific defects in different networks

[0062] During the training phase, captured wafer images undergo data augmentation before being fed into a dual-channel teacher-student network for training. Due to the large volume of data on ubiquitous defects, the deeper teacher network is chosen to detect these defects, guiding the student network to more effectively learn defect characteristics. The student network is lightweight by limiting its depth and adjusting the convolution kernel. Since the volume of data for each specific defect type is small, the lightweight student network is chosen to segment specific defects. The teacher network then distills defect-related information into the student network, enabling knowledge transfer. During the testing phase, the teacher network channel is removed, retaining only the student network channel. The wafer image is then fed directly into the student network to segment specific defects, ensuring the algorithm's real-time performance.

[0063] 3) Introduce the Focal loss function to alleviate sample imbalance and introduce the center loss function as a constraint

[0064] ①Introducing Focal loss function

[0065] This paper adopts a multi-task learning strategy to train the network and proposes a weighted combination loss function, which includes the following parts:

[0066] For the teacher network, it is necessary to detect ubiquitous defects, so the cross entropy loss function L is used. T , calculated as follows:

[0067]

[0068] Where yi represents the training sample label, pi represents the ubiquitous defect detection output probability, and K represents the number of training samples.

[0069] For the student network, it is necessary to achieve the segmentation of specific defects, so the Softmax loss function Ls is used, which is calculated as follows:

[0070]

[0071] Where M represents the number of subdivided defect types (M=10), Represents the one-hot encoding of the true label, and Zi represents the probability that the sample belongs to the i-th defect.

[0072] Considering the sample imbalance problem in target detection, that is, the imbalance in the number of positive and negative samples in the same image, the present invention introduces the Focal loss function LF in the teacher network to alleviate it, which is calculated as follows:

[0073] L F =-α(1-p i ) γ log(p i ) (7)

[0074] Among them, the hyperparameters ɑ and γ are used to alleviate the imbalance problem of positive and negative samples and difficult and easy samples, respectively.

[0075] ②Introducing center loss as a constraint

[0076] In order to further enhance the discriminability of features learned by the convolutional neural network, the present invention introduces the center loss function LC into the student network for constraint, increases the inter-class clustering of classification samples, and reduces the intra-class distance. The calculation is as follows:

[0077]

[0078] Among them, x i represents the sample code, c yi Represents x i The center of the corresponding category.

[0079] Finally, the weighted combination loss function used in training the dual-channel teacher-student network of the present invention is obtained by weighted method, and the formula is as follows:

[0080] L=v1L T +v2L S +v3L F +v4L C (9)

[0081] Among them, v1, v2, v3, and v4 are the set weights.

[0082] The above is a brief description of the present invention and is not limited to the above working scope. As long as the ideas and working methods of the present invention are simply modified and applied to other equipment, or improvements and embellishments are made without changing the main conceptual principles of the present invention, they are all within the scope of protection of the present invention.

[0083] The defect detection algorithm proposed in this paper can not only distinguish between normal and defective images, but also identify specific defect types. Therefore, this paper uses two metrics to evaluate the algorithm's performance: AUC (Area under the ROC curve) and mAP (mean average precision). AUC reflects the algorithm's ability to distinguish between normal wafer images and defective images, while mAP reflects the algorithm's ability to identify different types of defects.

[0084] The present invention tests the performance difference between the teacher network and the student network on a public dataset, and the results are shown in Table 1.

[0085] Table 1. Comparison of teacher-student network performance

[0086]

[0087] As can be seen from Table 1, although the student network results in a slight decrease in defect detection performance, it still meets the requirements for wafer defect detection. Furthermore, the student network's recognition time is approximately half that of the teacher network, making it feasible to run on embedded devices.

[0088] Finally, to verify the effectiveness of our algorithm, we introduced the mainstream Faster-RCNN and YOLOv7 for general object detection algorithms. The comparison results are shown in Table 2.

[0089] Table 2. Comparison of different defect detection algorithms

[0090]

[0091]

[0092] As shown in the table above, the object detection algorithms YOLOv7 and Faster-RCNN achieve good defect detection, outperforming the student network proposed in this paper. However, their time overhead is relatively high. The teacher network proposed in this paper achieves optimal wafer defect detection performance, while the student network provides a solution for detecting wafer defects with high accuracy on embedded devices.

Claims

1. A wafer defect detection method based on deep learning, characterized in that: Based on the existing YOLOv7 object detection algorithm, a teacher network and a student network are improved to form a dual-channel teacher-student network. The teacher network is trained on the input image to obtain ubiquitous defect regions, with the probability of a ubiquitous defect region being PA and the probability of the image being normal being PN = 1-PA. During the training process, the ubiquitous defect information obtained by the teacher network is transferred to the student network through knowledge distillation. The input image is trained on the student network to obtain the subdivided defects within the ubiquitous defect region, that is, the probability of specific defects Pj, j = 1,...,M, where M represents the number of ubiquitous defect regions. In the final testing phase, only the trained student network is retained, and the input image is used to determine whether the wafer is normal or not and the defect category of the wafer; The teacher network includes the backbone network of the existing YOLOv7 target detection algorithm, the attention enhancement mechanism, the multi-task learning mechanism, and the fusion module; the student network includes the backbone network of the improved YOLOv7 target detection algorithm, the attention enhancement mechanism, the multi-task learning mechanism, and the fusion module; The backbone network of the improved YOLOv7 object detection algorithm is a streamlined version of the existing YOLOv7 object detection algorithm, using only two sets of CSP bottleneck layer modules. The multi-task learning mechanism includes ubiquitous defect detection and segmented defect detection. When training the teacher network, only the ubiquitous defect detection task is performed, while when training the student network, only the segmented defect detection task is performed. The fusion module is used to splice the output results of the multi-task learning mechanism and output defect probabilities and defect images.

2. The wafer defect detection method based on deep learning according to claim 1, characterized in that: The attention enhancement mechanism includes spatial attention and channel attention; the feature map output by the backbone network is F, and the spatial attention weight As(F) and channel attention weight Ac(F) are as follows: Among them, MLP() represents a multi-layer perceptron, and Conv() represents a convolution operation with a kernel size of 7*7; represents the average value of the feature map F under the spatial attention weight; Represents the maximum value of the feature map F under the spatial attention weight; Represents the average value of the feature map F under the channel attention weight; Represents the maximum value of the feature map F under the channel attention weight; The attention enhancement mechanism is expressed as follows:

3. The wafer defect detection method based on deep learning according to claim 1 or 2, characterized in that: The fusion module fuses and splices the probabilities of ubiquitous defects and specific defects. During the testing phase, the fusion module fuses and splices the probability PN that the image is normal and the probability Pj of a specific defect, and normalizes them through the Softmax activation function to obtain the probability that the wafer image belongs to a normal sample or a specific defect.

4. The wafer defect detection method based on deep learning according to claim 1 or 2, characterized in that: The teacher network introduces the Focal loss function L when outputting the defect area results and defect area probabilities. F To alleviate the imbalance of training samples, the cross entropy loss function L is used. T Enables detection of ubiquitous flaws in teacher networks; L F =-α(1-p i ) γ log(p i ) (5) Among them, the hyperparameters ɑ and γ are used to alleviate the imbalance problem of positive and negative samples and difficult and easy samples respectively; y i represents the training sample label, p i represents the output probability of ubiquitous defect detection, and K represents the number of training samples; The student network relies on distilling information from the teacher network to mine multi-scale features, thereby reducing the spatiotemporal complexity of the student network. When the student network outputs the probability of segmentation defects and specific defects, the Softmax loss function Ls and the center loss function L are introduced. C ; Among them, M represents the number of subdivided defect types, Represents the one-hot encoding of the true label, Z i represents the probability that the sample belongs to the i-th defect; Center loss function L C Constraints are applied to increase the inter-class clustering of classified samples and reduce the intra-class distance; Among them, x i represents the sample code, c yi Represents x i The center of the corresponding specific defect category; The dual-channel teacher-student network obtains a weighted combination loss function in a weighted manner, and the formula is as follows: L=v1L T +v2L S +v3L F +v4L C (9) Among them, v1, v2, v3, and v4 are the set weights.

Images