Chip surface defect detection method

Abstract

The invention provides a chip surface defect detection method which comprises the following steps: acquiring image data of a chip surface, performing data processing on an acquired image, and performing image reconstruction by using a super-resolution reconstruction algorithm based on the processed acquired data to obtain an image data set; the method comprises the following steps: introducing a ResNet50 network, K-means clustering, a multi-scale feature pyramid, an improved EMSConv module and improved non-maximum suppression into a Faster R-CNN algorithm to obtain an improved network structure; obtaining a candidate region of the image data set by using the improved network structure; and processing the candidate region by utilizing a classification network, a confidence coefficient threshold value and improved non-maximum suppression to obtain the position, the category and the confidence coefficient of the chip surface defect. According to the method, the image quality and resolution can be improved, detail textures of tiny targets are increased, and rapid and accurate detection of chip surface defects is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of chip defect detection, and in particular to a chip surface defect detection method. Background Art

[0002] In the semiconductor manufacturing industry, the quality of chips is directly related to the performance and reliability of electronic products. With the advancement of semiconductor process technology, the defects on the chip surface are getting smaller and smaller, which puts higher requirements on detection technology. Traditional chip surface defect detection methods, such as manual inspection or simple machine vision systems, can no longer meet the high-precision and high-efficiency production requirements.

[0003] In addition, there are still many problems in the study of defects on the chip surface: 1) Differences in defect size: In the detection of chip surface defects, due to the wide variety of defects, their sizes also show significant differences. Compared with larger defects, their detection is relatively easy to achieve. However, current methods face greater challenges in detecting tiny defects with large size differences. These tiny defects are often difficult to accurately identify by existing detection technologies due to their small size, which increases the difficulty of detection. 2) Difficulties in detecting tiny defects: In semiconductor manufacturing, the detection of tiny defects is particularly important. Tiny defects have a small area and blurred details, and are often prone to missed detection or false detection during detection. 3) Similarity between defects and background: Some defects on the chip surface are similar to the texture of the background, which can easily cause missed detection or false detection. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a chip surface defect detection method that can improve image quality and resolution, increase the detail texture of tiny targets, achieve rapid and accurate detection of chip surface defects, improve detection efficiency, and ensure product quality.

[0005] To achieve the above object, the present invention provides the following solution: a chip surface defect detection method, comprising the following steps:

[0006] Collect image data on the chip surface, process the image data, and then reconstruct the image data using the SRGAN super-resolution reconstruction algorithm to obtain an image data set;

[0007] The ResNet50 network, K-means clustering, multi-scale feature pyramid, improved EMSConv module and improved non-maximum suppression are introduced into the Faster R-CNN algorithm to obtain the improved Faster R-CNN network structure.

[0008] Using the improved Faster R-CNN network structure, feature extraction, defect candidate and high-quality suppression are performed on the image data set to obtain a candidate region;

[0009] The candidate area is processed by using the classification network, confidence threshold and improved non-maximum suppression of Faster R-CNN to obtain the location, category and confidence of the chip surface defect.

[0010] Optionally, the K-means clustering processing method includes:

[0011] S1. Based on the image dataset and the Faster R-CNN algorithm, determine the number of clusters and the number of anchor box categories, and randomly select data points as the initial centroids of the clusters;

[0012] S2, calculating the distance between the size of each anchor box and the initial centroid, and assigning the anchor box to the cluster corresponding to the nearest initial centroid according to the calculated distance;

[0013] S3, calculating the average value of all anchor box sizes in the cluster, and taking the average value as the new centroid;

[0014] S4. Repeat steps S2-S3 until the change in the centroid is less than a preset threshold or reaches a preset number of iterations.

[0015] Optionally, the improved EMSConv module allocates channels according to the number of convolution kernels by utilizing the dynamically calculated number of channels so that each group of convolution operations increases the feature extraction capability, and performs multi-scale convolution calculations on the entire input tensor to obtain multi-scale features.

[0016] Optionally, the multi-scale feature pyramid processing step includes:

[0017] Using a feature pyramid network, extracting features from different levels of the image data set, and connecting the different levels horizontally to generate feature maps of multi-scale information, thus completing the construction of a multi-scale feature pyramid;

[0018] Utilizing the improved EMSConv module and convolution kernels of different sizes, the detail information in each feature map is extracted and enhanced, the sensitivity to small targets is increased, and detail processing is completed;

[0019] The region proposal network and detection network in the Faster R-CNN algorithm are used to generate candidate boxes of the feature map after detail processing, and the target is classified and regressed.

[0020] Optionally, the improved processing steps of the non-maximum suppression include:

[0021] The Rank&Sort function is introduced in non-maximum suppression to optimize the relative ranking between the candidate boxes, and the ranking loss function is used to optimize the score ranking of the candidate boxes to complete the candidate region processing; the formula of the ranking loss function is:

[0022] Ranking Loss = ∑ (i,j)∈Pairs max(0,1-(score i -score j ))

[0023] Among them, Ranking Loss is the pairwise sorting loss; Pairs is the overlapping detection box pair with IoU, and score i and score j are the scores of candidate boxes i and j respectively.

[0024] Optionally, the candidate region is processed using a classification network, a confidence threshold and the improved non-maximum suppression to obtain the location, category and confidence of the chip surface defect, including:

[0025] Use the region proposal network to generate candidate regions, and use non-maximum suppression to screen out high-quality proposal regions;

[0026] Use the classification network to determine whether the candidate area is a defect. If so, use bounding box regression to locate the defect position.

[0027] The confidence threshold is used to filter the low-probability detection results in the defect position, and the improved non-maximum suppression is used to remove the overlapping detection frames, and the position, category and confidence of the chip surface defect are output.

[0028] The present invention discloses the following technical effects by providing a chip surface defect detection method:

[0029] 1. Improve image resolution: Through the super-resolution reconstruction algorithm, the present invention can significantly enhance the details of tiny defects in the image, making the defects that were originally difficult to identify in the low-resolution image become clearly visible, thereby improving the image resolution and laying the foundation for subsequent defect detection.

[0030] 2. Enhanced defect recognition capability: The improved Faster R-CNN enhances the model’s ability to learn and extract defect features by introducing the ResNet50 network, improving anchor box clustering, introducing and improving feature pyramid networks, and improving non-maximum suppression, enabling the algorithm to more accurately identify and classify different types of defects.

[0031] 3. Reduce missed detections and false detections: Combined with super-resolution reconstruction and improved Faster R-CNN, the present invention can effectively reduce missed detections and false detections caused by defects that are too small or details are unclear, thereby improving the reliability of detection.

[0032] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0034] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a chip surface defect detection algorithm flow chart of an improved Faster R-CNN provided in an embodiment of the present invention;

[0036] Figure 3 A schematic diagram for comparing super-resolution reconstruction algorithm results provided by an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of the detection results provided in an embodiment of the present invention; wherein, (a) is a diagram showing the detection results of contamination defects; (b) is a diagram showing the detection results of line missing defects; (c) is a diagram showing the detection results of solid leakage defects; (d) is a diagram showing the detection results of grain tilt defects; and (e) is a diagram showing the detection results of skew line defects. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0040] like Figure 1-2 As shown, the present invention provides a chip surface defect detection method, comprising the following steps:

[0041] 1. Collect image data on the chip surface, process the collected images (rotate, zoom in, etc.), and then reconstruct the image based on the processed collected data using the SRGAN super-resolution reconstruction algorithm to enhance the detail information in the image and improve the visibility of defects to obtain an image data set.

[0042] 2. If Figure 3 As shown in the figure, the ResNet50 network, K-means clustering, multi-scale feature pyramid, improved EMSConv module and improved non-maximum suppression are introduced into the Faster R-CNN algorithm to obtain the improved Faster R-CNN network structure.

[0043] 2.1 Introducing the ResNet50 network

[0044] Since ResNet50 has a deeper network structure and more parameters, it has a higher generalization ability when processing various images, can extract richer and more abstract features, and help improve the network's detection performance for small targets. The introduction of ResNet50 enables the network to better learn the features and positions of small targets, and can improve the FasterR-CNN algorithm's detection of small target defects on chip surfaces by improving feature extraction performance, enhancing the generalization ability of the model, and increasing detection speed.

[0045] 2.2 Improved anchor box clustering

[0046] K-means clustering is used to cluster the target bounding boxes in the training set, and a set of anchor box sizes and proportions that are more consistent with the target distribution in the data set are found, thereby improving the match between the anchor box and the real target and further improving the detection performance of FasterR-CNN. The steps of using K-means clustering are as follows:

[0047] S1. Initialization: Based on the image dataset and the Faster R-CNN algorithm, determine the number of clusters K, where the K value also represents the number of anchor box categories, and randomly select K data points as the initial centroid of the cluster;

[0048] S2, allocation: calculate the distance between each anchor box size and the initial centroid, and allocate the anchor box to the cluster corresponding to the nearest initial centroid according to the calculated distance; this step will create K clusters, each cluster contains some anchor box sizes;

[0049] S3, Update: Calculate the average size of all anchor boxes in the cluster and get the new centroid to ensure that the sizes of anchor boxes in the cluster are as similar as possible;

[0050] S4, repeat steps S2-S3 until the change of the centroid is less than a preset threshold or reaches a preset number of iterations. Each iteration will make the centroid closer to the true center of the anchor box size within its cluster, thereby improving the quality of clustering.

[0051] 2.3 Introduction and Improvement of Feature Pyramid Network

[0052] By introducing the Feature Pyramid Network (FPN) and the improved EMSConv module based on Faster R-CNN, the effect of small target detection is improved. The improved EMSConv module uses a dynamically calculated number of channels and flexibly allocates channels according to the number of convolution kernels, so that each group of convolution operations can obtain more balanced and refined feature extraction capabilities. At the same time, multi-scale convolution is directly performed on the entire input tensor, avoiding channel segmentation and splicing steps, thereby simplifying the calculation process, which can not only reduce calculation and memory overhead, but also improve calculation efficiency and real-time response capabilities.

[0053] The steps of fusing FPN and improving EMSConv are as follows:

[0054] 1) In the feature extraction stage, FPN is used to construct a multi-scale feature pyramid. By extracting features from different levels of the original image and making horizontal connections between these levels, FPN can generate a series of high-resolution feature maps that contain rich multi-scale information and can provide target features at different scales.

[0055] 2) The feature maps generated by FPN are input into the improved EMSConv module. The improved EMSConv module applies convolution kernels of different sizes to each feature map, thereby further extracting and enhancing the detail information in these feature maps, increasing the sensitivity to small objects, and enabling the features of small objects to be better represented.

[0056] 3) The feature map processed by the improved EMSConv module is re-input into the detection network of Faster R-CNN. The region proposal network and detection network in the Faster R-CNN network will use the enhanced feature map to generate candidate boxes and classify and regress the target.

[0057] Since the input feature map contains more detailed information, Faster R-CNN can more accurately locate and identify small target defects on the chip surface, thereby achieving the goal of improving overall detection performance.

[0058] 2.4 Improved non-maximum suppression

[0059] The Rank&Sort function is introduced in non-maximum suppression. On the basis of considering the scores of candidate boxes, the relative ranking between candidate boxes is further optimized to improve the detection accuracy.

[0060] Calculate the ranking loss: Use the ranking loss function to optimize the score ranking of the candidate boxes. The basic formula of the ranking loss function can be the pairwise ranking loss, which is:

[0061] Ranking Loss = ∑ (i,j)∈Pairs max(0,1-(score i -score j ))

[0062] Among them, Ranking Loss is the pairwise sorting loss; Pairs is the overlapping detection box pair with higher IoU, and score i and score j are the scores of candidate boxes i and j respectively. This loss ensures that high-scoring boxes are better than low-scoring boxes and optimizes the ranking results.

[0063] 3. If Figure 3 As shown, the improved FasterR-CNN network structure is used to perform feature extraction, defect candidate (using the region proposal network to generate candidate defect regions) and high-quality suppression (screening out high-quality proposal regions through non-maximum suppression) on the image dataset to obtain candidate regions.

[0064] 4. If Figure 4 As shown, the candidate area is processed using the classification network of Faster R-CNN, the confidence threshold and the improved non-maximum suppression to obtain the location, category and confidence of the chip surface defects.

[0065] Defect classification and bounding box regression: For the candidate regions proposed by the region proposal network, a classification network is used to determine whether they are defects, and bounding box regression is used to accurately locate the location of the defect.

[0066] Post-processing: Post-process the results output by the classification network, including setting a confidence threshold to filter out low-probability detection results, and re-applying non-maximum suppression to remove overlapping detection boxes.

[0067] Result output: The location, category and confidence level of chip surface defects are finally output for subsequent quality control and defect analysis.

[0068] Table 1 is a comparison chart of FasterR-CNN algorithm evaluation indicators. As shown in Table 1, "A" means using ResNet50 network in the traditional Faster R-CNN model, "B" means using K-means clustering to cluster defect anchor boxes in the traditional Faster R-CNN model, "C" means integrating feature pyramid network into the traditional Faster R-CNN model and integrating improved EMSConv into the network, and "D" means adding Rank&Sort function to the RPN stage of NMS in the traditional Faster R-CNN model. "√" means introducing the module into the FasterR-CNN model, and unmarked means not introduced.

[0069] Table 1

[0070]

[0071] Table 2 is a comparison chart of the accuracy results of various defects of the Faster R-CNN algorithm. As shown in Table 2, "wr" represents contamination defect, "qx" represents missing line defect, "lg" represents leakage defect, "jlqx" represents grain tilt defect, and "wx" represents skew line defect.

[0072] Table 2

[0073]

[0074] Therefore, the present invention provides a chip surface defect detection method, which can improve image quality and resolution, increase the detail texture of tiny targets, realize fast and accurate detection of chip surface defects, improve detection efficiency, and ensure product quality.

[0075] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0076] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A chip surface defect detection method, characterized in that: The following steps are involved: Collect image data on the chip surface, process the image data, and then reconstruct the image data using the SRGAN super-resolution reconstruction algorithm to obtain an image data set; The ResNet50 network, K-means clustering, multi-scale feature pyramid, improved EMSConv module and improved non-maximum suppression are introduced into the Faster R-CNN algorithm to obtain the improved Faster R-CNN network structure. Using the improved Faster R-CNN network structure, feature extraction, defect candidate and high-quality suppression are performed on the image data set to obtain a candidate region; The candidate area is processed by using the classification network, confidence threshold and improved non-maximum suppression of Faster R-CNN to obtain the location, category and confidence of the chip surface defect.

2. A chip surface defect detection method according to claim 1, characterized in that: The K-means clustering processing method includes: S1. Based on the image dataset and the Faster R-CNN algorithm, determine the number of clusters and the number of anchor box categories, and randomly select data points as the initial centroids of the clusters; S2, calculating the distance between the size of each anchor box and the initial centroid, and assigning the anchor box to the cluster corresponding to the nearest initial centroid according to the calculated distance; S3, calculating the average value of all anchor box sizes in the cluster, and taking the average value as the new centroid; S4. Repeat steps S2-S3 until the change in the centroid is less than a preset threshold or reaches a preset number of iterations.

3. A chip surface defect detection method according to claim 2, characterized in that: The improved EMSConv module allocates channels according to the number of convolution kernels by utilizing the dynamically calculated number of channels so that each group of convolution operations increases the feature extraction capability and performs multi-scale convolution calculations on the entire input tensor to obtain multi-scale features.

4. A chip surface defect detection method according to claim 3, characterized in that: The processing steps of the multi-scale feature pyramid include: Using a feature pyramid network, extracting features from different levels of the image data set, and connecting the different levels horizontally to generate feature maps of multi-scale information, thus completing the construction of a multi-scale feature pyramid; Utilizing the improved EMSConv module and convolution kernels of different sizes, the detail information in each feature map is extracted and enhanced, the sensitivity to small targets is increased, and detail processing is completed; The region proposal network and detection network in the Faster R-CNN algorithm are used to generate candidate boxes of the feature map after detail processing, and the target is classified and regressed.

5. A chip surface defect detection method according to claim 4, characterized in that: The improved processing steps of the non-maximum suppression include: The Rank&Sort function is introduced in non-maximum suppression to optimize the relative ranking between the candidate boxes, and the ranking loss function is used to optimize the score ranking of the candidate boxes to complete the candidate region processing; the formula of the ranking loss function is: Ranking Loss=∑ (i,j)∈Pairs max(0,1-(score i -score j )) Among them, Ranking Loss is the pairwise sorting loss; Pairs is the overlapping detection box pair with IoU, and score i and score j are the scores of candidate boxes i and j respectively.

6. A chip surface defect detection method according to claim 5, characterized in that: The candidate region is processed using a classification network, a confidence threshold and the improved non-maximum suppression to obtain the location, category and confidence of the chip surface defect, including: Use the region proposal network to generate candidate regions, and use non-maximum suppression to screen out high-quality proposal regions; Use the classification network to determine whether the candidate area is a defect. If so, use bounding box regression to locate the defect position. The confidence threshold is used to filter the low-probability detection results in the defect position, and the improved non-maximum suppression is used to remove the overlapping detection frames, and the position, category and confidence of the chip surface defect are output.

Images