A method for zero-shot segmentation and classification of integrated circuit defects

Abstract

The application discloses a kind of integrated circuit defect's zero sample segmentation and classification method, multi-scale adjacent feature aggregation and self-contrast scoring (MFSC), aims at improving the detection accuracy and efficiency of defect in integrated circuit manufacturing.MFSC does not need additional neural network training, with strong generalization ability.The method comprises the following steps: first, using a pre-trained Vision Transformer as a feature extractor, multi-scale local feature extraction is performed on image blocks of different scales;Second, through the self-contrast scoring mechanism, the local abnormal image block features are represented by multi-scale adjacent feature aggregation, and the local features between different blocks of the same scale are compared with each other, and the abnormal score is calculated to realize the pixel-level defect segmentation.Finally, according to the experimental statistical results, whether to contain the threshold value of defect is judged.The experiment proves that the accuracy of this method in integrated circuit defect detection task reaches 91.5%.It meets the demand of integrated circuit defect detection in actual application process, and does not need any neural network training.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and intelligent information processing, and specifically relates to a method for zero-sample segmentation and classification of integrated circuit defects. Background Art

[0002] Integrated circuit (IC) manufacturing is a highly sophisticated and complex process, fabricated layer by layer using a so-called planar process. The detection and classification of IC defects is critical to ensuring product performance and improving production yield. Traditionally, defect detection relies on manual review after automated optical inspection (AOI). However, due to the millions of defect images processed daily, operators are prone to error, leading to misidentification or omission of wafers. Therefore, the development of automatic defect classification (ADC) systems has become particularly important to reduce the manual burden and improve detection accuracy.

[0003] In recent years, deep learning techniques, including convolutional neural networks (CNNs), Vision Transformer (ViT), and other machine learning models, have been widely used in defect detection and classification, demonstrating excellent performance. However, most of these techniques rely on either large amounts of labeled data for supervised learning or unsupervised feature learning, the latter of which also requires a certain number of unlabeled samples for differential analysis. The unique pattern repetitiveness and local symmetry of integrated circuits, as well as the asymmetry and scarcity of defects, pose challenges to traditional methods, particularly in terms of data-driven model generalization and adaptability to new types of defects. Summary of the Invention

[0004] In response to the limitations of existing technologies, such as reliance on large amounts of data for training, neglect of potential information in unlabeled images, and failure to fully consider the characteristics of integrated circuit patterns, the present invention proposes a zero-shot segmentation and classification method for integrated circuit defects. This method aims to efficiently and accurately classify and segment anomalies in integrated circuits without the need for additional neural network training. This method is particularly suitable for integrated circuit patterns with local symmetry and similarity, and can identify asymmetric and scarce defects. The present invention avoids overfitting and underfitting, and has high recognition accuracy. It not only reduces reliance on large amounts of labeled data, but also fully utilizes the inherent laws of integrated circuit patterns, improves the interpretability of the model, and demonstrates significant performance advantages in the detection of integrated circuit defects.

[0005] To achieve the above objectives, the present invention adopts the following specific technical solutions:

[0006] A method for zero-shot segmentation and classification of integrated circuit defects, comprising the following steps:

[0007] a) performing multi-scale cutting on the integrated circuit image to form multiple image blocks of different sizes;

[0008] b) using a pre-trained neural network feature extractor ViT to extract local features from the image blocks of different sizes to obtain multi-scale local features;

[0009] c) Perform Neighborhood Feature Aggregation (NFA) to average pool features of image blocks of different sizes at the same scale to enhance the ability to characterize defects of different sizes;

[0010] d) Applying a self-contrast scoring (SCS) mechanism to calculate an anomaly score for each image block by comparing local features of different image blocks at the same scale, thereby achieving defect segmentation and classification; wherein the self-contrast scoring mechanism includes:

[0011] i) Calculate the similarity between the local features of each image block and the local features of other image blocks in its neighborhood;

[0012] ii) Calculate the anomaly score of each image block based on the similarity, and identify the feature area with an anomaly score higher than 0.6 as the defect area;

[0013] iii) The maximum anomaly score of each image is used as the defect classification result; when the anomaly score is greater than 0.6, the image is classified as containing defects, and when it is less than or equal to 0.6, it is classified as not containing defects.

[0014] The multi-scale cutting includes cutting strategies of at least two different sizes to adapt to the local symmetry of the integrated circuit pattern and the similarity between different patterns.

[0015] The calculation of the anomaly score is specifically as follows: first, the similarity between the features is calculated. After the similarity between the features is calculated, the similarities of the n features that are most similar to each feature are extracted and averaged to obtain the anomaly score of the current feature. The formula is:

[0016]

[0017] Where s represents the abnormality score of the jth sub-image after it is cut into b*b blocks, the mth local feature of the lth layer after the neural network extraction and the clustering process with r aggregation degree, and the ith sub-image; Represents the local feature vector of the mth feature of the lth layer of the i-th image after being processed by the aggregation degree r; It means that the nth feature extracted by the lth layer of the neural network from the jth image is taken as the center, and the features in the surrounding r*r area are averagely pooled to represent the local features of the current point; It expresses similarity by calculating the paradigm distance between two features;

[0018] After calculating the anomaly score, values ​​with an anomaly score greater than 1 are treated as 1, then normalized and pseudo-colored using the OpenCV toolbox to visualize the defect area. The maximum anomaly score for each image is taken as the image's defect classification score. The higher the score, the greater the risk of containing a defect. When the score is higher than the set threshold, the current image is judged to contain a defect, which is used as the defect classification result.

[0019] Each area in the image with a feature score higher than the set threshold represents a defect area, and the defect classification result is obtained.

[0020] The local feature extractor (ViT) is a pre-trained model and does not require additional training for specific defect types to improve the generalization ability of the model.

[0021] The average pooling process is performed on the image block features of different sizes at the same scale, specifically, the average pooling is performed on the local features and their adjacent features, and the local features of the regions of different sizes are represented by averaging, thereby enhancing the recognition accuracy of defects of different sizes in the defect region.

[0022] i) in step d) is specifically to calculate the similarity between the features extracted from all image blocks at each scale; and to perform spatial distribution statistics of local features to capture the asymmetry and scarcity characteristics of defects.

[0023] The abnormality score of each image block is calculated according to the similarity in step ii) of step d), specifically, after calculating the similarity between each feature, for each feature, the similarities of a closest features are averaged to obtain the average value as the abnormality score of the current feature.

[0024] In step d) of iii), after calculating the anomaly score of each feature in the image, the maximum anomaly score is taken as the anomaly score of the current image.

[0025] The pre-trained neural network feature extractor ViT of the present invention is used to perform multi-scale image segmentation and extract local features; adjacent feature aggregation is used to perform adjacent feature aggregation to detect defects of different sizes; anomaly score calculation is used to calculate the anomaly score of each image block using a self-comparison scoring mechanism; segmentation decision is used to output defect classification and segmentation results based on the anomaly score.

[0026] The present invention also includes data storage for storing the pre-trained visual transformer model and intermediate feature data generated in the processing flow.

[0027] The present invention can achieve the following beneficial technical effects:

[0028] 1) The method provided by the present invention does not require any additional neural network training, which greatly reduces data requirements and computing costs.

[0029] 2) The accuracy of defect detection in the back-end process reaches 91.5%, significantly better than existing zero-sample methods, and the performance is close to or even exceeds some methods that require a large number of training samples.

[0030] 3) This invention not only provides a new approach to solving the defect problems encountered in the integrated circuit manufacturing process, but also demonstrates extremely high generalization ability and practicality, which is of great significance for promoting quality control in the semiconductor industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 Schematic diagram of multi-scale local feature extraction of the present invention;

[0033] Figure 3 This is a visualization effect diagram of the adjacent feature aggregation of the present invention;

[0034] Figure 4 This is a self-comparison scoring statistical result diagram of the present invention;

[0035] Figure 5 This is a visualization effect diagram of the multi-scale cutting result of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0037] The method of the present invention specifically comprises:

[0038] First, we propose multi-scale local feature extraction. This approach enhances the expression of local features by cutting the image to be inspected into multiple patches of varying sizes and then resizing them to the same size. Using a pre-trained Vision Transformer (ViT) as a feature extractor, we extract local features from these multi-scale image patches. This reduces the influence of global features on local features, thereby improving the detection of small defects.

[0039] To address the varying sensitivity of the ViT feature extractor to defects of different sizes, we designed a neighboring feature aggregation strategy. This compresses the feature representation by aggregating features from adjacent regions to varying degrees, allowing the model to better focus on defects of varying sizes. This process helps strengthen the signal from small defects and suppress the noise from larger ones.

[0040] Finally, a self-comparison scoring mechanism is designed. It calculates an anomaly score for each image patch by comparing the local features of different image patches at the same scale. This enables pixel-level defect segmentation and effectively distinguishes normal from defective areas.

[0041] Example

[0042] like Figure 1-5 As shown, this embodiment provides a method for zero-shot segmentation and classification of integrated circuit defects, which classifies and segments zero-shot integrated circuit defects based on multi-scale proximity feature aggregation and self-comparison mechanism, including the following steps:

[0043] First, if Figure 2 As shown in the figure, the integrated circuit image to be inspected is cut into multiple scales to form multiple image blocks of different sizes. Specifically, the image can be cut into sub-image blocks of different sizes such as 3x3, 4x4 and 5x5. After cutting, each sub-image block is uniformly adjusted to the same size (224x224 pixels), and a pre-trained Vision Transformer (ViT) is used as a feature extractor, such as ViT-L / 14-224 trained by DINO V2

[21] , to extract 16x16 local features from each multi-scale image block. For example, the image is first cut into 3x3 sub-images, and 16x16 local features are extracted from each sub-image. A total of 3x3x16x16 local features can be extracted.

[0044] Then, different local features are aggregated with adjacent features of different sizes. Convolution kernels of different sizes are used to perform average pooling on adjacent features to enhance small defect signals and suppress noise. Adjacent features are aggregated for image blocks of different sizes at the same scale. By using different aggregation degrees r, features of adjacent regions are aggregated to different degrees, so as to better focus on defects of different sizes. Figure 3 As shown in Figure 2, it can be observed that when the average pooling size r = 1, it is sensitive to features with smaller sizes. When r = 5, it is more sensitive to features with larger sizes. By aggregating adjacent features of different sizes, small defect signals can be strengthened and noise can be suppressed.

[0045] Afterwards, the comparative score is used to calculate the anomaly score of each local feature after clustering. Specifically, the similarity between different features is first calculated. For the anomaly score of each feature, the similarity of the 5 most similar local features is selected, and the average result is taken as the anomaly score. Because for integrated circuits, similar areas can usually be found in the circuits of normal areas, so their anomaly scores are low. However, defective areas are usually scarce and asymmetric, so even if the 5 most similar local features are found, their anomaly scores are still high. Figure 4 As shown in FIG, by performing abnormal score statistics on normal areas and abnormal areas, it can be found that the scores of normal areas and abnormal areas can be clearly distinguished.

[0046] Finally, the anomaly score heatmaps of different sizes are integrated to reduce the interference of ViT's global feature extraction on the anomaly score. Figure 5 As shown in Figure 2, after averaging the 3x3, 4x4, and 5x5 anomaly score results, it can be found that the boundaries between different sub-images of the anomaly score heat map disappear significantly.

[0047] The algorithm was deployed on a computer equipped with an NVIDIA 4090 graphics card, an Intel Core i7-13700kf CPU, and ROG 16GB of RAM. The method was developed using Python 3.9 and based on frameworks such as OpenCV, NumPy, and PyTorch. The system was run on Windows 10.

[0048] This invention can locally perform integrated circuit defect segmentation and classification tasks, eliminating the risks of information leakage and interference associated with transmission. It also eliminates the need for neural network training and adjustment. It demonstrates strong generalization and interpretability in the evolving field of integrated circuit process defect detection.

Claims

1. A method for zero-shot segmentation and classification of integrated circuit defects, characterized in that: The following steps are involved: a) Perform multi-scale segmentation on the integrated circuit image to form multiple image blocks of different sizes; b) extracting local features from the image blocks of different sizes using a pre-trained neural network feature extractor ViT to obtain multi-scale local features; c) Execute Neighbor Feature Aggregation (NFA) to perform average pooling on image block features of different sizes at the same scale to enhance the representation capability of defects of different sizes; d) applying a self-comparative scoring (SCS) mechanism to calculate an anomaly score for each image block by comparing local features of different image blocks at the same scale, thereby achieving defect segmentation and classification; wherein the self-comparative scoring mechanism includes: i) Calculate the similarity between the local features of each image block and the local features of other image blocks in its neighborhood; ii) Calculate the anomaly score of each image block based on the similarity, and identify the feature area with an anomaly score higher than 0.6 as the defect area; iii) The maximum anomaly score of each image is used as the defect classification result; when the anomaly score is greater than a specified threshold, the image is classified as containing a defect; when the anomaly score is less than or equal to the specified threshold, the image is classified as not containing a defect; wherein: The calculation of the anomaly score is specifically as follows: first, the similarity between the features is calculated, and then the similarities of the n features that are most similar to each feature are extracted and averaged to obtain the anomaly score of the current feature; the formula is: ； Where s represents the abnormal score of the jth sub-image after it is cut into b*b blocks, the mth local feature of the lth layer after the neural network extraction and the clustering process with r degree of aggregation compared with the i-th sub-image Represents the local feature vector of the mth feature of the lth layer of the i-th image after being processed by the aggregation degree r; It means that the nth feature extracted by the lth layer of the neural network from the jth image is taken as the center, and the features in the surrounding r*r area are averagely pooled to represent the local features of the current point; The representative represents the similarity by calculating the paradigm distance between two features; After calculating the anomaly score, values ​​with an anomaly score greater than 1 are treated as 1, then normalized and pseudo-colored using the OpenCV toolbox to visualize the defect area. The maximum anomaly score of each image is taken as the defect classification score of the image. When the score is higher than a certain threshold, the current image is judged to contain defects, which is used as the defect classification result.

2. The method according to claim 1, wherein The multi-scale cutting includes cutting strategies of at least two different sizes to adapt to the local symmetry of the integrated circuit pattern and the similarity between different patterns.

3. The method according to claim 1, wherein The average pooling process is performed on the image block features of different sizes at the same scale, specifically, the average pooling is performed on the local features and their adjacent features, and the local features of the regions of different sizes are represented by averaging, thereby enhancing the recognition accuracy of defects of different sizes in the defect region.

4. The method according to claim 1, wherein i) in step d) is specifically to calculate the similarity between the features extracted from all image blocks at each scale; and to perform spatial distribution statistics of local features to capture the asymmetry and scarcity characteristics of defects.

5. The method according to claim 1, wherein The abnormality score of each image block is calculated according to the similarity in step ii) of step d), specifically, after calculating the similarity between each feature, for each feature, the similarities of a closest features are averaged to obtain the average value as the abnormality score of the current feature.

Images