Zero sample segmentation and classification method for integrated circuit defects

Abstract

The invention discloses a zero sample segmentation and classification method for integrated circuit defects, and aims to improve the defect detection accuracy and efficiency in integrated circuit manufacturing. The MFSC does not need extra neural network training, and has strong generalization ability. The method comprises the following steps of: firstly, performing multi-scale local feature extraction on image blocks with different scales by using a pre-trained Vision Transform as a feature extractor; secondly, through a self-comparison scoring mechanism, features of local abnormal image blocks are represented by adopting multi-scale adjacent feature aggregation, mutual comparison of local features of different blocks with the same scale is carried out, and an abnormal score is calculated to realize pixel-level defect segmentation. And finally, dividing and judging a threshold value containing defects according to an experimental statistical result. Experiments prove that the precision of the method in an integrated circuit defect detection task reaches 91.5%. The requirement for defect detection of the integrated circuit in the actual application process is met, and any neural network training is not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and intelligent information processing, and specifically relates to a method for zero-sample segmentation and classification of integrated circuit defects. Background Art

[0002] The manufacturing of integrated circuits (ICs) is a highly sophisticated and complex process, which is prepared layer by layer by so-called planar processes. The detection and classification of integrated circuit defects is a key link to ensure product performance and improve production yield. Traditionally, defect detection relies on manual review after automatic optical inspection (AOI), but since millions of defect images need to be processed every day, operators are prone to make mistakes, resulting in misjudgment or omission of chips. Therefore, the development of automatic defect classification (ADC) systems has become particularly important to reduce the manual burden and improve detection accuracy.

[0003] In recent years, deep learning techniques, including convolutional neural networks (CNNs), vision transformers (ViTs), and other machine learning models, have been widely used in defect detection and classification, showing excellent performance. Nevertheless, most of these techniques rely on a large amount of labeled data for supervised learning, or rely on unsupervised feature learning, and the latter also requires a certain number of unlabeled samples for differential analysis. The unique pattern repetitiveness and local symmetry in integrated circuits, as well as the asymmetry and scarcity of defects, pose challenges to traditional methods, especially in terms of data-driven model generalization ability and adaptability to new types of defects. Summary of the invention

[0004] In view of the limitations of the prior art, such as reliance on a large amount of data training, ignoring the potential information of unlabeled images, and failing to fully consider the characteristics of integrated circuit patterns, the present invention proposes a method for zero-sample segmentation and classification of integrated circuit defects. It aims to efficiently and accurately classify and segment anomalies in integrated circuits without the need for additional neural network training. The method is particularly suitable for integrated circuit patterns with local symmetry and similarity, and can identify asymmetric and scarce defects. The present invention does not have overfitting or underfitting, and has high recognition accuracy. It not only reduces the dependence on a large amount of labeled data, but also makes full use of the inherent laws of integrated circuit patterns, improves the interpretability of the model, and shows significant performance advantages in the detection of integrated circuit defects.

[0005] To achieve the above object, the present invention adopts the following specific technical solutions:

[0006] A method for zero-sample segmentation and classification of integrated circuit defects comprises the following steps:

[0007] a) performing multi-scale cutting on the integrated circuit image to form multiple image blocks of different sizes;

[0008] b) using a pre-trained neural network feature extractor ViT to extract local features from the image blocks of different sizes to obtain multi-scale local features;

[0009] c) Perform Neighborhood Feature Aggregation (NFA) to average pool the features of image blocks of different sizes at the same scale to enhance the ability to characterize defects of different sizes;

[0010] d) Applying a self-contrast scoring (SCS) mechanism to calculate an anomaly score for each image block by comparing local features of different image blocks at the same scale, thereby achieving defect segmentation and classification; wherein the self-contrast scoring mechanism includes:

[0011] i) Calculate the similarity between the local features of each image block and the local features of other image blocks in its neighborhood;

[0012] ii) Calculate the anomaly score of each image block based on the similarity, and the feature area with anomaly score higher than 0.6 is marked as the defect area;

[0013] iii) The maximum anomaly score of each image is taken as the result of defect classification; when the anomaly score is greater than 0.6, the image is classified as containing defects, and when it is less than or equal to 0.6, it is classified as not containing defects.

[0014] The multi-scale cutting includes cutting strategies of at least two different sizes to adapt to the local symmetry of the integrated circuit pattern and the similarity between different patterns.

[0015] The calculation of the anomaly score is specifically as follows: first, the similarity between the features is calculated. After the similarity between the features is calculated, the similarities of the n features that are most similar to each feature are extracted and averaged to be the anomaly score of the current feature. The formula is:

[0016]

[0017] Where s represents the abnormal score of the j-th sub-image after it is cut into b*b blocks, the m-th local feature of the l-th layer after the neural network extraction and the clustering process with r aggregation degree and the i-th sub-image; Represents the local feature vector of the mth feature of the lth layer of the i-th image after being processed by the aggregation degree r; It means that the nth feature extracted from the jth image by the lth layer of the neural network is taken as the center, and the features in the surrounding r*r area are averagely pooled to represent the local features of the current point; It indicates the similarity by calculating the norm distance between two features;

[0018] After calculating the anomaly score, the values ​​with anomaly scores greater than 1 are treated as 1, and then normalized and pseudo-colored using the OpenCV toolbox to visualize the defect area. The maximum anomaly score of each image is taken as the defect classification score of the image. The higher the score, the greater the risk of containing defects. When the score is higher than the set threshold, the current image is judged to contain defects as the result of defect classification.

[0019] Each area in the image with a feature score higher than the set threshold represents a defect area, and the defect classification result is obtained.

[0020] The local feature extractor (ViT) is a pre-trained model and does not require additional training for specific defect types to improve the generalization ability of the model.

[0021] The average pooling process is performed on the image block features of different sizes at the same scale, specifically, the average pooling is performed on the local features and their adjacent features, and the local features of the regions of different sizes are represented by averaging, thereby enhancing the recognition accuracy of defects of different sizes in the defect region.

[0022] i) in step d) is specifically to calculate the similarity between the features extracted from all image blocks at each scale; and to perform spatial distribution statistics of local features to capture the asymmetry and scarcity characteristics of defects.

[0023] The step ii) of step d) calculates the abnormality score of each image block according to the similarity. Specifically, after calculating the similarity between each feature, for each feature, the similarities of a closest features are averaged to obtain the average value as the abnormality score of the current feature.

[0024] In step d) of iii), after calculating the anomaly score of each feature in the image, the maximum value of the anomaly score is taken as the anomaly score of the current image.

[0025] The pre-trained neural network feature extractor ViT of the present invention is used to perform multi-scale image cutting and extract local features; adjacent feature aggregation is used to perform adjacent feature aggregation and detect defects of different sizes; anomaly score calculation is used to apply a self-comparison scoring mechanism to calculate the anomaly score of each image block; segmentation decision is used to output defect classification and segmentation results according to the anomaly score.

[0026] The present invention also includes data storage for storing the pre-trained visual transformer model and the intermediate feature data generated in the processing flow.

[0027] The present invention can achieve the following beneficial technical effects:

[0028] 1) The method provided by the present invention does not require any additional neural network training, which greatly reduces data requirements and computing costs.

[0029] 2) The accuracy of defect detection in the back-end process reaches 91.5%, which is significantly better than the existing zero-sample method, and the performance is close to or even exceeds some methods that require a large number of training samples.

[0030] 3) The present invention not only provides a new approach to solve the defect problems encountered in the integrated circuit manufacturing process, but also demonstrates extremely high generalization ability and practicality, which is of great significance for promoting quality control in the semiconductor industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the present invention;

[0032] Figure 2 It is a schematic diagram of multi-scale local feature extraction of the present invention;

[0033] Figure 3 It is a visualization effect diagram of adjacent feature aggregation of the present invention;

[0034] Figure 4 This is a self-comparison scoring statistical result diagram of the present invention;

[0035] Figure 5 It is a visualization effect diagram of the result after multi-scale cutting of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention and the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0037] The method of the present invention specifically comprises:

[0038] First, a multi-scale local feature extraction is proposed. The image to be detected is cut into multiple patches according to different sizes and then uniformly adjusted to the same size to enhance the expression of local features. The pre-trained Vision Transformer (ViT) is used as a feature extractor to extract local features from these multi-scale image patches, which can reduce the impact of global features on local features and thus improve the detection capability of small-sized defects.

[0039] Then, in order to solve the different sensitivity of the ViT feature extractor to defects of different sizes, we designed a neighboring feature aggregation strategy, which compresses the feature representation by aggregating the features of adjacent regions to different degrees, so that the model can better focus on defects of different sizes. This process helps to strengthen the signal of small defects and suppress the noise interference caused by large defects.

[0040] Finally, a self-comparison scoring mechanism is designed, which calculates the anomaly score for each image block by comparing the local features of different image blocks at the same scale, thus achieving pixel-level defect segmentation and effectively distinguishing normal areas from defective areas.

[0041] Example

[0042] like Figure 1-5 As shown, this embodiment provides a method for zero-sample segmentation and classification of integrated circuit defects, which classifies and segments zero-sample integrated circuit defects based on multi-scale adjacent feature aggregation and self-comparison mechanism, including the following steps:

[0043] First, if Figure 2 As shown in the figure, the integrated circuit image to be inspected is cut into multiple scales to form multiple image blocks of different sizes. Specifically, the image can be cut into sub-image blocks of different sizes such as 3x3, 4x4 and 5x5. After cutting, each sub-image block is uniformly adjusted to the same size (224x224 pixels), and a pre-trained Vision Transformer (ViT) is used as a feature extractor, such as ViT-L / 14-224 trained by DINO V2

[21] , to extract 16x16 local features from each multi-scale image block. For example, the image is first cut into 3x3 sub-images, and 16x16 local features of each sub-image are extracted. A total of 3x3x16x16 local features can be extracted.

[0044] Then, different local features are aggregated with adjacent features of different sizes. Convolution kernels of different sizes are used to perform average pooling on adjacent features, strengthen small defect signals and suppress noise, and aggregate adjacent features of image blocks of different sizes at the same scale. Through different aggregation degrees r, the features of adjacent areas are aggregated to different degrees, so as to better focus on defects of different sizes. Figure 3 As shown in the figure, it can be observed that when the size of the average pooling process is r = 1, it is sensitive to features with smaller sizes. When r = 5, it is more sensitive to features with larger sizes. By aggregating adjacent features of different sizes, small defect signals can be strengthened and noise can be suppressed.

[0045] After that, the comparison score is used to calculate the anomaly score of each local feature after clustering. Specifically, the similarity between different features is first calculated. For the anomaly score of each feature, the similarity of the 5 most similar local features is selected, and the average result is taken as the anomaly score. Because for integrated circuits, similar areas can usually be found in the circuits of normal areas, so their anomaly scores are low. However, defective areas are usually scarce and asymmetric, so even if the 5 most similar local features are found, their anomaly scores are still high. Figure 4 As shown, by performing abnormal score statistics on normal areas and abnormal areas, it can be found that the scores of normal areas and abnormal areas can be clearly distinguished.

[0046] Finally, the anomaly score heatmaps of different sizes are integrated to reduce the interference of ViT's global feature extraction on the anomaly score. Figure 5 As shown in the figure, after averaging the anomaly score results of 3x3, 4x4, and 5x5, it can be found that the boundaries between different sub-images of the anomaly score heat map disappear significantly.

[0047] The algorithm is deployed on the computer as follows: The computer is equipped with NVIDIA 4090 graphics card, Intel Core i713700kf CPU, and ROG 16G memory. The method is built based on opencv, numpy, pytorch and other frameworks, and is developed based on python3.9. It runs on the window 10 system.

[0048] The present invention can locally complete the information leakage, interference and other risks brought by transmission in the task of integrated circuit defect segmentation and classification. And it does not require any training and adjustment of the neural network. It can show strong generalization ability and explainability in the evolving integrated circuit process defect detection.

Claims

1. A method for zero-sample segmentation and classification of integrated circuit defects, characterized in that: The following steps are involved: a) performing multi-scale cutting on the integrated circuit image to form multiple image blocks of different sizes; b) using a pre-trained neural network feature extractor ViT to extract local features from the image blocks of different sizes to obtain multi-scale local features; c) Execute neighbor feature aggregation (NFA) to perform average pooling on image block features of different sizes at the same scale to enhance the characterization capability of defects of different sizes; d) Applying the self-comparison scoring (SCS) mechanism, by comparing the local features of different image blocks at the same scale, calculating the anomaly score for each image block, and realizing the segmentation and classification of defects; wherein the self-comparison scoring mechanism includes: i) Calculate the similarity between the local features of each image block and the local features of other image blocks in its neighborhood; ii) Calculate the anomaly score of each image block based on the similarity, and the feature area with anomaly score higher than 0.6 is marked as the defect area; iii) The maximum anomaly score of each image is used as the defect classification result; when the anomaly score is greater than a specified threshold, the image is classified as containing defects, and when the anomaly score is less than or equal to the specified threshold, the image is classified as not containing defects.

2. The method according to claim 1, characterized in that The multi-scale cutting includes cutting strategies of at least two different sizes to adapt to the local symmetry of the integrated circuit pattern and the similarity between different patterns.

3. The method according to claim 1, characterized in that The calculation of the anomaly score is specifically as follows: firstly, the similarity between the features is calculated, and then, the similarities of the n features that are most similar to each feature are extracted and averaged to obtain the anomaly score of the current feature; The formula is: Where s represents the abnormal score of the j-th sub-image after it is cut into b*b blocks, the m-th local feature of the l-th layer after the neural network extraction and the clustering process with r aggregation degree and the i-th sub-image; Represents the local feature vector of the mth feature of the lth layer of the i-th image after being processed by the aggregation degree r; It means that the nth feature extracted from the jth image by the lth layer of the neural network is taken as the center, and the features in the surrounding r*r area are averagely pooled to represent the local features of the current point; The representative characterizes the similarity by calculating the paradigm distance between two features; After calculating the anomaly score, the values ​​with anomaly scores greater than 1 are treated as 1, and then normalized and pseudo-colored using the OpenCV toolbox to visualize the defect area. The maximum anomaly score of each image is taken as the defect classification score of the image. When the score is higher than a certain threshold, the current image is judged to contain defects as the result of defect classification.

4. The method according to claim 1, characterized in that The average pooling process is performed on the image block features of different sizes at the same scale, specifically, the average pooling is performed on the local features and their adjacent features, and the local features of the regions of different sizes are represented by averaging, thereby enhancing the recognition accuracy of defects of different sizes in the defect region.

5. The method according to claim 1, characterized in that i) in step d) is specifically to calculate the similarity between the features extracted from all image blocks at each scale; and to perform spatial distribution statistics of local features to capture the asymmetry and scarcity characteristics of defects.

6. The method according to claim 1, characterized in that The step ii) of step d) calculates the abnormality score of each image block according to the similarity. Specifically, after calculating the similarity between each feature, for each feature, the similarities of a closest features are averaged to obtain the average value as the abnormality score of the current feature.

Images