Small sample wafer defect classification method, system and equipment based on self-supervised feature constraint, medium and program product

Abstract

The invention belongs to the technical field of semiconductor manufacturing, and particularly relates to a self-supervised feature constraint-based small sample wafer defect classification method, system and device, a medium and a program product. The method comprises the following steps of: dividing images in a wafer data set into a known defect category and a new defect category; constructing a small sample wafer defect classification model based on self-supervised feature constraint, inputting known defect categories into the model for training, selecting an optimal model, and inputting new defect categories into the optimal model for wafer defect classification to obtain a classification result; systems, devices, and media for implementing the methods thereof; a program product comprising a computer program of the method; according to the invention, the accuracy of wafer defect identification is improved, and more defect feature representations are provided for the network model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of semiconductor manufacturing, and particularly relates to a small sample wafer defect classification method, system, device, medium and program product based on self-supervised feature constraint. BACKGROUND

[0002] As an important component of chips, wafers play an indispensable role in the semiconductor production process. However, due to environmental noise and material variability, scratches, circles, rings and other defects may occur on the wafer surface during the manufacturing process, which adversely affect the performance of semiconductors. Therefore, defect identification is a key step to ensure wafer quality. Traditional wafer defect detection methods largely rely on experienced workers for visual inspection, which is a time-consuming and labor-intensive process. At the same time, as the quality requirements of wafers increase and the complexity increases, traditional methods are difficult to meet the requirements of automated production.

[0003] At present, image classification methods based on convolutional neural networks have made significant progress and have been quickly applied to the field of wafer defect image classification. However, convolutional neural networks require a large amount of data to train a high-performance model. In the actual semiconductor production process, due to the high stability of the equipment, the probability of defect generation is very low, resulting in a lack of defect data. In recent years, as a potential method to deal with the problem of insufficient data, small sample learning has been applied in the field of image classification. Small sample learning uses a meta-learning mode to train a small amount of known class samples, and then tests in an unknown class data that does not overlap with the known class sample data to achieve high recognition accuracy. However, the effect of the meta-learning process is affected by the fully supervised pre-training model, and the resulting model does not have good generalization performance for a completely new class test set.

[0004] Chinese patent application CN119542163A discloses a wafer classification method and device for wafer rapid heating process. However, this method requires a large amount of data for model training, and has the disadvantage of low precision under small sample conditions.

[0005] Liang et al. disclosed Masked autoencoder with dynamic multi-loss adaptation mechanism for few shot wafer map pattern recognition (Liang Qi, Zhou Jian, Wang Yonglin. Masked autoencoder with dynamic multi-loss adaptation mechanism for few shot wafer map pattern recognition [J]. Engineering Applications of Artificial Intelligence, 2024, 137 (Part A): 109070), which proposed a small sample wafer image classification based on two-stage masked autoencoder. In the first stage, the effective representation of the defect wafer map image was obtained by using the masked autoencoder to reconstruct the pixel value of the mask patch based on the smooth loss. In the second stage, a fine-tuning mechanism was proposed to accelerate the rapid feature transfer in the small sample scene by using three kinds of collaborative loss. However, due to the multi-stage training of this method and the single feature extraction method, it has the shortcomings of insufficient feature extraction and feature fusion. SUMMARY

[0006] In order to overcome the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a small sample wafer defect classification method, system, device, medium and program product based on self-supervised feature constraint, which can constrain the training process of supervised learning by self-supervised learning and use adaptive feature alignment mechanism for feature fusion, and has the characteristics of realizing high-accuracy defect recognition under insufficient samples.

[0007] A small sample wafer defect classification method based on self-supervised feature constraint, comprising the following steps:

[0008] S1, dividing the images in the wafer dataset into known defect categories and new defect categories;

[0009] S2, constructing a small sample wafer defect classification model based on self-supervised feature constraint;

[0010] S3, inputting the known defect categories divided in step S1 into the model in step 2 for training, and selecting the optimal model for subsequent testing;

[0011] S4, inputting the new defect categories divided in step S1 into the optimal model selected in step 3 for wafer defect classification to obtain the classification result.

[0012] The step S1 specifically comprises:

[0013] 40-60% of the data set is divided into a known defect category, and the remaining data set is divided into a new defect category.

[0014] The small sample wafer defect classification model of step S2 comprises three branches, namely a supervised training network, a self-supervised training network and an information fusion branch network, and the backbone networks of the three branches all adopt ResNet12;

[0015] The construction process is as follows:

[0016] The supervised learning loss function of the supervised training network is as follows:

[0017]

[0018] Wherein, P(y=k|x i ) represents the probability that the sample x i belongs to the category k, represents the process of calculating the loss of the sample x i and its corresponding label y i .

[0019] The self-supervised training network adopts the SimSiam method to minimize the distance between the prediction vector and the projection vector, so that the small sample wafer defect classification model learns effective feature representation, and in this process, the stop gradient operation is used to ensure the stability of the back propagation in the self-supervised learning process. The self-supervised learning loss function of the self-supervised training network is as follows:

[0020]

[0021] Wherein, p1, p2 represent the prediction vector, z2, z1 represent the projection vector, and D(.) represents the negative cosine similarity. On the basis of supervised and self-supervised learning, the self-supervised learning is used to constrain the supervised learning, so that the features extracted by the small sample wafer defect classification model do not excessively depend on the supervised training network. In the training process of the self-supervised training network, the self-supervised constraint loss is as follows:

[0022]

[0023] Wherein, is the output of the feature extractor of the supervised learning, g ω (x) is the output of the feature extractor of the self-supervised learning.

[0024] The output of the feature extractor of the supervised learning and the output g ω (x) of the feature extractor of the self-supervised learning are input into the information fusion branch network for feature alignment, and the specific operation process is as follows:

[0025] Feature map F for supervised learning s Feature map F for self-supervised learning ss Preprocessing:

[0026]

[0027] where O m and O n are two learnable matrices, F′ S and F′ SS are the supervised feature map F S and the self-supervised feature map F SS respectively. s and F′ ss are the preprocessed supervised feature map F′ s and the preprocessed self-supervised feature map F′ ss respectively.

[0028]

[0029] Then the cosine similarity CS between F′ s1 and F′ ss1 is calculated:

[0030]

[0031] Meanwhile, the cosine similarity CS is negated and multiplied with F′ s and F′ ss to get F″ s2 and F″ ss2 .

[0032]

[0033] Finally, F″ s1 , F″ ss1 , F″ s2 and F″ ss2 are input into a multi-layer perceptron to get the output feature F out :

[0034] F out = MLP(Cat(F″ si , F″ ssi )), i = 1, 2

[0035] Wherein, MLP is a multi-layer perceptron, the multi-layer perceptron is composed of two fully connected layers and a ReLU activation function, and the ReLU activation function is located between the two fully connected layers; Cat is a short name of concatenate, which represents splicing features along the last dimension.

[0036] The step S3 is specifically:

[0037] Training is performed using known defect classification until an optimal model is obtained, and the loss function in the training process is composed of a self-supervised learning loss function And a self-supervised constraint loss The loss function is as follows:

[0038]

[0039] Wherein, D train represents known defect classification, and a and b are loss function weight coefficients, which are updated through each round of back propagation until the model converges to obtain an optimal model.

[0040] A small sample wafer defect classification system based on self-supervised feature constraint, comprising:

[0041] A data set division module divides images in a wafer data set into known defect categories and new defect categories;

[0042] A model construction module constructs a small sample wafer defect classification model based on self-supervised feature constraint;

[0043] A model training module inputs known defect categories into the model for training, and selects an optimal model for subsequent testing;

[0044] A defect classification module inputs new defect categories into the optimal model for wafer defect classification to obtain a classification result.

[0045] A small sample wafer defect classification device based on self-supervised feature constraint, comprising:

[0046] A memory for storing a computer program for implementing a small sample wafer defect classification method based on self-supervised feature constraint;

[0047] A processor for implementing a small sample wafer defect classification method based on self-supervised feature constraint when executing the computer program.

[0048] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps of a small sample wafer defect classification method based on self-supervised feature constraint.

[0049] A computer program product comprises a computer program which, when executed by a processor, implements a small sample wafer defect classification method based on self-supervised feature constraint.

[0050] The present application has the beneficial effect that, relative to the prior art,

[0051] 1. The small sample wafer defect classification method based on self-supervised feature constraint, adopts supervised learning to constrain self-supervised learning, and the self-supervised learning also provides more abundant features for the supervised learning, thereby improving the accuracy of wafer defect recognition under the condition of limited samples.

[0052] 2. The adaptive feature alignment mechanism in the information fusion branch of the classification method is based on cosine similarity, effectively fusing the self-supervised features and full supervision features, and providing more defect feature representations for the network model.

[0053] 3. In the overall training process of the model, the gradient stopping operation can effectively optimize the training process of the model, making the model more stable in the training process.

[0054] In summary, the present application improves the accuracy of wafer defect recognition, which is due to the method (RSSNet) that can use supervised learning to constrain self-supervised learning under the condition of limited samples, and the self-supervised learning also provides more abundant features for the supervised learning. At the same time, the adaptive feature alignment mechanism in the information fusion branch is based on cosine similarity, effectively fusing the self-supervised features and full supervision features, and providing more defect feature representations for the network model. In addition, in the overall training process of the model, the gradient stopping operation can effectively optimize the training process of the model, making the model more stable in the training process. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The step flow chart of the method of the present application.

[0056] Figure 2 The structure diagram of the small sample wafer defect classification model based on self-supervised feature constraint of the present application.

[0057] Figure 3 The adaptive feature alignment mechanism of the present application. DETAILED DESCRIPTION

[0058] The present application will be described in detail below with reference to the accompanying drawings.

[0059] As shown in Figure 1 A small sample wafer defect classification method based on self-supervised feature constraint comprises the following steps:

[0060] S1, dividing the images in the wafer data set into known defect categories and new defect categories;

[0061] Specifically, in order to train the small sample wafer defect recognition model in the mode of meta-learning, the embodiment divides the known defect categories and new defect categories for the public wafer defect pattern dataset WM811K (Wu, M.-J., Jang, J.-S.R., & Chen, J.-L. (2015). Wafer map failure pattern recognition and similarity ranking for large-scale datasets. IEEE Transactions on Semiconductor Manufacturing, 28(1), 1-12.) in the embodiment, which has nine wafer defect categories, namely: None (N), Scratch (S), Local (L), Random (R), Near-Full (NF), Edge-Local (EL), Edge-Ring (ER), Donut (D), Center (C), wherein L, R, ER, N are known defect categories, and S, EL, C, D, NF are new defect categories.

[0062] S2, a small sample wafer defect classification model based on self-supervised feature constraint is constructed, as shown in Figure 2 ;

[0063] The small sample wafer defect classification model includes three branches, namely a supervised training network, a self-supervised training network and an information fusion branch network, and the backbone networks of the three branches all adopt ResNet12;

[0064] The known defect categories are sequentially input into the supervised training network and the self-supervised training network for pre-training, wherein the self-supervised training network adopts a multi-resolution cropping data enhancement method for pre-training, and SimSiam (X. Chen and K. He. (2021) Exploring simple Siamese representation learning, in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., Jun. 2021, pp. 15750-15758.) is used for pre-training, after the pre-training is completed, the pre-trained data is input into the information fusion branch network, and an initialized small sample wafer defect classification model based on self-supervised feature constraint is constructed;

[0065] The supervised learning loss function of the supervised training network is as follows:

[0066]

[0067] where P(y=k|x) represents the probability that the sample x belongs to the class k, i i where L(x,y) represents the process of calculating the loss of the sample x and its corresponding label y, i i

[0068] The self-supervised training network uses the SimSiam method to minimize the distance between the prediction vector and the projection vector, so that the small sample wafer defect classification model learns effective feature representation. In this process, the stop gradient operation is used to ensure the stability of the self-supervised learning training back propagation process. The loss function is as follows:

[0069]

[0070] where p1, p2 represent the prediction vector, z2, z1 represent the projection vector, and D(.) represents the negative cosine similarity. On the basis of supervised and self-supervised learning, self-supervised learning is used to constrain supervised learning, so that the features extracted by the small sample wafer defect classification model do not rely too much on the supervised training network. In the training process of the self-supervised training network, the self-supervised constraint loss is as follows:

[0071]

[0072] where, is the output of the feature extractor of supervised learning, and g ω (x) is the output of the feature extractor of self-supervised learning.

[0073] The output of the feature extractor of supervised learning F and the output of the feature extractor of self-supervised learning g ω (x) are input into the information fusion branch network for feature alignment. The specific operation process is performed through the adaptive feature alignment mechanism shown in Figure 3 , and the specific operation process is as follows:

[0074] The feature map F s of supervised learning and the feature map F ss of self-supervised learning are preprocessed:

[0075]

[0076] where O m and O n are two learnable matrices, F′ s and F′ ss are the results of preprocessing the supervised feature map F s and the self-supervised feature map F ss respectively, and then F′​​​​s and F' ss between them:

[0077]

[0078] Then the numerical value of the similarity is multiplied by the pre-processed supervised feature map F' s and the pre-processed self-supervised feature map F' ss to obtain further supervised feature map F" s1 and further self-supervised feature map F" ss1 .

[0079]

[0080] At the same time, the cosine similarity value CS is negated and multiplied by the pre-processed supervised feature map F' s and the pre-processed self-supervised feature map F' ss to obtain feature F" s2 and feature F" ss2 .

[0081]

[0082] Finally, the further supervised feature map F" s1 , the further self-supervised feature map F" ss1 , the feature F" s2 and the feature F" ss2 are input into the multi-layer perception to obtain the output feature F out :

[0083] F out = MLP(Cat(F" si , F" ssi )), i = 1, 2

[0084] Where MLP is a multi-layer perception, the multi-layer perception consists of two fully connected layers and a ReLU activation function, and the ReLU activation function is located between the two fully connected layers; Cat is a short for concatenate, which means concatenating the features along the last dimension.

[0085] S3, input the known defect categories divided in step S1 into the model in step 2 for training, and select the optimal model for subsequent testing;

[0086] Train using known defect classification until the optimal model is obtained, and the loss function in this training process consists of a self-supervised learning loss function and a self-supervised constraint loss , and the loss function is as follows:

[0087]

[0088] wherein, D train The loss function is updated by each round of back propagation until the model converges, and the optimal model is obtained.

[0089] S4, input the new defect category divided in step S1 into the optimal model selected in step 3 for wafer defect classification to obtain a classification result. The present application can reduce the requirement of the model for the amount of wafer defect data, and realize accurate identification of wafer defect patterns under the condition of limited training sample quantity.

[0090] A small sample wafer defect classification system based on self-supervised feature constraint, comprising:

[0091] A data set division module divides the images in the wafer data set into known defect categories and new defect categories, which is used to realize step 1 of the small sample wafer defect classification method based on self-supervised feature constraint.

[0092] A model construction module constructs a small sample wafer defect classification model based on self-supervised feature constraint, which is used to realize step 2 of the small sample wafer defect classification method based on self-supervised feature constraint.

[0093] A model training module inputs the known defect categories into the model for training, and selects an optimal model for subsequent testing, which is used to realize step 3 of the small sample wafer defect classification method based on self-supervised feature constraint.

[0094] A defect classification module inputs the new defect categories into the optimal model for wafer defect classification to obtain a classification result, which is used to realize step 4 of the small sample wafer defect classification method based on self-supervised feature constraint.

[0095] A small sample wafer defect classification device based on self-supervised feature constraint, comprising:

[0096] A memory for storing a computer program for realizing the small sample wafer defect classification method based on self-supervised feature constraint;

[0097] A processor for realizing the small sample wafer defect classification method based on self-supervised feature constraint when the computer program is executed.

[0098] A computer readable storage medium storing a computer program, the computer program being executed by a processor to realize the steps of the small sample wafer defect classification method based on self-supervised feature constraint.

[0099] A computer program product comprising a computer program which, when executed by a processor, implements a small sample wafer defect classification method based on self-supervised feature constraint.

[0100] Experimental analysis

[0101] During the test, one or five pictures of the five new defect categories were randomly selected as the support set, and the remaining pictures were used as the query set to test the model in 5way-1shot or 5way-5shot.

[0102] The method of the embodiment is implemented on Pytorch 1.10.0 and Python 3.8.13. All experiments are performed on a workstation equipped with an Intel 4210R CPU and an NVIDIA RTX 3090 GPU (24 GB of memory). The small sample wafer defect classification method based on self-supervised feature constraint (RSSNet) proposed in the present application is trained using the Adam optimizer, and the weight decay and momentum are set to 0.9 and 0.0005, respectively.

[0103] The accuracy values obtained by the small sample wafer defect classification method based on self-supervised feature constraint of the embodiment and other methods are shown in Table 1:

[0104] Table 1 Comparison of accuracy results of various methods (%)

[0105]

[0106]

[0107] From the results in Table 1, it can be seen that the small sample wafer defect classification method based on self-supervised feature constraint (RSSNet) of the embodiment exhibits the optimal accuracy in both 5way-5shot and 5way-1shot experimental settings. Compared with the suboptimal method BECLR, the accuracy is improved by 2.1% in the 5way-5shot setting and by 2.09% in the 5way-1shot experimental setting. Overall, the method of the present example is significantly higher than other methods. It shows that the small sample wafer defect classification method of the embodiment can effectively improve the wafer defect recognition accuracy by using the self-supervised learning feature constraint to supervise the learning process in the case of limited sample quantity, and has high application value.

[0108] The comparison methods in Table 1 of the present experiment are all advanced methods in the field of small sample classification in the past five years, including:

[0109] SMFS (Rodriguez, P., Laradji, I., Drouin, A., & Lacoste, A. (2020). Embedding propagation: Smoother manifold for few-shot classification. In Proceedings of the Springer European Conference on Computer Vision (ECCV), 121-138.)

[0110] GIFA (Hu, Y., Gripon, V., & Pateux, S. (2021). Graph-based interpolation of feature vectors for accurate few-shot classification. In Proceedings of the 25th International Conference on Pattern Recognition (ICPR) (pp. 8164-8171).

[0111] EASE (Zhu, H., & Koniusz, P. (2022). EASE: Unsupervised discriminant subspace learning for transductive few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 9078-9088).

[0112] ProtoLP (Zhu, H., & Koniusz, P. (2023). Transductive few-shot learning with prototype-based label propagation by iterative graph refinement. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 23996-24006).

[0113] FaNet (Zhao, W., Song, K., Wang, Y., Liang, S., & Yan, Y. (2023). FaNet: Feature-aware network for few-shot classification of strip steel surface defects. Measurement, 208, Article 112446.)

[0114] FewVS (Li, Z., Wang, Y., & Li, K. (2024). FewVS: A vision-semantics integration framework for few-shot image classification. In Proceedings of the 32nd ACM International Conference on Multimedia, 1341-1350.)

[0115] LMEA (Askari, F., Fateh, A., & Mohammadi, M. R. (2024). Enhancing few-shot image classification through learnable multi-scale embedding and attention mechanisms. arXiv preprint arXiv:409.07989.)

[0116] BECLR (Poulakakis-Daktylidis, S., & Jamali-Rad, H. (2024). BECLR: Batch enhanced contrastive unsupervised few-shot learning. In Proceedings of The Twelfth International Conference on Learning Representations (ICLR).)

Claims

1. A small sample wafer defect classification method based on self-supervised feature constraint, characterized in that, The method comprises the following steps: S1, dividing the images in the wafer dataset into known defect categories and new defect categories; S2, constructing a small sample wafer defect classification model based on self-supervised feature constraints; S3, inputting the known defect categories divided in step S1 into the model in step 2 for training, and selecting the optimal model for subsequent testing; S4, inputting the new defect categories divided in step S1 into the optimal model selected in step 3 for wafer defect classification to obtain a classification result.

2. The method of claim 1, wherein, The step S1 specifically comprises: Divide 40-60% of the dataset into known defect categories, and the remaining dataset into new defect categories.

3. The method of claim 1, wherein, The small sample wafer defect classification model in step S2 comprises three branches, namely a supervised training network, a self-supervised training network, and an information fusion branch network, and the backbone networks of the three branches all adopt ResNet12; The construction process is as follows: The supervised learning loss function of the supervised training network is as follows: Where, P(y=k|x i ) represents sample x i The probability of belonging to category k. Represents the sample x i Its corresponding label y i The process of calculating loss; The self-supervised training network adopts the SimSiam method to minimize the distance between the prediction vector and the projection vector, so that the small sample wafer defect classification model learns effective feature representation, and in this process, the stop gradient operation is used to ensure the stability of the back propagation in the self-supervised learning process, and the self-supervised learning loss function of the self-supervised training network is as follows: Wherein, p1, p2 represent the prediction vector, z2, z1 represent the projection vector, D(.) represents the negative cosine similarity, and on the basis of supervised and self-supervised learning, the self-supervised learning is used to constrain the supervised learning, so that the features extracted by the small sample wafer defect classification model do not excessively depend on the supervised training network, and in the training process of the self-supervised training network, the self-supervised constraint loss is as follows: wherein, is the output of a supervised learning feature extractor, g ω (x) is the output of a self-supervised learning feature extractor; the output of the supervised learning feature extractor and the output g of the self-supervised learning feature extractor ω (x) input information fusion branch network for feature alignment, the specific operation process is as follows: Feature map F for supervised learning s Feature map F for self-supervised learning SS Preprocessing: where O m and O n are two learnable matrices, F′ s and F′ ss are the supervised feature map F s and the self-supervised feature map F ss respectively. After pre-processing, the cosine similarity between F′ s and F′ ss is calculated. The numerical value of the similarity is then multiplied with the pre-processed supervised feature map F' s and the pre-processed self-supervised feature map F' ss to obtain a further supervised feature map F" s1 and a further self-supervised feature map F" ss1 ; Meanwhile, the cosine similarity value CS is negated, and multiplied by the preprocessed supervised feature map F' S and the preprocessed self-supervised feature map F' ss to obtain a feature F" s2 and the feature F" ss2 ; Finally, the further supervised feature map F" s1 is input into a multi-layer perceptron to obtain an output feature F ss1 . s2 and the feature F" ss2 is input into a multi-layer perceptron to obtain an output feature F out . F out = MLP(Cat(F si ,F ssi )), i = 1, 2 Wherein, MLP is a multi-layer perception, the multi-layer perception comprises two fully connected layers and a ReLU activation function, and the ReLU activation function is located between the two fully connected layers; Cat is a concatenate, which means concatenating the features along the last dimension.

4. The method of claim 1, wherein, The step S3 specifically comprises: The training is performed using known defect classifications until an optimal model is obtained, the loss function during this training consisting of a self-supervised learning loss function and a self-supervised constraint loss comprising the following: where D train represent the known defect classification, a and b are loss function weight coefficients, which are updated by each round of back propagation until the model converges to obtain the optimal model.

5. A small sample wafer defect classification system based on self-supervised feature constraint based on the method of any one of claims 1-4, characterized in that, It comprises: A dataset division module for dividing the images in the wafer dataset into known defect categories and new defect categories; A model construction module for constructing a small sample wafer defect classification model based on self-supervised feature constraints; A model training module for inputting the known defect categories into the model for training, and selecting the optimal model for subsequent testing; A defect classification module for inputting the new defect categories into the optimal model for wafer defect classification to obtain a classification result.

6. A small sample wafer defect classification device based on self-supervised feature constraint, characterized in that, It comprises: A memory for storing a computer program for implementing the small sample wafer defect classification method based on self-supervised feature constraints according to any one of claims 1-4; A processor for implementing the small sample wafer defect classification method based on self-supervised feature constraints according to any one of claims 1-4 when the computer program is executed.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the small sample wafer defect classification method based on self-supervised feature constraints according to any one of claims 1-4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the small sample wafer defect classification method based on self-supervised feature constraint according to any one of claims 1-4.

Images