A machine learning-assisted approach for wafer manufacturing process assessment and anomaly detection

Abstract

The present invention discloses a machine learning-assisted wafer manufacturing process evaluation and anomaly detection method, comprising: constructing a text prompt library based on domain expert knowledge, including morphological descriptions of all possible defects, and obtaining a text feature vector through a CLIP text encoder; extracting customized image features to obtain original image features and new image features; converting the original image features based on a conversion layer fine-tuned by a small number of samples to obtain converted original image features, and using the converted original image features to calculate similarity with text features to obtain a defect map A F ; The new image features and text features are used to calculate the similarity of the redundant features to obtain the defect map A V ; Defect map A F and A V The final defect segmentation map is obtained by adding them together; based on the SEM defect image, the original image features are used to classify the defects, and the classification results are output. The method of the application can effectively reduce background interference and realize efficient and accurate classification and segmentation of nano-scale wafer surface defects.

Description

Technical Field

[0001] The present invention belongs to the field of semiconductor detection technology, and specifically relates to a machine learning-assisted wafer manufacturing process evaluation and anomaly detection method. Background Art

[0002] Integrated circuits continue to develop in the direction of ultra-high density. Tens of billions of transistors are integrated on a smaller and smaller chip, making more and smaller defects gradually become an important factor affecting the yield. From the end of production to the completion of shipment, the entire manufacturing process of the wafer needs to go through hundreds of process steps, and defects may occur in each step. Therefore, interspersed defect inspections between process steps can detect problems in time and avoid major losses. Defect detection collects scanning electron microscope (SEM) images of wafers, and then engineers determine the type of defects based on the appearance characteristics of the defects, and obtain information such as the location and size of the defects. This can help determine the specific process and equipment that caused the defects. Further combined with other conditions, the root cause of the defects can be analyzed and inferred, and these problems can be modified and adjusted in time to ensure the yield of the production line.

[0003] However, currently, manual defect detection is still mainly used on actual production lines. The accuracy of this method depends largely on the professional knowledge of the inspector, which is not only labor-intensive but also very time-consuming. Therefore, some scholars have begun to try to use deep learning models to replace manual work. Although some progress has been made, due to the complexity of the manufacturing process, the SEM defect textures and their backgrounds are complex and diverse, and it is difficult for traditional models to accurately distinguish them, resulting in a high false positive rate.

[0004] Pre-trained visual-language models are pre-trained on large-scale image-text pairs, and use contrastive learning techniques to map images and texts into the same embedding space, so that information from different modalities can be directly compared, thus having strong cross-modal understanding capabilities. Currently, visual-language models represented by CLIP and DALL-E have shown excellent performance in tasks such as image classification, generation, and image-text matching. Among them, the CLIP model has been pre-trained on 400 million image-text pairs and has strong generalization capabilities. Its structure mainly consists of two parts: an image encoder and a text encoder. These encoders are usually based on the Transformer structure and rely on the self-attention mechanism. The CLIP model obtains the image features and text features of the corresponding inputs through encoding, and then projects these features into a shared embedding space, where their similarity is calculated and trained using the contrastive loss function. In this process, the model learns how to align the features of matching image-text pairs closely and separate unmatched image-text pairs. Therefore, the pre-trained CLIP model has the ability to achieve zero-shot or few-shot classification and detection using only natural language prompts without a large amount of specific task data.

[0005] Few-shot learning and fine-tuning techniques are key methods in the field of machine learning for improving model performance when data is scarce. Few-shot learning effectively uses a small amount of labeled data to train the model to adapt to new tasks; while fine-tuning technology uses a small amount of task-related data for further training based on the pre-trained model, so that the model can be better adjusted for specific tasks. This method can significantly improve the performance of the model in specific fields while retaining the excellent generalization ability of the pre-trained model. It is currently widely used in industrial applications such as defect detection. By using few-shot learning and fine-tuning techniques, the dependence on large-scale labeled data can be effectively reduced, while improving the accuracy and robustness of the model.

[0006] WinCLIP was the first to propose the use of pre-trained visual-language models to achieve zero-shot and few-shot anomaly classification and segmentation through language prompts, and combined state words and prompt templates to accurately characterize normal and abnormal states. However, it is difficult to accurately focus on the defect area under a changing background environment because it cannot fully remove the interference caused by the background. AnomalyGPT achieves anomaly detection under few samples by simulating the generation of abnormal images and their corresponding text descriptions. The model uses natural language generation technology to capture abnormal features and enhances the generalization ability of the model through text descriptions. However, when faced with complex backgrounds and subtle defects in real industrial scenes, it shows insufficient generalization ability, and its application in actual production environments is limited due to high computing resource requirements. PromptAD learns text prompts through a data-driven approach, and uses a dual-path model structure and feature cutting technology to achieve anomaly detection. This method can flexibly adjust prompts to adapt to different detection tasks, but it cannot effectively distinguish defect categories with complex morphology and similar textures, especially when different types of defects have similar textures, they are more likely to be confused. Summary of the invention

[0007] In order to make up for the shortcomings of the prior art, the present invention provides a machine learning-assisted wafer manufacturing process evaluation and anomaly detection method to achieve efficient and accurate classification and segmentation of nano-scale wafer surface defects.

[0008] The technical problem solved by the present invention can be achieved by adopting the following technical solutions:

[0009] The machine learning-assisted wafer manufacturing process evaluation and anomaly detection method comprises the following steps:

[0010] S1. Build a text prompt library based on domain expert knowledge, which contains the morphological descriptions of all possible defects, and obtain the text feature vector through the CLIP text encoder;

[0011] S2. Customized image feature extraction to obtain original image features and new image features;

[0012] S3. The original image features are fine-tuned based on a small number of samples. The converted original image features are obtained, and the defect map is obtained by calculating the similarity between them and the text features. A F ; The new image features and text features are used to calculate the similarity of the redundant features to obtain the defect map A V ; Defect map A F and A V Add together to get the final defect segmentation map;

[0013] S4. Based on the SEM defect image, the original image features are used to classify the defects and the classification results are output.

[0014] Furthermore, in step S1, the prompt is split into two parts: template level and state level: the state level prompt describes in detail the possible morphology of each type of defect; the template level prompt describes the shooting effect of the SEM image; finally, the state level prompt replaces the {state} in the template level, thereby combining to obtain the final text prompt library with expert prior knowledge.

[0015] Furthermore, the specific operation steps of step S2 are:

[0016] S21. Based on the original CLIP model, VV self-attention mechanism is introduced;

[0017] S22. Transforming the image encoder to output original image features and new image features;

[0018] S23. Extract features of different levels from the output of each encoding block to obtain m original image features , and m new image features transformed by VV attention .

[0019] Furthermore, in step S21, in the original CLIP model, ViT based on the Transformer structure is usually used as the image encoder. Since ViT relies on the QK self-attention mechanism, the QK self-attention mechanism formula uses Query (Q) and Key (K) to capture the dependency between different parts, that is:

[0020] ,

[0021] The VV self-attention mechanism is introduced. The VV attention mechanism directly performs autocorrelation calculation on Value (V) to strengthen the connection between similar features, allowing the model to pay more attention to semantically similar or connected areas, that is:

[0022] .

[0023] Furthermore, in step S22, the specific process of modifying the image encoder is as follows:

[0024] First, while keeping the original Transformer structure unchanged, a branch is added, using the VV attention module, thus forming a new dual-path block that outputs two image features at the same time;

[0025] Then, n dual-path blocks are used to form a coding block. For the i-th dual-path block in the j-th coding block, the input is , there are two outputs, namely:

[0026] Original image features: ,

[0027] New image features: ,

[0028] in, Represents the QK block processing, is the output of this branch; represents a VV block, is its output;

[0029] Since the input of the j-th encoding block is the last layer output of the j-1-th encoding block, that is, , so for the jth encoding block, its output is:

[0030] ,

[0031] .

[0032] Furthermore, the specific operation process of step S3 is as follows:

[0033] S31. Use the pre-trained CLIP text encoder to convert our designed expert knowledge text guide T into text features t:

[0034] ;

[0035] S32. The input SEM image is passed through a customized image encoder to obtain the original image features F And new image features V ;

[0036] S33. For original image features ,in As token, aggregates the global features of the image and is used for defect classification tasks. Contains more detailed information; similarly for new image features, remove token, will , For pixel-level defect segmentation;

[0037] S34. Use the original image features to calculate the defect map and obtain the segmentation result of the original image features ;

[0038] S35. Use the new image features to calculate the defect map and obtain the segmentation result of the new image features ;

[0039] S36. Based on the two obtained image features, the final defect segmentation map is calculated as:

[0040] .

[0041] Furthermore, in step S34, a conversion layer is added to the original image features and fine-tuned using the SEM sample. Specifically, the conversion layer is implemented by a linear layer, which maps the image features to the joint embedding space. The input of the mapping is , where the output of the j-th encoding block is For example, the mapping process is as follows:

[0042] ;

[0043] Then the original image features after linear transformation Use cosine similarity to calculate the similarity between it and text feature t. The calculation formula is as follows:

[0044] ;

[0045] The defect map generated by the output of the j-th encoding block guided by the text prompt can be calculated:

[0046] ,

[0047] In the formula, Upsample refers to the upsampling operation, which is then normalized by the softmax layer;

[0048] Then sum the defect images corresponding to the m original image codes to obtain the segmentation result of the original image features. :

[0049] .

[0050] Furthermore, in step S35, for the new image feature, the text feature t and the image feature t are firstly compared. conduct Normalize and then multiply element by element to generate a fused feature vector containing both information:

[0051] ,

[0052] Fusion features Taking the average value to get redundant features :

[0053] ,

[0054] In the fusion feature Remove redundant features , get the defect image:

[0055] ,

[0056] Similarly, for m new image features The corresponding defect maps are summed to obtain the final segmentation result :

[0057] .

[0058] Furthermore, in step S4, an additional learnable token is used to aggregate information from global features during image encoding, which is located between the original image features The first encoding position of the classification is as follows:

[0059] S41. Since there are m encoding blocks in total, we Get the classification The vector is represented as:

[0060] ;

[0061] S42. Obtaining effective features After that, by fine-tuning the classification head, the classification prediction probability is obtained , thereby realizing the defect classification function:

[0062] ,

[0063] ,

[0064] Where W is the weight matrix of the classifier and b is the bias term.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] (1) The present invention uses a composite template to integrate domain expert knowledge and designs text prompts that can effectively guide the model to accurately identify defects, significantly improving the classification accuracy of defects with complex morphology and similar textures.

[0067] (2) The present invention optimizes image feature extraction by introducing the VV attention mechanism and adding a dual-path block, while transforming the model structure to perform multi-level feature extraction, thereby effectively reducing background interference and enhancing the model's ability to focus on defect areas.

[0068] (3) The present invention proposes a technical solution for fine-tuning a pre-trained model using a small amount of labeled samples by adaptively modifying the model structure. This method effectively enhances the model's understanding of the application scenarios of wafer surface defects and can maintain a high level of detection accuracy under conditions of data scarcity. It is a key technology for solving the problem of data dependence. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of the method of the present invention;

[0070] Figure 2 This is a dual-path block structure diagram of the present invention;

[0071] Figure 3 This is a structural diagram of the coding block of the present invention;

[0072] Figure 4 Schematic diagram of customized image feature extraction of the present invention, wherein (a) is the original CLIP image encoder, and (b) is the modified customized image encoder;

[0073] Figure 5 This is a diagram of the overall model architecture for defect classification and segmentation of the present invention;

[0074] Figure 6 It is the SEM data set in the embodiment of the present invention;

[0075] Figure 7 This is a visualization diagram of the segmentation results of different methods in Example 1 of the present invention. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to more clearly understand the technical solution of the present invention, a machine learning-assisted wafer manufacturing process evaluation and anomaly detection method of the present invention is further described below in conjunction with the accompanying drawings.

[0077] like Figure 1 As shown, a machine learning-assisted wafer manufacturing process evaluation and anomaly detection method includes the following steps:

[0078] (I) Text prompt design: A text prompt library is built based on domain expert knowledge, which contains the morphological descriptions of all possible defects, and the text feature vector is obtained through the CLIP (Contrastive Language-Image Pretraining) text encoder.

[0079] Due to the complexity of the integrated circuit manufacturing process, the appearance of wafer surface defects is highly variable. Defects of the same type often have very different morphologies, while defects of different types have very similar textures. In the past, text prompts often used rough prompts such as "abnormal" or "defect". Later, WinCLIP first proposed the use of state-level templates and template-level templates to form combined prompts to accurately characterize normal and abnormal states. However, due to the lack of relevant knowledge about semiconductor manufacturing defects in its description, it is not possible to guide the model to detect wafer surface defects.

[0080] This application proposes to use domain expert knowledge combined with a combined prompt structure to design text prompts, and refine rough defect names such as "scratches" and "holes" into more detailed defect morphology descriptions. These descriptions are based on relevant prior information about the target defect area. For example, for "scratches" type defects, they generally appear as thin and long straight scratches in the back-end process (BEOL) step, but in the front-end process (FEOL) step, they often appear in the form of fish scales. If these elliptical depressions are not carefully observed to show a continuous distribution, it is easy to misjudge them as "hole" type defects. Therefore, this application adopts a combined prompt integration structure to split the prompts into two parts: template level and state level:

[0081] ① In the state-level prompt, the possible morphology of each type of defect is described in detail, such as '{} image accompanied by linear scratches', '{} image accompanied by fish-scale scratches'.

[0082] ② Since the scanning electron microscope may also cause blurring due to inaccurate focus and changes in image brightness due to different electron beam intensities, the shooting effect of the SEM image is described in the template-level prompt, such as 'a blurry {state}.', 'a darker {state}.'.

[0083] ③ Finally, replace {state} in the template level with the state-level prompt to obtain the final text prompt with expert prior knowledge.

[0084] (II) Customized image feature extraction: When in use, the input SEM graphic data is passed through a customized dual-path graphic feature encoder to extract the original graphic features and the new graphic features obtained by VV self-attention from different levels. The specific operation steps are as follows:

[0085] (1) Based on the original CLIP model, the VV self-attention mechanism is introduced.

[0086] In SEM images, due to the variability and complexity of background patterns, it is easy to interfere with the detection of defects. In the original CLIP model, ViT (Vision Transformer) based on the Transformer structure is usually used as the image encoder. Since ViT relies on the QK self-attention mechanism, this may cause the model to establish incorrect associations in semantically irrelevant areas (such as between background and foreground) when processing images, resulting in distraction. The QK self-attention mechanism formula uses the query vector Query (Q) and the key vector Key (K) to capture the dependencies between different parts, namely:

[0087] ,

[0088] Since Q and K try to find matching features in the entire image, and this matching may be inaccurate due to insufficient model training or inappropriate parameters, it leads to misplaced attention, causing the model to focus on areas in the image that are irrelevant or opposite to the query.

[0089] Therefore, this application introduces the VV self-attention mechanism, which enables the model to focus more accurately on relevant feature areas by directly comparing and associating similar feature values, thereby reducing the interference of irrelevant background information. The VV attention mechanism directly performs autocorrelation calculations on the value vector Value (V) to strengthen the connection between similar features, allowing the model to pay more attention to semantically similar or connected areas to avoid interference from irrelevant areas, that is:

[0090] .

[0091] (2) Modify the image encoder to output the original image features and the new image features. The specific process of modifying the image encoder is as follows:

[0092] ① On the basis of keeping the original Transformer structure unchanged, a branch is added, where the VV attention module is used to form a new dual-path block, such as Figure 2 As shown, it can output two image features at the same time.

[0093] ② Use n dual-path blocks to form a coding block, such as Figure 3 As shown, for the i-th dual-path block in the j-th coding block, its input is , there are two outputs, namely:

[0094] Original image features: ,

[0095] New image features: ,

[0096] in, Represents the QK block processing, is the output of this branch; represents a VV block, is its output.

[0097] ③ Since the input of the j-th coding block is the output of the last layer (n-th dual-path block) of the j-1-th coding block, that is , so for the jth encoding block, its output is:

[0098] ,

[0099] .

[0100] (3) Extract features of different levels from the output of each encoding block to obtain m original image features , and m new image features transformed by VV attention ,like Figure 4 As shown, (a) is the original CLIP image encoder, and (b) is the modified customized image encoder.

[0101] (III) Original image features Based on the conversion layer fine-tuned with a small number of samples, the converted original image features are obtained, and the similarity between them and the text features is calculated to obtain the defect map A F ; The new image features and text features are used to calculate the similarity of the redundant features to obtain the defect map A V ; Defect map A F and A V The final defect segmentation map is obtained by adding them together.

[0102] When using the pre-trained CLIP model for zero-sample defect segmentation, the existing technology directly calculates the similarity between text and image features to generate defect maps, but this method is not suitable for wafer surface defect detection. This is due to the confidentiality of integrated circuit production line data. Among the 400 million pairs of data used for pre-training of the CLIP model, there are very few or even no scanning electron microscope images of wafer surface defects, so it cannot be directly migrated to this application and requires a small number of samples for fine-tuning.

[0103] This application improves the model's detection of SEM defects by adding a conversion layer and adopting a few-sample learning method, and uses a new similarity calculation method to reduce the interference of irrelevant background information on defect detection. The overall model architecture is as follows Figure 4 The specific operation process is as follows:

[0104] (1) Use the pre-trained CLIP text encoder to convert our designed expert knowledge text guide T into text features t:

[0105] .

[0106] (2) The input SEM image is passed through a customized image encoder to obtain the original image features. And new image features .

[0107] (3) For the original image features ,in As a token , aggregates the global features of the image for defect classification tasks, while contains more detailed information; the same is true for new image features. So we remove ,Will , For pixel-level defect segmentation.

[0108] (4) Use the original image features to calculate the defect map and obtain the segmentation result of the original image features .

[0109] For the original image features, in order to make the model better understand the application scenarios of wafer surface defects and achieve better alignment between images and text, a conversion layer is added and fine-tuned using SEM samples. Specifically, the conversion layer is implemented by a linear layer, which maps the image features to the joint embedding space. The input of the mapping is , where the output of the j-th encoding block is For example, the mapping process is as follows:

[0110] ;

[0111] Then the original image features after linear transformation Use cosine similarity to calculate the similarity between it and text feature t. The calculation formula is as follows:

[0112] ;

[0113] We can further calculate the defect map generated by the output of the j-th encoding block guided by the text prompt:

[0114] ,

[0115] Upsample here refers to the upsampling operation, which is then normalized by the softmax layer.

[0116] Then, the defect images corresponding to the m original image codes are summed to obtain the segmentation result of the original image features. :

[0117] .

[0118] (5) Use the new image features to calculate the defect map and obtain the segmentation result of the new image features .

[0119] For new image features, firstly, the text feature t and the image feature conduct Normalize and then multiply element by element to generate a fused feature vector containing both information:

[0120] ,

[0121] Then fusion features Taking the average value to get redundant features :

[0122] ,

[0123] In the fusion feature Remove redundant features , get the defect image:

[0124] ,

[0125] Similarly, for m new image features The corresponding defect maps are summed to obtain the final segmentation result :

[0126] .

[0127] (6) Based on the two image features obtained, the final defect segmentation map is calculated as:

[0128] .

[0129] (iv) Based on the SEM (Scanning Electron Microscope) defect image, the original image features are used to classify the defects and the classification results are output.

[0130] Although CLIP's contrastive learning capability can be directly applied to image classification tasks, as mentioned above, it is difficult for the pre-trained visual-language model to directly obtain satisfactory results in our application scenario, so it is necessary to use a small number of SEM defect images for fine-tuning. Therefore, this application uses an additional learnable token to aggregate information from global features during image encoding, which is located between the original image features. The first encoding position of the j-th encoding block For example, its that is The specific process of implementing classification is as follows:

[0131] (1) Since there are m encoding blocks in total, we select m original image features Get the classification The vector is represented as:

[0132] ;

[0133] (2) Obtaining effective features Finally, use it to fine-tune a simple classification head, such as a linear classifier, to get the classification prediction probability , thereby realizing the defect classification function:

[0134] ,

[0135] ,

[0136] Where W is the weight matrix of the classifier and b is the bias term.

[0137] In summary, the present application method builds a text-guided library based on domain expert knowledge, which contains morphological descriptions of all possible defects, and obtains text feature vectors through the CLIP text encoder. When in use, the input SEM graphic data is extracted from different levels through a customized dual-path graphic feature encoder to extract the original graphic features and the new graphic features obtained through VV self-attention. After removing the first CLS token from the two graphic features, the original image features are fine-tuned through a conversion layer using a small number of samples to obtain the converted original image features, which are used to calculate the similarity with the text features to obtain the defect image. ; For the new image features, directly use them to calculate the similarity of the redundant features with the text features to obtain the defect map . Defect map and The final defect segmentation map is obtained by adding them together, which completes the abnormal detection of wafer surface defects. Example 1

[0138] Due to the lack of publicly available SEM image datasets, this example collects some data from a 12-inch 55-nanometer CMOS manufacturing line. The dataset contains 1,332 grayscale images, including 226 defect-free images and 1,106 defective images. The defect data includes six common defect types: 59 bridges, 141 copper residues, 230 holes, 77 film defects, 455 particles, and 144 scratches. Figure 6 shown.

[0139] This embodiment illustrates the performance comparison of the method provided by the present invention and other few-sample models in defect detection applications:

[0140] This example was performed on an NVIDIA GeForce RTX 4090 GPU with an Intel Xeon Gold 6133 CPU and a single 24GB memory capacity, running in a PyTorch 1.10.0 environment supporting CUDA 11.3. The Adam optimizer was used for parameter update, the learning rate was set to 0.0001, the training lasted for 50 epochs, and the batch size was 1.

[0141] The pre-trained model publicly accessible by OpenCLIP3 CLIP is used, that is, the CLIP model based on LAION-400M and equipped with ViT-B / 16+ is used. The resolution of the input image is 240×240, which is divided into 15×15 patches, each patch size is 16×16, and a learnable classification [𝐶𝐿𝑆] tag is added at the beginning of the 225 features. The backbone structure of the image encoder contains 12 layers, which are divided into 4 encoding blocks, that is, 𝑚=4, so each encoding block contains 3 layers, corresponding to 3 dual-path blocks, that is, 𝑛=3.

[0142] In order to evaluate the defect segmentation performance, the proposed method is compared with some of the best anomaly detection models under different few-shot conditions, including WinCLIP+, PromptAD, DRA, and AnomalyGPT. Tables 1, 2, and 3 below show the comparison results of the proposed method and existing defect detection methods under three performance indicators: iAUROC, pAUROC, and F1-max scores. The results show that the proposed method outperforms other optimal models in iAUROC, pAUROC, and F1-max scores under all few-shot settings. For example, in the 1-shot (only 1 sample is used for training) setting, the proposed method improves the highest score of other models by 1.4% / 0.4% / 1.7%; in the 2-shot setting, it improves by 1.5% / 0.2% / 11.3%; in the 5-shot setting, it improves by 2.1% / 0.9% / 0.4%; in the 10-shot setting, it improves by 1.3% / 0.4% / 1.5%.

[0143] Table 1 Comparison of image-level AUROC performance of different methods under different few-shot condition settings

[0144] Model 1-shot 2-shot 5-shot 10-shot WinCLIP+ 51.4 55.5 64.9 72.2 PromptAD 94.1 96.1 96.3 97.8 DRA 96.6 97.3 97.6 98.5 AnomalyGPT 86.8 89.8 86.3 86.4 Ours 98.0 98.8 99.7 99.8

[0145] Table 2 Comparison of pixel-level AUROC performance of different methods under different few-sample condition settings

[0146] Model 1-shot 2-shot 5-shot 10-shot WinCLIP+ 84.5 85.3 86.1 87.0 PromptAD 95.8 96.5 96.9 97.3 DRA 81.2 91.7 96.9 98.2 AnomalyGPT 96.3 96.6 96.5 96.5 Ours 96.7 96.8 97.8 98.6

[0147] Table 3 Comparison of F1-max score performance of different methods under different few-shot condition settings

[0148] Model 1-shot 2-shot 5-shot 10-shot WinCLIP+ 28.5 29.5 29.7 31.1 PromptAD 58.2 60.4 61.5 62.7 DRA 67.9 70.5 78.2 82.3 AnomalyGPT 61.6 63.1 65.8 65.2 Ours 69.6 74.4 78.6 83.8

[0149] In this example, the few-shot condition is set to 10-shot, and the visual representative results of segmenting each defect category using different methods are shown, such as Figure 7 As shown, it can be seen that the method of the present application can clearly identify the fuzzy boundary between the normal image background and the defect area without being disturbed by irrelevant background information, thereby achieving accurate defect positioning and segmentation. Example 2

[0150] This embodiment illustrates the performance comparison of the method provided by the present invention and other few-sample models in defect classification applications:

[0151] The model training method and parameter settings of this embodiment are consistent with those in Example 1. Since the defect detection methods compared in Example 1 do not have the ability to classify multiple categories of defects, this embodiment selects models pre-trained on ImageNet-21K to compare classification performance, including ViT, ResNet101, ResNet50+ViT and EfficientNet. For a fair comparison, these models are fine-tuned using the same 10-shot sample on the SEM dataset of this application.

[0152] In order to evaluate the defect classification performance, accuracy, precision, recall and F1 score were selected as evaluation indicators. As shown in Table 4, the method of this application achieved the best results in accuracy, recall and F1 score, but the pre-trained EfficientNet model was better in accuracy, which may be due to the huge variability of the natural ImageNet dataset providing rich prior knowledge for the EfficientNet model, and the advanced regularization technology used in EfficientNet gave it an advantage; however, the overall accuracy of EfficientNet was low, indicating that its defect recognition ability was weak. The method of this application performed well in almost all indicators, especially in F1 score, which shows that it can identify defect categories and minimize false negatives. Therefore, the method of this application is more suitable for this SEM image classification application scenario containing a large number of unbalanced defect categories.

[0153] Table 4 Comparison of classification performance of different methods under 10-shot condition setting

[0154] Model Accuracy Accuracy Recall F1 score ViT 81.2 78.5 84.5 78.9 ResNet101 71.4 72.8 76.1 70.2 ResNet50+ViT 81.2 75.8 85.3 78.4 EfficientNet 78.5 89.5 83.3 81.6 Ours 83.7 87.2 86.7 84.4

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A machine learning-assisted wafer manufacturing process evaluation and anomaly detection method, characterized in that: The following steps are involved: S1. Build a text prompt library based on domain expert knowledge, which contains the morphological descriptions of all possible defects, and obtain the text feature vector through the CLIP text encoder; S2. Customized image feature extraction to obtain original image features and new image features. The specific steps are as follows: S21. Based on the original CLIP model, the VV self-attention mechanism is introduced; in the original CLIP model, ViT based on the Transformer structure is usually used as the image encoder. Since ViT relies on the QK self-attention mechanism, the QK self-attention mechanism formula uses Query (Q) and Key (K) to capture the dependencies between different parts of 1, that is: The VV self-attention mechanism is introduced. The VV self-attention mechanism directly performs autocorrelation calculation on Value(V) to strengthen the connection between similar features, allowing the model to pay more attention to semantically similar or connected areas, that is: S22. Modify the image encoder to output the original image features and the new image features; the specific process of modifying the image encoder is as follows: First, while keeping the original Transformer structure unchanged, a branch is added, using the VV attention module, thus forming a new dual-path block that outputs two image features at the same time; Then, n dual-path blocks are used to form a coding block. For the i-th dual-path block in the j-th coding block, the input is There are two outputs: Original image features: New image features: Among them, Arch QKV Represents the QK block processing, is the output of this branch; Arch VVV represents a VV block, is its output; Since the input of the j-th encoding block is the last layer output of the j-1-th encoding block, that is, Therefore, for the jth encoding block, its output is: S23. Extract features of different levels from the output of each encoding block to obtain m original image features [F1, F2, ..., F m ], and m new image features [V1, V2, …, V m ]; S3. The original image features are fine-tuned based on a small number of samples. The converted original image features are obtained, and the similarity between them and the text features is calculated to obtain the defect map A. F ; The new image features and text features are similarly calculated to remove redundant features, and the defect map A is obtained. V ; Defect image A F With A V Add together to get the final defect segmentation map; the specific operation process is as follows: S31. Use the pre-trained CLIP text encoder to convert our designed expert knowledge text guide T into text features t: t = TextEncoder(T); S32. The input SEM image is passed through a customized image encoder to obtain the original image feature F and the new image feature V; S33. For the original image feature F = {f CLS , f 1 , f 2 , ..., f T }, where f CLS As a CLS token, it aggregates the global features of the image for defect classification tasks, and F[1:] = {f 1 , f 2 , ..., f T } contains more detailed information; similarly for new image features, remove the CLS token and use F[1:] and V[1:] for pixel-level defect segmentation; S34. Use the original image features to calculate the defect map and obtain the segmentation result A of the original image features F ; S35. Use the new image features to calculate the defect map and obtain the segmentation result A of the new image features V ; S36. Based on the two obtained image features, the final defect segmentation map is calculated as: A=A F +A V ; S4. Based on the SEM defect image, the original image features are used to classify the defects and the classification results are output.

2. The method for wafer manufacturing process evaluation and anomaly detection assisted by machine learning according to claim 1, characterized in that: In step S1, the prompt is divided into two parts: template level and state level: the state level prompt describes in detail the possible morphology of each type of defect; the template level prompt describes the shooting effect of the SEM image; finally, the state level prompt replaces the {state} in the template level, thereby combining to obtain the final text prompt library with expert prior knowledge.

3. The method for wafer manufacturing process evaluation and anomaly detection assisted by machine learning according to claim 1, characterized in that: In step S34, a conversion layer is added to the original image features and fine-tuned using the SEM sample. Specifically, the conversion layer is implemented by a linear layer, which maps the image features to the joint embedding space. The input of the mapping is [F1[1:], F2[1:], ..., F m [1:]], where the output F of the j-th coding block is j For example, the mapping process is as follows: F′ j =Transformation(F j [1:]); Then the original image feature F after linear transformation j Use cosine similarity to calculate the similarity between it and text feature t. The calculation formula is as follows: The defect map generated by the output of the j-th encoding block guided by the text prompt can be calculated: In the formula, Upsample refers to the upsampling operation, which is then normalized by the softmax layer; Then sum the defect images corresponding to the m original image codes to obtain the segmentation result A of the original image features. F :

4. The machine learning-assisted wafer manufacturing process evaluation and anomaly detection method according to claim 1, characterized in that: In step S35, for the new image feature, the text feature t and the image feature V[1:] are first L2 normalized, and then element-by-element multiplication is performed to generate a fused feature vector containing information of both: Fusion features Taking the average value to get redundant features In the fusion feature Remove redundant features Get the defect image: Similarly, the defect images corresponding to the m new image features V are summed to obtain the final segmentation result A V :

5. The method for wafer manufacturing process evaluation and anomaly detection assisted by machine learning according to claim 1, characterized in that: In step S4, an additional learnable token is used to aggregate information from global features during the image encoding process. It is located at the first encoding position of the original image feature F. The specific process of implementing classification is as follows: S41. Since there are m coding blocks in total, we get the CLS vector for classification based on the m original image features F as follows: S42. Obtaining effective features F C After that, by fine-tuning the classification head, the classification prediction probability P is obtained. C , thereby realizing the defect classification function: F′ C =W·F C +b, P C =Softmax(F′ C ), Where W is the weight matrix of the classifier and b is the bias term.

Images