Training method, server, device, medium and product

Abstract

The invention provides a training method, a server, equipment, a medium and a product, and the method comprises the steps: obtaining the key information of a plurality of teacher models trained by a first server from the first server, and the training data of each teacher model comprises a first group of wafer sample images, training a first pre-trained student model based on the key information of each teacher model and each second group of wafer sample images to obtain each candidate student model, determining a target momentum parameter based on the performance index of each candidate student model and the momentum parameters of the plurality of teacher models, and obtaining a target momentum parameter of the candidate student model based on the candidate student model corresponding to the target momentum parameter. And determining a wafer defect detection model. The wafer defect detection model trained by the method can give full play to the potential adaptive ability, generalization ability, high availability and high reliability of AI, thereby ensuring that the model can adapt to the requirements of the semiconductor manufacturing industry.

Description

Technical Field

[0001] This application relates to the field of semiconductor manufacturing technology, and more particularly to a training method, server, device, medium, and product. Background Technology

[0002] Automated Defect Classification (ADC) systems are widely used in semiconductor manufacturing processes to classify wafer defects, thereby providing support for Defect Management System (DMS) for system management and Yield Management System (YMS) for product analysis.

[0003] In related technologies, a large distributed artificial intelligence (AI) model can be trained based on a large number of wafer images to obtain a wafer defect detection model. However, this wafer defect detection model cannot fully utilize the potential generalization ability, high availability, and high reliability of AI, making it difficult to meet the needs of the semiconductor manufacturing industry. Summary of the Invention

[0004] To achieve the above objectives, this application provides a training method, server, device, medium, and product to enable the wafer defect detection model to fully leverage the potential generalization ability, high availability, and high reliability of AI, thereby ensuring that the wafer defect detection model can meet the needs of the semiconductor manufacturing industry.

[0005] Firstly, this application provides a training method, the method comprising:

[0006] Key information of multiple teacher models trained by the first server is obtained from the first server, wherein the training data of each teacher model includes a first set of wafer sample images;

[0007] Based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the momentum parameters of each teacher model are determined.

[0008] Based on the momentum parameters of each teacher model and each second set of wafer sample images, the corresponding first pre-trained student model is trained to obtain each candidate student model, and each second set of wafer sample images is part of the training data of the corresponding teacher model.

[0009] Based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models, the target momentum parameters are determined.

[0010] Based on the teacher model corresponding to the target momentum parameter, the corresponding candidate student model is obtained, and the wafer defect detection model is determined.

[0011] Using the above technical solution, key information of multiple trained teacher models is obtained. The training data for each teacher model includes a first set of wafer sample images. This teacher model can not only learn various wafer defect details present in the first set of wafer sample images, but also has strong generalization ability and robustness. Therefore, based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the momentum parameter of each teacher model can be determined, so that the correlation between the key information of the teacher model and the key information of the corresponding first pre-trained student model is introduced into the momentum parameter of the teacher model. In this case, when training the corresponding first pre-trained student model based on the momentum parameter of each teacher model and each second set of wafer sample images, the convergence speed and training efficiency of the first pre-trained student model can be adjusted according to the degree of correlation between the key information of the teacher model and the key information of the first pre-trained student model, so that the wafer defect details learned by each teacher model can be transferred to the lightweight candidate student model. In this case, the target momentum parameter can be determined based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models. This process is essentially a process in which the second server obtains the momentum parameter selection process for transferring more teacher model knowledge to the candidate student model through the performance metrics of each candidate student model. Therefore, the knowledge transfer capability of the target momentum parameter is relatively good, and it shows excellent performance on the candidate student model. Based on the candidate student model obtained from the teacher model corresponding to the target momentum parameter, the wafer defect detection model is determined. This ensures that the wafer defect detection model not only fully learns the detailed knowledge of wafer defects, but also has the characteristics of strong generalization ability, high reliability and robustness of the teacher model.

[0012] In some possible implementations, the training method for the teacher model in this application is executed by a first server, while the training method for the wafer defect detection model in this application is executed by a second server. The first server can train multiple teacher models, which are large and require a large amount of training data. However, when the second server trains the wafer defect detection model, the training data required is only a portion of the training data from the teacher models. Therefore, the wafer defect detection model process in this application is not only a process of extracting detailed knowledge of wafer defects from multiple teacher models, but also a process of compressing a large number of teacher models into a lightweight wafer defect detection model.

[0013] In addition, in the technical solution proposed in this application, the second server selects the target momentum parameter with better knowledge transfer ability from the momentum parameters of multiple teacher models based on the performance indicators of each candidate student model, and participates in the determination process of the wafer defect detection model, so as to ensure that the wafer defect detail knowledge learned by the large number of teacher models can be transferred to the lightweight wafer defect detection model.

[0014] As can be seen, in the technical solution proposed in this application, the second server, even with low hardware and software configuration, does not need to train a large number of candidate teacher models. Instead, it directly obtains the key information of the already trained candidate teacher models from the first server and quickly completes the wafer defect detection model through knowledge extraction and model compression. This reduces the model training pressure on the second server, making the wafer defect detection model not only lightweight and easy to deploy, but also with strong generalization ability, high availability, and high reliability, facilitating the execution of inference tasks on low-configuration computer devices.

[0015] In some possible implementations, each first set of wafer sample images includes wafer sample images from at least one process stage in semiconductor manufacturing, and the portions of the first set of wafer sample images included in different first sets belong to different process stages. Therefore, the training data used by the candidate teacher models in this application exhibits certain differences, ensuring that the different candidate teacher models trained by the first server can learn the details of wafer defects at each process stage in semiconductor manufacturing.

[0016] The model weights of each teacher model are positively correlated with the probability of wafer defects in the process stage to which each of the first set of wafer sample images belongs. In this case, the higher the probability of wafer defects in the first set of wafer sample images, the more knowledge from the teacher model is transferred to the candidate student model; conversely, the lower the probability of wafer defects in the first set of wafer sample images, the less knowledge from the teacher model is transferred to the candidate student model. In this scenario, the teacher model can determine the degree to which its key information participates in the training process of the first pre-trained teacher model based on the probability of wafer defects in the selected training data, thereby reducing the problem of overfitting or underfitting in the first pre-trained student model.

[0017] In one possible implementation, multiple teacher models include a target teacher model and non-target teacher models, where the target teacher model is the teacher model corresponding to the target momentum parameter. The method of this application also includes:

[0018] Determine whether the performance indicators of the current wafer defect detection model meet the requirements;

[0019] When the requirements are not met, it indicates that the performance of the wafer defect detection model, based on the target momentum parameters, is not good enough. Therefore, a first weight update instruction is sent. This first weight update instruction instructs the knowledge transfer weight of the target teacher model to be increased, allowing the teacher model corresponding to the target momentum parameters to incorporate more knowledge into the training of the wafer defect detection model, thereby further improving the performance of the wafer defect detection model. For example, when the first server trains the teacher model and the second server trains the wafer defect detection model, the second server can send a first weight update instruction to the first server, instructing the first server to increase the knowledge transfer weight of the target teacher model.

[0020] When the requirements are met, it indicates that the wafer defect detection model determined based on the corresponding candidate student model obtained from the teacher model corresponding to the target momentum parameters has good performance. Therefore, a second weight update instruction is sent. This second weight update instruction is used to instruct the first server to increase the knowledge transfer weight of the non-target teacher model. For example, when the first server trains the teacher model and the second server trains the wafer defect detection model, the second server can send a second weight update instruction to the first server, so that the second weight update instruction is used to instruct the first server to increase the knowledge transfer weight of the non-target teacher model.

[0021] As can be seen, the technical solution proposed in this application can incorporate the knowledge of teacher models that do not correspond to the target momentum parameters into the training of the wafer defect detection model. This allows the wafer defect detection model to learn the knowledge of non-target teacher models after fully learning the knowledge of the target teacher model. Consequently, the wafer defect detection model can fully learn the knowledge of each teacher model and integrate the knowledge of each teacher model into the final lightweight wafer defect detection model. This enables the lightweight wafer defect detection model to effectively detect wafer defects at each process stage in wafer manufacturing.

[0022] In some possible implementations, the momentum parameters of each teacher model are determined based on key information of each teacher model and the corresponding key information of the first pre-trained student model, including:

[0023] Based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the reward parameters of each teacher model are determined.

[0024] Based on the reward parameters of each teacher model, the momentum parameters of each teacher model are determined.

[0025] With the above technical solution, this application can determine the reward parameter of each teacher model based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, so that the reward parameter of each teacher model can reflect the correlation between the key information of the teacher model and the key information of the corresponding first pre-trained student model. Therefore, when determining the momentum parameter of each teacher model based on the reward parameter of each teacher model, the correlation between the key information of the teacher model and the key information of the corresponding first pre-trained student model can be introduced into the momentum parameter of the teacher model, thereby accelerating the convergence of the first pre-trained student model or reducing the oscillation of the first pre-trained student model.

[0026] In some possible implementations, the first pre-trained student model includes a first branch network and a second branch network. Based on the momentum parameters of each teacher model and each second set of wafer sample images, the corresponding first pre-trained student model is trained to obtain each candidate student model, including:

[0027] Data augmentation is performed on each second set of wafer sample images to obtain a second set of wafer sample image pairs;

[0028] Each pair of second-group wafer sample images is input into the first branch network to obtain the projection feature prediction results of the two sets of wafer sample images included in each pair of second-group wafer sample images;

[0029] Each pair of second-group wafer sample images is input into the second branch network to obtain the mapping feature prediction results of the two sets of wafer sample images included in each pair of second-group wafer sample images under the second branch network;

[0030] The loss of the first pre-trained student model is determined based on the mapping feature prediction results and projection feature prediction results under the second branch network. When the loss of the first pre-trained student model meets the convergence condition of the first pre-trained student model, the first pre-trained student model is set as a candidate student model. When the loss of the first pre-trained student model does not meet the convergence condition of the first pre-trained student model, the backpropagation algorithm is used to update the first branch network, and the second branch network is updated based on the momentum parameters of the corresponding teacher model.

[0031] In some possible implementations, the first branch network includes a first prediction network and a feature projection network, and the second branch includes a second prediction network.

[0032] The first prediction network is used to determine the mapping feature prediction result of each second set of wafer sample images under the first branch network based on each of the two sets of wafer sample images included in the second set of wafer sample images;

[0033] The feature projection network is used to determine the projection feature prediction results of each second set of wafer sample images, based on the mapping feature prediction results under the first branch network, for the two sets of wafer sample images included.

[0034] The second prediction network is used to determine the mapping feature prediction result of each second set of wafer sample images under the second branch network, based on the second set of wafer sample images and the two sets of wafer sample images included.

[0035] In some possible implementations, the aforementioned feature projection network includes a batch normalization module and an activation module. The batch normalization module is used to batch normalize the mapping feature prediction results under the first branch network to obtain the batch normalization result corresponding to the two sets of wafer sample images included in each second set of wafer sample images. The activation module is used to activate the batch normalization result to obtain the projection feature prediction result of each second set of wafer sample images included in the two sets of wafer sample images. Since a batch normalization module is added before the activation module, negative samples can be automatically introduced for the first pre-trained student model. Therefore, when training the first pre-trained student model, it is not necessary to specifically provide negative samples for each second set of wafer sample images, thus achieving the purpose of training the first pre-trained student model using positive samples and providing a theoretical basis for the contrastive learning algorithm.

[0036] In some possible implementations, training a corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model further includes:

[0037] Based on the projection feature prediction results of one set of wafer sample images and the mapping feature prediction results of the other set of wafer sample images for each second set of wafer sample images, the loss corresponding to one set of wafer sample images is determined.

[0038] Based on the loss corresponding to the two sets of wafer sample images included in each second set of wafer sample images, the loss of the corresponding first pre-trained student model is determined.

[0039] In some possible implementations, the target momentum parameter is determined based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models, including:

[0040] Based on the performance metrics of each candidate student model, multiple target student models that meet the requirements of the first performance metric are obtained.

[0041] Obtain the target momentum parameters from the momentum parameters of the teacher models corresponding to multiple target student models.

[0042] Using the above technical solution, the method of this application can first select several target student models with better performance by referring to the performance indicators of candidate student models, and then obtain the target momentum parameters from the momentum parameters of the teacher models corresponding to the multiple target school models. This process can not only select the target momentum parameters with the best knowledge transfer ability, but also narrow down the selection range of target momentum parameters and improve the selection efficiency of target momentum parameters.

[0043] In some possible implementations, the aforementioned target momentum parameters include at least two target momentum parameters, and the method of this application further includes:

[0044] If all target momentum parameters correspond to the same candidate student model, a target candidate student model is determined based on the candidate student models corresponding to all target momentum parameters. If at least two target momentum parameters correspond to different candidate student models, and there are no candidate student models trained with momentum parameters from different teacher models, then a third set of wafer sample images is determined based on the training data of the teacher models corresponding to all target momentum parameters, and a first pre-trained student model is trained based on all target momentum parameters and the third set of wafer sample images to obtain the target candidate student model.

[0045] In some possible implementations, the wafer defect detection model is determined based on the target candidate student model obtained from the teacher model corresponding to the target momentum parameter. This may include:

[0046] Using the target candidate student model as the teacher model and the fourth set of wafer sample images as training data, the second pre-trained student model is trained using an iterative knowledge distillation method to obtain the target student model.

[0047] When the target student model satisfies the convergence condition of the target student model, the wafer defect detection model is determined based on the target student model.

[0048] When the second pre-trained student model is trained using an iterative approach, on the one hand, the wafer defect detection model can be further compressed, making it easier to deploy and improving the inference speed of the wafer defect detection model; on the other hand, the prediction results of the iterated target student model can be made closer to the prediction results of the target candidate student model or the prediction results of the old target student model, reducing the discrepancy.

[0049] In one possible implementation, the wafer defect detection model is determined based on the target candidate student model obtained from the teacher model corresponding to the target momentum parameter, including:

[0050] Based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, the momentum parameters of the target candidate student model are determined.

[0051] Using the fourth set of wafer sample images as training data, the knowledge of the target candidate student model is distilled into the corresponding second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model;

[0052] If the target candidate student model meets the convergence condition of the target student model, the wafer defect detection model is determined based on the target student model. If the target candidate student model does not meet the convergence condition of the target student model, the target candidate student model is updated based on the performance index of the target student model if the performance index of the target student model meets the second performance index requirement.

[0053] In employing the above-described technical solution, the present application's solution, during knowledge distillation, references the momentum parameters of the target candidate student model, distilling the knowledge of the target candidate student model into the corresponding second pre-trained student model. This is not simply model compression, but rather filtering out a significant amount of noise from the target candidate student model or the old target student model, transferring more realistic and useful features for the classification task to subsequent target student models, thereby optimizing the wafer defect detection model. Furthermore, each knowledge distillation uses the target student model obtained from the previous knowledge distillation as the teacher model, ensuring that the prediction results of the final wafer defect detection model converge with the prediction results of the target candidate student model or the old target student model, reducing discrepancies.

[0054] In one possible implementation, the fourth set of wafer sample images used for different knowledge distillations includes wafer sample images that belong to at least different process stages in the semiconductor manufacturing process. In this case, wafer defect details from different process stages can be distilled cumulatively into the final target student model. Therefore, when determining the wafer defect detection model based on the target student model, the wafer defect detection model not only has good wafer defect detection capabilities but also good generalization ability, high reliability, strong robustness, and relatively good adaptability.

[0055] In some possible implementations, the momentum parameters of the target candidate student model are determined based on key information of the target candidate student model and the corresponding key information of the second pre-trained student model, including:

[0056] Based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, the reward parameters of the target candidate student model are determined.

[0057] Based on the reward parameters of the target candidate student model, the momentum parameters of the target candidate student model are determined.

[0058] In some possible implementations, using the fourth set of wafer sample images as training data, the knowledge of the target candidate student model is distilled into the corresponding second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model, including:

[0059] Data augmentation is performed on the fourth set of wafer sample images to obtain the fourth set of wafer sample image pairs;

[0060] The fourth set of wafer sample images is input into the target candidate student model to obtain the feature similarity of the target candidate student model at the first distillation temperature.

[0061] Input the fourth set of wafer sample images into the second pre-trained student model to obtain the feature similarity of the second pre-trained student model at the first target distillation temperature and the feature similarity of the second pre-trained student model at the second distillation temperature;

[0062] Based on the feature similarity of the target candidate student model at the first distillation temperature, the feature similarity of the second pre-trained student model at the first target distillation temperature, and the feature similarity of the second pre-trained student model at the second distillation temperature, it is determined whether the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model.

[0063] When the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model, the target student model is determined based on the second pre-trained student model; otherwise, the second pre-trained student model is updated based on the momentum parameters of the target candidate student model.

[0064] In some possible implementations, using the fourth set of wafer sample images as training data, the knowledge of the target candidate student model is distilled into the second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model. Other implementations include:

[0065] The loss of the target candidate student model at the first distillation temperature is determined based on the feature similarity of the target candidate student model at the first distillation temperature and the feature similarity of the second pre-trained student model at the first target distillation temperature.

[0066] Based on the feature similarity of the second pre-trained student model at the second distillation temperature, the loss of the target candidate student model at the second distillation temperature is determined.

[0067] The distillation loss of the second pre-trained student model is determined based on the loss of the target candidate student model at the first distillation temperature and the loss of the target candidate student model at the second distillation temperature.

[0068] In some possible implementations, the aforementioned teacher model is a teacher model based on a contrastive learning algorithm, and the first pre-trained student model and the second pre-trained student model are student models based on the contrastive learning algorithm. In this case, it is not necessary to annotate the first, second, third, and fourth sets of wafer sample images to complete the training of the teacher model and the first pre-trained student model. This not only solves the problem of low annotation efficiency but also addresses the problem of discrepancies and misjudgments in wafer defect annotation caused by differences in the professional level and experience of the annotators.

[0069] Secondly, this application also provides a server, comprising:

[0070] A communication module is used to acquire key information of multiple trained teacher models, wherein the training data of each teacher model includes a first set of wafer sample images;

[0071] The training module is used to determine the momentum parameters of each teacher model based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, and to train the corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model. Each second set of wafer sample images is part of the training data of the corresponding teacher model.

[0072] The determination module is used to determine the target momentum parameter based on the performance indicators of each candidate student model and the momentum parameters of multiple teacher models; and to obtain the corresponding candidate student model based on the teacher model corresponding to the target momentum parameter, thereby determining the wafer defect detection model.

[0073] Thirdly, this application also provides an electronic device, comprising:

[0074] Processor; and,

[0075] Memory for stored programs;

[0076] The program includes instructions that, when executed by a processor, cause the processor to perform the method according to this application.

[0077] Fourthly, this application also provides a server, comprising:

[0078] Processor; and,

[0079] Memory for stored programs;

[0080] The program includes instructions that, when executed by a processor, cause the processor to perform the method according to this application.

[0081] The beneficial effects of the technical solutions provided in the second or fourth aspect of this application are in reference to the beneficial effects of the first aspect or any possible implementation thereof.

[0082] Fifthly, this application also provides a training system, including:

[0083] A first server is used to train a teacher model to be trained based on multiple first group wafer sample images, thereby obtaining multiple teacher models, each of which corresponds to each first group wafer sample image;

[0084] And a second server that communicates with the first server, the second server including the server of the fourth aspect of this application.

[0085] In one possible implementation, the first server is used to perform data augmentation on each first group of wafer sample images to obtain each first group of wafer sample image pairs. Each first group of wafer sample image pairs is input into the corresponding teacher model to be trained to obtain the mapping feature prediction result of each first group of wafer sample image pairs. When the contrastive learning loss of the corresponding teacher model determined based on the mapping feature prediction result of the first group of wafer sample image pairs satisfies the convergence condition of the teacher model to be trained, the teacher model is determined based on the teacher model to be trained.

[0086] In one possible implementation, the aforementioned teacher model to be trained includes a feature extraction network, a feature prediction network, and a feature mapping network.

[0087] The feature extraction network is used to extract features from each pair of first-group wafer sample images to obtain the feature representation of each pair of first-group wafer sample images;

[0088] The feature prediction network is used to determine the feature prediction result for each pair of first-group wafer sample images based on the feature representation of each pair.

[0089] The feature mapping network is used to determine the mapping feature prediction result for each first set of wafer sample image pairs based on the feature prediction result for each first set of wafer sample image pairs.

[0090] The beneficial effects of the technical solution provided by the fifth aspect of this application or any possible implementation thereof shall be in reference to the beneficial effects of the first aspect or any possible implementation thereof.

[0091] In a sixth aspect, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in this application.

[0092] In a seventh aspect, this application also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the method described in this application.

[0093] The beneficial effects of the technical solutions provided in the sixth and seventh aspects of this application are in reference to the beneficial effects of the first aspect or any possible implementation thereof. Attached Figure Description

[0094] In the following description of exemplary embodiments in conjunction with the accompanying drawings, further details, features, and advantages of this application are applied as illustrated in the drawings:

[0095] Figure 1 A schematic diagram of an example system in which the various methods described herein may be implemented according to exemplary embodiments of this application is shown;

[0096] Figure 2A A schematic diagram showing wafer sample images according to an embodiment of this application is provided.

[0097] Figure 2B This paper illustrates a color histogram of a wafer sample image according to an embodiment of this application.

[0098] Figure 3A A schematic diagram showing a comparison of stretched images of wafer samples from embodiments of this application is provided.

[0099] Figure 3B This paper shows a color histogram of a wafer sample comparison stretch map according to an embodiment of the present application;

[0100] Figure 4A A schematic diagram of wafer sample equalization diagram according to an embodiment of this application is shown;

[0101] Figure 4B This paper illustrates a color histogram of a wafer sample equalization map according to an embodiment of this application.

[0102] Figure 5 A flowchart illustrating a training method according to an embodiment of this application is shown;

[0103] Figure 6A This paper presents a schematic diagram showing a comparison of the first batch of wafer sample images before and after data enhancement, according to an embodiment of this application.

[0104] Figure 6B This paper presents a schematic diagram showing a comparison of the data enhancement before and after images of the second batch of wafer samples according to an embodiment of this application.

[0105] Figure 7 This illustration shows a schematic diagram of the training principle architecture of a teacher model according to an embodiment of this application;

[0106] Figure 8A schematic diagram of the structure of a feature prediction network according to an embodiment of this application is shown;

[0107] Figure 9 This paper illustrates a schematic diagram of an example training process for a candidate student model according to an embodiment of this application.

[0108] Figure 10 A schematic diagram illustrating the training principle of a first pre-trained student model according to an embodiment of this application is shown;

[0109] Figure 11 A schematic diagram illustrating the process of determining a target momentum parameter according to an embodiment of this application is shown;

[0110] Figure 12 This paper illustrates an example flowchart of training a wafer defect detection model using an iterative knowledge distillation method according to an embodiment of this application.

[0111] Figure 13 This illustration shows an example flowchart of training a target student model using a target candidate student model, according to an embodiment of this application.

[0112] Figure 14 This application illustrates a schematic diagram of the system principle based on reinforcement learning according to an embodiment of the present application;

[0113] Figure 15 A schematic block diagram of a training apparatus according to an exemplary embodiment of this application is shown;

[0114] Figure 16 A schematic block diagram of a chip according to an exemplary embodiment of this application is shown;

[0115] Figure 17 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of this application is shown. Detailed Implementation

[0117] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0118] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0119] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0120] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0121] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0122] Before introducing the embodiments of this application, the relevant terms involved in the embodiments of this application are first explained as follows:

[0123] Supervised learning is a machine learning method that learns predictive models from labeled data. Essentially, it learns the statistical patterns of the mapping from input to output. (Mapping: the relationship between elements in two sets).

[0124] Self-supervised learning (SSL) is a machine learning method that directly extracts self-supervised information from large-scale unsupervised data for supervised learning and training. It can be seen as a special case of unsupervised learning. Self-supervised learning requires labels, but these labels do not come from manual annotation, but from the data itself.

[0125] Knowledge distillation (KL) is a model compression method, a training approach based on the "teacher-student network" concept. It transfers knowledge from a complex deep learning model (the teacher model) to a simpler model (the student model), enabling the student model to simulate the output of the teacher model.

[0126] Reinforcement learning (RL) is a type of machine learning that uses continuous interaction with the environment and feedback to adjust its behavior in order to maximize cumulative rewards. It addresses the decision-making problem of finding the optimal behavior or behavior probability in the current state.

[0127] Contrastive learning (CL) is a special type of unsupervised learning method that aims to learn data representations by maximizing the similarity between relevant samples and minimizing the similarity between irrelevant samples. It typically uses a highly customizable, high-degree-of-freedom rule to generate positive and negative samples.

[0128] Momentum gradient descent is a method that updates model parameter values ​​by calculating the exponentially weighted average of the gradients. Compared to traditional gradient descent, momentum gradient descent offers a smoother trend, allows for a larger learning rate, and reduces the number of iterations.

[0129] Semiconductor manufacturing processes mainly include deposition, photolithography, etching, ion implantation, and polishing. After each process is completed, the wafer in that process segment can be measured to detect whether there are defects in the wafer in that process segment, thereby ensuring the quality of the final manufactured chip.

[0130] Given the numerous process steps and long processing cycles in semiconductor manufacturing, and the difficulty in manually detecting wafer defects, specialized imaging equipment, such as scanning electron microscopes (SEM) and critical dimension scanning electron microscopes (CD-SEM), can be used to perform microscopic imaging of the entire process flow, obtaining images of the wafers to be inspected. This provides a data source for subsequent analysis and management. For example, images of the wafers to be inspected at each process stage can be used, and then the Automatic Defect Classification (ADC) subsystem within systems such as Defect Management Systems (DMS) and Yield Management Systems (YMS) can be employed to perform defect detection on these images.

[0131] To improve the accuracy and efficiency of ADC systems in classifying wafer defects, cutting-edge deep learning technologies and algorithms are needed. A distributed artificial intelligence (AI) model can be trained using images of the wafer to be inspected as wafer sample images, and the trained model can be used for wafer defect classification inference tasks.

[0132] In related technologies, distributed AI large models can be trained based on machine learning (ML) technology. This can be achieved by manually analyzing image features, manually labeling and classifying a large number of wafer sample images, using the labeled wafer sample images as training source data, extracting features from the wafer sample images using a traditional ML model, and finally classifying the wafer sample image features based on the classifier in the ML model.

[0133] The distributed AI model trained using the aforementioned methods only has a binary classification task concept of positive and negative results, meaning it can only distinguish between defective and defect-free wafers, making it difficult to specifically classify wafer defects. Although this distributed AI model is lightweight, easy to train, and easy to deploy, it cannot fully leverage the potential adaptive, generalization, high availability, and high reliability of AI, making it difficult to adapt to the current digital and intelligent manufacturing transformation of the semiconductor manufacturing industry.

[0134] The second training method is to train a distributed AI large model based on supervised learning deep learning (DP) technology. This method involves manually analyzing image features and manually labeling a large number of wafer sample images. Then, the large number of labeled wafer sample images are used as training source data. The training source dataset is then divided into training set, validation set, and test set according to the proportion. Next, the size of the large number of labeled wafer sample images is scaled to an N×N M-channel image matrix, and a convolutional neural network (CNN) is used to map feature vectors to probabilities, thereby completing the classification task.

[0135] The distributed AI models trained by the above training methods are large in size and difficult to train, requiring deployment in scenarios with high hardware configurations. Therefore, this training method cannot fully leverage the potential adaptive, generalization, high availability, and high reliability of AI, making it difficult to adapt to the current stage of digital and intelligent manufacturing transformation in the semiconductor manufacturing industry.

[0136] To address the aforementioned issues, this application provides a training method that integrates knowledge transfer and reinforcement learning. It extracts wafer defect knowledge from a large teacher model and transfers it to a lightweight student model, thereby achieving the goals of knowledge extraction and model compression. This allows the wafer defect detection model to fully leverage the potential adaptive capabilities, generalization abilities, high availability, and high reliability of AI, ensuring that the wafer defect detection model can meet the needs of the semiconductor manufacturing industry.

[0137] For example, embodiments of this application can also use a lightweight student model as a teacher model, and perform knowledge distillation on the student model through knowledge distillation, thereby ensuring that the wafer defect detection model has strong generalization ability and high robustness.

[0138] Figure 1 A schematic diagram of an example system in which various methods described herein can be implemented according to exemplary embodiments of this application is shown. Figure 1 As shown, an embodiment of this application provides a training system 100 that may include a first server (e.g., Figure 1 The network server 101 shown) and the second server (e.g.) Figure 1 The local server 102 is shown.

[0139] like Figure 1 As shown, the network server 101 in this embodiment can be configured as a server in a computing center. For example, the server may include a single server, a distributed server, or a server cluster, and may belong to cloud servers such as private cloud, public cloud, hybrid cloud, or private cloud. The local server 102 can be deployed on a local server in a wafer fab, and can communicate with the network server 101 through different communication methods.

[0140] like Figure 1As shown, the local server 102 in an exemplary embodiment of this application can be connected to the network server 101 via one or more networks. The network in this embodiment can be any suitable network. By way of example and not limitation, one or more parts of the network may include an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless local area network (WLAN), wide area network (WAN), wireless wide area network (WWAN), metropolitan area network (MAN), a part of the Internet, a part of the public switched telephone network (PSTN), a cellular telephone network, or a combination of two or more of these.

[0141] For example, the communication links of one or more networks may include one or more wired lines (e.g., Digital Subscriber Line (DSL) or Data OverCable Service Interface Specifications (DOCSIS), wireless (e.g., Wireless Fidelity (WiFi) or Worldwide Interoperability for Microwave Access (WiMAX)), or optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links.

[0142] The communication links of one or more networks each include ad hoc networks, intranets, extranets, VPNs, LANs, WLANs, WANs, WWANs, MANs, a portion of the Internet, a portion of the PSTN, cellular-based networks, satellite-based networks, communication links of another network, or combinations of two or more such network communication links. The communication links of the networks are not necessarily identical throughout the searched network architecture. The communication links of one or more networks may differ from the communication links of one or more second networks in one or more aspects.

[0143] like Figure 1 As shown, the local server 102 in this embodiment is deployed in a wafer fab, for example, it can be placed in the wafer fab's server room. The local server 102 can communicate with the imaging device 103. It should be understood that the imaging device can be various imaging devices used for acquiring images of wafers, such as the SEM and CD-SEM mentioned above, but is not limited to these.

[0144] In the semiconductor manufacturing process, imaging equipment 103 in each wafer fab can acquire high-resolution wafer images at different process stages. These different process stages can be small or large. For example, forming a material layer on the wafer can be defined as a process stage. Or, for example, forming photoresist on the wafer and then exposing and developing the photoresist can be defined as a process stage.

[0145] like Figure 1 As shown, the local server 102 in this embodiment can also achieve software-level interconnection with the imaging device 103 through a message middleware, enabling the imaging device 103 to transmit wafer images to the local server 102 via a communication link. Alternatively, the wafer images acquired by the imaging device 103 can be stored in a storage device, and then imported into the local server 102 via the storage device.

[0146] For example: Figure 1 As shown, after the imaging device 103 acquires a wafer image on the machine, it can upload it to the local server 102 via the machine's Network Attached Storage (NAS) disk. The local server 102 stores the acquired wafer image as a wafer sample image in a database (i.e., the storage host address specified by the local server). The database records information such as the distributed identity document (ID) for each wafer image, machine ID, wafer ID, layer ID, step ID, storage path, scan time, and storage entry time, and performs master-slave replication. The ADC system can access the specified slave database through a message middleware to read the wafer image data.

[0147] like Figure 1As shown, when the number of wafer sample images reaches a certain amount, the local server 102 can transmit the wafer sample images to the network server 101 through a message middleware. At the same time, the database of the local server 102 can be cleared to reserve storage space for wafer images acquired by subsequent imaging equipment. For example, the local server 102 can periodically transmit wafer sample images to the network server 101 through the message middleware and clear the database, reducing the data storage pressure on the local server 102.

[0148] like Figure 1 As shown, when the number of wafer images received by the network server 101 reaches the training requirement, these wafer images can be used as wafer sample images to train n teacher models, such as the first teacher model 101a, the second teacher model 101b, ..., the (n-1)th teacher model 101n-1, and the nth teacher model 101n. Different teacher models can be trained using the same wafer sample images or using different sets of wafer sample images.

[0149] like Figure 1 As shown, in this embodiment of the application, the local server 102 can obtain key information of each teacher model from the network server 101 through the message middleware. The local server 102 reads the wafer image required for training from the network server 101 or the network database of cloud storage through the message middleware, and uses it as a wafer sample image and key information of each teacher model to train the student model.

[0150] In practical applications, such as Figure 1 As shown in the embodiment of this application, before the local server 102 saves the wafer image as a wafer sample image to the database, it can perform standardization processing to facilitate subsequent training of the teacher model and student model. Considering that the architectural differences between the teacher model and the student model may also affect the standardization process of the wafer image, the unified standardization processing method of the wafer sample images used by the teacher model and the student model in this embodiment of the application is introduced below.

[0151] Step 1: Standardize wafer sample images of different bit depths and formats into wafer sample images of a preset size and format. For example, a 16-bit Portable Network Graphics (Portable Network Graphics) format wafer sample image can be converted into an 8-bit RGB three-channel Joint Photographic Experts Group (JPEG) format wafer sample image, which facilitates the quantization of feature distribution in the [0, 255] color range during subsequent training.

[0152] The second step is to equalize the color histogram of the three-channel RGB format wafer sample image. For example, contrast stretching and histogram equalization can be performed on the three-channel RGB format wafer sample image to complete the wafer sample image processing.

[0153] Figure 2A A schematic diagram showing wafer sample images according to an embodiment of this application is provided. Figure 2B A color histogram of a wafer sample image according to an embodiment of this application is shown. Figure 2A and Figure 2B As shown, the three-channel RGB format wafer sample images exhibit an uneven distribution problem in the color histogram.

[0154] When the contrast of a three-channel RGB format wafer sample image is stretched, a wafer sample contrast-stretched image is obtained. Figure 3A This diagram illustrates a comparison of stretched wafer samples from embodiments of this application. Figure 3B This illustration shows a color histogram of a contrasting stretched image of a wafer sample according to an embodiment of this application. (See attached image.) Figure 3A and Figure 3B As shown, the uneven distribution of the three-channel RGB format wafer sample image on the color histogram is alleviated after contrast stretching.

[0155] After performing histogram equalization on the contrast stretching results of the wafer sample image, a wafer sample equalization image is obtained. Figure 4A This diagram illustrates a wafer sample balancing pattern according to an embodiment of this application. Figure 4B This diagram illustrates a color histogram of a wafer sample equalization map according to an embodiment of this application. (See attached diagram.) Figure 4A and Figure 4B As shown, after histogram equalization, the wafer sample contrast stretching image is more evenly distributed on the color histogram. Therefore, the wafer sample equalization image processed from each three-channel RGB format wafer sample image can be stored in the database.

[0156] like Figure 1As shown, the local server 102 in this embodiment can also communicate with a low-configuration computer 104. When conditions permit, the low-configuration computer 104 can be adapted to embedded devices, mobile terminals, and other terminals. For example, the terminal can be a mobile phone, tablet computer, wearable device, in-vehicle device, laptop computer, ultra-mobile personal computer (UMPC), netbook, PDA, and wearable devices based on augmented reality (AR) and / or virtual reality (VR) technologies.

[0157] For example, when the terminal is a wearable device, the term "wearable device" can also refer to any device that uses wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices worn directly on the body or integrated into a user's clothing or accessories. Wearable devices are not merely hardware devices; they achieve powerful functions through software support, data interaction, and cloud interaction. Broadly defined, wearable smart devices include those with comprehensive functions, large sizes, and the ability to perform complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those focused on a specific application function that require interaction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0158] like Figure 1 As shown, the local server 102 in this embodiment can use the trained student model as a wafer defect detection model, outputting pre-standardized deep learning model files such as *.pt, *.pth, and *.onnx, and transmit them to the low-configuration computer 104 where the ADC system is located. Simultaneously, various reference information of the wafer defect detection model (such as Top-1, Top-5, MIOU, learning rate chart, loss chart, and local feature visualization images) can be stored in the ADC system as a basis for subsequent analysis and improvement of the production environment model.

[0159] This application embodiment can also use the ADC system to upload a model to classify wafer images in the database to obtain highly professional wafer classification labels (Label-1, ..., Label-n). Professional personnel can then rename these image classification labels to more readable label names, such as wafer defect names. For example, the mapping relationship between wafer classification labels and wafer defect labels can be edited in the ADC system's mapping table (this operation can be performed once during the initialization of the wafer defect detection model).

[0160] When users need to perform defect detection on the wafer, they can upload one or more images of the wafer to be inspected through the ADC system's user interface (UI) and input them into the wafer defect detection model to obtain a highly readable wafer defect name. Because the wafer defect name is easily readable, no professional training is required, and users can readily identify the wafer defect. This reduces the need for specialized user expertise and is convenient for operators without high levels of technical skill.

[0161] The training method provided in this application can be applied to a system having a first server and a second server. Steps executed by the first server can also be executed by a chip applied to the first server; and steps executed by the second server can also be executed by a chip applied to the second server. The following embodiments use the first server and the second server as execution entities, and describe the method of this application in an interactive manner.

[0162] Figure 5 A flowchart illustrating a training method according to an embodiment of this application is shown. Figure 5 As shown, the training method in this application embodiment may include:

[0163] Step 501: The first server trains a teacher model based on multiple first group wafer sample images, obtaining multiple teacher models. Each teacher model corresponds to each first group wafer sample image. That is, each first group wafer sample image can be used to train one teacher model.

[0164] In practical applications, each first set of wafer sample images includes wafer sample images from at least one process stage in semiconductor manufacturing. Each process stage can have one or multiple wafer sample images, which can come from the same wafer fab or from different wafer fabs. For example, wafer sample images can come from twin fabs of the same period or twin fabs and their conventional fabs of different periods.

[0165] When the wafer sample images included in different first groups of wafer sample images belong to different process stages, and the training data used by different teacher models have certain differences, it can be ensured that the different teacher models trained by the network server can learn the details of wafer defects in each process stage of semiconductor manufacturing.

[0166] For example, one first set of wafer sample images in this application embodiment includes wafer sample images of the thin film deposition process stage, and another first set of wafer sample images may include wafer sample images of the wet etching process stage, or may include both wafer sample images of the thin film deposition process stage and wafer sample images of the wet etching process stage.

[0167] As can be seen, the teacher model in this application embodiment has a complex model architecture. By training a large number of teacher models to be trained using different first group wafer sample images, the scale of the teacher model is expanded, so that the obtained teacher model can learn deeper and more detailed features of wafer defects, providing the teacher model with powerful expressive and reasoning capabilities.

[0168] Step 502: The second server obtains key information of multiple teacher models trained from the first server, and the training data of each teacher model includes the first set of wafer sample images.

[0169] In the technical solution claimed in this application embodiment, when the second server sends a request to obtain key information to the first server through a message middleware, the first server can retrieve the key information of the relevant teacher model from the storage space and transmit it to the first server through the message middleware, so as to facilitate the first server to train the first pre-trained student model using the key information of different teacher models.

[0170] Step 503: The second server determines the momentum parameters of each teacher model based on the momentum parameters of each teacher model and the key information of the corresponding first pre-trained student model.

[0171] In the technical solution claimed in this application, there is a one-to-one correspondence between the teacher model and the first pre-trained student model. For example, the second server can select a student model to be trained. This student model can be an untrained lightweight model or a wafer defect detection model trained by the method of this application embodiment. Then, it obtains partial training data from the training data of a certain teacher model to form a second set of wafer sample images. The second set of wafer sample images is then used to pre-train the student model to be trained (e.g., complete one training generation) to obtain the first pre-trained student model. In this case, the teacher model corresponding to the first pre-trained student model can provide training data for the first pre-trained student model.

[0172] In addition, the training data used by the first pre-trained student model during the pre-training process can be wafer sample images read from the local database corresponding to the first server, or wafer sample images obtained from the server of the first server.

[0173] For example, the second server determines the reward parameters of each teacher model based on the key information of each teacher model and the corresponding key information of the first pre-trained student model, and then determines the momentum parameters of each teacher model based on the reward parameters of each teacher model.

[0174] When determining the reward parameter of each teacher model based on the key information of each teacher model and the key information of the first pre-trained student model, the magnitude of the reward parameter of each teacher model can reflect the degree of correlation between the key information of the teacher model and the key information of the first pre-trained student model (hereinafter referred to as the teacher model-student model correlation). Therefore, when determining the momentum parameter of each teacher model based on the reward parameter of each teacher model, the teacher model-student model correlation can be incorporated into the momentum parameter of that teacher model.

[0175] Step 504: The second server trains the corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images, obtaining each candidate student model. It should be understood that for each candidate student model, there is a one-to-one correspondence between the candidate student model, the teacher model, and the second set of wafer sample images.

[0176] The second server in this application embodiment trains the corresponding first pre-trained student model based on the momentum parameters of the teacher model and the second set of wafer sample images. Essentially, it can be regarded as inputting the second set of wafer sample images into the first pre-trained student model, obtaining the mapping feature prediction results of the second set of wafer sample images, and then using the momentum parameters of the teacher model as hyperparameters to update the first pre-trained student model using a gradient update method.

[0177] In practical applications, when a candidate student model corresponds to a second set of wafer sample images, the second set of wafer sample images can be part of the training data for the candidate student model. Therefore, when a candidate student model corresponds to a teacher model, the key information of the teacher model can be used as reference information, and the first pre-trained student model can be trained based on the second set of wafer sample images to obtain the candidate student model corresponding to the teacher model.

[0178] When a teacher model corresponds to a second set of wafer sample images, the second set of wafer sample images corresponding to the teacher model can be a portion of the training data of the corresponding teacher model. Using this second set of wafer sample images to train a first pre-trained student model can yield candidate teacher models corresponding to the second set of wafer sample images. For example, the second server can create multiple threads, with each thread running the training of the candidate student model. Here, the second set of wafer sample images can be a portion of the training data corresponding to the teacher model. For example, in this embodiment, when training n teacher models, all wafer sample images can be used as training data, and this training data can be divided into n parts, with each part used to train one teacher model. The first pre-trained student model can use a portion of the teacher model's training data for training. Therefore, in this embodiment, the second server can transmit a training data acquisition request to the first server via a message middleware before training the first pre-trained student model. The first server can respond to the training data acquisition request by sending a portion of the training data of each teacher model to the first server via the message middleware.

[0179] When training the first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images, the convergence speed and training efficiency of the first pre-trained student model can be adjusted according to the degree of correlation between the key information of the teacher models and the key information of the first pre-trained student model, thereby improving the training speed of the wafer defect detection model. Therefore, in this embodiment, when the key information of multiple teacher models participates in the training of the first pre-trained student model, the detailed knowledge of wafer defects learned by each teacher model can be transferred to the lightweight candidate student model.

[0180] Considering the long processing cycle of semiconductor wafer manufacturing, involving thousands of steps, and the diverse sources of defects including equipment, materials, and environment, the defect characteristics of wafer images will not be the same in different process flows and production environments. Therefore, this application embodiment can perform knowledge distillation on multiple teacher models trained by the first server, thereby transferring the detailed knowledge of wafer defects learned by multiple teacher networks into a lightweight model. Based on this idea, the method of this application embodiment may further include the following steps.

[0181] Step 505: The second server determines the target momentum parameters based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models. It should be understood that the performance metrics of the candidate student models in this embodiment may include Top-1 accuracy, Top-5 accuracy, Mean Intersection over Union (MIoU), learning rate, and loss, etc.

[0182] The process of determining the target momentum parameter in this application embodiment is essentially a process in which the second server obtains the momentum parameter selection process for transferring more teacher model knowledge to the candidate student model through the performance indicators of each candidate student model. Therefore, the knowledge transfer capability of the target momentum parameter is relatively good, and it shows excellent performance on the candidate student model.

[0183] Step 506: The second server determines the wafer defect detection model based on the target candidate student model obtained from the teacher model corresponding to the target momentum parameter. Here, because the target momentum parameter has good knowledge transfer capability and exhibits excellent performance on the candidate student model, using the corresponding candidate student model obtained from the teacher model corresponding to the target momentum parameter as a basis ensures that the wafer defect detection model not only fully learns the detailed knowledge of wafer defects but also possesses the strong generalization ability, high reliability, and robustness of the teacher model.

[0184] In practical applications, the technical solution claimed in this application can have one or at least two target momentum parameters that can be transferred to more teacher model knowledge. When there are at least two target momentum parameters, the target momentum can be determined based on the at least two target momentum parameters, and then the wafer defect detection model can be determined based on the target momentum and the first pre-trained student model. Here, when determining the target momentum based on two target momentum parameters, the two target momentum parameters can be weighted to obtain the target momentum. Of course, other methods can also be used to determine it.

[0185] Furthermore, based on the performance metrics of each candidate student model, the second server in this application selects the target momentum parameter with better knowledge transfer capability from the momentum parameters of multiple teacher models to participate in the determination process of the wafer defect detection model. This ensures that the detailed knowledge of wafer defects learned by the large-scale teacher model can be transferred to the lightweight wafer defect detection model. This process is not only a process of extracting detailed knowledge of wafer defects from multiple teacher models, but also a process of compressing the large-scale teacher model into a lightweight wafer defect detection model.

[0186] As can be seen, the second server in this application, even with low hardware and software configuration, does not need to train a large number of candidate teacher models. Instead, it directly obtains the key information of the already trained candidate teacher models from the first server and quickly completes the wafer defect detection model through knowledge extraction and model compression. This reduces the model training pressure on the second server, making the wafer defect detection model not only lightweight and easy to deploy, but also with strong generalization ability, high availability, and high reliability, making it convenient to perform inference tasks on low-configuration computer devices.

[0187] Furthermore, when the first server is a network server within a computing center or data center with high hardware and software configuration, and the second server is a local server with lower hardware and software performance, such as a server in a wafer fab's server room, the local server in this embodiment can upload all wafer sample images to the network server's storage space via middleware. The network server has a huge storage capacity, capable of storing a large number of wafer sample images that consume significant storage space, for the network server to perform the training task of the teacher model. A smaller portion of the image data is stored in the local server's storage space; for example, a small portion of the training data can be stored in the storage space of a local server in a local wafer fab's server room for the training task of the student model.

[0188] As can be seen, the method in this embodiment does not require using all training data to train the wafer defect detection model. Instead, through knowledge extraction and model compression, the second server can train the wafer defect detection model using only a portion of the training data, thereby solving the problem of high storage space consumption by training data. For example, the teacher model is trained using 4096 batches and trillions of parameters. After knowledge transfer and extraction, the student model only needs 512 batches of wafer sample images and hundreds of millions of parameters. Therefore, the training data used by the student model accounts for only 12.5% ​​of the total training data of the teacher model, thus significantly saving local storage space and reducing floating point operations per second (FLOPs).

[0189] In one alternative approach, to fully learn the knowledge of different teacher models, when multiple teacher models include a target teacher model and non-target teacher models, the target teacher model can be the teacher model corresponding to the target momentum parameter. There can be one or more target teacher models, and there can be one or more non-target teacher models. The training method in this embodiment may further include the following steps:

[0190] Step 507: The second server determines whether the performance metrics of the wafer defect detection model meet the requirements. The performance metrics of the wafer defect detection model here can include Top-1 accuracy, Top-5 accuracy, Mean Intersection over Union (MIoU), learning rate, and loss. The requirements here can be considered the performance requirements of the wafer defect detection model, which can be represented by preset Top-1 accuracy, preset Top-5 accuracy, preset MIoU, preset learning rate, and preset loss.

[0191] If the performance indicators of the wafer defect detection model do not meet the requirements, it means that the performance of the wafer defect detection model determined based on the target momentum parameters is not good enough. Therefore, steps 508 to 509 can be executed.

[0192] If the performance indicators of the wafer defect detection model meet the requirements, it means that the performance of the wafer defect detection model determined based on the target momentum parameter is already very good. Therefore, steps 510 to 511 can be executed.

[0193] In practical applications, the performance indicators of a wafer defect detection model can be designed according to actual needs, with a performance range for each indicator. Then, it is determined whether each performance indicator of the wafer defect detection model falls within its corresponding range. If a performance indicator exceeds its range, then that performance indicator of the wafer defect detection model is considered to fail to meet the corresponding performance requirement. When all performance indicators of the wafer defect detection model meet the corresponding requirements, or when a specified performance indicator meets the corresponding performance requirement, the wafer defect detection model can be considered to meet the requirements.

[0194] Step 508: The second server sends a first weight update instruction to the first server. For example, the first server can send the first weight update instruction to the first server via a message intermediary. Here, the first weight update instruction is used to instruct the first server to increase the knowledge transfer weight of the target teacher model, so that the target teacher model corresponding to the target momentum parameter can introduce more knowledge into the training of the wafer defect detection model.

[0195] Step 509: The first server responds to the first weight update instruction by increasing the knowledge transfer weights of the target teacher model. In this case, more knowledge can be introduced into the training of the wafer defect detection model, thereby further improving the performance of the wafer defect detection model.

[0196] For example, the knowledge transfer weights of the target teacher model can be changed from their current value to increase according to a certain update step size. If the knowledge transfer weights of the target teacher model in the current training period t of the wafer defect detection model are... If the update step size is Δ, then the increased knowledge transfer weight of the target teacher model is the knowledge transfer weight of the target teacher model in the next training cycle t+1 of the wafer defect detection model.

[0197] Step 510: The second server sends a second weight update instruction to the first server. For example, the second server can send the second weight update instruction to the first server via a message intermediary.

[0198] Step 511: The first server responds to the second weight update instruction by increasing the knowledge transfer weight of the non-target teacher models. Here, knowledge from teacher models whose momentum parameters do not correspond to the target model is added to the training of the wafer defect detection model. This allows the wafer defect detection model to learn from the non-target teacher models after fully learning the knowledge of the target teacher model. This enables the wafer defect detection model to fully learn the knowledge of each teacher model, concentrating the knowledge of each teacher model into the final lightweight wafer defect detection model. This lightweight wafer defect detection model can effectively detect wafer defects at various stages of the wafer manufacturing process.

[0199] Of course, the first server in this application embodiment can also, based on step 511, respond to the second weight update instruction to reduce the knowledge transfer weight of the target teacher model and reduce the knowledge transfer amount of the target teacher model.

[0200] For example: when the knowledge transfer weights of the non-target teacher model are changed from their current value and increased by a certain update step, if the knowledge transfer weights of the non-target teacher model in the current training period t of the wafer defect detection model are... If the update step size is Δ, then the increased knowledge transfer weight of the non-target teacher model is the knowledge transfer weight of the non-target teacher model in the next training cycle t+1 of the wafer defect detection model. The reduced knowledge transfer weights of the target teacher model, i.e., the knowledge transfer weights of the target teacher model in the next training cycle t+1 of the wafer defect detection model.

[0201] As one possible implementation, in the embodiments of this application, when training the teacher model, the wafer sample images can be divided according to the data preprocessing described above, in the following manner. For example, a large number of wafer sample images can be divided into training and test sets at a ratio (e.g., 10:1), and then the data can be labeled by relevant professionals on the ADC system interface as needed.

[0202] For example, when training a teacher model using a self-supervised approach, a small number of classification labels can be added to the test set. These labels can be in the form of Label1, Label2, ..., Labeln. Compared to supervised learning, this significantly reduces the number of labels required (saving 90% of storage space compared to traditional deep learning). This labeled data is used for automatic verification during subsequent teacher model training and to improve the accuracy of features extracted during the testing phase. This greatly reduces the probability of misjudgments caused by human factors and the workload, thus improving efficiency.

[0203] The first server in this embodiment can be used to perform data augmentation on each first group of wafer sample images to obtain each first group of wafer sample image pairs, and then train a teacher model based on each first group of wafer sample image pairs to obtain each teacher model. The two groups of wafer sample images included in the first group of image sample image pairs can be wafer sample images obtained after augmentation of the first group of wafer sample images.

[0204] In one alternative approach, the teacher model of this application embodiment uses the first server to perform data augmentation on each first group of wafer sample images to obtain each first group of wafer sample image pairs. Each first group of wafer sample image pairs is input into the corresponding teacher model to be trained to obtain the mapping feature prediction results of the two groups of wafer sample images included in each first group of wafer sample image pairs. When the loss of the corresponding teacher model determined based on the mapping feature prediction results of the two groups of wafer sample images included in the first group of wafer sample image pairs satisfies the convergence condition of the teacher model to be trained, the teacher model is determined based on the teacher model to be trained.

[0205] For example, the first server in this application embodiment can use a supervised learning model of a lightweight pre-trained network, such as Mobilenet or Xception, to perform data augmentation on each first wafer sample image. Of course, various image filters can also be used to perform one or more data augmentations on each first group of wafer sample images.

[0206] When various image filters are used to augment the wafer sample images included in each first group of wafer sample images, the data augmentation methods include, but are not limited to, random cropping at the original ratio, random color distortions, random Gaussian blur, random noise, random Roberts / Sobel filtering, edge enhancement, and other data augmentation methods. Since this embodiment of the application can use a combination of multiple large-scale data augmentation methods to augment the first group of wafer sample images, although this increases the training difficulty of the teacher model, it can also fully utilize the potential of the teacher model to learn larger and deeper pixel semantic features in the first wafer sample images.

[0207] This application uses two batches of wafer sample images as examples for illustration, in conjunction with the accompanying drawings. Figure 6AThis diagram illustrates a comparison of the first batch of wafer sample images before and after data enhancement, representing an embodiment of this application. Figure 6B This illustration shows a before-and-after comparison of data enhancement for the second batch of wafer sample images according to an embodiment of this application.

[0208] like Figure 6A As shown, performing two data augmentations on the first batch of wafer sample images x yields two enhanced images: First Batch Wafer Sample Enhanced Image 1 x1 and First Batch Wafer Sample Enhanced Image 2 x2. For example, data augmentation can be performed on the first batch of wafer sample images x using a Roberts filter and a Sobel filter to obtain First Batch Wafer Sample Enhanced Image 1 x1, and then data augmentation can be performed on the first batch of wafer sample images x using a Sobel filter and random flipping to obtain First Batch Wafer Sample Enhanced Image 2 x2.

[0209] like Figure 6B As shown, performing two data augmentations on the second batch of wafer sample images y yields two augmented images y1 and y2 for the first batch of wafer samples. For example, data augmentation of the second batch of wafer sample images y can be performed using a Roberts filter and a Sobel filter to obtain the first augmented image y1, and data augmentation of the second batch of wafer sample images y can be performed using a Sobel filter and random flipping to obtain the second augmented image y2.

[0210] In one alternative approach, the architecture of the teacher model to be trained in this application embodiment can be a supervised learning model, such as a Transformer-based supervised learning model, a DeepLab inverted pyramid structure-based supervised learning model, a YOLO-based object detection supervised learning model, or a Diffusion-based supervised learning model, etc. It can also be a self-supervised learning teacher model, such as a teacher model to be trained based on a contrastive learning algorithm.

[0211] The following description uses a contrastive learning algorithm as an example to illustrate the architecture of the teacher model to be trained in this embodiment of the application, thereby demonstrating the training process of the teacher model. It should be understood that the training process of the teacher model described below is for illustrative purposes only and is not intended to be limiting.

[0212] Figure 7 This diagram illustrates the training principle architecture of a teacher model according to an embodiment of this application. Figure 7 As shown, the teacher model to be trained in this embodiment may include a feature extraction network 701, a feature prediction network 702, and a feature mapping network 703. It should be understood that... Figure 7The teacher model structure involved in the training process, including feature extraction network 701, feature prediction network 702, and feature mapping network 703, all appear in pairs. This is primarily to illustrate the training principle and does not imply that feature extraction network 701, feature prediction network 702, and feature mapping network 703 must necessarily appear in pairs. For example, for each batch of wafer sample image pairs, one feature extraction network 701, one feature prediction network 702, and one feature mapping network 703 can be used in parallel through parameter sharing to achieve comparative learning training.

[0213] like Figure 7 As shown, the feature extraction network 701 is used to extract features from each first pair of wafer sample images to obtain a feature representation of each first pair of wafer sample images. For example, the feature extraction network 701 may include a backbone network 7011 and a convolutional neural network 7012. The backbone network 7011 is used to extract features from the two sets of wafer sample images included in each first pair of wafer sample images to obtain the features of the two sets of wafer sample images included in each first pair of wafer sample images. The convolutional neural network 7012 is mainly used to extract features based on the feature representations of the two sets of wafer sample images included in each first pair of wafer sample images, i.e., the backbone feature map of each first pair of wafer sample images. The backbone network 7011 may be a residual network such as ResNet-50, ResNet-101, or ResNet-152, or a backbone network with other structures.

[0214] like Figure 7 As shown, the first set of wafer sample images includes multiple wafer sample images. Taking a single wafer sample image as an example, data augmentation of this wafer sample image x can form a pair of wafer sample images, which are named the first wafer sample image x. i Second wafer sample image x j The first wafer sample image x i Second wafer sample image x j Sample pairs can be formed into a training batch.

[0215] like Figure 7 As shown, the first wafer sample image x i Second wafer sample image x j By inputting the feature extraction network 701, the first wafer sample image x can be obtained. i First main feature map f i Second wafer sample image x j Second main feature map f jThen, the feature prediction model is used for downstream tasks to achieve image classification. When the first set of wafer sample images includes 4096 batches of wafer sample images, then backbone feature maps of 4096 batches of wafer sample image pairs will be generated.

[0216] During neural network training, the problems of vanishing and exploding gradients are common. Vanishing gradients occur when the gradient of the weights updated during backpropagation shrinks to zero too quickly as the neural network depth increases. Exploding gradients occur when the gradient of the weights updated during backpropagation increases too rapidly and excessively as the neural network depth increases. Extracting the backbone features of wafer sample images using a residual network can reduce the likelihood of vanishing gradients.

[0217] When training a neural network, using a residual network to extract the backbone features of wafer sample images can prevent degradation problems. That is, as the depth of the neural network increases, the training error actually increases. Even with adaptive learning rate adjustment (LR) and a non-linear loss function, the teacher model still cannot fit the final feature map.

[0218] like Figure 7 As shown, the feature prediction network 702 in this embodiment can be used to determine the feature prediction results of the two sets of wafer sample images included in each first set of wafer sample image pairs based on the feature representation of each first set of wafer sample image pairs. This feature prediction network 702 can implement image classification tasks, and its model architecture can include an encoder 7021 and a decoder 7022. The encoder 7021 can provide the decoder 7022 with shallow effective feature layers and deep effective feature layers. It should be understood that the division between shallow and deep layers here is based on the scale of the feature layers.

[0219] like Figure 7As shown, the encoder 7021 in this embodiment can be a Deep Convolutional Neural Network (DCNN) or an Atrous Spatial Pyramid Pooling (ASPP) network, as long as it can provide two effective feature layers for the decoder 7022. These two effective feature layers can include shallow effective features and deep effective features. Compared to the deep effective feature layer, the shallow effective feature layer has a larger height and width, and can retain more features such as color, shape, and texture of the wafer sample image. The deep effective feature layer performs more operations on the basis of the shallow effective features, and has a smaller height and width. It performs more downsampling, and has a smaller height and width, and contains rich high-level semantic feature representations (i.e., abstract features). Based on this, the deep effective feature layer and the shallow effective feature layer are used as inputs to the decoder 7022. The decoder 7022 can combine the shallow and deep features of the wafer sample image to ensure that the decoder 7022 accurately outputs the feature prediction results of the first set of wafer sample image pairs. For example, as Figure 7 As shown, for the first main feature map f i The feature prediction network 702 can be based on the first backbone feature map f i Obtain the first deep feature map h i For the second main feature map f j The feature prediction network 702 can be based on the second backbone feature map f j Obtain the second deep feature map h j .

[0220] Figure 8 A schematic diagram of the structure of a feature prediction network according to an embodiment of this application is shown. Figure 8 As shown, the feature prediction network 800 of this application embodiment may include an encoder 801 and a decoder 802. The encoder 801 can provide the decoder 802 with a shallow effective feature layer and a deep effective feature layer, and the decoder 802 can determine the feature prediction result for each first set of wafer sample image pairs based on the shallow effective feature layer and the deep effective feature layer.

[0221] For example, such as Figure 8 As shown, the encoder 801 described above may include a deep convolutional neural network (DCNN) and a depthwise separable convolutional module. The DCNN can be used to process each backbone feature map (the first backbone feature map f). i Second main trunk feature map f jFeature extraction is performed to obtain the shallow and deep effective feature layers of the RGB three channels. The shallow effective feature layer can be fed into the decoder 802 for relevant feature extraction, while the deep effective feature layer can be extracted through a depthwise separable convolution module for multi-scale feature extraction. Then, the extracted multi-scale features are concatenated and the number of channels is adjusted using a 1×1 Conv module to obtain the deep effective feature layer.

[0222] For example, depthwise separable convolution can perform multiple operations in parallel on deep effective feature layers. For instance, the number of channels in the shallow effective feature layer can be adjusted using 1x1Conv, thereby reducing the dimensionality of the shallow effective feature layer with three RGB channels to a one-channel grayscale image. This one-channel grayscale image is essentially a pixel-level feature layer of the shallow effective feature layer. It can also extract feature maps of the shallow effective feature layer in different receptive fields by using three dilation convolutions with different dilation rates in parallel. For example, dilation convolutions with different dilation rates (rate = 6, 12, or 18) (with 3×3 kernels) can be performed on the shallow effective feature layer to obtain three different receptive field feature maps. At the same time, average pooling can be performed on the shallow effective feature layer to obtain a pooled feature layer.

[0223] A high-dimensional feature layer is obtained by stacking a single-channel grayscale image, three different receptive field feature maps, and a pooling feature layer. Then, a 1×1 convolution is performed to adjust the number of channels in the high-dimensional feature layer, resulting in a deeper effective feature layer than the shallow effective feature layer. Since the depthwise separable convolution module performs multi-scale feature extraction on the deep effective feature layer, and multi-scale features have different receptive fields, they contain richer semantic information. Therefore, after extraction of the deep effective feature layer through the Ineception network, this deep effective feature layer can provide the decoder with richer semantic information, facilitating feature prediction.

[0224] like Figure 8 As shown, the decoder 802 can perform a 4x upsampling operation on the deep effective feature layer to increase its scale. Simultaneously, the decoder 802 can also perform a 1×1 convolution on the shallow effective feature layer to adjust the number of channels in the shallow effective feature layer so that the number of channels in the shallow effective feature layer is the same as that in the deep effective feature layer. Then, the shallow and deep effective feature layers are fused to obtain a fused feature layer. Subsequently, a 3×3 convolution and a 4x upsampling operation are performed on the fused feature layer to obtain a deep effective feature map with the same scale as the input backbone features, which is used as the final feature prediction result.

[0225] The deep feature map used to represent the feature prediction result in this application embodiment can reflect the feature classification prediction result of each pixel in the input wafer sample image, which essentially enhances the semantics of the feature map. When the number of wafer sample images is 4096, after data augmentation, 4096 pairs of wafer sample images are generated, and the 4096 pairs of wafer sample images are processed as follows: Figure 7 The feature extraction network 701 and feature prediction network 702 shown can output 4096 pairs of deep feature maps (h i h j ).

[0226] like Figure 7 As shown, the feature mapping network 703 in this embodiment can be used to determine the mapping feature prediction result of each first set of wafer sample images on the two sets of included wafer sample images based on the feature prediction results of each first set of wafer sample images on the two sets of included wafer sample images. This process can essentially be viewed as converting pixel-level two-dimensional feature maps into pixel-level one-dimensional feature vectors to achieve pixel-level classification. Based on the candidate student model trained in this way, after determining the wafer defect detection model, it can be ensured that the wafer defect detection model can accurately detect small-sized fine structures in wafer images, reducing the possibility of misjudgment.

[0227] For example, such as Figure 7 As shown, the feature mapping network 703 in this embodiment may include a multi-layer projection head, and each feature mapping head can be regarded as a multilayer perceptron (MLP). By inputting the feature prediction results of the two sets of wafer sample images included in the first set of wafer sample image pairs into the multi-layer projection head, the feature vectors of the two sets of wafer sample images included in the first set of wafer sample image pairs can be obtained.

[0228] The multi-head projector mentioned above can have three layers, or fewer or more layers than three. The main purpose is to deepen the network depth of the teacher model being trained, so as to improve the quality of the feature map obtained by the previous layer, remove useless pixel features, such as feature direction and color, and reduce the two-dimensional feature map to one-dimensional feature vector through dimensionality reduction processing, retaining only the deep feature vectors that are useful for subsequent operations, thereby reducing the burden on the model.

[0229] For example: Figure 7 As shown, when the first wafer sample image x i Second wafer sample image x j When the teacher model to be trained is input, the teacher model can extract the first wafer sample image x through the feature extraction network 701, the feature prediction network 702, and the feature mapping network 703. iThe first mapping eigenvector z i Second wafer sample image x j The second mapping feature vector z j .

[0230] In this embodiment, when training the teacher model to be trained, a self-adjusting learner can be used to adaptively adjust the learning rate, thereby strengthening backpropagation and enhancing the adaptive fine-tuning of the teacher model. However, self-supervised deep learning models based on contrastive learning lack labeled data, therefore, the cross-entropy loss (CE loss) function cannot be used. Based on this, this embodiment can use a contrastive loss function to determine the loss of the teacher model to be trained, maximizing the similarity between positive samples and minimizing the similarity between positive and negative samples.

[0231] When calculating the contrastive loss, the positive sample pair loss function shown in Equation 1 can be used first to obtain the positive sample pair loss l. i,j Then, according to Equation 2, the arithmetic mean of the losses of all positive sample pairs in the same batch is taken as the final contrast loss L of the teacher model.

[0232]

[0233] In this process, data augmentation of the same wafer sample image can create two wafer sample images, which can form a positive sample pair. i Let z represent the eigenvector of the first mapping. i Let z represent the second mapping feature vector, τ represent the positive sample temperature, τ' represent the negative sample temperature, k represent the sequence number of the wafer sample image, N represent the total number of wafer sample images in a training batch, and z k This represents a negative sample of the first mapping feature vector.

[0234]

[0235] Where l represents the loss for any positive sample pair.

[0236] The embodiments of this application also use... Figure 6A and Figure 6B For example, Table 1 shows a combination table of different samples from embodiments of this application. When using... Figure 6A and Figure 6BWhen constructing sample pairs from the data augmentation results of the two batches of wafer sample images shown, sample pairs from different batches can be negative samples of each other, i.e., (x1, y1), (x2, y1), (x1, y2), (x2, y2) are negative samples of each other. Of course, in addition, the same wafer sample image can also be a negative sample pair after two data augmentations, for example: (x1, x2), (y1, y1).

[0237] The embodiments of this application can be used for... Figure 6A and Figure 6B Data augmentation was performed on the two batches of wafer sample images shown. Then, all the augmented wafer sample images were freely combined to determine the 16 pairs of samples shown in Table 1. These include 4 pairs of positive samples and 12 pairs of negative samples (wafer sample image pairs generated by data augmentation of the same wafer sample image can be considered as negative samples to some extent, and also as positive samples to some extent). Therefore, the positive sample temperature τ∈(0,1) and the negative sample temperature τ'∈(0,1) in the loss function of the positive sample pair shown in Equation 1 are within the interval to avoid overfitting. To identify positive and negative samples, cosine similarity can be used to judge each of the 16 pairs of samples, achieving the goal of grouping similar samples together and separating dissimilar samples.

[0238] Table 1. Combination table of different samples

[0239] / <![CDATA[x1]]> <![CDATA[x2]]> <![CDATA[y1]]> <![CDATA[y2]]> <![CDATA[x1]]> <![CDATA[(x1,x1)]]> <![CDATA[(x1,x2)]]> <![CDATA[(x1,y1)]]> <![CDATA[(x1,y2)]]> <![CDATA[x2]]> <![CDATA[(x2,x1)]]> <![CDATA[(x2,x2)]]> <![CDATA[(x2,x1)]]> <![CDATA[(x2,y2)]]> <![CDATA[y1]]> <![CDATA[(y1,x1)]]> <![CDATA[(y1,x2)]]> <![CDATA[(y1,y1)]]> <![CDATA[(y1,y2)]]> <![CDATA[y2]]> <![CDATA[(y2,x1)]]> <![CDATA[(y2,x2)]]> <![CDATA[(y2,y1)]]> <![CDATA[(y2,y2)]]>

[0240] As can be seen, the embodiments of this application can perform data augmentation on each wafer sample image without manual labeling of negative samples, forming wafer sample image pairs, which can introduce a large number of negative samples for model training. Combining with Equation 1, it can be seen that when the number of negative samples is particularly large, it is equivalent to increasing the denominator of Equation 1, making the loss value of positive sample pairs relatively small. Conversely, when the loss value of positive sample pairs is relatively small, the contrastive loss L of the teacher model is correspondingly smaller. Therefore, data augmentation can reduce overfitting of the teacher model under training, i.e., prevent the pre-trained teacher model from learning too many similar low-semantic shallow features. As can be seen, when the teacher model to be trained is a teacher model based on the contrastive learning algorithm, a large number of samples containing positive and negative samples can be automatically created for each batch of wafer sample images through data augmentation. Then, positive and negative sample pairs are directly constructed by comparing the positive and negative samples between wafer sample images of different batches. Finally, the feature distribution probability is obtained through parallel deep convolution operation and mapped to the label probability. Therefore, the embodiments of this application generate a large number of wafer sample images through data augmentation and combine the large number of wafer sample images to form wafer sample image pairs, thereby expanding the diversity of training data. This helps to improve the robustness and generalization ability of the model and reduce the risk of overfitting and underfitting during model training.

[0241] Considering the extremely large batch sizes and numerous convolutional layers in deep learning algorithms, which easily lead to internal covariate shift (ICS) problems, this embodiment of the application can uniformly add a batch normalization (BN) layer to the teacher model to be trained, normalizing the data in terms of channel count, feature map height, and feature map width. Simultaneously, the BN layer can also normalize the corresponding channels in the batched data and optimize the mean and variance of each convolutional layer.

[0242] During the adaptive adjustment phase of the teacher model to be trained, a strategy of freezing the Batch Normalization (BN) layer can be adopted to reduce the impact of batch size on the adaptive adjustment phase. Additionally, embodiments of this application can place the BN layer after the activation layer to ensure the nonlinear expressive power of the feature vector output by the first mapping network.

[0243] Considering that the teacher model to be trained in this embodiment is a self-supervised teacher model, which can be trained on servers located in data centers with strong computing power, the learning rate of the teacher model to be trained can be adaptively adjusted using a Layer-wise Adaptive Moments optimizer for Batch training (LAMB optmizer) in the multi-head mapper. This reduces the possibility that the teacher model to be trained learns too quickly or too slowly during training. In this way, the teacher model to be trained can perform gradient propagation in a non-linear manner, learn feature values, and achieve global convergence.

[0244] For example, in the embodiments of this application, when training the teacher model, the convergence condition of the teacher model to be trained may include not only the contrastive learning loss fluctuating within a certain preset range, but also the training generations of the teacher model to be trained being greater than or equal to the preset training generations.

[0245] After training the teacher model to be trained is completed, for a batch of data, when the first mapping network can output the first mapping feature vector z i The eigenvector pair (z) formed by the second mapping eigenvector i , z j Furthermore, the embodiments of this application can further compress the one-dimensional feature vector. For example, each feature vector can be mapped to the interval [0,1] through half-precision floating-point (fp16) or single-precision floating-point (fp32) operations. This mapping result essentially represents the feature probability of the wafer sample image, which can be mapped to the label probability.

[0246] For 4096 batches of wafer sample images, the probabilities of each 4096 wafer sample images can be obtained. Synchronization strategies such as `torch.cuda.synchronize()` are used to ensure synchronization between the graphics processing unit (GPU) and the central processing unit (CPU), thereby acquiring the feature probabilities of different batches of wafer sample images in parallel. Once the feature probabilities of the wafer sample images are obtained, they can be used to perform verification strategy operations with the test set, ultimately obtaining the self-supervised classification label output for that wafer sample image.

[0247] When different sets of wafer sample images belong to different process stages, the probability of defects in wafers at different process stages varies. Therefore, since the teacher model is relatively large, it can learn deeper and more detailed features in the first set of wafer sample images, providing powerful expressive and reasoning capabilities for subsequent student model training. Thus, in this embodiment, the knowledge transfer weight of each teacher model can be set to be positively correlated with the wafer defect probability of the process stage to which each first set of wafer sample image belongs.

[0248] The higher the probability of wafer defects in the first set of wafer sample images, the more knowledge from the teacher model is transferred to the candidate student model; conversely, the lower the probability of wafer defects in the first set of wafer sample images, the less knowledge from the teacher model is transferred to the candidate student model. In this case, the teacher model can determine the degree to which its key information participates in the training process of the corresponding first pre-trained teacher model based on the probability of wafer defects in its selected training data.

[0249] When the probability of wafer defects appearing in the training data of the teacher model is relatively low, the first pre-trained student model is trained based on the key information of the teacher model and each first wafer sample image. Each second set of wafer sample images serves as part of the training data for the corresponding teacher model. Although the training data for the first pre-trained student model is relatively small, the probability of wafer defects appearing in this training data is low. Therefore, the first pre-trained student model can train candidate student models relatively well with only a small amount of reference to the key information of the teacher model. However, if the key information of the teacher model is heavily involved in the training of the first pre-trained student model, it can easily cause excessive interference to the training process. For example, if the key information of the teacher model introduces too many high-dimensional features, the first pre-trained student model may overlearn the details of wafer defects, making it prone to overfitting during training.

[0250] When the probability of wafer defects appearing in the training data of the teacher model is relatively high, when training the first pre-trained student model based on the key information of the teacher model and each first wafer sample image, each second set of wafer sample images is part of the training data of the corresponding teacher model. The training data of the first pre-trained student model is relatively small and the model is lightweight, making it difficult to learn the wafer defect details hidden in the training data. Therefore, the first pre-trained student model can train the candidate student model relatively well when it references a large amount of key information from the teacher model. However, if the key information of the teacher model is only used in a small amount of the training of the first pre-trained student model, the first pre-trained student model is prone to underfitting during the training process.

[0251] As can be seen, the embodiments of this application can build multiple teacher models on the first server of a computing center or data center with high computing power, and configure personalized knowledge transfer weights for each teacher model based on the probability of wafer defects in the training data of the teacher model, thereby reducing the underfitting problem that the first pre-trained student model is prone to during the training process.

[0252] When training the teacher model, the first server can record key information about the teacher model, which may include procedural data from the training process. For example, this procedural data may include high-level semantic feature representation parameters, high-level semantic feature representations, and convergence parameters, which are stored as key information about the teacher model.

[0253] For example, the aforementioned high-level semantic feature representation may include: a high-level semantic feature map input to the wafer sample image, which may include a backbone feature map and feature vectors. Considering that the second set of wafer sample images is part of the training data corresponding to the teacher model, the high-level semantic feature representation may include the high-level semantic feature representation of the teacher model as the training data of the first pre-trained student model. This allows the second server to more accurately reference the high-level semantic features of the teacher model when training the first pre-trained student model, thereby improving the accuracy of knowledge transfer in the teacher model.

[0254] The high-level semantic feature representation parameters are the model parameters and gradient propagation parameters used in the teacher model to obtain the high-level semantic feature map. These model parameters can include the teacher model's learning rate and loss. When transmitting the key information of each teacher model, this key information can be considered as key information that has already been weighted by the teacher model's knowledge transfer weights. For example, the key information of each teacher model can be viewed as a weighting coefficient, and then multiplied by this weighting coefficient with the initial key information of the teacher model to obtain the teacher model's key information.

[0255] For example, the learning rate, loss, high-level semantic feature representation, and parameters of the high-level semantic feature representation of the teacher model can be used as key information of the teacher model. Based on the proportion of the training data of the first pre-trained student model to the training data of the teacher model, this key information of the teacher model is compressed and transmitted to the first server. This compression can be considered a data filtering process. For example, when both the first pre-trained student model and the teacher model use... Figure 8 The first wafer sample image x shown i Second wafer sample image x j Then the first main feature map f can be... i Second main trunk feature diagram f j The first mapping feature vector z i Second mapping eigenvector z j As a high-level semantic feature representation, it is transmitted to the second server.

[0256] As one possible implementation, the architecture of the first pre-trained student model in this embodiment can be partially identical to the architecture of the teacher model. In this case, the matching or difference between the key information of the teacher model and the key information of the first pre-trained student model can be determined by comparison, thereby determining the amount of knowledge that the teacher model needs to transfer to the first pre-trained student model. At this point, the correlation between the key information of the teacher model and the key information of the first pre-trained student model can be incorporated into the momentum parameters of the teacher model.

[0257] To ensure the amount of knowledge transfer in the teacher model, the reward value determined by the reward parameters of the teacher model in this embodiment is positively correlated with the degree of association between the teacher model and the student model. Furthermore, the reward value determined by the reward parameters of the teacher model is also positively correlated with the momentum determined by the momentum parameters of the teacher model. The specific reasons are as follows:

[0258] The greater the similarity between the key information of the teacher model and the key information of the first pre-trained student model, the stronger the correlation between the teacher and student models. In this case, the feature parameters of the teacher model in the same dimension, the same channel, the same pixel, or the same convolution kernel are similar to those of the first pre-trained student model. Therefore, the reward value determined by the reward parameter of the teacher model is larger, and correspondingly, the momentum of the teacher model determined by its momentum parameter is also larger.

[0259] The greater the similarity between the key information of the teacher model and the key information of the first pre-trained student model, the stronger the correlation between the teacher model and the student model. In this case, the gradient of the teacher model is similar in direction to the gradient of the first pre-trained student model, and the knowledge of the teacher model may have been transferred to the first pre-trained student model relatively well. Therefore, by increasing the momentum of the teacher model, the convergence of the first pre-trained student model can be accelerated.

[0260] The less similar the key information of the teacher model is to the key information of the first pre-trained student model, the more it indicates that the gradient of the teacher model is different from the gradient direction of the first pre-trained student model. The knowledge of the driving model cannot be effectively transferred to the first pre-trained student model. Therefore, by reducing the momentum of the teacher model, the oscillation of the first pre-trained student model during the training process can be reduced.

[0261] The first pre-trained student model in this application embodiment can be a student model based on a contrastive learning algorithm or a student model based on supervised learning. To reduce unnecessary training samples, this application embodiment can not only train the model using a student model based on a contrastive learning algorithm, but also, by further improving the contrastive learning algorithm, train the model using a student model based on a contrastive learning algorithm even when only positive samples are available.

[0262] When the teacher model and candidate student model are trained using an architecture based on contrastive learning algorithms, not only can the problem of low annotation efficiency be solved, but also the problem of differences and misjudgments in wafer defect annotation caused by differences in the professional level and experience of the annotators can be solved.

[0263] When using a student model based on a contrastive learning algorithm, the first pre-trained student model in this embodiment includes a first branch network and a second branch network. It should be understood that the first branch network and the second branch network use different hyperparameters (alpha, beta), and their weights are unrelated, do not need to share weights, and do not affect the training of the two branches or the parameter updates during the training process. Furthermore, the hyperparameters of the first branch network and the second branch network can be fixed or adjusted according to actual conditions.

[0264] Figure 9 A schematic diagram illustrating an example training process for a candidate student model according to an embodiment of this application is shown. Figure 9 As shown, in this embodiment of the application, the second server trains a first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model, including:

[0265] Step 901: The second server performs data augmentation on each second set of wafer sample images to obtain second set wafer sample image pairs. After data augmentation, each second set of wafer sample image pairs includes two matching sets of wafer sample images. For example, two different random data augmentation methods can be used to augment the second set of wafer sample images. Alternatively, the data augmentation method for the first set of wafer sample images described earlier can be referenced, but will not be elaborated here. After data augmentation on each second set of wafer sample images, a large number of wafer sample images can be obtained. Then, multiple wafer sample image pairs are obtained from this large number of wafer sample images, and these multiple wafer sample image pairs constitute the second set of wafer sample image pairs.

[0266] Step 902: The second server inputs each pair of second-group wafer sample images into the first branch network to obtain the projection feature prediction results of the two sets of wafer sample images within each pair of second-group wafer sample images. Here, the first branch network can be regarded as the prediction value branch, which is used to predict the mapping feature values ​​of the two sets of wafer sample images included in each pair of second-group wafer sample images.

[0267] Step 903: The second server inputs each pair of second-group wafer sample images into the second branch network to obtain the mapping feature prediction results of the two sets of wafer sample images within each pair of second-group wafer sample images under the second branch network. Here, the second branch network can be regarded as the ground truth branch, which is used to provide simulated ground truth values ​​for the mapping feature value prediction results of each pair of second-group wafer sample images.

[0268] Step 904: The second server determines the loss of the first pre-trained student model based on the mapping feature prediction results and projection feature prediction results under the second branch network.

[0269] In practical applications, the second server can determine the loss corresponding to one set of wafer sample images based on the projection feature prediction results of one set of wafer sample images included in each second set of wafer sample images and the mapping feature prediction results of the other set of wafer sample images in the second branch network. At the same time, the second server can determine the loss corresponding to the first pre-trained student model based on the loss corresponding to the two sets of wafer sample images included in each second set of wafer sample images.

[0270] For example, when the second set of wafer sample images includes a wafer sample image, the wafer sample image is data augmented to form a first wafer sample image and a second wafer sample image. The projection feature prediction result of the first wafer sample image and the projection feature prediction result of the second wafer sample image under the second branch network can be obtained through the first branch network. The mapping feature prediction result of the first wafer sample image under the second branch network and the mapping feature prediction result of the second wafer sample image under the second branch network can be obtained through the second branch network.

[0271] When determining the contrastive loss of the first pre-trained student model, the loss corresponding to the first wafer sample image can be determined based on the projection feature prediction results of the first wafer sample image and the mapping feature prediction results of the second wafer sample image under the second branch network. The loss corresponding to the second wafer sample image can be determined based on the mapping feature prediction results of the first wafer sample image under the second branch network and the projection feature prediction results of the second wafer sample image. Finally, the contrastive loss Loss of the first pre-trained student model can be obtained based on the loss loss-1 and the loss loss loss-2 corresponding to the first wafer sample image. For example, the loss loss-1 and the loss loss loss-2 corresponding to the first wafer sample image can be determined by a linear regression loss function, which can be expressed as Loss = loss-1 + loss-2, thus obtaining the loss of the first pre-trained student model.

[0272] Step 905: The second server determines whether the loss of the first pre-trained student model meets the convergence condition of the first pre-trained student model. It should be understood that the convergence condition of the first pre-trained student model in this embodiment may include the contrastive loss of the first pre-trained student model fluctuating within a certain preset range, and may also include the number of training generations of the first pre-trained student model being greater than or equal to a preset number of training generations.

[0273] Since the wafer sample image pairs generated by the same wafer sample image data augmentation essentially contain the same high-level semantics, the two sets of wafer sample images included in the wafer sample image pair can be considered positive samples of each other. When the two mutually positive wafer sample images are input into the first branch network and the second branch network respectively, if the second server determines that the loss of the first pre-trained student model does not meet the convergence condition of the first pre-trained student model, it indicates that the first branch network and the second branch network cannot accurately identify the two sets of mutually positive wafer sample images. Therefore, the first branch network and the second branch network need to be further updated, and thus, step 906 can be executed.

[0274] If the second server determines that the loss of the first pre-trained student model satisfies the convergence condition of the first pre-trained student model, it means that the first branch network and the second branch network can accurately identify two sets of wafer sample images that are positive samples to each other, and step 907 can be executed.

[0275] Step 906: The second server updates the first branch network using the backpropagation algorithm and updates the second branch network based on the momentum parameters of the corresponding teacher model. When updating the second branch network based on the momentum parameters of the teacher model, the learning rate and high-level semantic feature representation parameters of the teacher model can be incorporated. This method of updating the first and second branch networks avoids the possibility of model collapse.

[0276] For example, in the embodiments of this application, the model parameters of the first branch network and the model parameters of the second branch network can be updated using the backpropagation algorithm, and then the model parameters of the second branch network can be updated using the momentum determined by the momentum parameters of the teacher model.

[0277] For example, the learning rate (LR) and high-level semantic feature representation parameters of the teacher model can be obtained from the key information of the teacher model. The high-level semantic feature representation parameters are differentiated to obtain the change result dw of the high-level semantic feature representation parameters. Then, the updated weights of the second branch network are obtained by v = m*v0 + LR*dw and w = w0 + v. v0 represents the initial velocity of the teacher model, v represents the updated velocity, m represents the momentum determined by the momentum parameters of the teacher model, w0 represents the model parameters of the second branch network after updating through the backpropagation algorithm, and w represents the model parameters of the second branch network after adjusting through the momentum of the teacher model.

[0278] The second set of wafer sample images in this embodiment comes from a portion of the data of the first set of wafer sample images. Therefore, after updating the second branch network based on the momentum parameters of the corresponding teacher model, it can be guaranteed that when two wafer sample images that are positive samples of each other are input into the first pre-trained student model, the features of the two sets of wafer sample images that the first pre-trained student model can learn are approximately the same.

[0279] Step 907: The second server sets the first pre-trained student model as a candidate student model. For example, the first pre-trained student model at this time can be directly used as a candidate student model.

[0280] Considering that the architecture of the first pre-trained student model can be partially the same as that of the teacher model, the following will be based on... Figure 7 The architecture of the teacher model to be trained is used as a reference to design the architecture of the first pre-trained student model in this application embodiment. Examples are given below with reference to the accompanying drawings.

[0281] Figure 10A schematic diagram illustrating the training principle of a first pre-trained student model according to an embodiment of this application is shown. Figure 10 As shown, the first pre-trained student model in this application embodiment may include a first branch network and a second branch network. The first branch network may include a first prediction network and a feature projection network, and the second branch network includes a second prediction network.

[0282] The first prediction network is used to determine the mapping feature prediction result of each second set of wafer sample images under the first branch network, based on each of the two sets of wafer sample images included in the second set of wafer sample images.

[0283] For example, such as Figure 10 As shown, the first prediction network may include a first feature extraction network 1001A and a first multi-head mapper 1002A. The first feature extraction network 1001A can be referenced... Figure 7 The description of the feature extraction network shown is as follows. For example, this feature extraction network can also include a first backbone network and a first convolutional neural network. The first backbone network can extract features from each pair of second-set wafer sample images, which includes two sets of wafer sample images. The first convolutional neural network can determine the feature representation of each pair of second-set wafer sample images based on the features of each pair of second-set wafer sample images. It should be understood that the architecture of the first backbone network can be either a residual network or a lightweight pre-trained network such as Xception or MobileNet v3, to reduce the size of the first pre-trained student model and accelerate its training.

[0284] like Figure 10 As shown, the aforementioned first multi-head mapper 1002A can determine the mapping feature prediction result of each second set of wafer sample images on the two sets of included wafer sample images under the first branch network based on the feature representation of each second set of wafer sample images. This prediction result can be represented in the form of mapping features. The first multi-head mapper 1002A can employ... Figure 10 The example shows a single-layer multi-head mapper for feature mapping, but multi-layer multi-head mappers can also be used for feature mapping.

[0285] like Figure 10As shown, the aforementioned feature projection network 1003 is used to determine the projection feature prediction result of each second set of wafer sample images pair including the two sets of wafer sample images based on the mapping feature prediction result under the first branch network. For example, the feature projection network 1003 can transform the mapping feature vector of the second set of wafer sample images pair including the two sets of wafer sample images in a certain space into the projection feature vector of the second set of wafer sample images pair including the two sets of wafer sample images in another space, and use the projection feature vector of the second set of wafer sample images pair including the two sets of wafer sample images in another space as the projection feature prediction result of the second set of wafer sample images pair including the two sets of wafer sample images.

[0286] like Figure 10 As shown, the second prediction network is used to determine the mapping feature prediction result of each second set of wafer sample images in the second branch network based on the two sets of wafer sample images included in the second set of wafer sample images. The architecture of the second prediction network can be the same as that of the first prediction network, but parameters cannot be shared between the two. For example, the second prediction network may include a second feature extraction network 1001B and a second multi-head mapper 1002B, and the second feature extraction network 1001B may include a second backbone network and a second convolutional neural network.

[0287] like Figure 10 As shown, the second set of wafer sample images includes multiple wafer sample images. Taking a single wafer sample image as an example, data augmentation of this wafer sample image v can form a pair of wafer sample images, which are named the first wafer sample image v. i Second wafer sample image v j The first wafer sample image v i Second wafer sample image v j These can be combined to form a training batch of sample pairs.

[0288] like Figure 10 As shown, when the convolutional neural network is a deep convolutional neural network, the first wafer sample image v is... i Input the first feature extraction network 1001A to obtain the first deep feature map h of deep semantic features. alpha The second wafer sample image v j Input the second feature extraction network 1001B to obtain the second deep feature map h of deep semantic features. beta First depth feature map h alpha Second depth feature map h beta The resulting depth feature map pairs (h) alpha h beta ). The first depth feature map h alphaInput the first multi-head mapper 1002A for dimensionality reduction to obtain the first mapped feature vector z. alpha The second depth feature map h beta Input the second multi-head mapper 1002B for dimensionality reduction to obtain the second mapping feature vector z. beta The first mapping feature vector z alpha Second mapping eigenvector z beta It can form a first-dimensional mapping feature vector pair (z) alpha , z beta ).

[0289] Based on this, such as Figure 10 As shown, the first mapping feature vector z can also be... alpha Continue inputting into the feature projection network to obtain the projected feature vector q. beta This embodiment of the application can automatically introduce negative samples without requiring them by designing a feature projection network. For example, the feature projection network 1003 includes a batch normalization module 1003B and an activation module 1003C. The batch normalization module 1003B is used to batch normalize the mapping feature prediction results under the first branch network to obtain the batch normalization results corresponding to the two sets of wafer sample images included in each second set of wafer sample image pair. The activation module 1003C is used to activate the corresponding batch normalization results of each second set of wafer sample image pair to obtain the projection feature prediction results of the two sets of wafer sample images included in each second set of wafer sample image pair. Since the batch normalization module 1003B is added before the activation module 1003C, negative samples can be automatically introduced for the first pre-trained student model. Therefore, this embodiment of the application does not need to specifically provide negative samples for each second set of wafer sample images when training the first pre-trained student model, achieving the purpose of training the first pre-trained student model using positive samples, and providing a theoretical basis for the contrastive learning algorithm.

[0290] For example, for 512 batches of wafer sample image pairs, during batch normalization in the batch normalization module, a large number of mean and standard deviation values ​​are extracted from the 512 batches of wafer sample image pairs. This indirectly uses wafer sample images from multiple batches as positive and negative samples. Therefore, by setting a scheme of batch normalization first and then activation, this embodiment of the application can generate and reference a large number of negative samples, preventing model collapse. That is, it realizes the transfer of the feature representation space to the higher dimension latent space of the model, enabling the first pre-trained student model to learn a deeper semantic feature representation that is closer to the real world.

[0291] Of course, such as Figure 10As shown, the feature projection network 1003 may further include a first linear module 1003A and a second linear module 1003D. The first linear module 1003A can first linearize the mapping feature prediction result of each second set of wafer sample image pairs in the first branch network, and then input it into the batch normalization module 1003B. The second linear module 1003D can linearize the projection feature prediction result of each second set of wafer sample image pairs, so that the projection feature prediction result of each second set of wafer sample image pairs is represented in the form of a projection feature vector. In this case, the structure of the feature projection network 1003 can essentially include a first linear module, a batch normalization module 1003B, an activation module 1003C, and a second linear model, which can form a Linear-BN-ReLU-Linear structure to extract features from the mapping feature prediction result of each second set of wafer sample image pairs in the first branch network, thereby obtaining the projection feature prediction result of each second set of wafer sample image pairs represented by a mapping feature vector.

[0292] As can be seen, the architecture of the first pre-trained student model in this application embodiment is highly similar to that of the teacher model (both include a feature extraction module and a multi-head mapper). The difference is that the first pre-trained student model aims to compress the model size by using a lightweight backbone network to reduce the model size and accelerate model training.

[0293] In some alternative methods, such as Figure 10 As shown, since the first branch network can serve as the predicted value branch and the second branch network can serve as the true value branch, therefore, after acquiring the first wafer sample image v i The projected eigenvector q beta Second mapping eigenvector z beta Then, the projection feature vector q of the first wafer sample image can be used. beta The predicted value is the second mapping feature vector z corresponding to the second wafer sample image. beta To obtain the true value, acquire the first wafer sample image v. i The corresponding loss is loss-1. The loss calculated in this way guarantees the projection feature vector q of the first wafer sample image. beta The second mapping feature vector z corresponding to the second wafer sample image beta To draw closer and to converge.

[0294] As can be seen, in this embodiment of the application, the first branch network, which serves as the prediction branch, obtains the first mapping feature vector z. alpha After that, feature projection will be performed, and the second branch network, which serves as the ground truth branch, will obtain the second mapped feature vector z. betaAfter that, feature projection is no longer performed. This process is similar to that of adversarial learning, where the predicted value is updated (i.e., the first branch network moves from the mapped feature vector to the projected feature vector), while the true value is not updated (i.e., the second branch network obtains the second mapped feature vector z). beta This process stops, effectively freezing the second mapping feature vector z. beta (projection operation), therefore, when the first branch network can be used as the prediction branch and the second branch network can be used as the ground truth branch, the projection feature vector corresponding to the first wafer sample image can be used as the prediction value and the second mapping feature vector corresponding to the second wafer sample image can be used as the ground truth value, so that the loss corresponding to the first wafer sample image will never be reduced to 0.

[0295] like Figure 10 As shown in the embodiment of this application, the first wafer sample image v can also be used. i Input the second feature extraction network 1001B to extract the second wafer sample image v j Inputting the first feature extraction network 1001A yields the second deep feature map h, which represents the deep semantic features. beta The process is reset as described above until the first wafer sample image v is obtained. i Mapping feature vector of the second branch network and second wafer sample image v i The projected feature vector. This process can be combined with the first wafer sample image v. i Input the first feature extraction network 1001A and the second wafer sample image v j The second feature extraction network 1001B is input and executed in parallel, or sequentially. Since the first branch can be considered the prediction branch and the second branch the ground truth branch, the loss for the second wafer sample image (loss-2) can be obtained by using the first mapped feature vector corresponding to the first wafer sample image as the ground truth and the projected feature vector corresponding to the second wafer sample image as the prediction. This loss calculation ensures that the projected feature vector of the second wafer sample image converges to the second mapped feature vector corresponding to the first wafer sample image.

[0296] For example, such as Figure 10 As shown, for a batch of wafer sample image pairs, the first pre-trained student model, after a set number of training generations (such as one training generation or a specified number of training generations), can project the feature vector q of the first wafer sample image. beta The second mapping feature vector z corresponding to the second wafer sample image betaThe feature vector is activated and mapped using the softmax function or other functions to obtain the second feature probability [0,1]. Subsequently, the first and second branch networks can be inverted while keeping their two hyperparameters (alpha, beta) unchanged before and after the inversion. For example, the first wafer sample image v... i Image v, as a second wafer sample j The input is fed into the second branch network to obtain the first wafer sample image v. i The corresponding projected feature vector q beta The second wafer sample image v j Image v, as the first wafer sample i Input the first branch network to obtain the second wafer sample image v j The corresponding projected feature vector q beta Then, based on the first wafer sample image v i The corresponding projected feature vector q beta Second wafer sample image v j The corresponding projected feature vector q beta The second feature probability in the range [0,1] is obtained. Finally, the second feature probability and the second feature probability are averaged to obtain a more realistic and accurate first wafer sample image v. i Second wafer sample image v j The feature probability between them, i.e., feature similarity.

[0297] Meanwhile, synchronization strategies such as torch.cuda.synchronize() can be used to ensure synchronization between the GPU and CPU, ensuring that the feature probability values ​​of 512 batches of wafer sample images can be processed in parallel and synchronously, and finally returning the output result of the self-supervised classification label of the wafer sample image.

[0298] In addition, such as Figure 10 As shown, the contrastive loss Loss of the first pre-trained student model can be obtained by calculating the loss loss-1 corresponding to the first wafer sample image and the loss loss-2 corresponding to the second wafer sample image. This method of loss calculation ensures that the projected feature vector q of the first wafer sample image is accurate. beta The second mapping feature vector z corresponding to the second wafer sample image beta While ensuring convergence, it also guarantees that the projected feature vector of the second wafer sample image converges to the second mapped feature vector corresponding to the first wafer sample image.

[0299] As can be seen, the first pre-trained student model in this embodiment does not require manual classification and labeling of a second set of wafer sample images. Furthermore, the contrastive learning algorithm upon which this first pre-trained student model is based only requires positive samples, allowing for fully automatic and adaptive testing of feature accuracy during training. While a teacher model requires 4096 batches of wafer sample images per training round, the first pre-trained student model requires 512 batches per training round, with each batch including one wafer sample image. Therefore, the number of wafer sample images per batch is reduced by 87.5% compared to the teacher model, significantly reducing the FLOPs (Flat Operations Per Second) of the first pre-trained student model.

[0300] In one alternative approach, Figure 10 Based on the first pre-trained teacher model involved, when determining the momentum parameters of each teacher model based on the key information of the teacher model and the corresponding first pre-trained student model, various parameters involved in the process of generating the mapped feature vector from the wafer sample image pair formed by feature enhancement can be compared between the teacher model and the first pre-trained student model. For example, these parameters can be feature maps or model parameters.

[0301] When these parameters include feature maps, they can be the high-level semantic feature representations of the same wafer sample image pairs in the teacher model and in the student model. For example, they could be... Figure 8 The first main trunk feature map f shown i Second main trunk feature map f j ,and Figure 10 The first depth feature map h shown alpha Second depth feature map h beta The comparison determines the first reward parameter, which in turn determines the first momentum parameter. The first target momentum m1 is then determined from the first momentum parameter. Simultaneously, a comparison can also be made. Figure 8 The first mapping feature vector z shown i The second mapping feature vector z j ,and Figure 10 The first mapping feature vector z shown alpha Second mapping eigenvector z beta The second reward parameter is obtained, and the second momentum parameter can be determined through the second reward parameter. Then, the second target momentum m2 is determined by the first momentum parameter.

[0302] When updating the first and second branch networks, the first branch network can be updated using the backpropagation algorithm. After the second branch network is updated using the backpropagation algorithm, its model parameters need to be updated using the momentum of the teacher model. When the momentum of the teacher model includes a first target momentum m1 and a second target momentum m2, the momentum for updating the model parameters of the second branch network can be determined based on the first target momentum m1 and the second target momentum m2. Then, the model parameters of the second branch network after backpropagation are adjusted using the methods described above. Therefore, the momentum parameters of each teacher model in this embodiment actually consist of two momentum parameters. When the target momentum parameters are obtained, these target momentum parameters may belong to the same teacher model or different teacher models.

[0303] As one possible implementation method, Figure 11 A schematic diagram illustrating the process for determining a target momentum parameter according to an embodiment of this application is shown. Figure 11 As shown, the second server in this embodiment determines the target momentum parameter based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models, which may include:

[0304] Step 1101: The second server obtains multiple target student models that meet the first performance metric requirements based on the performance metrics of each candidate student model. It should be understood that the performance metrics in this embodiment can be referred to the relevant descriptions above, and will not be repeated here.

[0305] In practical applications, the target student model must meet the first performance metric requirement, which may be one or more performance metrics. Based on this, one or more performance metrics of different candidate student models can be compared according to actual needs, thereby obtaining multiple target student models that perform well in one or more aspects.

[0306] For example, the performance metrics can be designed according to actual needs, with a performance metric range for each metric. Then, it is determined whether each performance metric of each candidate student model falls within the corresponding performance metric range. If a performance metric exceeds the corresponding performance metric range, then the candidate student model is considered to have failed to meet the corresponding performance metric requirement for that performance metric. When all performance metrics of the candidate student model meet the corresponding performance metric requirement, or when a specified performance metric meets the corresponding performance metric requirement, the candidate student model can be considered as the target student model that meets the first performance metric requirement.

[0307] Step 1102: The second server obtains the target momentum parameters from the momentum parameters of the teacher models corresponding to multiple target student models.

[0308] The process of obtaining the target momentum parameter in this embodiment not only filters the target momentum parameter with the best knowledge transfer capability, but also narrows the selection range of the target momentum parameter, improves the selection efficiency of the target momentum parameter, and achieves the global optimal solution. It should be understood that the target momentum parameter can be obtained using various possible selection strategies in this embodiment. For example, it can be selected simply based on the magnitude of the target momentum parameter, or it can be selected by comprehensively considering the performance of various aspects; these will not be elaborated upon here.

[0309] In practical applications, the number of momentum parameters of the teacher model corresponding to the target student model in this application embodiment may include one or at least two. Correspondingly, the number of target momentum parameters may be one or at least two. If the number of target momentum parameters is at least two, it is possible that the selected target momentum parameters belong to different teacher models or correspond to different candidate student models. When training the candidate student model, if the combination of momentum parameters of multiple teacher models is not considered, it is impossible to obtain the target candidate student model directly based on the candidate student model corresponding to the target momentum parameters.

[0310] When the target momentum parameters include at least two target momentum parameters, if all target momentum parameters correspond to the same candidate student model, it means that all target momentum parameters come from the same teacher model. The second server can determine the target candidate student model based on the candidate student model corresponding to the target momentum parameters.

[0311] When the target momentum parameters include at least two target momentum parameters, and the second server determines that at least two target momentum parameters correspond to different candidate student models, if there are no candidate student models trained using the momentum parameters of different teacher models, then the second server can determine a third set of wafer sample images based on the training data of the teacher models corresponding to all target momentum parameters. The second server then trains a first pre-trained student model based on all target momentum parameters and the third set of wafer sample images to obtain the target candidate student model. Here, the third set of wafer sample images can be partial training data of the teacher models corresponding to all target momentum parameters.

[0312] For example, when the target momentum parameters include a first momentum parameter and a second momentum parameter, the first momentum parameter can determine the first target momentum m1 of the teacher model, and the second momentum parameter can determine the second target momentum m2 of the teacher model.

[0313] When the first momentum parameter and the second momentum parameter are both momentum parameters of the same teacher model, the candidate student model corresponding to the target momentum parameter can be directly reused as the target candidate student model.

[0314] When the first momentum parameter is the momentum parameter of one teacher model and the second momentum parameter is the momentum parameter of another teacher model, a portion of wafer sample images can be obtained from one teacher model and another portion from the other teacher model. These two portions of wafer sample images are then integrated to obtain a third set of wafer sample images. For example, when the first pre-trained student model has 512 wafer sample images, the required wafer samples can be extracted proportionally from the two teacher models to form the third set of wafer sample images.

[0315] Once the third set of wafer sample images is obtained, these images can be used as training data for the first pre-trained student model. The first pre-trained student model can then be trained using the methods described earlier, i.e., referencing... Figure 9 The candidate teacher model is used to train the first pre-trained student model, thereby obtaining the target candidate student model.

[0316] As one approach, considering the large amount of training data for the teacher model (e.g., 4096 batches) and the very small amount of training data for the student model (e.g., 512 batches), although the training data volumes of the two are proportional and a mapping can be achieved, for a large model like the teacher model, the different batches of training data contain three-dimensional features that are difficult to compress. When trillions of parameters are compressed to hundreds of billions or even billions of parameters, there may be losses. While larger model sizes generally lead to better performance, when averaging the benefit of a single parameter, larger models result in smaller average benefits per parameter, leading to lower training efficiency for each parameter.

[0317] Based on the above reasons, this application embodiment, from the perspective of balancing training efficiency and parameter gains, can use the target candidate student model as the teacher model and train the wafer defect detection model using knowledge distillation.

[0318] This application embodiment distills the knowledge of the target candidate student model into the target student model. While retaining the relevant characteristics of the first student model, it can further compress the size of the target student model. Therefore, when determining the wafer defect detection model based on the target student model, the wafer defect detection model has a more accurate wafer defect detection capability, is easier to deploy, has a faster inference speed, and is suitable for low-configuration servers in the local data center within the wafer fab.

[0319] In some alternative embodiments, the second server in this application embodiment can use the target candidate student model as the teacher model, and can train the second pre-trained student model using an iterative knowledge distillation method to obtain the target student model. When the second server determines that the target student model meets the convergence condition of the target student model, it determines the wafer defect detection model based on the target student model.

[0320] In practical applications, the convergence condition of the target student model in this application embodiment may include: when the number of iterations meets the iteration number constraint, the iteration number constraint may include the number of iterations of the target candidate student model being greater than or equal to a preset number. For example, when the preset number is n-1, if the number of iterations of the target candidate student model is less than n-1, it means that the target student model still needs to iterate to train a new target student model. If the number of iterations of the target candidate student model is equal to n-1, it means that the wafer defect detection model can be determined based on the target student model, or the target student model can be directly reused in the direct wafer defect detection model.

[0321] When using iterative knowledge distillation to train the target student model for determining the wafer defect detection model, on the one hand, the wafer defect detection model can be further compressed, making it easier to deploy and improving the inference speed of the wafer defect detection model; on the other hand, the prediction results of the iteratively generated target student model can be made closer to the prediction results of the target candidate student model or the prediction results of the old target student model, reducing the discrepancy.

[0322] To improve the generalization ability and robustness of the wafer defect detection model, this application employs an iterative knowledge distillation method. When distilling knowledge from the target candidate student model to the target student model, the wafer sample images included in the fourth set of wafer sample images used in different knowledge distillations (or iterations) belong to at least different process stages in the semiconductor manufacturing process. In this case, wafer defect details from different process stages can be distilled into the last target student model in an cumulative manner. Therefore, when determining the wafer defect detection model based on the target student model, the wafer defect detection model not only has good wafer defect detection function, but also good generalization ability, high reliability, strong robustness, and good adaptability. This results in a significant improvement in the performance of the wafer defect detection model determined by the target student model compared to each teacher network.

[0323] Moreover, the method of this application embodiment has good generalization ability. It can be adapted not only to classification tasks, but also, when conditions permit, as a pre-trained model to adapt to computer vision tasks such as object detection, semantic segmentation, and panoramic segmentation. It has strong application potential in the field of semiconductor wafer manufacturing.

[0324] Figure 12 This illustration shows an example flowchart of training a wafer defect detection model using an iterative knowledge distillation method, according to an embodiment of this application. Figure 12 As shown, in this embodiment of the application, the second server determines the wafer defect detection model based on the target candidate student model obtained from the teacher model corresponding to the target momentum parameter, including:

[0325] Step 1201: The second server determines the momentum parameters of the target candidate student model based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model. The method for obtaining the second pre-trained student model can refer to the method for obtaining the first pre-trained student model described above.

[0326] In practical applications, embodiments of this application can select a student model to be trained. This student model can be an untrained lightweight model or a wafer defect detection model trained by the method of this application. The second server can obtain a portion of the wafer sample images from the third set of wafer sample images, using these as the fourth set of wafer sample images. The fourth set of wafer sample images is then used to pre-train the student model to be trained (e.g., complete one training generation), thereby obtaining a second pre-trained student model. In this case, the target candidate student model corresponding to the second pre-trained student model can provide training data for the second pre-trained student model.

[0327] For example, the second server can determine the reward parameters of the target candidate student model based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, and then determine the momentum parameters of the target candidate student model based on the reward parameters. The method for determining the momentum parameters of the target candidate student model can refer to the process for determining the momentum parameters of the teacher model mentioned above, and will not be detailed here.

[0328] Step 1202: The second server uses the fourth set of wafer sample images as training data. Based on the momentum parameters of the target candidate student model, it distills the knowledge of the target candidate student model into the second pre-trained student model to obtain the target student model. Here, the architecture of the second pre-trained student model can be partially the same as that of the first pre-trained student model; both can be student models based on contrastive learning algorithms. In this case, it is not necessary to annotate the fourth set of wafer sample images to complete the training of the target student model. This not only solves the problem of low annotation efficiency but also addresses the issues of wafer defect annotation discrepancies and misjudgments caused by differences in the professionalism and experience of the annotators.

[0329] Step 1203: The second server determines whether the target candidate student model meets the convergence condition of the target student model. The convergence condition of the target student model may include the number of iterations of training the target candidate student model being greater than or equal to a preset number.

[0330] If the number of iterations of the target candidate student model is less than the preset number, it means that the target candidate student model does not meet the convergence condition of the target student model, and step 1204 can be executed. Otherwise, it means that the target candidate student model meets the convergence condition of the target student model, and step 1205 can be executed.

[0331] Step 1204: When the performance metrics of the target student model meet the second performance metric requirement, the second server updates the target candidate student model based on the target student model. This is actually the process of using the target student model as the teacher model and executing steps 1202 and 1203.

[0332] In practical applications, the second server in this embodiment can determine whether the performance indicators of the target student model meet the second performance indicator requirements. If the performance indicators of the target student model meet the second performance indicator requirements, it indicates that the performance of the target student model is relatively good, and the target candidate student model can be directly updated based on the target student model. For example, the target student model can be directly used as a new target candidate student model. If the second server determines that the performance indicators of the target student model do not meet the second performance indicator requirements, it indicates that the performance indicators of the target student model are relatively poor.

[0333] If the target candidate student model is updated directly based on the target student model, the performance of the target student model trained subsequently will deteriorate. Therefore, the second server in this embodiment can change the conditional parameters of knowledge distillation, such as the distillation loss function, the second distillation temperature, the momentum parameter of the target candidate student model, etc., but not limited to these. Then, based on the original target candidate student model, steps 1202 and 1203 are re-executed under the adjusted conditional parameters of knowledge distillation. This process is equivalent to filtering out interference noise, thereby optimizing the iterative distillation process of the target student model, enabling the wafer defect detection model to achieve better wafer defect detection results and reducing the interference of adverse factors.

[0334] Step 1205: The second server determines the wafer defect detection model based on the target student model. For example, the wafer defect detection model can directly reuse the target student model.

[0335] Using the above technical solution, the second server references the momentum parameters of the target candidate student model and distills the knowledge from the target candidate student model into the corresponding second pre-trained student model. This is not simply model compression, but rather filtering out a large amount of noise from the target candidate student model or the old target student model, passing on more realistic and useful features for the classification task to subsequent target student models, thereby optimizing the wafer defect detection model. Furthermore, each knowledge distillation uses the target student model obtained from the previous knowledge distillation as the teacher model, causing the prediction results of the final wafer defect detection model to converge with the prediction results of the target candidate student model or the old target student model, reducing discrepancies.

[0336] Figure 13This illustration shows an example flowchart of training a target student model using a target candidate student model, as per an embodiment of this application. Figure 13 As shown, in this embodiment of the application, the second server uses the fourth set of wafer sample images as training data, and distills the knowledge of the target candidate student model into the second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model, including:

[0337] Step 1301: The second server performs data augmentation on the fourth set of wafer sample images to obtain a fourth set of wafer sample image pairs. This fourth set of wafer sample images can be directly read from the second server's storage space; it does not need to be retrieved from the second server's storage space. For the data augmentation process of the fourth set of wafer sample images, please refer to the previous description of the data augmentation for the first set of wafer sample images.

[0338] Step 1302: The second server inputs the fourth set of wafer sample images into the target candidate student model and obtains the feature similarity of the target candidate student model at the first distillation temperature.

[0339] Step 1303: The second server inputs the second pre-trained student model based on the fourth set of wafer sample images, obtaining the feature similarity of the second pre-trained student model at the first target distillation temperature and the feature similarity of the second pre-trained student model at the second distillation temperature. Here, the architecture of the second pre-trained student model can be similar to that of the first pre-trained student model. For example, the second pre-trained student model can be obtained by replacing the network modules with the same functions in the first pre-trained student model with other, more lightweight network modules.

[0340] Step 1304: The second server determines the distillation loss of the second pre-trained student model based on the feature similarity of the target candidate student model at the first distillation temperature, the feature similarity of the second pre-trained student model at the first target distillation temperature, and the feature similarity of the second pre-trained student model at the second distillation temperature. Here, the process by which the second server determines the distillation loss of the second pre-trained student model in this embodiment may include:

[0341] First, the second server can determine the loss of the target candidate student model at the first distillation temperature based on the feature similarity of the target candidate student model at the first distillation temperature and the feature similarity of the second pre-trained student model at the first target distillation temperature. This can essentially be regarded as the soft loss in knowledge distillation.

[0342] Secondly, the second server determines the loss of the target candidate student model at the second distillation temperature based on the feature similarity of the second pre-trained student model at the second distillation temperature. This can essentially be seen as the hard loss in knowledge distillation.

[0343] Finally, the second server determines the distillation loss of the second pre-trained student model based on the loss of the target candidate student model at the first distillation temperature and the loss of the target candidate student model at the second distillation temperature.

[0344] Step 1305: The second server determines whether the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model. The convergence condition of the second pre-trained student model may include the distillation loss of the second pre-trained student model being within a preset loss range, and may also include the second pre-trained student model reaching a preset number of training generations.

[0345] If the distillation loss of the second pre-trained student model is within the preset loss range and / or reaches the preset training generations, it means that the second pre-trained student model meets the convergence condition of the second pre-trained student model, and step 1306 can be executed. If the distillation loss of the second pre-trained student model is outside the preset loss range and / or has not reached the preset training generations, it means that the second pre-trained student model does not meet the convergence condition of the second pre-trained student model, and step 1307 can be executed.

[0346] Step 1306: The second server determines the target student model based on the second pre-trained student model. For example, the second preset trained student model can be directly set as the target student model.

[0347] Step 1307: The second server updates the second pre-trained student model based on the momentum parameters of the target candidate student model. The momentum update method here can be found in relevant technologies and will not be elaborated upon here.

[0348] The embodiments of this application can be based on the idea of ​​reinforcement learning, and the selection of the target momentum model can be achieved through an intelligent agent. Figure 14 A schematic diagram illustrating the system principle based on reinforcement learning, according to an embodiment of this application, is shown. Figure 14 As shown, the system 1400 in this embodiment may include a computing center 1401 and a local data center server 1402. The computing center 1401 may include a first server, and the local data center server 1402 may include a second server. The intelligent agents used by the system include a first intelligent agent Ts2S, a second intelligent agent T2S, and a third intelligent agent S2S.

[0349] The first agent Ts2S and the third agent S2S can perform key information comparison and momentum parameter determination, while the third agent can select the target momentum parameters. The architecture of the first agent Ts2S, the second agent T2S, and the third agent S2S can be a supervised learning model architecture based on the attention mechanism, or a similar model architecture based on a supervised learning model architecture based on a Generative Adversarial Network (GAN). Similar model architectures can be trained by referring to relevant technologies to obtain the first agent Ts2S, the second agent T2S, and the third agent S2S.

[0350] In this embodiment, multiple teacher models can be constructed in the computing center 1401. Here, the n teacher models can respectively include the first teacher model Tea1, the second teacher model Tea2, ..., the (n-1)th teacher model Tea... n-1 And the nth teacher model Tea n This can be used to construct a large model environment. The n teacher models have different knowledge transfer weights, which are set according to the probability of experience deficiencies appearing in the training data of different teacher models. The setting method is described in the previous section.

[0351] For example, in this embodiment of the application, a single student model can also be constructed on the local data center server 1402. The computing center can record key information of the teacher model, such as: high-level semantic feature representation, learning rate, contrastive loss, and high-level semantic feature representation parameters (e.g., parameters before generating the mapped feature vector, such as...) when different teacher models reach the preset training generations and student models reach the preset training generations required for pre-training. Figure 8 The model parameters of the depthwise separable convolutional modules shown include pw and dw modules, main channel convolution, gradient propagation parameters, etc.

[0352] In this embodiment of the application, the local data center server 1402 can obtain key information of n teacher models from the computing center 1401, and then transmit the key information of the n teacher models to the first intelligent agent Ts2S. At the same time, the first intelligent agent Ts2S can also obtain key information of the pre-trained student models. Based on the key information of the first pre-trained teacher models and the key information of each teacher model, the reward value of each teacher model can be determined, and then the momentum of each teacher model can be determined based on the reward value of each teacher model.

[0353] The first agent Ts2S can compare the parameters from the data-augmented image to the parameter generation before feature vector generation of n teacher models with those of a single student model, including key information such as high-level semantic feature representation, learning rate, contrastive loss, and high-level semantic feature representation parameters, to determine the reward value of each teacher model. Here, the first agent Ts2S can dynamically select different teacher models to compare the parameters from the data-augmented image to the parameter generation before feature vector generation for a single student model.

[0354] For teacher models, high-level semantic feature representations can include, for example: Figure 7 The first trunk feature map f and the second trunk feature map f shown are shown. j First deep feature map h i and the second deep feature map h j High-level semantic feature representation parameters can include model parameters before generating feature vectors. For the student model, high-level semantic feature representation can include, for example, Figure 10 The first depth feature map h shown alpha Second depth feature map h beta The first mapping eigenvector z alpha Second mapping eigenvector z beta .

[0355] Once the momentum parameters of each teacher model are determined by the first agent Ts2S, the momentum parameters of each teacher model can be passed to a single student model. Then, following the training method for the student model described above, n candidate student models are obtained. At the same time, the performance indicators of the n candidate student models and the corresponding momentum parameters of the teacher models are recorded. Then, the data is transmitted to the second agent T2S. The second agent can refer to the relevant description above to determine the target weight parameters. Then, it uses the target weight parameters to update the model parameters of the student model, thereby obtaining the first student model std1. The first student model std1 can be considered as the globally optimal student model.

[0356] Once student model #1 (std1) is obtained, it can be used as the teacher model. Training data is retrieved from the local computer lab database, and student model #2 (std2), ..., student model #n-1 (std) are iteratively derived using iterative knowledge distillation. n-1 and the nth student model std nFor example, the key information of student model std1 can be compared with the key information of another student model through a third agent S2S to obtain the reward value of student model std1. Then, the momentum parameter of student model std1 is determined based on the reward value of student model std1. Next, based on the momentum parameter of student model std1 and the training data, another student model is trained by knowledge distillation to obtain student model std2. Student model std2 can be used as a new teacher model to carry out the next round of knowledge distillation training of student models in the same way that student model std1 trains another student model.

[0357] When using the student model n-1 std n-1 When the nth student model is obtained by knowledge distillation, the number of iterations has reached the preset number of iterations n. Therefore, the nth student model can be used as the wafer defect detection model and output to the low-configuration computer 1403 for inference tasks.

[0358] As can be seen, the first agent Ts2S and the second agent T2S combine to compress the large batch training data of the teacher model into small batch training data of the teacher model in a proportional manner. For example, when the teacher model uses 4096 batches of training data, while the student model uses only 512 batches, although the amount of training data is proportional and mapping can be achieved, for large models, the features extracted in each batch are three-dimensional features, and the compression accuracy of three-dimensional features is not high. Therefore, there will be a loss when compressing trillions of parameters into hundreds of billions and hundreds of millions of parameters. Although more parameters and a larger model result in better model performance, the average benefit of each parameter decreases when calculating the benefit of a single parameter. The training efficiency of each parameter is also lower. Therefore, the method in this embodiment comprehensively considers training efficiency and parameter benefits, and adopts iterative knowledge distillation between lightweight student models of the same batch size to ensure that the obtained nth student model std n It is lightweight, stable in deployment, and has low computational complexity, while also exhibiting good robustness and generalization ability.

[0359] In this embodiment, the third agent S2S uses the target candidate student model as the teacher model. During the iterative training of the target student model, it can decide whether to abandon the target student model generated by the local iterative training based on the performance indicators of the target student model. For example, when the performance indicators of the target student model are poor, the conditional parameters of knowledge distillation can be adaptively adjusted, such as adjusting the knowledge distillation loss function, contrastive learning rate, knowledge distillation learning rate, knowledge distillation temperature, reward value, and even the model parameters of the student model. Then, the target student model is retrained using the current old target candidate student model as the teacher model, reducing the transfer of unfavorable knowledge to the final wafer defect detection model. Therefore, the iterative knowledge distillation between student models in this embodiment can enhance adversarial competition, enabling the student model to achieve optimal feature extraction, corrective training, model optimization, knowledge extraction, capability generalization, robustness enhancement, and deployment stability, thereby comprehensively improving the performance of the wafer defect detection model.

[0360] The first server in this embodiment can train multiple teacher models. Since the teacher models are large and involve a large amount of training data, they can not only learn detailed knowledge of various wafer defects present in the first set of wafer sample images, but also possess strong generalization ability and robustness. Therefore, based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the momentum parameters of each teacher model can be determined, incorporating the correlation between the key information of the teacher model and the key information of the corresponding first pre-trained student model into the momentum parameters of the teacher model. In this case, when training the corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images, the convergence speed and training efficiency of the first pre-trained student model can be adjusted according to the degree of correlation between the key information of the teacher model and the key information of the first pre-trained student model, thereby transferring the detailed knowledge of wafer defects learned by each teacher model to the lightweight candidate student model. In this scenario, the target momentum parameter can be determined based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models. This process is essentially a screening process by the second server to obtain momentum parameters that allow the candidate student model to transfer knowledge from more teacher models through the performance metrics of each candidate student model. Therefore, the target momentum parameter has good knowledge transfer capability and exhibits excellent performance on the candidate student model. By determining the candidate student model based on the teacher model corresponding to the target momentum parameter, the wafer defect detection model can be determined. This ensures that the wafer defect detection model not only fully learns the detailed knowledge of wafer defects but also has the characteristics of strong generalization ability, high reliability, and robustness of the teacher model.

[0361] Furthermore, based on the performance metrics of each candidate student model, the second server in this application selects target momentum parameters with better knowledge transfer capabilities from the momentum parameters of multiple teacher models to participate in the determination process of the wafer defect detection model. This ensures that the detailed knowledge of wafer defects learned by the large-scale teacher models can be transferred to the lightweight wafer defect detection model. This process is not only about extracting detailed knowledge of wafer defects from multiple teacher models, but also about compressing the large-scale teacher models into a lightweight wafer defect detection model. Therefore, the second server in this application, even with low hardware and software configurations, does not need to train a large number of candidate teacher models. Instead, it directly obtains the key information of the already trained candidate teacher models from the first server and quickly completes the wafer defect detection model through knowledge extraction and model compression. This reduces the model training pressure on the second server, making the wafer defect detection model not only lightweight and easy to deploy, but also possessing strong generalization ability, high availability, and high reliability, facilitating inference tasks on lower-configuration computer devices.

[0362] In some embodiments, this application can determine the momentum parameter of the teacher model by comparing key information of the teacher model with key information of the first pre-trained student model, and then train the first pre-trained student model using this as a starting point to obtain candidate student models. Then, a target momentum parameter is selected from a large number of candidate models, and a target candidate student model is determined based on the target momentum parameter. Then, the momentum parameter of the target candidate student model is determined by comparing key information of the target candidate student model with key information of the second pre-trained student model. Based on this, a wafer defect detection model is iteratively distilled from knowledge.

[0363] It is evident that the key information of each model involved in the embodiments of this application plays an important role in the implementation of the method of the embodiments of this application. The key information here is essentially the training results of each model. Therefore, in the process of obtaining the teacher model, the first pre-trained student model and the second pre-trained student model, the embodiments of this application can collect the training results of each model to complete the technical solution of the embodiments of this application.

[0364] Furthermore, the process of selecting the target momentum parameter from multiple candidate student models utilizes the performance metrics of each candidate student model. Similarly, determining the knowledge transfer weights of the teacher model involves judging the performance metrics of the wafer defect detection model. Therefore, in both the training of candidate student models and the determination of the empirical defect detection model based on the target student model, the performance metrics of the models can be collected to fulfill the relevant functions of this embodiment. For example, the Top-1, Top-5, MIOU, loss chart, learning rate chart, and depth visualization convolutional images of each model can be collected, along with the final results. The necessary key information and performance metrics can then be selected to complete the relevant functions of this embodiment, thereby ensuring the effectiveness and superiority of the finally trained wafer defect detection model.

[0365] In some embodiments, both the teacher and student models in this application are trained using self-supervised methods such as contrastive algorithms. By comparing wafer sample images with themselves or other samples, discriminative and deep feature representations can be learned. Moreover, this training method does not require manually labeled training data. Instead, it uses input wafer sample images for data augmentation to automatically construct positive and negative sample pairs, thereby overcoming the problem of traditional supervised learning requiring a large amount of labeled data.

[0366] This application employs a self-supervised contrastive learning algorithm, which can learn the similarities and differences between wafer sample images, thereby improving the student model's ability to learn feature representations with better generalization performance. These feature representations can be transferred to other parallel tasks, such as object detection, semantic segmentation, and panoptic segmentation, exhibiting strong versatility. Therefore, this application's use of a self-supervised learning contrastive algorithm can improve the model's generalization ability.

[0367] In some embodiments of this application, during the training of multiple teacher models and individual student models (including target candidate student models and target student models) using a self-supervised contrastive learning approach, a large number of sample pairs can be generated through data augmentation techniques. These sample pairs can be a combination of source samples and data augmentation samples to form a sample set, thereby expanding the diversity of training data, helping to improve the robustness and generalization ability of the model, and reducing the risk of overfitting and underfitting during model training.

[0368] In some embodiments, when using a self-supervised contrastive learning algorithm, the present application employs unlabeled data for training and learning, enabling the obtained wafer defect detection model to complete the ADC automated wafer defect classification task. Simultaneously, the present application can also use a self-supervised contrastive learning algorithm to pre-train a student model, thereby obtaining a first pre-trained student model and a second pre-trained student model.

[0369] In some embodiments, during the training of multiple teacher models and student models (including target candidate student models and target student models) using self-supervised deep learning technology, multiple teacher models are obtained through a self-supervised contrastive learning algorithm. Their knowledge is then transferred in parallel to a smaller first pre-trained student model to obtain multiple corresponding candidate teacher models. Then, a second agent selects the target momentum parameters corresponding to the candidate student model with excellent performance based on the performance indicators such as Top-1, Top-5, and MIOU of the candidate student models.

[0370] In some embodiments, the present application uses a teacher model to transfer knowledge to a first pre-trained student model based on a contrastive learning algorithm. This can be seen as a self-supervised knowledge distillation algorithm, which can significantly improve the performance of the trained student model.

[0371] In some embodiments, the self-supervised knowledge distillation algorithm employed in this application can compress the teacher model into a smaller student model (i.e., the target candidate student model). Furthermore, the teacher model is trained in a computing center, while the student model is trained on a local server. Since the student model has fewer parameters, it exhibits more efficient computation and storage performance. Therefore, the method in this application is more suitable for resource-constrained environments in actual wafer fabs.

[0372] In some embodiments, since the student model (target candidate student model and target student model) has fewer parameters and fewer convolutional layers, its computational complexity is relatively lower than that of the teacher model. Therefore, the wafer defect detection model obtained based on the student model has excellent inference speed and is more suitable for deployment scenarios in actual wafer fabs.

[0373] In some embodiments, the teacher model involved in this application is a large-scale deep learning model related to computer vision, and the training source data images occupy a large amount of storage space, and the training process occupies a large amount of memory and video memory. Therefore, the teacher model in this application can be trained using a distributed self-supervised contrastive learning algorithm, thereby solving the problems of hardware computing power and cost.

[0374] In some embodiments of this application, when training the teacher model using a distributed self-supervised contrastive learning algorithm, multiple computing nodes and multiple GPUs / TPUs can be used for parallel training of the teacher model, thereby accelerating the training speed of the teacher model. When training the teacher model in parallel, the batch size can be increased, allowing the self-supervised contrastive learning algorithm to learn more diverse and deeper feature representation parameters. During the parallel knowledge distillation of knowledge from multiple teacher models into the student model (target candidate student model), data processing and computation tasks (such as data augmentation and data compression) of multiple teacher and student models are processed simultaneously. The training results from the computing center and the local computer room server are transmitted in parallel to the first agent through a network connection to participate in the training of the first pre-trained student model, improving learning efficiency.

[0375] In some embodiments, the distributed self-supervised contrastive learning algorithm introduced in this application expands one teacher model in knowledge distillation into n teacher models and one student model into n student models. Furthermore, as the parameters of the teacher and student models increase, the overall system architecture becomes more complex, ensuring that the teacher and student models can learn deeper and more features, providing the output model with stronger representation and reasoning capabilities. Therefore, the method in this application can train a more efficient and higher-performing experience defect detection model under limited conditions.

[0376] In some embodiments, the self-supervised contrastive learning algorithm used in this application is a type of self-supervised deep learning algorithm, which belongs to unsupervised learning methods. Combining the self-supervised deep learning algorithm with distributed technology can improve the training efficiency of the wafer defect detection model, empower wafer manufacturing plants, and improve work efficiency.

[0377] In some embodiments, the parameters required for large deep learning models in the AI ​​field increase exponentially with the size of the model, while the average training efficiency of the model decreases as the number of parameters increases. To address this problem, this application embodiment employs a self-supervised contrastive learning algorithm to train teacher models in a computing center. Then, the contrastive learning algorithm is combined with a knowledge distillation algorithm, enabling the knowledge from multiple large teacher models to be transferred to a local server. This knowledge is then transferred in parallel to a first pre-trained student model through a first intelligent agent, thereby obtaining multiple candidate teacher models. A second intelligent agent is used to compare and verify smaller candidate student models. Finally, the candidate student model with the best performance across various metrics is selected as the target candidate student model. Since the target candidate student model requires fewer parameters to train, the method of this application embodiment has high training efficiency.

[0378] As can be seen, in this embodiment of the application, a distributed contrastive learning algorithm is used to train a large teacher model in the computing center, while a smaller first pre-trained student model is trained on the local server in the wafer fab. The two models share parameters with the first agent through the network. Therefore, the method of this embodiment of the application can significantly reduce the training time and improve training efficiency.

[0379] In some embodiments, the method of this application uses a contrastive learning algorithm for model training. It only requires setting the number of classification labels to be identified, significantly reducing the reliance on manual annotation during the data preparation stage before model training and the preparation time required for preliminary work. In other words, before model training, it is no longer necessary to manually annotate a large number of classification labels. Only after the model training is completed do code labels such as DefectClass-0, DefectClass-1, and DefectClass-2 need to be manually identified and renamed to easily understandable label names used in wafer manufacturing and defect analysis. This allows users with limited professional knowledge to quickly identify wafer defect problems through the wafer defect detection model, solving the problem of insufficient experienced professionals in wafer fabs.

[0380] Moreover, compared with the training methods of wafer defect detection models in related technologies, the human intervention factors before training the wafer defect detection model in this application embodiment are greatly reduced (verified to reduce the workload of professionals by 90%), which can reduce the probability of human error and human bias, thereby reducing the probability of learning bias in the wafer defect detection model, making it easier for users to locate errors in the wafer defect detection model, reducing the communication costs between the wafer defect detection model maintenance department and various departments such as wafer manufacturing process, analysis, and integration, thereby improving work efficiency.

[0381] The above mainly describes the solution provided by the embodiments of this application from the perspective of the interaction between the first server and the second server. It is understood that, in order to achieve the above functions, the first server and the second server include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments applied herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0382] This application embodiment can divide the first server and the second server into functional units according to the above method example. For example, each function can be divided into a functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0383] By dividing each functional module according to its corresponding function, an exemplary embodiment of this application provides a server. Figure 15 A schematic block diagram of the functional modules of a server according to an exemplary embodiment of this application is shown. Figure 15 As shown, the training device 1500 includes a communication module and a training module.

[0384] When the training device 1500 is used to train the teacher model, the functions of each module included in the communication device of this application embodiment are as follows:

[0385] The communication module 1501 is used to obtain multiple first group wafer sample images from the first server, and the training module 1502 is used to train the teacher model to be trained based on the multiple first group wafer sample images to obtain multiple teacher models, each of which corresponds to each first group wafer sample image. The communication module 1501 is also used to send key information of the multiple teacher models to the first server.

[0386] In one possible implementation, the training module is used to perform data augmentation on each first group of wafer sample images to obtain each first group of wafer sample image pair, and input each first group of wafer sample image pair into the corresponding teacher model to be trained to obtain the mapping feature prediction results of the two groups of wafer sample images included in each first group of wafer sample image pair.

[0387] The training device 1500 in this application embodiment further includes a determination module 1503, which is used to determine a teacher model based on the mapping feature prediction results of the first set of wafer sample images for the two sets of wafer sample images, and when the contrastive learning loss of the corresponding teacher model determined by the mapping feature prediction results satisfies the convergence condition of the teacher model.

[0388] In one possible implementation, the aforementioned teacher model to be trained includes a feature extraction network, a feature prediction network, and a feature mapping module.

[0389] The feature extraction network is used to extract features from each of the two sets of wafer sample images included in the first set of wafer sample images, and obtain the feature representation of each of the two sets of wafer sample images included in the first set of wafer sample images;

[0390] The feature prediction network is used to determine the feature prediction result of each first set of wafer sample images for the two included sets of wafer sample images based on the feature representation of each first set of wafer sample images for the two included sets of wafer sample images;

[0391] The feature mapping module is used to determine the mapping feature prediction result of each first set of wafer sample images to the two sets of included wafer sample images based on the feature prediction result of each first set of wafer sample images to the two sets of included wafer sample images.

[0392] In one possible implementation, multiple teacher models include a target teacher model and non-target teacher models, where the target teacher model is the teacher model corresponding to the target momentum parameter.

[0393] In one possible implementation, the communication module 1501 is also used to receive a first weight update instruction when the performance indicators of the current wafer defect detection model do not meet the requirements.

[0394] The determination module 1503 is also used to increase the knowledge transfer weight of the target teacher model in response to the first weight update instruction when the performance indicators of the current wafer defect detection model do not meet the requirements.

[0395] In one possible implementation, the communication module 1501 is also used to receive a second weight update instruction when the performance indicators of the current wafer defect detection model meet the requirements.

[0396] The determination module 1503 is also used to increase the knowledge transfer weight of the non-target teacher model in response to the second weight update instruction.

[0397] When the server is used to train a wafer defect detection model, the server in this embodiment of the application may further include a determination module 1503, and the functions of each module are as follows:

[0398] The communication module 1501 is used to acquire key information of multiple teacher models obtained through training. The training data of each teacher model includes the first set of wafer sample images.

[0399] Training module 1502 is used to determine the momentum parameters of each teacher model based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, and to train the corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model. Each second set of wafer sample images is a part of the training data of the corresponding teacher model.

[0400] The determination module 1503 is used to determine the target momentum parameter based on the performance indicators of each candidate student model and the momentum parameters of multiple teacher models, and to determine the wafer defect detection model based on the target candidate student model obtained from the teacher model corresponding to the target momentum parameter.

[0401] In some possible implementations, each first set of wafer sample images includes wafer sample images from at least one process stage in the semiconductor manufacturing process, and the wafer sample images included in different first sets of wafer sample images belong to different process stages;

[0402] The knowledge transfer weight of each teacher model is positively correlated with the wafer defect probability of the process stage to which each first set of wafer sample images belongs.

[0403] In some possible implementations, multiple teacher models include a target teacher model and non-target teacher models. The target teacher model is the teacher model corresponding to the target momentum parameter. The communication module 1501 is also used to determine whether the performance index of the current wafer defect detection model meets the requirements. When the requirements are not met, a first weight update instruction is sent to instruct the knowledge transfer weight of the target teacher model to be increased. When the requirements are met, a second weight update instruction is sent to instruct the knowledge transfer weight of the non-target teacher model to be increased.

[0404] In some possible implementations, the training module 1502 is used to determine the reward parameter of each teacher model based on the key information of each teacher model and the corresponding key information of the first pre-trained student model, and to determine the momentum parameter of each teacher model based on the reward parameter of each teacher model.

[0405] In some possible implementations, the first pre-trained student model includes a first branch network and a second branch network. The training module 1502 is used to perform data augmentation on each second set of wafer sample images to obtain a pair of second set wafer sample images; input each pair of second set wafer sample images into the first branch network to obtain the projection feature prediction results of the two sets of wafer sample images within each pair of second set wafer sample images; input each pair of second set wafer sample images into the second branch network to obtain the mapping feature prediction results of the two sets of wafer sample images within each pair of second set wafer sample images under the second branch network; determine the loss of the first pre-trained student model based on the projection feature prediction results and the mapping feature prediction results under the second branch network; wherein, when the loss of the first pre-trained student model satisfies the convergence condition of the first pre-trained student model, the first pre-trained student model is set as the candidate student model; when the loss of the first pre-trained student model does not satisfy the convergence condition of the first pre-trained student model, the backpropagation algorithm is used to update the first branch network, and the second branch network is updated based on the momentum parameters of the corresponding teacher model.

[0406] In some possible implementations, the first branch network includes a first prediction network and a feature projection network, and the second branch includes a second prediction network;

[0407] The first prediction network is used to determine the mapping feature prediction results of each set of wafer sample images under the first branch network based on the two sets of wafer sample images within each second set of wafer sample image pairs.

[0408] The feature projection network is used to determine the projection feature prediction results of each set of wafer sample images based on the mapping feature prediction results of the two sets of wafer sample images in each second set of wafer sample image pairs under the first branch network.

[0409] The second prediction network is used to determine the mapping feature prediction results of the two sets of wafer sample images within each second set of wafer sample image pairs under the second branch network, based on the two sets of wafer sample images within the second set of wafer sample image pairs.

[0410] In some possible implementations, the training module 1502 is also used to determine the loss of one set of wafer sample images based on the projection feature prediction result of one set of wafer sample images in each second set of wafer sample image pairs and the mapping feature prediction result of the other set of wafer sample images under the second branch network, and to determine the loss of the corresponding first pre-trained student model based on the loss of the two sets of wafer sample images in each second set of wafer sample image pairs.

[0411] In some possible implementations, the determining module 1503 is used to obtain multiple target student models that meet the first performance index requirements based on the performance index of each candidate student model, and to obtain the target momentum parameters from the momentum parameters of the teacher models corresponding to the multiple target student models.

[0412] In some possible implementations, the target momentum parameters include at least two target momentum parameters. The determining module 1503 is further configured to determine the target candidate student model based on the candidate student models corresponding to all target momentum parameters if all target momentum parameters correspond to the same candidate student model, and to determine the third set of wafer sample images based on the training data of the teacher models corresponding to all target momentum parameters if at least two target momentum parameters correspond to different candidate student models, and to train the first pre-trained student model based on all target momentum parameters and the third set of wafer sample images to obtain the target candidate student model.

[0413] In some possible implementations, the determining module 1503 is used to determine the momentum parameters of the target candidate student model based on the key information of the target candidate student model and the key information of the corresponding second pre-trained student model. Using the fourth set of wafer sample images as training data, the knowledge of the target candidate student model is distilled into the corresponding second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model. If the target student model meets the convergence condition of the target student model, the wafer defect detection model is determined based on the target student model. If the target candidate student model does not meet the convergence condition of the target student model, if the performance index of the target student model meets the second performance index requirement, the target candidate student model is updated based on the target student model.

[0414] In some possible implementations, the fourth set of wafer sample images used for different knowledge distillations includes wafer sample images that belong to at least different process stages in the semiconductor manufacturing process.

[0415] In some possible implementations, the determining module 1503 is used to determine the reward parameter of the target candidate student model based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, and to determine the momentum parameter of the target candidate student model based on the reward parameter of the target candidate student model.

[0416] In some possible implementations, the determining module 1503 is used to determine the reward parameters of the target candidate student model based on the key information of the target candidate student model and the key information of the second pre-trained student model, and the second server determines the momentum parameters of the target candidate student model based on the reward parameters of the target candidate student model.

[0417] In some possible implementations, the determining module 1503 is used to perform data augmentation on the fourth set of wafer sample images to obtain a fourth set of wafer sample image pairs; input the fourth set of wafer sample image pairs into the target candidate student model to obtain the feature similarity of the target candidate student model at the first distillation temperature; input the fourth set of wafer sample image pairs into the second pre-trained student model to obtain the feature similarity of the second pre-trained student model at the first target distillation temperature and the feature similarity of the second pre-trained student model at the second distillation temperature; based on the feature similarity of the target candidate student model at the first distillation temperature, the feature similarity of the second pre-trained student model at the first target distillation temperature, and the feature similarity of the second pre-trained student model at the second distillation temperature, determine whether the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model; when the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model, determine the target student model based on the second pre-trained student model; otherwise, update the second pre-trained student model based on the momentum parameters of the target candidate student model.

[0418] In some possible implementations, the determining module 1503 is further configured to determine the loss of the target candidate student model at the first distillation temperature based on the feature similarity of the target candidate student model at the first distillation temperature and the feature similarity of the second pre-trained student model at the first target distillation temperature; to determine the loss of the target candidate student model at the second distillation temperature based on the feature similarity of the second pre-trained student model at the second distillation temperature; and to determine the distillation loss of the second pre-trained student model based on the loss of the target candidate student model at the first distillation temperature and the loss of the target candidate student model at the second distillation temperature.

[0419] In some possible implementations, the teacher model described above is a teacher model based on a contrastive learning algorithm, and the first pre-trained student model and the second pre-trained student model are student models based on a contrastive learning algorithm.

[0420] Figure 16 A schematic block diagram of a chip according to an exemplary embodiment of this application is shown. Figure 16 As shown, the chip 1600 includes one or more processors 1601 and a communication interface 1602.

[0421] In one example, the communication interface 1602 can support the first server in performing the data transmission and reception steps in the above training method, and the processor 1601 can support the first server in performing the data processing steps in the above training method.

[0422] In another example, the communication interface 1602 can support the second server to perform the data transmission and reception steps in the above training method, and the processor 1601 can support the second server to perform the data processing steps in the above training method.

[0423] Optional, such as Figure 16 As shown, the chip 1600 also includes a memory 1603, which may include read-only memory and random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0424] In some implementations, such as Figure 16As shown, processor 1601 executes corresponding operations by calling operation instructions stored in memory (which may be stored in the operating system). Processor 1601 controls the processing operations of any terminal device; processor can also be called a central processing unit (CPU). Memory 1603 may include read-only memory and random access memory, and provides instructions and data to processor 1601. A portion of memory 1603 may also include NVRAM. For example, in applications, memory, communication interfaces, and other components are coupled together via a bus system, which may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for clarity, in... Figure 16 The general labeled all buses as Bus System 1604.

[0425] The methods disclosed in the embodiments of this application can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams of the application in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods applied in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0426] An exemplary embodiment of this application also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of this application.

[0427] An exemplary embodiment of this application also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this application.

[0428] An exemplary embodiment of this application also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of this application.

[0429] refer to Figure 17 The present invention describes a structural block diagram of an electronic device 1700 that can serve as a server or client of this application, which is an example of a hardware device that can be applied to various aspects of this application. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0430] like Figure 17 As shown, the electronic device 1700 includes a computing unit 1701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1702 or a computer program loaded from a storage unit 1708 into a random access memory (RAM) 1703. The RAM 1703 may also store various programs and data required for the operation of the device 1700. The computing unit 1701, ROM 1702, and RAM 1703 are interconnected via a bus 1704. An input / output (I / O) interface 1705 is also connected to the bus 1704.

[0431] like Figure 17As shown, multiple components in electronic device 1700 are connected to I / O interface 1705, including: input unit 1706, output unit 1707, storage unit 1708, and communication unit 1709. Input unit 1706 can be any type of device capable of inputting information to electronic device 1700. Input unit 1706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1708 may include, but is not limited to, disks and optical discs. Communication unit 1709 allows electronic device 1700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0432] like Figure 17 As shown, computing unit 1701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 1701 include, but are not limited to, central processing unit (CPU), graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 1701 performs the various methods and processes described above. For example, in some embodiments, the methods of the embodiments of this application can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1708. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 1700 via ROM 1702 and / or communication unit 1709. In some embodiments, computing unit 1701 can be configured to perform the methods of the embodiments of this application by any other suitable means (e.g., by means of firmware).

[0433] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0434] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0435] As used in this application, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0436] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0437] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0438] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0439] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0440] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

Claims

1. A training method, characterized in that, The method includes: Obtain key information of multiple teacher models obtained through training, wherein the training data of each teacher model includes a first set of wafer sample images; Based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the momentum parameters of each teacher model are determined. Based on the momentum parameters of each teacher model and each second set of wafer sample images, the corresponding first pre-trained student model is trained to obtain each candidate student model, and each second set of wafer sample images is part of the training data of the corresponding teacher model. Based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models, the target momentum parameter is determined. Based on the teacher model corresponding to the target momentum parameters, the target candidate student model is obtained, and the wafer defect detection model is determined.

2. The method according to claim 1, characterized in that, Each of the first set of wafer sample images includes wafer sample images from at least one process stage in the semiconductor manufacturing process, and the wafer sample images included in different first sets of wafer sample images belong to different process stages; The knowledge transfer weight of each teacher model is positively correlated with the wafer defect probability of the process stage to which each of the first set of wafer sample images belongs.

3. The method according to claim 2, characterized in that, The teacher model corresponding to the target momentum parameter is determined as the target teacher model, and the remaining teacher models are determined as non-target teacher models; The method further includes: Determine whether the performance indicators of the current wafer defect detection model meet the requirements; When the requirements are not met, a first weight update instruction is sent to instruct the knowledge transfer weight of the target teacher model to be increased. When the requirement is met, a second weight update instruction is sent to instruct the knowledge transfer weight of the non-target teacher model to be increased.

4. The method according to claim 1, characterized in that, Based on the key information of each teacher model and the corresponding key information of the first pre-trained student model, the momentum parameters of each teacher model are determined, including: Based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, the reward parameters of each teacher model are determined. Based on the reward parameters of each teacher model, the momentum parameters of each teacher model are determined.

5. The method according to claim 1, characterized in that, The first pre-trained student model includes a first branch network and a second branch network. Based on the momentum parameters of each teacher model and each second set of wafer sample images, the corresponding first pre-trained student model is trained to obtain each candidate student model, including: Data augmentation is performed on each of the second set of wafer sample images to obtain a pair of second set wafer sample images; Each pair of second-group wafer sample images is input into the first branch network to obtain the projection feature prediction results of the two sets of wafer sample images within each pair of second-group wafer sample images; Each pair of second-group wafer sample images is input into the second branch network to obtain the mapping feature prediction results of the two sets of wafer sample images in each pair of second-group wafer sample images under the second branch network; The loss of the first pre-trained student model is determined based on the projection feature prediction results and the mapping feature prediction results under the second branch network; wherein, when the loss of the first pre-trained student model satisfies the convergence condition of the first pre-trained student model, the first pre-trained student model is set as the candidate student model; when the loss of the first pre-trained student model does not satisfy the convergence condition of the first pre-trained student model, the first branch network is updated using the backpropagation algorithm, and the second branch network is updated based on the momentum parameters of the corresponding teacher model.

6. The method according to claim 5, characterized in that, The first branch network includes a first prediction network and a feature projection network, and the second branch network includes a second prediction network; The first prediction network is used to determine the mapping feature prediction result of each set of wafer sample images under the first branch network based on the two sets of wafer sample images within each pair of second set wafer sample images; The feature projection network is used to determine the projection feature prediction results of each set of wafer sample images based on the mapping feature prediction results of the two sets of wafer sample images within each second set of wafer sample image pairs under the first branch network. The second prediction network is used to determine the mapping feature prediction results of the two sets of wafer sample images in each of the second set of wafer sample image pairs under the second branch network, based on the two sets of wafer sample images within the second set of wafer sample image pairs.

7. The method according to claim 5, characterized in that, The step of training the corresponding first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model further includes: Based on the projection feature prediction result of one set of wafer sample images in each of the second set of wafer sample image pairs and the mapping feature prediction result of the other set of wafer sample images under the second branch network, the loss of the one set of wafer sample images is determined. Based on the loss corresponding to the two sets of wafer sample images within each pair of second set wafer sample images, the loss of the corresponding first pre-trained student model is determined.

8. The method according to claim 1, characterized in that, The determination of the target momentum parameter based on the performance metrics of each candidate student model and the momentum parameters of multiple teacher models includes: Based on the performance metrics of each candidate student model, multiple target student models that meet the requirements of the first performance metric are obtained. The target momentum parameters are obtained from the momentum parameters of the teacher models corresponding to the multiple target student models.

9. The method according to claim 1, characterized in that, The target momentum parameters include at least two target momentum parameters, and the method further includes: If all target momentum parameters correspond to the same candidate student model, the target candidate student model is determined based on the candidate student models corresponding to all target momentum parameters. If at least two of the target momentum parameters correspond to different candidate student models, a third set of wafer sample images is determined based on the training data of the teacher models corresponding to all target momentum parameters, and the first pre-trained student model is trained based on all target momentum parameters and the third set of wafer sample images to obtain the target candidate student model.

10. The method according to any one of claims 1 to 9, characterized in that, The target candidate student model obtained based on the teacher model corresponding to the target momentum parameter, and the wafer defect detection model are determined, including: Based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, the momentum parameters of the target candidate student model are determined. Using the fourth set of wafer sample images as training data, the knowledge of the target candidate student model is distilled into the corresponding second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model; If the target student model meets the convergence condition of the target student model, the wafer defect detection model is determined based on the target student model; if the target candidate student model does not meet the convergence condition of the target student model, if the performance index of the target student model meets the second performance index requirement, the target candidate student model is updated based on the target student model.

11. The method according to claim 10, characterized in that, The fourth set of wafer sample images used in the different knowledge distillation includes some wafer sample images that belong to at least different process stages in the semiconductor manufacturing process.

12. The method according to claim 10, characterized in that, Based on the key information of the target candidate student model and the corresponding key information of the second pre-trained student model, the momentum parameters of the target candidate student model are determined, including: Based on the key information of the target candidate student model and the key information of the corresponding second pre-trained student model, the reward parameters of the target candidate student model are determined. Based on the reward parameters of the target candidate student model, the momentum parameters of the target candidate student model are determined.

13. The method according to claim 10, characterized in that, The step of using the fourth set of wafer sample images as training data, and distilling the knowledge of the target candidate student model into the corresponding second pre-trained student model based on the momentum parameters of the target candidate student model to obtain the target student model includes: Data augmentation is performed on the fourth set of wafer sample images to obtain the fourth set of wafer sample image pairs; The fourth set of wafer sample images is input into the target candidate student model to obtain the feature similarity of the target candidate student model at the first distillation temperature. The fourth set of wafer sample images is input into the second pre-trained student model to obtain the feature similarity of the second pre-trained student model at the first target distillation temperature and the feature similarity of the second pre-trained student model at the second distillation temperature. Based on the feature similarity of the target candidate student model at the first distillation temperature, the feature similarity of the second pre-trained student model at the first target distillation temperature, and the feature similarity of the second pre-trained student model at the second distillation temperature, it is determined whether the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model; when the distillation loss of the second pre-trained student model satisfies the convergence condition of the second pre-trained student model, the target student model is determined based on the second pre-trained student model; otherwise, the second pre-trained student model is updated based on the momentum parameter of the target candidate student model.

14. A server, characterized in that, include: A communication module is used to acquire key information of multiple trained teacher models, wherein the training data of each teacher model includes a first set of wafer sample images; The training module is used to determine the momentum parameters of each teacher model based on the key information of each teacher model and the key information of the corresponding first pre-trained student model, and to train the first pre-trained student model based on the momentum parameters of each teacher model and each second set of wafer sample images to obtain each candidate student model, wherein each second set of wafer sample images is a portion of the training data of the corresponding teacher model. The determination module is used to determine the target momentum parameter based on the performance indicators of each of the candidate student models and the momentum parameters of multiple teacher models; Based on the teacher model corresponding to the target momentum parameter, the corresponding candidate student model is obtained, and the wafer defect detection model is determined.

15. An electronic device, characterized in that, include: processor; as well as, Memory for stored programs; The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 13.

16. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1 to 13.

17. A computer program product, characterized in that, Includes a computer program, wherein the computer program, when executed by a processor, implements the method of any one of claims 1 to 13.

Images