Privacy-Preserving Imbalanced Wafer Defect Pattern Recognition Method and Apparatus

Abstract

This invention provides a privacy-preserving method and apparatus for identifying unbalanced wafer defect patterns, relating to the field of pattern recognition technology. The method includes: a first client acquiring a first sample subset and a second sample subset from a sample dataset; performing data augmentation on sample wafer images for each defect pattern in the first sample subset, and downsampling on sample wafer images for each defect pattern in the second sample subset; performing federated learning training based on the connected topology matrix between the client and each second client, as well as the augmented first sample subset and the downsampled second sample subset, to obtain an optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework. This invention improves the robustness, generalization, and accuracy of wafer defect pattern recognition.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition technology, and in particular to a privacy-preserving method and apparatus for identifying unbalanced wafer defect patterns. Background Technology

[0002] As the fundamental raw material for the IC (Integrated Circuit Chip) and chip industry, wafers are made from high-purity semiconductors through extremely complex and rigorous processes such as polishing and slicing. These wafers then undergo a series of stringent manufacturing processes to form extremely small circuit structures, which are finally cut and packaged into integrated circuits and chips. The quality of the wafer directly affects the performance of subsequent ICs and chips. Therefore, wafer defect pattern recognition, as an important branch of computer vision guiding the intelligent manufacturing industry, has become a hot research area in semiconductors due to its practical significance in industrial production, promoting intelligent industrial transformation, guiding wafer production, and reducing the defect rate of ICs and chips.

[0003] In the field of wafer identification, although feature extraction based on convolutional neural networks has demonstrated impressive performance, wafer defect pattern recognition methods based on deep convolutional neural networks typically rely on long-term data collection and extensive manual annotation to obtain rich training sample datasets, thereby training a wafer defect pattern recognition model with good recognition performance.

[0004] The wafer manufacturing process is complex, and class imbalance is very common. Furthermore, due to potential conflicts of interest, data privacy management in various manufacturing industries is becoming increasingly strict, and defect data cannot be shared between different manufacturing industries. This results in insufficient training sample datasets and a severe imbalance in the number of samples for various defect patterns, which in turn leads to poor recognition performance of the trained wafer defect pattern recognition system, making it difficult to accurately identify wafer defect patterns. Summary of the Invention

[0005] This invention provides a privacy-preserving method and apparatus for identifying unbalanced wafer defect patterns. It addresses the shortcomings of existing technologies, such as insufficient training sample datasets and severe imbalance in the number of samples for various defect patterns, which make it difficult to accurately identify wafer defect patterns under data privacy constraints. This invention aims to improve the accuracy of wafer defect pattern identification while protecting data privacy.

[0006] This invention provides a privacy-preserving method for identifying unbalanced wafer defect patterns, comprising:

[0007] The first client obtains a first sample subset and a second sample subset from the sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value.

[0008] Data augmentation is performed on the sample wafer images of each defect mode in the first sample subset, and downsampling is performed on the sample wafer images of each defect mode in the second sample subset.

[0009] Based on the connectivity topology matrix between the client and each second client, and the augmented first sample subset and the downsampled second sample subset, federated learning is performed to obtain the optimal defect pattern recognition model. The optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image. The first client and the second client are located in the same federated learning framework.

[0010] According to the privacy-preserving method for identifying unbalanced wafer defects provided by the present invention, the method involves federated learning training based on the connected topology matrix between the wafer and each second client, as well as the augmented first sample subset and the downsampled second sample subset, to obtain an optimal defect pattern recognition model, including:

[0011] For the current model training, based on the connected topology matrix, the target client connected to the first client is obtained from multiple second clients;

[0012] Receive local model parameters obtained by each target client during the current model training process; the local model parameters are obtained by each target client through local training of its initial defect pattern recognition model based on its own locally stored sample dataset after data augmentation and downsampling, and the global model parameters obtained by the client connected to each target client during the previous model training process.

[0013] Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the local model parameters, the global model parameters corresponding to the current model training are obtained.

[0014] The global model parameters corresponding to the current model training are sent to each target client, and the model training steps are executed iteratively until the defect pattern recognition model corresponding to the global model parameters meets the preset termination condition.

[0015] The optimal defect pattern recognition model is constructed based on the global model parameters corresponding to the last model training.

[0016] According to a privacy-preserving method for identifying imbalanced wafer defect patterns, the step of obtaining global model parameters corresponding to the current model training based on the augmented first sample subset, the downsampled second sample subset, and the local model parameters includes:

[0017] Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the global model parameters corresponding to the previous model training stored locally, local training is performed to obtain the local model parameters corresponding to the current model training.

[0018] Based on the number of times each target client participates in federated learning training and the number of samples each target client participates in in the current model training, the degree of contribution of each target client to the current model training is obtained.

[0019] Based on the degree of contribution, the local model parameters and the local model parameters are aggregated and updated to obtain the global model parameters corresponding to the current model training.

[0020] According to a privacy-preserving method for identifying class-imbalanced wafer defect patterns provided by the present invention, the step of data augmentation of the sample wafer images for each type of defect pattern in the first sample subset includes:

[0021] Extract the image to be augmented from the sample wafer images of each defect mode in the first sample subset;

[0022] The image to be augmented is input into the encoder to obtain the feature map of the image to be augmented;

[0023] The feature map is input into the decoder to obtain a simulated image of the image to be augmented;

[0024] Based on the simulated images, data augmentation is performed on the sample wafer images of each defect mode in the first sample subset.

[0025] According to the present invention, a privacy-preserving method for identifying unbalanced wafer defect patterns is provided, wherein the encoder is constructed based on deformable convolutional layers and pooling layers;

[0026] The step of inputting the image to be augmented into the encoder to obtain the feature map of the image to be augmented includes:

[0027] The image to be augmented is input into the deformable convolutional layer to obtain the convolutional features output by the deformable convolutional layer;

[0028] The convolutional features are input into the pooling layer to obtain the feature map output by the pooling layer.

[0029] According to the present invention, a privacy-preserving method for identifying unbalanced wafer defect patterns is provided, wherein the decoder is constructed based on a first deconvolution layer, an upsampling layer, and a second deconvolution layer;

[0030] The step of inputting the feature map into the decoder to obtain a simulated image of the image to be augmented includes:

[0031] The feature map is input into the first deconvolution layer to obtain the deconvolution features output by the first deconvolution layer; the feature dimension of the deconvolution features is the same as the feature dimension of the feature map.

[0032] The deconvolutional features are input into the upsampling layer to obtain the upsampling features output by the upsampling layer; the feature dimension of the upsampling features is the same as the feature dimension of the convolutional features.

[0033] The upsampled features are input into the second deconvolution layer to obtain the simulated image output by the second deconvolution layer.

[0034] According to the present invention, a privacy-preserving method for identifying unbalanced wafer defects is provided, wherein the optimal defect pattern recognition model includes a visual transformer (VIT) model constructed based on a multi-head attention mechanism.

[0035] The VIT model includes an embedding layer, an encoding layer, and a classification layer;

[0036] The embedding layer is used to extract embedding features from the target wafer image and output the embedding matrix of the target wafer image;

[0037] The encoding layer is used to extract feature maps from the target wafer image based on the multi-head self-attention mechanism and the embedding matrix, and output the feature map of the target wafer image;

[0038] The classification layer is used to perform defect pattern recognition based on the feature map of the target wafer image and output the defect pattern recognition result of the target wafer image.

[0039] The present invention also provides a privacy-preserving device for identifying unbalanced wafer defect patterns, comprising:

[0040] The data acquisition module is used by the first client to obtain a first sample subset and a second sample subset from the sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value.

[0041] The data balancing module is used to augment the sample wafer images under each defect mode in the first sample subset and downsample the sample wafer images under each defect mode in the second sample subset.

[0042] The pattern recognition module is used to perform federated learning training based on the connected topology matrix between the module and each second client, as well as the first sample subset after data augmentation and the second sample subset after downsampling, to obtain an optimal defect pattern recognition model. The optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image. The first client and the second client are located in the same federated learning framework.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the privacy-preserving unbalanced wafer defect pattern recognition method as described above.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the privacy-preserving class-unbalanced wafer defect pattern recognition method as described above.

[0045] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the privacy-preserving class-unbalanced wafer defect pattern recognition method as described above.

[0046] The present invention provides a privacy-preserving method and apparatus for class-imbalanced wafer defect pattern recognition. On the one hand, it divides the sample dataset into a first sample subset and a second sample subset according to the number of samples in each defect pattern class. Data augmentation is performed on the sample wafer images of each defect pattern class in the first sample subset, and downsampling is performed on the sample wafer images of each defect pattern class in the second sample subset. This addresses the class imbalance problem, reduces overfitting in defect pattern recognition, and improves the robustness, generalization, and accuracy of defect pattern recognition. On the other hand, it achieves local model parameter sharing through a connected topology matrix with a second client. Federated learning training is then performed using the augmented first sample subset and the downsampling second sample subset to obtain the optimal defect pattern recognition model. This ensures data privacy while enabling the sharing and aggregation of model parameters among clients, solving the data silo problem and further improving the robustness, generalization, and accuracy of defect pattern recognition. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts of the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the simulation results of data balancing in the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0050] Figure 3 This is a schematic diagram of the federated learning framework in the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention.

[0051] Figure 4 This is a schematic diagram of the VIT model in the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0052] Figure 5 This is a schematic diagram of the coding layer structure in the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0053] Figure 6 This is one of the simulation results of the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0054] Figure 7 This is the second schematic diagram of the simulation results of the privacy-preserving unbalanced wafer defect pattern recognition method provided by the present invention;

[0055] Figure 8 This is a schematic diagram of the structure of the privacy-preserving unbalanced wafer defect pattern recognition device provided by the present invention;

[0056] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] The booming development of the Internet of Things and intelligent manufacturing is stimulating progress in the integrated circuit industry. As the fundamental raw material for the IC and chip industry, the quality of wafers directly affects the performance of ICs and chips. Wafer manufacturing is large-scale, complex, and costly, with a trend towards miniaturization. The wafer manufacturing process is intricate and precise; even minor changes can cause serious damage. Furthermore, wafer defects are difficult to avoid; even in cleanroom environments, processing variations still exist. Therefore, wafer surfaces are highly susceptible to defects. Identifying the sources of failure in production based on wafer surface defects and adjusting process parameters can effectively reduce the defect rate of wafers and their derivatives, such as integrated circuit chips, and improve chip reliability. Therefore, wafer defect pattern recognition has become a hot research area in semiconductor engineering.

[0059] In recent years, deep learning has made groundbreaking progress in computer vision, finding widespread applications in image classification, image localization and detection, and defect detection. While feature extraction based on convolutional neural networks has demonstrated impressive performance in wafer recognition, the increasing complexity of recognition tasks means that shallow neural networks are no longer effective at extracting deep features from images. Deep neural networks, such as the ResNet50 series of residual networks and the GoogleNet deep network, stack more convolutional modules to capture important feature information, leading to a dramatic increase in computational complexity with each convolutional operation, posing new challenges to computer hardware computing power. Furthermore, excellent wafer defect pattern recognition methods often rely on long-term data collection and extensive manual annotation, consuming significant human and time resources. Moreover, due to the complexity of wafer manufacturing processes, widespread class imbalance, and constraints imposed by data silos and data privacy, obtaining wafer labels and training samples is difficult, and the severe class imbalance further hinders accurate wafer defect pattern recognition.

[0060] Therefore, how to improve the accuracy of wafer defect modes while protecting data privacy, given the constraints of data privacy and security, limited communication bandwidth, and a severe imbalance in the number of samples for various defect modes, is one of the urgent problems to be solved in the industry.

[0061] To address the aforementioned problems, this invention provides a privacy-preserving method and apparatus for identifying unbalanced wafer defect patterns, which will be described below.

[0062] The following is combined Figure 1 This invention describes a privacy-preserving method for identifying unbalanced wafer defect patterns.

[0063] like Figure 1The diagram shown is one of the flowcharts illustrating the privacy-preserving method for identifying unbalanced wafer defects provided by this invention. This method can be applied to a federated learning framework that includes a first client and a second client. The federated learning framework contains multiple clients; the first client is any client within the federated learning framework, and the second client is any client in the federated learning framework other than the first client. The execution entity of this method is the first client within the federated learning framework.

[0064] The method specifically includes the following steps:

[0065] Step 101: The first client obtains a first sample subset and a second sample subset from the sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value.

[0066] Among them, the defect pattern classes include, but are not limited to, the defect-free pattern class (hereinafter referred to as None), the center defect pattern class (hereinafter referred to as Center), the ring defect pattern class (hereinafter referred to as Donut), the edge-local defect pattern class (hereinafter referred to as Edge-ring), the local defect pattern class (hereinafter referred to as Local), the near-full defect pattern class (hereinafter referred to as Near-full), the random defect pattern class (hereinafter referred to as Random), and the scratch defect pattern class (hereinafter referred to as Scratch). This embodiment does not specifically limit these.

[0067] The first preset value can be set according to the testing requirements and is the threshold value for marking defect pattern classes with small samples. If the number of samples under a certain defect pattern class is less than the first preset value, it indicates that the defect pattern class is a small sample class. If the number of samples under a certain defect pattern class is greater than or equal to the first preset value, it indicates that the defect pattern class is a non-small sample class.

[0068] Optionally, the first client can retrieve a sample dataset from local memory. This dataset contains sample wafer images and defect pattern labels for various defect pattern classes. The number of samples varies across different defect pattern classes, and there may be significant class imbalance.

[0069] To eliminate class imbalance in the sample dataset, we can first divide the small sample defect pattern class and the non-small sample defect pattern class for classification preprocessing, so as to achieve a basic consistency in the number of samples in each defect pattern, that is, to achieve class balance, thereby improving the accuracy of defect pattern recognition.

[0070] Optionally, the number of samples in each defect pattern category in the sample dataset is compared with a first preset value. A first sample subset is constructed based on the sample wafer images and defect pattern labels of the sample wafer images under the defect pattern class with a sample number less than the first preset value. A second sample subset is constructed based on the sample wafer images and defect pattern labels of the sample wafer images under the defect pattern class with a sample number greater than or equal to the first preset value. It is understood that the first sample subset contains multiple small sample defect pattern classes, and the second sample subset contains multiple non-small sample defect pattern classes.

[0071] Step 102: Perform data augmentation on the sample wafer images of each defect mode in the first sample subset, and downsample the sample wafer images of each defect mode in the second sample subset.

[0072] Optionally, to eliminate class imbalance in the sample dataset, data augmentation can be performed on the sample wafer images of each defect pattern in the first sample subset. This increases the number of samples for each defect pattern in the first sample subset, improves the learning ability of the defect pattern recognition model for that defect pattern, and thus enhances the robustness and accuracy of defect pattern recognition. Data augmentation includes, but is not limited to, image transformation and image simulation; this embodiment does not specifically limit these methods.

[0073] Furthermore, random downsampling is performed on the sample wafer images of each defect pattern in the second sample subset to reduce the number of samples in each defect pattern in the second sample subset. This means balancing the number of majority class samples to a number similar to the number of minority class samples after data augmentation, thereby reducing the overfitting of the defect pattern recognition model by majority class defect samples, and thus improving the robustness and accuracy of defect pattern recognition.

[0074] It is understood that the number of samples under each defect pattern in the first sample subset after data augmentation and the number of samples under each defect pattern in the second sample subset after downsampling are balanced, such as the number of samples under each defect pattern in the first sample subset after data augmentation and the number of samples under each defect pattern in the second sample subset after downsampling are both equal to the first preset value; or the number of samples under each defect pattern in the first sample subset after data augmentation and the number of samples under each defect pattern in the second sample subset after downsampling are both within the same preset value range. This embodiment does not specifically limit this.

[0075] like Figure 2 The image shows the distribution of the sample datasets before and after downsampling and data augmentation. The horizontal axis represents the defect pattern class, and the vertical axis represents the number of samples. Figure 2As can be seen, compared with the original sample dataset and the sample dataset that only underwent downsampling, the number of samples in each defect pattern after downsampling and data augmentation in this example is basically balanced.

[0076] Step 103: Based on the connectivity topology matrix between the client and each second client, and the augmented first sample subset and the downsampled second sample subset, federated learning training is performed to obtain the optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework.

[0077] The optimal defect pattern recognition model can be built and generated based on various deep learning models such as multi-head attention mechanism, embedding layer, encoding layer and classification layer. This embodiment does not make specific limitations on this.

[0078] The connected topology matrix is ​​used to simulate the interconnected topology between the first client and each of the second clients. For example... Figure 3 The diagram shown illustrates the communication topology between clients in a federated learning framework. Figure 3 It can be seen that the connected topology matrix is ​​generated based on the time-varying communication topology between the first client and each second client. That is, during each model training process, the interconnection relationship between the first client and each second client changes in real time according to their respective performance or user settings.

[0079] In the federated learning framework, the initial defect pattern recognition model and the sample dataset are stored in the local memory of both the first client and the local memory of both clients; the first client and the second client jointly participate in federated learning.

[0080] Optionally, the first client can share local model parameters with the second client through a connected topology matrix, and combine the local model parameters shared by the second client with the first sample subset after data augmentation and the second sample subset after downsampling to perform federated learning training on the locally stored initial defect pattern recognition model, so as to obtain the optimal defect pattern recognition model that can accurately capture wafer defect features and accurately output the defect pattern class of the wafer based on the wafer defect features.

[0081] After obtaining the optimal defect pattern recognition model, the optimal defect pattern recognition model can be used to perform defect pattern recognition on the target wafer image to obtain the defect pattern recognition result of the target wafer image, that is, to determine the defect pattern class to which the target wafer belongs, such as the no-defect pattern class, the center defect pattern class, the annular defect pattern class, the edge local defect pattern class, the edge annular defect pattern class, the local defect pattern class, the near-full defect pattern class, the random defect pattern class, or the scratch defect pattern class.

[0082] The privacy-preserving class-imbalanced wafer defect pattern recognition method provided in this embodiment addresses the class imbalance problem by dividing the sample dataset into a first subset and a second subset based on the number of samples for each defect pattern class. Data augmentation is performed on the sample wafer images for each defect pattern class in the first subset, while downsampling is performed on the sample wafer images for each defect pattern class in the second subset. This reduces overfitting in defect pattern recognition and improves its robustness, generalization, and accuracy. Furthermore, by using a connected topology matrix with a second client, local model parameters are shared. Federated learning training is then performed using the augmented first subset and the downsampled second subset to obtain the optimal defect pattern recognition model. This approach ensures data privacy while enabling the sharing and aggregation of model parameters among clients, solving the data silo problem and further improving the robustness, generalization, and accuracy of defect pattern recognition.

[0083] In some embodiments, the federated learning training based on the connectivity topology matrix between each second client, and the augmented first sample subset and the downsampled second sample subset, to obtain the optimal defect pattern recognition model includes:

[0084] For the current model training, based on the connected topology matrix, the target client connected to the first client is obtained from multiple second clients;

[0085] Receive local model parameters obtained by each target client during the current model training process; the local model parameters are obtained by each target client through local training of its initial defect pattern recognition model based on its own locally stored sample dataset after data augmentation and downsampling, and the global model parameters obtained by the client connected to each target client during the previous model training process.

[0086] Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the local model parameters, the global model parameters corresponding to the current model training are obtained.

[0087] The global model parameters corresponding to the current model training are sent to each target client, and the model training steps are executed iteratively until the defect pattern recognition model corresponding to the global model parameters meets the preset termination condition.

[0088] The optimal defect pattern recognition model is constructed based on the global model parameters corresponding to the last model training.

[0089] In the federated learning framework, undirected topological graphs are used to represent data communication in real-world scenarios, enabling clients to share model parameters and achieve federated learning. This means that data sharing no longer depends on the existence of a central server. Each connected client is both a local model training end and a data aggregation end for federated learning.

[0090] Optionally, the training steps for federated learning are as follows:

[0091] For each of the first and second clients, a defect pattern recognition model and a sample dataset containing the first and second sample subsets are deployed in their local memory; the number and type of samples in each client's sample dataset are different. The number of clients in the federated learning framework can be set according to actual needs, such as K, but this embodiment does not specifically limit this.

[0092] For the current model training iteration, the first client can determine the second client interconnected with it during the training process based on the connected topology matrix, and use that client as the target client. It should be noted that the connected topology matrix can be changed in real-time based on client performance and user settings for each model training iteration.

[0093] Then, the first client communicates with each target client to receive the global model parameters obtained by other clients (such as the first client and other second clients) during the previous model training process, as well as the local model parameters obtained by local iterative training of the initial defect pattern recognition model stored locally using the sample dataset after data augmentation and downsampling.

[0094] The first client can perform global training or aggregation updates based on the local model parameters and the augmented first sample subset and downsampled second sample subset of locally stored data to obtain the global model parameters corresponding to the current model training.

[0095] Global training may include the following steps:

[0096] The sample dataset D is based on the local storage of the first client k. k The local loss function on can be characterized as Where M is the initial defect pattern recognition model stored locally by the first client k, and G k Let be the local model parameters stored locally by the first client k. If we use the parameters ω of model M in units, then the global loss function for all clients in federated learning can be characterized as: The initial defect pattern recognition model stored locally on the first client k is globally trained based on the global loss function, so as to obtain the global model parameters corresponding to the current model training based on the training results.

[0097] The aggregate update may include the following steps:

[0098] The first client k can perform local training on the initial defect pattern recognition model stored locally, based on the augmented first sample subset and the downsampled second sample subset, as well as the global model parameters corresponding to the previous model training, to obtain local model parameters; then, the local model parameters and local model parameters are aggregated and updated to obtain the global model parameters corresponding to the current model training.

[0099] Alternatively, after receiving the local model parameters transmitted by each second client, the global and local model parameters corresponding to the previous model training of the first client k are weighted and summed. The weighted sum is used as the initial model parameters for the current model training. Based on the initial model parameters and the first client k's performance on the sample dataset D... k The local loss function is updated to obtain the global model parameters corresponding to the current model training iteration. The specific calculation formula is as follows:

[0100]

[0101] Among them, w k ′(t+1) represents the global model parameters obtained by the first client k during the (t+1)th model training iteration, W kj Let w be the adjacency matrix between the first client k and the second client j, used to characterize the communication between the first client k and the second client j. j (t) represents the local model parameters obtained during the t-th model training process transmitted by the second client j. Let k be the learning rate of the first client k during the t-th model training iteration. For the t-th model training, the first client k is on the sample dataset D k The local loss function on, w k (t) represents the local model parameters obtained by the first client k during the t-th model training iteration.

[0102] It should be noted that the local model parameters under initial conditions The choice is arbitrary. The specific update rule for the first client during the federated learning training process is as follows: it runs its local training algorithm locally, such as gradient descent, and shares the globally acquired model parameters with the second client it communicates with, according to the connected topology matrix. Once the first client receives the local weights and local gradients from the second client it communicates with, it aggregates and updates the received local weights and local gradients, and generates its aggregated weights in the next iteration to obtain the global model parameters.

[0103] After obtaining the global model parameters corresponding to the current model training, the first client sends the global model parameters corresponding to the current model training to each target client and iteratively executes the model training steps until the defect pattern recognition model corresponding to the global model parameters meets the preset termination conditions. The preset termination conditions include, but are not limited to, the number of model training times being greater than the preset number, model convergence, or wafer defect pattern recognition performance meeting the preset performance, etc. This embodiment does not specifically limit these conditions.

[0104] Then, based on the global model parameters corresponding to the last model training, the optimal defect pattern recognition model is constructed.

[0105] In this embodiment, the first client directly shares model parameters with the second client through a connected topology matrix to achieve federated learning training and obtain an optimal defect pattern recognition model that can accurately identify wafer defect patterns. Throughout the federated learning training process, there is no need to aggregate and transmit third-party central clients in any iteration, effectively solving the current situation of data silos and the problems of privacy protection and dependence on central servers. Furthermore, based on federated learning, data communication and model aggregation are simulated based on a time-varying communication topology graph, further reducing the high communication pressure and data privacy leakage risks of traditional federated learning central servers.

[0106] In some embodiments, obtaining the global model parameters corresponding to the current model training based on the augmented first sample subset, the downsampled second sample subset, and the local model parameters includes:

[0107] Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the global model parameters corresponding to the previous model training stored locally, local training is performed to obtain the local model parameters corresponding to the current model training.

[0108] Based on the number of times each target client participates in federated learning training and the number of samples each target client participates in in the current model training, the degree of contribution of each target client to the current model training is obtained.

[0109] Based on the degree of contribution, the local model parameters and the local model parameters are aggregated and updated to obtain the global model parameters corresponding to the current model training.

[0110] Optionally, after obtaining the local model parameters obtained by each target client during the current model training process, the first client can perform local training on the locally stored initial defect pattern recognition model based on the first sample subset after data augmentation, the second sample subset after downsampling, and the global model parameters corresponding to the previous model training, to obtain the local model parameters corresponding to the current model training; and jointly determine the contribution of each target client to the current model training based on the number of times each target client participates in federated learning training and the number of samples participating in the current model training.

[0111] For example, we can calculate the first ratio of the number of times each target client currently participates in federated learning training to the total number of times all clients currently participate in federated learning training, and calculate the second ratio of the number of samples each target client participates in the current model training to the total number of samples all clients participate in the current model training. We can then add the first and second ratios together with weights to obtain the degree of contribution.

[0112] Then, based on the degree of contribution, the local model parameters are weighted and added together with the local model parameters transmitted by each target client to obtain the global model parameters, and then the optimal defect pattern recognition model that can accurately identify wafer defect pattern classes is trained.

[0113] In this implementation, a contribution mechanism is established based on the number of times each target client participates in federated learning training and the number of samples each target client participates in the current model training. This cuts off the possibility of multiple clients communicating and connecting on the basis of server trust, enhances the fairness and enthusiasm among clients jointly building a shared model in federated learning, and thus enhances the privacy protection effect of federated learning and the accuracy of wafer defect pattern recognition.

[0114] In some embodiments, step 102, which involves data augmentation of the sample wafer images for each type of defect mode in the first sample subset, includes:

[0115] Extract the image to be augmented from the sample wafer images of each defect mode in the first sample subset;

[0116] The image to be augmented is input into the encoder to obtain the feature map of the image to be augmented;

[0117] The feature map is input into the decoder to obtain a simulated image of the image to be augmented;

[0118] Based on the simulated images, data augmentation is performed on the sample wafer images of each defect mode in the first sample subset.

[0119] The encoder is used to extract deep features from images to accurately learn various defect patterns. The encoder's structure can be customized according to specific needs, such as being built based on convolutional layers, deformable convolutional layers, pooling layers, etc.

[0120] The decoder is used to decode and reconstruct the image to generate the corresponding analog image. The encoder structure can be configured according to actual needs, such as being built based on deconvolutional layers, upsampling layers, etc.

[0121] Optionally, during the data augmentation process, at least one sample wafer image can be randomly selected from the sample wafer images of each defect mode in the first sample subset as the image to be augmented.

[0122] The image to be augmented is input into the encoder, which performs feature encoding on the image to be augmented in order to extract the feature map of the image to be augmented.

[0123] Based on this, the decoder is used to decode the feature map encoded by the encoder, and random noise is combined to reconstruct a simulated image to generate the image to be augmented.

[0124] Then, based on the simulated image of the image to be augmented, data augmentation is performed on the sample wafer images of each defect pattern in the first sample subset, so that each defect pattern in the first sample subset has sufficient sample data, thereby solving the current situation of wafer class imbalance, improving the training performance of the defect pattern recognition model, and thus improving the robustness and accuracy of wafer defect pattern recognition.

[0125] In this embodiment, by using an encoder to extract depth feature maps from the image to be augmented, and using a decoder to decode the feature maps encoded by the encoder, combined with random noise, a rich simulated image is reconstructed, which effectively alleviates class imbalance, reduces feature learning overfitting, and thus improves the robustness and accuracy of wafer defect pattern recognition.

[0126] In some embodiments, the encoder is constructed based on deformable convolutional layers and pooling layers;

[0127] The step of inputting the image to be augmented into the encoder to obtain the feature map of the image to be augmented includes:

[0128] The image to be augmented is input into the deformable convolutional layer to obtain the convolutional features output by the deformable convolutional layer;

[0129] The convolutional features are input into the pooling layer to obtain the feature map output by the pooling layer.

[0130] The encoder consists of at least one deformable convolutional layer and one pooling layer stacked in sequence.

[0131] The deformable convolutional layer effectively expands the receptive field of view and accurately samples defect areas by adding an offset matrix to the sampling positions in the standard convolution operation, thereby outputting the convolutional features of the image to be augmented. The specific calculation formula is as follows:

[0132]

[0133]

[0134]

[0135]

[0136] Where y out It is the convolutional feature output by the deformable convolutional layer, where R is the set of sampling points at all locations obtained by regular convolution, and p n For the sampling point at position n, offset(Δp) n ) represents sampling point p n The offset at point β, where β is the sampling point at any location, i.e., β = p i +p n +offset(Δp n X(·) is the feature map after convolution; X(β) is the feature vector of sampling point β after convolution; α is the enumeration of the set of sampling points at all positions; G(α, β) is the bilinear insertion kernel, α x and α y These represent the spatial positions of sampling point α along the x-axis and y-axis, respectively; β x and β y These represent the spatial positions of sampling point β along the x-axis and y-axis, respectively.

[0137] Then, the convolutional features are input into the pooling layer, which performs pooling operations on the convolutional features to obtain the feature map of the sample wafer image.

[0138] Table 1. Encoder Parameter Table

[0139] Layer name Output scale Input layer 96×96×3 Standard convolutional layer 96×96×32 Deformable convolutional layers 96×96×32 Pooling layer 48×48×32

[0140] It should be noted that the scale of the image to be augmented input to the encoder, the scale of the convolutional features output by the deformable convolutional layer, and the scale of the feature map output by the pooling layer can be set according to actual needs. For example, as shown in Table 1, the scale of the image to be augmented can be set to 96×96×3, the scale of the convolutional features output by the deformable convolutional layer can be 96×96×32, and the scale of the feature map output by the pooling layer can be 48×48×32.

[0141] In this embodiment, the offset matrix introduced in the deformable convolutional layer effectively expands the receptive field and improves the feature extraction capability, enabling better capture of the defect features of the sample wafer image, thereby better simulating and generating the corresponding simulated wafer image, better solving the class imbalance problem of the sample dataset, reducing the overfitting phenomenon of feature learning, and improving the accuracy of wafer defect pattern recognition.

[0142] In some embodiments, the decoder is constructed based on a first deconvolution layer, an upsampling layer, and a second deconvolution layer;

[0143] The step of inputting the feature map into the decoder to obtain a simulated image of the image to be augmented includes:

[0144] The feature map is input into the first deconvolution layer to obtain the deconvolution features output by the first deconvolution layer; the feature dimension of the deconvolution features is the same as the feature dimension of the feature map.

[0145] The deconvolutional features are input into the upsampling layer to obtain the upsampling features output by the upsampling layer; the feature dimension of the upsampling features is the same as the feature dimension of the convolutional features.

[0146] The upsampled features are input into the second deconvolution layer to obtain the simulated image output by the second deconvolution layer.

[0147] The decoder includes at least a first deconvolution layer, an upsampling layer, and a second deconvolution layer stacked in sequence, used to decode the feature map output by the encoder to obtain a reconstructed analog image.

[0148] Optionally, the first deconvolution layer is used to perform a deconvolution operation on the feature map output by the encoder to output deconvolution features with the same feature dimensions as the feature map. For example, the feature dimensions of the deconvolution features and the feature dimensions of the feature map output by the encoder are both 48×48×32.

[0149] The upsampling layer is used to upsample the deconvolutional features output by the first deconvolutional layer, so as to output upsampled features with the same feature dimensions as the convolutional features output by the deformable convolutional layer. For example, the feature dimensions of the upsampled features and the feature dimensions of the convolutional features output by the deformable convolutional layer are both 96×96×32.

[0150] The second deconvolutional layer performs a deconvolution operation on the upsampled features output by the upsampled layer to reconstruct the image to be augmented, i.e., outputting a simulated image with the same feature dimensions as the image to be augmented. For example, the feature dimensions of the upsampled features and the convolutional features output by the deformable convolutional layer are both 96×96×3.

[0151] Table 2 Parameter Table of Decoder

[0152] Layer name Output scale First deconvolutional layer 48×48×32 upsampling layer 96×96×32 First deconvolutional layer 96×96×3

[0153] In this embodiment, by combining a convolutional autoencoder with deformable convolution, the corresponding simulated wafer images can be effectively generated, which can better solve the problem of class imbalance in the sample dataset, reduce the overfitting phenomenon of feature learning, and improve the accuracy of wafer defect pattern recognition.

[0154] In some embodiments, the optimal defect pattern recognition model includes a Visual Transformer (VIT) model built based on a multi-head attention mechanism;

[0155] The VIT model includes an embedding layer, an encoding layer, and a classification layer;

[0156] The embedding layer is used to extract embedding features from the target wafer image and output the embedding matrix of the target wafer image;

[0157] The encoding layer is used to extract feature maps from the target wafer image based on the multi-head self-attention mechanism and the embedding matrix, and output the feature map of the target wafer image;

[0158] The classification layer is used to perform defect pattern recognition based on the feature map of the target wafer image and output the defect pattern recognition result of the target wafer image.

[0159] The optimal defect pattern recognition model includes the VIT (VisionTransformer) model, which is built based on a multi-head attention mechanism. This model is used to establish the mapping relationship between wafer images and defect pattern labels.

[0160] The embedding layer includes a class embedding layer, a position embedding layer, and a patch embedding layer. The class embedding layer is used to label classification information; the patch embedding layer is used to transform the target wafer image to divide the target wafer image into a series of two-dimensional planes (patches); and the position embedding layer is used to record the position information of the patches.

[0161] The two-dimensional planar embedding layer is constructed and generated based on two-dimensional convolutional sub-layers and linear mapping sub-layers.

[0162] like Figure 4 As shown, the coding layer contains multiple coding blocks, such as three; Figure 5 As shown, each encoding block can be based on a stacked attention layer and a multilayer perceptron layer. The attention layer includes a regularized linear sublayer, a multi-head attention mechanism sublayer, and a random deactivation sublayer. The multilayer perceptron layer includes a regularized linear sublayer, a multilayer perceptron sublayer, and a random deactivation sublayer. The output of the attention layer or the input of the multilayer perceptron layer is generated by fusing the input information of the attention layer and the output information of the random deactivation sublayer. The output of the multilayer perceptron layer is generated by fusing the output information of the attention layer and the output information of the random deactivation sublayer. The multilayer perceptron sublayer is constructed from multiple perceptron networks stacked together, consisting of a linear sublayer, a Gaussian error linear unit activation function, and a random deactivation sublayer.

[0163] The classification layer is generated based on the multilayer perceptron sublayer.

[0164] Optionally, in the defect pattern recognition process, the target wafer image can be divided into a series of two-dimensional planes through the two-dimensional plane embedding layer in the ViT model. Since image positional information is lost during the data division into patches, positional and category information is added to the two-dimensional planes based on the positional and category embedding layers in ViT. Then, the patches pass through a linear layer to output a learnable embedding matrix, which is input into the encoding layer. A multi-head self-attention mechanism combined with the linear layer captures important feature information and outputs a feature map. Finally, defect pattern recognition is performed based on the classification layer to obtain the defect pattern recognition result of the target wafer image. The loss function chosen for ViT is the Sparse Categorical Cross Entropy (SCCE) function. The formula for ViT is as follows:

[0165]

[0166] z l ′=MHSA(LN(zl-1 ))+z l-1 , l = 1·L;

[0167] z l =MLP(LN(z) l ′))+z l ′,l=1,…,L;

[0168]

[0169]

[0170] Where E is a fully connected layer; E pos For location embedding layer; MLP is a multilayer perceptron layer, MHSA is an attention layer, LN is a linear layer, and z0 is the embedding matrix output after the category embedding layer, location embedding layer, and 2D plane embedding layer; z l ′ represents the feature map output after the attention layer of layer l; z t The feature map is the output of the coding block after the l-th layer; L is the number of coding blocks; The output of the classification layer is the wafer defect pattern recognition result; This is the feature map output by the last encoded block. Attention is a multi-head self-attention sub-layer; Q, K, and V are the query vector, key vector, and value vector of the multi-head self-attention sub-layer, respectively; d = D / k is a constraint parameter used to ensure that the number of computations and parameters remains unchanged when k changes; D is the dimension of the patch embedding, k is the number of heads in the multi-head attention mechanism, and Softmax is the activation function.

[0171] In this embodiment, the VIT model is used to extract the intrinsic features of the target wafer image through a multi-head self-attention mechanism, achieving high-precision detection of pattern recognition tasks under high-dimensional and complex defect features without relying on convolution stacking.

[0172] Based on a specific example, the complete process of the privacy-preserving unbalanced wafer defect pattern recognition method in this embodiment is described, and the specific steps include:

[0173] Step (1) Data augmentation is performed on the sample wafer images of the minority class (i.e., the first sample subset) under class imbalance. Specifically, a convolutional autoencoder combined with deformable convolution is used to reconstruct the sample wafer images to generate simulated images, thereby augmenting the sample wafer images.

[0174] Step (2) involves downsampling the sample wafer images of the majority class (i.e., the second sample subset) to balance the number of majority class samples to a number similar to that of the minority class after data augmentation.

[0175] Step (3) involves using a ViT model based on a multi-head attention mechanism to perform defect pattern recognition on the balanced sample dataset, thereby establishing a mapping relationship between wafer images and defect labels. The training steps for the defect pattern recognition model include steps (31) and (32), as follows:

[0176] Step (31): The first client uses the locally balanced sample dataset to train the ViT model locally.

[0177] Step (32): Based on the connected topology matrix, determine the communication relationship between the first client and each second client, so that the interconnected clients can perform global model fusion, and the non-connected clients do not participate in global model fusion, so as to obtain the optimal defect pattern recognition model.

[0178] Step (4): Based on the optimal defect pattern recognition model, perform defect pattern recognition on the target wafer image and visualize the defect pattern recognition results.

[0179] In summary, the wafer defect pattern recognition method proposed in this embodiment effectively expands the receptive field by introducing an offset matrix. Furthermore, it effectively addresses the class imbalance problem and reduces overfitting during feature learning by using a deformable convolutional network training method combined with random noise simulation to generate minority class defect samples. In addition, the VIT model based on an attention mechanism achieves high-precision detection of pattern recognition tasks with high-dimensional and complex defect features without relying on convolution stacking. Finally, it simulates data communication and model aggregation based on a time-varying communication topology graph, further reducing the high communication pressure and data privacy leakage risks associated with traditional federated learning center servers, thereby further improving the accuracy of wafer defect pattern recognition.

[0180] To further verify the effectiveness of the privacy-preserving imbalanced wafer defect pattern recognition method (hereinafter referred to as DCAE-VIT) provided in this embodiment, the widely used public dataset WM-811K in the field of wafer defect pattern recognition was selected. The WM-811K dataset consists of 811,457 wafer images collected by experts in industrial production. The WM-811K dataset records information such as defect category, production batch, chip size, and image pixel count for the relevant wafer images. A total of nine patterns are included in all wafer images, covering defect-free patterns, center defect patterns, circular defect patterns, edge local defect patterns, edge ring defect patterns, local defect patterns, near-full defect patterns, random defect patterns, and scratch defect patterns. Among the 811,457 wafer images in the dataset, 78.7% of the wafers are unlabeled, 3.1% of the wafers have actual defect patterns, and 18.2% of the wafers have no category label. Therefore, 14,312 labeled images were randomly selected to form the experimental dataset. Of the images, 13,436 were defect-free wafers, accounting for 93.87%, while images of other defect modes accounted for 6.13%, indicating a significant class imbalance. The majority of normal wafer images were downsampled to 436 images. For the minority defect samples, image reconstruction was performed using deformable convolution and an autoencoder. The optimizer chosen was Adam (Adaptive Moment Estimation), with 40 training epochs, a batch size of 20, and the mean squared error loss function. The specific calculation formula is as follows:

[0181]

[0182] Among them, L DCAE (·) is the loss function, y is the true label, y′ is the predicted label value, and n is the number of classification categories.

[0183] To fully verify the effectiveness of the defect pattern recognition method provided in this embodiment, the following ablation studies were conducted, divided into five cases: (1) the performance of the defect pattern recognition model trained under imbalanced data; (2) the performance of the defect pattern recognition model trained only with downsampled data; (3) the performance of the defect pattern recognition model trained with data augmentation and downsampled data; (4) the performance of the defect pattern recognition model obtained under different recognition algorithms; and (5) the performance of the defect pattern recognition model trained by centralized learning, ordinary federated learning, and decentralized federated learning.

[0184] Specifically, the following defect pattern recognition methods are compared: (1) a defect pattern recognition method that trains the VIT model on imbalanced data (Unbalanced Dataset+VIT); (2) a defect pattern recognition method that trains the Inception V3 asymmetric convolution model on a sample dataset augmented and downsampled by Deformable Convolution (DC) and Autoencoder (AE) (DCAE+Inception). (3) A defect pattern recognition method for a defect pattern recognition model trained on the Visual Geometric Group Network VGG16 using a sample dataset augmented and downsampled by deformable convolution and autoencoder (abbreviated as DCAE+VGG16); (4) A defect pattern recognition method for a defect pattern recognition model trained on the ResNet50 series of residual networks using a sample dataset augmented and downsampled by deformable convolution and autoencoder (abbreviated as DCAE+ResNet50); (5) A defect pattern recognition method for a defect pattern recognition model trained on the ResNet50 series of residual networks using a sample dataset augmented and downsampled by deformable convolution and autoencoder (abbreviated as DCAE+ResNet50); (6) A defect pattern recognition method for a defect pattern recognition model trained on the EfficientNet series of efficient networks using a sample dataset augmented and downsampled by deformable convolution and autoencoder (abbreviated as DCAE+EfficientNet).

[0185] like Figure 6 The image shown is one of the simulation diagrams of the privacy-preserving, unbalanced wafer defect pattern recognition method proposed in this application. The horizontal axis represents the number of iterations (Epochs), and the vertical axis represents the accuracy. Figure 6 It can be seen that the defect pattern recognition results obtained by training the model on balanced data have a higher accuracy than those obtained by training the algorithm on imbalanced data. Although the model trained on imbalanced data converges in accuracy faster, it gets stuck in local optima for the majority class, resulting in overfitting. The overall accuracy stops increasing after reaching 84%. The model trained on balanced data shows a significant improvement in both classification performance and accuracy.

[0186] like Figure 7 The image shown is a simulation diagram of the privacy-preserving unbalanced wafer defect pattern recognition method proposed in this application.

[0187] Figure 7The study includes multiple sets of control experiments, where N1, N2, N3, and N4 represent the number of clients participating in federated learning, respectively, being 1, 2, 3, and 4. DeecFL represents the accuracy of the defect pattern recognition model obtained by weighted aggregation and updating a decentralized distributed adjacency matrix, as proposed in this embodiment. Centralized Learning represents the accuracy of the defect pattern recognition model obtained through centralized learning. FedAvg represents the accuracy of the defect pattern recognition model obtained through centralized federated learning.

[0188] Depend on Figure 7 The comparison results in Table 3 demonstrate that the decentralized distributed adjacency matrix iterative training proposed in this application can achieve training results similar to traditional centralized federated learning and centralized learning. It should be noted that although both federated learning algorithms and centralized learning have achieved satisfactory performance, and decentralized federated learning has a longer convergence time, the decentralized federated learning algorithm can obtain more reliable results while avoiding data leakage from central servers and communication barriers between users in real-world scenarios. This is beneficial for solving data privacy and data silo problems in real-world scenarios.

[0189] Table 3 Wafer defect identification performance under different training strategies

[0190]

[0191] The privacy-preserving unbalanced wafer defect pattern recognition device provided by the present invention will be described below. The privacy-preserving unbalanced wafer defect pattern recognition device described below can be referred to in correspondence with the privacy-preserving unbalanced wafer defect pattern recognition method described above.

[0192] like Figure 8 As shown, this embodiment provides a privacy-preserving, unbalanced wafer defect pattern recognition device, which includes:

[0193] The data acquisition module 801 is used by the first client to obtain a first sample subset and a second sample subset from the sample dataset; the number of samples in each defect mode in the first sample subset is less than a first preset value, and the number of samples in each defect mode in the second sample subset is greater than or equal to the first preset value.

[0194] The data balancing module 801 is used to perform data augmentation on the sample wafer images of each type of defect mode in the first sample subset, and to perform downsampling on the sample wafer images of each type of defect mode in the second sample subset.

[0195] The pattern recognition module 803 is used to perform federated learning training based on the connected topology matrix between the module and each second client, as well as the first sample subset after data augmentation and the second sample subset after downsampling, to obtain an optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework.

[0196] The privacy-preserving class-imbalanced wafer defect pattern recognition device provided in this embodiment addresses the class imbalance problem by dividing the sample dataset into a first sample subset and a second sample subset based on the number of samples for each defect pattern class. Data augmentation is performed on the sample wafer images for each defect pattern class in the first sample subset, while downsampling is performed on the sample wafer images for each defect pattern class in the second sample subset. This reduces overfitting in defect pattern recognition and improves its robustness, generalization, and accuracy. Furthermore, by using a connected topology matrix with a second client, local model parameters are shared. Federated learning training is then performed using the augmented first sample subset and the downsampled second sample subset to obtain the optimal defect pattern recognition model. This approach ensures data privacy while enabling the sharing and aggregation of model parameters among clients, solving the data silo problem and further improving the robustness, generalization, and accuracy of defect pattern recognition.

[0197] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9As shown, the electronic device may include: a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904. Processor 901 can call logic instructions in memory 903 to execute a privacy-preserving class-unbalanced wafer defect pattern recognition method. The method includes: a first client obtaining a first sample subset and a second sample subset from a sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value; data augmentation is performed on the sample wafer images in each defect pattern of the first sample subset, and downsampling is performed on the sample wafer images in each defect pattern of the second sample subset; federated learning training is performed based on the connected topology matrix between the processor and each second client, as well as the data-augmented first sample subset and the downsampling second sample subset, to obtain an optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework.

[0198] Furthermore, the logical instructions in the aforementioned memory 903 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the privacy-preserving class-unbalanced wafer defect pattern recognition method provided by the above methods. The method includes: a first client obtaining a first sample subset and a second sample subset from a sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value; performing data augmentation on the sample wafer images in each defect pattern in the first sample subset, and downsampling on the sample wafer images in each defect pattern in the second sample subset; performing federated learning training based on the connected topology matrix between the first client and each second client, as well as the data-augmented first sample subset and the downsampled second sample subset, to obtain an optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework.

[0200] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the privacy-preserving class-unbalanced wafer defect pattern recognition method provided by the above methods. The method includes: a first client obtaining a first sample subset and a second sample subset from a sample dataset; the number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value; performing data augmentation on the sample wafer images in each defect pattern of the first sample subset, and downsampling the sample wafer images in each defect pattern of the second sample subset; performing federated learning training based on the connected topology matrix between the first client and each second client, and the data-augmented first sample subset and the downsampled second sample subset, to obtain an optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image; the first client and the second client are located in the same federated learning framework.

[0201] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A privacy-preserving method for identifying unbalanced wafer defect patterns, characterized in that, include: The first client obtains a first subset of samples and a second subset of samples from the sample dataset; The number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value. Data augmentation is performed on the sample wafer images of each defect mode in the first sample subset, and downsampling is performed on the sample wafer images of each defect mode in the second sample subset. Based on the connectivity topology matrix between the client and each second client, and the augmented first sample subset and the downsampled second sample subset, federated learning is performed to obtain the optimal defect pattern recognition model; the optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image. The first client and the second client are located in the same federated learning framework; The training steps of the optimal defect pattern recognition model include: For the current model training, based on the connected topology matrix, the target client connected to the first client is obtained from multiple second clients; Receive local model parameters obtained by each target client during the current model training process; the local model parameters are obtained by each target client through local training of its initial defect pattern recognition model based on its own locally stored sample dataset after data augmentation and downsampling, and the global model parameters obtained by the client connected to each target client during the previous model training process. Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the local model parameters, the global model parameters corresponding to the current model training are obtained. The global model parameters corresponding to the current model training are sent to each target client, and the model training steps are executed iteratively until the defect pattern recognition model corresponding to the global model parameters meets the preset termination condition. The optimal defect pattern recognition model is constructed based on the global model parameters corresponding to the last model training. The optimal defect pattern recognition model includes a Visual Transformer (VIT) model built based on a multi-head attention mechanism; The VIT model includes an embedding layer, an encoding layer, and a classification layer; The embedding layer is used to extract embedding features from the target wafer image and output the embedding matrix of the target wafer image; The encoding layer is used to extract feature maps from the target wafer image based on the multi-head self-attention mechanism and the embedding matrix, and output the feature map of the target wafer image; The classification layer is used to perform defect pattern recognition based on the feature map of the target wafer image and output the defect pattern recognition result of the target wafer image.

2. The privacy-preserving method for identifying unbalanced wafer defect patterns according to claim 1, characterized in that, The step of obtaining the global model parameters corresponding to the current model training based on the augmented first sample subset, the downsampled second sample subset, and the local model parameters includes: Based on the first sample subset after data augmentation and the second sample subset after downsampling, as well as the global model parameters corresponding to the previous model training stored locally, local training is performed to obtain the local model parameters corresponding to the current model training. Based on the number of times each target client participates in federated learning training and the number of samples each target client participates in in the current model training, the degree of contribution of each target client to the current model training is obtained. Based on the degree of contribution, the local model parameters and the local model parameters are aggregated and updated to obtain the global model parameters corresponding to the current model training.

3. The privacy-preserving method for identifying unbalanced wafer defect patterns according to any one of claims 1-2, characterized in that, The data augmentation of the sample wafer images for each defect mode in the first sample subset includes: Extract the image to be augmented from the sample wafer images of each defect mode in the first sample subset; The image to be augmented is input into the encoder to obtain the feature map of the image to be augmented; The feature map is input into the decoder to obtain a simulated image of the image to be augmented; Based on the simulated images, data augmentation is performed on the sample wafer images of each defect mode in the first sample subset.

4. The privacy-preserving method for identifying unbalanced wafer defect patterns according to claim 3, characterized in that, The encoder is built based on deformable convolutional layers and pooling layers; The step of inputting the image to be augmented into the encoder to obtain the feature map of the image to be augmented includes: The image to be augmented is input into the deformable convolutional layer to obtain the convolutional features output by the deformable convolutional layer; The convolutional features are input into the pooling layer to obtain the feature map output by the pooling layer.

5. The privacy-preserving method for identifying unbalanced wafer defect patterns according to claim 4, characterized in that, The decoder is constructed based on a first deconvolution layer, an upsampling layer, and a second deconvolution layer; The step of inputting the feature map into the decoder to obtain a simulated image of the image to be augmented includes: The feature map is input into the first deconvolution layer to obtain the deconvolution features output by the first deconvolution layer; the feature dimension of the deconvolution features is the same as the feature dimension of the feature map. The deconvolutional features are input into the upsampling layer to obtain the upsampling features output by the upsampling layer; the feature dimension of the upsampling features is the same as the feature dimension of the convolutional features. The upsampled features are input into the second deconvolution layer to obtain the simulated image output by the second deconvolution layer.

6. A privacy-preserving device for identifying unbalanced wafer defect patterns using the privacy-preserving unbalanced wafer defect pattern recognition method as described in any one of claims 1 to 5, characterized in that, include: The data acquisition module is used by the first client to obtain a first sample subset and a second sample subset from the sample dataset; The number of samples in each defect pattern in the first sample subset is less than a first preset value, and the number of samples in each defect pattern in the second sample subset is greater than or equal to the first preset value. The data balancing module is used to augment the sample wafer images under each defect mode in the first sample subset and downsample the sample wafer images under each defect mode in the second sample subset. The pattern recognition module is used to perform federated learning training based on the connected topology matrix between the module and each second client, as well as the first sample subset after data augmentation and the second sample subset after downsampling, to obtain an optimal defect pattern recognition model. The optimal defect pattern recognition model is used to output the defect pattern recognition result of the target wafer image based on the target wafer image. The first client and the second client are located in the same federated learning framework.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the privacy-preserving class-unbalanced wafer defect pattern recognition method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the privacy-preserving class-unbalanced wafer defect pattern recognition method as described in any one of claims 1 to 5.

Images