Chip surface defect detection method and system based on federal semi-supervised learning

Abstract

The invention relates to the technical field of chip surface defect detection methods, in particular to a chip surface defect detection method and system based on federal semi-supervised learning, and the method comprises the steps: carrying out the model training of a target client on a supplier server through employing supervised data, an auxiliary agent model is introduced to participate in the training, a local model of a target client and the auxiliary agent model predict unlabeled data, and probability distributions of the local model and the auxiliary agent model are combined to obtain a most credible pseudo label; the target client side calculates consistency loss according to the local model and the auxiliary agent model of the target client side, and local model unsupervised parameters of the target client side are updated by minimizing the consistency loss; the unsupervised model parameters are uploaded to the supplier server to update the whole model of the unsupervised model, and the accuracy of chip surface defect detection can be improved under the condition that no enough annotation data is available.

Description

Technical Field

[0001] The present invention relates to the technical field of chip surface defect detection methods, and in particular to a chip surface defect detection method and system based on federated semi-supervised learning. Background Art

[0002] In the chip manufacturing process, surface defect detection is a key link in ensuring product quality and reliability. However, the chip manufacturing process involves multiple production lines or different testing equipment. The data collected by these devices often have sensitivity and privacy protection requirements. Directly sharing raw data may lead to privacy leaks or violations of industry regulations. Therefore, how to efficiently utilize decentralized data resources while protecting data privacy has become an important direction of current research.

[0003] Traditional chip defect detection methods are mostly based on traditional machine learning algorithms. These methods usually require a large amount of labeled data. However, in actual production environments, obtaining labeled data is often difficult and time-consuming. Therefore, in the chip surface defect detection task, due to the wide variety of defects and scarcity of labeled samples, the existing chip defect detection methods based on traditional machine learning are insufficient in the accuracy of chip surface defect detection without sufficient labeled data. Summary of the Invention

[0004] The purpose of the present invention is to provide a chip surface defect detection method and system based on federated semi-supervised learning, which can improve the accuracy of chip surface defect detection in the absence of sufficient labeled data.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a chip surface defect detection method based on federated semi-supervised learning, comprising:

[0006] The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, and uses supervised data on the supplier server to train the model for the target client;

[0007] In each round of training, the supplier server introduces auxiliary agent models from the user client set to participate in this round of training;

[0008] The target client's local model and auxiliary proxy model predict the unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo labels and passes them to the target client for local training.

[0009] The target client calculates the consistency loss based on its local model and the auxiliary proxy model, fixes the supervised parameters, and updates the unsupervised parameters of its local model by minimizing the consistency loss;

[0010] The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to defect classification and detection in the chip surface defect detection task.

[0011] The supplier server decomposes the model parameters into supervised learning and unsupervised learning. The specific steps for model training using supervised data on the supplier server include:

[0012] The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, initializes the supervised model parameters and the unsupervised learning model parameters, and prepares them for use in the chip surface defect detection task;

[0013] After receiving the model initialization parameters, the consumer client is ready for local training, where each consumer client has a large amount of unlabeled data;

[0014] After the model is initialized, the supplier's server uses supervised data to train the model on the user's client, fix the unsupervised learning model parameters, and update the supervised learning model parameters.

[0015] In each round of training, the supplier server introduces the auxiliary agent model from the user client set to participate in the step of this round of training. The introduction process of the auxiliary agent model includes:

[0016] For each consumer client model, generate a model embedding vector by adding Gaussian noise to the input data.

[0017] Based on the embedding vector, a BallTree data structure is constructed. For the target client, a nearest neighbor search is performed, and at least one user client model that is most similar to its embedding vector is selected as a similar auxiliary model. An additional user client model is randomly selected as a random auxiliary model. The similar auxiliary model and the random auxiliary model are jointly used as auxiliary proxy models.

[0018] The target client's local model and auxiliary proxy model predict unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo labels and passes them to the target client for local training. The specific steps include:

[0019] Obtain the predicted probability distribution of the target user client model, all similar auxiliary models, and a random auxiliary model;

[0020] Based on the predicted probability distribution, defining a basic probability distribution for each model possessed by the target consumer client;

[0021] The basic probability assignments are recursively merged across all similar auxiliary models and a random auxiliary model;

[0022] According to the combined basic probability distribution, the confidence of each category is calculated, the category with the largest confidence is selected as the pseudo label, and passed to the target client for local training.

[0023] The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server then updates its entire model based on information from similar clients. After the training process is complete, the supplier server outputs the final global model and applies it to the chip surface defect detection task for defect classification and detection. The specific steps include:

[0024] The target client uploads its locally updated unsupervised model parameters to the supplier server, and the supplier server updates its entire model based on the information of similar clients;

[0025] Each time the predetermined number of training rounds is reached, the supplier server selects the auxiliary proxy model that is most similar to the target client and updates it;

[0026] After receiving the updated global model and auxiliary agent model, the target client enters the next round of training;

[0027] When the training reaches the predetermined number of rounds, the training process ends, the supplier server outputs the final global model, and applies the model to the chip surface defect detection task for defect classification and detection.

[0028] The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. The specific steps include:

[0029] The target client uploads its locally updated unsupervised model parameters to the supplier server and updates the model uploaded to the supplier server by aggregating node and neighbor information.

[0030] The updated unsupervised parameters of each user client are uploaded to the supplier server. The user client weight is dynamically calculated using the local data volume of the user client, the category range covered by the label, and the global confidence of the pseudo label. The updated results of all user clients are weighted averaged, and the user client models are aggregated to generate a global model.

[0031] In a second aspect, the present invention further provides a chip surface defect detection system based on federated semi-supervised learning, comprising a model initialization training module, an auxiliary model introduction module, a pseudo-label generation module, a parameter update module, and a model iterative optimization module; the model initialization training module, the auxiliary model introduction module, the pseudo-label generation module, the parameter update module, and the model iterative optimization module are connected in sequence;

[0032] The model initialization training module is used by the supplier server to decompose the model parameters into a supervised learning part and an unsupervised learning part, and to perform model training on the target client using supervised data on the supplier server;

[0033] The auxiliary model introduction module is used in each round of training, where the supplier server introduces the auxiliary agent model from the user client set to participate in this round of training;

[0034] The pseudo-label generation module is used for the target client's local model and auxiliary agent model to predict unlabeled data and generate pseudo-labels respectively. The supplier server combines the probability distributions of the two to obtain the most reliable pseudo-label and transmits it to the target client for local training;

[0035] The parameter update module is used by the target client to calculate the consistency loss based on its local model and the auxiliary proxy model, fix the supervised parameters, and update the unsupervised parameters of its local model by minimizing the consistency loss;

[0036] The model iteration optimization module is used for the target client to upload its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to defect classification and detection in the chip surface defect detection task.

[0037] The present invention provides a chip surface defect detection method and system based on federated semi-supervised learning. When performing chip defect detection using federated semi-supervised learning, better pseudo-labels can be obtained, and the accuracy and robustness of chip surface defect detection can be improved through innovative data utilization methods, model aggregation strategies, and algorithm designs. In the absence of sufficient labeled data, unsupervised data can be fully utilized, and when processing uneven user client data, information from similar user clients can be effectively aggregated into a global model, effectively improving the ability to identify defects. By implementing the present invention, users can, in chip surface defect detection tasks, make full use of unsupervised data to improve the detection capability of the model when there are many types of defects and labeled samples are scarce, or when there is insufficient labeled data. At the same time, in federated learning, due to the large differences in data distribution between different clients, the present invention can set a federated learning aggregation algorithm in a semi-supervised environment to cope with the challenges of Non-IID data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.

[0039] Figure 1 It is an overall flow chart of a chip surface defect detection method based on federated semi-supervised learning of the present invention.

[0040] Figure 2 This is a flow chart of the supplier server of the present invention decomposing model parameters into a supervised learning part and an unsupervised learning part, and performing model training using supervised data on the supplier server.

[0041] Figure 3 It is a flowchart of the process of introducing the auxiliary agent model in the step of introducing the auxiliary agent model from the user client set to participate in the current round of training in each round of training of the present invention.

[0042] Figure 4 The flow chart of the present invention is that the local model of the target client and the auxiliary agent model predict the unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most credible pseudo labels and transmits them to the target client for local training.

[0043] Figure 5 The target client of the present invention uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to perform defect classification and detection in the chip surface defect detection task. Flowchart

[0044] Figure 6 It is a flowchart of the target client of the present invention uploading its locally updated unsupervised model parameters to the supplier server, and the supplier server updating its entire model according to the information of similar clients.

[0045] Figure 7 This is a flowchart of the overall idea of ​​a chip surface defect detection method based on federated semi-supervised learning of the present invention.

[0046] Figure 8 It is a structural diagram of a chip surface defect detection system based on federated semi-supervised learning of the present invention.

[0047] 1-Model initialization training module, 2-Auxiliary model introduction module, 3-Pseudo label generation module, 4-Parameter update module, 5-Model iterative optimization module. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be understood as limiting the present invention.

[0049] First, see Figure 1-Figure 7 The present invention provides a chip surface defect detection method based on federated semi-supervised learning, comprising:

[0050] The S100 supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, and uses supervised data on the supplier server to train the model for the user client;

[0051] The specific steps include:

[0052] S101 The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, initializes the supervised model parameters and the unsupervised learning model parameters, and prepares them for use in the chip surface defect detection task;

[0053] S102 After receiving the model initialization parameters, the user client is ready to perform local training, wherein each user client has a large amount of unlabeled data;

[0054] After the model is initialized in S103, the supplier's server uses supervised data to train the model on the user's client, fix the unsupervised learning model parameters, and update the supervised learning model parameters;

[0055] In the embodiment of the present invention, during the initialization phase, the supplier server decomposes the model parameters θ into a supervised learning part σ and an unsupervised learning part ψ, so that the supervised learning and unsupervised learning parts are performed independently. The supervised model parameters σ0 and the unsupervised learning model parameters ψ0 are initialized using the pre-trained lightweight defect detection model WDD-Net, ready for use in the chip surface defect detection task. After receiving the model initialization parameters, the client is ready for local training. Each user client has a large amount of unlabeled data D l .

[0056] Define the decomposition relationship of model parameters θ:

[0057] θ=σ+ψ

[0058] After the model is initialized, the supplier uses the supervised data S to train the model on the server, fixes the unsupervised learning model parameters ψ, and uses the following formula to update the supervised learning model parameters σ.

[0059] minimizeL s (σ)=λ s CrossEntropy(y,p σ+ψ* (y|x))

[0060] Here, x and y come from the labeled set S, and CrossEntropy(·,·) represents the cross entropy loss, which measures the difference between the probability distribution predicted by the supplier model and the true label.

[0061] In each round of training of S200, the supplier server introduces auxiliary agent models from the user client set to participate in this round of training;

[0062] The introduction process of the auxiliary agent model includes:

[0063] S201 generates a model embedding vector for each user client model by adding Gaussian noise to the input data as a prediction result;

[0064] In the embodiment of the present invention, for each user client model M l (The parameter is θ l ) is represented by its prediction result m on any input a (with Gaussian noise added), namely:

[0065]

[0066] in, represents expectation, θ l represents the l-th user client model, N(0,∈) is a Gaussian noise distribution with mean pseudo-zero and standard deviation ∈.

[0067] S202 constructs a BallTree data structure based on the embedding vector, performs a nearest neighbor search on the target client, selects at least one user client model that is most similar to its embedding vector as a similar auxiliary model, and then randomly selects an additional user client model as a random auxiliary model. The similar auxiliary model and the random auxiliary model are jointly used as auxiliary proxy models.

[0068] In the embodiment of the present invention, based on the embedding vector {m1,m2,…,m k}, construct BallTree data structure to achieve efficient nearest neighbor search. For each user client l, perform H-nearest neighbor search and select the nearest neighbor with m l The H most similar consumer client models are used as auxiliary agents, and an additional consumer client model is randomly selected to increase diversity.

[0069] The local model and auxiliary proxy model of the S300 target client predict the unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo labels and passes them to the target client for local training;

[0070] The specific steps include:

[0071] S301 obtains the predicted probability distribution of the target user client model, all similar auxiliary models, and a random auxiliary model;

[0072] In the embodiment of the present invention, the target user client model θ is obtained. l , H similar auxiliary models And 1 random auxiliary model The predicted probability distribution p k =[p k (C1|u),p k (C2|u),p k (C3|u)], where k represents the total number of models owned by the consuming client, and C i Indicates category, p k =(C i |u) represents the model θ k The predicted result of input u belongs to category C i probability.

[0073] S302 defines a basic probability distribution for each model of the target user client based on the predicted probability distribution;

[0074] In the embodiment of the present invention, each model θ of the target user client is defined k The basic probability allocation (BPA) is:

[0075] mk (C i )=p k (C i |u)

[0076] That is, model θ k Predict that input u belongs to category C i probability.

[0077] S303 recursively merges all similar auxiliary models and the basic probability distribution of a random auxiliary model;

[0078] In this embodiment of the present invention, in order to fuse the evidence of H+1 models, the BPAs of the H+1 models are merged recursively:

[0079] m fused =m0⊕m1⊕…⊕m H ⊕m rand

[0080] Among them, ⊕ represents the Dempster merge operation.

[0081] S304 calculates the confidence of each category based on the combined basic probability distribution, selects the category with the highest confidence as the pseudo label, and transmits it to the target client for local training;

[0082] In the embodiment of the present invention, BPAm is allocated according to the combined basic probability fused , calculate the trust level of each category Bel(C i ), select the category with the highest confidence as the pseudo label:

[0083]

[0084] The S400 target client calculates the consistency loss based on its local model and the auxiliary proxy model, fixes the supervised parameters, and updates the unsupervised parameters of its local model by minimizing the consistency loss;

[0085] In this embodiment of the present invention, each user client calculates the consistency loss based on its local model and the auxiliary proxy model (from similar clients) to improve the consistency of the model. The consistency loss of the prediction difference between the local model and the auxiliary proxy model calculated by each user client is expressed as:

[0086]

[0087] Among them, H+1 represents the total number of auxiliary agents, H similar auxiliary models and 1 random auxiliary model. h represents the predicted probability distribution of the jth auxiliary agent model for the output y given the input u. The auxiliary agent model is one of the H models selected from the supplier server based on model similarity.j represents the jth auxiliary agent, represents the predicted probability distribution of the output y by the l-th user client model when given input u, l represents the l-th user client, θ l denotes the parameters of the lth consumer client model. KL[·||·] denotes the KL divergence, which measures the difference between the predicted distribution of the auxiliary agent model and the predicted distribution of the client model. KL is not trained on the consumer client (* indicates that we freeze the parameters). The supplier server selects and broadcasts H auxiliary agents in each communication round.

[0088] This paper uses the semi-supervised scheme proposed by Sohn and Berthelot (K. Sohn, D. Berthelot, C.-L. Li, Z. Zhang, N. Carlini, E. D. Cubuk, A. Kurakin, H. Zhang, and C. Raffel, “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence,” in Advances in Neural Information Processing Systems, vol. 33, 2020.) to generate pseudo-labeled images. Strong enhancement (such as more complex image distortion and cropping) is performed on the image, and the model is required to predict the results of the strongly enhanced image consistent with the pseudo-label, achieving data-level consistency regularization. The final consistency loss is:

[0089]

[0090] in, is a pseudo label generated based on evidence theory, p θl (y|π(u)) represents the probability distribution predicted by the model on the input π(u) (the result of random data augmentation on the input u), and CrossEntropy(·, ·) represents the cross entropy loss, which measures the difference between the probability distribution predicted by the model and the true label.

[0091] Fix the supervised parameters and update the unsupervised parameters of its local model by minimizing the consistency loss:

[0092]

[0093] Among them, λ ICCS is a hyperparameter that controls the consistency loss. is L2 regularization, which is used to control ψ not to deviate too far from the knowledge learned by σ. L1||ψ||1 is an L1 regularization term used to make the ψ parameter sparse. This parameter decomposition and optimization strategy enables the model to use different parameter settings in supervised and unsupervised learning phases, thereby reducing the impact of noise in unlabeled data on supervised learning while maintaining the consistency and sparsity of the model.

[0094] The S500 target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server then updates its entire model based on information from similar clients. After the training process is complete, the supplier server outputs the final global model and applies it to the chip surface defect detection task for defect classification and detection.

[0095] The specific steps include:

[0096] S501: The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. The specific steps include:

[0097] S511 The target client uploads its locally updated unsupervised model parameters to the supplier server, and updates the model uploaded to the supplier server by aggregating node and neighbor information;

[0098] In an embodiment of the present invention, the device user client uploads its locally updated unsupervised model parameters to the supplier server. During this process, the model uploaded to the supplier server is updated using the method of aggregating node and neighbor information proposed by Thomas N. Kipf and Max Welling (TNKipf and M. Welling, "Semi-supervised classification with graph convolutional networks," in Proc. International Conference on Learning Representations, 2017.). This method uses the model parameters uploaded by the user client as the embedding of each node, calculates the similarity of each user client model as the edge of each node, and uses graph convolution operation to

[0099]

[0100] Among them, θ l It is the overall representation of the lth user client model. is the adjacency matrix with self-loops added. yes The degree matrix of θ′. W is the learnable weight matrix. σ(·) is the nonlinear activation function. lis the updated overall representation of the i-th consumer client model.

[0101] S512 uploads the updated unsupervised parameters of each user client to the supplier server, dynamically calculates the user client weight based on the amount of local data of the user client, the category range covered by the label, and the global confidence of the pseudo label, and takes the weighted average of all the updated results of the user clients to aggregate the user client models and generate a global model;

[0102] In this embodiment of the present invention, the updated unsupervised parameters of each user client are uploaded to the supplier server. The user client weight is dynamically calculated using the amount of local data on the user client, the category range covered by the label, and the global confidence of the pseudo label (i.e., the average confidence of all pseudo labels on the data of the user client). The updated results of all user clients are weighted and averaged to more reasonably aggregate the user client models and generate a global model. The weight calculation method is as follows:

[0103] λ l =α·DataSize l +β·LabelCoverage l +γ·PseudoLabelConf l

[0104] DataSize: The amount of local data on the client. LabelCoverage: The range of categories covered by the pseudo-label. PseudoLabelConf: The global consistency confidence of the pseudo-label (i.e., the average confidence of all pseudo-labels on the client). α, β, and γ: Balance weight parameters.

[0105] The aggregated global model is expressed as:

[0106]

[0107] Where A is the number of consuming clients, θ global is the aggregated global model, ψ l represents the unsupervised part of the l-th user client, σ l Indicates that the lth user client has a supervisory part.

[0108] S502: Every time a predetermined number of training rounds is reached, the supplier server selects the auxiliary proxy model that is most similar to the target client and updates it;

[0109] In this embodiment of the present invention, every 10 rounds, the supplier server selects the auxiliary proxy model that is most similar to each user client and updates it. The server uses the Ball-Tree method to efficiently find the most similar model based on the embedding vector of each user client model.

[0110] S503 After the target client receives the updated global model and auxiliary agent model, it enters the next round of training;

[0111] In this embodiment of the present invention, after receiving the updated global model and auxiliary proxy model, the user client enters the next round of training. The client repeats the supervised learning, unsupervised learning, pseudo-label generation, and consistency loss calculation processes until the predetermined number of training rounds is reached.

[0112] S504: When the training reaches the predetermined number of rounds, the training process ends, and the supplier server outputs the final global model and applies the model to the chip surface defect detection task for defect classification and detection;

[0113] In the embodiment of the present invention, when the training reaches the predetermined number of rounds, the training process ends and the server outputs the final global model θ global , and applied the model to perform defect classification and detection in the chip surface defect detection task.

[0114] In order to better understand the present invention, the present invention is described below through a specific implementation process.

[0115] In this embodiment, the data is labeled by the supplier, and the supervised data S is only on the supplier's server. The user does not need to label the data, that is, each user client only has unsupervised data D. l .

[0116] The present invention trained a lightweight defect detection model WDD-Net (X.Chen et al., "A Light-Weighted CNN Model for Wafer Structural Defect Detection," in IEEE Access, vol. 8, pp. 24006-24018, 2020, doi: 10.1109 / ACCESS.2020.2970461.), which is designed specifically for chip surface defect detection. With a total parameter count of 17,200 and a model size of only 307KB, the model is efficient, fast, and small in size, making it very suitable for deployment and operation on user devices. WDD-Net consists of four main modules: an input preprocessing module, a feature extraction module, a feature aggregation module, and a classification decision module to ensure the efficient capture and classification of defect information.

[0117] Input preprocessing module: This module receives the original chip surface image and normalizes it to reduce the impact of illumination changes on the detection results. At the same time, it improves the generalization ability of the model through data augmentation technology.

[0118] Feature extraction module: This module consists of a standard 3×3 convolution layer and six depthwise separable convolution modules (including 3×3 depthwise convolution and 1×1 standard convolution). This design reduces the computational workload while ensuring that the model can efficiently extract local and global features of chip surface defects.

[0119] Feature aggregation module: To avoid the loss of shallow features and improve the recognition ability of low-brightness defect areas, this module uses global average pooling (GAP) instead of the traditional fully connected layer to reduce the number of parameters, lower the computational cost, and enhance the model's adaptability to different defect morphologies.

[0120] Classification Decision Module: After feature aggregation, the model uses the Softmax function to calculate the probability of defect categories and output the final defect detection results. This module can effectively distinguish different types of chip surface defects, ensuring detection accuracy and stability.

[0121] The raw chip surface image first enters the preprocessing module for normalization and enhancement, then enters the feature extraction module for multi-layer convolution calculations to extract the deep features of the defect. The extracted features are then globally integrated by the feature aggregation module, and finally the classification decision module calculates the category probability and outputs the defect category. This process ensures efficient data flow from input to classification, enabling the model to achieve efficient computation while maintaining detection accuracy.

[0122] Step 1: Initialization phase: The supplier server decomposes the model parameters θ into a supervised learning part σ and an unsupervised learning part ψ, allowing the supervised learning and unsupervised learning parts to proceed independently. The supervised model parameters σ0 and the unsupervised learning model parameters ψ0 are initialized using the pre-trained lightweight defect detection model WDD-Net, ready for use in the chip surface defect detection task. After receiving the model initialization parameters, the client is ready for local training. Each user client has a large amount of unlabeled data D l .

[0123] Step 2: After the model is initialized, the supplier uses the supervised data S to train the model on the server, fixes the unsupervised learning model parameters ψ, and uses the following formula to update the supervised learning model parameters σ.

[0124] minimizeL s (σ)=λ s CrossEntropy(y,p σ+ψ* (y|x))

[0125] Here, x and y come from the labeled set S, and CrossEntropy(·,·) represents the cross entropy loss, which measures the difference between the probability distribution predicted by the supplier model and the true label.

[0126] Step 3: In each round of training, the supplier server randomly selects A=5 user clients from the user client set L to participate in this round of training.

[0127] Each device user client has a portion of unlabeled data. The user client receives the supervised model parameters σ and unsupervised learning model parameters ψ from the supplier server. l , H auxiliary model unsupervised parameters and random auxiliary model unsupervised parameters After that, the client merges the supervised parameters with the unsupervised parameters to restore the complete model parameters.

[0128] Step 4: Each user client uses its local model to predict the unlabeled data and generate a pseudo label. For example, user client 1 predicts an unlabeled image and obtains a prediction result of "defect", and the pseudo label is set to "defect". The supplier server merges the pseudo labels from different user clients using the Dempster merging rule. For example, user client 1 predicts "defect" and user client 2 predicts "no defect". The supplier server merges the probability distributions of the two and finally obtains the most credible pseudo label and updates the pseudo label of the user client. Assuming model θ k For Category C i The predicted probability is p k =[p k (C1|u),p k (C2|u),p k (C3|u)], then the prediction result of each model p k This is represented as the model's basic probability allocation (BPA) for each category. These probability distributions allow us to quantify the predictions of each model and express the degree of support each model has for each category.

[0129] For example, suppose the target client model θ0 predicts the input sample u as follows:

[0130] p0=[0.1,0.7,0.2]

[0131] This means that the model believes that the probability that the sample is "non-defective" is 0.7, and the probabilities of being "defective" and "unrecognizable" are 0.1 and 0.2 respectively.

[0132] Auxiliary model θ1 prediction results:

[0133] p1=[0.2,0.6,0.2]

[0134] Auxiliary model θ2 prediction results:

[0135] p2=[0.15,0.75,0.1]

[0136] Random auxiliary model θ rand Prediction results:

[0137] p rand =[0.25,0.5,0.25]

[0138] According to the predicted probability of each model, we can convert it into a basic probability allocation (BPA). For example, for the target client model θ0, its basic probability allocation is:

[0139] m0(C1)=p0(C1|u)=0.1, m0(C2)=p0(C2|u)=0.7, m0(C3)=p0(C3|u)=0.2

[0140] Similarly, the BPA of the auxiliary model θ1 is:

[0141] m1(C1)=0.2, m1(C2)=0.6, m1(C3)=0.2

[0142] Random auxiliary model θ rand The BPA is:

[0143] m rand (C1)=0.25,m rand (C2) = 0.5, m rand (C3) = 0.25

[0144] These BPAs represent the support of each model for different categories.

[0145] Next, the Dempster merging rule is used to fuse the BPAs of multiple models. Assume that the BPA of the target user client model θ0 is m0, the BPA of the auxiliary model θ1 is m1, and the BPAs of other auxiliary models are m2, m2, ..., m H , and BPAm with random auxiliary model rand , the merged BPAm fused :

[0146] m fused =m0⊕m1…⊕m H ⊕m rand

[0147] The BPA obtained after the combination is:

[0148] m fused =[0.22, 0.66, 0.12]

[0149] This means that after the combined BPA, the model believes that the sample most likely belongs to the category "no defect" (C2), which has a support of 0.66.

[0150] According to the combined BPA, the confidence of each category is calculated, and the category with the largest confidence is selected as the final pseudo label:

[0151]

[0152] In this example, the combined confidence levels are:

[0153] Bel(C1)=0.22, Bel(C2)=0.66, Bel(C3)=0.12

[0154] Therefore, the one with the highest confidence is finally selected as the final pseudo label, and the image is marked as “defect-free” (C2) and passed to the client for local training.

[0155] Step 5: Each consumer client calculates the consistency loss based on its local model and the auxiliary proxy model (from similar clients) to increase the consistency of the model. Each consumer client calculates the prediction difference consistency loss of the local model and the auxiliary proxy model as:

[0156]

[0157] Among them, H+1 represents the total number of auxiliary agents, H similar auxiliary models and 1 random auxiliary model. h represents the predicted probability distribution of the jth auxiliary agent model for the output y given the input u. The auxiliary agent model is one of the H models selected from the supplier server based on model similarity. j represents the jth auxiliary agent, represents the predicted probability distribution of the output y by the l-th user client model given the input u, l represents the l-th user client, θ l denotes the parameters of the lth consumer client model. KL[·||·] denotes the KL divergence, which measures the difference between the predicted distribution of the auxiliary agent model and the predicted distribution of the client model. KL is not trained on the consumer client (* indicates that we freeze the parameters). The supplier server selects and broadcasts H auxiliary agents in each communication round.

[0158] This paper uses the semi-supervised scheme proposed by Sohn and Berthelot (K.Sohn, D.Berthelot, C.-L.Li, Z.Zhang, N.Carlini, E.D.Cubuk, A.Kurakin, H.Zhang, and C.Raffel, “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence,” in Advances in Neural Information Processing Systems, vol. 33, 2020.) to generate pseudo-labeled images. Strong enhancement (such as more complex image distortion and cropping) is performed, and the model is required to predict the results of the strongly enhanced images consistent with the pseudo-labels, achieving data-level consistency regularization. The final consistency loss is:

[0159]

[0160] in, It is a pseudo-label generated based on evidence theory. represents the probability distribution predicted by the model on the input π(u) (the result of random data augmentation on the input u), and CrossEntropy(·,·) represents the cross entropy loss, which measures the difference between the probability distribution predicted by the model and the true label.

[0161] Fix the supervised parameters and update the unsupervised parameters of its local model by minimizing the consistency loss:

[0162]

[0163] where λ ICCS is a hyperparameter that controls the consistency loss. is L2 regularization, which is used to control ψ not to deviate too far from the knowledge learned by σ. L1 ||ψ||1 is an L1 regularization term used to make the ψ parameter sparse. This parameter decomposition and optimization strategy enables the model to use different parameter settings in supervised and unsupervised learning phases, thereby reducing the impact of noise in unlabeled data on supervised learning while maintaining the consistency and sparsity of the model.

[0164] Step 6: The device user client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server uses graph theory to absorb information from similar clients (neighbors) based on model similarity, thereby updating its entire model. As shown in the figure, model aggregation consists of two stages: the parameter update stage and the weighted aggregation stage. The details are as follows:

[0165] Parameter update phase:

[0166] In this process, the model uploaded to the supplier's server is updated using the method proposed by Thomas N. Kipf and Max Welling (TNKipf and M. Welling, "Semi-supervised classification with graph convolutional networks," in Proc. International Conference on Learning Representations, 2017.) to aggregate node and neighbor information. This method uses the model parameters uploaded by the user client as the embedding of each node, calculates the similarity of each user client model as the edge of each node, and uses graph convolution operations to

[0167]

[0168] Among them, θ l It is the overall representation of the lth user client model. is the adjacency matrix with self-loops added. yes The degree matrix of θ′. W is the learnable weight matrix. σ(·) is the nonlinear activation function. l is the updated overall representation of the i-th consumer client model.

[0169] Weighted aggregation stage:

[0170] The updated unsupervised parameters of each user client are uploaded to the supplier server. The user client weight is dynamically calculated using the amount of local data of the user client, the category range covered by the label, and the global confidence of the pseudo label (that is, the average confidence of the pseudo label of all data on the user client). The updated results of all user clients are weighted and averaged to more reasonably aggregate the user client models and generate a global model. The weight calculation method is as follows:

[0171] λ l =α·DataSize l +β·LabelCoverage l +γ·PseudoLabelConf l

[0172] DataSize: The amount of local data on the client. LabelCoverage: The range of categories covered by the pseudo-label. PseudoLabelConf: The global consistency confidence of the pseudo-label (i.e., the average confidence of all pseudo-labels on the client). α, β, and γ: Balance weight parameters.

[0173] The aggregated global model is expressed as:

[0174]

[0175] Where A is the number of consuming clients, θ global is the aggregated global model, ψ l represents the unsupervised part of the l-th user client, σ l Indicates that the lth user client has a supervisory part.

[0176] Step 7: Every 10 rounds, the supplier server selects the auxiliary proxy model that is most similar to each consumer client and updates it. The server uses the Ball-Tree method to efficiently find the most similar model based on the embedding vector of each consumer client model.

[0177] Step 8: After receiving the updated global model and auxiliary proxy model, the client enters the next round of training. The client repeats the supervised learning, unsupervised learning, pseudo label generation, consistency loss calculation, etc. until the predetermined number of training rounds is reached.

[0178] Step 9: When the training reaches the predetermined number of rounds, the training process ends and the server outputs the final global model θ global , and applied the model to perform defect classification and detection in the chip surface defect detection task.

[0179] The present invention provides a chip surface defect detection method based on federated semi-supervised learning. When performing chip defect detection using federated semi-supervised learning, better pseudo-labels can be obtained, and the accuracy and robustness of chip surface defect detection can be improved through innovative data utilization methods, model aggregation strategies, and algorithm designs. In the absence of sufficient labeled data, unsupervised data can be fully utilized, and when processing uneven user client data, information from similar user clients can be effectively aggregated into a global model, effectively improving the ability to identify defects. By implementing the present invention, users can, in chip surface defect detection tasks, make full use of unsupervised data to improve the detection capability of the model when there are many types of defects and labeled samples are scarce, or when there is insufficient labeled data. At the same time, in federated learning, due to the large differences in data distribution between different clients, the present invention can set a federated learning aggregation algorithm in a semi-supervised environment to cope with the challenges of Non-IID data.

[0180] Second, see Figure 8 The present invention also provides a chip surface defect detection system based on federated semi-supervised learning, comprising a model initialization training module 1, an auxiliary model introduction module 2, a pseudo-label generation module 3, a parameter update module 4, and a model iterative optimization module 5; the model initialization training module 1, the auxiliary model introduction module 2, the pseudo-label generation module 3, the parameter update module 4, and the model iterative optimization module 5 are connected in sequence;

[0181] The model initialization training module 1 is used by the supplier server to decompose the model parameters into a supervised learning part and an unsupervised learning part, and to perform model training on the target client using supervised data on the supplier server;

[0182] The auxiliary model introduction module 2 is used in each round of training, where the supplier server introduces the auxiliary agent model from the user client set to participate in this round of training;

[0183] The pseudo-label generation module 3 is used for the target client's local model and auxiliary agent model to predict the unlabeled data and generate pseudo-labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo-label and transmits it to the target client for local training;

[0184] The parameter updating module 4 is used for the target client to calculate the consistency loss based on its local model and the auxiliary proxy model, fix the supervised parameters, and update the unsupervised parameters of its local model by minimizing the consistency loss;

[0185] The model iteration optimization module 5 is used for the target client to upload its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to defect classification and detection in the chip surface defect detection task.

[0186] The present invention provides a chip surface defect detection system based on federated semi-supervised learning. The equipment supplier server and the user client perform collaborative training through federated learning to ensure that the sensitive data of the equipment user does not leave the local area. The system includes client-to-client consistency loss, pseudo-label generation mechanism, parameter decomposition strategy, and client fusion algorithm. The client-to-client consistency loss is used to introduce an auxiliary agent model and a consistency regularization mechanism. The pseudo-label generation technology uses evidence theory to generate more stable and accurate pseudo-labels by combining the prediction information of different user clients. The parameter decomposition strategy decomposes the model parameters into supervised learning and unsupervised learning, and respectively trains supervised parameters on labeled data and unsupervised parameters on unsupervised data. The graph convolution client fusion algorithm uses a graph convolution network to aggregate information of similar user clients, update the client model, and dynamically calculate client weights for aggregation. When using federated semi-supervised learning to detect chip defects, better pseudo-labels can be obtained, and the accuracy and robustness of chip surface defect detection can be improved through innovative data utilization methods, model aggregation strategies, and algorithm design. In the absence of sufficient labeled data, full use can be made of unsupervised data, and when dealing with uneven user client data, information from similar user clients can be effectively aggregated into the global model, effectively improving the ability to identify defects.

[0187] The above disclosure is merely one or more preferred embodiments of the present application and is not intended to limit the scope of the present application. A person skilled in the art will understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the present application.

Claims

1. A chip surface defect detection method based on federated semi-supervised learning, characterized in that: include: The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, and uses supervised data on the supplier server to train the model for the target client; In each round of training, the supplier server introduces auxiliary agent models from the user client set to participate in this round of training; The target client's local model and auxiliary proxy model predict the unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo labels and passes them to the target client for local training. The target client calculates the consistency loss based on its local model and the auxiliary proxy model, fixes the supervised parameters, and updates the unsupervised parameters of its local model by minimizing the consistency loss; The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to defect classification and detection in the chip surface defect detection task.

2. The chip surface defect detection method based on federated semi-supervised learning according to claim 1, characterized in that: The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts. The specific steps for model training using supervised data on the supplier server include: The supplier server decomposes the model parameters into supervised learning and unsupervised learning parts, initializes the supervised model parameters and the unsupervised learning model parameters, and prepares them for use in the chip surface defect detection task; After receiving the model initialization parameters, the consumer client is ready for local training, where each consumer client has a large amount of unlabeled data; After the model is initialized, the supplier's server uses supervised data to train the model on the user's client, fix the unsupervised learning model parameters, and update the supervised learning model parameters.

3. The chip surface defect detection method based on federated semi-supervised learning according to claim 2, characterized in that: In each round of training, the supplier server introduces the auxiliary agent model from the user client set to participate in the steps of this round of training. The introduction process of the auxiliary agent model includes: For each consumer client model, generate a model embedding vector by adding Gaussian noise to the input data. Based on the embedding vector, a Ball Tree data structure is constructed. For the target client, a nearest neighbor search is performed, and at least one user client model that is most similar to its embedding vector is selected as a similar auxiliary model. An additional user client model is randomly selected as a random auxiliary model. The similar auxiliary model and the random auxiliary model are jointly used as auxiliary proxy models.

4. The chip surface defect detection method based on federated semi-supervised learning according to claim 3, characterized in that: The target client's local model and auxiliary proxy model predict the unlabeled data and generate pseudo labels respectively. The supplier server merges the probability distributions of the two to obtain the most reliable pseudo labels and passes them to the target client for local training. The specific steps include: Obtain the predicted probability distribution of the target user client model, all similar auxiliary models, and a random auxiliary model; Based on the predicted probability distribution, defining a basic probability distribution for each model possessed by the target consumer client; The basic probability assignments are recursively merged across all similar auxiliary models and a random auxiliary model; According to the combined basic probability distribution, the confidence of each category is calculated, the category with the largest confidence is selected as the pseudo label, and passed to the target client for local training.

5. The chip surface defect detection method based on federated semi-supervised learning according to claim 4, characterized in that: The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server then updates its entire model based on information from similar clients. After the training process is complete, the supplier server outputs the final global model and applies it to the chip surface defect detection task for defect classification and detection. The specific steps include: The target client uploads its locally updated unsupervised model parameters to the supplier server, and the supplier server updates its entire model based on the information of similar clients; Each time the predetermined number of training rounds is reached, the supplier server selects the auxiliary proxy model that is most similar to the target client and updates it; After receiving the updated global model and auxiliary agent model, the target client enters the next round of training; When the training reaches the predetermined number of rounds, the training process ends, the supplier server outputs the final global model, and applies the model to the chip surface defect detection task for defect classification and detection.

6. The chip surface defect detection method based on federated semi-supervised learning according to claim 5, characterized in that: The target client uploads its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. The specific steps include: The target client uploads its locally updated unsupervised model parameters to the supplier server and updates the model uploaded to the supplier server by aggregating node and neighbor information. The updated unsupervised parameters of each user client are uploaded to the supplier server. The user client weight is dynamically calculated using the local data volume of the user client, the category range covered by the label, and the global confidence of the pseudo label. The updated results of all user clients are weighted averaged, and the user client models are aggregated to generate a global model.

7. A chip surface defect detection system based on federated semi-supervised learning, using the chip surface defect detection method based on federated semi-supervised learning according to any one of claims 1 to 6, characterized in that: It includes a model initialization training module, an auxiliary model introduction module, a pseudo label generation module, a parameter update module and a model iterative optimization module; the model initialization training module, the auxiliary model introduction module, the pseudo label generation module, the parameter update module and the model iterative optimization module are connected in sequence; The model initialization training module is used by the supplier server to decompose the model parameters into a supervised learning part and an unsupervised learning part, and to perform model training on the target client using supervised data on the supplier server; The auxiliary model introduction module is used in each round of training, where the supplier server introduces the auxiliary agent model from the user client set to participate in this round of training; The pseudo-label generation module is used for the target client's local model and auxiliary agent model to predict unlabeled data and generate pseudo-labels respectively. The supplier server combines the probability distributions of the two to obtain the most reliable pseudo-label and transmits it to the target client for local training; The parameter update module is used by the target client to calculate the consistency loss based on its local model and the auxiliary proxy model, fix the supervised parameters, and update the unsupervised parameters of its local model by minimizing the consistency loss; The model iteration optimization module is used for the target client to upload its locally updated unsupervised model parameters to the supplier server. The supplier server updates its entire model based on the information of similar clients. After the training process is completed, the supplier server outputs the final global model and applies the model to defect classification and detection in the chip surface defect detection task.

Images