A flip chip defect detection method, system, device and medium
By constructing a method of expanding data sets with multiple loss functions and pseudo-labels, the problem of poor generalization of model in flip-chip defect detection is solved, and the detection accuracy and model generalization ability are improved.
Patent Information
- Application Number
- CN202410766498.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-06-14
AI Technical Summary
In the flip chip defect detection method, due to the small data and diverse defect types, the model generalization is poor and the defect detection accuracy is low.
By collecting one-dimensional vibration data of the flip chip, dividing it into a training set, a test set and an unlabeled sample set, the original classification loss, supervised comparison loss, self-calibration loss and cross-calibration loss, training and improving the prototype network, and Augmenting the data sets through pseudo-labels to enhance model generalization capabilities.
It significantly improves the accuracy of flip chip defect detection in small samples, enhances the generalization ability of the model, and improves the reliability of the detection results.
Smart Images

Figure CN118445624B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of chip defect detection, and in particular to a flip chip defect detection method, system, equipment and medium. Background Art
[0002] As flip-chip technology develops toward high-density solder joints and ultra-fine pitch, its scale and surface effects become more obvious. With new features such as the lead-free requirements for packaging materials and the introduction of Low-K materials, the thermal or stress mismatch of the packaging structure will become more serious, and deformation, warping or scratches will be more likely to occur at the packaging interface, which will cause solder joint fatigue and stress concentration, leading to solder joint defects. The mechanical reliability of chip packaging will deteriorate as the chip scale continues to shrink.
[0003] Intelligent defect detection methods based on deep learning have developed rapidly in recent years. They are non-destructive, safe, and can realize online monitoring. Compared with traditional machine learning methods such as SVM, Kmeans, KNN, spectral clustering, etc., deep learning methods have more powerful sensitive feature extraction and fault identification capabilities without the need for artificially designed features. Deep learning methods, such as autoencoders, convolutional neural networks, recursive neural networks, etc., have been widely used in mechanical defect detection and have achieved good performance when there are enough labeled fault samples.
[0004] However, the defect detection of flip chips is characterized by a large amount of unlabeled data and diverse defect types, which leads to conflicts in fault information and difficulty in extracting characteristic patterns, making the existing intelligent defect detection methods poorly adaptable. In actual application environments, the number of labeled samples is small, and the existing data volume cannot match the intelligent model, resulting in weak model generalization and low defect detection accuracy, which restricts the application of intelligent defect detection technology in flip chip defect detection. Summary of the invention
[0005] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that there is less flip chip defect data, resulting in weak model generalization and low defect detection accuracy.
[0006] In order to solve the above technical problems, the present invention provides a flip chip defect detection method, comprising:
[0007] Step S1: collecting one-dimensional vibration data of the flip chip, and dividing the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set;
[0008] Step S2: dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training the preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss;
[0009] Step S3: assigning pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, adding the pseudo label samples to the training set to expand the training set, and further training the improved prototype network with the expanded training set to obtain a trained improved prototype network;
[0010] Step S4: importing the test set into the trained improved prototype network to perform defect detection and obtain detection results.
[0011] In one embodiment of the present invention, the method of dividing the training set into a support set and a query set in step S2 is:
[0012] The training set is defined as ,in Indicates i training samples, express The corresponding label, Represents the total number of samples in the training set;
[0013] Randomly select from each category of the training set The samples form the support set ,in, , Indicates i Support samples, express The corresponding label, represents the total number of samples in the support set, N is the number of categories contained in the training set;
[0014] The remaining samples in each category of the training set are taken as query sets ,in, , Indicates i A sample of queries, express The corresponding label, Indicates that each category is extracted The number of remaining samples after samples, Indicates the total number of samples in the query set.
[0015] In one embodiment of the present invention, the method for constructing the original classification loss includes:
[0016] The support samples and the query samples are processed by a pre-set feature extractor of the improved prototype network, mapped to a high-dimensional feature space, and generated support embedding features. and query embedding features ;
[0017] Calculate the support embedding features for each category The average vector of is used as the support set prototype of this category. The formula is:
[0018] ;
[0019] in, Belong to category The support set prototype, A set of support set prototypes for all categories, represents the indicator function, To improve the feature extractor of the prototype network;
[0020] Calculate the query embedding feature The Euclidean distance between each of the support set prototypes , convert the Euclidean distance into the posterior probability to obtain the predicted label of the query sample. The formula is:
[0021] ;
[0022] ;
[0023] in, represents the two-norm, represents the posterior probability of the query sample, Indicates the category label;
[0024] The cross entropy loss between the true label and the predicted label of the query sample is calculated and used as the original classification loss. The formula is:
[0025] ;
[0026] in, represents the cross entropy function.
[0027] In one embodiment of the present invention, the method for constructing the supervised contrast loss includes:
[0028] For each category , each support set prototype As an anchor point, c The query sample of the class is taken as a positive sample ,in Indicates iPositive samples, Indicates i The labels corresponding to the positive samples are calculated, and the distance between the anchor point and each positive sample is calculated using the formula:
[0029] ;
[0030] in, represents the cosine distance;
[0031] Will not belong to c The query sample of the class is taken as a negative sample ,in Indicates i negative samples, Indicates i The labels corresponding to the negative samples are calculated, and the distance between the anchor point and each of the negative samples is calculated using the formula:
[0032] ;
[0033] according to and Construct a supervised contrast loss, the formula is:
[0034] ;
[0035] in, Represents a hyperparameter.
[0036] In one embodiment of the present invention, the self-calibration loss and the cross-calibration loss are used to impose constraints on the support set prototype to improve the support set prototype, wherein:
[0037] The method for constructing the self-calibration loss includes: calculating the support embedding feature With each of the support set prototypes The Euclidean distance between them is then converted into the posterior probability using the formula:
[0038] ;
[0039] in, represents the posterior probability of supporting the sample, Indicates Support set prototypes;
[0040] The self-calibration loss is calculated as follows:
[0041] ;
[0042] in, represents the cross entropy function;
[0043] The method for constructing the cross calibration loss includes: generating support embedding features and query embedding features , calculate the query embedding features in each category The average vector of is taken as the prototype of the query set of this category, and the formula is:
[0044] ;
[0045] in, Belong to category The support set prototype, A collection of support set prototypes for all categories;
[0046] The Euclidean distance between the query set prototype and the support set prototype is calculated, and the Euclidean distance is converted into a posterior probability, the formula is:
[0047] ;
[0048] in, Indicates i query set prototypes;
[0049] The cross calibration loss is calculated as follows:
[0050] ;
[0051] in, Indicates i The labels corresponding to the query set prototypes.
[0052] In one embodiment of the present invention, the method for obtaining the trained improved prototype network in step S3 is:
[0053] The unlabeled sample set is defined as ,in, , Indicates i unlabeled samples, represents the number of samples in each category, Represents the total number of samples in the unlabeled sample set;
[0054] During the training process of the improved prototype network, the unlabeled samples are input into the feature extractor of the improved prototype network for processing to obtain unlabeled embedded features, and the Euclidean distance between the unlabeled embedded features and each of the support set prototypes is calculated. The Euclidean distance is converted into a probability vector for the prediction of the unlabeled sample set, and the prediction probability of each unlabeled sample corresponding to the unlabeled embedded feature is recorded every several equal iteration steps. The formula is: ,in Indicates the number of recorded steps. Indicates The predicted probability of unlabeled samples; then calculate the predicted label, the formula is:
[0055] ;
[0056] in, Represents a function that returns the label corresponding to the maximum probability value;
[0057] Each unlabeled sample is The predicted labels ; The predicted label with the highest frequency of occurrence is recorded as , the frequency is recorded as ,like Not less than the frequency threshold , then the unlabeled sample is selected as an easy sample, denoted as ; Otherwise, select another unlabeled sample to determine whether to assign a pseudo label;
[0058] Information entropy is used to measure the easy samples The uncertainty of the predicted probability is given by:
[0059] ;
[0060] in Indicates j The predicted probabilities belong to k The probability of the class; an uncertainty threshold is set for each class to constrain the assignment of pseudo labels to the unlabeled samples, and the uncertainty threshold formula is:
[0061] ;
[0062] in, Indicates i The predicted labels corresponding to the easy samples are represents the indicator function, is the category label;
[0063] If the uncertainty of the predicted probability is not greater than the uncertainty threshold, the easy sample is assigned Pseudo-labels are used as pseudo-label samples and added to the training set to further train the improved prototype network to obtain a trained improved prototype network; otherwise, another unlabeled sample is selected to determine whether to assign a pseudo-label.
[0064] In one embodiment of the present invention, the method of dividing the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set in step S1 is:
[0065] The collected one-dimensional vibration data are cut into samples of preset length and normalized; the normalized samples are divided into a training set, a test set and an unlabeled sample set, wherein the number of samples in the training set is much smaller than that in the test set and the unlabeled sample set.
[0066] In order to solve the above technical problems, the present invention provides a flip chip defect detection system, comprising:
[0067] Collection and division module: used to collect the one-dimensional vibration data of the flip chip and divide the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set;
[0068] A first training module: used for dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training a preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss;
[0069] The second training module is used to assign pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, add the pseudo label samples to the training set to expand the training set, and further train the improved prototype network through the expanded training set to obtain a trained improved prototype network;
[0070] Detection module: used to import the test set into the trained improved prototype network for defect detection to obtain the detection result.
[0071] To solve the above technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above flip chip defect detection method when executing the computer program.
[0072] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above flip chip defect detection method are implemented.
[0073] The above technical solution of the present invention has the following advantages compared with the prior art:
[0074] The flip chip defect detection method of the present invention constructs original classification loss, supervised comparison loss, self-calibration loss and cross-calibration loss, and effectively trains the improved prototype network based on the four losses, continuously improves the model, enhances the generalization ability of the prototype improved network, and greatly improves the accuracy of flip chip defect detection in small sample conditions;
[0075] The flip chip defect detection method of the present invention obtains rich semi-supervised information based on the pseudo-label sample screening mechanism of cognitive uncertainty and entropy, which can expand the data set and realize further effective training of the prototype improvement network. BRIEF DESCRIPTION OF THE DRAWINGS
[0076] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.
[0077] Figure 1 is a flow chart of the method of the present invention;
[0078] Figure 2 is a schematic diagram of the construction of an improved prototype network loss function in an embodiment of the present invention;
[0079] Figure 3 4 is a flow chart of a pseudo-label sample screening mechanism in an embodiment of the present invention. DETAILED DESCRIPTION
[0080] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1
[0081] Reference Figure 1 As shown, the present invention relates to a flip chip defect detection method, comprising:
[0082] Step S1: collecting one-dimensional vibration data of the flip chip, and dividing the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set;
[0083] Step S2: dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training the preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss;
[0084] Step S3: assigning pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, adding the pseudo label samples to the training set to expand the training set, and further training the improved prototype network with the expanded training set to obtain a trained improved prototype network;
[0085] Step S4: importing the test set into the trained improved prototype network to perform defect detection and obtain detection results.
[0086] It should be noted that the reason why the model is named as improved prototype network in step S2 in this embodiment is that four loss functions are pre-constructed in this embodiment, which is equivalent to improving the prototype network.
[0087] The following is a detailed introduction to this embodiment:
[0088] Step S1: Collect one-dimensional vibration data of the flip chip, including solder ball cold soldering fault signals, solder ball crack fault signals and solder ball leaking fault signals; cut the one-dimensional vibration data of the flip chip into samples of preset length and perform normalization processing; divide the normalized flip chip samples into a training sample set (i.e., training set), a test sample set (i.e., test set) and an unlabeled sample set, wherein the number of samples in the training sample set is much smaller than that in other sample sets.
[0089] Step S2: Divide the training sample set into a support set and a query set. The training set is defined as: ,in Indicates training samples, express The corresponding label, Represents the total number of samples in the training set. Randomly selected from each category samples form the support set: ,in Indicates Support samples, express The corresponding label, represents the total number of samples in the support set, Represents the number of categories contained in the training set. The remaining samples in each category are taken as the query set: ,in Indicates A sample of queries, express The corresponding label, Represents the number of remaining samples in each category (i.e., each category is extracted samples), Indicates the total number of samples in the query set.
[0090] Step S3: using the support set and the query set as inputs of the improved prototype network, constructing four loss functions, training the model through the four loss functions, and updating the model parameters.
[0091] (1) Original classification loss
[0092] The support samples and the query samples are processed by the feature extractor of the improved prototype network, mapped to a high-dimensional feature space, and generated support embedding features. and query embedding features Then, we calculate the support embedding features for each category The average vector of the support set is used to obtain the prototype of the corresponding category in the support set (support set prototype), and the formula is:
[0093] ;
[0094] in, Belong to category The support set prototype, A set of support set prototypes for all categories, represents the indicator function, To improve the feature extractor of the prototype network, the feature extractor is mainly composed of a convolutional layer for extracting features. Since it is a prior art, this embodiment will not be repeated. Calculate the query embedding feature The Euclidean distance between each of the support set prototypes , the Euclidean distance is converted into a posterior probability for prediction of the query sample. The specific formula is as follows:
[0095] ;
[0096] ;
[0097] in, represents the two-norm, represents the posterior probability of the query sample, Indicates the category number. What needs to be explained here is: is the label of the query sample, is the number of samples, is the label of the category, , a total of categories.
[0098] The cross entropy loss between the true label and the predicted label of the query sample is calculated and used as the original classification loss. The formula is:
[0099] ;
[0100] in, represents the cross entropy function.
[0101] (2) Supervised Contrastive Loss
[0102] For each category , the support set prototype of each category As an anchor point, The query sample of the class is considered a positive sample ,in Indicates Positive samples, Indicates The label corresponding to the positive sample. Then the distance calculation formula between the anchor point and each positive sample is:
[0103] ;
[0104] in, Represents the cosine distance.
[0105] Will not belong to The query samples of the class are considered negative samples ,in Indicates negative samples, Indicates The labels corresponding to the negative samples. Then the distance calculation formula between the anchor point and each negative sample is:
[0106] ;
[0107] Finally, according to and Construct the supervised contrast loss, the formula is:
[0108] ;
[0109] in, represents a hyperparameter.
[0110] (3) Self-calibration loss and cross-calibration loss
[0111] This embodiment introduces self-calibration loss and cross-calibration loss to impose constraints on the support set prototype to improve the support set prototype.
[0112] For the self-calibration loss, we first calculate the support embedding feature With each of the support set prototypes The Euclidean distance between them is then converted into the posterior probability. The specific formula is as follows:
[0113] ;
[0114] Calculate the self-calibration loss as follows:
[0115] ;
[0116] in, represents the posterior probability of supporting the sample.
[0117] For the cross calibration loss, the query is embedded into the feature The average vector of is regarded as the prototype of the category (query set prototype). The query set prototype is regarded as the sample to be predicted, and the label of the query set prototype is predicted in the same way as the query sample. Specifically, the query set prototype calculation formula for each category is:
[0118] ;
[0119] The Euclidean distance between the query set prototype and the support set prototype is calculated, and the Euclidean distance is converted into the posterior probability. The specific formula is as follows:
[0120] ;
[0121] Calculate the cross calibration loss as:
[0122] .
[0123] (4) Training and improving the prototype network
[0124] The support set and the query set are used as inputs of an improved prototype network, and the improved prototype network is trained according to the above four losses.
[0125] Step S4: introduce semi-supervised pseudo-label learning to assign pseudo-labels to the samples in the unlabeled sample set, add high-quality pseudo-label samples to the training sample set to expand the data set, and further train and improve the prototype network.
[0126] The flowchart of the pseudo-label sample screening mechanism based on cognitive uncertainty and entropy in this embodiment is as follows: Figure 3 As shown. The unlabeled sample set is defined as: ,in Indicates unlabeled samples, represents the number of samples in each category, Indicates the total number of samples in the unlabeled sample set. During the training process of the improved prototype network, the unlabeled samples are input into the feature extractor of the improved prototype network for processing to obtain unlabeled embedded features, and the Euclidean distance between the unlabeled embedded features and each of the support set prototypes is calculated. The Euclidean distance is converted into a probability vector for the prediction of the unlabeled sample set. The prediction probability of each unlabeled sample is recorded every several equal iteration steps (for example, 100 iterations, and the prediction probability is recorded every 10 iterations). Indicates that Indicates the number of recorded steps. Indicates The predicted probability of unlabeled samples is then calculated:
[0127] ;
[0128] in, Represents a function that returns the label corresponding to the maximum probability value.
[0129] Each of the unlabeled samples obtains Prediction labels The predicted label with the highest frequency of occurrence is recorded as , and record the frequency as ,like Not less than the frequency threshold , then the unlabeled sample is selected as an easy sample, denoted as ; Otherwise, reselect another unlabeled sample to determine whether to assign a pseudo label.
[0130] Information entropy is used to measure the easy samples The uncertainty of the predicted probability (i.e. The information entropy of easy samples) is:
[0131] ;
[0132] in Indicates The predicted probabilities belong to An uncertainty threshold is set for each class to control the strictness of assigning pseudo labels to the unlabeled samples:
[0133] ;
[0134] in, Indicates The predicted labels corresponding to the easy samples are represents the indicator function, The category number.
[0135] If the uncertainty of the predicted probability is not greater than the uncertainty threshold, the easy sample is assigned The pseudo label is added to the training sample set to expand the data set, and the prototype network is further trained and improved. Otherwise, another unlabeled sample is selected to determine whether to assign a pseudo label.
[0136] Step S5: Preferably, in this embodiment, steps S2-S4 are repeated until all the unlabeled sample sets are assigned pseudo labels. The finally trained improved prototype network is saved, and the test sample set is imported into it for defect detection to obtain the diagnosis results of all samples. Embodiment 2
[0137] This embodiment provides a flip chip defect detection system, including:
[0138] Collection and division module: used to collect the one-dimensional vibration data of the flip chip and divide the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set;
[0139] A first training module: used for dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training a preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss;
[0140] The second training module is used to assign pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, add the pseudo label samples to the training set to expand the training set, and further train the improved prototype network through the expanded training set to obtain a trained improved prototype network;
[0141] Detection module: used to import the test set into the trained improved prototype network for defect detection to obtain the detection result. Embodiment 3
[0142] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the flip chip defect detection method described in the first embodiment are implemented. Embodiment 4
[0143] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the flip chip defect detection method described in the first embodiment are implemented.
[0144] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0145] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0146] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0147] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0148] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0149] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.
Claims
1. A flip chip defect detection method, characterized in that: include: Step S1: collecting one-dimensional vibration data of the flip chip, and dividing the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set; Step S2: dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training the preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss; Step S3: assigning pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, adding the pseudo label samples to the training set to expand the training set, and training the improved prototype network with the expanded training set to obtain a trained improved prototype network; The method for obtaining the trained improved prototype network in step S3 is: The unlabeled sample set is defined as ,in, , Indicates unlabeled samples, represents the number of samples in each category, Represents the total number of samples in the unlabeled sample set; During the training process of the improved prototype network, the unlabeled samples are input into the feature extractor of the improved prototype network for processing to obtain unlabeled embedded features, and the Euclidean distance between the unlabeled embedded features and each of the support set prototypes is calculated. The Euclidean distance is converted into a probability vector for the prediction of the unlabeled sample set, and the prediction probability of each unlabeled sample corresponding to the unlabeled embedded feature is recorded every several equal iteration steps. The formula is: ,in Indicates the number of recorded steps. Indicates The predicted probability of unlabeled samples; then calculate the predicted label, the formula is: ; in, Represents a function that returns the label corresponding to the maximum probability value; Each unlabeled sample is The predicted labels ; The predicted label with the highest frequency of occurrence is recorded as , the frequency is recorded as ,like Not less than the frequency threshold , then the unlabeled sample is selected as an easy sample, denoted as ; Otherwise, select another unlabeled sample to determine whether to assign a pseudo label; Information entropy is used to measure the easy samples The uncertainty of the predicted probability is: ; in Indicates The predicted probabilities belong to The probability of the class; an uncertainty threshold is set for each class to constrain the assignment of pseudo labels to the unlabeled samples, and the uncertainty threshold formula is: ; in, Indicates The predicted labels corresponding to the easy samples are represents the indicator function, is the category label; If the uncertainty of the predicted probability is not greater than the uncertainty threshold, the easy sample is assigned Pseudo-label, take it as a pseudo-label sample and add it to the training set, train the improved prototype network, and obtain a trained improved prototype network; otherwise, reselect another unlabeled sample to determine whether to assign a pseudo-label; Step S4: importing the test set into the trained improved prototype network to perform defect detection and obtain detection results.
2. The flip chip defect detection method according to claim 1, characterized in that: The method for dividing the training set into a support set and a query set in step S2 is: The training set is defined as ,in Indicates training samples, express The corresponding label, Represents the total number of samples in the training set; Randomly select from each category of the training set The samples form the support set ,in, , Indicates Support samples, express The corresponding label, represents the total number of samples in the support set, is the number of categories contained in the training set; The remaining samples in each category of the training set are taken as query sets ,in, , Indicates A sample of queries, express The corresponding label, Indicates that each category is extracted The number of remaining samples after samples, Indicates the total number of samples in the query set.
3. The flip chip defect detection method according to claim 2, characterized in that: The method for constructing the original classification loss includes: The support samples and the query samples are processed by a pre-set feature extractor of the improved prototype network, mapped to a high-dimensional feature space, and generated support embedding features. and query embedding features ; Calculate the support embedding features for each category The average vector of is used as the support set prototype of this category. The formula is: ; in, Belong to category The support set prototype, A set of support set prototypes for all categories, represents the indicator function, To improve the feature extractor of the prototype network; Calculate the query embedding feature The Euclidean distance between each of the support set prototypes , convert the Euclidean distance into the posterior probability to obtain the predicted label of the query sample. The formula is: ; ; in, represents the two-norm, represents the posterior probability of the query sample, Indicates the category label; The cross entropy loss between the true label and the predicted label of the query sample is calculated and used as the original classification loss. The formula is: ; in, represents the cross entropy function.
4. The flip chip defect detection method according to claim 3, characterized in that: The method for constructing the supervised contrast loss includes: For each category , each support set prototype As an anchor point, The query sample of the class is taken as a positive sample ,in Indicates Positive samples, Indicates The labels corresponding to the positive samples are calculated, and the distance between the anchor point and each positive sample is calculated using the formula: ; in, represents the cosine distance; Will not belong to The query sample of the class is taken as a negative sample ,in Indicates negative samples, Indicates The labels corresponding to the negative samples are calculated, and the distance between the anchor point and each of the negative samples is calculated using the formula: ; according to and Construct a supervised contrast loss, the formula is: ; in, Represents a hyperparameter.
5. The flip chip defect detection method according to claim 3, characterized in that: The self-calibration loss and the cross-calibration loss are used to impose constraints on the support set prototype to improve the support set prototype, wherein, The method for constructing the self-calibration loss includes: calculating the support embedding feature With each of the support set prototypes The Euclidean distance between them is then converted into the posterior probability using the formula: ; in, represents the posterior probability of supporting the sample, Indicates Support set prototypes; The self-calibration loss is calculated as follows: ; in, represents the cross entropy function; The method for constructing the cross calibration loss includes: generating support embedding features and query embedding features , calculate the query embedding features in each category The average vector of is taken as the prototype of the query set of this category, and the formula is: ; in, Belong to category The support set prototype, A collection of support set prototypes for all categories; The Euclidean distance between the query set prototype and the support set prototype is calculated, and the Euclidean distance is converted into a posterior probability, the formula is: ; in, Indicates queryset prototypes; The cross calibration loss is calculated as follows: ; in, Indicates The labels corresponding to the query set prototypes.
6. The flip chip defect detection method according to claim 1, characterized in that: The method for dividing the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set in step S1 is: The collected one-dimensional vibration data are cut into samples of preset length and normalized; the normalized samples are divided into a training set, a test set and an unlabeled sample set, wherein the number of samples in the training set is smaller than that in the test set and the unlabeled sample set.
7. A flip chip defect detection system, characterized in that: include: Collection and division module: used to collect the one-dimensional vibration data of the flip chip and divide the collected one-dimensional vibration data into a training set, a test set and an unlabeled sample set; A first training module: used for dividing the training set into a support set and a query set, constructing a plurality of loss functions based on the support set and the query set, and training a preset improved prototype network through the plurality of loss functions; wherein the plurality of loss functions include original classification loss, supervised contrast loss, self-calibration loss and cross-calibration loss; The second training module is used to assign pseudo labels to samples in the unlabeled sample set to obtain pseudo label samples, add the pseudo label samples to the training set to expand the training set, and train the improved prototype network through the expanded training set to obtain a trained improved prototype network; The method for obtaining the trained improved prototype network in the second training module is: The unlabeled sample set is defined as ,in, , Indicates unlabeled samples, represents the number of samples in each category, Represents the total number of samples in the unlabeled sample set; During the training process of the improved prototype network, the unlabeled samples are input into the feature extractor of the improved prototype network for processing to obtain unlabeled embedded features, and the Euclidean distance between the unlabeled embedded features and each of the support set prototypes is calculated. The Euclidean distance is converted into a probability vector for the prediction of the unlabeled sample set, and the prediction probability of each unlabeled sample corresponding to the unlabeled embedded feature is recorded every several equal iteration steps. The formula is: ,in Indicates the number of recorded steps. Indicates The predicted probability of unlabeled samples; then calculate the predicted label, the formula is: ; in, Represents a function that returns the label corresponding to the maximum probability value; Each unlabeled sample is The predicted labels ; The predicted label with the highest frequency of occurrence is recorded as , the frequency is recorded as ,like Not less than the frequency threshold , then the unlabeled sample is selected as an easy sample, denoted as ; Otherwise, select another unlabeled sample to determine whether to assign a pseudo label; Information entropy is used to measure the easy samples The uncertainty of the predicted probability is: ; in Indicates The predicted probabilities belong to The probability of the class; an uncertainty threshold is set for each class to constrain the assignment of pseudo labels to the unlabeled samples, and the uncertainty threshold formula is: ; in, Indicates The predicted labels corresponding to the easy samples are represents the indicator function, is the category label; If the uncertainty of the predicted probability is not greater than the uncertainty threshold, the easy sample is assigned Pseudo-label, take it as a pseudo-label sample and add it to the training set, train the improved prototype network, and obtain a trained improved prototype network; otherwise, reselect another unlabeled sample to determine whether to assign a pseudo-label Detection module: used to import the test set into the trained improved prototype network for defect detection to obtain the detection result.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the flip chip defect detection method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the flip chip defect detection method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Small sample remote sensing image classification method based on restrictive prototype comparison network
CN115984621A
Small sample metal defect detection method based on self-supervised learning
CN117611536A
Self-adaptive prototype network and pseudo tag semi-supervised learning diagnosis method and device
CN117708694A
Deep-learning-based surface defect detection method for mobile phone battery
WO2023155069A1
