Image-based defect detection method for electronic components

Abstract

The present invention provides an image-based electronic component defect detection method, comprising: step 1: initial training data set; step 2: training initial classification model; step 3: judging whether a new category image appears, if so, executing step 4, otherwise executing step A1; step 4: collecting new category training data; step 5: incrementally adjusting the initial model; step 6: updating the classification model, and then executing step A1; step A1: classifying the test image; step A2: the image enters the corresponding preprocessing model. The beneficial effects of the present invention are: the present invention introduces the continuous learning method into the capacitor recognition task to effectively alleviate catastrophic forgetting and improve the model recognition accuracy. In the field of image classification, the trained deep neural network classification model can recognize objects of different categories in the image, and can already achieve a high recognition rate.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an image-based electronic component defect detection method. Background Art

[0002] In the field of image classification, trained deep neural network classification models can identify objects of different categories in images and have achieved a high recognition rate. However, most of the current deep neural networks cannot meet the requirements of identifying untrained objects. That is, deep neural network models can only recognize objects of known categories. For untrained objects, deep neural network models will mistakenly classify the objects into known categories, resulting in a decrease in recognition accuracy.

[0003] At present, the industry's solution to the above-mentioned catastrophic forgetting problem is basically based on the model's ability to train all of these massive data at the same time. However, in practical applications, models are often faced with continuously increasing streaming data. Due to some uncontrollable factors, the model cannot be trained using the full amount of data in most such cases; because the data update accumulates too quickly, high-frequency full training to update the model greatly reduces the efficiency of the model; due to the limited computing and storage resources of the machine, the model cannot always achieve full training. In this way, according to the learning steps of general deep neural networks, the model can only continuously learn and train in the order in which the data appears. This not only requires the deep neural network to be able to quickly adapt to new knowledge on new data, but also to still have good performance on all learned data. How to apply the continuous learning method to the thin film capacitor recognition task to solve the problem of catastrophic forgetting is the main research content of this invention. Summary of the invention

[0004] The present invention provides an image-based electronic component defect detection method, comprising the following steps:

[0005] Step 1: Initial training data set;

[0006] Step 2: Train the initial classification model;

[0007] Step 3: Determine whether a new category of image appears. If so, execute step 4, otherwise execute step A1;

[0008] Step 4: Collect new category training data;

[0009] Step 5: Incrementally adjust the initial model;

[0010] Step 6: Update the classification model and then execute step A1;

[0011] Step A1: classify the test image;

[0012] Step A2: The image enters the corresponding preprocessing model.

[0013] As a further improvement of the present invention, step 4 and step 5 include:

[0014] Training steps: If a dataset of new category samples is added to the dataset for the first time, use the dataset of new category samples to train the incremental learning neural network; at the same time, use the dataset of new category samples to train the open set recognition model, so that the open set recognition model can effectively identify the datasets of old category samples and the datasets of new category samples.

[0015] As a further improvement of the present invention, in the steps 4 and 5, if the data set of the new category samples added to the data set for the second time is different from the data set in the training step, the data set of the new category samples in the training step automatically becomes the data set of the old category samples, and the class incremental learning neural network is trained using the data set of the new category samples, and the open set recognition model is trained using the data set of the new category samples, and the open set recognition model trained in the training step is discarded, so that the new open set recognition model can effectively recognize the data sets of the old category samples and the data sets of the new category samples; the class incremental learning neural network is saved and recorded as the class incremental learning network updated in the most recent stage of continuous learning, and at the same time, the class incremental learning neural network in the training step is saved and recorded as the class incremental learning neural network retained after the last increment.

[0016] As a further improvement of the present invention, in step A1, the test set first passes through the Model_A model, and then the open set recognition model is used to classify the data set. The classified old category and new category data pass through the feature extractor of Model_B, and enter the corresponding fully connected layers W1 and W2 after recognition enhancement, respectively. The output results are combined to finally obtain the predicted category of the test set image.

[0017] The beneficial effects of the present invention are as follows: the present invention introduces a continuous learning method into the capacitance recognition task to effectively alleviate catastrophic forgetting and improve the recognition accuracy of the model. In the field of image classification, the trained deep neural network classification model can recognize objects of different categories in the image and can achieve a relatively high recognition rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the present invention;

[0019] Figure 2a and Figure 2b It is a model framework diagram;

[0020] Figure 3 This is the LWF algorithm framework diagram;

[0021] Figure 4 It is the enhanced discrimination method diagram;

[0022] Figure 5 It is a parallel Resnet model diagram;

[0023] Figure 6 This is the model diagram after adding the SE module;

[0024] Figure 7 It is a graph showing the impact of modifying the model structure on the recognition accuracy of continuous learning;

[0025] Figure 8 It is a comparative experimental diagram of the film capacitor data set. DETAILED DESCRIPTION

[0026] The present invention discloses an image-based electronic component defect detection method, which mainly studies how to introduce a continuous learning method into a capacitor recognition task to effectively alleviate catastrophic forgetting and improve the model recognition accuracy.

[0027] The present invention mainly comprises:

[0028] Dataset preparation: The data set has been collected and comes from the thin film capacitor image dataset collected at the industrial site.

[0029] Preprocessing of the data set: After manual cleaning and preprocessing, the final data is currently formed. The capacitor images contain 5 different colors, including blue, new white, new yellow, old yellow and dark yellow, and old gray; and also include 8 pictures taken from different angles, including top, bottom, left, right, front, back, pin front, and pin back. The total number of samples is 102,926.

[0030] Model structure design: Preliminarily complete the overall design and coding implementation of the continuous learning model framework.

[0031] Model training and testing: The model proposed in this invention was trained and tuned on the data set, and preliminary results were obtained. At the same time, the current mainstream model was reproduced, and the relevant thin film capacitor classification tasks were tested on the data set of this invention, and preliminary evaluation results were obtained.

[0032] In the current thin film capacitor defect detection task, multi-class incremental learning (MCCL) is a relatively important task in defect detection, which aims to solve the practical problem of allowing the previous model to continuously learn new knowledge from new samples. The disadvantage of MCCL is that they are affected by catastrophic forgetting. Secondly, since the requirement for device storage data is associated with the category recognized by the model, continuous continuous learning makes it impossible for devices with limited memory to implement the MCCL algorithm. The research of the present invention shows that this is due to the combination of two factors: (A) the imbalance between the new and old model categories, and (B) most MCCL algorithms use old sample sets, which increases the memory requirements of the incremental steps in continuous learning. Therefore, the present invention proposes a simple and effective method, called the (Open Set Continual Learning) OSCL algorithm, to preserve the knowledge of existing classes without storing any of their basic category data, while allowing the classifier to learn new category knowledge and forget less basic category information. In OSCL, we then added an open set incremental strategy: it uses enhanced discrimination and open set recognition methods to overcome the imbalance between the new and old model categories without storing any data. The flow chart of the model designed by the present invention is as follows Figure 1 shown.

[0033] An image-based electronic component defect detection method comprises the following steps:

[0034] Step 1: Initial training data set;

[0035] Step 2: Train the initial classification model;

[0036] Step 3: Determine whether a new category of image appears. If so, execute step 4, otherwise execute step A1;

[0037] Step 4: Collect new category training data;

[0038] Step 5: Incrementally adjust the initial model;

[0039] Step 6: Update the classification model and then execute step A1;

[0040] Step A1: classify the test image;

[0041] Step A2: The image enters the corresponding preprocessing model.

[0042] The model framework diagram proposed by the present invention is as follows Figure 2a and Figure 2b As shown. Figure 2a As shown, during the training phase:

[0043] (1) If the dataset only contains images of old categories, do not train the open set recognition model, but only train the initial classification network.

[0044] (2) If a new category of samples is added to the dataset for the first time, the new category of samples is used to train the incremental learning neural network. At the same time, the new category of samples is used to train the open set recognition model, so that the open set recognition model can effectively identify the old category of samples and the new category of samples.

[0045] (3) If the dataset of new category samples is added for the second time (different from the dataset in (2)), the dataset of new category samples in (2) automatically becomes the dataset of old category samples, and the class incremental learning neural network is trained using the dataset of new category samples (this dataset is the dataset of new category samples added for the second time, and has nothing to do with (2), the same below). At the same time, the open set recognition model is trained using the dataset of new category samples, and the open set recognition model trained in (2) is discarded, so that the new open set recognition model can effectively recognize the datasets of old category samples and the datasets of new category samples. The class incremental learning neural network is saved and recorded as the class incremental learning network updated in the most recent stage of continuous learning. At the same time, the class incremental learning neural network in (2) is saved and recorded as the class incremental learning neural network retained after the last increment.

[0046] (4) If a new category of images is added to the dataset for the nth time (n>2), the training process is the same as the steps in (3).

[0047] like Figure 2b As shown in the figure, in the test phase, the test set first passes through the Model_A model, and then uses the open set recognition model to classify the data set. The classified old category and new category data pass through the feature extractor of Model_B, and enter the corresponding fully connected layers W1 and W2 after recognition enhancement. The output results are combined to finally obtain the predicted category of the test set image.

[0048] The first major problem studied in this invention is how to train on a new category of data set without an old data set so that the newly obtained model not only maintains the knowledge of the old category samples but also learns the knowledge of the new category samples. For a data set with m old categories (m old types of thin film capacitor images) and n new categories (n new types of thin film capacitor images), the knowledge extracted from the initial neural network that classifies the m old categories is used to learn a new neural network to classify the m+n categories. Specifically, the model parameters are updated through a loss function, in which the loss function is a fusion of the distillation loss and the cross entropy loss function. The following example illustrates the details of the steps: First, we represent the new category of thin film capacitor image samples as:

[0049]

[0050] Where N represents the number of samples of new type of film capacitors, x i and i They represent the image and label of the sample respectively. The old category film capacitor sample is represented as:

[0051]

[0052] Where M represents the number of samples of old-category film capacitors. In most practical application scenarios, M is much smaller than N, which means that the data set of new-category film capacitors will become larger and larger. The logits output of the classifier for old-category film capacitors and the logits output of the classifier for new-category film capacitors are expressed as:

[0053] O m (x) = [o1(x), ..., o m (x)] (2-3)

[0054] O m+n (x) = [o1(x), ..., o m (x), o m+1 (x), ..., o m+n (x)] (2-4)

[0055] We update the parameters of the continuous learning model by optimizing the loss function, where the loss function of the continuous learning algorithm is expressed as:

[0056] Loss = αLoss D +βLoss CE (2-5)

[0057] The distillation loss function is expressed as:

[0058]

[0059] in The cross entropy loss function is expressed as:

[0060]

[0061] where p k (x) represents the probability value obtained by sending the logits output of the kth category into the softmax layer among the m+n new and old categories of film capacitors.

[0062] The schematic diagram of the LWF algorithm used in the training phase of the present invention is as follows Figure 3As shown in the figure, for the convenience of explanation, we choose the structure diagram of AlexNet, where θ Shared Represents the first five convolutional layers and the last two fully connected layers of the network. The last layer is the category-related output layer, whose parameters are represented by θ alone. Old It means that if a new classification task is added, the parameters of the new task will be randomly initialized, denoted as θ New During the training process, we use the SGD optimizer with regularization to train the network. The steps are as follows:

[0063] (1) First, fix θ Sharef ,θ Old Unchanged, and then use the new task dataset to train θ New Until convergence.

[0064] (2) Then jointly train all parameters θ Sharef ,θ Old ,θ New until the network converges.

[0065] There are two loss functions here. The first is the loss function of the normal classification network for samples in the new task, and the second is the distillation loss function, which are expressed as follows:

[0066]

[0067]

[0068]

[0069] Here in equation 2-9 represents the softmax layer output of the network, and yn represents the one-hot groundtruth label vector. In equation 2-10, l represents the number of labels. The meaning of this constraint is that for a picture, the new model should try to learn or retain the old category that may be predicted by the old model.

[0070] Introduce how the open set recognition model distinguishes new categories of data sets. First, define the model. After continuous learning in the i-th stage, the logits output of the fully connected layer is

[0071] O n (x)=[O1(x),O2(x),...,O k (x), ..., O n (x)] (2-11)

[0072] Then we define the critical value coefficient based on experimental experience, usually a constant. In the test phase, we perform softmax processing on the logits output obtained after inputting the test data. For the kth film capacitor test image in the test data, the following conditions are met:

[0073] η≥max(softmax(o k (x))) (2-12)

[0074] The kth image is considered to belong to the old category thin film capacitor dataset, otherwise it is determined that the image belongs to the new category thin film capacitor dataset.

[0075] In order to achieve continuous learning of the model for large-scale thin film capacitor data, after updating the network parameters, we input the logits output of the new category into the bias correction structure, which ensures that the classification network can more accurately classify new and old categories of thin film capacitors. Figure 4 Schematic diagram of the enhanced discriminant method to distinguish between new and old category data.

[0076] The specific steps of this method are as follows: For the logits output of the classifier for the old category film capacitor samples and the new category film capacitor samples, we define it as:

[0077] O m+n (x) = [o1(x), ..., o m (x), o m+1 (x), ..., o m+n (x)]=(W Model ) T φ(x) (2-13)

[0078] Where φ(x) represents the feature extractor of the classification network Resnet, W Model Represents the weight of the fully connected layer of the classification network model Model, which can be described as W Model ={w i , 1≤i≤m+n}, wi represents the weight vector of the i-th class.

[0079] In the training phase, we first set the weights W of the fully connected layer of the classification network model Model Model Rewrite it as follows:

[0080] W Model =(W old , W new ) (2-14)

[0081] Here W old =(w1, w2, ..., w m ), W new =(w m+1 , wm+2 , ..., w m+n ). We can do this by changing W ola and W new The relative norm of enables the model to better distinguish samples of new and old categories of film capacitors. Here we make the following transformations: and Here μ is the sum of the average norms of the two:

[0082]

[0083] This operation is done so that the model can better understand the samples of new and old types of film capacitors. At this point we can get

[0084] O m+n (x) = [o1(x), ..., o m (x), o m+1 (x), ..., o m+n (x)]=(W old , W new ) T φ(x) (2-16)

[0085] At this point, we retain the model’s feature extractor φ(x), and then set the weights of the model’s fully connected layer to and Define the classification network model Model. The logits output after the above steps is

[0086]

[0087]

[0088] As shown in the above equation, by changing and norm, on the old category film capacitor dataset, O m+n (x) The classification effect is better; on the new category film capacitor dataset, O m+n (x) The classification effect is better.

[0089] At this point, the defect detection model based on continuous learning is completed.

[0090] To analyze the impact of modifying the model structure and attention mechanism on continuous learning, we conducted experiments on the CIFAR-100 dataset, where the CIFAR-100 experiment includes 5 incremental steps and 20 classes per step. First, we divide the method of modifying the model into the following one baseline and three variants:

[0091] Benchmark 1: Continuous learning using a standard 32-layer ResNet model.

[0092] Variation1: Uses the existing OSCL method based on Benchmark1.

[0093] Variation2: Uses the existing OSCL method based on benchmark 2.

[0094] Variation3: Calculate the upper limit of the OSCL method based on Benchmark2, that is, assume that the open set recognition algorithm in the OSCL algorithm can correctly classify all new and old category data.

[0095] The SE attention mechanism module and parallel structure diagram are as follows Figure 5 and Figure 6 shown.

[0096] Figure 7 The results of these experiments are summarized, with the benchmark performing the worst. First, we analyzed the case without using old class data: in Variation1, we used a parallel convolutional network structure, which does not reduce the impact of catastrophic forgetting on the gain during continuous learning. However, in Variation2, we used a ResNet model with an SE module added, and the overall performance of the model improved significantly with each additional module compared to Benchmark1, which is completely different from the average improvement in Variation 2. As shown in Variation3, when we combine the contents of Variation1 and Variation2. The accuracy of the combination is much greater than that of each module alone. This indirectly proves that our method is more effective in solving the problem of catastrophic forgetting.

[0097] In order to analyze the effect of the OSCL algorithm, we first use CIFAR-100 to conduct experiments. In the CIFAR-100 experiment, we set up two sets of benchmark tests and three sets of comparative experiments to demonstrate the effect of OSCL, comparing:

[0098] Benchmark 1: Continuous learning using only knowledge distillation and cross entropy loss function.

[0099] Benchmark2: Incremental learning using a model that combines the SE module with knowledge distillation and cross entropy loss function.

[0100] Comparison1: Using the existing OSCL method based on Benchmark1.

[0101] Comparison2: Uses the existing OSCL method based on Benchmark2.

[0102] Comparison3: Calculate the upper limit of the OSCL method based on Benchmark2, assuming that the open set recognition algorithm in the OSCL algorithm can correctly classify all new and old category data.

[0103] Table 1 Recognition accuracy of OSCL algorithm benchmark and comparative experiment (set to increment 20 categories at a time)

[0104]

[0105] Table 1 summarizes the experimental results of the OSCL algorithm. Comparing the results of Benchmark1 and Comparison1, we find that using knowledge distillation and cross entropy loss function alone cannot solve the imbalance problem between new and old data. Our OSCL method can overcome this problem through open set recognition. Comparing the results of Benchmark2 and Comparison2, we believe that the OSCL method can further improve the accuracy of model recognition on the basis of overcoming catastrophic forgetting by using the attention mechanism, which once again verifies the effectiveness of the OSCL method. Comparing the results of Comparison2 and Comparison3, we believe that the OSCL method still has a lot of room for improvement, which depends on the accuracy of the open set recognition algorithm. If we can perfectly identify new and old categories, then the OSCL algorithm will be further improved.

[0106] In order to analyze the prediction effect of the OSCL algorithm on thin film capacitor images, we use the thin film capacitor image dataset for experiments. In the experiment, we set 220 images in the training set and 80 images in the test set for each type of capacitor image, totaling 68 types of capacitor images. The experiments are compared respectively: Benchmark: Continuous learning using only knowledge distillation and cross entropy loss function.

[0107] OSCL: Use existing OSCL methods based on Benchmark.

[0108] Table 2 summarizes the thin-film capacitor images collected at different angles and the different colors included in the existing thin-film capacitor image dataset.

[0109] Table 2 Contents of the film capacitor image dataset

[0110]

[0111] Figure 8The experimental results of OSCL algorithm verified on thin film capacitor image dataset are summarized. Comparing the results of Benchmark and OSCL method, we found that OSCL method can further improve the accuracy of model recognition based on overcoming catastrophic forgetting by using attention mechanism.

[0112] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. An image-based electronic component defect detection method, characterized in that: The steps include: Step 1: Initial training data set; Step 2: Train the initial classification model; Step 3: Determine whether a new category of image appears. If so, execute step 4, otherwise execute step A1; Step 4: Collect new category training data; Step 5: Incrementally adjust the initial model; Step 6: Update the classification model and then execute step A1; Step A1: classify the test image; Step A2: The image enters the corresponding preprocessing model; In the steps 4 and 5, if the data set of the new category samples added to the data set for the second time is different from the data set in the training step, the data set of the new category samples in the training step automatically becomes the data set of the old category samples, and the class incremental learning neural network is trained using the data set of the new category samples, and the open set recognition model is trained using the data set of the new category samples, and the open set recognition model trained in the training step is discarded, so that the new open set recognition model can effectively recognize the data set of the old category samples and the data set of the new category samples; the class incremental learning neural network is saved and recorded as the class incremental learning network updated in the most recent stage of continuous learning, and at the same time, the class incremental learning neural network in the training step is saved and recorded as the class incremental learning neural network retained after the last increment; In step 2, the AlexNet structure diagram is selected, where Represents the first five convolutional layers and the last two fully connected layers of the network. The last layer is the category-related output layer, and its parameters are used separately. It means that if a new classification task is added, the parameters of the new task will be randomly initialized, expressed as ,During the training process, the network is trained using the SGD optimizer with regularization; The step 2 comprises: First fix , Unchanged, then trained using the new task dataset Until convergence; Then all parameters are trained jointly , , Until the network converges; There are two loss functions. The first is the loss function of the normal classification network for samples in the new task, and the second is the distillation loss function, which are expressed as follows: ， ， ， In equation 2-9 represents the softmax layer output of the network, Represents the one-hot type ground truth label vector; in Equation 2-9 Represents the number of labels. The meaning of this constraint is that for a picture, the new model should try to learn or retain the old categories that may be predicted by the old model. The electronic component defect detection method further includes an enhanced discrimination step, wherein the enhanced discrimination step includes: The logits output of the classifier for the old category film capacitor samples and the new category film capacitor samples is defined as: ， in Represented as the feature extractor of the classification network Resnet, Represents a classification network model The weights of the fully connected layer can be described as , Represents the weight vector of the i-th class; In the training phase, the classification network model is first The weights of the fully connected layer Rewrite it as follows: ， here , , by changing and The relative norm of enables the model to better distinguish samples of new and old categories of film capacitors; the following transformations are performed respectively: ,here Take the sum of the two average norms: ， This operation is done so that the model can better understand the samples of new and old types of film capacitors. At this time, we can get ， At this point, keep the model’s feature extractor , and then the weights of the fully connected layer of the model are and ; Define the classification network model After the above steps, the logits output is ， ， As shown in the above equation, by changing and norm, on the old category film capacitor dataset, Compare The classification effect is better; on the new category film capacitor dataset, Compare The classification effect is better.

2. The electronic component defect detection method according to claim 1, characterized in that: The steps 4 and 5 include: Training steps: If a dataset of new category samples is added to the dataset for the first time, use the dataset of new category samples to train the incremental learning neural network; at the same time, use the dataset of new category samples to train the open set recognition model, so that the open set recognition model can effectively identify the datasets of old category samples and the datasets of new category samples.

3. The electronic component defect detection method according to claim 1, characterized in that: In step A1, the test set first passes through the Model_A model, and then the open set recognition model is used to classify the data set. The classified old category and new category data pass through the feature extractor of Model_B, and enter the corresponding fully connected layers W1 and W2 after recognition enhancement, respectively. The output results are combined to finally obtain the predicted category of the test set image.

4. The electronic component defect detection method according to claim 3, characterized in that: The old category data is the old thin film capacitor image category, and the new category data is the new thin film capacitor image category. In the step A1, first, the thin film capacitor image sample of the new category is represented as: ， Where N represents the number of samples of new category film capacitors. and They represent the image and label of the sample respectively; while the old category film capacitor sample is represented as: ， Where M represents the number of samples of the old category film capacitors. The logits output of the classifier for the old category film capacitors and the logits output of the classifier for the new category film capacitors are expressed as: ， ， The parameters of the continuous learning model are updated by optimizing the loss function, where the loss function of the continuous learning algorithm is expressed as: ， The distillation loss function is expressed as: ， in , , and the cross entropy loss function is expressed as: ， in It represents the probability value obtained by sending the logits output of the kth category into the softmax layer among the m + n new and old categories of film capacitors.

5. The electronic component defect detection method according to claim 4, characterized in that: In the open set recognition model, for the kth film capacitor test image in the test data, the following conditions are met: ， The kth image is considered to belong to the old category thin film capacitor dataset, otherwise it is determined that the image belongs to the new category thin film capacitor dataset.

Images