A detection algorithm for small sample defects in QFN chips

Abstract

The present invention provides a detection algorithm for small sample defects in QFN chips, which belongs to the field of computer vision technology. The technical solution specifically includes: obtaining pictures of the upper and lower surfaces of the frame, selecting the ROI area and cropping it; using annotation software to perform data annotation; dividing the annotated pictures into a multi-sample defect data set and a small sample defect data set; constructing a twin Faster-RCNN network model, and using the multi-sample defect data set for meta-learning training; using the trained model to test the small sample defect data set to obtain the defect detection results of the picture and evaluate them; repeating the above operations according to actual production needs and detection effects to obtain the final model. The present invention reduces the demand for small sample defect data by improving the existing Faster-RCNN network structure and the data sampling method in the training process. The inference model is converted into a similarity calculation model, and the algorithm has good scalability.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a detection algorithm for small sample defects in QFN chips. Background Art

[0002] As the chip carrier of integrated circuits, the semiconductor lead frame is a key structural component that uses bonding materials (gold wire, aluminum wire, copper wire) to achieve electrical connection between the internal circuit leads of the chip and the external leads to form an electrical circuit. It acts as a bridge connecting to external wires and is an important basic material for integrated circuit packaging and testing.

[0003] There are many problems that are difficult to avoid in the semiconductor manufacturing process. Defect detection technology based on machine vision is conducive to improving production technology and reducing production costs. It is also the premise and necessary technology to avoid excessively high product defective rates.

[0004] Faster RCNN, a work jointly developed by He Kaiming and the authors of RCNN, is a well-known and widely used deep learning framework in the field of object detection. Since its introduction in 2016, it has been widely used in the field of object detection.

[0005] However, Faster RCNN often has the following problems when used in the field of industrial defect detection:

[0006] First, model training requires a large amount of defect data, but images with defects are not easily available. In highly automated production scenarios, product yields are extremely high, and collecting defect samples is extremely time-consuming. Furthermore, because defects are caused by uncontrolled factors in the production process, they take many different forms, making it difficult to collect comprehensive samples of all these forms. Current deep learning methods for defect detection mostly build models based on large numbers of defect samples. This lack of defect samples makes it difficult to train and launch these models, further limiting the application of deep learning in industrial inspection.

[0007] Secondly, model training requires a large amount of labeled data, and defect labels are not easy to obtain because data labeling of defect images requires manual supervision, that is, a large number of experienced professionals are needed to perform manual labeling, which is very expensive and time-consuming. Summary of the Invention

[0008] To address these issues, the Twin Faster-RCNN object detection algorithm was applied to QFN chip defect detection. The key point is to transform the inference problem into a similarity calculation problem, reducing the requirement for small-sample defect data. This makes the network highly scalable, allowing it to locate and classify small-sample defects that have not been trained on them. Even with only a few images, this network model can achieve very good detection results, surpassing many other algorithms.

[0009] To achieve the above objectives, the present invention proposes a detection algorithm for small sample defects in QFN chips, which is characterized by comprising the following steps:

[0010] (1) Obtain images of the upper and lower surfaces of the frame, select the ROI area of the image and crop it;

[0011] (2) using annotation software to annotate the ROI area of the cropped image and generate a location category label;

[0012] (3) dividing the labeled images into a multi-sample defect dataset and a small-sample defect dataset;

[0013] (4) constructing a twin Faster-RCNN network model, inputting the images in the multi-sample defect dataset into the twin Faster-RCNN network model for meta-learning training to obtain a trained model;

[0014] (5) Using the trained model to test the images in the small sample defect dataset, obtain defect detection results for the images and perform evaluation;

[0015] (6) Repeat steps (3) to (5) several times according to actual production requirements and detection results to obtain the final model for performing small sample defect detection on QFN chips;

[0016] Furthermore, the ROI area in step 1 is a QFN chip area, and the method for selecting the ROI area from the image is a template matching algorithm.

[0017] Furthermore, in step 2, data annotation is performed on the cropped ROI area, that is, the position and category of the defects in the image are marked using a rectangular frame.

[0018] Furthermore, in step 3, the multi-sample defect dataset and the small-sample defect dataset are divided according to data volume. Defects with a large number of defect samples are divided into the multi-sample defect dataset, and defects with only a small number of samples are divided into the small-sample defect dataset.

[0019] Furthermore, the twin Faster-RCNN network model in step 4 is to add a twin branch to the Faster-RCNN network, and the twin branch shares a parameter structure with the backbone network of the Faster-RCNN for extracting image feature vectors.

[0020] Furthermore, the meta-learning process in step 4 specifically includes:

[0021] (4.1) Randomly select N types of defects from the multi-sample defect dataset, randomly select one type of defect from the N types of defects and extract K+1 images, of which K images are added to the support set SupportSet, and the remaining image is used as the query set QuerySet. For each of the remaining N-1 types of defects, K images are randomly selected and added to the support set SupportSet;

[0022] (4.2) Inputting N*K images in the support set SupportSet and their corresponding position category labels into the twin branches to obtain N*K feature vectors;

[0023] (4.3) Input the images in the query set QuerySet into the Faster-RCNN network, calculate the similarity between the feature vectors generated by ROIPooling and the N*K feature vectors obtained by the twin branches, and then input the obtained similarity vectors into the predictor head layer for classification and precise positioning;

[0024] (4.4) Calculate the position and category loss through the loss function;

[0025] (4.5) Set the number of cycles, repeat (4.1) to (4.4) and calculate the average loss of multiple cycles and update the parameters by gradient backpropagation.

[0026] Furthermore, the calculation of similarity in the step (4.3) is specifically to perform channel-by-channel dot product between the feature vector generated by the ROI Pooling and the N*K feature vectors obtained by the twin branches, and calculate the vector similarity in the feature dimension.

[0027] Furthermore, the evaluation in step 5 uses missed detection rate and false detection rate as evaluation criteria.

[0028] Beneficial effects of the present invention:

[0029] This paper improves the existing Faster-RCNN network structure and data sampling methods in the training process, fully utilizing the information of small-sample defects and reducing the required amount of small-sample defect data. By converting the inference model into a similarity calculation model, the algorithm has good scalability and can locate and classify small-sample defects that have not been trained. Even defects with only a few images can achieve a very good detection rate using this network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of a detection algorithm framework for small sample defects in QFN chips according to an embodiment of the present invention.

[0031] Figure 2 The figure is a flow chart of a detection algorithm for a small number of sample defects in QFN chips according to an embodiment of the present invention.

[0032] Figure 3 Schematic diagram of the detection framework of the classic Faster RCNN target detection algorithm in the prior art of the embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of the detection framework of the twin Faster RCNN target detection algorithm provided by an embodiment of the present invention.

[0034] Figure 5 Schematic diagram of the network structure of twin branches for extracting features provided in an embodiment of the present invention.

[0035] Figure 6 It is a flow chart of the meta-learning training eposide process provided by an embodiment of the present invention.

[0036] Figure 7 This is a flowchart of sampling a data set in the meta-learning training eposide process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings and examples.

[0038] Figure 1 This is a schematic diagram of a detection algorithm framework for small sample defects in QFN chips according to an embodiment of the present invention. Figure 2 The following is a flow chart of a detection algorithm for rare sample defects in QFN chips according to an embodiment of the present invention. Assume that there are 10 types of defects in a certain model of QFN chip: 6 samples of each of the 3 types of rare defects and 200 samples of each of the 7 types of common defects. Figure 1 、 Figure 2 It can be seen that the processing flow is as follows:

[0039] S101: Obtain images of the upper and lower surfaces of the frame, select a ROI area from the image, and crop it.

[0040] Based on the QFN chip defect detection hardware system, images of the upper and lower surfaces of the frame are obtained, and the QFN chip area is selected and cropped using a template matching algorithm.

[0041] S102: Using annotation software to perform data annotation on the ROI area of the cropped image to generate a location category label.

[0042] Use the corresponding annotation software to annotate the data of the cut ROI area, and annotate the defects of each component image, that is, use a rectangular box to mark the position and category of the defects in the image, and generate a position category label.

[0043] S103: Divide the labeled images into a multi-sample defect dataset and a small-sample defect dataset.

[0044] The defects with a large number of defect samples in the annotated image dataset are divided into the multi-sample defect dataset base class, and the defects with only a small number of samples are divided into the few-sample defect dataset novel class.

[0045] S104: Construct a twin Faster-RCNN network model, input the images in the multi-sample defect dataset into the twin Faster-RCNN network model for meta-learning training, and obtain a trained model.

[0046] like Figures 3 to 5 As shown in the figure, the twin Faster-RCNN network model is constructed by improving the existing Faster-RCNN network and adding a twin branch. The parameter structure shared with the backbone network of the existing Faster-RCNN is used to extract the image feature vector. The feature vector output by the twin branch is dot-producted with the feature vector obtained by the ROIPooling layer in the original Faster-RCNN through channel-by-channel. The vector similarity is calculated in the feature dimension and the similarity feature vector replaces the feature vector obtained through the ROIPooling layer and is input into the predictor head layer for classification and precise positioning.

[0047] Figure 6 A schematic diagram of the meta-learning eposide process provided by an embodiment of the present invention is given. Figure 7 This is a flow chart of sampling a dataset in the meta-learning eposide process provided by an embodiment of the present invention. The specific meta-learning episode training process is as follows:

[0048] (1) Randomly select three types of defects from the majority sample defect dataset base class, randomly select one of the three defects and extract six images, of which five images are added to the support set SupportSet, and the other image is used as the query set QuerySet. For the remaining two defects, five images are randomly selected for each defect and added to the support set SupportSet;

[0049] (2) Input the 15 images in the support set SupportSet and the corresponding location category labels into the twin branches to obtain 15 feature vectors;

[0050] (3) Input the query set QuerySet image into the original Faster-RCNN network, calculate the similarity between the feature vector generated by ROIPooling and the 15 feature vectors mentioned in (2), and then input the obtained similarity vector into the predictor head layer for classification and precise positioning;

[0051] (4) Calculate position and category losses through loss function;

[0052] (5) Repeat (1) to (4) 2000 times, calculate the average loss of multiple cycles and update the parameters by gradient back propagation;

[0053] S105: Using the trained model to test the data in the small sample defect dataset, obtain defect detection results for the image and perform evaluation.

[0054] The testing and evaluation process specifically includes:

[0055] (1) Randomly select three types of defects from the novel class of the minority sample defect dataset, randomly select one of the three defects and extract six images, of which five images are added to the support set SupportSet, and the other image is used as the query set QuerySet. For the remaining two defects, five images are randomly selected for each defect and added to the support set SupportSet;

[0056] (2) Input the 15 images and location category labels in the support set SupportSet into the twin branch to obtain 15 feature vectors.

[0057] (3) Input the query set QuerySet image into the original Faster-RCNN network, calculate the similarity between the feature vector generated by ROIPooling and the 15 feature vectors described in (2), and then input the obtained similarity vector into the predictor head layer for classification and precise position correction;

[0058] (4) Evaluation is conducted using missed detection rate and false detection rate as evaluation criteria;

[0059] S106: Repeat S103 to S105 several times according to actual production requirements and detection results to obtain a final model for performing small sample defect detection on QFN chips.

[0060] The above describes the specific embodiments of the present invention in conjunction with the accompanying drawings. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A detection algorithm for small sample defects in QFN chips, characterized by: The steps include: (1) Obtain images of the upper and lower surfaces of the frame, select the ROI area of the image and crop it; (2) using annotation software to annotate the ROI area of the cropped image and generate a location category label; (3) dividing the labeled images into a multi-sample defect dataset and a small-sample defect dataset; (4) constructing a twin Faster-RCNN network model, inputting the images in the multi-sample defect dataset into the twin Faster-RCNN network model for meta-learning training to obtain a trained model; The twin Faster-RCNN network model adds a twin branch to the Faster-RCNN network. The twin branch and the Faster-RCNN backbone network share a parameter structure for extracting image feature vectors. The feature vector output by the twin branch is dot-producted with the feature vector obtained by the ROIPooling layer in the original Faster-RCNN. The vector similarity is calculated in the feature dimension and the similarity feature vector is input into the predictor head layer instead of the feature vector obtained by the ROIPooling layer. (4.1) Randomly select N defects from the multi-sample defect dataset, randomly select one defect from the N defects and extract K+1 images, of which K images are added to the support set SupportSet, and the remaining image is used as the query set QuerySet. For each of the remaining N-1 defects, K images are randomly selected and added to the support set SupportSet; (4.2) Inputting N*K images in the support set SupportSet and their corresponding position category labels into the twin branches to obtain N*K feature vectors; (4.3) Input the images in the query set QuerySet into the Faster-RCNN network, calculate the similarity between the feature vectors generated by ROIPooling and the N*K feature vectors obtained by the twin branches, and then input the obtained similarity vectors into the predictor head layer for classification and precise positioning; (4.4) Calculate the position and category loss through the loss function; (4.5) Set the number of cycles, repeat (4.1) to (4.4) and calculate the average loss of multiple cycles and update the parameters by gradient backpropagation. (5) Using the trained model to test the images in the small sample defect dataset, obtain defect detection results for the images and perform evaluation; (6) Repeat steps (3) to (5) several times according to actual production needs and test results to obtain the final model for production.

2. The detection algorithm for small sample defects in QFN chips according to claim 1, characterized in that: The ROI area in step (1) is the QFN chip area, and the method for selecting the ROI area in the image is a template matching algorithm.

3. The detection algorithm for small sample defects in QFN chips according to claim 1, characterized in that: In the step (2), the cropped ROI area is labeled with data, that is, the defects in the image are labeled with positions and categories using rectangular boxes.

4. The detection algorithm for small sample defects in QFN chips according to claim 1, characterized in that: In step (3), the multi-sample defect data set and the small-sample defect data set are divided according to the data volume. Defects with a large number of defect samples are divided into the multi-sample defect data set, and defects with only a small number of samples are divided into the small-sample defect data set.

5. The detection algorithm for small sample defects in QFN chips according to claim 1, characterized in that: The specific calculation of similarity in the step (4.3) is to perform channel-by-channel dot product between the feature vector generated by the ROIPooling and the N*K feature vectors obtained by the twin branches, and calculate the vector similarity in the feature dimension.

6. The detection algorithm for small sample defects in QFN chips according to claim 1, characterized in that: The evaluation in step (5) uses missed detection rate and false detection rate as evaluation criteria.

Images