Joint detection method for chip bond wire defects based on rotation target detection and semantic segmentation

Abstract

The present invention discloses a joint detection method for chip bonding wire defects based on rotating target detection and semantic segmentation. The method assigns labels to bonding wire defects in X-ray images from a top-down perspective of the chip interior, divides the data set, and performs offline multi-scale enhancement processing to obtain a rotating target detection data set; trains on the rotating target detection data set to obtain a rotating target detection model; uses the prediction box predicted by ST-O-RCNN on the data set image to cut out pictures of the focused bonding wire, construct a semantic segmentation data set, and train a UNet semantic segmentation network to obtain a semantic segmentation model; during prediction, the X-ray top view of the chip to be predicted is subjected to offline data enhancement processing, and the bonding wire position is located by rotating target detection, the bonding wire morphology is obtained by semantic segmentation, and the bonding wire defect is identified by a program to obtain the bonding wire defect detection result. The present invention can achieve accurate positioning and identification of bonding wire defects.

Description

Technical Field

[0001] The present invention belongs to the technical field of semiconductor chip defect detection, and in particular relates to a chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation. Background Art

[0002] During semiconductor chip manufacturing, due to the complex and diverse manufacturing processes, chips may develop internal defects that are invisible from the outside due to various factors, such as packaging techniques, environment, and materials. These defects can affect chip performance to varying degrees. Bonding wires, due to their dense and slender nature, are particularly difficult to inspect, and the resulting defects are also difficult to detect.

[0003] Defect detection of bonding wires in chips is an important quality inspection process in chip production. However, in previous inspection processes, it was difficult to accurately determine the specific defect type of the bonding wires by relying solely on target detection. Summary of the Invention

[0004] The object of the present invention is to provide a chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation.

[0005] The technical solution to achieve the purpose of the present invention is as follows: In a first aspect, the present invention provides a joint detection method for chip bonding wire defects based on rotation target detection and semantic segmentation, comprising the following steps:

[0006] S1. Obtain an X-ray top view of the chip;

[0007] S2. Obtain a rotating object detection dataset: assign labels and manually annotate the chip X-ray top view from step S1, and obtain its DOTA format txt label file. Divide all chip X-ray top views into three original datasets: training set, validation set, and test set.

[0008] S3. Dataset data augmentation: Crop the chip X-ray top view in step S2 into multiple smaller sub-chip X-ray top views at 0.7, 1.0, 1.3, and 1.5 times the predetermined training size, respectively. Adjust the corresponding rotation box annotations of the cropped images to form a multi-scale dataset. Merge the three uncut original datasets of the chip X-ray top view into the corresponding multi-scale datasets to increase the overall information of the images, forming the final training set, validation set, and test set.

[0009] S4. Train the rotation object detection model: Use the swin-tiny backbone network to replace the Resnet-50 backbone network in oriented-rcnn to build the ST-O-RCNN network, increase the number of layers of the feature pyramid in the network structure, and change the anchor box size. Use weak data augmentation to train the network on the final training set in step S3. Under the guidance of the test set, obtain the trained object detection model;

[0010] S5. Obtain a semantic segmentation dataset: Use the object detection model in step S4 to predict the rotation box of the bonding wires in the dataset, cut out the position of the predicted rotation box on the image, and obtain angled tilted bonding wire images. Use perspective transformation to convert the angled tilted bonding wire images into horizontal bonding wire images so that they can be used to train the network. After labeling the bonding wires in the horizontal bonding wire images, obtain a semantic segmentation dataset.

[0011] S6. Training semantic segmentation model: Build a UNet semantic segmentation network and train it using the dataset from step S5 to obtain a trained semantic segmentation model.

[0012] S7. Actual prediction: The X-ray top view of the chip to be predicted is processed according to steps S2 and S3 to obtain a multi-scale data set. This data set is input into the rotation target detection network of step S4 to obtain the prediction box of the bonding wire related defects. The prediction box is cut at the corresponding position on the image and converted into a horizontal image through perspective transformation. The semantic segmentation model of step S6 is used to predict the horizontal image to obtain a semantic segmentation result image. Finally, the program performs maximum connected area related judgment on the semantic segmentation result image to predict the specific class of the bonding wire defect.

[0013] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0014] In a third aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0015] Compared with the existing technology, the significant advantages of the present invention are: the present invention proposes a chip bonding wire defect joint detection method based on rotating target detection and semantic segmentation, which includes data processing, model construction and joint identification scheme to achieve accurate positioning and identification of bonding wire defects. Compared with traditional defect detection technology, the joint detection method based on rotating target detection and semantic segmentation has higher accuracy and more stable detection results, providing a new idea for defect detection of slender objects in chips.

[0016] The present invention will be described in further detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the method of the present invention.

[0018] Figure 2 This is a diagram of bond wire label assignments.

[0019] Figure 3 This is the cutting plan of the original picture.

[0020] Figure 4 It is a cutting image edge feature processing diagram.

[0021] Figure 5 This is the oriented-RCNN structure diagram.

[0022] Figure 6 It is the anchor box ratio change map.

[0023] Figure 7 It is the conversion diagram from tilted image to horizontal image. DETAILED DESCRIPTION

[0024] Based on the characteristics of dense distribution and large scale span of bonding wires in chips, the present invention proposes a joint detection method for chip bonding wire defects based on rotational target detection and semantic segmentation, which greatly enhances the learnability of the dataset and the model's learning ability for such targets. The method is based on an oriented-RCNN target detection network (ST-O-RCNN network) with a swin-tiny backbone network and a UNet semantic segmentation network. First, a large number of X-ray images of the chip's internal top-down perspective are obtained, and then the images are annotated and divided into datasets. The images in the dataset are processed by offline data enhancement to obtain a final dataset. The processed dataset is placed in the ST-O-RCNN network for training to obtain a rotational target detection network model; the prediction box predicted by the ST-O-RCNN on the dataset images is used to cut out pictures of the focused bonding wires, construct a semantic segmentation dataset, and use it to train the UNet semantic segmentation network to obtain a semantic segmentation model. During prediction, the X-ray top view of the chip to be predicted is subjected to the same offline data enhancement processing. Then, the bonding wire position is located through rotating target detection, the bonding wire morphology is obtained through semantic segmentation, and the bonding wire defects are identified by the program to obtain the bonding wire defect detection results.

[0025] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0026] like Figure 1 As shown, a chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation includes the following steps:

[0027] Step 1: Obtain a large number of chip X-ray top views;

[0028] Step 2: Obtain a rotating target detection dataset: Based on the principle of "detailed differentiation of relevant features and ignoring irrelevant features as appropriate", label the chip X-ray top view from step 1 and manually annotate it. Obtain its DOTA format txt label file. Divide all chip X-ray top views into three original datasets: training set, validation set, and test set in a ratio of 3:1:1.

[0029] Step 3: Dataset Augmentation: The chip X-ray top view from step 2 is cropped into multiple smaller sub-chip X-ray top views at 0.7, 1.0, 1.3, and 1.5 times the predetermined training size, respectively. The corresponding rotation box annotations of the cropped images are adjusted to form a multi-scale dataset. Subsequently, to alleviate the image fragmentation problem caused by the cropped images, the three original uncropped chip X-ray top view datasets are merged into the corresponding multi-scale datasets to increase the overall information of the images, forming the final training set, validation set, and test set.

[0030] Step 4: Train the rotated object detection model: Use the more expressive swin-tiny backbone network to replace the Resnet-50 backbone network in oriented-rcnn to build the ST-O-RCNN network. Based on the dense and slender characteristics of bond wires, increase the number of feature pyramid layers in the network structure. At the same time, change the anchor box size to favor slender objects to optimize the network's ability to handle bond wires. Use weak data augmentation to train the network on the final training set in step 3. Under the guidance of the test set, obtain a trained object detection model.

[0031] Step 5: Obtain a semantic segmentation dataset: Use the object detection model from step 4 to predict the rotation box of the bonding wires in the dataset. Cut out the position of the predicted rotation box on the image to obtain angled bond wire tilt images. Use perspective transformation to convert the angled bond wire tilt images into horizontal bond wire images so that they can be used to train the network. After annotating the bonding wires in the horizontal bond wire images, obtain a semantic segmentation dataset.

[0032] Step 6: Train the semantic segmentation model: Build a UNet semantic segmentation network and train it using the dataset from step 5 to obtain a trained semantic segmentation model.

[0033] Step 7. Actual prediction: Process the X-ray top view of the chip to be predicted according to steps 2 and 3 to obtain a multi-scale dataset. Input this dataset into the rotation target detection network of step 4 to obtain the prediction box of the bonding wire related defects. Cut the prediction box at the corresponding position on the image and convert it into a horizontal image through perspective transformation. Use the semantic segmentation model of step 6 to predict the horizontal image to obtain the semantic segmentation result image. Finally, use the program to perform maximum connected area related judgment on the semantic segmentation result image to predict the specific type of bonding wire defect.

[0034] In this embodiment, step 2 includes the following process:

[0035] (1) Based on the principle of “detailed differentiation of relevant features and ignoring irrelevant features as appropriate”, the labels of the chip X-ray top view in step 1 are assigned: in order to enhance the information of the bonding wire in the image, the defects related to the bonding wire should be subdivided into multiple labels, such as bonding wire, bonding wire breakage, bonding wire bridging, bonding wire deviation, etc., and the features unrelated to the bonding wire should be ignored if they are too different from the bonding wire morphology. If they are similar to the bonding wire morphology, they should also be assigned labels to enhance the network's ability to distinguish them;

[0036] If "bonding wire broken" is used as a defect label, then "bonding wire normal", "bonding wire deviation", and "bonding wire crossover" should also be assigned as labels. "Bonding wire normal" is not a defect we need to identify. Its role is to serve as an auxiliary label to help the network distinguish the semantic information in the image. The annotation example is as follows Figure 2 As shown;

[0037] (2) Obtain the txt label file of the manually annotated rotation box: The format of the txt label file follows the format of the label file in the DOTA dataset. The label format is as follows:

[0038] The label information on each sub-defect image will be output to a txt file. The format of each line in the txt file is:

[0039] The x coordinate of the upper left vertex of the rotation box;

[0040] The y coordinate of the upper left vertex of the rotation box;

[0041] The x coordinate of the lower left vertex of the rotation box;

[0042] The y coordinate of the lower left vertex of the rotation box;

[0043] The x coordinate of the upper right vertex of the rotation box;

[0044] The y coordinate of the upper right vertex of the rotation box;

[0045] The x coordinate of the lower right vertex of the rotation box;

[0046] The y coordinate of the lower right vertex of the rotation box;

[0047] Whether this rotating box is a difficult-to-predict box;

[0048] This spin box predicts the category;

[0049] From the above, we can get Figure 2 The label of the fault wire mark box is: 1045 381 866 620 933 681 1151 4430 fault wire;

[0050] In this embodiment, step three includes the following process:

[0051] (1) Cutting the chip X-ray top view: For a given image, take the left vertex as the starting point, cut out the image with the specified width and height, then move the starting point to the right by a specified number of pixels, and cut out the image with the specified width and height again, and repeat this process until the image is cut to the end in the horizontal direction. This operation is called horizontal cutting. Then move the starting point down by a specified number of pixels, and cut horizontally again. After cutting to the end of the image, move the starting point down again, and repeat this process. The specific steps are as follows: Figure 3 As shown, each box represents a cropped sub-image area.

[0052] (2) Adjust the corresponding rotation box annotation of the cropped image: Only the rotation box at the edge of the cut image needs to be adjusted. The rotation box should be retained according to its proportion in the cut image. If less than 30% of the rotation box is left in the cut image, the rotation box should be discarded. Figure 4 The objects selected in the middle box are removed because the remaining parts are too small.

[0053] (3) X-ray the chip downward Figure 3 The uncut original datasets are merged into the corresponding multi-scale datasets: Due to the operation of process 2, the images in the multi-scale dataset are all incomplete, and edge targets are discarded to a certain extent. The scale of the bonding wire spans a large range, and the bonding wire in an image may span half of the image. This leads to a serious discard of the bonding wire in the cut image, which will mislead the model. Therefore, three original datasets are merged into this dataset to make up for the defects of the dataset itself.

[0054] In this embodiment, step 4 includes the following process:

[0055] (1) Use the more expressive swin-tiny backbone network to replace the Resnet-50 backbone network in oriented-rcnn: Figure 5 The structure of the oriented-rcnn network is used. The Resnet-50 backbone network has a slightly weak feature extraction capability for chip detection tasks, so it is replaced by the swin-tiny backbone network to form the ST-O-RCNN network.

[0056] (2) According to the characteristics of dense and slender bonding wires, the number of feature pyramid layers in the network structure is increased, and the size of the anchor frame is changed to favor slender objects: the bonding wire itself is an extremely slender object, and its annotated rotation frame also has an extreme aspect ratio, which is extremely inconsistent with the rotation frame of general objects. The oriented-rcnn network is based on the anchor frame network, and it needs to manually set the anchor frame aspect ratio in advance to generate the rotation frame. Therefore, the general anchor frame aspect ratio is not suitable for the detection of slender objects such as bonding wires. Therefore, the anchor frame aspect ratio is changed from the general 0.5, 1, 2.0 to 0.3, 0.5, 1, 2.0, 3.0, which will produce an anchor frame with a larger aspect ratio; In addition, for swin-tiny The feature pyramid structure in the oriented-RCNN network is changed from the traditional downsampling multiples of 4, 8, 16, 32, and 64 to 4, 8, 16, 32, 64, and 128 times. This allows the network to have a larger receptive field, which can produce better detection effects for particularly long objects such as bonding wires and objects with particularly large spans in images. Figure 6 Comparison of anchor box sizes generated for different anchor box settings.

[0057] (3) Use weak data enhancement to train the network on the final training set: General target detection data enhancement strategies include image horizontal flip enhancement, random rotation enhancement, Mixup enhancement, mosaic enhancement, etc. However, due to the characteristics of the chip dataset environment, small amount of defect data, and single usage scenario, it will be difficult to learn target features using a strong data enhancement model. Therefore, we only use weak data enhancement operations such as horizontal flip and random rotation enhancement. Bond wire defect detection also has the characteristic that the number of normal samples is much larger than the number of defective samples. Therefore, the model's fitting of the data will be greatly biased towards normal samples, while ignoring the errors of defective samples. Therefore, we adopt the category balanced sampler to alleviate this problem:

[0058] The class-balanced sampler is a resampling method that solves the problem of data imbalance. Its principle is to dynamically adjust the sampling probability based on the number of samples in each class, so that each class has the same probability of being sampled. Its general formula is:

[0059]

[0060] Among them, C is the number of categories in the dataset, n i is the total number of samples of category i, p jis the probability of sampling a sample from class j, and q is a hyperparameter controlling the weight of each class. When q = 0, all classes have the same sampling probability; when q = 1, the sampling probability of each class is proportional to the number of samples in it; when q < 0, the sampling probability of the minority class increases and the sampling probability of the majority class decreases. The class-balanced sampler can effectively reduce the influence of the majority class on model training and improve the recognition performance of the minority class. It also avoids overfitting and information loss because it does not delete or duplicate any samples.

[0061] In this embodiment, step seven includes the following process:

[0062] (1) Use perspective transformation to convert the angled bond wire image into a horizontal bond wire image: Use the getPerspectiveTransform method of opencv to convert the tilted image into a horizontal image. The specific transformation is as follows: Figure 7 .

[0063] (2) The specific type of bonding wire defect is predicted by performing the maximum connected area related judgment on the semantic segmentation result graph through the program: For overly dense bonding wires, it is difficult to distinguish the boundaries between the bonding wires through target detection alone, let alone distinguishing specific defects. Therefore, it is necessary to perform semantic segmentation in the rotating detection frame to obtain the semantic segmentation result graph. Then, a programmatic solution is used to judge the connected area where the bonding wires are located and the relationship between the connected areas where each bonding wire is located on the result graph to obtain the specific judgment of normal bonding wires, bonding wire deviation, and bonding wire cross-connection. The judgment method has different ways according to different defect designs.

Claims

1. A joint detection method for chip bonding wire defects based on rotation target detection and semantic segmentation, characterized in that: The following steps are involved: S1. Obtain an X-ray top view of the chip; S2. Obtain a rotating object detection dataset: assign labels and manually annotate the chip X-ray top view from step S1, and obtain its DOTA format txt label file. Divide all chip X-ray top views into three original datasets: training set, validation set, and test set. S3. Dataset data augmentation: Crop the chip X-ray top view in step S2 into multiple smaller sub-chip X-ray top views at 0.7, 1.0, 1.3, and 1.5 times the predetermined training size, respectively. Adjust the corresponding rotation box annotations of the cropped images to form a multi-scale dataset. Merge the three uncut original datasets of the chip X-ray top view into the corresponding multi-scale datasets to increase the overall information of the images, forming the final training set, validation set, and test set. S4. Train the rotation object detection model: Use the swin-tiny backbone network to replace the Resnet-50 backbone network in oriented-rcnn to build the ST-O-RCNN network, increase the number of layers of the feature pyramid in the network structure, and change the anchor box size. Use weak data augmentation to train the network on the final training set in step S3. Under the guidance of the test set, obtain the trained object detection model; S5. Obtain a semantic segmentation dataset: Use the object detection model in step S4 to predict the rotation box of the bonding wires in the dataset, cut out the position of the predicted rotation box on the image, and obtain angled tilted bonding wire images. Use perspective transformation to convert the angled tilted bonding wire images into horizontal bonding wire images so that they can be used to train the network. After labeling the bonding wires in the horizontal bonding wire images, obtain a semantic segmentation dataset. S6. Training semantic segmentation model: Build a UNet semantic segmentation network and train it using the dataset from step S5 to obtain a trained semantic segmentation model. S7. Actual prediction: The X-ray top view of the chip to be predicted is processed according to steps S2 and S3 to obtain a multi-scale data set. This data set is input into the rotation target detection network of step S4 to obtain the prediction box of the bonding wire related defects. The prediction box is cut at the corresponding position on the image and converted into a horizontal image through perspective transformation. The semantic segmentation model of step S6 is used to predict the horizontal image to obtain a semantic segmentation result image. Finally, the program performs maximum connected area related judgment on the semantic segmentation result image to predict the specific class of the bonding wire defect.

2. The chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation according to claim 1 is characterized in that: In step S2, labels are assigned to the chip X-ray top view in step S1. Bonding wire-related defects need to be subdivided, with bonding wire, bonding wire breakage, bonding wire bridging, and bonding wire deviation as labels.

3. The chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation according to claim 1 is characterized in that: In step S2, all chip X-ray top views are divided into three original data sets at a ratio of 3:1:1: a training set, a validation set, and a test set.

4. The chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation according to claim 1 is characterized in that: Step S3 specifically includes the following process: S31, cutting the chip X-ray top view: for a given image, using the left vertex as the starting point, cut out an image of specified width and height, then move the starting point right by a specified number of u pixels, and cut out an image of specified width and height again, repeating this process until the image is cut horizontally to the end. This operation is called horizontal cutting. Then, move the starting point downward by a specified number of v pixels, and cut horizontally again. After cutting to the end of the image, move the starting point downward again, and repeat this process. S32, adjusting the corresponding rotation box annotations of the cropped image: only adjusting the rotation box at the edge of the cropped image, and deciding whether to retain it based on the proportion of the rotation box in the cropped image. If less than 30% of the rotation box is left in the cropped image, the rotation box is discarded; S33, merge the three uncut original data sets of the chip X-ray top view into the corresponding multi-scale data set.

5. The chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation according to claim 1 is characterized in that: The step S4 includes the following process: S41, use the swin-tiny backbone network to replace the Resnet-50 backbone network in oriented-rcnn to form the ST-O-RCNN network; S42 increases the number of feature pyramid layers in the network structure and changes the aspect ratio of the anchor box to generate an anchor box with a larger aspect ratio. S43, use up and down horizontal flipping and random rotation to enhance the training network on the final training set.

6. The chip bonding wire defect joint detection method based on rotation target detection and semantic segmentation according to claim 1 is characterized in that: In step S7, the semantic segmentation result graph is subjected to the maximum connected area related judgment to predict the specific type of bonding wire defects, specifically: semantic segmentation is performed in the rotation detection frame to obtain the semantic segmentation result graph, and then the relationship between the connected area where the bonding wire is located and the connected area where each bonding wire is located is judged on the result graph to obtain the specific judgment of normal bonding wire, bonding wire deviation, and bonding wire cross-connection.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images