Rotating target detection method and system for chip internal component fault

Abstract

The invention discloses a rotating target detection method and system for faults of internal parts of a chip, and the method specifically comprises the steps: 1, collecting an internal image of the chip through an X-ray imaging technology, carrying out the precise marking, and obtaining labels of a defect region and a normal region, and a rotating frame; 2, dividing the marked image into a training set, a verification set and a test set according to a proportion, and performing multi-scale cutting; 3, setting rotating frame labels for the cut sub-images, and combining the rotating frame labels to form a final data set; 4, an ROI-Transform network based on Swin-Tiny is constructed and optimized, and data enhancement technologies such as random rotation, disturbance rotation, Mixup and random erasure are adopted for training; and 5, inputting a to-be-detected image into the trained model to obtain a fault detection result. By introducing various data enhancement technologies and targeted label distribution and data processing, the learnability of the data set and the detection precision of the model are remarkably improved, the quality monitoring efficiency in the chip manufacturing process is improved, and the method is suitable for internal part fault detection in the field of semiconductor chip manufacturing.

Description

Technical Field

[0001] The present invention relates to the technical field of semiconductor chip manufacturing quality monitoring, and in particular to a rotating target detection method and system for chip internal component failure. Background Art

[0002] In the process of manufacturing semiconductor chips, due to the complexity and diversity of chip manufacturing processes, the chips may produce some defects that are invisible to the outside due to various factors such as packaging technology, environment, and materials. These defects will affect the performance of the chip to varying degrees. Therefore, chip defect detection is an indispensable part.

[0003] Traditional chip defect detection methods usually rely on manual screening of chip X-ray photos. This method is not only inefficient, but also prone to missed detection and false detection. In contrast, the quality inspection method based on rotating target detection has obvious advantages in efficiency and accuracy, providing an innovative solution for chip defect detection.

[0004] With the advancement of computer vision and deep learning technology, rotating object detection technology has made significant progress and has the ability to be applied to chip defect detection. Compared with horizontal box-based object detection tasks, rotating box object detection has more advantages in handling densely overlapping objects. Rotating object detection can more accurately describe and locate the actual shape and direction of the object, so when detecting complex structures such as a large number of bonding wires, rotating object detection technology can provide better results.

[0005] However, the current rotating target detection technology is not very perfect in the field of chip defect detection. It has the defects of high computational complexity, poor model generalization ability, and insufficient flexibility in defect detection. Summary of the invention

[0006] The purpose of the present invention is to provide a rotating target detection method and system for chip internal component failures, which can more accurately capture the characteristics of the chip's internal image and improve the detection accuracy of the chip's internal top-view image and the generalization ability of the model.

[0007] The technical solution to achieve the purpose of the present invention is:

[0008] A rotating target detection method for internal component failure of a chip comprises the following steps:

[0009] Step 1: Collecting a top-view image of the chip interior taken by quantitative X-ray imaging technology;

[0010] Step 2: Label the top view images of the chip and divide all images into training set, validation set and test set in a ratio of 3:1:1;

[0011] Step 3: Preprocess the training set data;

[0012] Step 4, constructing a rotation target detection network with Region of Interest Transformer as the core and Swin-Tiny network as the backbone, and iteratively training the rotation target detection network through the training set; wherein the Swin-Tiny network includes a partitioning module, a linear embedding layer, and multiple layered Swin Transformer Blocks, the partitioning module divides the input image into multiple image blocks of fixed sizes, and the Swin Transformer Block uses a local window attention mechanism, a sliding window mechanism, and a multi-layer perceptron to process the image blocks;

[0013] Step 5: Perform back propagation through the results of the validation set and the test set. If the rotation target detection network does not meet the requirements, adjust the parameters of the rotation target detection network and repeat step 4 for multiple iterative training. Otherwise, execute step 6.

[0014] Step 6: Process the X-ray top view of the chip to be predicted according to steps 1 to 3, input the processed data into the rotating target detection network to obtain the final defect prediction result.

[0015] Furthermore, the step 2 comprises:

[0016] Step 2-1, assigning primary labels and auxiliary labels to defective areas and normal areas;

[0017] Step 2-2, make the format of the label file follow the format of the DOTA dataset, and output the label information of each sub-graph as a .txt file. The data format is the x-coordinate of the upper left vertex of the rotating box, the y-coordinate of the upper left vertex of the rotating box, the x-coordinate of the lower left vertex of the rotating box, the y-coordinate of the lower left vertex of the rotating box, the x-coordinate of the upper right vertex of the rotating box, the y-coordinate of the upper right vertex of the rotating box, the x-coordinate of the lower right vertex of the rotating box, the y-coordinate of the lower right vertex of the rotating box, whether the rotating box belongs to a difficult-to-predict box, and the prediction category of the rotating box.

[0018] Furthermore, the preprocessing includes random rotation, perturbed random rotation, Mixup and random erasure data enhancement techniques.

[0019] Furthermore, the random rotation includes: flipping the image along a vertical axis, flipping the image along a horizontal axis, and flipping the image along a diagonal line.

[0020] Furthermore, the perturbed random rotation is:

[0021] x′=C x +(xC x )cos(θ)-(yC y)sin(θ)

[0022] y′=C y +(xC x )sin(θ)+(yC y )cos(θ)

[0023] Among them, (x, y) is the original point, (x', y') is the new coordinate after the perturbation random rotation, (C x ,C y ) is the image center and θ is the rotation angle.

[0024] Furthermore, the segmentation module specifically includes: starting from the left vertex of the image, cutting out a 1024*1024 image block, and gradually moving the starting point rightward and downward until the cutting is completed, and each image block is projected to SwinTransformerBlock through a linear embedding layer.

[0025] Furthermore, the sliding window mechanism includes: continuously moving the starting point to the right by 512 pixels through the sliding window, and cutting out a 1024*1024 image block again until the cutting is completed in the horizontal direction, and then moving the starting point downward by 512 pixels again, and repeating the cutting process in the horizontal direction until the entire area of ​​the picture is cut.

[0026] Furthermore, the attention weight of the local window attention mechanism is Among them, Q and K are two matrices, which are transformed by the input data through the fully connected layer of Region of Interest Transformer, softmax() is the normalization function, and d k is the square of the number of key-value pairs k.

[0027] Furthermore, the loss function of the rotating target detection network includes:

[0028] CrossEntropyLoss:

[0029]

[0030] SmoothL1Loss:

[0031]

[0032] Among them, x is the output of the network, class is the target category, and x[j] represents the jth component of x.

[0033] A rotating target detection system for chip internal component failure, comprising:

[0034] An image acquisition unit, used to collect top-view images of the inside of the chip taken using quantitative X-ray imaging technology;

[0035] The data set construction unit is used to annotate the top-view images of the chip interior and divide all images into training set, validation set and test set in a ratio of 3:1:1;

[0036] A preprocessing unit, used for preprocessing the training set data;

[0037] A rotating target detection network construction unit is used to construct a rotating target detection network with Region of Interest Transformer as the core and Swin-Tiny network as the backbone, and the rotating target detection network is iteratively trained through a training set; wherein the Swin-Tiny network includes a partitioning module, a linear embedding layer, and a plurality of layered Swin Transformer Blocks, the partitioning module divides the input image into a plurality of image blocks of fixed sizes, and the Swin Transformer Block uses a local window attention mechanism, a sliding window mechanism, and a multi-layer perceptron to process the image blocks;

[0038] The verification and testing unit performs back propagation through the results of the verification set and the test set. If the rotation target detection network does not meet the requirements, the parameters of the rotation target detection network are adjusted;

[0039] The detection unit inputs the pre-processed X-ray top view data of the chip to be predicted into the rotating target detection network to obtain the final defect prediction result.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. Using Swin-Tiny as the backbone network and combining it with the improved ROI Transformer network can more accurately capture the features of the chip's internal image and improve the detection accuracy of the chip's internal top-view image;

[0042] 2. Through data enhancement techniques such as random rotation, perturbation rotation, Mixup and random erasure, the learnability of the data set is enhanced, so that the model can better learn the characteristics of chip defects and improve the generalization ability of the model;

[0043] 3. Adjust the edge rotation frame and set the parameter iof-thr to require that at least 70% of the rotation frame after cutting be saved in the cut image. Otherwise, the rotation frame will be discarded, which improves the accuracy of rotation frame annotation and thus improves the accuracy of detection.

[0044] 4. The local window attention mechanism calculates attention within a small block, and the sliding window mechanism enables information between different windows to interact, thereby capturing a wider range of features.

[0045] The following content describes the specific implementation process of the present invention in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a step diagram of the key technology of the present invention.

[0047] Figure 2 This is an example of annotated chip dataset.

[0048] Figure 3 yes Figure 2 Schematic diagram of the chip example after cutting and trimming the outline.

[0049] Figure 4 yes Figure 2 Schematic diagram of the chip example image showing the marked box being cut.

[0050] Figure 5 This is the network structure diagram of Region of Interest (ROI) Transformer.

[0051] Figure 6 (a) Diagram showing the network structure steps of Swin Transformer.

[0052] Figure 6 (b) Represents the two-stage Swin Transformer Block structure.

[0053] Figure 7 This is a graph showing the mPA comparison of the results before and after the experiment.

[0054] Figure 8 Schematic diagram showing the test results of the training model. DETAILED DESCRIPTION

[0055] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The embodiments are only used to explain the present invention and do not limit the protection scope of the present invention.

[0056] Combination Figure 1The present invention discloses a rotating target fault detection technology for detecting internal components of a chip, which adopts a Region of Interest (ROI) Transformer network based on Swin-Tiny as the backbone network. First, based on an X-ray picture of the chip's internal top-down perspective, labels are assigned and annotated to the defective parts and normal parts in the picture to assist the model in learning defect labels. Subsequently, the data set is divided into a training set, a validation set, and a test set in a ratio of 3:1:1, and the processed data set is placed in the Region of Interest (ROI) Transformer network with the adjusted Swin-Tiny as the backbone network for training to obtain the best network model. Finally, the X-ray top view of the chip to be predicted is processed in the same way and placed in the best network model for prediction to obtain the final defect detection result; specifically including:

[0057] Step 1: Collect a large number of top-down images of the inside of the chip taken using X-ray imaging technology.

[0058] Step 2: Accurately label the X-ray top view image, including assigning main labels and auxiliary labels to defect areas and normal areas to assist the model in learning defect features. At the same time, manually draw the rotation box corresponding to the defect and generate the corresponding txt label file. After that, all images are divided into training set, validation set and test set in a ratio of 3:1:1. Specifically:

[0059] 1. Label Allocation

[0060] When detecting defects, assign key labels for defects and auxiliary labels for normal areas. For example, assign "bonding wire broken" as a defect label and "bonding wire normal" as an auxiliary label. Auxiliary labels help the network distinguish semantic information in the image.

[0061] (2) Generate .txt tag file

[0062] The format of the label file follows the format of the DOTA dataset. The label information of each sub-graph is output as a .txt file in the following format:

[0063] Table 2

[0064]

[0065] Step 3. Based on the characteristics of small chip image data sets, dense targets in a single image, and large differences in the scales of different targets, the present invention proposes a targeted label allocation, data processing, and model building scheme. By introducing a variety of data enhancement techniques, applying random rotation (RRandomFlip), perturbed random rotation (PolyRandomRotate), Mixup, and random erasure data enhancement techniques on the training set, the learnability of the data set and the model's ability to learn the target are significantly improved.

[0066] Specifically include:

[0067] According to the characteristics of the data set, specific data enhancement processing is added to it.

[0068] Given the characteristics of the chip dataset, which has a simple environment, few defective data, and a single usage scenario, it is difficult to learn the target features using a powerful data augmentation model. Therefore, this technology uses the following methods for data augmentation:

[0069] 1) Random rotation (RRandomFlip)

[0070] This technique implements three special flip types through random flip (RRandomFlip):

[0071] a. Horizontal flip

[0072] Horizontal flipping flips the image along the vertical axis. Assuming the pixel position of the original image is I(x,y), the width of the image is W, the pixel position after flipping becomes I(Wx-1,y).

[0073] I(x,y)→I(Wx-1,y)

[0074] b. Vertical flip

[0075] Vertical flipping flips the image along the horizontal axis. Assuming the pixel position of the original image is I(x,y), and the height of the image is H, the pixel position after flipping becomes I(x,Hy-1).

[0076] I(x,y)→I(x,Hy-1)

[0077] c. Diagonal flip

[0078] Diagonal flipping swaps the x and y coordinates and inverts their values. Assuming the original image pixel position is I(x,y), the image width is W, and the height is H, the flipped pixel position becomes I(Hy-1,Wx-1).

[0079] I(x,y)→I(Hy-1,Wx-1)

[0080] 2) Perturbed random rotation (PolyRandomRotate)

[0081] Since the chip dataset contains images with polygonal labels, this technique specifically uses a random rotation operation of the polygons, which involves using a rotation matrix to transform the coordinates of each pixel in the image.

[0082] a. Rotation operation

[0083] Assuming the rotation angle is θ, the rotation operation can be expressed by the following rotation matrix:

[0084]

[0085] For each point (x, y) on the image, the new coordinates (x', y') after the rotation transformation can be calculated by the following formula:

[0086]

[0087] The above formula assumes that the center of rotation is at the origin (0,0). In this technology, the center of the image is used as the center of rotation. Assume that the center of the image is (C x ,C y ), we need to perform a translation operation to move the rotation center to the origin, rotate it, and then move it back to its original position.

[0088] Assume that the original point is (x, y) and the center of the image is (C x ,C y ), then the new coordinates (x', y') after rotation are calculated as follows:

[0089] x′=C x +(xC x )cos(θ)-(yC y )sin(θ)

[0090] y′=C y +(xC x )sin(θ)+(yC y )cos(θ)

[0091] Step 4: Construct a rotation target detection network with Region of Interest (ROI) Transformer as the core and Swin-Tiny network as the backbone. Divide the input image into small patches of fixed size (Patch Embedding), and then pass through multiple layered Swin Transformer Blocks. These blocks include local window attention mechanism, sliding window mechanism and multi-layer perceptron (MLP). The local window attention mechanism calculates attention within the small patch, and the sliding window mechanism enables information between different windows to interact, thereby capturing a wider range of features. The network is back-propagated through the results on the validation set and test set, so as to adjust the parameters and iterate the training multiple times.

[0092] In this embodiment, the step 4 includes the following features:

[0093] 1) Use the Swin-Tiny backbone network to replace the Region of Interest (ROI) Transformer backbone network.

[0094] Figure 5 The structure of Region of Interest (ROI) Transformer. The Region of Interest (ROI) Transformer network generally uses the ResNet backbone network, but because it is slightly weak in local feature extraction for chip detection tasks, it is replaced by the Swin-Tiny backbone network to form the Swin-TinyROI-Transformer network. Figure 6 (a) represents the algorithm structure steps of SwinTiny. Figure 6 (b) Represents the two-stage SwinTransformerBlock structure.

[0095] The specific structure of SwinTiny includes:

[0096] 1. Image block division

[0097] Starting from the left vertex of the image, cut out a 1024*1024 image block, and gradually move the starting point right and down until the cutting is completed. Each image block is projected into a high-dimensional vector space through a linear embedding layer to form an initial feature embedding. The specific cutting is as follows Figure 3 As shown, each box represents a cropped sub-image area.

[0098] 2. Linear Embedding

[0099] The projected image block is embedded into the Swin Transformer Block in the high-dimensional space to generate embedded features.

[0100] 3.Swin-Transformer Block

[0101] The Swin Transformer Block is a backbone module for embedding features. It uses a local window attention mechanism, a sliding window mechanism, and a multi-layer perceptron to process image blocks. The features are processed by a local window attention mechanism (attention is calculated on the features within the local window, each head processes a part of the features independently, and the results are spliced).

[0102] For each attention calculation within window w, the attention weight

[0103] The Swin Transformer Block uses a sliding window mechanism to continuously move the starting point to the right by 512 pixels, and cut out a 1024*1024 image block again until the cropping is completed in the horizontal direction. Then, the starting point is moved down by 512 pixels and the cropping process in the horizontal direction is repeated until the entire area of ​​the image is cropped.

[0104] After sliding cutting, the cropped image can be marked and adjusted with a rotating frame, including:

[0105] 1. Adjust the rotation box annotation

[0106] Adjust the edge rotation frame to decide whether to keep its annotation. Set the parameter iof-thr to require that at least 70% of the rotation frame after cutting is saved in the cut image, otherwise the rotation frame will be discarded. Figure 4 The object b selected in the middle box is removed because the remaining part is too small; Figure 4 The object a selected in the middle box meets the requirements and is retained.

[0107] 2. Finally adjust the rotation box annotation

[0108] Adjust the edge rotation frame and set the parameter iof-thr to require that at least 70% of the rotation frame after cutting is saved in the cut image, otherwise the rotation frame is discarded.

[0109] When training the Swin-Tiny ROI-Transformer network, CrossEntropyLoss is used as the loss function for classification loss. This loss function combines the functions of LogSoftmax and NLLoss. For a given input x and target target, the calculation formula of CrossEntropyLoss is:

[0110]

[0111] In the bounding box regression loss, SmoothL1Loss is selected as the loss function. SmoothL1Loss is more robust when dealing with outliers and is smoother when the gradient is close to zero. The mathematical expression of SmoothL1Loss is:

[0112]

[0113] Step 5: Process the X-ray top view of the chip to be predicted according to steps 1 to 4 to obtain a multi-scale data set, input this data set into the rotating target detection network of step 5, and merge the output results of the multi-scale data set to filter out the final defect prediction result.

[0114] The present invention is aimed at components with complex internal structures of chips, and can accurately identify and locate faults at different rotation angles, thereby improving the production quality and efficiency of semiconductor chips. The present invention significantly improves the learnability of the data set and the detection accuracy of the model by introducing a variety of data enhancement technologies and targeted label allocation and data processing, and improves the quality monitoring efficiency in the chip manufacturing process. It is suitable for internal component fault detection in the field of semiconductor chip manufacturing.

[0115] The description of the above preferred embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without using creative capabilities. Therefore, the present invention is not limited to the embodiments shown herein, but conforms to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A rotating target detection method for chip internal component failure, characterized in that: The following steps are involved: Step 1: Collecting a top-view image of the chip interior taken by quantitative X-ray imaging technology; Step 2: Label the top view images of the chip and divide all images into training set, validation set and test set in a ratio of 3:1:1; Step 3: Preprocess the training set data; Step 4, constructing a rotating target detection network with Region of Interest Transformer as the core and Swin-Tiny network as the backbone, and iteratively training the rotating target detection network through the training set; wherein the Swin-Tiny network includes a partitioning module, a linear embedding layer, and multiple layered Swin Transformer Blocks, the partitioning module divides the input image into multiple image blocks of fixed sizes, the image blocks are projected to the Swin Transformer Block through the linear embedding layer, and the Swin Transformer Block uses a local window attention mechanism, a sliding window mechanism, and a multi-layer perceptron to process the image blocks; Step 5: Perform back propagation through the results of the validation set and the test set. If the rotation target detection network does not meet the requirements, adjust the parameters of the rotation target detection network and repeat step 4 for multiple iterative training. Otherwise, execute step 6. Step 6: Process the X-ray top view of the chip to be predicted according to steps 1 to 3, input the processed data into the rotating target detection network to obtain the final defect prediction result.

2. The rotating target detection method for chip internal component failure according to claim 1, characterized in that: The step 2 comprises: Step 2-1, assigning primary labels and auxiliary labels to defective areas and normal areas; Step 2-2, make the format of the label file follow the format of the DOTA dataset, and output the label information of each sub-graph as a .txt file. The data format is the x-coordinate of the upper left vertex of the rotating box, the y-coordinate of the upper left vertex of the rotating box, the x-coordinate of the lower left vertex of the rotating box, the y-coordinate of the lower left vertex of the rotating box, the x-coordinate of the upper right vertex of the rotating box, the y-coordinate of the upper right vertex of the rotating box, the x-coordinate of the lower right vertex of the rotating box, the y-coordinate of the lower right vertex of the rotating box, whether the rotating box belongs to a difficult-to-predict box, and the prediction category of the rotating box.

3. The rotating target detection method for chip internal component failure according to claim 1, characterized in that: The preprocessing includes random rotation, perturbed random rotation, Mixup and random erasure data enhancement techniques.

4. The rotating target detection method for chip internal component failure according to claim 3, characterized in that: The random rotation includes: flipping the image along a vertical axis, flipping the image along a horizontal axis, and flipping the image along a diagonal line.

5. The rotating target detection method for chip internal component failure according to claim 3, characterized in that: The perturbed random rotation is: x′=C x +(xC x )cos(θ)-(yC y )sin(θ) y′=C y +(xC x )sin(θ)+(yC y )cos(θ) Among them, (x, y) is the original point, (x', y') is the new coordinate after the perturbation random rotation, (C x ,C y ) is the image center and θ is the rotation angle.

6. The rotating target detection method for chip internal component failure according to claim 3, characterized in that: The segmentation module specifically includes: starting from the left vertex of the image, cutting out a 1024*1024 image block, and gradually moving the starting point rightward and downward until the cutting is completed, and each image block is projected to the Swin Transformer Block through a linear embedding layer.

7. The rotating target detection method for chip internal component failure according to claim 1, characterized in that: The sliding window mechanism includes: continuously moving the starting point to the right by 512 pixels through the sliding window, cutting out a 1024*1024 image block again until the cutting is completed in the horizontal direction, then moving the starting point downward by 512 pixels, repeating the cutting process in the horizontal direction until the entire area of ​​the image is cut, and marking and adjusting the cut image block with a rotating frame.

8. The rotating target detection method for chip internal component failure according to claim 1, characterized in that: The attention weight of the local window attention mechanism is Among them, Q and K are two matrices, which are transformed by the input data through the fully connected layer of Region of Interest Transformer, softmax() is the normalization function, and d k is the square of the number of key-value pairs k.

9. The rotating target detection method for chip internal component failure according to claim 1, characterized in that: The loss function of the rotation target detection network includes: CrossEntropyLoss: SmoothL1Loss: Among them, x is the output of the network, class is the target category, and x[j] represents the jth component of x.

10. A rotating target detection system for chip internal component failures implementing any of the methods described in claims 1-9, characterized in that: include: An image acquisition unit, used to collect top-view images of the inside of the chip taken using quantitative X-ray imaging technology; The data set construction unit is used to annotate the top-view images of the chip interior and divide all images into training set, validation set and test set in a ratio of 3:1:1; A preprocessing unit, used for preprocessing the training set data; A rotating target detection network construction unit is used to construct a rotating target detection network with Region of Interest Transformer as the core and Swin-Tiny network as the backbone, and the rotating target detection network is iteratively trained through a training set; wherein the Swin-Tiny network includes a partitioning module, a linear embedding layer, and a plurality of layered Swin Transformer Blocks, the partitioning module divides the input image into a plurality of image blocks of fixed sizes, and the Swin Transformer Block uses a local window attention mechanism, a sliding window mechanism, and a multi-layer perceptron to process the image blocks; The verification and testing unit performs back propagation through the results of the verification set and the test set. If the rotation target detection network does not meet the requirements, the parameters of the rotation target detection network are adjusted; The detection unit inputs the pre-processed X-ray top view data of the chip to be predicted into the rotating target detection network to obtain the final defect prediction result.

Images