An adaptive rotation defect detection method for chip x-ray side view

Abstract

This invention discloses an adaptive rotation defect detection method for chip X-ray side views. The method includes: constructing a ResNet-50-based backbone network for image feature extraction; adding an FPN module to fuse and enhance feature maps from different levels; constructing a candidate box generation module, which reduces the number of channels in the feature maps and obtains a set of high-quality rotation pre-selected boxes; building a detection head module to correct the pre-selected boxes and determine the type of anomaly within the boxes; introducing KLD to measure the prediction loss of the rotation boxes; and determining the anomaly location and type based on the obtained correction information and classification score. This invention designs a candidate box generation module and a KLD-based loss function. The candidate box generation module aims to generate high-quality candidate boxes using fewer parameters, while the KLD-based loss function dynamically calculates loss weights for targets with different aspect ratios and tilt angles, thereby guiding the model to generate a rotation box that better fits the target.

Description

Technical Field

[0001] This invention relates to the field of semiconductor chip defect detection technology, and more specifically, to an adaptive rotation defect detection method for chip X-ray side views. Background Technology

[0002] In the field of semiconductor chips, chip manufacturing processes are complex and diverse. Chips may develop invisible defects due to factors such as packaging techniques, environment, and materials. These defects can affect chip performance to varying degrees.

[0003] Chip defect detection is a key quality monitoring technology in chip manufacturing. However, in the past, the detection process relied heavily on manual screening using X-ray images of chips, a method that is inefficient and prone to errors. Compared to traditional defect detection techniques, computer vision-based detection offers advantages such as non-contact, flexibility, and high efficiency, pointing to a new direction for chip defect detection. However, current computer vision-based chip defect detection methods work by extracting image features from multiple convolutional neural networks and then identifying and classifying defects. The classification results are presented as horizontal bounding boxes. This method suffers from a high false negative rate when dealing with chips with numerous, small, and varied bonding wires, and the detection accuracy is insufficient to replace manual methods. Furthermore, chip defect detection faces challenges such as difficulty and high cost in obtaining training samples, making it difficult to generalize to the detection of different types of chips.

[0004] With the development of computer vision in the field of deep learning, object detection tasks have seen significant advancements, making them well-suited for chip defect detection. Unlike horizontal bounding box-based object detection, rotated bounding box-based object detection is a more challenging task. It is better able to adapt to different aspect ratios of targets in chip X-ray side views and also exhibits better performance in distinguishing overlapping targets. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive rotational defect detection method for chip X-ray side views.

[0006] The technical solution for achieving the present invention is as follows: In a first aspect, the present invention provides an adaptive rotation defect detection method for chip X-ray side view, comprising the following steps:

[0007] The first step is to construct a ResNet-50-based backbone network for extracting image features;

[0008] The second step is to add the FPN module after obtaining the feature maps to fuse and enhance the feature maps at different levels respectively.

[0009] The third step is to input the feature map into a candidate box generation module. This network reduces the number of channels in the feature map through a 3×3 convolution and then uses two 1×1 convolutions to obtain a set of high-quality rotated preselected boxes.

[0010] The fourth step is to build a detection head module to correct the preselected box and determine what kind of anomalies exist within the box;

[0011] The fifth step is to introduce KLD to measure the prediction loss of the rotated box, so as to adaptively calculate the side length weight and angle weight of the box.

[0012] The sixth step is to determine the location and type of the anomaly based on the obtained correction information and classification score.

[0013] In a second aspect, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect.

[0014] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, the significant features of this invention are: (1) The chip defect detection of this invention is based on the chip X-ray side view. Compared with other defect detection based on the chip top view, the chip detection based on the side view can better identify the chip's free particles, internal lead sagging and other problems; (2) The candidate box generation module is designed to accelerate the generation efficiency of rotating candidate boxes to improve the inference speed of the model; (3) Since most of the detection targets in the chip side view have a large aspect ratio and distribution density, the KLD loss function is designed to adapt to the shape of the target and dynamically adjust the weight of the loss function so that the model converges faster and has higher accuracy.

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a flowchart of the adaptive rotation defect detection method for chip X-ray side view according to the present invention.

[0018] Figure 2 This is the fast candidate box generation module of the present invention.

[0019] Figure 3The images show the results of the method of this invention on a chip X-ray side view dataset. (a) is an image of the chip X-ray side view dataset; (b) is the detection map of the chip X-ray side view dataset using the Rotated-Retinanet method; (c) is the detection map of the chip X-ray side view dataset using the Rotated-FRCNN method; (d) is the detection map of the chip X-ray side view dataset using the Oriented R-CNN method; and (e) is the detection map of the chip X-ray side view dataset using the method of this invention. Detailed Implementation

[0020] This invention proposes an adaptive rotation defect detection method for chip X-ray side views. By constructing a candidate box generation module and a KLD-based loss function, it addresses the slow inference speed and low accuracy of conventional algorithms. In the candidate box generation stage, this invention abandons the scheme of generating pre-selected boxes based on 54 anchor points, instead using three anchor points with the same area but different aspect ratios to directly generate pre-selected boxes, thus accelerating the generation speed. In the loss function design, this invention employs a KLD function with mutually coupled parameters, allowing the weights of the loss function to be dynamically adjusted according to the target situation, achieving faster convergence and higher accuracy.

[0021] Combination Figure 1 The implementation process of this invention is described in detail below, and the steps are as follows:

[0022] The first step involves constructing a ResNet-50-based backbone network for image feature extraction. The ResNet-50 network consists of four residual blocks, and the input image is a chip X-ray side view. The network input passes through the ResNet-50 to obtain the basic feature map.

[0023] The second step, after obtaining the feature maps, involves adding the FPN module to fuse and enhance feature maps from different levels. The specific process is as follows:

[0024] After using ResNet-50, each residual block yields features at different levels, but these features lack correlation with each other. Considering the FPN module's ability to fuse and enhance feature maps from different levels, after obtaining the basic features, these features are imported into the FPN module for enhancement. The FPN operates as follows:

[0025] F i =C(f) i )+U(F i+1 )

[0026] Here, f represents the i-th layer feature map output by FPN, C(·) represents a convolution operation with a kernel size of 1×1, and f iThis represents the i-th basic feature layer extracted by ResNet-50. U(·) represents a doubling upsampling operation. When i represents the highest layer feature, no upsampling operation is performed.

[0027] The third step involves inputting the feature map into a candidate box generation module. This network reduces the number of channels in the feature map using a 3×3 convolution, followed by two 1×1 convolutions to obtain a set of high-quality rotated pre-selected boxes, such as... Figure 2 As shown, the specific process is as follows:

[0028] We will obtain image feature maps F at different levels from the FPN module. i For each feature map, we first reduce the number of channels to 64 using a 3×3 convolution to reduce subsequent computational complexity. Then, we perform convolution operations on two 1×1 convolutions on the reduced feature maps to obtain a set of regression parameters for the preselected boxes and the probability that the corresponding predicted box is foreground, expressed by the following formula:

[0029] B(F i ) = C 1×1R (C 3×3 (F i ))

[0030] P(F i ) = C 1×1P (C 3×3 (F i ))

[0031] Among them, C 1×1R (·) indicates a convolution operation with a kernel size of 1×1, and the subscript R indicates the regression parameters used to generate the preselection box, with 6 output channels; C 1×1P (·) represents a convolution operation with a kernel size of 1×1, where the subscript P indicates the probability that the preselected box is foreground, and its output channels are 2; C 3×3 (·) represents a convolution operation with a kernel size of 3×3 and an output channel of 64, used for dimensionality reduction; B(·) and P(·) represent the regression parameters of the preselected box and the probability that the corresponding candidate preselected box is the foreground, respectively.

[0032] The fourth step is to build a detection head module to correct the preselected box and determine what kind of anomalies exist within the box. The specific process is as follows:

[0033] In the third step, a set of rotated preselected boxes are output. These preselected boxes and the feature maps of the corresponding layers are used as input to the detection head module. Rotated RoIAligin is used to extract a fixed-size feature vector, which is then input into two consecutive fully connected networks for classification and regression, respectively.

[0034] The fifth step involves introducing KLD (Kullback-Leibler Divergence) to measure the prediction loss of the rotated bounding box. This allows for adaptive calculation of the side length and angle weights of the bounding box, thereby achieving target detection performance that adapts to different aspect ratios. The specific process is as follows:

[0035] The predicted bounding boxes are transformed from a vector form into a Gaussian distribution representation. The same operation is performed on the ground truth bounding boxes. Then, the KLD (Knowledge, Distribution, and Allocation) of the two Gaussian distributions is calculated. This KLD is used as the loss function to guide the model's prediction, achieving adaptive calculation of the side length and angle weights of the bounding boxes, thus achieving target detection performance suitable for different aspect ratios. The loss function is shown below:

[0036]

[0037] in Let p and t represent the Gaussian distributions of the predicted and ground truth bounding boxes, respectively. The subscripts p and t represent the predicted and ground truth bounding boxes, respectively. The Gaussian distribution of the bounding box is denoted as... μ t μ represents the center point of the true target bounding box. p ∑ represents the center point of the predicted bounding box. t Let ∑ be the positive semidefinite real symmetric covariance matrix of the Gaussian distribution representation of the true bounding box. p Let be the semi-positive definite real symmetric covariance matrix under the Gaussian distribution representation of the predicted target box; Tr(·) denotes finding the trace of the matrix.

[0038] The coordinates of the bounding box are represented as B(x, y, w, h, θ), and the Gaussian distribution of the bounding box is represented as... The Gaussian representation and coordinate representation can be converted according to the following formula:

[0039] μ = (x, y) T

[0040]

[0041] Where μ = (x, y) T This represents the coordinates of the center point of the target bounding box, w and h represent its width and height respectively, and θ is the tilt angle of the target bounding box.

[0042] The reason it can adaptively adjust the weights of gradient backpropagation is that each term of the function is composed of coupled partial parameters, which makes all parameters form a chain of coupling relationships.

[0043] When D kl For μ p When differentiating, we set θ t =0°, we can get:

[0044]

[0045] Where D kl (μ p ) represents the KLD value between the predicted bounding box and the ground truth bounding box, μ p w represents the center point of the predicted bounding box. t h t Δx and Δy represent the width and height of the true target bounding box, respectively, and Δx and Δy represent the changes in the independent variables on the x-axis and y-axis, respectively.

[0046] From the formula, we can see that when the width is large, the weight of the corresponding x-axis will become smaller, and the change of the corresponding y-axis will be amplified, thus achieving adaptive aspect ratio.

[0047] Similarly, other parameters influence each other and optimize together during the optimization process of the KLD-based detector, making the model's optimization mechanism self-regulating.

[0048] However, if the smooth L1 loss function is used, this adaptive capability is lost. The formula for smooth L1 is as follows:

[0049]

[0050] The sixth step involves determining the location and type of anomalies based on the obtained correction information and classification scores. For each predicted bounding box, a correction vector and a classification vector are generated.

[0051] t=(Δx,Δy,Δw,Δh,Δθ)

[0052] P = (p1...p i ...p n )∈R n*1

[0053] Where t represents the correction vector, (Δx, Δy, Δw, Δh, Δθ) where each component represents the offset value along the s-axis, y-axis, width, height, and deflection angle, respectively; P represents the classification vector, n represents the number of anomaly categories, and p i Let p represent the probability that the target belongs to class i. i If the value is the largest, it indicates that the exception belongs to type i.

[0054] The effects of this invention can be further illustrated by the following simulation experiments:

[0055] Simulation conditions

[0056] The simulation experiment uses a chip X-ray side view dataset. The detection dataset contains 100 chip X-ray side views, each with a size of 2792×2400. In the experiment, the 100 images are divided into training, validation, and test sets in a 3:1:1 ratio. This invention implements the proposed defect detection method using the PyTorch framework. During training, the learning rate is set to 5×10⁻⁶. -05 The simulation experiments were conducted on an NVIDIA Titan V for 50 iterations.

[0057] For quantitative evaluation, this invention uses the mean average precision (MAP) score to measure detection accuracy.

[0058] Simulation content

[0059] This invention uses a chip X-ray side view dataset to evaluate the algorithm's performance. Three representative change detection algorithms are used for comparison: Rotated-RetinaNet, Oriented R-CNN, and Rotated-FRCNN.

[0060] Analysis of Simulation Experiment Results

[0061] Table 1 shows the comparison results of different object detection algorithms on the chip X-ray side view dataset. As shown in Table 1, compared with the three change detection algorithms, the adaptive rotation defect detection method for chip X-ray side view proposed in this invention performs best on the chip X-ray side view dataset and achieves the highest accuracy of 80.6%. The settings of the candidate box generation module and the KLD loss function reduce the number of model parameters and update the model by adaptively adjusting the weights of the loss function, which is the main reason why the model achieves the best results. Rotated-Retinanet performs the worst on the chip X-ray side view dataset, with a map of 52.1%, which is 28.5% lower than that of this invention. This is because it uses a one-stage object detection method. The metrics of Oriented R-CNN and Rotated-FRCNN are 74.4% and 67.3%, respectively, which are 6.2% and 13.3% lower than that of this invention, respectively.

[0062] Table 1. Quantitative evaluation of different target detection algorithms on the chip X-ray side view dataset.

[0063] Methods backbone loss map(%) Rotated-Retinanet Resnet50 Smooth L1 52.1 Rotated-FRCNN Resnet50 Smooth L1 67.3 Oriented R-CNN Resnet50 Smooth L1 74.4 Ours Resnet50 KLD 80.6

[0064] The results of the method of this invention on the chip X-ray side view dataset are shown in the figure below. Figure 3 As shown. Figure 3In the diagram, (a) represents the X-ray side view image of the chip under test, and (b) represents the detection result of Rotated-RetinaNet. It can be seen that this detection result did not detect any bonding wire drooping anomalies, while (e) shows the detection result of the proposed method, which effectively identifies bonding wire drooping anomalies. However, (b) is inaccurate in detecting free particle anomalies, misdetecting them as bonding wire solder joints. While the Rotated-FRCNN detection result in (c) also correctly detects free particles, its confidence level for this detection result is lower than that of Oriented R-CNN and the proposed method in (d). Although Oriented R-CNN in (d) detects a more comprehensive range of targets compared to Rotated-RetinaNet and Rotated-FRCNN when detecting bonding wire drooping anomalies, its target fit is still inferior to that of the proposed method. The simulation results on the real chip X-ray side view dataset demonstrate the effectiveness of the proposed method.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it. Parts not described in detail are common knowledge to those skilled in the art. The scope of protection of the present invention is determined by the claims, and any equivalent changes made based on the technical teachings of the present invention are also within the scope of protection of the present invention.

Claims

1. An adaptive rotational defect detection method for chip X-ray side view, characterized in that, The method comprises the following steps: In the first step, a backbone network based on ResNet-50 is constructed to extract image features. In the second step, after the feature map is obtained, an FPN module is added to fuse and enhance the feature maps of different levels respectively. The third step involves inputting the feature map into a candidate box generation module, which uses a... Convolutions are used to reduce the number of channels in the feature map, and then two convolutions are used... The convolution yields a set of high-quality rotated preselection boxes; In the fourth step, a detection head module is constructed to correct the pre-selected frame and determine what kind of anomaly exists in the frame. In the fifth step, KLD is introduced to measure the prediction loss of the rotated frame, and the edge length weight and angle weight of the frame are adaptively calculated to achieve the target detection effect of adapting to different aspect ratios, and the specific process is as follows: The predicted target frame obtained by prediction is converted from a vector form to a Gaussian distribution form, and the same operation is performed on the real target frame, then the KLD of the two Gaussian distributions is calculated, and the KLD is used as a loss function to guide the prediction of the model, and the edge length weight and angle weight of the frame are adaptively calculated, and the loss function is as follows: ； wherein respectively denote the Gaussian distribution of the predicted target box and the real target box, and the subscript respectively denote the predicted target and the real target; the Gaussian distribution of the target box is represented as , denotes the center point of the real target box, denotes the center point of the predicted target box, is a semi-positive definite real symmetric covariance matrix under the Gaussian distribution representation of the real target box, is a semi-positive definite real symmetric covariance matrix under the Gaussian distribution representation of the predicted target box; denotes the trace of a matrix; The coordinates of the target box are represented as The Gaussian distribution of the target box is represented as ; wherein the Gaussian representation and the coordinate representation can be converted according to the following formula: ； ； wherein represents the center point coordinate of the target frame, respectively represent the width and height thereof, is the inclination angle of the target frame; When To When taking the derivative, set , we get: ； wherein represents a KLD value of a predicted target frame with respect to a real target frame, represents a center point of a predicted target frame, respectively represent a real target frame width and height, respectively represent a change amount of an independent variable on an axis and a change amount of an independent variable on an axis; In the sixth step, the abnormal position and abnormal type are determined according to the obtained correction information and classification score.

2. The method of claim 1, wherein the method further comprises: In the first step, a backbone network based on ResNet-50 is constructed to extract image features, that is, ResNet-50 is used as a feature extractor, and the input of the network is a chip X-ray side view, and the feature extractor can obtain the feature map of the image.

3. The method of claim 1, wherein the method further comprises: In the second step, after the feature map is obtained, an FPN module is added to fuse and enhance the feature maps of different levels respectively, and the specific process is as follows: After using ResNet-50, different levels of features are obtained for each residual block, and the features are input into the FPN module for enhancement, and the operation mode of the FPN is as follows: ； wherein represents the first layer feature map output by the FPN; represents a convolution operation with a kernel size of ; represents the first layer basic feature layer extracted by the ResNet-50; represents a two-fold up-sampling operation; when the highest layer feature is represented, no up-sampling operation is performed.

4. The method of claim 1, wherein the method further comprises: The third step involves inputting the feature map into a candidate box generation module, which uses a... Convolutions are used to reduce the number of channels in the feature map, and then two convolutions are used... The convolution process yields a set of high-quality rotated preselection boxes. The specific process is as follows: Obtaining image feature maps of different levels from the FPN module For each feature map, a convolution is first performed to reduce the number of channels to 64; then the reduced dimension feature map is respectively convolved on two convolutions to obtain a set of regression parameters of pre-selected frames and the probability that the corresponding predicted frame is foreground, which is expressed as follows: ； wherein, represents a convolution operation with a kernel size of , subscript represents regression parameters for generating pre-bounding boxes, with an output channel number of 6; represents a convolution operation with a kernel size of , subscript represents probabilities for pre-bounding boxes being foreground, with an output channel number of 2; represents a convolution operation with a kernel size of , with an output channel number of 64, for dimension reduction; respectively represent regression parameters of pre-bounding boxes and probabilities of corresponding pre-bounding boxes being foreground.

5. The method of claim 1, wherein the method further comprises: In the fourth step, a detection head module is constructed to correct the pre-selected frame and determine what kind of anomaly exists in the frame, and the specific process is as follows: In the third step, a group of rotated pre-selected frames are output, and the pre-selected frames and the corresponding level feature maps are jointly input into the detection head module, and a fixed size feature vector is extracted by using Rotated RoIAligin, and then the vector is input into two consecutive fully connected networks for classification and regression.

6. The method of claim 1, wherein the method further comprises: In the sixth step, the abnormal position and abnormal type are determined according to the obtained correction information and classification score; for each predicted frame, a correction vector and a classification vector are given. ； ； wherein represents a correction vector, each component represents a respective axis, axis, width, height, deflection angle; represents a classification vector, represents the number of classes of anomalies, represents the probability that the target belongs to class, if maximum, then it represents that the anomaly is of class.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the method of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the steps of the method of any one of claims 1-6.

Images