Chip defect detection method based on improved YOLOv3 model

Abstract

The application discloses a chip defect detection method based on an improved YOLOv3 model and belongs to the image detection field.The application solves the problem of low chip defect detection precision in complex situations.First, the images of different chip defects are acquired, a data set is constructed, and a training set and a test set are divided.Second, an EMO (Efficient Model) module is improved, and the SE (Squeeze and Excitation) attention inside the model is updated to CA (Coordinate Attention).Third, the improved EMO is used to replace the Darknet-53 backbone network of the original YOLOv3.Fourth, a SIoU (Smoothed Intersection over Union) loss function is used as the loss function in the original YOLOv3 network model, and the improved YOLOv3 model is used to train the data set to obtain a chip defect detection model.Finally, the trained model is used for detection.The model can enhance the expression ability of network learning features, improve the accuracy and real-time performance of the model, and realize accurate detection of chip defects in complex situations.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of chip surface defect detection. BACKGROUND

[0002] Chip is the core device in the electronic industry, and a large number of defective products will appear in the mass production of chips. Chip packaging test is an important link to exclude defective products, ensure product quality and production reliability. The traditional chip surface defect detection method usually adopts manual visual detection method, but this method is low in efficiency and prone to misjudgment.

[0003] In recent years, the chip surface defect detection method based on deep learning has become one of the important means of chip production quality control. Compared with the traditional rule detection algorithm, the deep learning model can train the neural network model to detect product quality and find out the defective products. It can more accurately identify various defects, reduce the product failure rate and improve the efficiency of the production line.

[0004] However, there are some challenges in deploying deep learning models to factory sites. The production equipment is usually difficult to upgrade, and improving the model means replacing the equipment. This will increase the deployment cost. A feasible method is to use edge computing to deploy a lightweight deep learning model compatible with the current hardware at the end of the production line, so that the flexibility of model upgrading can be obtained. At present, many deep learning models are deployed in embedded devices, but the system model has the characteristics of high complexity and large amount of calculation, which is difficult to deploy. There are problems of large model calculation and poor detection accuracy. SUMMARY

[0005] The application is to solve the problem of large model and poor detection accuracy of existing embedded device deployment, and provides a chip defect detection method based on improved YOLOv3 model.

[0006] The chip defect detection method based on the improved YOLOv3 model comprises:

[0007] Step one, acquire images of different chip defects, label and label, construct a data set, and divide the training set and the test set;

[0008] Step two, improve the EMO (Efficient Model) module, update the SE (Squeeze and Excitation) attention in the model to CA (Coordinate Attention) to enhance the expression ability of mobile network learning features;

[0009] Step three, using improved EMO replaces the original YOLOv3 Darknet-53 backbone network, improves the average precision of network training model, reduces the computational complexity;

[0010] Step four, using SIoU(Smoothed Intersection over Union, Smoothed Intersection over Union) loss function replaces the original YOLOv3 network model CIoU(Complete Intersection over Union, Complete Intersection over Union) loss function, uses the improved YOLOv3 model to train the data set to obtain the chip defect detection model;

[0011] Step five, using the chip defect detection model to detect the test data set image, obtains the detection result.

[0012] Further, in the present application, in step one, the Japanese Toshiba industrial camera BU040M is used to collect chip surface defect images, a total of 400 images, the image pixel is 720x540, there are 7 common defects, including character defect, scratch defect, deformation hole, tape defect, chip reverse, chip loss, cutting defect, label annotation is carried out on different defect images, 90% of the data set is used as the training set, and 10% is used as the test set.

[0013] Further, in the present application, in step two, the EMO module is improved, the SE attention in the model is updated to CA attention, since the SE attention does not consider the spatial dimension, the CA attention can consider the attention on the channel dimension and the spatial dimension at the same time, and can learn adaptive channel weight, so that the model pays more attention to useful channel information. CA attention formula:

[0014] First, the global average pooling is decomposed, and the input feature map with the size of CxHxW is respectively pooled according to X and Y directions, respectively generating feature maps with the size of CxHx1 and Cx1xW;

[0015]

[0016]

[0017] Wherein, The cth channel output with a height of h is, The cth channel output of w, x c The cth channel input, i is the iteration variable in the width direction, j is the iteration variable in the height direction, and m is the maximum value function.

[0018] The generated Cx1xW feature map is transformed, and then the concat operation is performed. z h And zw After concat, the feature map is generated, then the dimension is reduced by using 1x1 convolution kernel, which is called F1, and finally the activation operation is performed to generate the feature map f C / r×(H+W)×1 ;

[0019] f = d (F1 ([z h ,z w ]))

[0020] Along the spatial dimension, f is split into f h ∈R C / r×H×1 and fw∈R C / r×1×W , then the dimension is increased by using 1x1 convolution, represented by F h and F w . Then, the final attention tensor g h ∈R C×H×1 and g h ∈R C×1×W is obtained by combining the sigmoid activation function.

[0021] g h =σ(F h (f h ))

[0022] g w =σ(F w (f w ))

[0023] The obtained attention weight g h in the height direction and the attention weight g w in the width direction are weighted by multiplication with the original feature map x c (i,j), and the output formula of CA is finally obtained:

[0024]

[0025] Further, in the step three, in the YOLOv3 network model, the improved EMO backbone network is used to replace the original Darknet-53 backbone network of YOLOv3, in order to improve the average precision mean of the network training model and reduce the calculation complexity, the formula is:

[0026]

[0027] Wherein, the W-MSA (Window Multi-head Self-Attention, window multi-head self-attention) module divides the h x w x c feature map into several small windows. Assuming that the side length of each square window is M, the division obtains windows, and attention calculation is performed on each window respectively. Each window corresponds to a calculation amount of a feature map with a height of h, a width of w, and a channel depth of c, and the calculation amount becomes 4hwc 2 + 2(hw) 2 c, and the total calculation amount of all windows is 4(Ac) 2 + 2A 4 c.

[0028] Further, in the present application, in step four, the SIoU loss function is used to replace the CIoU loss function in the original YOLOv3 network model, and the improved model is used to train the data set to obtain the final chip defect detection model. The specific calculation method of the angle cost, distance cost and shape cost of the SIoU loss function is as follows:

[0029] Angle cost Λ:

[0030]

[0031]

[0032]

[0033]

[0034] Wherein, x is the sine value of α, σ is the distance between the center points of the real box and the predicted box, and represent the center point coordinates of the target box; and represent the center point coordinates of the predicted box; c w and c h represent the difference between the target box and the predicted box center point in the width and height direction.

[0035] Distance cost Δ:

[0036]

[0037]

[0038]

[0039] γ=2-Λ

[0040] Wherein, p t is the square of the difference between the center point coordinates of the two boxes, γ is a control parameter, p x represents the depth gradient in the x-axis direction, p y represents the depth gradient in the y-axis direction, and Λ is calculated from the angle cost.

[0041] Shape cost Ω:

[0042]

[0043]

[0044]

[0045] wherein, w is the predicted frame width, h is the predicted frame height, w gt is the real frame width, h gt is the real width height, the weight of the shape cost in the positioning loss is controlled by the parameter θ, the greater the value is, the less important the shape cost in the positioning loss is. w ω represents the ratio of the difference between the real frame and the predicted frame width and the maximum value, ω h represents the ratio of the difference between the real frame and the predicted frame height and the maximum value.

[0046] The IoU loss function is:

[0047]

[0048] The SIou loss function is:

[0049]

[0050] wherein, L SIoU represents the SIoU loss function of the model, IoU is the intersection over union of the predicted frame and the real frame, B represents the predicted frame, B GT represents the real frame, Δ is the distance cost, and Ω is the shape cost.

[0051] Further, in the present application, in step five, the training method includes freezing the weight of the skeleton feature network part, training multiple epochs, then unfreezing all weights, and the network training uses the following training parameters: learning rate 0.001, batch_size is 8, the optimizer uses Adam, the model iteration number is set to 220 times, when the model loss function and the average precision mean tend to be stable, the final training weight best.pt is obtained, and the test set is verified with the training weight.

[0052] Advantages:

[0053] The improved EMO backbone network is used to update the chip surface defect detection method of the original YOLOv3 network Darknet-53 backbone network, an end-to-end deep neural network EMO-YOLOv3 model is designed, the improved lightweight EMO backbone network is used for feature extraction, the SIoU loss function is used to help the confidence loss function of YOLOv3 to speed up the convergence speed, and the learning feature is further enhanced, so that the chip surface defect detection can be realized, the defects detected include scratches, character defects and the like, and the detection accuracy and calculation complexity are greatly optimized compared with other algorithms. The detection accuracy of the chip defect is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a chip defect detection method flow chart based on an improved YOLOv3 model in the method of the application.

[0055] Figure 2 is a principle diagram of a chip defect detection method based on an improved YOLOv3 model in the method of the application.

[0056] Figure 3 is an EMO structure principle diagram in the method of the application.

[0057] Figure 4 is an IRMB structure principle diagram in the method of the application.

[0058] Figure 5 is a CA attention diagram in the method of the application.

[0059] Figure 6 is a comparison diagram of improved YOLOv3 and original YOLOv3 chip defect generation effect in the method of the application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application. It should be noted that, in the case of no conflict, the embodiments in the application and the features in the embodiments can be combined with each other.

[0061] Specific implementation method one: refer to Figure 1 and Figure 2 The chip defect detection method based on the improved YOLOv3 model described in this embodiment includes:

[0062] Step one, acquire different chip defect images, label annotation, construct data set, and divide training set and test set;

[0063] Step two, improve the EMO (Efficient Model) module, update the SE (Squeeze and Excitation) attention in the model to CA (Coordinate Attention) attention to enhance the expression ability of mobile network learning features;

[0064] Step three, use the improved EMO to replace the original YOLOv3 Darknet-53 backbone network, improve the average precision mean of the network training model, and reduce the computational complexity;

[0065] Step four, replace the CIoU (Complete Intersection over Union) loss function in the original YOLOv3 network model with the SIoU (Smoothed Intersection over Union) loss function, and use the improved YOLOv3 model to train the data set to obtain a chip defect detection model;

[0066] Step five, use the chip defect detection model to detect the test data set image to obtain the detection result.

[0067] Further, in the present application, in step one, Japanese Toshiba industrial camera BU040M is used to collect chip surface defect images, a total of 400 images, the image pixels are 720x540, there are 7 common defects, including character defects, scratch defects, deformation holes, tape defects, chip reverse, chip loss and cutting defects, different defect images are labeled, 90% of the data set is used as the training set, and 10% is used as the test set.

[0068] Further, in the present application, in step two, the EMO module is improved, the SE attention in the model is updated to CA attention, as shown in Figure 3 、 4 As shown in the formula, since the SE attention does not consider the spatial dimension, the CA attention can consider the attention in the channel dimension and the spatial dimension at the same time, and can learn adaptive channel weights, so that the model pays more attention to useful channel information. The CA attention is as shown in Figure 5 ,

[0069] First, the global average pooling is decomposed, and the input feature map with the size of CxHxW is respectively pooled according to the X and Y directions, respectively generating feature maps with the sizes of CxHx1 and Cx1xW;

[0070]

[0071]

[0072] wherein, is the c-th channel output of height h, is the c-th channel output of width w, x c is the c-th channel input, i is the iteration variable in the width direction, j is the iteration variable in the height direction, and m is the maximum function.

[0073] The generated Cx1xW feature map is transformed and then subjected to a concat operation. z h and z w are concatenated to generate a feature map, which is then subjected to dimension reduction using a 1x1 convolution kernel, referred to as F1, and finally subjected to an activation operation δ to generate a feature map f∈R C / r×(H+W)×1 ;

[0074] f = δ (F1([z h ,z w ]))

[0075] Along the spatial dimension, f is subjected to a split operation to divide it into f h ∈R C / r×H×1 and f w ∈R C / r×1×W Then, 1x1 convolution is used to perform dimension increasing operations, denoted by F h and F w . In combination with a sigmoid activation function, the final attention tensors g h ∈R C×H×1 and g h ∈R C×1×W are obtained.

[0076] g h = σ (F h (f h ))

[0077] g w = σ (F w (f w ))

[0078] The obtained attention weight in the height direction g h and the attention weight in the width direction g w are combined with the original feature map x c (i,j) to perform multiplication weighting calculation, and finally the output formula of CA is obtained:

[0079]

[0080] Further, in the present application, in step three, in the YOLOv3 network model, the improved EMO backbone network is used to replace the original YOLOv3 Darknet-53 backbone network, in order to improve the average precision mean value of the network training model, and the formula for reducing the calculation complexity is:

[0081]

[0082] Among them, the W-MSA (Window Multi-head Self-Attention, window multi-head self-attention) module divides the h x w x c feature map into several small windows. Assuming that the length of each square window is M, the division obtains Window, then the attention calculation is performed on each window respectively. Each window corresponds to the calculation amount of a feature map with a height of h, a width of w, and a channel depth of c, and the calculation amount becomes 4hwc 2 +2(hw) 2 c, then the total calculation amount of all windows is 4(Ac) 2 +2A 4 c.

[0083] Further, in the present application, in step four, the SIoU loss function is used to replace the CIoU loss function in the original YOLOv3 network model, and the improved model is used to train the data set to obtain the final chip defect detection model. The specific calculation method of the angle cost, distance cost and shape cost of the SIoU loss function is as follows:

[0084] Angle cost Lambda:

[0085]

[0086]

[0087]

[0088]

[0089] Among them, x is the sine value of alpha, sigma is the distance between the center points of the real box and the predicted box, And Indicate the target box center point coordinates; And Indicate the predicted box center point coordinates; c w And c h Indicate the difference between the target box and the predicted box center point in the width and height directions.

[0090] Distance cost Delta:

[0091]

[0092]

[0093]

[0094] γ = 2 - Λ

[0095] wherein, p t is the square of the difference of the center point coordinates of the two boxes, γ is a control parameter, p x represents the depth gradient in the x-axis direction, p y represents the depth gradient in the y-axis direction, and Λ is obtained by angle cost calculation.

[0096] Shape cost:

[0097]

[0098]

[0099]

[0100] wherein, w is the predicted box width, h is the predicted box height, w gt is the real box width, h gt is the real width and height, and the weight of the shape cost in the positioning loss is controlled by the parameter θ, and the greater the value, the less important the shape cost in the positioning loss. ω w represents the ratio of the difference between the real box and the predicted box width to the maximum value, and ω h represents the ratio of the difference between the real box and the predicted box height to the maximum value.

[0101] The IoU loss function is:

[0102]

[0103] The SIou loss function is:

[0104]

[0105] wherein, L SIoU represents the SIoU loss function of the model, IoU is the intersection over union of the predicted box and the real box, B represents the predicted box, B GT represents the real box, Δ is the distance cost, and Ω is the shape cost.

[0106] Further, in the present application, in step five, the training method includes freezing the weights of the skeleton feature network part, training multiple epochs, and then unfreezing all weights. The network training uses the following training parameters: learning rate 0.001, batch_size is 8, the optimizer uses Adam, the model iteration number is set to 220 times, when the model loss function and the average precision mean tend to be stable, the final training weight best.pt is obtained, and the training weight is used to verify the test set.

[0107] Further, first, the picture to be detected needs to be standardized for easy input into the trained network. Then, the network will output the standardized picture and filter the output area threshold, filtering out positions where chip defects are unlikely to exist. Then, use the confidence threshold to filter the final result, and only chip defect targets with a confidence greater than the threshold will be output as the final prediction. The output area threshold includes a width direction threshold and a height direction threshold, which are defined as the ratio of the distance between the center coordinates of the chip defect target and the boundaries of the picture to the width and height of the entire picture.

[0108] Precision of the chip defect model The calculation formula is as follows, A refers to the number of times of correct detection of chip defects, and B refers to the number of actual targets of a given class in the chip defect image:

[0109]

[0110] Among them, the Precision of a given chip defect image class C is c N(True Positives) is the number of correct predictions (True Positives) of the chip defect image c Divided by the total number of targets N(Total Objects) in the chip defect class c ;

[0111] For the entire test set, the average precision of the chip defect model is used to evaluate the chip defect model, as follows:

[0112]

[0113] Among them, the Average Precesion of a C class is c ∑Precesion is the sum of the precision values of all chip defect images for class C on the test set c Divided by the number of all face images N(Total Objects) of the target class C c ; For the test set and the chip defect model, Average Precesion is used cThe measurement is made.

[0114] The mAP formula is:

[0115]

[0116] Where, the sum of AP of all categories is divided by the total number of categories K is mAP

[0117] Table 1 shows the comparative results of the proposed method and other object detection algorithms Faster-RCNN, SSD (Single Shot Detector), Poolformer-YOLOv3, Mobilevit-YOLOv3, VanillaNet-YOLOv3 for detecting the chip defect dataset. The results show that the detection performance of the proposed method is better than that of other methods, and the detection accuracy of the seven types of defects of the dataset reaches 95.4%.

[0118] Table 1 Performance comparison of the proposed method and other algorithms

[0119]

[0120] As shown in Table 1, the mAP of the proposed EMO-YOLOv3 is 95.4%, and the accuracy is 91.2%, which is improved by 4.1% and 2.8% respectively compared with the original YOLOv3. In addition, the parameter amount of EMO-YOLOv3 is optimized compared with other algorithms, and the detection speed is improved from 53 frames of the standard YOLO to 60 frames, which shows obvious superiority.

[0121] The model is applied to the constructed chip surface defect dataset, and the results are shown in Table 1. Figure 6 The EMO-YOLOv3 model proposed in the present application can recognize all targets and classify correctly.

Claims

1. A chip defect detection method based on an improved YOLOv3 model, characterized in that, The method comprises the following steps: Step one, obtain images of different chip defects, label, construct a data set, and divide the training set and test set; Step two, improve the EMO (Efficient Model) module, update the SE (Squeeze and Excitation) attention in the model to CA (Coordinate Attention) to enhance the expression ability of mobile network learning features; Step three, use the improved EMO to replace the original YOLOv3 Darknet-53 backbone network, improve the average precision of the network training model, and reduce the computational complexity; Step four, use the SIoU (Smoothed Intersection over Union) loss function to replace the CIoU (Complete Intersection over Union) loss function in the original YOLOv3 network model, and use the improved YOLOv3 model to train the data set to obtain a chip defect detection model; Step five, use the chip defect detection model to detect the test data set image to obtain the detection result.

2. The chip defect detection method based on the improved YOLOv3 model according to claim 1, characterized in that, In step one, 400 chip surface defect images are collected by using a Toshiba industrial camera BU040M, and the image size is 720x540 pixels, including character defects, scratch defects, deformation holes, tape defects, chip reverse, chip loss, and cutting defects. A total of 7 types of defects are labeled for different defect images, and 90% of the data set is used as the training set and 10% as the test set.

3. The chip defect detection method based on the improved YOLOv3 model according to claim 1, characterized in that, In step two, the EMO module is improved, and the SE attention in the model is updated to CA attention. Since the SE attention does not consider the spatial dimension, the CA attention can consider the attention in the channel dimension and the spatial dimension at the same time, and can learn adaptive channel weights, so that the model pays more attention to useful channel information. The CA attention formula is: First, decompose the global average pooling, and pool the input feature map with a size of CxHxW according to the X and Y directions respectively, to generate feature maps with sizes of CxHx1 and Cx1xW respectively: wherein, is the c-th channel output of height h, is the c-th channel output of width w, x c is the c-th channel input, i is an iteration variable in the width direction, j is an iteration variable in the height direction, and m is a maximum function. The generated Cx1xW feature map is transformed, and then a concat operation is performed on z h and z w The generated feature map is subjected to a concat operation, and then a 1x1 convolution kernel is used for dimension reduction, referred to as F1, and finally an activation operation δ is performed to generate a feature map f∈R C / r×(H+W)×1 ; f = δ(F1([z h ,z w ])) Along the spatial dimension, f is split into fh∈R C / r×H×1 and f w ∈R C / r×1×W Then, dimension is increased by 1×1 convolution, and F h and F w are denoted, and the final attention tensor g h ∈R C×H×1 and g w ∈R C×1×W are obtained by combining the sigmoid activation function. g h = σ (F h (f h )) g w = σ (F w (f w )) The attention weight g in the height direction obtained h and the attention weight g in the width direction w and the original feature map x c (i,j) is calculated by multiplication weighting, and finally the output formula of CA is obtained, 4. The chip defect detection method based on the improved YOLOv3 model according to claim 1, characterized in that, In step three, in the YOLOv3 network model, the improved EMO backbone network is used to replace the original YOLOv3 Darknet-53 backbone network. In order to improve the average detection accuracy of the network training model and reduce the computational complexity, the formula is: Wherein, the W-MSA (Window Multi-head Self-Attention) module divides the h x w x c feature mapping into several small windows, assuming that the side length of each square window is M, and the division is performed to obtain Window, and then attention calculation is performed on each window respectively, and the calculation amount of each window corresponds to a feature map with a height of h, a width of w, and a channel depth of c, and the calculation amount becomes 4hwc 2 +2(hw) 2 c, then the total calculation amount of all windows is 4(Ac) 2 +2A 4 c, A refers to the number of times of correct detection of chip defects.

5. The chip defect detection method based on the improved YOLOv3 model according to claim 1, characterized in that, In step four, the SIoU loss function is used to replace the CIoU loss function in the original YOLOv3 network model, and the improved model is used to train the data set to obtain the final chip defect detection model. The specific calculation method of the angle cost, distance cost, and shape cost of the SIoU loss function is as follows: Angle cost Lambda: wherein x is a sine value of a, and s is a distance between a center point of a real frame and a center point of a predicted frame, and denotes a center point coordinate of a target frame; and denotes a center point coordinate of a predicted frame; c w and c h denotes a difference between a target frame and a predicted frame in a width and height direction. Distance cost Delta: Gamma = 2 - Lambda wherein, ρ t is the square of the difference of the center point coordinates of two blocks, γ is a control parameter, ρ x represents the depth gradient in the x-axis direction, ρ y represents the depth gradient in the y-axis direction, and Λ is obtained by angle cost calculation; Shape cost Omega: where w is the predicted bounding box width, h is the predicted bounding box height, w gt is the real bounding box width, h gt is the real bounding box height, and the weight of the shape cost in the localization loss is controlled by the parameter θ, the greater the value of which, the less important the shape cost in the localization loss, ω w represents the ratio of the width difference between the real bounding box and the predicted bounding box to the maximum value, ω h represents the ratio of the height difference between the real bounding box and the predicted bounding box to the maximum value. IoU loss function is: SIoU loss function is: wherein L SIoU represents the SIoU loss function of the model, IoU is the intersection over union of the predicted box and the real box, B represents the predicted box, B GT represents the real box, Δ is the distance cost, and Ω is the shape cost.

6. The chip defect detection method based on the improved YOLOv3 model according to claim 1, characterized in that, In step five, the training method includes freezing the weights of the skeleton feature network part, training multiple epochs, and then unfreezing all the weights. The network training uses the following training parameters: a learning rate of 0.001, a batch size of 8, an Adam optimizer, a model iteration number of 220 times, and when the model loss function and the average precision mean tend to be stable, the final training weight best.pt is obtained, and the training weight is used to verify the test set.

Images