Improved chip small target defect detection method and system based on YOLOv5 and Swin Transform fusion

Abstract

The invention discloses an improved chip small target defect detection method and system based on YOLOv5 and Swin Transform fusion, and belongs to the technical field of chip defect detection, and the method comprises the steps: obtaining images of different types of defects on the surface of a chip; constructing a chip defect data set, and dividing the chip defect data set into a training set, a test set and a verification set; the collected chip defect data set is preprocessed; constructing a deep learning detection model, and performing pre-training; optimizing the deep learning detection model; and performing channel pruning and fine tuning on the deep learning detection model. According to the invention, the small target defect of the chip can be rapidly and accurately detected.

Description

Technical Field

[0001] The present invention relates to the field of chip defect detection technology, and in particular to a chip small target defect detection method and system based on an improved fusion of YOLOv5 and SwinTransformer. Background Art

[0002] In the field of integrated circuit manufacturing, chips are the core components of electronic products. Various types of defects may occur during the production process, such as welding problems, component damage, and line short circuits. Timely detection and resolution of these defects are crucial to ensuring product quality, improving production efficiency, and reducing production costs. However, current chip defect detection methods mainly rely on manual inspection or special inspection equipment. These methods have problems such as low efficiency, high cost, and insufficient accuracy in detecting small target defects, which can no longer meet the production needs of modern chips.

[0003] With the rapid development of computer vision and deep learning technologies, automated defect detection methods based on these technologies have gradually become an effective way to solve the above problems. By using advanced image processing algorithms and deep learning models with automated inspection systems, various defects in the chip manufacturing process can be quickly and efficiently identified and located, thereby greatly improving the inspection efficiency and accuracy, reducing the inspection cost, and significantly improving the reliability of chip quality. To this end, researchers apply deep learning algorithms to the field of chip defect detection to improve the detection rate and efficiency of chip defects. However, the existing algorithms are affected by the small size defects of the chip and the complex texture background of the chip, resulting in low detection accuracy. At the same time, the detection model is too complex, the reasoning time is too long, and the calculation parameters are too large, making it difficult to deploy on industrial control equipment and unable to meet the real-time detection of chip defects. Summary of the invention

[0004] To solve the above problems, the present invention provides a chip small target defect detection method and system based on the fusion improvement of YOLOv5 and Swin Transformer.

[0005] To achieve the above object, the technical solution of the present invention is: The chip small target defect detection method based on the fusion of YOLOv5 and Swin Transformer is improved, including: Step S1, acquiring images of different types of defects on the chip surface; Step S2, constructing a chip defect data set and dividing it into a training set, a test set, and a validation set; Step S3, preprocessing the collected chip defect data set; Step S4, constructing a deep learning detection model and performing pre-training; Step S5, optimizing the deep learning detection model; The step S5 includes, step S54, adding a small target detection layer based on the Swin Transformer architecture at the 4-fold downsampling of the initial deep learning detection model YOLOv5; Step S6: perform channel pruning and fine-tuning on the deep learning detection model.

[0006] Furthermore, the above step S5 also includes step S51, improving the backbone network, and concatenating cross-layer feature maps of ResNet and CSP in the backbone network.

[0007] Furthermore, the above step S5 also includes, step S52, embedding an attention mechanism module, embedding the attention mechanism module before the last SPPF module in the backbone network, and embedding the attention mechanism module between the bottleneck layer and the prediction box.

[0008] Furthermore, the above step S5 also includes a calculation method for optimizing the loss function.

[0009] Furthermore, in the above step S53, the optimized positioning loss function The calculation formula is as follows: , Among them, x and y is the center point of the prediction box, x gt and y gt is the center point of the real frame, and are the height and width of the bounding box.

[0010] Furthermore, in the above step S53, the expression of the optimized classification loss function is as follows: In the formula, q represents the true label, p For the predicted results, α is the balancing factor for negative samples, γ represents the modulation factor.

[0011] Furthermore, the above step S3 includes: Step S31, performing spatial transformation on the image of the chip defect data set; Step S32, performing data enhancement on the image of the chip defect data set; Step S33, performing sliding segmentation and cutting on the image of the chip defect data set; Step S34, stitching the images.

[0012] Furthermore, the above step S33 includes: Step S331, setting a sliding window of 640×640 pixels at a vertex of the original chip defect image, the vertex is the initialization position, and the X coordinate of the vertex is the starting X coordinate; Step S332, starting from the initialization position, slide the window along the X-axis direction, sliding 320 pixels each time until the entire X-axis is covered; Step S333, after the sliding operation in the X-axis direction is completed, the sliding window returns to the starting X coordinate and moves down 320 pixels along the Y-axis direction, and repeats the sliding operation in the X-axis direction; Step S334, continuously repeating the sliding operation in the X-axis and Y-axis directions until the entire chip image is divided into a plurality of sub-images of 640×640 pixels.

[0013] Furthermore, the above step S4 includes: Step S41, constructing an initial model and optimizing the anchor frame; Step S42, pre-training the initial model.

[0014] Furthermore, the above step S41 includes: Step S411, collecting and arranging coordinate information of various defects in the chip defect data set as data points; Step S412, randomly selecting a data point as an initial cluster center; Step S413, calculate the distance between each initialized cluster center and the sample, and select the shortest distance d min ; Step S414, whether K cluster centers can be selected, if not, return to step S413; if yes, proceed to step S415; Step S415, preferentially selecting the point with the largest distance as the new cluster center; Step S416, calculating the probability of the next cluster center, and using the roulette method to find the next cluster center; Step S417, determine whether the convergence or iteration number reaches n, if not, go to step S415; if yes, go to step S418; Step S418, outputting the clustering result; Step S419, based on the clustering results, the number and size of the anchor boxes are optimized and adjusted to make them more suitable for various defect shapes and proportions in the chip defect dataset.

[0015] Furthermore, the above step S6 includes: Step S61, performing sparse training on the model; Step S62: according to the scaling factor γCompare the lightweighting effect with the pruning ratio, and select the one with the best lightweighting effect as the pruning strategy; Step S63, fine-tuning the pruning network; Step S64, based on the evaluation results of the validation set, the training is repeated continuously to obtain the optimal pruning result.

[0016] Furthermore, the above step S63 includes: Step S631, train a high-precision teacher model, and compare the logits output of the large model with the temperature T After the division, the soft target is obtained by performing SoftMax calculation; Step S632, train the student network and divide its logits output into two parts, one of which is directly used for SoftMax calculation, and the other is divided by the same temperature and then subjected to SoftMax calculation to obtain the predicted value; Step S633, compare the predicted value with the hard index, add the loss functions of the two parts to obtain a new loss function, and update the calculation parameters.

[0017] The present invention also provides a chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer, which applies the above-mentioned chip small target defect detection method based on the fusion improvement of YOLOv5 and Swin Transformer, including an image detection module, a video detection module, a camera detection module, a model initialization module, a model weight selection module, and a detection result display module; the image detection module obtains chip images from a chip defect data set; the video detection module obtains chip videos from the chip defect data set; the camera detection module obtains chip images through a high-speed industrial camera; the model initialization module initializes the detection model; the model weight selection module selects a weight file of a corresponding model according to different chip samples or different detection requirements; and the detection result display module displays the defect detection effect of the chip.

[0018] Beneficial effect: The present invention is based on the improved chip small target defect detection method and system based on the fusion of YOLOv5 and Swin Transformer. By expanding image data, sliding segmentation and improving loss function, the problem of low robustness of deep learning model caused by small chip data and unbalanced defect types is effectively solved. YOLOv5 is used as the benchmark model, a cross-stage partially connected residual network is used to replace the backbone network of the original model, the SimAM attention mechanism is integrated into the model, WIoU and Varifocal Loss loss functions are introduced, and a small target detection layer is added to achieve the result of quickly and accurately detecting small target defects of chips. Based on the characteristics of the BN layer, the scaling factor is used to γTo evaluate the importance of different channels, the original model was pruned, and the knowledge distillation fine-tuning method was used to improve the accuracy loss caused by pruning. A chip defect detection system was designed based on the PyQt5 framework to realize functions such as chip defect detection and recording of quantity and location, making chip defect detection more convenient.

[0019] In order to make the above features and advantages of the invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 The present invention is a flowchart of a chip small target defect detection method based on the fusion improvement of YOLOv5 and Swin Transformer.

[0021] Figure 2 This is a specific flow chart of step S3.

[0022] Figure 3 This is a specific flow chart of step S33.

[0023] Figure 4 This is a specific flow chart of step S4.

[0024] Figure 5 Schematic diagram of the network structure of the deep learning detection model YOLOv5.

[0025] Figure 6 This is a specific flow chart of step S41.

[0026] Figure 7 This is a specific flow chart of step S5.

[0027] Figure 8 Schematic diagram of the embedded attention mechanism module.

[0028] Fig. 9 This is the workflow diagram of Swin Transformer.

[0029] Fig.10 is a detailed flow chart of step S6.

[0030] Fig.11 is a detailed flowchart of step S63.

[0031] Fig.12 This is an operation flow chart of the chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer in the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose and technical solution of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] Figure 1 The figure is a flow chart of a chip small target defect detection method based on the fusion of YOLOv5 and Swin Transformer. Figure 1 As shown, a flowchart of a chip small target defect detection method based on the fusion improvement of YOLOv5 and Swin Transformer of the present invention includes the following steps.

[0034] Step S1, acquiring images of different types of defects on the chip surface.

[0035] In a specific embodiment, images of different types of defects on the chip surface may be acquired by a high-speed industrial camera.

[0036] Step S2, constructing a chip defect data set and dividing it into a training set, a test set, and a validation set.

[0037] In a specific embodiment, according to the VOC2007 dataset standard, LabelImg software is used to annotate defects in the collected images and generate corresponding label information to construct a chip defect dataset, and the chip defect dataset is divided into a training set, a test set, and a validation set in a ratio of 7:2:1.

[0038] Step S3, preprocessing the collected chip defect data set.

[0039] Furthermore, if Figure 2 As shown, step S3 includes the following steps.

[0040] Step S31, performing spatial transformation on the image of the chip defect data set.

[0041] More specifically, spatial transformation includes image rotation, flipping, translation and shearing. Through spatial transformation, multiple different images can be obtained, thereby improving the generalization of the model.

[0042] Step S32, performing data enhancement on the images of the chip defect dataset.

[0043] More specifically, the image is enhanced by adjusting image contrast, adding different noises, gamma transformation, etc., so as to enhance the robustness of the model.

[0044] Step S33, performing sliding segmentation and cutting on the image of the chip defect data set.

[0045] More specifically, the computing power of existing deep learning algorithm frameworks and hardware devices is difficult to effectively process large-size images. If large-size images are directly reduced to the input size required by the detection model, the image will lose a lot of feature information, making it difficult for the detection model to effectively learn and identify defects. In order to improve detection efficiency, reduce the amount of calculation, and expand the data set samples, while highlighting the detection effect of small target defects, the present invention proposes sliding segmentation and cutting of images of chip defect data sets. In this way, the larger-size whole image is divided into several smaller sub-images, thereby retaining the detailed feature information in the image, so that the detection model can learn and identify defects more effectively.

[0046] like Figure 3 As shown, step S33 further includes the following steps.

[0047] Step S331, setting a sliding window of 640×640 pixels at a vertex of the original chip defect image, the vertex is the initialization position, and the X coordinate of the vertex is the starting X coordinate.

[0048] Step S332, starting from the initialization position, slide the window along the X-axis direction, sliding 320 pixels each time until the entire X-axis is covered.

[0049] Step S333: After the sliding operation in the X-axis direction is completed, the sliding window returns to the starting X coordinate and moves down 320 pixels along the Y-axis direction, and repeats the sliding operation in the X-axis direction.

[0050] Step S334, continuously repeating the sliding operation in the X-axis and Y-axis directions until the entire chip image is divided into a plurality of sub-images of 640×640 pixels.

[0051] The above sliding segmentation method can not only improve the processing capability of large-size chip images and reduce the amount of calculation, but also improve the recognition effect of the detection model on small target defects by increasing the number of sub-image samples, thereby significantly improving the accuracy and efficiency of chip defect detection.

[0052] Step S34, stitching the images.

[0053] More specifically, in order to enhance the generalization of the detection model and improve the randomness of chip image data, the present invention also uses a splicing method to process chip images during training, randomly combining four different chip images into one picture, and the cropping size of each picture is random. This enhancement method enables a combined picture to contain different defect types, which can alleviate the impact of insufficient chip defect image data and enable the detection model to train better results with less chip defect data.

[0054] Step S4, construct a deep learning detection model and perform pre-training.

[0055] Furthermore, if Figure 4 As shown, step S4 includes the following steps.

[0056] Step S41, constructing an initial model and optimizing the anchor frame.

[0057] More specifically, a deep learning detection model YOLOv5 is constructed. The deep learning detection model YOLOv5 is a deep learning detection model that does not require recognition of regions and can directly implement target detection and classification based on regression problems. Its network structure is as follows Figure 5 As shown in the figure, the deep learning detection model YOLOv5 includes input, backbone network, bottleneck layer and prediction box.

[0058] Among them, the input end is used to process the input target chip image, adjust the input size of images of different sizes to 640×640 pixels, and adaptively fill the original samples during the scaling and filling process. After adaptive filling, the preprocessed image is sent to the backbone network for feature extraction and processing.

[0059] The backbone network consists of a series of convolutional layers and pooling layers, which are mainly used to extract feature information and output the results to the bottleneck layer. Spatial pyramid pooling SPPF uses cubic convolution and maximum pooling to expand the receptive field and improve nonlinear expression capabilities.

[0060] The bottleneck layer uses a superposition structure of a feature pyramid network (PAN) and a feature pyramid network (FPN) to achieve multi-dimensional prediction and output the results to the prediction box. The feature pyramid layer is responsible for strengthening the top-down semantic information transmission, and the feature layer path aggregation network structure further enhances the feature positioning information transmission capability.

[0061] In the prediction frame, the present invention uses three feature maps of different sizes to give prediction results for defect features of different sizes, and adds a weight to the weight through the weighted non-maximum suppression method to obtain a more accurate position.

[0062] In the deep learning detection model based on the prior frame, the number of anchor frames and their sizes have a great influence on the detection accuracy. If the number of anchor frames is too small or the size is too small, the feature information of the defects may be lost; if the number of anchor frames is too large or the size is too large, invalid information may be introduced and the complexity of the model may be increased. The original anchor frame size of the deep learning detection model is fixed. Since the defects in the chip defect dataset have many characteristics of extreme aspect ratios and the defects are very small, the initial anchor frame is difficult to adapt to the chip defect dataset of the present invention. Therefore, the present invention calculates the distance between all cluster center points to ensure that the initial cluster center points are selected as far away from each other as possible, thereby improving the clustering effect and obtaining suitable chip defects.

[0063] like Figure 6 As shown, optimizing the anchor frame specifically includes the following steps.

[0064] Step S411, collecting and organizing coordinate information of various defects in the chip defect data set as data points.

[0065] Step S412: randomly select a data point as the initial cluster center.

[0066] Step S413, calculate the distance between each initialized cluster center and the sample, and select the shortest distance d min .

[0067] Step S414, whether K cluster centers can be selected, if not, return to step S413; if yes, proceed to step S415.

[0068] Step S415: preferentially select the point with the largest distance as the new cluster center.

[0069] Step S416, calculate the probability of the next cluster center, and use the roulette method to find the next cluster center.

[0070] Step S417, determine whether the convergence or iteration times reaches n, if not, proceed to step S415; if yes, proceed to step S418, where n is an integer.

[0071] Step S418: output the clustering result.

[0072] Step S419, based on the clustering results, the number and size of the anchor boxes are optimized and adjusted to make them more suitable for various defect shapes and proportions in the chip defect dataset.

[0073] By optimizing the anchor frames, the present invention effectively solves the problem of mismatch between the size and number of anchor frames, improves the model's ability to detect defects of different shapes and proportions, and thus enhances the accuracy and reliability of chip defect detection. This improvement is not only applicable to chip defect datasets, but can also be extended to other datasets with special aspect ratios and shape features.

[0074] Step S42, pre-training the initial model.

[0075] More specifically, the deep learning detection model YOLOv5 constructed in step S41 is used as a pre-training model for training, and appropriate parameters such as batch size, training rounds, and optimizer need to be set during training.

[0076] The batch size refers to the number of samples used in each iteration during the deep learning detection model training process. In a specific embodiment, the batch size is selected to be 32, which achieves a good balance between training speed, hardware resource utilization, model convergence and stability, and detection accuracy and generalization ability.

[0077] Among them, the influence of the training round Epoch setting on chip defect detection is multifaceted. A reasonable training round Epoch setting needs to balance the training time, model convergence, detection accuracy and overfitting risk. In a specific embodiment, the training round Epoch is set to 150, which meets the convergence condition of model training.

[0078] In one embodiment, stochastic gradient descent is used as the optimizer.

[0079] When the deep learning detection model YOLOv5 constructed in the present invention is trained normally from the 1st round to the 50th round, the loss value decreases rapidly; the loss value decreases slowly between the 50th round and the 150th round; after 150 rounds, the model is in a convergence state; it can be seen that this model is effective in the chip defect recognition algorithm training, and no overfitting or underfitting occurs. The deep learning detection model YOLOv5 constructed in the present invention meets the basic chip defect detection effect.

[0080] Under the premise of ensuring the performance of the model, by reasonably setting the training parameters, the training efficiency can be improved and the resource consumption can be reduced, verifying the effectiveness and feasibility of the chip defect recognition algorithm based on the deep learning detection model constructed by the present invention.

[0081] Step S5, optimizing the deep learning detection model.

[0082] Furthermore, if Figure 7 As shown, step S5 includes the following steps.

[0083] Step S51, improve the backbone network, and combine the cross-layer feature maps of ResNet and CSP in the backbone network.

[0084] The backbone network is the most critical part of the convolutional neural network model. It is responsible for extracting the semantic features of different layers of the image. The quality of the backbone network design will directly affect the feature extraction capability of the detection model.

[0085] The backbone network of the deep learning detection model YOLOv5 constructed in step S4 includes Conv module, C3 module and SPPF module, which adopts the idea of ​​cross-stage link structure (CSP). Among them, the C3 module contains 3 standard convolutional layers and a BottleNeck; the input of the Conv module passes through a convolution module composed of convolution, BatchNorm, and Leaky activation function; the SPPF module inputs multiple MaxPools of different sizes in parallel, and then further fuses them, which can solve the multi-scale problem of detecting targets to a certain extent. The last model of the backbone network enters the bottleneck layer through the SPPF structure output.

[0086] ResNet aims to solve the problem of gradient explosion caused by feature loss and increased depth in the process of information transmission in traditional neural networks. ResNet continuously enhances the ability to express multi-scale features through residual learning, thus having great advantages in generalization. ResNet constructs residual hierarchical links in a single residual block.

[0087] The present invention combines the cross-layer feature map splicing of ResNet and CSP to effectively solve the gradient vanishing problem and prevent the gradient information duplication of multi-layer learning. In addition, this method not only increases the depth of the convolutional layer, but also enriches the nonlinear expression ability of deep features, thereby further improving the model's ability to autonomously learn features.

[0088] Step S52, embedding the attention mechanism module, embedding the attention mechanism module before the last SPPF module in the backbone network, and embedding the attention mechanism module between the bottleneck layer and the prediction box.

[0089] In the actual production process, the complex texture background of the chip will increase the difficulty of defect detection. Therefore, the present invention introduces an attention mechanism module to enable the model to accurately extract the defect features of the chip and reduce the impact of the complex background on defect detection.

[0090] like Figure 8 As shown in the figure, the attention mechanism module is embedded before the last SPPF module in the backbone network. Since the features are mainly extracted by the backbone network and the feature map here contains rich underlying semantic information, the attention mechanism can improve the detection accuracy and eliminate target interference. In addition, embedding the attention mechanism module between the bottleneck layer and the prediction box can effectively compensate for the loss of semantic information caused by downsampling, which is beneficial to the regression of the target box, reduces the loss of the regression box, and improves the detection accuracy of the model.

[0091] In order to avoid the situation where the introduction of the module causes the number of parameters to increase too much on the basis of the original algorithm and the operation speed to decrease, you can choose to add a lightweight attention mechanism module.

[0092] In one specific embodiment, the SimAM attention mechanism is selected.

[0093] The SimAM attention mechanism is superior to the current two-dimensional attention mechanism. It can directly calculate the three-dimensional attention weight, so that the model can better capture the characteristics of the target. At the same time, the theory of the SimAM attention mechanism is based on the theory of visual neuroscience, and the interaction between adjacent neurons has also been effectively applied to chip defect detection.

[0094] The SimAM attention mechanism searches for more important neurons through the linear relationship between different neurons. Assuming that the pixels in the channel can follow the same distribution and the target neuron removes the mean and variance, the minimum value of the neuron energy function can be obtained. , and its calculation formula is as follows: in t Target neurons ， is the calculated mean of neurons in the channel, To calculate the variance ,λ is a hyperparameter applicable to the chip data of the present invention. e t The smaller the value of, the t The more different it is from the adjacent neurons, the higher its weight. Finally, the Sigmoid function enhances the features after the neuron is derived. X As input features, 1 / e t The importance of neurons is weighted, and the result is is the weight of the neuron, so as to improve the ability to extract features, and the output results are as follows: .

[0095] The SimAM attention mechanism can directly evaluate the weight of each neuron, so that it can more accurately extract the required features and suppress useless features. In addition, because the SimAM attention mechanism focuses more on the extraction of chip defect feature information without adding additional model parameters, it is conducive to suppressing the interference of complex background on the network, thereby more efficiently completing the extraction and reconstruction of chip defect features, improving network classification accuracy, and reducing model complexity. Therefore, the SimAM attention mechanism is suitable for neural networks with more model parameters and can meet the requirements of low computing power for detection equipment.

[0096] Therefore, the present invention introduces a SimAM attention mechanism before the SPPF module and between the bottleneck layer and the prediction box to improve the feature extraction capability of the model.

[0097] Step S53, optimizing the calculation method of the loss function.

[0098] Loss function of the model of the present invention By the positioning loss function , classification loss function And the confidence loss function The composition is expressed as follows: , In the above formula, Nc is the number of detection layers, λ a , b , c represents the weight value of each loss function.

[0099] The loss function is an indicator used to evaluate the results of the current training round. Its loss value can be used to speed up the training of the model and adjust the weights. Therefore, the result of the loss function will directly affect the final detection effect of the model. However, in the process of chip defect learning, it is inevitable that there are some low-quality chip defect labeled samples, which will cause the model to misclassify, thereby reducing the generalization ability of the model.

[0100] The present invention replaces the original positioning loss function by constructing a dynamic non-monotonic focusing mechanism WIoU. By reducing the weight of the position information when the anchor frame and the target frame overlap, the interference of the wrong position information on the pre-training is reduced, and the generalization ability of the model is improved. The optimized positioning loss function The calculation formula is as follows: , In the above formula, x and y is the center point of the prediction box, x gt and y gt is the center point of the real frame, and are the height and width of the bounding box.

[0101] The optimized positioning loss function not only takes into account the intersection between the real box and the predicted box, but also dynamically adjusts the size and position of the predicted box to make the predicted box closer to the real box, thereby improving the detection accuracy.

[0102] In the chip defect recognition task, after the chip image is processed and extracted by the previous layers, the classifier will generate a large number of prediction frames, but only a few detection frames contain defective targets, and the remaining frames are of no help for learning. The loss function is obtained by calculating the error between the true frame and the predicted frame of the positive sample. In chip defect detection, the defective area accounts for a very small proportion of the entire chip image. During training, the model regards the area containing defects as positive samples and the defect-free chip area as negative samples. The number of defective positive samples is much lower than the number of negative samples in the normal area, so the loss value obtained by the loss function is mostly the loss value of useless samples, and the detection difficulty of positive and negative samples varies greatly.

[0103] Therefore, the present invention optimizes the classification loss function and introduces adjustable parameters to balance the weights of difficult and easy samples. The weights of difficult and easy samples are adjusted according to the distribution of samples within the dataset, so it has stronger generalization ability and can adapt to a wider range of datasets and tasks. The expression of the optimized classification loss function is as follows: In the formula, q represents the true label, p For the predicted results, α is the balancing factor for negative samples, γ Represents the modulation factor. By introducing adjustable parameters, the weights of difficult and easy samples are balanced, and the weights of difficult and easy samples are adjusted according to the distribution of samples within the dataset. Therefore, it has stronger generalization ability and can adapt to a wider range of datasets and tasks.

[0104] Step S54, adding a small target detection layer based on the SwinTransformer architecture at the 4-fold downsampling of the initial deep learning detection model YOLOv5.

[0105] The Swin Transformer framework consists of two parts, a decoder and an encoder. It was originally designed for the field of natural language processing, but the Resnet structure is also added to the Swin Transformer framework. Therefore, the present invention uses stacking multiple decoders to construct a convolutional network and adds it to the model constructed by the present invention.

[0106] In order to improve the recognition ability of small target defects, the present invention proposes to add the Swin Transformer overall framework to the original deep learning detection model YOLOv5. The workflow of Swin Transformer is as follows: Fig. 9 shown.

[0107] First, an RGB three-channel image of size H×W is input, and the input image is divided into 4×4 patches through the segmentation layer, and its feature dimension is H / 4×W / 4. In stage 1, the output dimension is changed to C through linear embedding, and then output to the Transformer Block in stage 2. The process from stage 2 to stage 4 is the same. In stage 2, 2×2 adjacent blocks are merged and spliced ​​through image block patches, and the high-dimensional features are reduced in dimension through convolution, and the output dimension becomes 2C; in stage 3, it is repeated 6 times, and the output after dimensionality reduction is 4C; in stage 4, it is repeated twice, and the output dimension becomes 8C.

[0108] In a convolutional network, the feature map size of the deep network is small, so the geometric information obtained is limited. The shallow network has more geometric information, but lacks semantic feature information. This may weaken the classification ability of the image, especially when detecting weak and small targets. When the model wants to obtain high-level features, it needs to downsample the image multiple times. The present invention combines the feature pyramid layer in the bottleneck layer with the feature layer path aggregation network structure to enhance the feature fusion between different scales, thereby improving the detection accuracy. However, the size of some defects in the chip is too small. In order to enhance the feature fusion between different scales and improve the recognition accuracy of small target defects, the present invention adopts a weighted bidirectional feature network (BidirectionalFeature Network) and uses weighted feature fusion and cross-scale connections (Cross-Scale Connections) to obtain global features of multiple levels of semantic information. The weighted bidirectional feature network (BiFPN) uses cross-scale connections to reduce the burden on the network. An attention mechanism is added to extract deeper feature information. Feature fusion is performed in both directions by upsampling and downsampling to improve the feature fusion effect between different layers, thereby improving the detection accuracy of chip defects of different scales.

[0109] Therefore, the present invention adds a small target detection layer based on the Swin Transformer architecture at the 4-fold downsampling of the original deep learning detection model YOLOv5. The downsampling ratio of the detection layer here is low, and it contains more small target feature information. In the bottleneck layer, the detailed representation of shallow features and the semantic features of high-level features are fully integrated through the CONCAT operation to fully extract information of features of different scales. When the input chip image is 640×640 pixels unchanged, the size of the feature map after four times downsampling is 160×160 pixels. The grid of this detection layer is tighter, and the defect feature information of the chip is more accurately divided, which further improves the ability to extract small pixel defects in chip images.

[0110] The improved model after adding the small target detection layer has a total of four detection layers, and the corresponding four feature sizes are 160×160 pixels, 80×80 pixels, 40×40 pixels, and 20×20 pixels.

[0111] Step S6: perform channel pruning and fine-tuning on the deep learning detection model.

[0112] As the number of network layers increases, the performance of the detection model will usually improve, but the increase in the number of model layers will lead to too many calculation parameters and longer inference time. This makes it difficult to deploy deep learning models in lightweight computing devices, so lightweight operation of the model becomes extremely important. The present invention also optimizes the model size and computing speed.

[0113] Since there are a large number of redundant parameters in the convolutional network model, pruning these redundant parameters not only has a small impact on the detection ability of the model, but also improves the generalization of the model to a certain extent. Therefore, the present invention adopts the channel pruning method to reduce the size of the detection model, which is conducive to real-time detection.

[0114] Sparsity enables lossless pruning at different levels. The Batch Normalization (BN) layer uses a regularization method to scale the scaling factor γ As a regularization coefficient, so that the scaling factor γ Gradually tends to zero. Scaling factor γ The value of is proportional to the importance of the channel, where Z in is the simplified input value of the BN layer, Z out is the output of the BN layer, Represents the channel coefficient, and the expression is as follows: , From the above formula, we can get the scaling factor γ The smaller it is, the less important the weight of the channel is, and these channels can be pruned. The loss during pruning is L The calculation formula is as follows:.

[0115] , In the above formula, x is the output during training, y is the training target, W is the convolution kernel, f () is convolution, l is the loss value of the network basic training, g () is sparse, is a constant, using the hyperparameters of g() for sparseness λ To balance the output values, the L1 norm is used to achieve the purpose of sparsification.

[0116] like Fig.10 As shown, step S6 further includes the following steps.

[0117] Step S61, performing sparse training on the model.

[0118] Step S62: according to the scaling factor γ Compare the lightweighting effect with the pruning ratio, and select the result with the best lightweighting effect as the pruning strategy.

[0119] More specifically, the scaling factor based on the BN layer γ The weight of the channel can be evaluated, the scaling factor γ The closer the value is to zero, the lower the weight of the channel is, and these channels can be pruned to compress the size of the model. When sparse training is not performed, the scaling factor γ The parameter value is kept around 1.0, and the scaling factor γ The distance from zero is far, which means that only a small number of channels can be pruned, and it is difficult to effectively reduce the number of parameters. -4 In the case of , the model sparse speed performs normally, and most of the scaling factors after sparse γ is close to 0.15, which indicates that these scaling factors γ The corresponding channels are of low importance, so they can be pruned.

[0120] As the pruning ratio increases, the weight of the model is significantly reduced and the inference time is greatly reduced, but the decrease in weight will lead to a decrease in the model detection accuracy.

[0121] In a specific embodiment, the pruning ratio can be 70%, which greatly reduces the total number of channels of the model while ensuring the minimum reduction in detection accuracy.

[0122] Step S63, fine-tune the pruned network.

[0123] Since the precise measurement accuracy of the detection model after pruning will fluctuate and decrease, and the generalization ability of the detection model is poor, the present invention fine-tunes the lightweight model to improve the model detection accuracy.

[0124] Furthermore, if Fig.11 As shown, step S63 includes the following steps.

[0125] Step S631, train a high-precision teacher model, and compare the logits output of the large model with the temperature T After division, SoftMax calculation is performed to obtain the soft target.

[0126] Step S632, train the student network and divide its logits output into two parts, one of which is directly used for SoftMax calculation, and the other part is divided by the same temperature and then subjected to SoftMax calculation to obtain the predicted value.

[0127] Step S633, compare the predicted value with the hard target, add the loss functions of the two parts to obtain a new loss function, and update the calculation parameters.

[0128] In a specific embodiment, the teacher model selects a large model with a similar network structure to the student model to perform knowledge distillation on a model with a channel pruning ratio of 70% to fine-tune the accuracy of the pruned model.

[0129] Step S64, based on the evaluation results of the validation set, the training is repeated continuously to obtain the optimal pruning result.

[0130] The present invention also provides a chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer.

[0131] Optionally, based on the Windows 10 operating system, PyQt5 can be used in the PyCharm development environment to design a chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer of the present invention.

[0132] The invention discloses a chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer, and applies the above-mentioned chip small target defect detection method based on the fusion improvement of YOLOv5 and Swin Transformer, comprising an image detection module, a video detection module, a camera detection module, a model initialization module, a model weight selection module, and a detection result display module; wherein the image detection module obtains chip images from a chip defect data set; the video detection module obtains chip videos from the chip defect data set; the camera detection module obtains chip images through a high-speed industrial camera; the model initialization module initializes the detection model; the model weight selection module selects a weight file of a model corresponding to different chip samples or different detection requirements; and the detection result display module displays the defect detection effect of the chip.

[0133] like Fig.12 As shown, the operation process of the chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer of the present invention includes the following steps.

[0134] Step P1, start; Step P2, logging in to a chip small target defect detection system based on the fusion improvement of YOLOv5 and Swin Transformer of the present invention; Step P3, select a picture or video to obtain a chip image; Step P4, select the model; Step P5, load cfg file; Step P6, detecting defects; Step P7, display the test results; Step P8, end.

[0135] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person having ordinary knowledge in the technical field may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the scope of the attached patent application.

Claims

1. An improved chip small target defect detection method based on the fusion of YOLOv5 and Swin Transformer, characterized in that: include, Step S1, acquiring images of different types of defects on the chip surface; Step S2, constructing a chip defect data set and dividing it into a training set, a test set, and a validation set; Step S3, preprocessing the collected chip defect data set; Step S4, constructing a deep learning detection model and performing pre-training; Step S5, optimizing the deep learning detection model; Step S5 The method comprises: step S54, adding a small target detection layer based on the Swin Transformer architecture at the 4-fold downsampling of the initial deep learning detection model YOLOv5; Step S6: perform channel pruning and fine-tuning on the deep learning detection model.

2. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 1 is characterized in that: The step S5 further comprises: Step S51, improve the backbone network, and combine the cross-layer feature maps of ResNet and CSP in the backbone network.

3. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 2 is characterized in that: The step S5 further comprises: Step S52, embedding the attention mechanism module, embedding the attention mechanism module before the last SPPF module in the backbone network, and embedding the attention mechanism module between the bottleneck layer and the prediction box.

4. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 3 is characterized in that: The step S5 also includes optimizing the calculation method of the loss function.

5. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 4 is characterized in that: In step S53, the optimized positioning loss function The calculation formula is as follows: ， Among them, x and y is the center point of the prediction box, x gt and y gt is the center point of the real frame, and are the height and width of the bounding box.

6. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 4 is characterized in that: In step S53, the expression of the optimized classification loss function is as follows: In the formula, q represents the true label, p For the predicted results, α is the equalization factor of negative samples, γ represents the modulation factor.

7. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 1, characterized in that: The step S3 comprises: Step S31, performing spatial transformation on the image of the chip defect data set; Step S32, performing data enhancement on the image of the chip defect data set; Step S33, performing sliding segmentation and cutting on the image of the chip defect data set; Step S34, stitching the images.

8. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 7, characterized in that: The step S33 comprises: Step S331, setting a sliding window of 640×640 pixels at a vertex of the original chip defect image, the vertex is the initialization position, and the X coordinate of the vertex is the starting X coordinate; Step S332, starting from the initialization position, slide the window along the X-axis direction, sliding 320 pixels each time until the entire X-axis is covered; Step S333, after the sliding operation in the X-axis direction is completed, the sliding window returns to the starting X coordinate and moves down 320 pixels along the Y-axis direction, and repeats the sliding operation in the X-axis direction; Step S334, continuously repeating the sliding operation in the X-axis and Y-axis directions until the entire chip image is divided into a plurality of sub-images of 640×640 pixels.

9. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 1, characterized in that: The step S4 comprises: Step S41, constructing an initial model and optimizing the anchor frame; Step S42, pre-training the initial model.

10. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 9, characterized in that: The step S41 includes: Step S411, collecting and arranging coordinate information of various defects in the chip defect data set as data points; Step S412, randomly selecting a data point as an initial cluster center; Step S413, calculate the distance between each initialized cluster center and the sample, and select the shortest distance d min ; Step S414, whether K cluster centers can be selected, if not, return to step S413; if yes, proceed to step S415; Step S415, preferentially selecting the point with the largest distance as the new cluster center; Step S416, calculating the probability of the next cluster center, and using the roulette method to find the next cluster center; Step S417, determine whether the convergence or iteration number reaches n, if not, go to step S415; if yes, go to step S418; Step S418, outputting the clustering result; Step S419, based on the clustering results, the number and size of the anchor boxes are optimized and adjusted to make them more suitable for various defect shapes and proportions in the chip defect dataset.

11. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 1, characterized in that: The step S6 comprises: Step S61, performing sparse training on the model; Step S62: according to the scaling factor γ Compare the lightweighting effect with the pruning ratio, and select the one with the best lightweighting effect as the pruning strategy; Step S63, fine-tuning the pruning network; Step S64, based on the evaluation results of the validation set, the training is repeated continuously to obtain the optimal pruning result.

12. The chip small target defect detection method based on the improved fusion of YOLOv5 and Swin Transformer as claimed in claim 11, characterized in that: The step S63 includes: Step S631, train a high-precision teacher model, and compare the logits output of the large model with the temperature T After the division, the soft target is obtained by performing SoftMax calculation; Step S632, train the student network and divide its logits output into two parts, one of which is directly used for SoftMax calculation, and the other is divided by the same temperature and then subjected to SoftMax calculation to obtain the predicted value; Step S633, compare the predicted value with the hard index, add the loss functions of the two parts to obtain a new loss function, and update the calculation parameters.

13. The chip small target defect detection system based on the fusion of YOLOv5 and Swin Transformer is characterized by: An improved chip small target defect detection method based on the fusion of YOLOv5 and Swin Transformer as claimed in any one of claims 1 to 12 is applied, comprising an image detection module, a video detection module, a camera detection module, a model initialization module, a model weight selection module, and a detection result display module; the image detection module obtains chip images from a chip defect data set; the video detection module obtains chip videos from a chip defect data set; the camera detection module obtains chip images through a high-speed industrial camera; the model initialization module initializes the detection model; the model weight selection module selects a weight file of a corresponding model according to different chip samples or different detection requirements; and the detection result display module displays the defect detection effect of the chip.

Images