Chip appearance defect automatic detection method, electronic device and storage medium

Abstract

The application discloses a kind of chip appearance defect automatic detection method, electronic equipment and storage medium, comprising: chip image is collected using optical camera, and using Retinex light balance and median filter is carried out image pre-processing;Chip detection module based on single-stage anchor-free neural network structure is constructed, and chip area is extracted from the image after pre-processing;Pin detection module based on multi-scale single-stage neural network structure is constructed, and pin is extracted from chip area;Pin abnormality classification module based on neural network structure is constructed, and pin abnormality situation is classified;Encapsulation surface anomaly detection module based on multi-scale single-stage neural network structure is constructed to detect the anomaly of encapsulation surface;Based on the data set of pre-processed data, each model is trained.The trained model is used to detect the chip appearance defect of input new image.The application solves the defects of low efficiency, low robustness, high false detection rate and the like of artificial and traditional automatic detection mode.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of pattern recognition, and particularly relates to a chip appearance defect automatic detection method, an electronic device and a storage medium. BACKGROUND

[0002] Integrated circuit (IC) chips are the core devices of the information industry, and are of great significance. However, due to the precision and complexity of the manufacturing process of IC chips, various defects are inevitably generated, which directly affect the normal use of the chips. In order to improve the reliability of the chips, defect detection needs to be performed on the chips in the quality control link. Chip defect detection includes internal defect detection and appearance defect detection. The former can be detected by X-ray, and the latter is often realized by an optical camera.

[0003] Traditional chip appearance defect detection is mainly realized by manual methods, which need to train the testers at high cost, and the actual operation is seriously dependent on the experience of the testers, and cannot objectively make consistent and stable judgments. In addition, long-term work under strong light can easily cause fatigue and even serious eye damage. Therefore, the manual appearance defect detection method cannot guarantee the detection accuracy, and the efficiency is relatively low. With the innovation of IC chip design and packaging technology, IC chips tend to be smaller in size, lighter in weight, and have more pins and smaller pin spacing. This trend has increased the difficulty of chip appearance defect detection. Therefore, the traditional manual detection method is more difficult to meet the actual needs. Therefore, the automatic chip appearance defect detection technology based on computer vision has gradually replaced the manual appearance defect detection method.

[0004] At present, the automatic chip appearance defect detection technology based on computer vision often uses classical image preprocessing and pattern recognition methods, such as manual selection and design of features and operators. However, the classical preprocessing method has great limitations, and usually has high requirements for the quality of the image, and cannot handle larger external interference factors such as light intensity, shooting angle, etc., resulting in a serious decline in detection performance in the actual environment. In addition, the image features obtained by traditional pattern recognition methods do not have strong robustness and universality, and the algorithm is difficult to apply to the detection of special defects, and has very poor generalization ability on different data sets, so it has great limitations.

[0005] In recent years, with the emergence of artificial intelligence theories and technologies such as deep learning, various computer vision problems have developed rapidly and achieved remarkable results superior to traditional manually designed operators and features, such as image classification, object detection and other problems. However, so far, deep learning has rarely been used in the research and application of chip appearance defect detection. The few chip appearance defect detection networks based on deep learning are not optimized enough, resulting in low accuracy. Therefore, it is necessary to design and implement an automatic and high-performance chip appearance defect detection technology based on the latest theories and tools such as deep neural networks. SUMMARY

[0006] In order to realize automatic and high-performance chip appearance defect detection, the present application provides a chip appearance defect detection method, an electronic device and a storage medium. The technical problem to be solved by the present application is solved by the following technical scheme:

[0007] The implementation process of the present application is as follows: a plurality of target detection models and an image classification model are constructed, the system is constructed in a mixed way of series and parallel connection, and each model is fine-tuned based on transfer learning to obtain a trained model, thereby performing chip appearance automatic defect detection work.

[0008] Further, the chip appearance defect detection method comprises the following steps:

[0009] Step one, construct an image acquisition and preprocessing module, use an optical camera to acquire an image containing a chip, and perform contrast enhancement and noise reduction preprocessing on the chip image to obtain an enhanced chip image;

[0010] Step two, construct a chip detection module based on a single-stage, anchor-free deep neural network structure (denoted as YOLO-Chip), and correspondingly construct an IoU perception classification loss function, input the preprocessed enhanced image, use YOLO-Chip to accurately locate and extract the chip region in the image to obtain the chip region;

[0011] Step three, construct a pin detection module based on a multi-scale, single-stage deep neural network structure (denoted as SSD-Pin), and correspondingly construct a bounding box regression loss function (SIoU), use SSD-Pin to detect and extract the chip pins in the chip region, and obtain each pin in the chip region, so that each chip image obtains a set of pin images;

[0012] Step four, construct a pin abnormality classification module based on a deep neural network structure (denoted as CNN-Pin), use CNN-Pin to classify the abnormality of all pins corresponding to each chip image to complete pin abnormality detection;

[0013] Step five, a multi-scale, single-stage deep neural network structure based package surface anomaly detection module (denoted as SSD-Surface) is constructed, the chip region detected in step two is taken as input, and the anomaly positions of the package surface are detected by using the SSD-Surface, so that the chip surface anomaly positions are positioned and classified;

[0014] Step six, a data set making and model training module is constructed, the data sets required for training of various models are constructed based on the preprocessed enhanced images, for example, a chip detection data set, a chip pin detection data set, a chip pin classification data set and a package surface anomaly detection data set. And the fine-tuning training of the chip detection module, the pin detection module, the pin anomaly classification module and the package surface anomaly detection module is completed accordingly.

[0015] Step seven, the trained chip detection model, the pin detection model, the pin anomaly classification model and the package surface anomaly detection model constitute a complete chip appearance defect detection system, and the chip appearance defect detection system is used for testing the input new chip image, so as to realize the chip appearance defect detection function.

[0016] Further, the image acquisition and preprocessing module in step one uses an optical camera to acquire an image containing a chip, and uses Retinex light balance and median filtering to preprocess the image. Specifically, the steps include: acquiring an RGB color space image containing a chip by a conventional optical camera, estimating the reflection classification component of the image using a single-scale Retinex algorithm, completing the Retinex light balance operation to obtain a contrast-enhanced chip image. On the basis of the contrast-enhanced chip image, the image detail protection and denoising are completed by the median filtering algorithm, and the preprocessed image with better contrast and lower noise is obtained.

[0017] Further, the chip detection module YOLO-Chip based on a single-stage, anchor-free deep neural network structure in step two, the corresponding neural network model specifically includes: a feature extraction backbone layer, a multi-scale feature fusion layer and an IoU perception detection head;

[0018] The feature extraction backbone layer is based on CSPDarkNet, which is used for preliminary feature extraction of the input image and generates three-level multi-scale feature maps to be fused;

[0019] The multi-scale feature fusion layer is based on PAFPN, and the three-level multi-scale feature maps are taken as input to perform path aggregation feature fusion, so as to obtain a feature map with scale robustness;

[0020] The IoU-aware detection head takes the three-level feature fusion result as input, decouples the foreground classification and the bounding box regression task, and uses the IoU-aware classification loss function to complete the training of the classification and regression branches of the task alignment, and during the test, the anchor point is used to predict the classification and regression task.

[0021] The post-processing module is used to decode the anchor-free prediction result, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip region detection prediction result.

[0022] Further, the pin detection module SSD-Pin based on the multi-scale single-stage deep neural network structure in step three specifically includes a feature extraction backbone layer, a multi-scale feature prediction head, and a post-processing module.

[0023] The feature extraction backbone layer is based on VGG16 and is used to extract 6 layers of feature maps of different scales.

[0024] The multi-scale feature prediction head takes 6 layers of feature maps of different scales as input, uses the SIoU loss (SCYLLA-IoU) to complete the learning of the high-precision bounding box regression task, and during the test, the anchor box is used to perform encoding prediction of classification and regression.

[0025] The post-processing module is used to decode the anchor-based prediction result, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip pin detection result.

[0026] Further, the pin anomaly classification module CNN-Pin based on the deep neural network structure in step four specifically includes a convolutional layer, a down-sampling layer, a fully connected layer, and a classification layer.

[0027] The convolutional layer is based on 3x3 and 1x1 convolution stacking to obtain a plurality of convolutional blocks, which are used to extract edge, detail, texture, and other features related to pin anomalies of the chip pin image.

[0028] The down-sampling layer is based on the max-pooling operation to reduce the size of the feature map, reduce the amount of subsequent calculation, and prevent overfitting caused by information redundancy.

[0029] The fully connected layer expands the final two-dimensional feature map into one dimension and performs three fully connected operations. It is used to integrate the global features of the chip pin image for subsequent classification.

[0030] The classification layer uses the softmax activation on the final one-dimensional feature vector output by the fully connected layer to generate the final pin anomaly multi-classification probability.

[0031] Further, the package surface anomaly detection module SSD-Surface based on the multi-scale single-stage deep neural network structure in the fifth step corresponds to a neural network model specifically including a feature extraction backbone layer, a multi-scale feature prediction head, and a post-processing module.

[0032] The feature extraction backbone layer is constructed based on VGG16 and is used to extract 6 layers of feature maps of different scales.

[0033] The multi-scale feature prediction head takes 6 layers of feature maps of different scales as input respectively and encodes and predicts based on anchor boxes.

[0034] The post-processing module decodes the prediction results based on anchor boxes, performs non-maximum suppression, discards low-quality prediction values, and generates the final package surface anomaly detection results.

[0035] Further, the model training module in the sixth step specifically includes training dataset construction and transfer learning fine-tuning of each model.

[0036] The training dataset construction is used to construct the dataset required for training each detection and classification model, wherein the detection dataset required for the YOLO-Chip chip detection model, the SSD-Pin pin detection model, and the SSD-Surface package surface anomaly detection model is labeled with horizontal rectangular boxes. The classification dataset required for the CNN-Pin pin anomaly classification model is labeled with simple classification storage.

[0037] The transfer learning fine-tuning, wherein the YOLO-Chip chip detection model, the SSD-Pin pin detection model, and the SSD-Surface package surface anomaly detection model are fine-tuned with the pre-trained model trained based on the COCO2017 target detection dataset. The CNN-Pin pin anomaly classification model is fine-tuned with the pre-trained model trained based on the large-scale image classification dataset of ImageNet.

[0038] Another object of the present application is to provide an electronic device comprising an optical image acquisition device, a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to cause the processor to perform the following steps: controlling the optical image acquisition device to acquire chip image data, and performing Retinex light balance and median filter denoising preprocessing on the image data; constructing a chip detection module based on a single-stage, anchor-free deep neural network structure to accurately locate the chip in the image; constructing a pin detection module based on a multi-scale single-stage deep neural network structure to detect the pins of the chip; constructing a pin anomaly classification module based on a deep neural network structure to classify the chip pin anomaly situations; constructing a packaging surface anomaly detection module based on a multi-scale single-stage deep neural network structure to locate and classify the chip surface abnormal parts; and constructing a model training module to construct chip detection data sets, pin detection data sets, pin classification data sets, and packaging surface anomaly data sets required for training of each model based on the preprocessed enhanced image, and to complete fine-tuning training of each model.

[0039] Another object of the present application is to provide a computer-readable storage medium storing a computer program, the computer program being executed by a processor to cause the processor to perform the following steps: performing Retinex light balance and median filter denoising preprocessing on image data; constructing a chip detection module based on a single-stage, anchor-free deep neural network structure to accurately locate the chip in the image; constructing a pin detection module based on a multi-scale single-stage deep neural network structure to detect the pins of the chip; constructing a pin anomaly classification module based on a deep neural network structure to classify the chip pin anomaly situations; constructing a packaging surface anomaly detection module based on a multi-scale single-stage deep neural network structure to locate and classify the chip surface abnormal parts; and constructing a model training module to construct chip detection data sets, pin detection data sets, pin classification data sets, and packaging surface anomaly data sets required for training of each model based on the preprocessed enhanced image, and to complete fine-tuning training of each model.

[0040] In combination with all the above technical solutions, the present application has the following advantages and positive effects:

[0041] First, in view of the technical problems and difficulties existing in the above-mentioned existing chip appearance defect detection method, the present application proposes a complete automatic chip appearance defect detection technology based on deep learning, which can solve the problems of poor robustness, complicated steps, slow response, low intelligence, low efficiency and other shortcomings commonly existing in current chip appearance defect detection technology. The specific description is as follows:

[0042] (1) Traditional chip appearance defect inspection is mainly carried out manually. This method consumes a lot of manpower and is costly. It also relies heavily on the experience of the inspectors and cannot make objective, consistent and stable judgments. In addition, manual methods are prone to fatigue and even serious eye damage. At the same time, manual appearance defect inspection cannot guarantee the accuracy of the inspection and is relatively inefficient.

[0043] (2) While current automated chip appearance defect detection methods using computer vision have overcome the drawbacks of manual methods, most of these methods rely on traditional image processing techniques, requiring manual design of operators and feature extractors, feature classification, etc., to achieve defect detection and classification tasks. Traditional operators' feature extractors have low robustness, thus placing stringent requirements on the image acquisition environment, or requiring redesign of operators and feature extractors for different images. In contrast, the deep learning-based defect detection method used in this invention greatly improves the performance and stability of the model. After the network model is trained, users only need to upload the chip image to be detected to complete the chip appearance defect detection task with one click, without any other additional complex operation steps.

[0044] Second, the technical solution protected by this invention introduces a series of core technologies and theories, such as Retinex-based image enhancement theory, deep neural networks, single-stage object detection networks, loss construction for sub-tasks in object detection tasks, and model training modules based on transfer learning, enabling the corresponding method of this invention to possess technical advantages such as high performance, noise resistance, and high robustness, as described in detail below:

[0045] The chip appearance defect detection method provided by this invention constructs an image preprocessing module based on single-scale Retinex illumination equalization and median filtering, which improves the system's ability to process low-quality chip images; it also constructs a chip detection module based on a single-stage, anchor-free deep neural network structure, which enables the system to have scale invariance for chips of different sizes; based on the theories and tools of single-stage target detection and CNN-based image classification, this invention constructs a chip pin anomaly detection module, which has higher accuracy for small-sized pins compared to traditional manual operators; the embodiments of this invention provide new theoretical and methodological support for how to use deep learning for chip appearance defect detection, making deep learning-based chip appearance defect detection technology more practical, reliable, and widespread.

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0047] Figure 1 This is a flowchart of the chip appearance defect detection method provided in the embodiments of the present invention.

[0048] Figure 2is the overall architecture diagram of the chip appearance defect detection method provided by the embodiment of the application.

[0049] Figure 3 is the schematic diagram of the single-scale Retinex light balance algorithm provided by the embodiment of the application.

[0050] Figure 4 is the operation result display diagram of the Retinex light balance algorithm provided by the embodiment of the application, wherein (a) is before Retinex contrast enhancement, and (b) is after Retinex contrast enhancement.

[0051] Figure 5 is the operation result display diagram of the median filtering algorithm provided by the embodiment of the application, wherein (a) is before median filtering, and (b) is after median filtering.

[0052] Figure 6 is the YOLO-Chip chip detection network model structure diagram provided by the embodiment of the application.

[0053] Figure 7 is the YOLO-Chip chip detection result display diagram provided by the embodiment of the application.

[0054] Figure 8 is the SSD-Pin pin detection network model structure diagram provided by the embodiment of the application.

[0055] Figure 9 is the pin abnormality classification result display diagram provided by the embodiment of the application, wherein (a) is normal, (b) is bent, (c) is incomplete, and (d) is upwarping.

[0056] Figure 10 is the packaging mark abnormality detection result display diagram provided by the embodiment of the application, wherein (a) is scratch, (b) is hole, and (c) is stain.

[0057] Figure 11 is the schematic diagram of the electronic device provided by the embodiment of the application.

[0058] Figure 12 is the schematic diagram of the readable storage medium provided by the embodiment of the application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical scheme and advantages of the application clearer, the application will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and are not used to limit the application.

[0060] In view of the problems in the prior art, the chip appearance defect detection method, system, electronic equipment and storage medium are provided, and the chip appearance defect detection method provided by the present application is described in detail below with reference to the drawings.

[0061] As shown in Figure 1 The chip appearance defect detection method provided by the present application comprises the following steps:

[0062] S101, an optical camera is used to collect an image containing a chip, and Retinex light balance and median filtering methods are used to perform contrast enhancement and noise reduction preprocessing on the chip image;

[0063] S102, a chip detection module YOLO-Chip based on a single-stage and anchor-free deep neural network structure is constructed to accurately locate and extract the chip in the image;

[0064] S103, a pin detection module SSD-Pin based on a multi-scale and single-stage deep neural network structure is constructed to extract the pin image of the chip;

[0065] S104, a pin anomaly classification module CNN-Pin based on a deep neural network structure is constructed to complete the classification of the abnormal situation of the pin;

[0066] S105, a packaging surface anomaly detection module SSD-Surface based on a multi-scale and single-stage deep neural network structure is constructed to complete the anomaly detection of the packaging surface.

[0067] S106, a data set is constructed based on the preprocessed data to complete the fine-tuning training of each model.

[0068] The chip appearance defect detection method provided by the present application can also be implemented by other steps by those skilled in the art, Figure 1 The chip appearance defect detection method provided by the present application is only one specific embodiment.

[0069] The technical solutions of the present application are further described below in combination with specific embodiments.

[0070] Embodiment 1

[0071] Current manual methods for chip appearance defect detection suffer from fatal flaws such as low efficiency, error-proneness, and poor reliability. Automated methods based on traditional machine vision also suffer from numerous weaknesses, including complex procedures, slow speed, poor robustness, low generalization ability, and high false detection rates. This project investigates how to use deep learning techniques, such as deep neural networks, to achieve efficient, fast, and accurate automated chip appearance defect detection technology. Specific content includes: designing preprocessing algorithms for raw chip images, such as illumination equalization and image noise reduction; designing neural networks to detect and locate chip regions; designing neural networks to perform pin detection operations and determine the presence of pin anomalies, such as pin breakage, bending, or tilting; and designing neural networks to detect chip surface defects, such as scratches, stains, and damage.

[0072] This invention provides a method for detecting chip appearance defects. Figure 2 This is a diagram illustrating the overall architecture of the chip appearance defect detection method provided in this embodiment of the invention. The method includes the following five steps:

[0073] Step 1: Construct an image acquisition and preprocessing module. Use an optical camera to acquire images containing the chip, and use Retinex illumination equalization and median filtering for noise reduction to preprocess the images and obtain the preprocessed and enhanced images.

[0074] Specifically, a single-scale Retinex (SSR) algorithm is used to achieve illumination equalization. For example... Figure 3 As shown, the Retinex model can be expressed by the formula S(x,y)=R(x,y)*L(x,y), where the original image is S(x,y), the reflection component is R(x,y), and the incident ray component is L(x,y). Applying a log operation converts the multiplication into addition, resulting in the SSR algorithm formula as follows:

[0075] r(x,y)=logR(x,y)

[0076] =logS(x,y)-logL(x,y)

[0077] =logS(x,y)-logS(x,y)*G(x,y)

[0078] Where G(x,y) is a Gaussian function with a scale parameter of σ, i.e. r(x,y) is the logarithm of the reflection component R(x,y). After obtaining r(x,y), r(x,y) is transformed from the logarithmic field to the real number field R(x,y).

[0079] Then the image after light balance is removed by noise using median filtering, since median filtering will not produce edge blur phenomenon, and the operation is relatively simple, therefore, the effect of median filtering is better than that of mean filtering and Gaussian filtering. Figure 4 and Figure 5 The actual operation effects of Retinex light balance algorithm and median filtering algorithm are respectively shown.

[0080] Step two, a chip detection module YOLO-Chip based on a single-stage and anchor-free deep neural network structure, takes the preprocessed enhanced image as input, uses the YOLO-Chip model to accurately locate and extract the chip region in the image, so as to obtain the image corresponding to the chip region of interest;

[0081] Specifically, the embodiment of the application constructs a chip detection model YOLO-Chip based on a single-stage target detection model YOLOX-S, as shown in the figure, the module contains a feature extraction backbone network CSPDarkNet, a multi-scale feature fusion network PAFPN, an IoU perception detection head based on an anchor-free paradigm, and a post-processing part. Figure 6

[0082] In the feature extraction backbone network, DarkNet is a target detection backbone that balances speed and accuracy, and after introducing the CSPN (Cross Stage Partial Network) structure, the model optimizes the repeated gradient information, greatly reduces the parameter quantity and calculation quantity, and does not reduce the accuracy of the model. Taking a 640*640 size RGB image as input, continuously performing convolution and downsampling operations on the CSPN block to reduce the feature map size and extract higher level abstract features. After 8, 16 and 32 times of downsampling, C1:80*80, C2:40*40 and C3:20*20 scale feature maps are generated.

[0083] In the multi-scale feature fusion network PAFPN (Path Aggregation Feature Pyramid Network), C1, C2 and C3 are taken as input, the fusion of detailed features and high-level semantic features is enhanced by top-down path aggregation to generate three feature maps F1, F2 and F3.

[0084] ​In the structure of the IoU-aware detection head based on the anchor-free paradigm, the foreground-background classification and the bounding box regression task are designed as parallel convolution branches, which reduces the competition of the two tasks for the network learning ability, while greatly enhancing the independence of optimization during training of different tasks, and improving the convergence speed during training. Specifically, two convolution branches are introduced on each output feature map of the PAFPN. If the size of the current feature map is HxW, the output dimension of the foreground-background classification branch is HxWx1, and the output dimension of the bounding box regression branch is HxWx4, representing the distance from the four corners of the chip bounding box to the feature point.

[0085] In addition, VariFocalLoss (VFL) is constructed to train the classification branch, and the alignment of the predictions of the two branches is enhanced. Each sample of the IoU-aware classification loss function uses VFL, and the VFL is defined as follows:

[0086]

[0087] where p is the IoU-aware classification score (IACS) of the classification branch, and q is the true value of the IACS. For the anchor points assigned to the foreground target, q is set to the IoU between the predicted box of the regression branch of the anchor point and its true box, and for the anchor points not assigned to the foreground target, q is set to 0. γ is the scaling factor of VFL. The loss of positive samples is weighted using q, so that the focus of training is placed on the positive anchor points with high-quality predicted boxes, avoiding the introduction of too many outlier gradients. Alpha is the balance coefficient between positive sample loss and negative sample loss, which plays a role in balancing the positive sample loss and the negative sample loss. Finally, the classification loss L cls can be written as:

[0088]

[0089] where p k,i and q k,i represent the IACS of the i-th anchor point and the true value of the IACS, respectively; N pos represents the total number of positive samples, which is obtained by the simOTA positive and negative sample assignment algorithm.

[0090] The CIoU loss is used to train the bounding box regression branch, which comprehensively considers the center distance, diagonal distance of the minimum closed region, and aspect ratio of the two bounding boxes, providing a better gradient direction. Let the predicted box and the true box be B and B gt respectively, then the IoU calculation formula of the two is as follows:

[0091]

[0092] Let v be the measure of aspect ratio similarity, which can be written as:

[0093]

[0094] where w, h, w gt , h gt represent the width and height of the predicted and ground truth bounding boxes, respectively.

[0095] Let a be the trade-off coefficient of v, which is defined as:

[0096]

[0097] Let p represent the distance between the centers of two bounding boxes, and l be the diagonal distance of the minimum enclosing box. Then the final CIoU loss is:

[0098]

[0099] Similar to the classification loss, the final bounding box regression loss can be written as:

[0100]

[0101] where q k,i represents the IACS ground truth value at the anchor point, to weight the loss value of positive samples.

[0102] The final total loss is defined as follows:

[0103] L sum = L cls + b reg L reg

[0104] where L cls , L reg represent the classification loss and bounding box regression loss, respectively, and b reg represents the balance parameter of the bounding box regression loss.

[0105] Other techniques during training include cosine annealing learning rate decay, multi-scale input data, Mosaic and MixUp strong data augmentation, and using the exponential moving average (EMA) rule to update the learnable parameters. The detailed parameter design of the chip detection network is shown in Table 1:

[0106] Table 1 Training settings of the chip detection model (YOLO-Chip)

[0107]

[0108]

[0109] In the post-processing part, the prediction result based on anchor-free needs to be decoded and non-maximum suppression is performed to obtain the final prediction result at inference time. Figure 7 is the YOLO-Chip chip detection result display diagram provided by the embodiment of the application.

[0110] Step three, a pin detection module SSD-Pin based on a multi-scale single-stage deep neural network structure is constructed, and the pin detection module SSD-Pin is used to detect pins in a plurality of chip images, so that each chip image obtains a set of pin images;

[0111] Specifically, the SSD-Pin model adopts an anchor-based single-stage target detection structure, and combines prediction information from feature maps of a plurality of different resolutions to process the detection requirements of targets of different sizes. The SSD-Pin model is simpler and more direct than the RPN method, and the model does not need to perform region extraction or subsequent pixel or feature resampling, and encapsulates all calculations in one network. This makes the SSD easy to train and can be directly integrated into a system that needs to detect components.

[0112] The SSD-Pin model takes VGG16 as a model backbone, and adds a convolutional layer to obtain more feature maps on the basis of VGG16. Specifically, the full connection layers FC6 and FC7 of VGG16 are converted into a 3×3 convolutional layer conv6 and a 1×1 convolutional layer conv7, then the dropout layer and the fc8 layer are removed, and a series of convolutional layers are added. Figure 8 is the SSD-Pin pin detection network structure diagram provided by the embodiment of the application.

[0113] The loss function of the SSD-Pin includes a foreground-background classification loss and a bounding box regression loss. The total loss function is defined as:

[0114]

[0115] wherein, L conf represents the foreground-background classification loss, L loc represents the bounding box regression loss, and α is a balance coefficient; l represents a prediction box, g represents a real box; N represents the number of anchor boxes assigned as positive samples, and if N is 0, the total loss is set to 0.

[0116] The classification loss L conf uses a softmax binary classification loss:

[0117]

[0118] wherein, represents the probability value of the i th prediction box being judged as the k th class; Ind is an indicator function, represents whether the i-th prediction box matches the j-th ground truth box as the k-th class, represents the i-th prediction box is a positive sample, otherwise is a negative sample.

[0119] SSD as an anchor-based algorithm, its training target is to regress to the offset of the relative anchor center and its width and height of the ground truth box. The bounding box regression loss L loc is defined as:

[0120]

[0121] The constructed SIoU (SCYLLA-IoU) based bounding box regression loss, characterized in that the SIoU loss comprehensively considers the intersection over union, center point position, width-height ratio (shape) and angle difference between the ground truth box and the prediction box, and constrains the degree of freedom of the optimal gradient direction. Let respectively represent the center point coordinates of the ground truth box and the prediction box, then the SIoU loss is defined as:

[0122]

[0123] wherein, When the diagonal lines of the two bounding boxes are aligned with the x-axis or y-axis, Λ = 0. When the diagonal line is at 45° with the x-axis, Λ = 1.

[0124] The SIoU loss is defined as:

[0125]

[0126] wherein, γ = 2-Λ, R h ,R w respectively represent the height and width of the minimum closed frame formed by the prediction box and the ground truth box. This penalty term describes the distance between the center points, and its penalty cost is positively correlated with the angle cost. When α→0, the distance difference penalty is greatly reduced. On the contrary, the closer α is to The greater the distance difference penalty contribution is.

[0127] The SIoU loss is defined as:

[0128]

[0129] wherein, θ is the weight adjustment parameter of this term, w, h represent the width and height of the bounding box.

[0130] Finally, the total bounding box regression loss L SIoU is defined as:

[0131]

[0132] wherein, B, B GT respectively represent the prediction frame and the real frame.

[0133] Step four, a pin abnormality classification module CNN-Pin based on a deep neural network structure is constructed, a 20-layer convolutional neural network (CNN-Pin) is used to perform abnormality classification on the images in the pin image set of each chip image, so as to complete pin abnormality situation detection;

[0134] The CNN-Pin is composed of a convolution block, a pooling layer and a full connection layer, the convolution block uses multiple small convolution kernels (3x3) to replace a convolution layer with a large convolution kernel, which can reduce the parameters on the one hand and perform more nonlinear mapping on the other hand, thereby increasing the fitting / expression ability of the network.

[0135] The convolution layer is all 3x3 convolution kernel with a step of 1, and the size of the input and output is filled with a fixed size. The kernel size of the pooling layer is 2x2. The detailed structure design of the CNN-Pin model is shown in Table 2. Finally, a 4-class output is obtained, representing the classification probability of normal, bending, missing and upwarping pins. A softmax multi-classification loss function is constructed. The CNN-Pin detection result is shown in Figure 9 .

[0136] Table 2 Detailed parameter design of pin abnormality classification model (CNN-Pin)

[0137]

[0138] Step five, a packaging surface abnormality detection module SSD-Surface based on a multi-scale single-stage deep neural network structure is constructed, a plurality of chip images are used as input, and the SSD-Surface model is used to detect the packaging surface abnormality position;

[0139] Specifically, the feature extraction backbone layer is constructed based on VGG16, which is used to extract 6 layers of feature maps of different scales; the multi-scale feature prediction head takes 6 layers of feature maps as input respectively, and performs encoding prediction based on anchor boxes; the post-processing module is used to decode the prediction results based on anchor boxes, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip packaging surface abnormality result. Figure 10 is a packaging surface abnormality detection result display diagram provided by the embodiment of the present application.

[0140] Step six, the model training module is constructed, and the chip detection dataset, the pin detection dataset, the pin classification dataset and the packaging surface anomaly detection dataset required by each model are constructed based on the pre-processed enhanced image. The detection dataset required by the YOLO-Chip chip detection model uses a horizontal rectangular frame as a single-class target label for a chip, and is converted into the format of a COCO2017 dataset for storage. The detection dataset required by the SSD-Pin pin detection model uses a horizontal rectangular frame as a single-class target label for a pin, and is converted into the format of a COCO2017 dataset for storage. The classification dataset required by the CNN-Pin pin anomaly classification model is stored by simply classifying 4 for labeling. The detection dataset required by the SSD-Surface packaging surface anomaly detection model uses a horizontal rectangular frame as a single-class target label for an anomaly area, and is converted into the format of a COCO2017 dataset for storage.

[0141] The transfer learning fine-tuning, wherein the YOLO-Chip chip detection model, the SSD-Pin pin detection model and the SSD-Surface packaging surface anomaly detection model are fine-tuned with a pre-trained model trained based on a COCO2017 target detection dataset. The CNN-Pin pin anomaly classification model is fine-tuned with a pre-trained model trained based on a large-scale image classification dataset of ImageNet. The parameters obtained by transfer are not frozen during training, so that the model has higher modeling freedom.

[0142] Embodiment 2:

[0143] Based on the above embodiment 1, please refer to Figure 11 , Figure 11 is a schematic diagram of a chip appearance defect detection electronic device provided by the embodiment of the present application. The chip appearance defect detection device provided by the embodiment of the present application, the electronic device comprises a chip to be defect detected 1, an optical image acquisition device 2, a display 3, a host (containing a processor CPU and a GPU) 4, a read-only memory (ROM) 5, a router 6, a power supply 7 and an external hard disk 8.

[0144] Among them, the host 4 and the optical image acquisition device 2, the display 3, the read-only memory 5, the router 6, the external hard disk 8 and the like complete the communication among each other through the communication interface; the optical image acquisition device 2 is used for acquiring chip image data; the display 3 is used for displaying the chip image data and the appearance defect detection result thereof; the read-only memory 5 is used for storing computer program codes; the router 6 is used for the networking function of the electronic device and external devices; the external hard disk 8 is used for storing the registration result; the power supply 7 is used for the power supply of the electronic device; the host 4 is used for executing the computer program stored on the memory, and the computer program is executed by the processor to realize the following steps:

[0145] Step one, open the optical image acquisition device to collect chip image data and use the electronic device to preprocess the image.

[0146] Specifically, step one of the embodiment of the application uses an optical image acquisition device to collect chip image data; and the collected image data is preprocessed using Retinex light balance and median filter denoising;

[0147] Step two, a chip detection module YOLO-Chip model based on a single-stage, anchor-free deep neural network structure is constructed to accurately locate and extract the chip in the preprocessed image;

[0148] Specifically, step two of the embodiment of the application uses CSPDarkNet to preliminarily extract features of the chip in the preprocessed image, uses PAFPN to perform multi-scale feature fusion, uses an IoU perception detection head to generate an output result in an anchor-free paradigm, and completes output value decoding and non-maximum suppression in a post-processing module to generate a final prediction result to complete the location and extraction of the chip image;

[0149] Step three, a pin detection module SSD-Pin based on a multi-scale single-stage deep neural network structure is constructed to detect the pins in the chip image using the SSD-Pin model to obtain a set of pin images of the chip;

[0150] Specifically, the pin detection module of step three of the embodiment of the application uses the backbone network of the SSD-Pin model to preliminarily extract features of the chip image, obtains features of 6 scales, and further performs convolution operation on the features to obtain an anchor-based chip pin detection result set.

[0151] Step four, a pin anomaly classification module CNN-Pin based on a deep neural network structure is constructed to use the CNN-Pin model to classify the pin images and complete pin anomaly situation detection;

[0152] Specifically, step four of the embodiment of the application uses a convolution layer and a down-sampling layer to extract features of the chip pin image, uses a fully connected layer to integrate the features, and finally uses a softmax classification layer to complete probability output of multi-classification to obtain a classification result.

[0153] Step five, a packaging surface anomaly detection module SSD-Surface based on a multi-scale single-stage deep neural network structure is constructed to use the SSD-Surface model to detect the chip with a packaging surface anomaly;

[0154] Specifically, in step five of this embodiment of the invention, the SSD-Surface model backbone network is used to perform preliminary feature extraction on the chip image to obtain features at six scales, and further convolution operations are performed on these features to obtain the detection results of abnormal parts on the chip package surface based on the anchor frame.

[0155] Step six involves constructing a dataset based on the preprocessed data to fine-tune the training of each model and then conducting tests.

[0156] Specifically, in step six of this embodiment of the invention, image annotation software and preprocessed images are used to construct the datasets required for training various detection and classification models. Fine-tuning is then performed using pre-trained models trained on the COCO2017 object detection dataset and pre-trained models trained on the ImageNet image classification dataset.

[0157] Step 7: Respond to the request from the peripheral application system via the router, and send the chip appearance defect detection results from Steps 4 and 5 to the peripheral system to achieve other peripheral application purposes based on the chip appearance defect detection results. The implementation of this peripheral application system is not limited, and the following are just examples: (1) A chip quality assessment system based on chip appearance defect detection, which can be used to evaluate the quality of manufactured chips and ensure production quality; (2) A chip recycling and sorting system based on chip appearance defect detection, used to sort scrapped or recycled chips according to defect categories.

[0158] This invention provides an electronic device for detecting chip appearance defects, which can execute the embodiments corresponding to the aforementioned chip appearance defect detection method. The electronic device provided by this invention employs the aforementioned chip appearance defect detection method, overcoming the shortcomings of low automation in existing chip appearance defect detection methods. It also addresses the shortcomings of low robustness and poor adaptability to actual conditions in existing chip appearance defect detection methods, and can be widely applied to various chip appearance defect detection scenarios and needs.

[0159] Example 3:

[0160] Based on the above embodiment 3, please refer to Figure 12 , Figure 12 This is a schematic diagram of a computer-readable storage medium for detecting chip appearance defects according to an embodiment of the present invention. The computer-readable storage medium for detecting chip appearance defects provided in this embodiment of the present invention stores a computer program, which, when executed by a processor, performs the following steps:

[0161] Step one: The program starts and performs necessary initializations.

[0162] Step 2: Turn on the optical image acquisition device to begin acquiring chip images.

[0163] Specifically, step two of the embodiment of the present application adopts a light optical image acquisition device to collect chip image data, and pre-processes the collected image by using single-scale Retinex light balance and median filtering.

[0164] Step three, a chip detection module YOLO-Chip based on a single-stage, anchor-free deep neural network structure is used to accurately locate and extract the chip in the pre-processed image;

[0165] Specifically, step three of the embodiment of the present application uses CSPDarkNet to preliminarily extract features of the chip in the pre-processed image, uses PAFPN to perform multi-scale feature fusion, uses an IoU perception detection head to generate an output result in an anchor-free paradigm, and completes output value decoding and non-maximum suppression in a post-processing module to generate a final prediction result to complete the location and extraction of the chip image.

[0166] Step four, a pin detection module SSD-Pin based on a multi-scale single-stage deep neural network structure is used to detect the pins in the chip image by using the SSD-Pin model to obtain a set of pin images of the chip.

[0167] Specifically, the pin detection module of step four of the embodiment of the present application uses the backbone network of the SSD-Pin model to preliminarily extract features of the chip image, obtains features of 6 scales, and further performs convolution operation on the features to obtain an anchor-based chip pin detection result set.

[0168] Step five, a pin anomaly classification module CNN-Pin based on a deep neural network structure is used to classify the pin image by using the CNN-Pin model to complete pin anomaly situation detection.

[0169] Specifically, step five of the embodiment of the present application uses a convolution layer and a down-sampling layer to extract features of the chip pin image, uses a fully connected layer to integrate the features, and finally uses a softmax classification layer to complete probability output of multi-classification to obtain a classification result.

[0170] Step six, a packaging surface anomaly detection module SSD-Surface based on a multi-scale single-stage deep neural network structure is used to detect the chip with packaging surface anomaly by using the SSD-Surface model.

[0171] Specifically, step six of the embodiment of the present application uses the backbone network of the SSD-Surface model to preliminarily extract features of the chip image, obtains features of 6 scales, and further performs convolution operation on the features to obtain an anchor-based chip packaging surface anomaly part detection result.

[0172] Step seven, based on the pre-processed data to build a data set to complete the fine-tuning training of each model. And testing.

[0173] Specifically, step seven of the embodiment of the present application utilizes image annotation software and pre-processed images to construct a data set required for training of each detection and classification model. And the pre-trained model trained based on COCO2017 target detection data set and the pre-trained model trained based on ImageNet image classification data set are used for fine-tuning training.

[0174] Step eight, the program needs to end when the program is exited.

[0175] The computer readable storage medium provided by the embodiment of the present application can execute the above-mentioned chip appearance defect detection method embodiment, and the computer program of the chip appearance defect detection is stored thereon, which can be used by various chip appearance defect detection electronic devices, thereby improving the security and accuracy of the chip appearance defect detection device, and bringing good experience to users.

[0176] It should be noted that the embodiments of the present application can be realized by hardware, software or a combination of software and hardware. The hardware part can be realized by special logic; the software part can be stored in a memory and executed by a suitable instruction execution system, such as a microprocessor or a specially designed hardware. Those skilled in the art can understand that the above-mentioned devices and methods can be realized by computer executable instructions and / or included in processor control code, such as provided on a carrier medium, such as a magnetic disk, CD or DVD-ROM, a programmable memory, such as a read-only memory (firmware), or a data carrier, such as an optical or electronic signal carrier. The devices of the present application and their modules can be realized by hardware circuits, such as very large scale integrated circuits or gate arrays, semiconductors, such as logic chips, transistors, etc., or programmable hardware devices, such as field programmable gate arrays, programmable logic devices, etc., can also be realized by software executed by various types of processors, and can also be realized by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0177] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any modification, equivalent replacement and improvement made by those skilled in the art within the technical range disclosed by the present application, as long as it is within the spirit and principles of the present application, should be covered within the protection scope of the present application.

Claims

1. An automatic chip appearance defect detection method, characterized by, The method comprises the following steps: An image acquisition and preprocessing module is constructed, an optical camera is used to acquire an image containing a chip, and contrast enhancement and noise reduction preprocessing are performed on the chip image; A chip detection module YOLO-Chip based on a single-stage and anchor-free deep neural network structure is constructed, and an IoU-aware classification loss function is correspondingly constructed, the chip detection module is used to locate and extract the chip region in the preprocessed image; A pin detection module SSD-Pin based on a multi-scale and single-stage deep neural network structure is constructed, and a bounding box regression loss function SIoU is correspondingly constructed, the pin detection module is used to detect and extract each pin in the chip region; A pin anomaly classification module CNN-Pin based on a deep neural network structure is constructed, and the pin anomaly classification module is used to classify the anomaly of each pin; A packaging surface anomaly detection module SSD-Surface based on a multi-scale and single-stage deep neural network structure is constructed, and the packaging surface anomaly detection module is used to locate and classify the abnormal part on the chip surface; A dataset making and model training module is constructed, a dataset is constructed based on the preprocessed image, and the chip detection module is fine-tuned and trained; A dataset is constructed based on the extracted chip image, and the pin detection module is fine-tuned and trained; A dataset is constructed based on the pin image, and the pin anomaly classification module is fine-tuned and trained; A dataset is constructed based on the extracted chip image, and the packaging surface anomaly detection module is fine-tuned and trained; A complete chip appearance defect detection system is formed by using the trained chip detection model, pin detection model, pin anomaly classification model and packaging surface anomaly detection model, and the chip appearance defect detection system is used to test the input new chip image to realize the chip appearance defect detection function.

2. The chip appearance defect automatic detection method according to claim 1, characterized in that, The steps of acquiring an image containing a chip by using an optical camera and preprocessing the image by using Retinex light balance and median filtering comprise: An RGB color space image containing a chip is acquired by using an optical camera, a single-scale Retinex algorithm is used to estimate the reflection component of the image, light balance is performed, and a contrast-enhanced chip image is obtained; On the basis of the contrast-enhanced chip image, a median filtering algorithm is used to complete detail protection and noise reduction of the image, and a high-quality chip image with better contrast and lower noise is obtained.

3. The chip appearance defect automatic detection method according to claim 1, characterized in that, The neural network model corresponding to the chip detection module comprises: a feature extraction backbone layer, a multi-scale feature fusion layer, an IoU-aware detection head and a post-processing module; The feature extraction backbone layer is based on CSPDarkNet and is used for preliminary feature extraction of an input image and generation of three-level multi-scale feature maps to be fused; The multi-scale feature fusion layer is based on PAFPN and performs path aggregation feature fusion with the three-level multi-scale feature maps as input to obtain a scale-robust feature map; The IoU-aware detection head takes the three-level feature fusion result as input, decouples the foreground classification and the bounding box regression task, and uses the IoU-aware classification loss function to complete the learning of the classification and regression branches of the task alignment, and performs classification and regression prediction based on the anchor point during testing. The post-processing module is used to decode the anchor-free prediction result, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip region detection result.

4. The chip appearance defect automatic detection method according to claim 3, characterized in that, Each sample of the IoU-aware classification loss function uses VFL, and the formula is: where p is the classification score IACS of the classification branch with IoU-aware ability, q is the true value of IACS; a is the balance coefficient between positive sample loss and negative sample loss; for the anchor point of the foreground target, q is set as the IoU between the predicted box of the anchor point by the regression branch and its true box, and the anchor point not assigned to the foreground target q is 0, p γ is the scaling factor of VFL, and finally the classification loss L cls is: where p k,i and q k,i are the ith kth type predicted IACS and IACS ground truth of the anchor point, respectively; N pos represents the total number of positive samples, which is obtained by the simOTA positive and negative sample assignment algorithm.

5. The chip appearance defect automatic detection method according to claim 1, wherein The neural network model corresponding to the pin detection module includes a feature extraction backbone layer, a multi-scale feature prediction head, and a post-processing module. The feature extraction backbone layer is constructed based on VGG16 and is used to extract 6 layers of feature maps of different scales. The multi-scale feature prediction head takes 6 layers of feature maps of different scales as input, completes the learning of the high-precision bounding box regression task based on the SIoU loss, and performs encoding prediction of classification and regression based on the anchor box during testing. The post-processing module is used to decode the prediction result based on the anchor box, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip pin detection result.

6. The chip appearance defect automatic detection method according to claim 5, wherein The SIoU loss comprehensively considers the intersection over union, the center point position, the width-height ratio, and the geometric difference of the angle between the real box and the predicted box. respectively represent the center point coordinates of the real box b gt and the predicted box b, and the SIoU loss defines an angle difference penalty term Λ between the real box and the predicted box as follows: wherein The SIoU loss defines a distance difference penalty term△ as: where ρ t is the ratio of the height of the smallest enclosing box of the predicted and ground truth boxes, respectively, to the height of the image, and x is the ratio of the width of the smallest enclosing box of the predicted and ground truth boxes, respectively, to the width of the image. y The ratio γ = 2 - Λ, R h , R w represent the height and width of the smallest enclosing box of the predicted and ground truth boxes, respectively. The SIoU loss defines a length-width ratio difference penalty termΩ as: where ω t is the sum of ω w , ω h , θ is the weight adjustment parameter of the term, w, h represent the width and height of the prediction box, w gt , h gt represent the width and height of the real box; Finally, the total bounding box regression loss L SIoU is defined as: wherein B, B GT respectively represent the predicted and real boxes.

7. The chip appearance defect automatic detection method according to claim 1, wherein The neural network model corresponding to the pin anomaly classification module includes a convolution layer, a down-sampling layer, a fully connected layer, and a classification layer. The convolution layer extracts features related to pin anomalies in the chip pin image based on 3×3 and 1×1 convolution. The down-sampling layer reduces the size of the feature map based on the max-pooling operation to reduce the amount of subsequent calculation and prevent overfitting caused by information redundancy. The fully connected layer expands the final two-dimensional feature map into one dimension and performs three fully connected operations to integrate the global features of the chip pin image. The classification layer uses softmax activation on the one-dimensional feature vector output by the fully connected layer to generate the final pin anomaly multi-classification probability.

8. The chip appearance defect automatic detection method according to claim 1, wherein The neural network model corresponding to the packaging surface anomaly detection module includes a feature extraction backbone layer, a multi-scale feature prediction head, and a post-processing module. The feature extraction backbone layer is constructed based on VGG16 and is used to extract 6 layers of feature maps of different scales. The multi-scale feature prediction head takes 6 layers of feature maps of different scales as input and performs encoding prediction based on the anchor box. The post-processing module is used to decode the prediction result based on the anchor box, perform non-maximum suppression, discard low-quality prediction values, and generate the final chip packaging surface anomaly result.

9. An electronic device, comprising: The electronic device includes a memory and a processor; the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the chip appearance defect automatic detection method in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to make the processor execute the chip appearance defect automatic detection method in any one of claims 1-8.

Images