An internal defect detection method for plastic encapsulated chips combined with deep learning

Abstract

The present invention discloses a method for detecting internal defects of plastic encapsulated chips combined with deep learning, belonging to the technical field of chip defect detection. The present invention includes: S1: Obtain the ultrasonic scan image of the plastic encapsulated chip, and use the improved neural network algorithm to perform chip target detection on the image; S2: Perform traditional image algorithm analysis on the obtained chip image to achieve the detection of chip defects; the neural network includes a backbone extraction network, an improved feature fusion network, and a YOLOHead structure. The improved feature fusion network strengthens the fusion of the three feature vectors obtained by the backbone extraction network through the pyramid feature fusion FPN and PAN networks in the YOLOv5 network to obtain enhanced feature vectors, and fuses the original feature vectors to obtain an improved feature fusion vector. The present invention combines deep learning and traditional algorithms to process the ultrasonic scan image, and can quickly identify the defect types and distributions at the pixel level.

Description

Technical Field

[0001] The present invention relates to the technical field of semiconductor chip defect detection, and more specifically, to an internal defect detection method for plastic-encapsulated chips combined with deep learning. Background Art

[0002] In the field of semiconductor chip manufacturing, the internal defect detection of plastic-encapsulated chips is a major problem. During the chip packaging process, it is easily affected by factors such as mechanical vibration and temperature change, resulting in delamination defects between the plastic encapsulant and the chip, and between the plastic encapsulant and the substrate, seriously affecting the performance of the plastic-encapsulated chips.

[0003] Since defects such as delamination belong to internal defects of the chip, it is necessary to use ultrasonic scanning imaging technology to scan the inside of the chip to obtain ultrasonic scanning images. Then, based on the amplitude intensity distribution of the ultrasonic scanning images, the quality of the chips is judged. Commonly, manual recognition is used to judge the defect types of the chips, which is time-consuming and laborious.

[0004] After retrieval, the patent application number is 202210402961.X, the application date is April 18, 2022, and the invention creation name is: A method for identifying internal defects of plastic-encapsulated chips; the identification method of this application case includes: S1: Obtain the original data set of ultrasonic detection images inside the plastic-encapsulated chips; S2: Perform data augmentation processing on the original data set to obtain the augmented data set; S3: Perform normalization processing on the augmented data set to obtain a large image data set; S4: Use the large image data set to train and test multiple convolutional neural network models, select the optimal model among them for improvement, and obtain the improved convolutional neural network model; S5: Use the large image data set to train and test the improved convolutional neural network model to obtain the identification result of the internal defects of the plastic-encapsulated chips. This application case uses the improved MobileNetV2 neural network to identify and classify the internal defects of plastic-encapsulated chips. However, in actual production detection, there are many chips in the ultrasonic scanning images, they are distributed densely, have small sizes, and the defect areas on small-sized chips are only a few pixels. Therefore, the missed detection rate is relatively high, the detection accuracy is low, which is not conducive to the realization of detection automation. In addition, to implement object detection by the neural network algorithm, a large number of defect images are required. In the plastic-encapsulated chip manufacturing industry, it is difficult and costly to obtain defect images, and it is difficult to meet the requirements of various types of plastic-encapsulated chips. Summary of the Invention

[0005] 1. Technical Problems to be Solved by the Invention

[0006] To solve the problems of low efficiency, slow detection, and difficulty for neural networks to distinguish pixel-level defects in manual detection of plastic-encapsulated chips in the prior art, the present invention provides a method for detecting internal defects of plastic-encapsulated chips combined with deep learning. In the process of chip plastic encapsulation, the present invention uses the ultrasonic reflection intensity map for defect detection, and combines deep learning and traditional algorithms to process the ultrasonic scan image, which can effectively and quickly identify the types and distributions of pixel-level defects.

[0007] 2. Technical Solution

[0008] To achieve the above object, the technical solution provided by the present invention is as follows:

[0009] A method for detecting internal defects of plastic-encapsulated chips combined with deep learning according to the present invention includes the following steps:

[0010] S1: Obtain the ultrasonic scan image of the plastic-encapsulated chip, and use the improved neural network algorithm to perform chip target detection on the ultrasonic scan image to obtain each target chip image in the ultrasonic scan image;

[0011] S2: Analyze the obtained chip image by traditional image algorithms to achieve the detection of chip defects;

[0012] The improved neural network includes a backbone extraction network, an improved feature fusion network, and a YOLOHead structure. The improved feature fusion network performs enhanced fusion on the three feature vectors obtained by the backbone extraction network through the pyramid feature fusion FPN and PAN networks in the YOLOv5 network to obtain enhanced feature vectors, and fuses the original feature vectors to obtain an improved feature fusion vector.

[0013] Furthermore, step S1 includes the following steps:

[0014] S11: Use an ultrasonic scanner to obtain the ultrasonic scan image inside the chip;

[0015] S12: Label the chips in the image obtained in step S11 using binary classification, set the "chip" label to 1 and the "non-chip" label to 0, and classify the labeled images into a training set, a validation set, and a test set;

[0016] S13: Augment the obtained data set, and use image cropping, rotation, and stitching to obtain an enhanced data set;

[0017] S14: Use the constructed data set to train the improved YOLOv5s neural network to obtain a trained model;

[0018] S15: Use the trained model to calculate and predict the ultrasonic scan image, and obtain the target chip image containing the complete chip area.

[0019] Furthermore, the backbone extraction network includes: Focus module, first convolution module, first C3 module, second convolution module, second C3 module, third convolution module, third C3 module, fourth convolution module, SPP module and fourth C3 module. The outputs of the second C3 module, the third C3 module, and the fourth C3 module are used as three feature vectors of different sizes.

[0020] Furthermore, the Focus module extracts one pixel every other pixel from the input image; all convolution modules are composed of two-dimensional convolution, batch normalization module, and activation function; all C3 modules are composed of multiple residual structures; the SPP module is formed by stacking after convolutions of multiple different sizes.

[0021] Furthermore, the three feature vectors of different sizes obtained by the backbone extraction network are redistributed for training resources through the CBAM attention mechanism to obtain the redistributed feature vectors, which are respectively named the first feature vector, the second feature vector, and the third feature vector. The three feature vectors are input into the FPN+PAN pyramid for feature fusion. The three feature vectors after feature fusion are superimposed and fused with the corresponding input feature vectors again to obtain enhanced features, and finally three outputs are obtained through the YOLOHead structure.

[0022] Furthermore, the process of training the improved neural network is as follows:

[0023] S141: Normalize the image and input the normalized image into the improved YOLOv5s neural network.

[0024] S142: Three loss functions are used in the training process, namely classification loss, localization loss, and confidence loss. The original localization loss is replaced by the CIOU loss function, and the classification loss function uses the cross-entropy loss function.

[0025] S143: Use the cross-entropy loss function and the CIOU loss function to evaluate the prediction results, and use the Adam optimization function to adjust and optimize the model parameters to gradually reduce the loss value. When the loss value is gradually fitted and no longer decreases, the network model training is completed.

[0026] S144: Use the test set to test the trained model and evaluate the detection performance of the network.

[0027] Furthermore, step S2 includes:

[0028] S21: Implement simple segmentation of the chip area and the background using image grayscale and image binarization algorithms, remove image noise by combining image dilation and erosion, perform edge detection on the segmented binary image to obtain the edge area of the chip, further extract the minimum bounding rectangle, obtain the minimum bounding rectangle of the chip area, and crop according to the minimum bounding rectangle to obtain the accurate chip area;

[0029] S22: Grayscale the accurate chip area image, and accumulate the pixel values in the row and column directions of the image respectively to obtain two one-dimensional vectors; Solve the minimum value and the second minimum value of the two one-dimensional vectors respectively, perform threshold segmentation according to the second minimum value, solve the rising edge and falling edge positions of the two segmented one-dimensional vectors, calculate the width greater than the threshold, and compare these two widths to obtain the direction position of the chip; After obtaining the chip direction, rotate the image to keep the chip direction consistent to obtain the rotated image;

[0030] S23: Extract the potential defect area from the rotated and corrected image, obtain the three channels of the image, process the red channel and the green channel, use twice the pixel value of the red channel minus the pixel value of the green channel to obtain a single-channel image; Perform threshold segmentation on the obtained single-channel image to obtain the binary image of the red defect area;

[0031] S24: For the obtained binary image, perform image connected component calculation to obtain the size and position of all potential defect area connected components; Calculate the intersection-over-union ratio of the potential defect area and the ROI area and the size of the potential defect connected component to determine whether the potential defect area is a chip defect; And perform calculation and judgment on each chip, and combine the chip position to display whether each chip has a defect on the ultrasonic scan image.

[0032] 3. Beneficial effects

[0033] Adopting the technical solution provided by the present invention, compared with the existing well-known technologies, it has the following remarkable effects:

[0034] (1) A method for detecting internal defects of plastic encapsulated chips combined with deep learning according to the present invention can realize automatic recognition of ultrasonic internal scan defects of plastic encapsulated chips, reduce the time of manual detection, and improve the detection efficiency and detection accuracy; Moreover, the present invention can accurately locate and classify defects at the pixel level on the chip, avoid the problem of labeling a large number of defect images, and realize defect recognition of small-sized plastic encapsulated chips.

[0035] (2) A method for detecting internal defects of plastic encapsulated chips combined with deep learning according to the present invention does not require a large number of dataset annotations, can save the pre-preparation time, and can adjust traditional image algorithms according to different chips without relying on a large number of defect images. Description of the drawings

[0036] Figure 1 This is the flowchart of ultrasonic scanning defects of plastic encapsulated chips provided by the present invention.

[0037] Figure 2 This is the ultrasonic scanning pattern of plastic encapsulated chips provided by the present invention.

[0038] Figure 3 This is the deep learning neural network model diagram provided by the present invention.

[0039] Figure 4 This is the structural schematic diagram of the C3 module provided by the present invention.

[0040] Figure 5 This is the structural schematic diagram of the SPP module provided by the present invention.

[0041] Figure 6 This is the process diagram of the image processing algorithm provided by the present invention. Detailed implementation manners

[0042] To further understand the content of the present invention, the present invention will be described in detail in combination with the accompanying drawings and embodiments.

[0043] Embodiment 1

[0044] A method for detecting internal defects of plastic encapsulated chips combined with deep learning in this embodiment has a processing flow as Figure 1 shown, and the specific steps are as follows:

[0045] S1: Obtain the ultrasonic scanning image of the plastic encapsulated chip, and use the deep learning method to perform chip target detection on the ultrasonic scanning image to obtain each target chip image in the ultrasonic scanning image. Among them, step S1 includes:

[0046] S11: Use an ultrasonic scanner to obtain the ultrasonic scanning map inside the chip. In this embodiment, the electroplated chip is placed in deionized water, and the ultrasonic scanner is used for positioning and scanning to obtain an ultrasonic amplitude intensity distribution map as Figure 2 shown. Figure 2 In the figure, the larger the pixel value of the defect area, the smaller the intensity of the reflected wave, and the more serious the defect. A total of 100 ultrasonic scanning images are obtained in this embodiment.

[0047] S12: Use binary classification to label the chips in the image obtained in step S11. In this embodiment, the lableme software is used, the "chip" label is set to 1, the "non-chip" label is set to 0, the data set is established according to the VOC format, and the labeled images are classified into a training set, a validation set, and a test set.

[0048] S13: Augment the obtained dataset. The methods for augmenting the dataset include: performing rotation transformation on the image, randomly cropping and splicing the image, and adjusting the brightness of the image to achieve the uneven brightness of the image caused by uneven scanning that can occur in actual situations, so as to obtain the augmented dataset.

[0049] S14: Classify the format of the obtained augmented dataset, and classify it into a training set, a validation set, and a test set according to the ratio of 8:1:1. Use the constructed dataset to train the improved YOLOv5s neural network to obtain a trained model.

[0050] The improved YOLOv5s neural network structure is as Figure 3 shown. Specifically: the input size of the neural network is 640*640*3. Send the input image into the Backbone main extraction network, where the main extraction network includes: Focus module, first convolution module, first C3 module, second convolution module, second C3 module, third convolution module, third C3 module, fourth convolution module, SPP module, fourth C3 module. The main extraction network can realize the extraction of main features. Among them, the Focus module extracts one pixel every other pixel of the input image, and can convert the 640*640*3 input image into 320*320*12; all convolution modules are composed of two-dimensional convolution, batch normalization module, and activation function; all C3 modules are as Figure 4 shown, and are composed of multiple residual structures; the SPP module is as Figure 5 shown, and is formed by stacking after convolutions of multiple different sizes. Take the outputs of the second C3 module, the third C3 module, and the fourth C3 module as three feature vectors of different sizes.

[0051] Obtain three feature vectors of different sizes and reallocate the training resources through the CBAM (Convolutional Block Attention Module) attention mechanism to obtain the reallocated feature vectors, which are named the first feature vector, the second feature vector, and the third feature vector respectively. Input the three feature vectors into the FPN (Feature Pyramid Network) combined with the PAN (Path Aggregation Network) for feature fusion. The three feature vectors after feature fusion are superimposed and fused with the corresponding input feature vectors again to obtain enhanced features, and finally three outputs are obtained through the YOLOHead (decoupled).

[0052] The network training process of this embodiment is as follows:

[0053] S141: Normalize the image to the 640*640*3 format, and input the normalized image into the improved YOLOv5s neural network.

[0054] S142: The improved YOLOv5s neural network includes a backbone network, an improved feature fusion network, and the YOLOHead structure to obtain three outputs. Among them, the improved feature fusion network strengthens and fuses the three feature vectors obtained by the backbone extraction network through the pyramid feature fusion FPN and PAN networks in the YOLOv5 network to obtain enhanced feature vectors, and fuses the original feature vectors to obtain the improved feature fusion vectors.

[0055] Three loss functions are used in the training process, namely classification loss, localization loss, and confidence loss. During training, due to the small size of the target and inaccurate localization, the original localization loss is replaced by the CIOU loss function from the GIOU loss function to improve the accurate localization of the target chip. The classification loss function uses the cross-entropy loss function, as shown in Equation (1):

[0056]

[0057] where y represents the predicted value, represents the true value.

[0058] The localization loss is the CIOU loss function, and the expression is:

[0059]

[0060] where ρ 2 (b, b gt ) represents the Euclidean distance between the predicted box and the true box, c represents the distance of the diagonal of the minimum bounding rectangle of the predicted box and the true box, and ν is a proportionality parameter measuring the ratio of the predicted box to the true box.

[0061] S143: Use the cross-entropy loss function and the CIOU loss function to evaluate the prediction results, and use the Adam optimization function to adjust and optimize the model parameters to gradually reduce the loss value. When the loss value gradually fits and no longer decreases, it indicates that the network model has been trained.

[0062] S144: Use the test set to test the trained model and evaluate the detection performance of the network.

[0063] S15: Use the trained model to calculate and predict the ultrasonic scan image and obtain the target chip image containing the complete chip area.

[0064] S2: Perform traditional image algorithm analysis on the obtained chip image to achieve chip defect detection. The S2 step includes:

[0065] S21: Further process the obtained chip image using traditional image algorithms. First, use image grayscale conversion and image binarization algorithms to simply segment the chip area from the background. Combine image dilation and erosion to remove image noise. Perform edge detection on the segmented binary image to obtain the edge area of the chip. Further extract the minimum bounding rectangle to get the minimum bounding rectangle of the chip area, and crop according to the minimum bounding rectangle to obtain the accurate chip area as shown in Figure 6 as shown.

[0066] S22: Grayscale the accurate chip area image, and perform pixel accumulation respectively in the row and column directions of the image to obtain two one-dimensional vectors. Solve the minimum value and the second minimum value for the two one-dimensional vectors respectively, perform threshold segmentation according to the second minimum value, solve the rising edge and falling edge positions for the two segmented one-dimensional vectors, calculate the width greater than the threshold, and compare these two widths to obtain the direction position of the chip. After obtaining the chip direction, rotate the image to keep the chip direction consistent to obtain the image after rotation in the direction as shown in Figure 4 as shown in.

[0067] S23: Extract the potential defect area from the rotated and corrected image to obtain the three channels of the image. Since the ultrasonic scan image returns an amplitude intensity distribution map, all pixel values in the blue channel are 0. Therefore, only process the red channel and the green channel. Subtract the pixel value of the green channel from twice the pixel value of the red channel to obtain a single-channel image, and its calculation formula is as follows:

[0068] M = M R - M G (3)

[0069] where M represents the obtained grayscale pixel value, M R represents the pixel value corresponding to the red channel, and M G represents the pixel value corresponding to the green channel. Perform threshold segmentation on the obtained single-channel image to obtain the binary image of the potential defect area as shown in Figure 6 as shown in.

[0070] S24: For the obtained binary image, perform image connected component calculation. Obtain the size and position of all connected components of the potential defect areas. Calculate the intersection-over-union ratio of the potential defect areas and the ROI area and the size of the potential defect connected components to determine whether the potential defect areas are chip defects. In the binary image, the areas outside the ROI area are the potential defect areas of the plastic encapsulated chip, and the areas in the ROI area are interferences caused by other devices such as capacitors in the plastic encapsulated chip itself. Calculate and judge for each chip, and combine the chip position to display whether each chip has defects on the ultrasonic scan image.

[0071] The above has schematically described the present invention and its embodiments. This description is not restrictive. What is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Therefore, if those of ordinary skill in the art are inspired by it and, without departing from the gist of the present invention, design similar structural modes and embodiments to this technical solution without creative efforts, they shall fall within the protection scope of the present invention.

Claims

1. An internal defect detection method for plastic encapsulated chips combined with deep learning, characterized in that It includes the following steps: S1: Obtain the ultrasonic scan image of the encapsulated chip, and use the improved neural network algorithm to perform chip target detection on the ultrasonic scan image to obtain each target chip image in the ultrasonic scan image; S2: Perform traditional image algorithm analysis on the obtained chip images to achieve chip defect detection; The process of using traditional image algorithms for analysis includes: S21: Use image grayscale conversion and image binarization algorithms to achieve simple segmentation of the chip area and the background, combine image dilation and erosion to remove image noise, perform edge detection on the segmented binary image to obtain the edge area of the chip, further extract the minimum bounding rectangle, obtain the minimum bounding rectangle of the chip area, and crop according to the minimum bounding rectangle to obtain the accurate chip area; S22: Grayscale the accurate chip area image, and accumulate the pixel values in the row and column directions of the image respectively to obtain two one-dimensional vectors; Solve the minimum value and the second minimum value of the two one-dimensional vectors respectively, perform threshold segmentation according to the second minimum value, solve the rising edge and falling edge positions of the two segmented one-dimensional vectors, calculate the width greater than the threshold, and compare these two widths to obtain the direction position of the chip; After obtaining the chip direction, rotate the image to keep the chip direction consistent to obtain the rotated image; S23: Extract the potential defect area from the rotated and corrected image, obtain the three channels of the image, process the red channel and the green channel, use twice the pixel value of the red channel minus the pixel value of the green channel to obtain a single-channel image; Perform threshold segmentation on the obtained single-channel image to obtain the binary image of the red defect area; S24: For the obtained binary image, perform image connected component calculation to obtain the size and position of all potential defect area connected components; Calculate the intersection-over-union ratio of the potential defect area and the ROI area and the size of the potential defect connected component to determine whether the potential defect area is a chip defect; And perform calculation and judgment on each chip, and combine the chip position to display whether each chip has defects on the ultrasonic scan image; The improved neural network includes a backbone extraction network, an improved feature fusion network, and a YOLOHead structure. The improved feature fusion network strengthens and fuses the three feature vectors obtained by the backbone extraction network through the pyramid feature fusion FPN and PAN networks in the YOLOv5 network to obtain strengthened feature vectors, and fuses the original feature vectors to obtain improved feature fusion vectors; The backbone extraction network includes: a Focus module, a first convolutional module, a first C3 module, a second convolutional module, a second C3 module, a third convolutional module, a third C3 module, a fourth convolutional module, an SPP module, and a fourth C3 module. The outputs of the second C3 module, the third C3 module, and the fourth C3 module are used as three feature vectors of different sizes; The Focus module extracts every other pixel from the input image; all convolutional modules are composed of two-dimensional convolution, batch normalization module, and activation function; all C3 modules are composed of multiple residual structures; the SPP module is formed by stacking multiple convolutions of different sizes; The three feature vectors of different sizes obtained by the backbone extraction network are reallocated training resources through the CBAM attention mechanism to obtain the reallocated feature vectors, which are named the first feature vector, the second feature vector, and the third feature vector respectively; the three feature vectors are input into the FPN+PAN pyramid for feature fusion, and the three feature vectors after feature fusion are superimposed and fused with the corresponding input feature vectors again to obtain enhanced features, and finally three outputs are obtained through the YOLOHead structure.

2. The internal defect detection method of the plastic encapsulated chip combined with deep learning according to claim 1, characterized in that, Step S1 includes the following steps: S11: Use an ultrasonic scanner to obtain an ultrasonic scan image inside the chip; S12: Label the chip in the image obtained in step S11 using binary classification, set the "chip" label to 1 and the "non-chip" label to 0, and classify the labeled images into training set, validation set, and test set; S13: Augment the obtained dataset, and use image cropping, rotation, and stitching to obtain an enhanced dataset; S14: Use the constructed dataset to train the improved YOLOv5s neural network to obtain a trained model; S15: Use the trained model to calculate and predict the ultrasonic scan image, and obtain a target chip image containing the complete chip area.

3. The internal defect detection method of the plastic encapsulated chip combined with deep learning according to claim 2, characterized in that: The process of training the improved neural network is as follows: S141: Normalize the image and input the normalized image into the improved YOLOv5s neural network; S142: Three loss functions are used in the training process, namely classification loss, localization loss, and confidence loss. The original localization loss is replaced by the CIOU loss function, and the classification loss function uses the cross-entropy loss function; S143: Use the cross-entropy loss function and the CIOU loss function to evaluate the prediction results, and use the Adam optimization function to adjust and optimize the model parameters to gradually reduce the loss value. When the loss value gradually fits and no longer decreases, the network model training is completed; S144: Use the test set to test the trained model and evaluate the detection performance of the network.

Images