A method and device for detecting defects in chip adhesion area

Abstract

The present invention discloses a method and device for detecting chip adhesion area defects, the method comprising: preprocessing the collected X-ray image to obtain a first image; using a trained target detection model to locate the position of the adhesion area in the first image; repairing the adhesion area defect in the first image based on an unsupervised image repair model to obtain a second image; using an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine the pixels corresponding to the target defect. The method for detecting chip adhesion area defects provided in the embodiment of the present application has accurate and reliable detection results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and more particularly to a method and device for detecting chip adhesion area defects based on deep learning. Background Art

[0002] In the production process of component packaging, the chip must be fixed on the solder plate by adhesive materials to ensure its working reliability. If the chip bonding material is insufficient, the chip adhesion strength will be reduced, and it will easily fall off during operation, causing fatal failure of the device. In addition, insufficient bonding material may cause local voids in the bonding material under the chip. These voids will cause uneven heat dissipation of the chip and induce secondary breakdown and burning of the device. Therefore, accurately detecting defects in the chip bonding area is of great significance to the quality and safety of components.

[0003] The existing method for detecting defects in the adhesion area of ​​chips is as follows: manually place the components to be inspected on the inspection workbench of the X-ray inspection equipment, then operate the equipment to aim at the components to be inspected, take pictures of the components, and transfer the pictures to the computer. Finally, manually browse the pictures formed by the X-rays to detect defects in the adhesion area. This method of manual naked eye inspection not only consumes a lot of time of the inspectors, but also has the problem of human misjudgment caused by visual fatigue or lack of experience. The quality of components is difficult to effectively guarantee, which brings risks and hidden dangers to the safe operation of components on site. Summary of the invention

[0004] The embodiment of the present invention provides a method for detecting defects in a chip adhesion area, so as to solve the problem of inaccurate detection results of defects in a chip adhesion area in the prior art, and can improve detection stability and detection accuracy.

[0005] In order to solve the above technical problems, the present invention discloses a method for detecting defects in chip adhesion areas, wherein the method comprises: preprocessing the collected X-ray image to obtain a first image; using a trained target detection model to locate the position of the adhesion area in the first image; repairing the adhesion area defects in the first image based on an unsupervised image repair model to obtain a second image; and using an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine the pixels corresponding to the target defects.

[0006] Optionally, before the step of preprocessing the collected X-ray image to obtain the first image, the method also includes: preprocessing a preset number of image samples to obtain a third image; for each of the third images, based on template matching the adhesion area position in the third image, converting the matching coordinates into a label file, and saving the label file; and training the target detection model based on the labeled image samples.

[0007] Optionally, before the step of training the target detection model based on the labeled image samples, the method further includes: performing sample amplification processing on the labeled image samples, wherein the sample amplification processing includes: at least one of rotation, flipping, scaling, brightness adjustment and contrast adjustment.

[0008] Optionally, the step of preprocessing the collected X-ray image to obtain the first image includes: using Gaussian filtering to filter out Gaussian noise in the collected X-ray image to obtain a fourth image; and using median filtering to filter out salt and pepper noise in the fourth image to obtain the first image.

[0009] Optionally, the step of performing defect detection on the first image and the second image using a segmentation algorithm with an adaptive threshold to determine pixels corresponding to target defects includes: performing pixel-level difference processing on the first image and the second image to determine a defect score map; and segmenting the image according to an adaptive threshold to determine pixels corresponding to the target defect.

[0010] In order to solve the above technical problems, the present invention also discloses a device for detecting defects in chip adhesion areas, the device comprising: a first preprocessing module, used to preprocess the collected X-ray image to obtain a first image; a positioning module, used to use a trained target detection model to locate the position of the adhesion area in the first image; a repair module, used to repair the adhesion area defects in the first image based on an unsupervised image repair model to obtain a second image; a determination module, used to use an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image, and determine the pixels corresponding to the target defects.

[0011] Optionally, the device also includes: a second preprocessing module, used to preprocess a preset number of image samples to obtain a third image before the first preprocessing module preprocesses the collected X-ray image to obtain the first image; a labeling module, used to convert the matching coordinates into a label file based on the template matching the adhesion area position in the third image for each of the third images, and save the label file; a training module, used to train the target detection model based on the labeled image samples.

[0012] Optionally, the device also includes: an amplification module, used to perform sample amplification processing on the labeled image samples before the training module trains the target detection model based on the labeled image samples, wherein the sample amplification processing includes: at least one of rotation, flipping, scaling, brightness adjustment and contrast adjustment.

[0013] Optionally, the first preprocessing module includes: a first submodule, used to use Gaussian filtering to filter out Gaussian noise in the collected X-ray image to obtain a fourth image; and a second submodule, used to use median filtering to filter out salt and pepper noise in the fourth image to obtain the first image.

[0014] Optionally, the determination module includes: a third submodule, used to perform pixel-level difference processing on the first image and the second image to determine a defect score map; and a fourth submodule, used to segment the image according to an adaptive threshold to determine pixels corresponding to the target defect.

[0015] The solution for detecting chip adhesion area defects provided by an embodiment of the present invention preprocesses the collected X-ray image to obtain a first image; uses a trained target detection model to locate the position of the adhesion area in the first image; repairs the adhesion area defects in the first image based on an unsupervised image repair model to obtain a second image; uses an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine the pixels corresponding to the target defects, and the detected results are accurate and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of the steps of the method for detecting chip adhesion area defects provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of an automatic standard flow based on template matching provided in an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the template matching effect provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of a training curve of a target detection model provided in an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the chip adhesion area defect detection effect provided by an embodiment of the present invention;

[0021] Figure 6 A network structure diagram provided by an embodiment of the present invention;

[0022] Figure 7 This is a repaired image effect diagram provided by an embodiment of the present invention;

[0023] Figure 8 A detection effect diagram provided by an embodiment of the present invention;

[0024] Fig. 9 This is a structural block diagram of a device for detecting chip adhesion area defects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below based on specific embodiments and in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] The method for detecting chip adhesion area defects in the embodiment of the present application is based on a deep learning network structure, and mainly includes: acquiring the collected original image and template image, and preprocessing them to match them using a preset matching algorithm to obtain the position of the adhesion area, and saving the label file; building a target detection model, training it with labeled samples, and locating the position of the adhesion area using the trained target detection model; building an image repair model, training it with images of the adhesion area, and using the trained model to repair defects in the image to obtain a corresponding defect-free image; using an adaptive threshold segmentation algorithm to perform defect detection on the images before and after repair to obtain the pixel-level positioning of the final defect.

[0027] like Figure 1 As shown, a flow chart of the steps of a method for detecting chip adhesion area defects according to an embodiment of the present invention is provided. The method comprises the following steps:

[0028] Step 101: pre-process the collected X-ray image to obtain a first image.

[0029] Since there is a lot of white point noise in X-ray images, it is necessary to preprocess the X-ray images to eliminate image noise.

[0030] An optional way of preprocessing the collected X-ray image to obtain the first image may be: using Gaussian filtering to filter out Gaussian noise in the collected X-ray image to obtain a fourth image; using median filtering to filter out salt and pepper noise in the fourth image to obtain the first image, and the obtained first image is the image after noise is eliminated.

[0031] Step 102: Using the trained object detection model, locate the position of the adhesion area in the first image.

[0032] Due to the complex and changeable imaging conditions in the actual environment, traditional detection algorithms are difficult to adapt to all images and lack robustness. By utilizing the powerful feature extraction capabilities of convolutional neural networks and building a target detection model based on deep learning, the adhesion area features can be accurately extracted to achieve detection of adhesion areas.

[0033] The target detection model is trained before step 101. An optional method for training the target detection model may be as follows:

[0034] First, a preset number of image samples are preprocessed to obtain a third image.

[0035] The preset number can be set by those skilled in the art according to actual needs, and is not specifically limited in the embodiments of the present application. The method of preprocessing the sample image can refer to the relevant description of preprocessing the X-ray image in step 101, and will not be repeated here.

[0036] Secondly, for each third image, the position of the adhesion area in the third image is matched based on the template, the matching coordinates are converted into a label file, and the label file is saved to complete the labeling of the image sample.

[0037] Since the position of the chip in the entire image is not fixed, template matching is used to roughly locate the position of the adhesion area, and the matching coordinates are converted into a label file. For the mismatched structure, manual correction is performed to complete the construction of the data set. This technology can greatly improve the labeling efficiency and reduce the workload of manual labeling.

[0038] Finally, the object detection model is trained based on the labeled image samples.

[0039] An optional way to obtain the image sample may be to perform sample augmentation processing on the labeled image sample, wherein the sample augmentation processing includes at least one of rotation, flipping, scaling, brightness adjustment, and contrast adjustment. This method of performing sample augmentation processing on the labeled image sample can reduce the resources consumed by image sample processing.

[0040] Step 103: Repair the adhesion area defect in the first image based on the unsupervised image restoration model to obtain a second image.

[0041] The normal defect-free adhesion area images screened out are put into the image repair model for training. The trained images can repair the defects in the images well and obtain the corresponding defect-free images.

[0042] Step 104: Use an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine pixels corresponding to the target defects.

[0043] The image before and after repair is differentiated at the pixel level, the defect score map is calculated, and the image is segmented according to the automatically selected threshold to obtain the pixel-level area location of the final defect.

[0044] The method for detecting chip adhesion area defects provided in the embodiment of the present application pre-processes the collected X-ray image to obtain a first image; uses a trained target detection model to locate the position of the adhesion area in the first image; repairs the adhesion area defects in the first image based on an unsupervised image repair model to obtain a second image; uses an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine the pixels corresponding to the target defects, and the detected results are accurate and reliable.

[0045] Combine the following Figure 2 to Figure 8 The method for detecting chip adhesion area defects of the present application is described. The method includes the following process:

[0046] S1 Image Preprocessing

[0047] Before template matching, the original image and the template image need to be preprocessed. First, Gaussian filtering is used to filter out Gaussian noise, and median filtering is used to filter out salt and pepper noise. To reduce the influence of illumination, the image is normalized to [0,1] by maximum and minimum values. The normalization formula is as follows:

[0048]

[0049] Among them, x i ,y i They represent the pixels before and after normalization respectively, min(x) and max(x) represent the minimum and maximum grayscale values ​​of the image respectively.

[0050] S2 automatically annotates image data based on template matching

[0051] In order to reduce the workload of manual annotation and improve annotation efficiency, the embodiment of the present application proposes an automatic annotation technology based on template matching, using a correlation coefficient based on normalized cross-correlation:

[0052]

[0053] Among them, Gx T ,Gy T Respectively represent the gradient amplitude of the template image in the x and y directions, Gx S ,Gy S Represent the gradient magnitude of the original image in the x and y directions respectively.

[0054] First, calculate the gradient images of the template image w(x,y) and the standard image f(x,y) respectively. Then, starting from the origin of the original image, obtain a sub-image of the same size as the template image, calculate their correlation coefficients, and slide the template image from left to right and from top to bottom in the search image, traverse the entire image in turn, calculate all the correlation coefficients, and compare them to find the best matching position. When the correlation coefficient γ(x,y) takes the maximum value, the corresponding (x,y) is the best matching position. The specific flow chart is as follows: Figure 2 Shown:

[0055] However, template matching can only be used for images with a small rotation angle. When the chip in the image is rotated at a large angle, a large error will occur, and it needs to be manually re-labeled using labeling software such as Labelme or Labelimg. The template matching result is shown in the figure below. Figure 3 shown.

[0056] S3 locates the adhesion area based on the deep learning target detection model

[0057] Due to the complex and changeable imaging conditions in the actual environment, traditional detection algorithms are difficult to adapt to all images and lack robustness. Therefore, the powerful feature extraction capability of convolutional neural networks is used to construct the Yolov4 target detection network to realize the detection and positioning of adhesion areas. The backbone feature extraction network of YOLOv4 is a CSP Daeknet53 structure, which contains 5 large residual blocks. The number of small residual units contained in the 5 large residual blocks is 1, 2, 8, 8, and 4 respectively. YOLOv4 has three outputs with different feature scales, namely 19×19, 38×38, and 76×76. For each feature layer, three scales of anchor box frames are set for discrimination to determine whether it contains the target and the type of target, and non-maximum suppression processing and anchor box adjustment are performed to obtain the final prediction frame. Its CSP Daeknet53 backbone feature structure is shown in Table 1:

[0058] Table 1

[0059] Inputs 608×608×3 DarknetConv2D_BN_Mish 608×608×32 Resblock_body×1 304×304×64 Resblock_body×2 152×152×128 Resblock_body×8 76×76×256 Resblock_body×8 38×38×512 Resblock_body×4 19×19×1024

[0060] The loss function in YOLOv4 network training is composed of the target box regression loss L ciou , confidence loss L conf and classification loss L class The target box regression function uses CIOU_Loss, and the distance between the target and the anchor, the overlap rate, the scale, and the penalty term are all taken into account, making the target box regression more stable.

[0061] L ciou The calculation formula is as follows:

[0062]

[0063] Among them, v is a parameter to measure the consistency of aspect ratio:

[0064]

[0065] Where c is the diagonal distance between the predicted box and the actual box closure area;

[0066] b, w, h——the center coordinates, width and height of the prediction box;

[0067] b gt 、w gt 、h gt ——The center coordinates, width and height of the actual frame.

[0068] Since it is difficult to collect a large number of image samples in an actual industrial environment, it is necessary to perform data augmentation on the automatically annotated data, such as rotation, flipping, scaling, adjusting brightness and contrast, to obtain a training data set. The constructed data set is put into the YOLOv4 network for training, with the learning rate set to 0.0013 and the weight decay coefficient set to 0.0005. A total of 6000 training times are performed, and the training curve is as follows: Figure 4 shown.

[0069] Finally, the trained network is used to detect and locate the adhesion area. The final network detection result is as follows: Figure 5 shown.

[0070] S4 repairs and reconstructs adhesion area defects based on an unsupervised image repair model

[0071] The chip adhesion area detected by YOLOV4 in the previous step is cropped from the image to obtain the image of the adhesion area, and the image is divided into 32×32 sub-image blocks. First, a lightweight autoencoder network model is constructed. The autoencoder is mainly composed of three parts: encoder, fully connected layer and decoder. It reconstructs the input data through encoding and decoding operations to achieve the purpose of feature extraction with the goal of reducing reconstruction error. Among them, the encoder consists of 4 convolutional layers and 4 pooling layers, and the decoder consists of 4 upsampling layers and 4 convolutional layers. Each convolutional layer is processed by a normalization layer, and the pooling layer uses average pooling to retain more background information. The activation function uses the ReLu activation function except the last convolutional layer, and the last layer uses the Sigmoid activation function. The activation function formula is as follows:

[0072] relu(x)=max(x,0)

[0073]

[0074] The network structure of the autoencoder is shown in the figure Figure 6 As shown:

[0075] Network training: Use defect-free image blocks to train the autoencoder so that the network can learn the features of defect-free image blocks. Since no data labels are used in the training process, and only the input data is used as the reconstruction target, it is a typical unsupervised learning. The loss function during training uses smooth L1 loss, which not only solves the problem of easy derivation of L1 loss, but also avoids the gradient explosion problem of L2 loss. The formula is as follows:

[0076]

[0077] During the detection process, the autoencoder will repair the defects of the input image to generate a defect-free image similar to the input image. The network's repair results are shown in Figure 2. Figure 7 As shown, Figure 7 (a) is the original image, and (b) is the image restored by the network.

[0078] S5 Pixel-level defect localization based on adaptive threshold

[0079] First, the residual of the input image and the image reconstructed by the previous network is calculated to obtain a residual image. Then the residual image is binarized using an adaptive threshold. The formula for the adaptive threshold is:

[0080] T=μ+λσ

[0081] Among them, T is the adaptive threshold, μ and σ are the mean and standard deviation of the residual image, and λ is the adjustment coefficient.

[0082] The binarization formula is as follows:

[0083]

[0084] Among them, x(i,j) is the pixel value of the residual image at (i,j), and y(i,j) is the pixel value of the binary image at (i,j) after segmentation.

[0085] The final defect segmentation result is as follows: Figure 8 shown.

[0086] Fig. 9 A structural block diagram of a device for detecting chip adhesion area defects provided by an embodiment of the present invention.

[0087] like Fig. 9 As shown, a device for detecting chip adhesion area defects according to an embodiment of the present invention includes the following modules:

[0088] A first preprocessing module 901 is used to preprocess the collected X-ray image to obtain a first image;

[0089] A positioning module 902, configured to locate the position of the adhesion area in the first image using the trained object detection model;

[0090] A repair module 903 is used to repair the adhesion area defect in the first image based on an unsupervised image repair model to obtain a second image;

[0091] The determination module 904 is used to perform defect detection on the first image and the second image using a segmentation algorithm with an adaptive threshold value, and determine pixels corresponding to target defects.

[0092] Optionally, the device further comprises:

[0093] A second preprocessing module, configured to preprocess a preset number of image samples to obtain a third image before the first preprocessing module preprocesses the collected X-ray image to obtain the first image;

[0094] a labeling module, configured to, for each of the third images, match the position of the adhesion region in the third image based on the template, convert the matching coordinates into a label file, and save the label file;

[0095] The training module is used to train the target detection model based on the labeled image samples.

[0096] Optionally, the device further comprises:

[0097] The amplification module is used to perform sample amplification processing on the labeled image samples before the training module trains the target detection model based on the labeled image samples, wherein the sample amplification processing includes: at least one of rotation, flipping, scaling, brightness adjustment and contrast adjustment.

[0098] Optionally, the first preprocessing module includes:

[0099] The first submodule is used to filter out Gaussian noise in the collected X-ray image by using Gaussian filtering to obtain a fourth image;

[0100] The second submodule is used to filter out the salt and pepper noise in the fourth image by using median filtering to obtain the first image.

[0101] Optionally, the determining module includes:

[0102] The third submodule is used to perform pixel-level difference processing on the first image and the second image to determine a defect score map;

[0103] The fourth submodule is used to segment the image according to the adaptive threshold value and determine the pixels corresponding to the target defect.

[0104] The device for detecting chip adhesion area defects provided in the embodiment of the present application can be used to execute the relevant processes of the above-mentioned method for detecting chip adhesion area defects. The relevant parts between the two can be referred to the relevant description in the above-mentioned method for detecting chip adhesion area defects, which will not be repeated here.

[0105] It should be noted that the above is only the best specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0106] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. A method for detecting defects in a chip adhesion area, characterized in that: The method comprises: Preprocessing the collected X-ray image to obtain a first image; Using the trained object detection model, locate the position of the adhesion area in the first image; Repairing the adhesion area defect in the first image based on an unsupervised image repair model to obtain a second image; Using an adaptive threshold segmentation algorithm to perform defect detection on the first image and the second image to determine pixels corresponding to target defects; Before the step of preprocessing the acquired X-ray image to obtain the first image, the method further includes: Preprocessing a preset number of image samples to obtain a third image; For each of the third images, matching the adhesion region position in the third image based on the template, converting the matching coordinates into a label file, and saving the label file; Based on the labeled image samples, train the target detection model; The step of preprocessing the collected X-ray image to obtain the first image includes: Using Gaussian filtering to remove Gaussian noise in the collected X-ray image to obtain a fourth image; Using median filtering to filter out salt and pepper noise in the fourth image to obtain a first image; The step of using the segmentation algorithm with an adaptive threshold to perform defect detection on the first image and the second image to determine the pixels corresponding to the target defects includes: Performing pixel-level difference processing on the first image and the second image to determine a defect score map; The image is segmented based on an adaptive threshold to determine the pixels corresponding to the target defects.

2. The method according to claim 1, characterized in that: Before the step of training the target detection model based on the labeled image samples, the method further includes: The labeled image samples are subjected to sample augmentation processing, wherein the sample augmentation processing includes: at least one of rotation, flipping, scaling, brightness adjustment, and contrast adjustment.

3. A device for detecting defects in chip adhesion areas, characterized in that: The device comprises: A first preprocessing module, used for preprocessing the collected X-ray image to obtain a first image; A positioning module, used to locate the position of the adhesion area in the first image using a trained object detection model; A repair module, used to repair the adhesion area defect in the first image based on an unsupervised image repair model to obtain a second image; A determination module, used to perform defect detection on the first image and the second image using a segmentation algorithm with an adaptive threshold, and determine pixels corresponding to target defects; The device also includes: A second preprocessing module, configured to preprocess a preset number of image samples to obtain a third image before the first preprocessing module preprocesses the collected X-ray image to obtain the first image; a labeling module, configured to, for each of the third images, match the position of the adhesion region in the third image based on the template, convert the matching coordinates into a label file, and save the label file; A training module is used to train the target detection model based on the labeled image samples; The first preprocessing module comprises: The first submodule is used to filter out Gaussian noise in the collected X-ray image by using Gaussian filtering to obtain a fourth image; The second submodule is used to filter out the salt and pepper noise in the fourth image by using median filtering to obtain the first image; The determination module comprises: The third submodule is used to perform pixel-level difference processing on the first image and the second image to determine a defect score map; The fourth submodule is used to segment the image according to the adaptive threshold value and determine the pixels corresponding to the target defect.

4. The device according to claim 3, characterized in that The device also includes: The amplification module is used to perform sample amplification processing on the labeled image samples before the training module trains the target detection model based on the labeled image samples, wherein the sample amplification processing includes: at least one of rotation, flipping, scaling, brightness adjustment and contrast adjustment.

Images