Method and apparatus for detecting defects of bumps on substrate

Abstract

A method and apparatus for detecting defects of bumps on a substrate are provided. According to the embodiment of the invention, the method comprises the steps that an image of a substrate is acquired, and the substrate comprises salient points used for bonding; providing the image to a segmentation model to obtain an image of a salient point region; and providing the obtained image of the salient point area to an object detection model to determine whether the salient point has a defect or not.

Description

Technical Field

[0001] The present invention generally relates to the field of semiconductor technology, and more particularly, to a method and apparatus for detecting defects of bumps on a substrate. Background Art

[0002] In the semiconductor packaging process, the direct stiffener attachment (DSA) process is usually used to attach stiffeners to the four sides of the substrate to control the package warpage of the substrate. However, the direct stiffener attachment process will cause problems such as foreign matter, bridging and scratches on the bumps on the substrate. After the die is bonded to the substrate, foreign matter, bridging and scratches on the bumps may cause a short circuit, thereby making the bonded die scrapped.

[0003] To solve this problem, after the ribs are attached to the substrate and before the die is bonded to the substrate, the substrates with defective bumps are picked out by manual visual inspection. However, the size of the bumps continues to decrease with the advancement of technology, which greatly reduces the reliability of manual visual inspection. In addition, the overall defect rate of such bumps is usually not high, for example, its defects per million (DPM) may be around 300, which also makes manual visual inspection time-consuming and inefficient.

[0004] Currently, there are systems designed specifically for rib quality inspection, but there are no systems designed specifically for detecting defects in bumps on substrates. In addition, the pixels of the cameras used in systems for rib quality inspection are usually not high, so it is impossible to use the images captured by the camera to detect whether there are defects in the bumps on the substrate, let alone detect the type of defects. Figure 1 As shown in FIG. 1 , the photo is taken using a camera in a current system for inspecting the quality of reinforcing ribs. The reinforcing ribs 110 are attached around the substrate 120. During the attachment process of the reinforcing ribs 110, defects may be generated in the bump areas 130 and 140 of the substrate 120. However, it is difficult to use Figure 1 The photographs shown in are used to detect whether there are defects in the bumps in the bump areas 130 and 140, let alone the type of defects.

[0005] Therefore, there is a need for a method and apparatus for detecting defects in bumps on a substrate, which can use images captured by existing equipment to quickly and accurately detect various defects in bumps on the substrate, regardless of whether the pixels of the captured images are high or low. Summary of the invention

[0006] According to an embodiment of the present invention, a method for detecting defects in bumps on a substrate is provided, comprising: acquiring an image of the substrate, the substrate including bumps for bonding; providing the image to a segmentation model to acquire an image of the bump area; and providing the acquired image of the bump area to an object detection model to determine whether the bump has defects.

[0007] In one embodiment, determining whether the bump has a defect includes determining the location, type, and probability of the defect.

[0008] In one embodiment, the type includes one or more of foreign matter, bridges, scratches, and conductive adhesive on the bumps.

[0009] In one embodiment, if the size of the defect is smaller than a threshold, the method further includes: cutting out an image of a defect area whose size is smaller than the threshold in the output image of the object detection model; and providing the image of the defect area to a defect classification model to further determine the type of the defect.

[0010] In one embodiment, the segmentation model is any one of U-net, Mask R-CNN, and SOLO segmentation models.

[0011] In one embodiment, the object detection model is any one of the YOLO, SSD, and EfficientDet detection models.

[0012] In one embodiment, the defect classification model is any one of DenseNet, VGG, and MobileNet classification models.

[0013] In one embodiment, the segmentation model is obtained by training using a plurality of first sample images, and each first sample image includes a label value for indicating a position of a salient point region included in the first sample image.

[0014] In one embodiment, the object detection model is obtained by training with a plurality of second sample images, and each second sample image includes a label value indicating whether the second sample image contains a defect and the type and location of the defect when the defect is contained.

[0015] In one embodiment, the defect classification model is obtained by training with a plurality of third sample images, and each third sample image includes a label value indicating whether the third sample image contains a defect and the type and location of the defect when the defect is contained.

[0016] In one embodiment, each of the first sample image, the second sample image and the third sample image further includes a label value indicating a product name corresponding to each sample image, a lighting condition when each sample image is captured, and a substrate supplier.

[0017] In one embodiment, before providing the image of the substrate to the segmentation model, the method further comprises pre-processing the image.

[0018] In one embodiment, the pre-processing comprises calculating a rotation matrix of the image.

[0019] According to an embodiment of the present invention, there is provided an apparatus for detecting defects of bumps on a substrate, comprising: a memory; and a control unit, the control unit being coupled to the memory and configured to execute the method according to the present invention.

[0020] According to an embodiment of the present invention, there is provided a computer readable medium storing a computer program comprising instructions which, when executed by a control unit, cause the control unit to be configured to perform the method according to the present invention.

[0021] According to an embodiment of the present invention, a computer program product for detecting defects of bumps on a substrate is provided, comprising a computer program code executable by a processor, wherein the computer program code is used to execute the method according to the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments of the present disclosure and, together with the description, further serve to explain the principles of the present disclosure and enable those skilled in the relevant art to make and use the present disclosure.

[0023] Figure 1 A schematic diagram showing defects that may be generated on a bump of a substrate during the process of attaching a stiffener to the periphery of the substrate;

[0024] Figure 2 A flow chart of a method for detecting defects of bumps on a substrate according to an embodiment of the present invention is shown;

[0025] Figure 3(a)-3(b) A schematic diagram showing rotation processing of an image according to an embodiment of the present invention is shown;

[0026] Figure 4(a)-4(b) A schematic diagram showing segmentation processing of a substrate image using a segmentation model according to an embodiment of the present invention is shown;

[0027] Figure 5(a)-5(b)A schematic diagram showing an image of a salient point area after object detection processing is performed using an object detection model according to an embodiment of the present invention;

[0028] Figure 6(a)-6(d) A schematic diagram is shown of using a defect classification model to process an image of a defect region whose size is smaller than a threshold value determined by an object detection model to further determine the defect type according to an embodiment of the present invention.

[0029] Various embodiments will be described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0030] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and is not a limitation of the scope of protection, applicability or examples set forth in the claims. The function and arrangement of the elements discussed can be changed without departing from the scope of protection of the present disclosure. Various examples can omit, replace or add various processes or components as needed. For example, the described method can be performed in an order different from the described order, and various steps can be added, omitted or combined. In addition, the features described relative to some examples can also be combined in other examples.

[0031] It should be noted that the references to "one embodiment", "an embodiment", "some embodiments", etc. in the specification indicate that the described embodiments may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, such wordings do not necessarily refer to the same embodiment. In addition, when specific features, structures or characteristics are described in conjunction with an embodiment, it should be within the knowledge of technicians in the relevant field to implement such features, structures or characteristics in conjunction with other embodiments that are explicitly or not explicitly described.

[0032] Embodiments of a method and apparatus for detecting defects of bumps on a substrate according to the present invention will now be described with reference to the accompanying drawings.

[0033] Figure 2 A flow chart of a method 200 for detecting defects of bumps on a substrate according to an embodiment of the present invention is shown. Figure 3(a)-6(d) 2 to describe the method 200. It should be understood that the operations shown in the method 200 are not exclusive, and other operations may be performed before, after, or between any of the operations shown.

[0034] like Figure 2As shown in , method 200 starts at step S1, where an image of a substrate is obtained, the substrate including bumps for bonding. According to an embodiment of the present invention, the substrate can be photographed, for example, using a camera used in an existing system for detecting the quality of reinforcing ribs attached to the periphery of the substrate, but the present invention is not limited thereto, and the image of the substrate can also be obtained by any other device capable of photographing the substrate. The obtained image can be processed by the next steps of method 200 to determine whether the bumps on the substrate have defects such as scratches, foreign matter, bridging, conductive glue, etc. and the location of the defects. If it is determined that the bumps on the substrate have the above-mentioned defects (for example, foreign matter), the defective substrates should be picked out in time to avoid these defects affecting the final product quality (for example, causing a short circuit) during the subsequent process of bonding the die to the substrate.

[0035] Next, the method 200 proceeds to step S2, where the image obtained in step S1 is preprocessed. In some embodiments, the preprocessing includes denoising the obtained image. For example, a filter model-based method or a model-based method can be used to denoise the image.

[0036] In some embodiments, the preprocessing also includes rotating the obtained image to perform horizontal correction on the image. In the semiconductor packaging process, the substrate is usually placed in a tray pocket. During the transmission of the substrate, the substrate may rotate, as shown in Figure 3(a). Before processing the captured substrate image, such image needs to be rotated to the horizontal. According to an embodiment of the present invention, as shown in Figure 3(a), by identifying the reference circles 31 and 32 on the substrate 30, the rotation angle θ of the substrate 30 can be obtained. After obtaining the rotation angle θ, the rotation matrix M can be calculated. The substrate image can be rotated to the horizontal by the rotation matrix M, as shown in Figure 3(b).

[0037] In one embodiment, the reference circles 31 and 32 on the substrate 30 may be identified by Hough transformation, but the present invention is not limited thereto.

[0038] Next, the method 200 proceeds to step S3, where the image processed in step S2 is provided to the segmentation model to obtain an image of the bump region. As shown in FIG4(a), regions 41 and 42 are regions where bumps exist on the substrate, and are also regions of interest. By providing the substrate image to the trained segmentation model, an image of the bump region of interest can be obtained, as shown in FIG4(b). Figure 4(a)-4(b)As shown in FIG. 4 , only two bump regions 41 and 42 are shown. However, this is only indicative, and the number of bump regions on the substrate depends on the need for bonding with the die. For example, there may be more (e.g., three, four or more) bump regions on the substrate, and the present invention is not limited thereto.

[0039] According to an embodiment of the present invention, the segmentation model may be, for example, any one of U-net, Mask R-CNN (Region-based Convolutional Neural Network with Mask), and SOLO (Segmenting Objects by Locations) segmentation models, but the present invention is not limited thereto.

[0040] According to an embodiment of the present invention, the segmentation model can be obtained by training with multiple sample images, and each sample image can include a label value for indicating the position of the salient point area included in the sample image. During the training process, the model will continuously iteratively update the weights to better fit the training data. After the training is completed, the model can be evaluated using a validation set including label values ​​indicating the position of the salient point area, and the model can be optimized based on the evaluation results.

[0041] According to an embodiment of the present invention, int8 instead of float32 data type can be used to represent weight parameters in the training process, thereby greatly shortening the training process of the segmentation model.

[0042] According to an embodiment of the present invention, each sample image used for training the segmentation model may also include a label value for indicating the product name corresponding to each sample image, the lighting conditions when each sample image was collected, and the substrate supplier. According to an embodiment of the present invention, for example, training can be performed using substrate sample images of the same product of the same substrate supplier taken under the same lighting conditions to obtain a segmentation model of the corresponding substrate. After the training of the segmentation model is completed, the image of the convex point area of ​​interest can be obtained by simply providing the obtained substrate image to the trained segmentation model.

[0043] According to an embodiment of the present invention, by using a segmentation model trained with a large number of sample images to segment the convex point region of interest from the substrate image, not only the defect detection speed but also the defect detection accuracy can be improved. For example, if such segmentation processing is not performed, not only the detection speed is slow, but also various markers in the substrate image may affect the final detection result.

[0044] Next, the method 200 proceeds to step S4, where the acquired image of the salient point region is provided to the object detection model to determine whether the salient point has a defect. According to an embodiment of the present invention, when multiple salient point region images are obtained in step S3, they are provided to the trained object detection model respectively to determine whether the salient point in each salient point region has a defect.

[0045] According to an embodiment of the present invention, determining whether there is a defect in a bump on a substrate may include, for example, determining the location, type, and probability of the defect. For example, the type of defect may include one or more of foreign matter, bridging, scratches, and conductive adhesive on the bump, but the present invention is not limited thereto.

[0046] According to an embodiment of the present invention, the object detection model may be, for example, any one of the YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), and EfficientDet detection models, but the present invention is not limited thereto. For example, for the SSD detection model, it uses a multi-scale feature map for prediction, can detect targets of different sizes at the same time, and is fast; for the EfficientDet detection model, it combines EfficientNet and BiFPN to achieve efficient target detection. There are significant improvements in both speed and accuracy, but more adjustments are required. According to an embodiment of the present invention, a suitable object detection model can be selected according to the needs of detection.

[0047] According to an embodiment of the present invention, the object detection model can be obtained by training with multiple sample images, and each sample image can include a label value indicating whether the sample image contains a defect and the type and location of the defect when the defect is contained. According to an embodiment of the present invention, the sample image used for the object detection model is an image of the salient point area obtained by the segmentation model in step S3.

[0048] According to an embodiment of the present invention, each sample image used for object detection model training may also include a label value indicating a product name corresponding to each sample image, a lighting condition when each sample image was captured, and a substrate supplier.

[0049] According to an embodiment of the present invention, after the images of the plurality of convex point regions obtained in step S3 are respectively input into the trained object detection model, images with the defect type, location and probability of the defect being present are respectively output. As shown in FIG5(a), the image output by the object detection model indicates that the probability of a scratch defect being present in the convex point of the sub-region marked by the box in the convex point region of the substrate is 66%; in addition, as shown in FIG5(b), the image output by the object detection model indicates that the probability of a bridging defect being present in the convex point of the sub-region marked by the box in the convex point region of the substrate is 87%.

[0050] In some embodiments, before providing the image of the salient point region obtained by the segmentation model to the trained object detection model, the image of the salient point region may need to be processed to obtain an image of a suitable size (e.g., 640×640 pixels). To this end, the obtained image of the salient point region may be scaled, and in this process, in order to keep the image aspect ratio unchanged, the edge may be filled with an appropriate grayscale (e.g., about 140) using a letterbox.

[0051] Next, if the size of the defect determined in step S4 is smaller than a threshold, method 200 proceeds to step S5, where an image of a defect area whose size is smaller than the threshold in the output image of the object detection model is cut out; and the image of the defect area is provided to a defect classification model to further determine the type of the defect.

[0052] According to an embodiment of the present invention, the size of the defect may be determined, for example, according to Figure 5(a)-5(b) According to an embodiment of the present invention, the threshold of the defect may be related to, for example, the substrate supplier, the product packaged with the substrate, the direct stiffener attachment process parameters, etc., but the present invention is not limited thereto.

[0053] According to an embodiment of the present invention, further determining the type of the defect may, for example, include determining whether a defect exists, and if a defect exists, further determining the type of the defect.

[0054] When using a trained object detection model for defect detection, in addition to larger defects (e.g., scratch defects), some small defects (e.g., bridges, conductive adhesives, etc.) may be incorrectly labeled. For example, the object detection model may incorrectly label a single bright spot as a bridge defect, or incorrectly label a bridge defect as conductive adhesive, etc. In order to avoid incorrectly identifying these smaller defects, the image of the defect area whose size is smaller than a threshold in the output image of the object detection model can be cut out; and the image of the defect area is provided to the defect classification model to further determine the type of the defect.

[0055] As shown in FIG6(a), after the defect area image 61 whose size is less than the threshold and marked as a bridging defect in the output image of the object detection model is cut out and provided to the trained defect classification model, the defect classification model confirms the existence of a bridging defect, and FIG6(b) shows that a bridging defect does exist in the corresponding bump area of ​​the substrate. However, as shown in FIG6(c), after the defect area image whose size is less than the threshold and marked as a bridging defect in the output image of the object detection model is cut out and provided to the trained defect classification model, the defect classification model confirms that there is no bridging defect, and FIG6(d) shows that there is no bridging defect in the corresponding bump area of ​​the substrate, and it is actually just a single bright spot. Therefore, according to an embodiment of the present invention, defects whose size is less than the threshold can be further confirmed by the defect classification model, thereby avoiding the false detection of such small-sized defects, thereby greatly improving the detection accuracy.

[0056] According to an embodiment of the present invention, the defect classification model is a lightweight weight classification deep learning model. For example, the defect classification model is any one of DenseNet, VGG (Visual Geometry Group), and MobileNet classification models, but the present invention is not limited thereto.

[0057] According to an embodiment of the present invention, the defect classification model can be obtained by training, for example, using multiple sample images, and each sample image can include a label value indicating whether the sample image contains a defect and the type and location of the defect when the defect is contained. According to an embodiment of the present invention, the sample image used for the defect classification model is an image of a defect area whose size is smaller than a threshold in the output image of the object detection model.

[0058] According to an embodiment of the present invention, each sample image used for defect classification model training may also include a label value indicating a product name corresponding to each sample image, a lighting condition when each sample image is collected, and a substrate supplier.

[0059] According to the method of the embodiment of the present invention, it is possible to use the substrate image captured by existing equipment (for example, a camera in a system for reinforcing rib quality inspection) to quickly and accurately detect various defects of bumps on the substrate without the need for separate hardware equipment, even if the pixel of the captured substrate image is not high.

[0060] According to an embodiment of the present invention, there is also provided a device for detecting defects of bumps on a substrate, comprising: a memory; and a control unit, the control unit being coupled to the memory and configured to perform the method according to the present invention. The control unit according to an embodiment of the present invention can be implemented using electronic hardware, computer software, or any combination thereof. Whether the control unit is implemented as hardware or software will depend on the specific application. As an example, the control unit according to the present invention can be implemented as a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, a gate logic, a discrete hardware circuit, and other suitable processing components configured to perform the various functions described in the present disclosure. The control unit according to an embodiment of the present invention can also be implemented as software executed by a microprocessor, a microcontroller, a DSP, or other suitable platforms.

[0061] According to one embodiment, a computer-readable storage medium is provided, on which a program code is stored. When the program code is executed by a processor, the processor is enabled to perform the Figure 2-6(d) Specifically, a system or device equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system or device reads and executes the instructions stored in the readable storage medium.

[0062] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A method for detecting defects of bumps on a substrate, comprising: Acquiring an image of the substrate, wherein the substrate includes bumps for bonding; Providing the image to a segmentation model to obtain an image of a salient point region; as well as The acquired image of the salient point area is provided to the object detection model to determine whether the salient point has defects.

2. The method according to claim 1, wherein: Determining whether the bump has a defect includes determining the location, type, and probability of the defect.

3. The method according to claim 2, wherein: The types include one or more of foreign matter, bridges, scratches and conductive glue on the bumps.

4. The method according to claim 3, if the size of the defect is smaller than a threshold value, the method further comprises: Cut out an image of a defect area whose size is smaller than the threshold value in the output image of the object detection model; as well as The image of the defect area is provided to a defect classification model to further determine the type of the defect.

5. The method according to claim 1, wherein: The segmentation model is any one of U-net, Mask R-CNN, and SOLO segmentation models.

6. The method according to claim 1, wherein: The object detection model is any one of the YOLO, SSD, and EfficientDet detection models.

7. The method according to claim 4, wherein: The defect classification model is any one of DenseNet, VGG, and MobileNet classification models.

8. The method according to claim 5, wherein: The segmentation model is obtained by training with a plurality of first sample images, and each first sample image includes a label value for indicating a position of a salient point area included in the first sample image.

9. The method according to claim 6, wherein: The object detection model is obtained by training with a plurality of second sample images, and each second sample image includes a label value indicating whether the second sample image contains a defect and the type and position of the defect when the defect is contained.

10. The method according to claim 7, wherein: The defect classification model is obtained by training with a plurality of third sample images, and each third sample image includes a label value indicating whether the third sample image contains a defect and the type and position of the defect when the defect is contained.

11. The method according to any one of claims 8 to 10, wherein: Each of the first sample image, the second sample image and the third sample image further includes a label value for indicating a product name corresponding to each sample image, a lighting condition when each sample image is captured, and a substrate supplier.

12. The method according to claim 1, wherein: Prior to providing the image of the substrate to the segmentation model, the method further comprises pre-processing the image.

13. The method according to claim 12, wherein: The pre-processing includes calculating a rotation matrix of the image.

14. A device for detecting defects of bumps on a substrate, comprising: Memory; as well as A control unit is coupled to the memory and is configured to execute the method according to any one of claims 1-13.

15. A computer readable medium storing a computer program comprising instructions which, when executed by a control unit, cause the control unit to be configured to perform the method according to any one of claims 1-13.

16. A computer program product for detecting defects of bumps on a substrate, comprising a computer program code executable by a processor, wherein the computer program code is used to execute the method according to any one of claims 1 to 13.

Images