Program, computer, inspection system, and inspection method

Abstract

To provide a program, a computer, an inspection system, and an inspection method for reliably detecting an abnormality of a bump 103.SOLUTION: Reception means 32 receives an image 100 including an image 100a of a gap g between a substrate 101 and an object 102. Pre-processing means 34 performs: identifying a first boundary line b1 in the image 100 by using a first learned model 220 generated by machine learning with the use of teacher data 210 including the image which is the image of the gap between the substrate bonded by bumps and the object and also is the image for obtaining a first boundary line between the bumps and the object; and identifying a second boundary line b2 from the first boundary line b1, so as to include a lower end of the bumps 103 with the first boundary line b1 based on the size of the gap g. Determination means 36 determines whether the bumps 103 are normal from the image part 100a by using a second learned model 320 generated by machine learning with the use of teacher data 310 including the image of the bump being normal.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a program, a computer, an inspection system, and an inspection method. [Background technology]

[0002] Conventionally, when mounting a semiconductor chip on a substrate, the semiconductor chip is attached to the substrate by bumps, and whether the bumps are properly attaching the semiconductor chip to the substrate or not (checking the melted state and remaining amount of bumps, and judging the appearance such as gloss) has been visually inspected by a person using a microscope. For example, Patent Document 1 discloses a technique for magnifying and observing the main surface side of a chip on which bumps are formed using a microscope. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-270628 Summary of the Invention [Problem to be solved by the invention]

[0004] As such, in the past, whether or not bumps used to attach objects such as semiconductor chips to substrates were normal was inspected by a person using a microscope. However, since most bumps are normal to begin with and abnormalities rarely occur, there was a problem that abnormalities in the bumps were easily overlooked when inspected by a person visually.

[0005] The present invention has been made in consideration of the above points, and an object of the present invention is to provide a program, a computer, an inspection system, and an inspection method that can more reliably detect abnormalities in bumps. [Means for solving the problem]

[0006] The program of the present disclosure is A program that causes a computer to function as a receiving means, a preprocessing means, and a determining means, the receiving means receives an image including an image of a gap between a substrate and an object bonded to the substrate by a bump, the image being captured by an imaging device; the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the lower end of the bump is included between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including images of normal bumps to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; The first trained model is characterized by being trained to detect all bumps with bounding boxes of the same size and shape.

[0007] In the program of the present disclosure, The bounding box may be rectangular in shape.

[0008] In the program of the present disclosure, The bounding boxes do not have to overlap.

[0009] In the program of the present disclosure, The first learned model may identify the first boundary line based on the edges of all of the bounding boxes on the object side.

[0010] In the program of the present disclosure, The first trained model may identify the first boundary line for each predetermined area of ​​the image accepted by the accepting means.

[0011] In the program of the present disclosure, the second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image; the determining means uses the second trained model to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in an image including a real image of the bump and a mirror image of the bump projected onto the substrate, which image is received by the receiving means; The second boundary line may be a boundary line between a mirror image of the bump and a mirror image of the object in an image including a real image of the bump received by the receiving means and a mirror image of the bump projected onto the substrate.

[0012] In the program of the present disclosure, The second trained model may be generated by non-defective learning using only training data including images of normal bumps.

[0013] In the program of the present disclosure, The object may be a semiconductor chip.

[0014] The computer of the present disclosure includes: A computer that functions as a receiving means, a preprocessing means, and a determining means by executing a program, the receiving means receives an image including an image of a gap between a substrate and an object bonded to the substrate by a bump, the image being captured by an imaging device; the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the lower end of the bump is included between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including images of normal bumps to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; The first trained model is characterized by being trained to detect all bumps with bounding boxes of the same size and shape.

[0015] The inspection system of the present disclosure comprises: An inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, the imaging device acquires an image by capturing an image including an image of a gap between a substrate and an object bonded to the substrate by a bump; The computer functions as a receiving unit, a preprocessing unit, and a determining unit by executing a program; the receiving means receives an image captured by the imaging device, the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the lower end of the bump is included between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including images of normal bumps to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; The first trained model is characterized by being trained to detect all bumps with bounding boxes of the same size and shape.

[0016] The inspection method of the present disclosure includes: An inspection method using an inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, acquiring an image by the imaging device capturing an image including an image of a gap between a substrate and an object bonded to the substrate by bumps; a step in which the computer receives an image captured by the imaging device; a step in which the computer identifies a first boundary line from the received image using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the received image based on a predetermined size of the gap between the substrate and the object, so as to include a lower end of the bump between the first boundary line and the second boundary line; a step in which the computer determines whether the bump is normal or not from an image portion of the received image in a region between the first boundary line and the second boundary line, using a second trained model generated by machine learning using training data including an image of a normal bump; Equipped with The first trained model is characterized by being trained to detect all bumps with bounding boxes of the same size and shape.

[0017] In the inspection method of the present disclosure, the second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image; The imaging device may capture an image obliquely downward when capturing an image to be sent to the computer, and may obtain an image including a real image of the bump and a mirror image of the bump projected onto the substrate. [Effects of the Invention]

[0018] According to the program, computer, inspection system, and inspection method disclosed herein, abnormalities in bumps can be detected more reliably. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a diagram schematically illustrating a configuration of an inspection system according to an embodiment of the present disclosure. [Figure 2] 1 is a diagram illustrating an exemplary flow of a detection method and information processing in the detection method according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates an exemplary flow of information processing in an inspection system and detection method according to an embodiment of the present disclosure. [Figure 4] 1A and 1B illustrate exemplary arrangements of a substrate, object, gap, and bumps imaged in an inspection system and detection method according to an embodiment of the present disclosure. [Figure 5]1A-1C illustrate exemplary images captured and determined in an inspection system and detection method according to an embodiment of the present disclosure. [Figure 6] 10A and 10B are conceptual diagrams showing a method for identifying a first boundary line in an inspection system and a detection method according to a conventional example and an embodiment of the present disclosure. [Figure 7] 1A to 1C are diagrams for explaining a cutout process in an inspection system and a detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0020] Embodiments of the present disclosure will be described below, but the invention according to the present disclosure is not limited thereto. FIG. 1 is a diagram schematically illustrating the configuration of an inspection system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a detection method according to an embodiment of the present disclosure and an exemplary flow of information processing in the detection method. FIG. 3 is a diagram illustrating an inspection system according to an embodiment of the present disclosure and an exemplary flow of information processing in the detection method. FIG. 4 is a diagram illustrating an exemplary arrangement of a substrate, an object, gaps, and bumps imaged in the inspection system and detection method according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an exemplary image imaged and determined in the inspection system and detection method according to an embodiment of the present disclosure.

[0021] [Inspection System 1] FIG. 1 shows an inspection system 1 according to the present disclosure. The inspection system 1 more reliably detects whether bumps 103 (e.g., FIG. 4(B)) for bonding an object 102 such as a semiconductor chip to a substrate 101 are normal. As shown in FIG. 1, the inspection system 1 includes an imaging device 2 and a computer 3 to which an image 100 (e.g., FIGS. 5(A) and 5(B)) captured by the imaging device 2 is sent. The computer 3 and the imaging device 2 are connected so as to be able to communicate with each other. The computer 3 determines whether the bumps 103 on the image 100 captured by the imaging device 2 are normal (e.g., FIG. 5(C)). The inspection system 1 allows the process from imaging to determination to be performed automatically and more reliably than visual inspection.

[0022] FIG. 4A is a top view of a substrate 101 and an object 102 to be imaged, with the object 102 and an obstacle 104 disposed on the substrate 101. Examples of the object 102 to be bonded to the substrate 101 include semiconductor chips and chip molds. FIG. 4 shows an example of an FCCSP. As in this example, defective products can be excluded through a judgment inspection of the bonded state before underfill filling. Examples of the obstacle 104 disposed on the substrate 101 include, but are not limited to, chip capacitors. FIG. 4B is a diagram showing an example of the positional relationship between the substrate 101, the object 102, and bumps 103. The bumps 103 are interposed between the substrate 101 and the object 102, and the object 102 is bonded to the substrate 101 by the bumps 103. The example of the substrate 101 is not particularly limited. Furthermore, a metal layer or the like may be laminated on the substrate 101, or the substrate 101 may be laminated with an epoxy resin or the like. 4(A), the accuracy of the determination can be improved by using images from up to four directions in the determination process. The bumps 103 are the joints between the substrate 101 and the object 102, and are usually made of solder, but the bumps 103 are not limited to this.

[0023] The inspection system 1 can be incorporated into a manufacturing line and can determine the quality of the bumps 103 and therefore the target object 102 in real time.

[0024] <Imaging device 2> Next, the configuration of the imaging device 2 will be described. In the inspection system 1, the imaging device 2 acquires the image 100 by capturing an image 100 (an imaging range including the gap g) including an image 100a of a gap g (see FIG. 4(C)) between the substrate 101 and an object 102 bonded to the substrate 101 by a bump 103 (hereinafter, also simply referred to as "capturing an image of the gap g"). As shown in FIG. 1, the imaging device 2 includes a camera 21 and a communication unit 22.

[0025] There are no particular limitations on the camera 21, as long as it can capture an image of the gap g between the substrate 101 and the object 102 as shown in FIG. 4(C). Generally, information from at least 3 to 4 pixels is required for defect detection using image processing. Therefore, for example, if one wishes to visualize a 10 μm defect, this can be achieved by attaching a 1x telecentric lens to the camera 21 in relation to the size of the mounted CMOS (object 102), 2.7 μm / pix.

[0026] FIG. 4(C) shows the direction (indicated by the solid and dotted arrows) in which the imaging device 2 (camera 21) captures the gap g. While FIG. 4(C) illustrates an example in which no bumps 103 are present on the periphery of the object 102, the present invention is not limited to this example and can also be applied to cases in which bumps 103 are present on the periphery of the object 102. To determine whether the bumps used to bond the object to the substrate are normal, it is conceivable to capture the bumps 103 horizontally (sideways), i.e., from the front, as indicated by the dotted arrows in FIGS. 4(B) and 4(C). However, capturing the bumps 103 horizontally can be difficult if there is an obstacle 104, such as a chip capacitor, in the imaging direction. Therefore, as indicated by the solid arrow in FIG. 4(C), an image 100 to be sent to the computer 3 can be captured obliquely downward. The imaging angle (camera angle), i.e., "obliquely downward," is determined by the distance between the object 102 and the obstacle 104 and is not limited thereto. It is preferable to adjust the camera angle so that the light reflected by the obstacle 104 does not overlap with the bump 103 and so that the bump 103 is not hidden too much by the object 102.

[0027] The communication unit 22 includes a communication interface for transmitting and receiving signals to and from an external device wirelessly or via a wired connection. The image 100 captured by the camera 21 is transmitted by the communication unit 22 to the computer 3 (reception means 32).

[0028] <Computer 3> The configuration of the computer 3 of the present disclosure will be described with reference to Fig. 1. The computer 3 of this embodiment is configured as an industrial computer, a tablet terminal, or the like, and includes a control unit 30, a storage unit 40, a communication unit 42, a display unit 44, and an operation unit 46, as shown in Fig. 1.

[0029] The control unit 30 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an AI inference device, etc., and controls the operation of the computer 3. Specifically, the control unit 30 executes programs stored in a storage unit 40, which will be described later, to function as a reception unit 32, a preprocessing unit 34, a determination unit 36, and an output unit 38. Furthermore, the control unit 30 may execute programs stored in the storage unit 40 to function as a first model generation unit 200 or a second model generation unit 300. Each of these units will be described later.

[0030] The storage unit 40 is configured with, for example, a hard disk drive (HDD), a random access memory (RAM), a read-only memory (ROM), a solid state drive (SSD), etc. Furthermore, the storage unit 40 is not limited to being built into the computer 3, and may be a storage medium (for example, a USB memory) that is detachably attached to the computer 3. In this embodiment, the storage unit 40 is configured to store the program executed by the control unit 30, the first trained model 220, the second trained model 320, etc.

[0031] The communication unit 42 includes a communication interface for transmitting and receiving signals to and from an external device wirelessly or via a wire. The communication unit 42 allows the control unit 30 to transmit and receive signals to and from the imaging device 2.

[0032] The display unit 44 is, for example, a monitor or the like, and displays various screens in response to a display command signal received from the control unit 30. The operation unit 46 is, for example, a keyboard or the like, and is capable of issuing various commands to the control unit 30. Note that in this embodiment, a display operation unit such as a touch panel in which the display unit 44 and the operation unit 46 are integrated may also be used.

[0033] (Details of the control unit 30) (Reception means 32) The receiving means 32 receives an image 100 captured by the imaging device 2, the image 100 including an image 100a of a gap g between a substrate 101 and an object 102 bonded to the substrate 101 by a bump 103. Fig. 5(A) shows an example of the image 100 including the image 100a of the gap g.

[0034] (Pre-processing means 34) The preprocessing means 34 (and the preprocessing step in the present disclosure) aims to extract (mask) a determination region by identifying a first boundary line b1 and a second boundary line b2. The preprocessing means 34 identifies the first boundary line b1 in the image 100 accepted by the accepting means 32 using a first trained model 220 generated by machine learning using training data 210 including an image of a gap g between the substrate 101 and the object 102 bonded by the bumps 103, the image being an image from which the first boundary line b1 between the bumps 103 and the object 102 is obtained (process 51 for identifying the first boundary line b1 in FIG. 2 ), and identifies a second boundary line b2 from the first boundary line b1 in the image accepted by the accepting means 32 based on a preset size of the gap g between the substrate 101 and the object 102 so that the lower end of the bumps 103 are included between the first boundary line b1 and the second boundary line b2 (process 52 for identifying the second boundary line b2 in FIG. 2 ). The "first boundary line b1" and the "second boundary line b2" define the upper and lower ends of the image portion 100a that is the determination area.

[0035] ((First trained model 220)) The first trained model 220 is trained to detect all bumps 103 (which may include mirror images, as described below) using bounding boxes (BBs) of the same size and shape. Here, "detecting" a bump 103 using the first trained model 220 refers to identifying (annotating) each bump 103 that appears in the image 100 accepted by the accepting means 32 with a BB. This includes surrounding each bump 103 with a BB, including each bump 103 together with a margin (background), and partially identifying each bump 103 (see FIG. 6(B)). The shape of these BBs is preferably rectangular. It is also preferable that the BBs do not overlap each other. This annotation process includes surrounding a single standard bump 103 in the image 100 as the sole reference for the BB, and applying (detecting) this BB to all bumps 103 in the image 100 without changing the size or starting Y position (top edge). Here, the term "standard bump" is not particularly limited, but excludes bumps that are clearly larger, smaller, or more unusual than other bumps.

[0036] ((Identification of the first boundary line b1)) After detecting (annotating) all bumps 103 using the BBs, the first trained model 220 can identify the first boundary line b1 based on the edges (top edges) of all BBs facing the object 102. For example, if the heights of the captured bumps 103 are the same as shown in Figures 5(A) and 5(B), all bumps 103 are uniformly surrounded by the BBs, and the top edges of the BBs can be connected to form the first boundary line b1. Even if the heights of all bumps 103 are not the same, if all bumps 103 are detected using BBs of the same size and shape, the top edges of the BBs will be roughly aligned (Figure 6(B)). In this case, the first trained model 220 can identify the first boundary line b1 so that it passes through the top edge of the BB with the highest Y position (top edge) in the image 100 (e.g., the center of Figure 6(B)) or the top edge of the BB with the lowest Y position (top edge) (e.g., a BB other than the center of Figure 6(B)). It can also identify the first boundary line b1 so that it passes through the average height (Y position) of the top edges of the BBs. As long as the first boundary line b1 can be identified as described above, the machine learning method for generating the first trained model 220 is not particularly limited, and various methods such as deep learning can be used. The first trained model 220 is an object detection AI that precedes the second trained model 320 for pass / fail judgment, which will be described later, and is trained using various images (which may include mirror images) containing the bumps described above as training data 210. Note that the first boundary line b1 is not necessarily identified in the image serving as training data 210 from which the first boundary line b1 between the bump 103 and the object 102 is obtained. As will be described later, the image serving as training data 210 shows BB. In other words, the "image from which the first boundary line b1 can be obtained" refers to an image from which the first boundary line b1 would be obtained by the preprocessing means 34 of the present disclosure.

[0037] ((Identification of the second boundary line b2)) The phrase "determining the second boundary line b2 based on the size of the gap g between the substrate 101 and the object 102" means that, once the first boundary line b1 (upper edge), which is the boundary line between the bump 103 and the object 102, is known, the second boundary line b2 (lower edge) can be derived from the size of the gap g. Here, the size of the gap g can be defined based on the number of pixels (pix) in the image. Furthermore, the setting for deriving the second boundary line b2 (lower edge), which includes the lower edge of the bump 103, from the image received by the receiving unit 32 can be arbitrarily changed depending on the type, purpose, and operation method of the object 102 and the bump 103. For example, if only rough defects are desired, the amount of information can be reduced (the second boundary line b2 can be determined to be in the upper position), or, as described below, a larger amount of mirror image information can be included to capture information not visible in the actual image (the second boundary line b2 can also be determined to be in the lower position; see b2' in Figure 5(B)). 5(A), which is an exemplary side view (one-directional side view) of the substrate 101, the object 102, etc., captured from the direction indicated by the dotted arrow in FIG. 4(C), shows exemplary first boundary lines b1 and second boundary lines b2 identified by the preprocessing means 34, as well as an image portion 100a of the area between them. In this image portion 100a, the upper and lower ends of all bumps 103 are in contact with the first boundary line b1 and the second boundary line b2. As in this example, in order to remove as much unnecessary information as possible from the object 102, the substrate 101, etc., it is particularly preferable to identify the first boundary line b1 and the second boundary line b2 so that they are in contact with the upper and lower ends of the bumps 103.

[0038] ((The significance of identifying the first boundary line b1 and the second boundary line b2)) In the present invention, the reason for identifying the first boundary line b1 and the second boundary line b2 before the determination process is as follows: In the image 100, the areas (amount of information) of the object 102 and the substrate 101 are larger than the area (amount of information) of the bump 103. Therefore, if the determination area remains the image 100, it would be difficult for the AI ​​to determine the bump 103 portion. In particular, if the cut surface of the object 102 is not managed during the manufacturing process, the appearance of the object 102 in the image will vary, making the difference between normal and abnormal bumps 103 more noticeable, resulting in false detection or false determination. Therefore, the first trained model 220 according to the present invention identifies the first boundary line b1 and the second boundary line b2 in order to perform masking of the bump 103 in the image 100.

[0039] Typically, when creating a learning model for object detection, bumps 103 on the screen are circled one by one with a mouse. The inventor attempted to create a learning model using this conventional manual method, but the first boundary line b1 (multiple bumps 103) was not neatly aligned. In other words, the more precisely the human tried to circle the bumps and the more samples were used, the more uneven the first boundary line b1 became (Figure 6(A)). The inventor then discovered two main challenges in aligning the first boundary line b1 for masking (identifying the first boundary line b1). First, there is significant individual variation between the object 102 (chip) and the substrate 101, and in some cases, they may be warped (not perfectly flat). Also, differences in the amount of solder on the left and right bumps 103 can cause the object 102 to tilt relative to the substrate 101. Therefore, the first boundary line b1 cannot be simply determined by drawing a straight line between the left and right edges of the chip. Another reason is that there are variations among the individual bumps 103 (even the same type of bumps can vary across all four sides, and even on a single side), and naturally the appearance of the glossy portion illuminated by the imaging light will differ. In other words, there are areas behind the glossy portion (dark areas) that are not illuminated, and an example of this area is the top (rounded part) of the bump 103. Furthermore, there are cases where the bump 103 is imaged in different positions at the front and back. In this way, the joint between the bump 103 and the object 102 may not be captured in the image, and the first boundary line b1 does not exist between all of the bumps 103 and the object 102.

[0040] Therefore, after extensive research, the inventors changed their approach and decided to circle only one bump 103 with the mouse (Figure 6(B)). Then, using such a reference value bump 103 from various images as training data 210, they constructed a first trained model 220 and found that the first boundary line b1 (multiple bumps 103) aligned. In other words, when this first trained model 220 was used, the inventors found that the first boundary line b1 aligned as if the reference value were sliding from one end of the chip to the other. This is thought to be because encircling each bump 103, most of which are non-defective, ultimately increased the influence of human error. While Figures 6(A) and (B) show an example in which a mirror image is also reflected, as in Figure 5(B), the presence or absence of a mirror image is not necessary. Furthermore, to identify the first boundary line b1, it is preferable that the judgment area of ​​the bump 103 be as large as possible, extending to the top (rounded portion).

[0041] ((Other processing)) Furthermore, by identifying the first boundary line b1 with high accuracy, in addition to the masking process, the position of the inspection image to be input to the second trained model 320 may also be corrected. For example, the Y coordinates (vertical direction) of the first boundary line b1 and the second boundary line b2 in the image may be corrected. This allows for more accurate determination by the anomaly detection AI of the second trained model 320.

[0042] ((Cut-out processing)) However, if the left and right edges of the object 102 or the substrate 101 are not horizontal, it may be difficult to align the first boundary line b1 when using the captured image 100 (one field of view). Therefore, the first trained model 220 may specify the first boundary line b1 for each predetermined region (e.g., a region of Y pixel (pix) length in FIG. 7) of the image 100 accepted by the accepting means 32. That is, the difference between the left and right edges of the object 102 or the substrate 101 can be reduced by cropping the learning area or the judgment area.

[0043] This cropping process may be performed as follows: First, a feature point search is performed using pattern matching. The "feature point" here refers to, but is not limited to, a unique shape found in the image 100. Depending on the product type, a feature that is unique to the bump array (e.g., different gaps between adjacent bumps 103) may be used as a feature point. Next, a cropping region is determined based on the searched feature points. Specifically, the left and right end positions of the cropping region (i.e., the relative distances from the feature point in the X coordinate) are determined at positions offset (relative) by a predetermined pixel length from the detected X coordinate of the feature point in the image 100 (for example, if the detected feature point is in the center of the image, the start point of the cropping region can be expressed as minus X and the end point as plus X). Next, the distance from the left end to the right end of the cropping region is set to a fixed size (Y pixel length in Figure 7), and the cropping region is sequentially cropped from the image 100 while overlapping. In Figure 7, Z pixel indicates the length of the overlapping region. The values ​​of "Y" and "Z" can be set appropriately according to the size of the bump 103, the image 100, etc. Next, in each cut-out region of Y pixel length, the bump 103 is detected, and the first boundary line b1 and the second boundary line b2 are identified and masked. After the masking process, each cut-out region may be further divided into two, left and right, depending on the size of the defect to be determined and the resolution of the camera 21 (with a predetermined area overlapping in the center), and used as the determination region. This division process can avoid a situation where the defect becomes invisible due to reduction when there is a limit to the input size to the AI ​​(first trained model 220).

[0044] (Determination means 36) The determination means 36 uses the second trained model 320 generated by machine learning using training data 310 including an image of a normal bump 103 to determine whether the bump 103 is normal from the image portion 100a between the first boundary line b1 and the second boundary line b2 of the image 100 received by the receiving means 32 (determination process 53 in FIG. 2). This "determination" corresponds to the appearance inspection originally performed by humans, and specifically includes checking the melting state and remaining amount of the bump 103, and determining the appearance such as gloss. Various types of machine learning, such as deep learning, can be used to generate the second trained model 320. Note that FIG. 5(C) shows examples of bumps 103 determined to be normal (OK) and abnormal (NG).

[0045] In this way, instead of using the image 100 itself accepted by the accepting means 32, the judgment range is limited by masking, and the judgment means 36 makes judgment from the image portion 100a (FIG. 5(A)) in the region (gap g) between the first boundary line b1 and the second boundary line b2. Alternatively, the image 100a may be further divided (cut out) so that each divided image contains only one bump 103. This division method is not particularly limited, and the aforementioned cutting process may be performed. In this way, it is possible to prevent the features of the substrate 101 and the object 102 captured in the image 100 from being detected in error. More specifically, the second trained model 320 is used for judgment to detect the position of the bump 103 from the image portion 100a, and erroneous judgment can be prevented by AI object detection.

[0046] If the substrate 101 reflects light during imaging and projects a mirror image (virtual image) of the bumps 103, the second trained model 320 may be generated by machine learning using an image of a normal bump 103 that includes both a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image. As shown in the example of FIG. 4(C) with the solid arrow, when an image of the gap g is captured diagonally downward to inspect the bumps 103, the mirror image of the bumps 103 projected onto the substrate 101 is also captured and can be used for judgment. In the example shown in FIG. 5(B), each bump 103 is reflected in the image 100 in a daruma shape. That is, when the gap g is imaged obliquely downward, the area above the dotted line between b1' and b2' shows the real image of the bump 103, and the area below the dotted line shows the mirror image of the bump 103. (As shown in FIG. 4(C), when the gap g is imaged obliquely downward without any bumps 103 present on the periphery of the object 102, the upper ends of the bumps 103 are shown with dotted lines to show that the upper ends of the real images of the bumps 103 are hidden by the object 102.) Here, as shown in FIG. 5(B), the upper ends of the bumps 103, i.e., part of the area of ​​the bumps 103 in the real image, may not be visible (the amount of information is small). Even in this case, b1' between the hidden upper ends of the bumps 103 and the object 102 in the image 100 may be specified as the first boundary line, and b2' may be specified as the second boundary line so that the mirror image area is also included in the determination area. This increases the amount of information indicating the difference between good and bad bumps 103, improving or maintaining high judgment accuracy. Therefore, when a mirror image is also captured as in the example shown in FIG. 5B, the second boundary line b2 can be a second boundary line b2' indicating the boundary line between the mirror image of the bump 103 and the mirror image of the object 102, in order to improve judgment accuracy by utilizing the mirror image. In other words, the image portion 100a' serving as the judgment area in this case corresponds to the area between the first boundary line b1' based on the real image and the second boundary line b2' based on the mirror image. This second boundary line b2' can also be specified based on the size of the gap g between the substrate 101 and the object 102, which is set in advance.

[0047] Alternatively, the second trained model 320 may be generated by non-defective product training using only the training data 310 including images of normal bumps 103 (as described above, the upper ends of the bumps 103 may be hidden and / or may include a mirror image of the bumps 103). The determination means 36 may use the second trained model 320 described above to determine whether the bumps 103 are normal or not from the image portion 100a of the region between the first boundary line b1 or b1' and the second boundary line b2 or b2' in the image 100 including the real image of the bumps 103 received by the receiving means 32 and the mirror image of the bumps 103 projected onto the substrate 101.

[0048] If the determination is impossible, the bump 103 is determined to be abnormal (NG).

[0049] (output means 38) The output means 38 outputs the determination result via the display unit 44 .

[0050] (First model generation means 200, second model generation means 300) The first model generation means 200 and the second model generation means 300 generate the first trained model 320 and the second trained model 220 through machine learning using the teacher data 210 and 310, respectively (FIG. 2). The first trained model 220 and / or the second trained model 320 may function by a device other than the computer 3 executing a predetermined program, or may function by the control unit 30 executing a program stored in the memory unit 40. In the former case, the first trained model 220 and / or the second trained model 320 generated by the other device are transmitted to the computer 3 and stored in the memory unit 40 of the computer 3.

[0051] [Detection method] Next, a description will be given of a detection method using such a computer 3 (inspection system 1). Note that the following processing is performed by the control unit 30 executing a program stored in the storage unit 40.

[0052] First, the imaging device 2 captures an image 100 by capturing an image of the gap g between the substrate 101 and the object 102 bonded to the substrate 101 by the bumps 103 (FIG. 3, step S1). Here, as described above, the imaging device 2 can capture an image obliquely downward when capturing an image to be sent to the computer 3, thereby capturing an image including a real image of the bumps 103 and a mirror image of the bumps 103 projected onto the substrate 101.

[0053] Next, the computer 3 (accepting means 32) accepts the image captured by the imaging device 2 (FIG. 3, step S2).

[0054] Next, the computer 3 (preprocessing means 34) identifies the first boundary line b1 from the received image 100 using a first trained model 220 generated by machine learning using training data including an image of the gap g between the substrate 101 and the object 102 bonded by the bump 103, the image being an image from which the first boundary line b1 between the bump 103 and the object 102 is obtained (FIG. 2, process 51 for identifying the first boundary line b1). Based on the preset size of the gap g between the substrate 101 and the object 102, the computer 3 identifies the second boundary line b2 from the first boundary line b1 in the received image 100 so that the lower end of the bump 103 is included between the first boundary line b1 and the second boundary line b2 (FIG. 2, process 51 for identifying the second boundary line b2; FIG. 3, step S3). As described above, the first trained model 220 has been trained to detect all bumps 103 using bounding boxes of the same size and shape. In this way, an image portion 100a that becomes a determination area is extracted from the first boundary line b1 and the second boundary line b2. Furthermore, as preprocessing for the image 100, angle measurement and rotation correction by contour extraction may be performed so that the substrate 101 and the object 102 are horizontal, or the image 100 may be enlarged.

[0055] Next, the computer 3 (determination means 36) determines whether the bump 103 is normal or not from the image portion 100a of the region between the first boundary line b1 and the second boundary line b2 of the received image, using the second trained model 320 generated by machine learning using training data including an image of the normal bump 103 (FIG. 3, step S4). Note that, as described above, the second trained model 320 may be generated by machine learning using an image of the normal bump 103 that includes both the real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image.

[0056] Next, the computer 3 (output means 38) outputs the determination result via the display unit 44. The computer 3 (output means 38) may also transmit the determination result via the communication unit 42 to an external device.

[0057] According to the program, computer 3, inspection system 1, and inspection method of the present disclosure configured as described above, the inspection system 1 includes an imaging device 2 and a computer 3 to which an image captured by the imaging device 2 is sent. The imaging device 2 acquires the image 100 by capturing the image 100 including an image 100a of the gap g between the substrate 101 and the object 102 bonded to the substrate 101 by the bumps 103. The computer 3 (controller 30) executes the program to function as a receiving means 32, a preprocessing means 34, and a determination means 36. The receiving means 32 receives the image 100 captured by the imaging device 2, the image 100a including the image 100a of the gap g between the substrate 101 and the object 102 bonded to the substrate 101 by the bumps 103. The preprocessing means 34 uses a first trained model 220 generated by machine learning using training data including an image of the gap g between the substrate 101 and the object 102 bonded by the bump 103, where the first boundary line b1 between the bump 103 and the object 102 is obtained, to identify the first boundary line b1 in the image 100 received by the receiving means 32, and also identifies a second boundary line b2 from the first boundary line b1 in the image 100 received by the receiving means 32, based on a preset size of the gap g between the substrate 101 and the object 102, so that the lower end of the bump 103 is included between the first boundary line b1 and the second boundary line b2. The determination means 36 uses a second trained model 320 generated by machine learning using training data including an image of a normal bump 103, to determine whether the bump 103 is normal from an image portion 100a of the region between the first boundary line b1 and the second boundary line b2 in the image 100 received by the receiving means 32. The first trained model 220 is trained to detect all bumps 103 with bounding boxes of the same size and shape.

[0058] Conventionally, a person has visually inspected bumps for bonding objects such as semiconductor chips to a substrate using a microscope to determine whether they are normal. However, because most bumps are normal and abnormalities are rare, there is a problem that abnormalities in the bumps are easily overlooked. In contrast, the program, computer 3, inspection system 1, and inspection method disclosed herein can detect abnormalities in bumps from a received image more reliably than visual inspection. In particular, according to the present invention, a first boundary line b1 and a second boundary line b2 are identified in a received image 100, and whether a bump 103 is normal is determined from an image portion 100a in the region between the first boundary line b1 and the second boundary line b2, thereby achieving efficient inspection.

[0059] In the program, computer 3, inspection system 1, and inspection method disclosed herein, the bounding box may be rectangular. Bounding boxes of various rectangular shapes may be set to fit the bumps 103.

[0060] Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, it is preferable that the bounding boxes do not overlap each other as a means of preventing unnecessary bounding boxes from being detected when identifying the first boundary line b1. Non-maximum suppression (NMS) can be used as a method of selecting or clarifying an appropriate BB for identifying the first boundary line b1 from the many BBs generated.

[0061] Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the first trained model 220 may identify the first boundary line b1 based on the sides (top sides) of all bounding boxes facing the object 102. In this way, the first boundary line b1 can be easily identified based on the top sides of each bounding box.

[0062] Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the first trained model 220 may identify the first boundary line b1 for each predetermined area of ​​the image 100 accepted by the accepting means 32. By trimming as described above, the image size can be adjusted, making it possible to ignore warping or tilting of the object 102 such as a chip (because short sections approximate a straight line).

[0063] Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the second trained model 320 is generated by machine learning by using an image of a normal bump 103 that includes both a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image, and the determination means 36 may use this second trained model 320 to determine whether the bump 103 is normal or not from the image portion 100a' of the area between the first boundary line b1 (b1') and the second boundary line b2' in the image 100 that includes the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 and that has been accepted by the acceptance means 32, where the second boundary line b2' is the boundary line between the mirror image of the bump 103 and the mirror image of the object 102 in the image 100 that includes the real image of the bump 103 and the mirror image of the bump 103 projected onto the substrate 101 and that has been accepted by the acceptance means 32 (Figure 5(B)). While it is possible to photograph the bump 103 from the side as shown in FIG. 4(B), photographing the bump 103 from the side may be difficult if there is an obstacle 104, such as a chip capacitor, in the image. Therefore, even if the image 100 is photographed diagonally downward as shown in FIG. 4(C), it is possible to determine whether the bump 103 is normal by using the actual image of the bump 103 contained in the image 100 and the mirror image of the bump 103 projected onto the substrate 101. In other words, the program, computer 3, inspection system 1, and inspection method disclosed herein enable more reliable determination even when there is an obstacle 104 that obstructs the imaging of the bump 103 or the gap g. Furthermore, by using the mirror image, the amount of information required for determination increases, improving the accuracy of the determination.

[0064] Furthermore, in the program, computer 3, inspection system 1, and inspection method disclosed herein, the second trained model 320 may be generated by non-defective product learning using only training data including images of normal bumps 103. The program, computer 3, inspection system 1, and inspection method disclosed herein are also suitable when the actual number of defective products is small and it is difficult to define defective products or when it is difficult to collect data on defective products.

[0065] Furthermore, in the program, computer 3, inspection system 1, and inspection method of the present disclosure, the object 102 may be a semiconductor chip. In this way, when there is another object (such as an obstacle 104) placed in addition to the object 102 on the substrate 101, the program, computer 3, inspection system 1, and inspection method of the present disclosure can be suitably applied.

[0066] Furthermore, in the inspection method of the present disclosure, the second trained model 320 is generated by machine learning using an image of a normal bump 103 that includes both a real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 in the same image, and the imaging device 2 may capture an image obliquely downward when capturing an image to send to the computer 3, thereby acquiring an image that includes both the real image of the bump 103 and a mirror image of the bump 103 projected onto the substrate 101 (see FIG. 4(C)). In this way, even if there is an obstacle 104 as shown in FIG. 4 that obstructs the imaging of the bump 103, it is possible to accurately determine whether the bump 103 is normal or not.

[0067] The computer 3, the program, and the detection method according to the present disclosure are not limited to the above-described aspects and combinations, and various modifications can be made.

[0068] For example, as described above, even when a mirror image is also included in the image 100, the preprocessing means 34 may identify the first boundary line b1 (b1') and the second boundary line b2' in the image 100, but it is sufficient that the height of the first boundary line b1 (b1') be specified so that it includes about half (neither too much nor too little) of the bump 103 in the mirror image. Because there is variation in the gap g even within the same sample, the first boundary line b1 (b1') and the second boundary line b2' cannot always be identified with high accuracy, and good judgment results can still be obtained in this case. [Explanation of symbols]

[0069] 1. Inspection system 2. Imaging device 21 Camera 22 Communications Department 3. Computer 30 Control Unit 32 Reception methods 34 Pretreatment means 36 Judgment means 38 Output Method 22a Model generation section 24a Reception 26a Estimation part 28a Output section 40 Storage section 42 Communications Department 44 Display section 46 Control section 100, 100a, 100a' images 101 Substrate 102 Object 103 Bump 104 Obstacles 200 First model generation means 210 Teacher Data 220 First trained model 300 second model generation means 310 Teacher Data 320 Second trained model b1, b1' 1st border b2, b2' second border g Gap

Claims

1. A program that causes a computer to function as a receiving means, a preprocessing means, and a determining means, the receiving means receives an image including an image of a gap between a substrate and an object bonded to the substrate by a bump, the image being captured by an imaging device; the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the second boundary line includes a lower end of the bump between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including an image of a normal bump to determine whether the bump is normal from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; The first trained model is trained to detect all bumps with bounding boxes of the same size and shape.

2. The program according to claim 1 , wherein the bounding box has a rectangular shape.

3. The program of claim 1 , wherein the bounding boxes do not overlap.

4. The program according to claim 1 , wherein the first learned model identifies the first boundary line based on the edges of all the bounding boxes on the object side.

5. The program according to claim 1 , wherein the first trained model identifies the first boundary line for each predetermined area of ​​the image accepted by the accepting means.

6. the second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image; the determining means uses the second trained model to determine whether the bump is normal or not from an image portion of a region between the first boundary line and the second boundary line in an image including a real image of the bump and a mirror image of the bump projected onto the substrate, which image is received by the receiving means; The program according to claim 1, wherein the second boundary line is a boundary line between the mirror image of the bump and the mirror image of the object in an image including the real image of the bump accepted by the accepting means and the mirror image of the bump projected onto the substrate.

7. The program according to claim 1 , wherein the second trained model is generated by non-defective learning using only training data including images of normal bumps.

8. The program according to claim 1 , wherein the object is a semiconductor chip.

9. A computer that functions as a receiving means, a preprocessing means, and a determining means by executing a program, the receiving means receives an image including an image of a gap between a substrate and an object bonded to the substrate by a bump, the image being captured by an imaging device; the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the second boundary line includes a lower end of the bump between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including an image of a normal bump to determine whether the bump is normal from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; The first trained model is trained to detect all bumps with bounding boxes of the same size and shape.

10. An inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, the imaging device acquires an image by capturing an image including an image of a gap between a substrate and an object bonded to the substrate by a bump; The computer functions as a receiving unit, a preprocessing unit, and a determining unit by executing a program; the receiving means receives an image captured by the imaging device, the preprocessing means identifies a first boundary line in the image received by the receiving means using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the image received by the receiving means based on a predetermined size of the gap between the substrate and the object, such that the second boundary line includes a lower end of the bump between the first boundary line and the second boundary line; the determination means uses a second trained model generated by machine learning using training data including an image of a normal bump to determine whether the bump is normal from an image portion of a region between the first boundary line and the second boundary line in the image received by the reception means; An inspection system, wherein the first trained model is trained to detect all bumps with bounding boxes of the same size and shape.

11. An inspection method using an inspection system including an imaging device and a computer to which an image captured by the imaging device is sent, acquiring an image by the imaging device capturing an image including an image of a gap between a substrate and an object bonded to the substrate by bumps; a step in which the computer receives an image captured by the imaging device; a step in which the computer identifies a first boundary line from the received image using a first trained model generated by machine learning using training data including an image of a gap between a substrate and an object bonded by a bump, the image being an image from which a first boundary line between the bump and the object can be obtained, and identifies a second boundary line from the first boundary line in the received image based on a predetermined size of the gap between the substrate and the object so as to include the lower end of the bump between the first boundary line and the second boundary line; a step in which the computer determines whether the bump is normal or not from an image portion of the received image in a region between the first boundary line and the second boundary line, using a second trained model generated by machine learning using training data including an image of a normal bump; Equipped with An inspection method, wherein the first trained model is trained to detect all bumps with bounding boxes of the same size and shape.

12. the second trained model is generated by machine learning using an image of a normal bump that includes a real image of the bump and a mirror image of the bump projected onto a substrate in the same image; The inspection method according to claim 11, wherein the imaging device captures the image to be sent to the computer obliquely downward, and obtains an image including a real image of the bump and a mirror image of the bump projected onto the substrate.

Images