Contour extraction system and contour extraction method

Abstract

This system provides a contour extraction system that can more accurately detect the contours of objects. [Solution] The contour extraction system 10 for extracting the contour of an object 100 from an image of the object 100 comprises an imaging device 14 for imaging the object 100, an illumination unit 16 for illuminating the object 100, the illumination unit 16 being able to switch the illumination light between a first light having a first color and a second light of a second color different from the first light, and a controller 20 for calculating the contour of the object 100 from the image, wherein the controller 20 is configured to extract the contour of the object 100 in the image based on a comparison result between the brightness value Lr in a first image 30R obtained by imaging under the first light and the brightness value Lb in a second image 30B obtained by imaging under the second light.

Description

Technical Field

[0001] This specification discloses a contour extraction system and a contour extraction method for extracting the contour of an object in an image obtained by imaging the object.

Background Art

[0002] Conventionally, many techniques for extracting the contour of an object based on an image obtained by imaging the object have been proposed. For example, a technique for extracting the contour of an object by binarizing an image obtained by imaging the object with a luminance value is known.

[0003] However, when only binarized with a luminance value, there are cases where the contour of the object cannot be clearly extracted due to the influence of the shadow of other members, dirt, etc. Therefore, in some cases, techniques for improving the extraction accuracy of the contour by changing the color of the light irradiated on the object have been proposed.

[0004] For example, in Patent Document 1, a technique for changing the color of the light irradiated on the visual field according to the inspection content is disclosed. Further, in Patent Document 2, a technique for imaging one object a plurality of times by changing the color of the light, binarizing each image, and synthesizing the obtained plurality of binarized images is disclosed. Furthermore, in Patent Document 3, a technique for imaging an object by changing the color of the light, detecting an inspection region based on one of the obtained two images, and detecting the presence or absence of a defect based on the other is disclosed.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

[0006] However, the technologies described in Patent Documents 1 to 3 made it difficult to accurately detect the contours of objects. Therefore, this specification discloses a contour extraction technology that can more accurately detect the contours of objects. [Means for solving the problem]

[0007] The contour extraction system disclosed herein is a contour extraction system for extracting the contour of an object from an image of the object, comprising: an imaging device for imaging the object; an illumination unit for illuminating the object, the illumination unit being able to switch the illumination light between a first light having a first color and a second light of a second color different from the first light; and a controller for calculating the contour of the object from the image, wherein the controller is configured to extract the contour of the object in the image based on a comparison result between the brightness value in a first image obtained by imaging under the first light and the brightness value in a second image obtained by imaging under the second light.

[0008] In this case, the controller may be configured to store in advance as a reference value the correspondence between the brightness value of an image obtained by imaging an object of the same quality as the object under the first light and the brightness value of an image obtained by imaging the same object under the second light, calculate as a corresponding value the correspondence between the brightness value of each of the multiple pixels in the first image and the brightness value of each of the multiple pixels in the second image, and extract the contour of the object in the image based on a comparison between the reference value and the corresponding value.

[0009] Furthermore, the controller may be configured to calculate the similarity between the reference value and the corresponding value for each of the multiple pixels, and to extract the contour of the object in the image based on the binarized image obtained by binarizing the image based on the similarity.

[0010] Furthermore, at least one of the first color and the second color may be a color of the same type as the color of the object, but may not be a color of the same type as the color of any non-objects that appear in the image together with the object.

[0011] In this case, the object is made of gold or copper, the non-object is made of aluminum, the first color is a reddish color, and the second color may be a blued color.

[0012] Furthermore, the object may be a crimp ball formed on a semiconductor device, and the non-object may be an aluminum pad to which the crimp ball is pressed.

[0013] Furthermore, the object is a crimp ball formed on a semiconductor device, and the non-object that appears in the image together with the object is an aluminum pad to which the crimp ball is crimped. The controller may be configured to extract the contour of the ball neck interposed between the crimp ball and the bonding wire based on the first image or the second image, and to consider the corresponding value inside the contour of the ball neck as the same as the reference value.

[0014] Furthermore, the contour extraction method for extracting the contour of an object from an image of the object is characterized by: acquiring a first image by illuminating the object with a first light having a first color; acquiring a second image by illuminating the object with a second light of a second color different from the first color; and extracting the contour of the object in the image based on the comparison result between the brightness value in the first image and the brightness value in the second image. [Effects of the Invention]

[0015] The technique disclosed herein allows for more accurate detection of the contour of an object by utilizing the comparison result between the brightness value of the first image and the brightness value of the second image. [Brief explanation of the drawing]

[0016] [Figure 1]It is a schematic diagram showing the configuration of the contour extraction system. [Figure 2] It is a diagram showing an example of the first image and the second image. [Figure 3] It is a diagram with explanatory lines added to the first image. [Figure 4] It is a diagram showing the correspondence between the luminance value in red light and the luminance value in blue light for gold and aluminum respectively. [Figure 5] It is a diagram showing a reference value and an example of the similarity C based on the reference value. [Figure 6] It is a schematic diagram showing the flow of contour extraction. [Figure 7] It is a diagram showing an example of a normalized image. [Figure 8] It is a diagram showing an example of a binary image. [Figure 9] It is a flowchart showing the flow of contour extraction.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, referring to the drawings, the configuration of the contour extraction system 10 will be described. FIG. 1 is a schematic diagram showing the configuration of the contour extraction system 10. This contour extraction system 10 images the object 100 and extracts the contour of the object 100 based on the obtained image. The contour extraction system 10 in this example extracts the contour of the bonding ball 102 formed on the semiconductor device as the object 100. As will be described in detail later, the bonding ball 102 is a portion where the end of the bonding wire connecting between the electrodes is pressure-bonded to the pad 112 that functions as an electrode.

[0018] The contour extraction system 10 comprises a microscope 12, an imaging device 14, an illumination unit 16, and a controller 20. The microscope 12 magnifies the image of the fine crimped ball 102 so that it can be observed with the naked eye. Such a microscope 12 is, for example, an optical microscope. The imaging device 14 captures the image of the crimped ball 102 obtained by the microscope 12 and outputs image data. Such an imaging device 14 has an image sensor such as a CMOS or CCD. In this example, the imaging device 14 is a monochrome imaging device that generates grayscale image data. However, naturally, the imaging device 14 may also be a color imaging device that generates color image data. The image captured by this imaging device 14 is transmitted to the controller 20.

[0019] The irradiation unit 16 irradiates light onto the object 100, which is the crimped ball 102. This irradiation unit 16 may be incorporated inside the microscope 12 or placed outside the microscope 12. In either case, the irradiation unit 16 is positioned and oriented so as to irradiate the entire object 100.

[0020] The irradiation unit 16 is capable of emitting at least a first light, which is a first color, and a second light, which is a second color different from the first color. The colors of the first and second lights are not particularly limited, as long as they are different from each other. Typically, the first and second colors are colors that are clearly recognized as "different colors." For example, the first and second colors are colors that are located in different regions when the color wheel is divided into three or four equal parts. In this example, a reddish color is used as the first color, and a blued color is used as the second color. Therefore, the irradiation unit 16 in this example is capable of emitting red and blue light. In order to selectively emit two different colors of light, the irradiation unit 16 may have two types of light sources. For example, the irradiation unit 16 may have a red LED and a blue LED. Alternatively, the irradiation unit 16 may have a color filter placed in the optical path of the light from the light source that changes the color of the transmitted light. In any case, the irradiation unit 16 changes the color of the light irradiated onto the crimping ball 102 in response to a command from the controller 20.

[0021] The controller 20 is physically a computer having a processor 22 and memory 24. Although Figure 1 shows the controller 20 as a single computer, the controller 20 may be configured by combining multiple physically separated computers. In addition to the processor 22 and memory 24, the controller 20 may also have a communication interface for exchanging data with external devices and a user interface for presenting information to the user or receiving operation instructions from the user.

[0022] The controller 20 analyzes the image captured by the imaging device 14 and extracts the contour of the object 100, which is the crimped ball 102. In order to accurately extract the contour, the controller 20 drives the imaging device 14 and the illumination unit 16 to acquire a first image 30R captured under red light illumination and a second image 30B captured under blue light illumination. The controller 20 then extracts the contour of the object 100 by comparing the brightness values ​​Lr and Lb of the first image 30R and the second image 30B, respectively. The principle of this contour extraction will be explained below.

[0023] First, let's describe the crimp ball 102, which is the object 100 in this example. The crimp ball 102 is formed by melting the end of the bonding wire 106 to create a sphere, which is then pressed onto a metal plate called a pad 112. Therefore, the crimp ball 102 itself is a flat disc shape. There is also a roughly conical portion between the crimp ball 102 and the bonding wire 106 that is formed when pressed with a capillary (not shown). Hereafter, this roughly conical portion will be referred to as the ball neck 104. In this example, the pad 112 is made of aluminum, and the bonding wire 106, which includes the crimp ball 102, is made of gold or copper. Therefore, the pad 112 is white or silver, and the crimp ball 102 is reddish-yellow or reddish-brown. Below, we will explain using the case where the crimp ball 102 is made of gold as an example.

[0024] Figure 2 shows images obtained by imaging the crimp ball 102 with the imaging device 14. The left image in Figure 2 is the first image 30R, captured under red light illumination, and the right image is the second image 30B, captured under blue light illumination. Figure 3 shows the first image 30R with explanatory lines added. As described above, in this example, the imaging device 14 captures grayscale images. The contour extraction system 10 extracts the contour of the crimp ball 102 from these images 30R and 30B and detects the shape and diameter of the crimp ball 102. The detected shape and diameter are important indicators for evaluating the quality of the semiconductor device.

[0025] Here, images 30R and 30B are grayscale images and are composed of multiple pixels with brightness values ​​corresponding to the reflected light from the subject. Image 30 shows the pad 112 and the crimping ball 102. Inside the crimping ball 102, there is a roughly circular image (see Figure 3) corresponding to the ball neck 104. Furthermore, the shadow of the bonding wire 106 is visible diagonally to the upper left from the ball neck 104. If there is such a shadow from the bonding wire 106 or dirt on the pad 112, the boundary between the crimping ball 102 and the pad 112 becomes unclear, and the outline of the crimping ball 102 cannot be properly extracted. This problem occurs not only in the first image 30R but also in the second image 30B.

[0026] Therefore, in order to accurately extract the contour of the object 100, the contour extraction system 10 in this example obtains the correspondence (e.g., ratio, etc.) between the luminance value Lr in the first image 30R and the luminance value Lb in the second image 30B as the correspondence value RB, and compares it with a pre-stored reference value RBref. The reference value RBref is a value that shows the correspondence between the luminance value Lr in red light and the luminance value Lb in blue light of an object of the same quality as the object 100.

[0027] The correspondence between the two luminance values ​​Lr and Lb varies greatly depending on the properties of the material, particularly its color. This will be explained with reference to Figure 4. In Figure 4, the solid lines show the luminance values ​​Lr and Lb when an object made of gold is imaged, and the dashed lines show the luminance values ​​Lr and Lb when an aluminum object is imaged.

[0028] Even when imaging the same object, the brightness value L of the resulting image will differ depending on the color of the light used for illumination. For example, suppose that when a gold object is imaged under red light, the brightness value of one pixel P1 is Lr1 and the brightness value of another pixel P2 is Lr2. Subsequently, when this gold object is imaged under blue light, the brightness value of one pixel P1 becomes Lbg1, which is significantly lower than Lr1, and the brightness value of another pixel P becomes Lbg2, which is significantly lower than Lr2. This is because the surface color of a gold object contains a red component, and while it reflects a lot of long-wavelength light in the visible range (light that appears red), it easily absorbs short-wavelength light (light that appears blue).

[0029] On the other hand, silvery objects like aluminum reflect a lot of both long-wavelength and short-wavelength light. Therefore, the brightness value L obtained from an aluminum object does not change significantly whether it is illuminated with red or blue light. For example, suppose that when an aluminum object is imaged under red light, the brightness value of one pixel P1 is Lr1 and the brightness value of another pixel P2 is Lr2. Subsequently, when this aluminum object is imaged under blue light, the brightness value of one pixel P1 becomes Lba1, which is approximately the same as Lr1, and the brightness value of another pixel P becomes Lba2, which is approximately the same as Lr2.

[0030] Thus, the correspondence between luminance values ​​for two types of light (i.e., the correspondence value RB) differs depending on the material. Therefore, in this example, the correspondence between an object of the same quality as the object 100 is stored in memory 24 as the reference value RBref. In this example, since the object 100 is a gold crimped ball 102, the reference value RBref is the correspondence between luminance values ​​Lr and Lb obtained by imaging a material of the same quality as the object 100, i.e., a gold object, under red and blue light. Such a reference value RBref may also be obtained, for example, by plotting points corresponding to luminance values ​​Lr and Lb for each pixel P of an image obtained by imaging a gold object, generating a scatter plot, and obtaining the approximate curve of this scatter plot as the reference value RBref.

[0031] The controller 20 sets the similarity score C based on this reference value RBref. Figure 5 shows the reference value RBref and an example of the similarity score C based on the reference value RBref. The controller 20 also calculates the correspondence between the luminance value Lr in red light and the luminance value Lb in blue light (i.e., the corresponding value RB) for the object 100, and calculates the similarity score C with respect to the reference value RBref. The similarity score C is an index that represents the strength of the relationship between the reference value RBref and the corresponding value RB. The formula for calculating the similarity score C is not particularly limited, as it changes according to the degree of similarity of the corresponding value RB with respect to the reference value RBref. For example, the similarity score C may change linearly or non-linearly, taking "1" when the corresponding value RB matches the reference value RBref, and approaching 0 as it moves away from the reference value RBref.

[0032] For example, in the case of the example in Figure 5, if the corresponding value RB of a particular pixel P1 in the image 30 of the object 100 is point M1 in Figure 5, then the similarity C of that pixel P is 1. Also, if the corresponding value RB of another pixel P2 is M2 in Figure 5, then the similarity C of that other pixel P is 0.5, and if it is M3 in Figure 5, then the correlation coefficient of that other pixel P is 0.

[0033] The controller 20 calculates the similarity C for all pixels P that make up the image 30 of the object 100. Then, based on the obtained similarity C, the controller 20 binarizes the image 30 and extracts the contour of the object 100 by extracting edges from the resulting binarized image.

[0034] For example, suppose that for object 100, a first image 30R and a second image 30B are obtained as shown in the upper part of Figure 6. In Figure 6, for simplicity of explanation, image 30 is composed of 9 pixels in a 3x3 grid. Also, in the first image 30R, the brightness value L of all 9 pixels is Lr1. On the other hand, in the second image 30B, the brightness value L varies in the range of Lb1 to Lb4.

[0035] In this case, as shown in the middle right of Figures 5 and 6, the similarity C of pixel P in Lb1 is "1", the similarity C of pixel P in Lb2 is "0.8", the similarity C of pixel P in Lb3 is "0.2", and the similarity C of pixel P in Lb4 is "0". Hereafter, the data to which the similarity C has been applied to each pixel P will be referred to as the "similarity image 31".

[0036] The controller 20 further normalizes the similarity image 31 to a range of 0 to 255. Hereafter, this normalized data will be referred to as the "normalized image 40". The middle left of Figure 6 shows the normalized image 40. The controller 20 further binarizes the normalized image 40. The bottom of Figure 6 shows the binarized image of the normalized image 40. Based on the obtained binarized image 42, the controller 20 extracts the contour of the object 100. In the example in Figure 6, line K is extracted as the contour of the object 100.

[0037] Figures 7 and 8 show the normalized image 40 and the binarized image 42 calculated from the first image 30R and the second image 30B of Figure 2. As is clear from the comparison of Figures 2 and 7, in the normalized image 40 derived from the similarity C between the corresponding value RB and the reference value RBref, the effects of the shadow of the wire 106 and the dirt on the pad 112 are eliminated, and the boundary between the crimp ball 102 and the pad 112 is clarified. In the example of Figure 7, the inner portion of the ball neck 104 is pre-processed with a similarity C=1. That is, the portion corresponding to the ball neck 104 appears darker than the crimp ball 102, and the boundary of the ball neck 104 can be clearly distinguished to some extent in the first image 30R and the second image 30B. Therefore, prior to generating the similarity image 31, the controller 20 extracts the contour of the ball neck 104 based on the first image 30R or the second image 30B. The controller 20 then generates a similarity image 31, and subsequently a normalized image 40, for the pixels P inside the contour, with a similarity score C=1.

[0038] The controller 20 binarizes the normalized image 40 shown in Figure 7 and extracts the contour of the crimped ball 102. Figure 8 is the binarized image 42 generated from the normalized image 40. The controller 20 extracts the contour of the crimped ball 102 based on this binarized image 42. The controller 20 may perform general image processing, such as histogram averaging, blob processing, and morphological transformation processing, before or after the binarization process. Furthermore, the controller 20 may perform general preprocessing, such as sharpening, tone adjustment, and filtering, on the first image 30R and the second image 30B before generating the similarity image 31 and the normalized image 40.

[0039] Next, the flow of the contour extraction process of the crimped ball 102 by the contour extraction system 10 will be explained with reference to Figure 9. In this case, the contour extraction system 10 has in advance stored a reference value RBref and a similarity value C to the reference value RBref in the memory 24, as shown in Figure 5. When the contour extraction system 10 extracts the contour of the crimped ball 102, it first acquires a first image 30R (S10). That is, the controller 20 irradiates the illumination unit 16 with red light and causes the imaging device 14 to image the crimped ball 102. Subsequently, the controller 20 irradiates the illumination unit 16 with blue light and causes the imaging device 14 to image the crimped ball 102, acquiring a second image 30B (S12).

[0040] The controller 20 binarizes the first image 30R or the second image 30B and extracts the contour of the ball neck 104 based on the obtained binarized image (S14). Subsequently, the controller 20 generates a similarity image 31 and a normalized image 40 based on the first image 30R and the second image 30B (S16, S18). That is, for each pixel P, the controller 20 applies the brightness value Lr of the first image 30R and the brightness value Lb of the second image 30B to the graph in Figure 5 and calculates the similarity C. However, for the area inside the contour of the ball neck 104, S=1.0 regardless of the brightness values ​​Lr and Lb. Subsequently, the controller 20 normalizes the similarity C calculated for each pixel P between 0 and 255 and calculates the normalized image 40. Subsequently, the controller 20 binarizes the normalized image 40 using the same procedure as for normal contour extraction (S20), and then extracts areas where the brightness value L changes from the binarized image 42 as contours of the object 100 (S22).

[0041] As is clear from the above explanation, in this example, a first image 30R captured under red light and a second image 30B captured under blue light are obtained, and a binarized image 42 is generated according to the similarity C between the corresponding value RB of the two images 30R and 30B and the reference value RBref, thereby enabling accurate extraction of the object 100 in the image.

[0042] It should be noted that the configurations described above are all examples, and other configurations may be changed as appropriate, as long as they meet the requirements of the configuration described in claim 1. For example, in the above description, red and blue light are used for illumination, but the color of the light used for illumination may be changed as appropriate. For example, the color of the light used for illumination may be changed according to the colors of the object 100 and the non-object 110 that are captured in the image together with the object 100. For example, when illuminating with a first color and a second color, the first color may be a color of the same type as the object 100, but not a color of the same type as the surface color of the non-object 110. Furthermore, in this case, the second color may be a color that is not of the same type as the surface color of either the object 100 or the non-object 110. Alternatively, the second color may be a color of the same type as the non-object 110, but not a color of the same type as the surface color of the object 100. "Similar colors" are colors that are located in the same area when the color wheel is divided into three or four equal parts, while "non-similar colors" are colors that are located in different areas when the color wheel is divided into three or four equal parts.

[0043] Furthermore, although the above explanation assumes two types of light colors are used, it is also possible to use three or more colors of light. For example, one could obtain the first, second, and third images by illuminating with light of three different colors, and then combine the contours obtained from comparing the brightness values ​​of the first and second images with the contours obtained from comparing the brightness values ​​of the first and third images.

[0044] Furthermore, although the crimped ball 102 is treated as the object 100 in the above description, the technology disclosed herein may also be applied to contour extraction of objects other than the crimped ball 102. Also, although the object 100 is imaged in grayscale in this example, it may be imaged in color. Moreover, the procedure described above is just one example, and the contour extraction procedure may be modified as appropriate if the comparison result between the luminance value Lr of the first image 30R and the luminance value Lb of the second image 30B is used. For example, the ratio of the luminance value Lr to the luminance value Lb may be calculated as the luminance ratio for each pixel P, and the binarized image 42 may be generated based on that luminance ratio. That is, since the yellow luminance ratio Lb / Lr of gold is around 0.3 and the luminance ratio Lb / Lr of aluminum is around 1, even if the luminance ratio is binarized with a value of around 0.6, it is possible to distinguish the gold region from the aluminum region and extract the contour of a gold object.

[0045] Furthermore, in the above explanation, the brightness values ​​Lr and Lb are calculated by comparing the pixels P of the first image 30R and the second image 30B, respectively. However, it is also possible to combine multiple pixels P of the first image 30R and the second image 30B into a single region and compare the statistical values ​​(e.g., mean or median) of the brightness values ​​of multiple pixels included in each of these combined regions. [Explanation of Symbols]

[0046] 10 Contour extraction system, 12 Microscope, 14 Imaging device, 16 Irradiation unit, 20 Controller, 22 Processor, 24 Memory, 30B Second image, 30R First image, 31 Similarity image, 40 Normalized image, 42 Binarized image, 100 Object, 102 Crimping ball, 104 Ball neck, 106 Bonding wire, 110 Non-object, 112 Pad.

Claims

1. A contour extraction system for extracting the contour of an object from an image of the object, An imaging device for imaging the aforementioned object, An irradiation unit for irradiating the aforementioned object, wherein the irradiation light is switchable between a first light having a first color, and a second light having a second color different from the first light. A controller that calculates the contour of the object from the captured image, The controller is equipped with, Based on a comparison of the brightness values ​​in the first image obtained by imaging under the first light source and the brightness values ​​in the second image obtained by imaging under the second light source, the contour of the object in the image is extracted. A contour extraction system characterized by being configured in such a way.

2. A contour extraction system according to claim 1, The aforementioned controller, The correspondence between the brightness value of a reference image obtained by imaging an object identical in quality to the object under the first light and the brightness value in the reference image under the second light is stored in advance as a reference value. The correspondence between the brightness values ​​of each of the multiple regions in the first image and the brightness values ​​of each of the multiple regions in the second image is calculated as a corresponding value. Based on the comparison between the aforementioned reference value and the aforementioned corresponding value, the contour of the object in the image is extracted. A contour extraction system characterized by being configured in such a way.

3. A contour extraction system according to claim 2, The aforementioned controller, For each of the multiple pixels, the similarity between the reference value and the corresponding value is calculated. Based on the binarized image obtained by binarizing the image using the aforementioned similarity as a criterion, the contour of the object within the image is extracted. A contour extraction system characterized by being configured in such a way.

4. A contour extraction system according to claim 1, At least one of the first color and the second color is a color of the same type as the color of the object, while being a color of a non-object that appears in the image together with the object. A contour extraction system characterized by the following features.

5. A contour extraction system according to claim 4, The aforementioned object is an object made of gold or copper, The aforementioned object is an object made of aluminum, The first color is a reddish color, and the second color is a blued color. A contour extraction system characterized by the following features.

6. A contour extraction system according to claim 5, The object is a crimped ball formed on a semiconductor device, The non-object is an aluminum pad to which the crimping ball is crimped. A contour extraction system characterized by the following features.

7. A contour extraction system according to claim 3, The object is a crimped ball formed on a semiconductor device, The non-objects that appear in the image along with the aforementioned object are aluminum pads to which the crimping balls are pressed. The aforementioned controller, Based on the first or second image, the contour of the ball neck interposed between the crimping ball and the bonding wire is extracted. The corresponding value inside the contour of the ball neck is considered to be the same as the reference value. A contour extraction system characterized by being configured in such a way.

8. A contour extraction method for extracting the contour of an object from an image of the object, While irradiating the object with a first light having a first color, the object is imaged to acquire a first image. While irradiating the object with a second light of a second color different from the first color, the object is imaged to acquire a second image. Based on the comparison result between the brightness value in the first image and the brightness value in the second image, the contour of the object in the image is extracted. A contour extraction method characterized by the following features.

Images