Appearance inspection device, machine learning model learning method, teaching image generation method, and program

Abstract

To provide an appearance inspection device and a program that can highly accurately determine the quality of the appearance of an inspection object and suppresses a decrease in a processing speed.SOLUTION: An appearance inspection device 10 comprises: an imaging unit 20 that acquires a camera image obtained by imaging an inspection object; a primary determination image generation unit 30 that generates a primary determination image IMG12 including blobs obtained through blob analysis of the camera image; a rule base determination unit 40 that determines whether or not the blobs included in the primary determination image IMG12 correspond to defects; a secondary determination image generation unit 50 that obtains a position P1 of the blob determined as a defect by the rule base determination unit 40, extracts an image from the camera image so that the position P1 is located within a predetermined range A, and generates it as a secondary determination image IMG22 in a predetermined size; and an AI determination unit 60 that determines whether or not the secondary determination image IMG22 indicates a defect using a machine learning model based on a convolution neural network.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an appearance inspection apparatus, a method for training a machine learning model, a method for generating teacher images, and a program.

Background Art

[0002] Patent Document 1 describes an inspection apparatus. This inspection apparatus includes an imaging unit that images an object to be inspected, an image processing unit that processes the imaging data of the object to be inspected obtained by the imaging unit and generates processed image data, a first determination unit that determines the presence or absence of defects in the object to be inspected based on the processed image data and predefined void and defect data, and a second determination unit that determines the presence or absence of voids and defects in the object to be inspected based on the processed image data and a pre-constructed machine learning model. For an object to be inspected determined by the second determination unit to have a void and be defective, the first determination unit determines the presence or absence of the void. For an object to be inspected determined by the second determination unit to have a scratch and be defective, the first determination unit determines the presence or absence of the scratch.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present invention is to provide an appearance inspection apparatus and a program that can determine the quality of the appearance of an object to be inspected with high accuracy while suppressing a decrease in processing speed. Another object of the present invention is to provide a method for training a machine learning model and a method for generating teacher images that can determine the quality with high accuracy when a plurality of defects are included in the object to be inspected.

Means for Solving the Problems

[0005] The invention according to claim 1 includes an imaging unit that acquires a camera image obtained by imaging an object to be inspected, a primary determination image generation unit that generates a primary determination image including blobs obtained by blob analysis of the camera image, a rule-based determination unit that determines whether or not the blobs included in the primary determination image correspond to defects based on a preset criterion, and obtains the position P1 of the blob determined to be a defect by the rule-based determination unit, and cuts out an image from the camera image so that the obtained position P1 is within a preset range A, May include a plurality of blobs a secondary determination image generation unit that generates the image as a secondary determination image, and an AI determination unit that determines whether or not the secondary determination image indicates a defect using a machine learning model based on a convolutional neural network, and discriminates the type of the defect if the image indicates a defect. 、 The learning method of the machine learning model includes steps of respectively performing blob analysis on a plurality of pre-prepared dataset images of the objects to be inspected, generating a plurality of analysis images including blobs, obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, and cutting out an image from the dataset image so that the obtained position P2 is within the range A, Include at least one defect generating a teacher defect candidate image, and Located at the position P2 generating an annotated teacher image by classifying the teacher defect candidate image according to the type of defect, and performing learning of the machine learning model based on the teacher image. It is an appearance inspection device including these steps.

[0006]

[0007] Claim 2 The invention described in An imaging unit that acquires a camera image obtained by imaging an object to be inspected, a primary determination image generation unit that generates a primary determination image including blobs obtained by blob analysis of the camera image, and based on a preset criterion, a rule-based determination unit that determines whether the blobs included in the primary determination image correspond to defects, and a secondary determination image generation unit that obtains the position P1 of the blob determined to be a defect by the rule-based determination unit, cuts out an image from the camera image so that the obtained position P1 is within a predetermined range A, and generates it as a secondary determination image that may include a plurality of blobs, and an AI determination unit that determines whether the secondary determination image indicates a defect using a machine learning model based on a convolutional neural network, and when it indicates a defect, discriminates the type of the defect The learning method of the machine learning model includes steps of respectively performing blob analysis on a plurality of pre-prepared dataset images of the objects to be inspected, generating a plurality of analysis images including blobs having an area equal to or larger than a preset size, obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, and cutting out an image from the dataset image so that the obtained position P2 is within the range A, Include at least one defectA step of generating a teacher defect candidate image, and the teacher defect candidate image Located at the position P2 A step of generating an annotated teacher image by classifying according to the type of defect, and a step of training the machine learning model based on the teacher image, including An appearance inspection device .

[0008] Claim 3 The invention according to claim is a computer for generating a teacher image of a machine learning model used in an AI determination unit provided in the appearance inspection apparatus according to claim 1, and respectively performing blob analysis on a plurality of pre-prepared dataset images of the inspection object, generating a plurality of analysis images including a plurality of blobs, obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, and cutting out the image from the dataset image so that the obtained position P2 is within a predetermined range A, Include at least one defect A program for executing a step of obtaining as a teacher defect candidate image.

[0009] Claim 4 The invention according to claim is a learning method of a machine learning model configured by a convolutional neural network, for determining whether the appearance of an inspection object is good or bad based on a determination image, and respectively performing blob analysis on a plurality of pre-prepared dataset images of the inspection object, generating a plurality of analysis images including blobs, obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, and cutting out the image from the dataset image so that the obtained position P2 is within a predetermined range A, And may include defects other than the defect A step of obtaining as a teacher defect candidate image, and the teacher defect candidate image Include at least one defect A step of creating an annotated teacher image by classifying according to the type of defect, and a step of training the machine learning model based on the teacher image, including Located at the position P2 A learning method of a machine learning model.

[0010] Claim 5The invention described in is composed of a convolutional neural network, and the position P1 of the defect of the object to be inspected is within a predetermined range A. And may include defects other than the defect A method for generating a teacher image used for training a machine learning model for determining the quality of the appearance of an object to be inspected based on a determination image, the method including: respectively performing blob analysis on a plurality of pre-prepared dataset images of the object to be inspected to generate a plurality of analysis images including blobs; obtaining the position P2 of the blobs included in the plurality of analysis images by the method for calculating the position P1, and cutting out an image from the dataset image so that the obtained position P2 is within a predetermined range A, Include at least one defect obtaining it as a teacher defect candidate image; and classifying the teacher defect candidate image Located at the position P2 according to the type of defect to create an annotated teacher image. The method for generating a teacher image includes the above steps.

Effect of the Invention

[0011] According to the present invention, it is possible to provide an appearance inspection apparatus and a program that can determine the quality of the appearance of an object to be inspected with high accuracy while suppressing a decrease in processing speed. Further, according to the present invention, it is possible to provide a method for training a machine learning model and a method for generating a teacher image that can determine the quality with high accuracy when a plurality of defects are included in the object to be inspected.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0013] Subsequently, with reference to the attached drawings, embodiments embodying the present invention will be described to facilitate understanding of the present invention. In the figures, parts not relevant to the description may be omitted from illustration.

[0014] An appearance inspection apparatus 10 according to an embodiment of the present invention includes, as shown in FIG. 1, an imaging unit 20, a primary determination image generation unit 30, a rule-based determination unit 40, a secondary determination image generation unit 50, and an AI determination unit 60, and can inspect for defects on the appearance of a semiconductor 12 (an example of an object to be inspected) conveyed, for example, in the direction of the arrow. Defects are, as shown in FIG. 2, for example, scratches, voids, dirt, foreign matter, and filling defects. Note that the object to be inspected is not limited to the semiconductor 12. Other objects to be inspected include, for example, arbitrary machine parts and foods.

[0015] The imaging unit 20 is a camera capable of imaging the appearance of the semiconductor 12 and is disposed above the semiconductor 12. The number of pixels of the camera is, for example, 19 million to 33 million pixels. The image of the semiconductor 12 captured is acquired as a camera image by the imaging unit 20.

[0016] The primary determination image generation unit 30 can perform blob analysis on the camera image obtained by the imaging unit 20 and generate a primary determination image IMG11 (see FIG. 3) including the blobs obtained as a result.

[0017] The rule-based determination unit 40 (see FIG. 1) can determine whether or not the blobs included in the primary determination image IMG11 correspond to defects based on a preset criterion. The preset criterion is, for example, that the area of the blob is larger than a predetermined threshold value. The preset criterion is set more strictly than an appearance inspection apparatus that determines the quality of appearance only based on rules without using AI. In some cases, the rule-based determination unit 40 may determine that a semiconductor 12 that should originally be a non-defective product is a defective product including a defect. The blobs determined by the rule-based determination unit 40 to correspond to defects are treated as defect candidates.

[0018] As shown in FIG. 3, the secondary determination image generation unit 50 obtains the position of the center of gravity of the blob (an example of the position P1) determined to be a defect by the rule-based determination unit 40, and cuts out the image from the camera image so that the position of this center of gravity is in the central part (an example of a predetermined range A), and can generate a secondary determination image (an example of a determination image) IMG12 with a predetermined size S. The vertical and horizontal sizes of the secondary determination image IMG12 (the image cut out from the camera image) are, for example, 256 pixels each. Note that the position of the center of gravity of this blob is an example of the position P1 of the blob. The position P1 of the blob is not limited to the position of the center of gravity of the blob, and may be, for example, the center obtained from the maximum width (the maximum size in the horizontal direction) and the maximum height (the maximum size in the vertical direction) of the blob. That is, the position P1 of the blob may be arbitrary as long as it is the position P1 calculated by a predetermined position calculation method. Also, the predetermined range A is not limited to the central part of the image. However, the narrower the range, the better.

[0019] The AI determination unit 60 (see FIG. 1) can determine whether or not the secondary determination image IMG12 indicates a defect using a machine learning model, and when it indicates a defect, it can discriminate the type of the defect (such as scratches, voids, stains, foreign matters, and filling defects). The machine learning model is configured by a convolutional neural network.

[0020] Here, a learning method of this machine learning model will be described based on FIGS. 4 and 5. Note that at least the following steps SA1, SA2, and step SA4 are implemented by a computer program.

[0021] (Step SA1) Blob analysis is performed on each of a plurality of pre-prepared images of the semiconductor 12 (hereinafter referred to as "dataset images"), and a plurality of analysis images IMG21 including blobs are generated. Note that the analysis image IMG21 may be an image selected to include a blob having an area equal to or larger than a predetermined size after blob analysis.

[0022] (Step SA2) For each blob included in the analysis image IMG21, the position of the center of gravity is obtained, and an image in which the obtained position of the center of gravity is in the central part is cut out from the dataset image and generated as a teacher defect candidate image IMG22. The teacher defect candidate image IMG22 cut out from the dataset image has the same size S as the secondary determination image IMG12. Here, "the same size" does not mean exactly the same. "The same size" means that some differences are allowed as long as they do not practically affect the determination accuracy by the AI determination unit 60, which means "substantially the same" (the same applies hereinafter). That is, the teacher defect candidate image IMG22 includes a defect candidate in the central part. In addition, it should be noted that other defect candidates may enter around the defect candidate in the central part in the teacher defect candidate image IMG22.

[0023] Note that in this step SA2, in order to generate the teacher defect candidate image IMG22, it is not necessarily cut out so that the center of gravity is located in the central part of the image, and it may be cut out so that the position P2 of the blob obtained by the position calculation method for obtaining the above-mentioned position P1 is within a predetermined range A. That is, in this step SA2, the position P2 of the blobs included in the plurality of analysis images IMG21 is obtained by the above-described method for calculating the position P1, and an image in which the obtained position P2 is within a predetermined range A is cut out from the dataset image and generated as a teacher defect candidate image IMG22 having substantially the same size S as the secondary determination image IMG12.

[0024] (Step SA3) The defect in the central part of the obtained teacher defect candidate image IMG22 is classified according to its type (see FIG. 2). The classification work is performed manually. Thereby, an annotated teacher image IMG23 is generated. However, the classification work may be performed by AI instead of manually. In addition, a teacher image IMG23 showing a non-defective semiconductor 12 without defects and having the same size S as the secondary determination image IMG12 is also prepared separately.

[0025] (Step SA4) Based on the teacher image IMG23, deep learning based on a convolutional neural network is performed. That is, the machine learning model learns the appearance of the semiconductor 12 without defects and the semiconductor 12 with defects based on the teacher image IMG23, and when the teacher image IMG23 indicates a defect, learns the type of the defect (see FIG. 2).

[0026] According to the learning method described above, in order to create a teacher image IMG23 showing a defect corresponding to a blob whose center of gravity position (an example of the position P1) is in the central part (an example of the predetermined range A), it is not necessary to create a plurality of teacher images IMG23 that differ only in position for the same defect, and the number of teacher images IMG23 is reduced.

[0027] Next, the operation of the appearance inspection apparatus 10 will be described with reference to FIG. 6. The appearance inspection apparatus 10 operates according to the following steps SB1 to SB5. However, if possible, each of the steps SB1 to SB5 may be performed in a different order, may be performed in parallel, or some of them may be omitted. Note that at least the following steps SB2 to SB5 are realized by a computer program.

[0028] (Step SB1) The imaging unit 20 images the semiconductor 12 before shipment and acquires a camera image.

[0029] (Step SB2) The primary determination image generation unit 30 performs blob analysis on the image of the imaged semiconductor 12. By the blob analysis, the image is binarized and blobs are detected. The image in which blobs are detected is generated as the primary determination image IMG11 shown in FIG. 3.

[0030] (Step SB3) The rule-based determination unit 40 determines whether or not the blobs detected in step SB2 correspond to defects. The blobs determined to correspond to defects are specified as defect candidates.

[0031] (Step SB4) The secondary determination image generation unit 50 obtains the centroid (the intersection of the dashed-dotted lines in the enlarged view of FIG. 3) of each blob determined by the rule-based determination unit 40 to correspond to a defect. Thereafter, the secondary determination image generation unit 50 cuts out from the camera image an image in which the position of the obtained centroid is in the central part, and generates it as a secondary determination image IMG12 having a predetermined size S. That is, the secondary determination image IMG12 includes defect candidates in the central part of the image. Incidentally, when there are a plurality of adjacent defects in the camera image, the secondary determination image IMG12 may include defect candidates other than the defect candidates whose centroids are located in the central part.

[0032] Note that the generation of the secondary determination image IMG12 is not limited to being based on the blob whose center of gravity is located in the central part of the image. As long as the secondary determination image IMG12 is generated based on the blob whose aforementioned position P1 is in the predetermined range A, that is sufficient.

[0033] (Step SB5) The AI determination unit 60 determines whether the defect candidates included in the secondary determination image IMG12 correspond to true defects based on a machine learning model using a convolutional neural network. If it is a defect, it specifies up to the type of the defect (such as scratches, voids, stains, foreign objects, and filling defects). At this time, since the machine learning model learns the type of defect based on the teacher image IMG23 whose center of gravity of the defect is in the central part during prior learning, even if the secondary determination image IMG12 includes a plurality of defect candidates, it shows a strong response to the defect candidate located in the central part of the secondary determination image IMG12. As a result, the determination accuracy is improved as compared with the case where learning is not performed by the aforementioned learning method.

[0034] Note that when comparing the processing speed of the AI determination unit 60 in this step SB5 with the processing speeds in steps SB1 to SB4, the processing speed of the AI determination unit 60 in step SB5 is slower.

[0035] By executing steps SB1 to SB5 in this way, the semiconductor 12 inspected by the appearance inspection device 10 is finally determined by the AI determination unit 60 whether it corresponds to a non-defective product without defects or a defective product with defects. If it is a defective product, the type of the corresponding defect is specified.

[0036] As described above, according to the appearance inspection device 10 according to the present embodiment, when the semiconductor 12 includes a plurality of defects, it is possible to determine whether it is good or bad with high accuracy using a learning model configured by a convolutional neural network. Also, according to the appearance inspection device 10, first, candidates for defects are quickly selected by the aforementioned SB1 to SB4, and then, for the selected candidates for defects, the rule - based determination unit 40 makes a determination in step SB5 where the processing becomes slow. Therefore, compared with the case where the rule - based determination unit 40 makes a determination after the AI determination unit 60 makes a determination, the process of determining the quality of the semiconductor 12 is executed more quickly as a whole. That is, according to the appearance inspection device 10, compared with an appearance inspection device different from the appearance inspection device 10, it is possible to determine the quality of the appearance of the semiconductor 12 with high accuracy, while suppressing a decrease in the processing speed required for the determination.

[0037] As described above, the embodiments of the present invention have been described. However, the present invention is not limited to the above - described forms, and all changes and the like that do not deviate from the gist are within the scope of application of the present invention. The aforementioned appearance inspection device 10 can also determine, for example, chips, stains, black foreign objects, white foreign objects, marks, resin chips, wire scratches, and dot scratches by causing a machine learning model to learn appropriate teacher images.

Explanation of Reference Numerals

[0038] 10 Appearance inspection device 12 Semiconductor 20 Imaging unit 30 Primary determination image generation unit 40 Rule - based determination unit 50 Secondary determination image generation unit 60 AI determination unit IMG11 Primary determination image IMG12 Secondary determination image IMG21 Analysis image IMG22 Teacher defect candidate image IMG23 Teacher image

Claims

1. An imaging unit that acquires a camera image obtained by imaging an object to be inspected; A primary determination image generation unit that generates a primary determination image including blobs obtained by blob analysis of the camera image; A rule-based determination unit that determines whether or not the blobs included in the primary determination image correspond to defects based on a preset criterion; A secondary determination image generation unit that obtains the position P1 of the blobs determined to be defective by the rule-based determination unit, cuts out an image from the camera image so that the obtained position P1 is within a preset range A, and generates it as a secondary determination image of a preset size; An appearance inspection apparatus comprising: an AI determination unit that determines whether or not the secondary determination image indicates a defect using a machine learning model based on a convolutional neural network, and discriminates the type of the defect if the secondary determination image indicates a defect.

2. In the appearance inspection apparatus according to Claim 1, The learning method of the machine learning model includes steps of performing blob analysis on a plurality of dataset images of the objects to be inspected prepared in advance, respectively, and generating a plurality of analysis images including blobs; Obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, cutting out an image from the dataset image so that the obtained position P2 is within the range A, and generating it as a teacher defect candidate image having substantially the same size as the secondary determination image; Generating an annotated teacher image by classifying the teacher defect candidate image according to the type of defect; A step of learning the machine learning model based on the teacher image.

3. In the appearance inspection apparatus according to Claim 1, The learning method of the machine learning model includes steps of performing blob analysis on a plurality of dataset images of the objects to be inspected prepared in advance, respectively, and generating a plurality of analysis images including blobs having an area equal to or larger than a preset size; Obtaining the position P2 of the blobs included in the plurality of analysis images by the calculation method of the position P1, cutting out an image from the dataset image so that the obtained position P2 is within the range A, and generating it as a teacher defect candidate image having substantially the same size as the secondary determination image; Generating an annotated teacher image by classifying the teacher defect candidate image according to the type of defect; A step of training the machine learning model based on the teacher image, and an appearance inspection apparatus including the same.

4. A computer for generating a teacher image of a machine learning model used in an AI determination unit included in the appearance inspection apparatus according to Claim 1, A step of respectively performing blob analysis on a plurality of pre-prepared dataset images of the inspection object to generate a plurality of analysis images including a plurality of blobs; A program for causing the computer to execute a step of obtaining the position P2 of the blob included in the plurality of analysis images by the calculation method of the position P1, cutting out the image from the dataset image so that the obtained position P2 is within a predetermined range A, and obtaining a teacher defect candidate image having substantially the same size as the secondary determination image.

5. A method for training a machine learning model configured by a convolutional neural network to determine whether the appearance of an inspection object is good or bad based on a determination image of a predetermined size in which the position P1 of a defect of the inspection object is within a predetermined range A, A step of respectively performing blob analysis on a plurality of pre-prepared dataset images of the inspection object to generate a plurality of analysis images including blobs; A step of obtaining the position P2 of the blob included in the plurality of analysis images by the calculation method of the position P1, cutting out the image from the dataset image so that the obtained position P2 is within a predetermined range A, and obtaining a teacher defect candidate image having substantially the same size as the determination image; A step of creating an annotated teacher image by classifying the teacher defect candidate image according to the type of defect; A method for training a machine learning model including a step of training the machine learning model based on the teacher image.

6. A method for generating a teacher image used for training a machine learning model configured by a convolutional neural network to determine whether the appearance of an inspection object is good or bad based on a determination image of a predetermined size in which the position P1 of a defect of the inspection object is within a predetermined range A, A step of respectively performing blob analysis on a plurality of pre-prepared dataset images of the inspection object to generate a plurality of analysis images including blobs; Determine the position P2 of the blob included in the plurality of analysis images by the method for calculating the position P1, cut out the image from the dataset image so that the obtained position P2 is within a predetermined range A, and obtain it as a teacher defect candidate image having substantially the same size as the determination image; A method for generating a teacher image, comprising: creating an annotated teacher image by classifying the teacher defect candidate image according to the type of defect.

Images