Appearance inspection device and appearance inspection method

Abstract

To easily verify an inference model.SOLUTION: A first flag indicating a defective product or good product is given to each defective product image or good product image for verification for verifying the quality determination performance of an inference model. A defective product image or good product image for verification is input into the inference model to obtain a quality determination result. An input of a second flag to be given to an image for verification that is more difficult in quality determination than the defective product image or good product image with the first flag given thereto is accepted. An image with the second flag given thereto is input into the inference model to obtain a quality determination result. A verification result for the image with the second flag given thereto is displayed on a display device so as to be distinguishable from a verification result for the image with the first flag given thereto.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to an appearance inspection apparatus and an appearance inspection method for inspecting the appearance of a workpiece.

Background Art

[0002] For example, Patent Document 1 discloses a processing apparatus that uses machine learning by a computer to determine whether a workpiece is a good product or a defective product. The processing apparatus of Patent Document 1 performs supervised machine learning on good product data to generate a good product learning model, and performs supervised machine learning on defective product data to generate a defective product learning model. After that, the data of the workpiece to be determined is input, and it is configured to be able to determine whether the workpiece is a good product or a defective product by the good product learning model and the defective product learning model. Such an apparatus is also called an appearance inspection apparatus for a workpiece.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in an appearance inspection apparatus, after generating a good product learning model and a defective product learning model (hereinafter referred to as an inference model), a verification image is input to the inference model to confirm the pass / fail determination performance. Among the images input to the inference model, there are images that even the user is confused about whether they are good products or defective products. For example, when there is a small scratch on the workpiece, but a scratch of that degree may be judged as a good product, or when it is once judged as a defective product and later the user may return it to the good product line. In the inspection of such images, it doesn't matter whether they are judged as good products or defective products. That is, in the case of images without a true correct answer, although the inference model cannot determine whether it is a good product or a defective product, the conventional method had no choice but to assign a binary flag of good product or defective product. As a result, even if the inference model was performing sufficiently, it was difficult to determine whether trial and error in adjusting the inference model was really necessary because of the images without a true correct answer.

[0005] The present disclosure is in view of such a point, and its object is to enable easy verification of the inference model.

Means for Solving the Problem

[0006] To achieve the above object, in one aspect of the present disclosure, it is possible to premise an appearance inspection apparatus that inputs a work image obtained by photographing a work to be inspected into a machine learning network and performs a pass / fail determination of the work based on the input work image. The appearance inspection apparatus includes a storage unit that stores an inference model after learning processing, an input unit that assigns a first flag indicating a defective product or a non-defective product to a defective product image or a non-defective product image for verification to verify the pass / fail determination performance of the inference model, an inspection unit that inputs the defective product image or the non-defective product image for verification into the inference model to obtain a pass / fail determination result, and a display control unit that causes the display unit to display a verification result of the pass / fail determination performance of the inference model based on the pass / fail determination result obtained by the inspection unit. The input unit can receive an input of a second flag to be assigned to a verification image for which it is more difficult to determine pass / fail than the defective product image or the non-defective product image to which the first flag is assigned. Further, the inspection unit can input the image to which the second flag is assigned into the inference model to obtain a pass / fail determination result. Furthermore, the display control unit can display the verification result for the image to which the second flag is assigned on the display unit separately from the verification result for the image to which the first flag is assigned.

[0007] According to this configuration, for example, a first flag is assigned to a verification image for which it is possible to clearly determine whether it is a defective product image or a non-defective product image. On the other hand, a second flag is assigned to a verification image that does not have a true correct answer, such as an image for which it is impossible to clearly determine whether it is a defective product image or a non-defective product image, or an image for which either determination is acceptable. The inspection unit inputs one or more images to which the first flag is assigned into the inference model to obtain a pass / fail determination result, and also inputs one or more images to which the second flag is assigned into the inference model to obtain a pass / fail determination result. Since the display unit can display the verification result of the image to which the first flag is assigned and the verification result of the image to which the second flag is assigned separately, the user can grasp the pass / fail determination performance of the inference model excluding images that do not have a true correct answer. As a result, the user can easily determine whether the inference model already exhibits sufficient performance or whether trial and error in adjusting the inference model is still necessary.

[0008] The display control unit according to another aspect is configured to be able to display information regarding the number of correct answers in the pass / fail determination for the image to which the first flag is assigned, excluding the verification result for the image to which the second flag is assigned.

[0009] According to this configuration, it is possible to exclude the verification result for an image that does not have a correct answer and display the pass / fail determination result for an image that can clearly determine whether it is a defective product image or a non-defective product image, so it is possible to accurately determine whether the inference model is already exhibiting sufficient performance.

[0010] The input unit according to another aspect is configured to be able to receive a selection as to whether to exclude the pass / fail determination result for the image to which the second flag is assigned. Further, when the input unit receives a selection to exclude the pass / fail determination result for the image to which the second flag is assigned, the display control unit excludes the verification result for the image to which the second flag is assigned and displays information regarding the number of correct answers in the pass / fail determination for the image to which the first flag is assigned. On the other hand, when the input unit receives a selection not to exclude the pass / fail determination result for the image to which the second flag is assigned, the display control unit is configured to be able to display information regarding the number of correct answers combining the verification result for the image to which the second flag is assigned and the pass / fail determination for the image to which the first flag is assigned.

[0011] That is, when there are users who want to judge the performance of the inference model by completely ignoring the verification results for images that do not have true correct answers and users who also want to refer to the verification results for images that do not have true correct answers, it is possible to accommodate both types of users.

[0012] In another aspect, the display control unit causes the display unit to display a list of the plurality of verification images, and the input unit can receive the assignment of the first flag or the second flag to the image selected from the plurality of verification images displayed in a list on the display unit, so that it is possible to easily perform the process of assigning a flag by the user.

[0013] The display control unit according to another aspect may select an image to which neither the first flag nor the second flag is assigned from among the verification images and cause the display unit to display the image.

[0014] That is, it is possible to prompt the user to assign the first flag or the second flag to an image to which neither the first flag nor the second flag is assigned. As a result, if the pass / fail determination is accurately performed on the image to which the first flag is assigned, it can be determined that the inference model already exhibits sufficient performance. On the other hand, if the pass / fail determination is not accurately performed on the image to which the first flag is assigned, it can be determined that the performance of the inference model is insufficient.

[0015] The display control unit according to another aspect may select an image that the inference model could not perform a pass / fail determination on from among the verification images and cause the display unit to display the image.

[0016] According to this configuration, it is possible to determine whether an image that the inference model could not perform a pass / fail determination on is an image without a correct answer. If, by chance, an image that the inference model could not perform a pass / fail determination on is an image without a correct answer, the performance of the inference model may be determined by excluding the determination result. On the other hand, if an image that the inference model could not perform a pass / fail determination on is an image with a correct answer, it can be determined that the performance of the inference model is insufficient.

[0017] The display control unit according to another aspect can generate a cumulative histogram based on the frequency of images determined to be good products and the frequency of images determined to be defective products and cause the display unit to display the histogram.

[0018] The display control unit according to another aspect may generate a user interface screen that is selectable as to whether to include the frequency of the image to which the second flag is assigned when generating the cumulative histogram, and cause the display unit to display the screen. When it is selected to include the frequency of the image to which the second flag is assigned, the cumulative histogram is generated including the frequency of the image to which the second flag is assigned. On the other hand, when it is selected not to include the frequency of the image to which the second flag is assigned, the cumulative histogram may be generated without including the frequency of the image to which the second flag is assigned.

[0019] According to this configuration, the user can freely select and display a cumulative histogram including the frequency of an image having no true correct answer and a cumulative histogram not including the frequency of an image having no true correct answer.

[0020] In another aspect, the apparatus may further include a learning unit that inputs learning data to the machine learning network for learning to generate the inference model.

[0021] Further, the input unit may receive an input of the second flag to be assigned to a verification image having a lower defect degree than the defective product image to which the first flag is assigned, or may receive an input of the second flag to be assigned to a verification image having a lower non-defective product degree than the non-defective product image to which the first flag is assigned.

Advantages of the Invention

[0022] As described above, since the verification result of an image that can clearly determine whether it is a defective product image or a non-defective product image and the verification result of an image having no correct answer can be separately displayed on the display unit, the verification of the inference model can be easily performed.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0024] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. It should be noted that the following description of the preferred embodiments is merely illustrative in nature and is not intended to limit the present invention, its applications, or its uses.

[0025] FIG. 1 is a schematic diagram showing the configuration of an appearance inspection apparatus 1 according to an embodiment of the present invention. The appearance inspection apparatus 1 is an apparatus for determining the quality of a work image obtained by imaging a work that is an inspection target, such as various parts and products, and can be used at a production site such as a factory. Specifically, a machine learning network is constructed inside the appearance inspection apparatus 1. A work image obtained by imaging a work to be inspected is input to the generated machine learning network, and the machine learning network can perform a quality determination of the work image.

[0026] The entire work may be the inspection target, or only a part of the work may be the inspection target. Also, a single work may include a plurality of inspection targets. Further, a work image may include a plurality of works.

[0027] The appearance inspection device 1 includes a control unit 2 that serves as the device main body, an imaging unit 3, a display device (display unit) 4, and a personal computer 5. The personal computer 5 is not essential and can also be omitted. Instead of the display device 4, the personal computer 5 can be used to display various information and images, or the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0028] In FIG. 1, as an example of the configuration example of the appearance inspection device 1, the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are described. However, any plurality of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Also, the control unit 2 can be divided into a plurality of units and a part of it can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into a plurality of units and a part of it can be incorporated into other units.

[0029] (Configuration of Imaging Unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that executes acquisition of a workpiece image. The camera module 14 includes an AF motor 141 that drives the imaging optical system and an imaging substrate 142. The AF motor 141 is a part that automatically performs focus adjustment by driving the lens of the imaging optical system, and can perform focus adjustment by a method such as conventional contrast autofocus. The imaging substrate 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to be able to acquire a color image. Instead of the CMOS sensor 143, for example, a light receiving element such as a CCD sensor can also be used.

[0030] The illumination module 15 includes an LED (light-emitting diode) 151 as a light emitter that illuminates an imaging area including the workpiece, and an LED driver 152 that controls the LED 151. The light emission timing, light emission time, and light emission amount by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided as an external illumination unit separately from the imaging unit 3.

[0031] (Configuration of the display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel, an organic EL panel, or the like. The workpiece image, user interface image, etc. output from the control unit 2 are displayed on the display device 4. Also, when the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used as an alternative to the display device 4.

[0032] (Operating device) Examples of the operating device for the user to operate the appearance inspection device 1 include the keyboard 51 and mouse 52 of the personal computer 5, but are not limited thereto, and any device configured to be able to receive various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operating device.

[0033] Operations by the user on the keyboard 51 and mouse 52 can be detected by the control unit 2. Also, the touch panel 41 is a conventionally well-known touch-type operation panel equipped with, for example, a pressure sensor, and the touch operation of the user can be detected by the control unit 2. The same applies when other pointing devices are used.

[0034] (Configuration of the control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. A processor 13a is provided on the main board 13. The processor 13a controls the operations of the connected respective boards and modules. For example, the processor 13a outputs an illumination control signal for controlling the lighting / extinguishing of the LED 151 to the LED driver 152 of the illumination module 15. The LED driver 152 switches the lighting / extinguishing of the LED 151 and adjusts the lighting time according to the illumination control signal from the processor 13a, and also adjusts the light quantity of the LED 151 and the like.

[0035] Also, the processor 13a outputs an imaging control signal for controlling the CMOS sensor 143 to the imaging board 142 of the camera module 14. The CMOS sensor 143 starts imaging according to the imaging control signal from the processor 13a, and performs imaging while adjusting the exposure time to an arbitrary time. That is, the imaging unit 3 images within the field of view of the CMOS sensor 143 according to the imaging control signal output from the processor 13a. If there is a workpiece within the field of view, the workpiece will be imaged. However, if there are other objects within the field of view, they can also be imaged. For example, the appearance inspection device 1 can image a good product image corresponding to a good product and a defective product image corresponding to a defective product by the imaging unit 3 as learning images for the machine learning network. The learning images do not have to be images captured by the imaging unit 3, and can also be images captured by other cameras or the like.

[0036] On the other hand, during the operation of the appearance inspection device, the imaging unit 3 can image the workpiece. Also, the CMOS sensor 143 is configured to be able to output a live image, that is, the currently captured image, at any time at a short frame rate.

[0037] When the imaging by the CMOS sensor 143 is completed, the image signal output from the imaging unit 3 is input to and processed by the processor 13a on the main board 13, and is also stored in the memory 13b on the main board 13. Details of the specific processing content by the processor 13a on the main board 13 will be described later. Note that a processing device such as an FPGA or a DSP may be provided on the main board 13. The processor 13a may be integrated with a processing device such as an FPGA or a DSP.

[0038] A display control unit 13c is provided on the main board 13. The display control unit 13c is a part that generates a display screen, controls the display device 4, and causes the display device 4 to display the display screen. Details of the specific operation of the display control unit 13c will be described later.

[0039] The connector board 16 is a part that receives power supply from the outside through a power connector (not shown) provided on the power interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and module, etc. Specifically, it distributes power to the lighting module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 includes an AF motor driver 181. The AF motor driver 181 supplies driving power to the AF motor 141 of the camera module 14 to realize autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a on the main board 13.

[0040] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, communication between the main board 13 and an external control device (not shown), etc. The external control device can include, for example, a programmable logic controller or the like. The communication may be wired or wireless, and any communication form can be realized by a conventionally well-known communication module.

[0041] The control unit 2 is provided with a storage device (storage unit) 19 composed of, for example, a solid state drive, a hard disk drive, etc. The storage device 19 stores a program file 80, a setting file, etc. (software) for enabling each control and process described later to be executed by the above hardware. The program file 80 and the setting file are stored, for example, in a storage medium 90 such as an optical disk, and the program file 80 and the setting file stored in this storage medium 90 can be installed in the control unit 2. The program file 80 may be downloaded from an external server using a communication line. Further, the storage device 19 can store, for example, the above image data, etc., and can also store parameters, etc. for constructing an inference model obtained after the learning process described later.

[0042] In FIG. 2, the storage device 19 is shown as being integrated with the control unit 2, but the storage device 19 may be a separate entity from the control unit 2. Examples of such a storage device 19 include a network-attached storage (NAS). The NSA and the control unit 2 are connected by a communication line such as a wired LAN or a wireless LAN.

[0043] That is, in the appearance inspection device 1, by using learning data to train a machine learning network, the parameters of the machine learning network are adjusted to generate an inference model. A work image obtained by photographing a work to be inspected is input to the inference model, and the quality of the work can be determined based on the input work image. By using this appearance inspection device 1, an appearance inspection method for determining the quality of a work based on a work image can be executed.

[0044] (Configuration of the processor) As shown in FIG. 2, the processor 13a is provided with a learning unit 13d, an input unit 13e, and an inspection unit 13f. The learning unit 13d, the input unit 13e, and the inspection unit 13f may be parts configured by hardware, or may be parts configured by executing software. Also, the learning unit 13d, the input unit 13e, and the inspection unit 13f do not necessarily have to be provided on the main board 13, and a part or all of these may be provided on a board other than the main board 13. Note that, in this example, the learning unit 13d is not essential.

[0045] The learning unit 13d is a part that inputs learning data into a machine learning network for learning, and generates an inference model for determining the quality of an input image. The learning unit 13d may be configured by, for example, a learning computer separate from the control unit 2. The learning computer is configured to be capable of performing machine learning at high speed. By communicably connecting the learning computer and the control unit 2, the parameters for constructing the inference model generated by the learning computer can be transmitted to and stored in the storage device 19. Thereby, the control unit 2 can construct an inference model.

[0046] (Learning Process) Next, an example of the procedure of the learning process will be described based on the flowchart shown in FIG. 3. In step SA1 after starting, the learning unit 13d prepares an unlearned machine learning network. The unlearned machine learning network has, for example, initial values of parameters randomly determined. In step SA2, the learning unit 13d reads out a plurality of learning images stored in the storage device 19 and sequentially inputs them into the machine learning network. Then, in step SA3, the parameters of the machine learning network are adjusted. At this time, only good product images may be input for good product learning, only defective product images may be input for defective product learning, or both types of learning may be performed. The parameters obtained in step SA3 are the parameters for constructing a learned inference model, and are stored in the storage device 19 in step SA4. Step SA4 is a storage step.

[0047] For example, when the learning image is a defective product image, before inputting it into the machine learning network in step SA2, annotations are executed on the defective product image. That is, the user performs in advance processes such as a process of explicitly indicating that it is a defective product image and a process of designating defective parts of the defective product image on the defective product image. The information given by the annotation is stored in the storage device 19 in a state associated with the corresponding defective product image. Therefore, when performing defective product learning, the parameters of the machine learning network may be adjusted using the information given in advance.

[0048] The learning method of the above machine learning network is not particularly limited, but for example, the following method can be used. That is, the machine learning network can be learned by minimizing the Loss function. Although there are various definitions of Loss, Mean Square Error (MSE) can be cited as an example.

[0049]

Number

[0050] Here, T is the target abnormality map, 0 is the output image (abnormality map), n is the number of pixels of the image T that are 0, and x and y are pixel positions. Note that Loss functions such as Binary Cross Entropy can also be used. The above is merely an example.

[0051] (Example of work image) FIG. 4 shows a good product image 200 of a good product work W1, a defective product image 201 of a defective product work W2, and an image (image causing confusion in judgment) 202 of a work W3 for which it is difficult to determine whether it is a good product or a defective product. Since there are no scratches or the like on the good product work W1 shown in FIG. 4, the user can easily determine that it is a good product. Also, since there is a large and noticeable scratch Wa on the defective product work W2, the user can easily determine that it is a defective product. However, there is a hardly noticeable scratch Wb on the work W3 for which it is difficult to make a determination. The work W3 with such a scratch Wb may be determined as a good product without any problem, but there are cases where it may be determined as a defective product and then returned to the good product line later. That is, even for the user, it is difficult to determine whether the work W3 is a good product or a defective product, and whichever determination is made, no particular problem occurs. It is a so-called work without a true correct answer. Even if the image 202 of such a work W3 is input to a learned inference model, it may not be possible to make a determination.

[0052] Here, basically, the learning process is carried out aiming to eliminate misjudgments between good products and defective products. For example, if a good product image is determined to be a defective product or an image for which no determination can be made, trial and error for adjusting the inference model is carried out aiming to determine that the good product image is a good product. Also, if a defective product image is determined to be a good product or an image for which no determination can be made, trial and error for adjusting the inference model is carried out aiming to determine that the defective product image is a defective product.

[0053] However, if the image with a judgment error is an image for which a judgment error may occur in the first place, such as the image 202 that is confusing in the judgment shown in FIG. 4, even though no further adjustment of the inference model is necessary, the user will try to adjust the inference model through trial and error, imposing an excessive burden on the user. On the other hand, if the image with a judgment error is an image that should be clearly judged, such as the non-defective image 200 or the defective image 201 shown in FIG. 4, further adjustment of the inference model is necessary, so it is a situation where the user should try through trial and error. Thus, in conventional machine learning, it has been difficult to determine whether it is a situation where the user should try through trial and error.

[0054] (Verification process) In order to solve the problem in conventional machine learning, that is, the problem that it is difficult to determine whether it is a situation where the user should still try to adjust through trial and error, in this embodiment, efforts are made in the verification process of the inference model. Hereinafter, it will be described based on the flowchart shown in FIG. 5.

[0055] In step SB1 after starting, the input unit 13e acquires a verification image for verifying the pass / fail judgment performance of the inference model. The verification image may be only defective images, only non-defective images, or both defective images and non-defective images. The verification image may be an image acquired before the verification process, an image newly acquired for the verification process, or an image acquired during the operation of the appearance inspection device 1. When the verification image is stored in the storage device 19, the input unit 13e can acquire it by reading the verification image from the storage device 19.

[0056] In step SB2, a process of attaching a defective product label or a non-defective product label to the verification image is executed. The defective product label and the non-defective product label indicate attributes related to the quality of the verification image. The defective product label is a label indicating that the verification image is a defective product image corresponding to a defective product. On the other hand, the non-defective product label is a label indicating that the verification image is a non-defective product image corresponding to a non-defective product. The user operates a mouse 52 or the like while displaying the verification image on the display device 4 and checking each one, and designates a defective product label if it is a defective product image corresponding to a defective product, and designates a non-defective product label if it is a non-defective product image corresponding to a non-defective product. Also, a plurality of defective product images can be grouped into one folder, and it is possible to collectively specify the attachment of defective product labels to the defective product images in that folder, or a plurality of non-defective product images can be grouped into one folder, and it is possible to collectively specify the attachment of non-defective product labels to the non-defective product images in that folder. The designated defective product label or non-defective product label is respectively attached to the corresponding verification image by the input unit 13e. Information indicating which label is attached is stored in the storage device 19 in a state associated with the corresponding image as information regarding label attachment.

[0057] To explain an example of a method for collectively attaching a defective product label or a non-defective product label, in the process of step SB2, first, the input unit 13e generates a user interface screen 250 as shown in FIG. 6 and the display control unit 13c causes it to be displayed on the display device 4. The user interface screen 250 is provided with a number information display area 251, a list display area 252, and an enlarged display area 253. A first table 260 is displayed in the number information display area 251. In the first table 260 displayed in the number information display area 251, the number of images with "no label" to which neither a non-defective product label nor a defective product label has been attached as information regarding label attachment, the number of images with "non-defective product label" to which a non-defective product label has been attached, and the number of images with "defective product label" to which a defective product label has been attached are respectively displayed.

[0058] In the first table 260 displayed in the quantity information display area 251, there is also a "display" column for making selections on whether to display the image of "no label" in the list display area 252, whether to display the image of "good product label" in the list display area 252, and whether to display the image of "defective product label" in the list display area 252. When the user operates the "display" column with the mouse 52 or the like, the operation is received by the input unit 13e.

[0059] In the first table 260 displayed in the quantity information display area 251, there is also a "batch assignment" column. Since neither the good product label nor the defective product label is assigned to the image of "no label", when the user operates "batch assignment" with the mouse 52 or the like, the "good product label" or "defective product label" can be batch-assigned to all of the images of "no label". The images with the "good product label" or "defective product label" assigned are classified into the image of "good product label" or the image of "defective product label" in the first table 260 and the "quantity" is updated. The information of the batch-assigned label is stored in the storage device 19 in a state associated with the corresponding image.

[0060] In the list display area 252, the images selected by operating the "display" column of the first table 260 are displayed. Specifically, when the input unit 13e detects that the "display" column of the first table 260 has been operated by the user, it receives the operation and displays the corresponding image in the list display area 252. For example, when the image of "good product label" is selected, the image of "good product label" is displayed in the list display area 252; when the image of "defective product label" is selected, the image of "defective product label" is displayed in the list display area 252; when the image of "no label" is selected, the image of "no label" is displayed in the list display area 252. It is also possible to display multiple types of images simultaneously.

[0061] In the enlarged display area 253, any one of the images displayed in the list display area 252 that is selected by the user is enlarged and displayed. Specifically, when the input unit 13e detects that one image has been selected from the images displayed in the list display area 252 by an operation of the user's mouse 52 or the like, the selected image is made larger than that displayed in the list display area 252 and is displayed in the enlarged display area 253.

[0062] The above is the processing in step SB2. After that, the process proceeds to step SB3. In step SB3, a process of assigning a first flag or a second flag to the verification image is executed. The first flag is assigned to a verification image for which it is easy to determine whether it is good or bad, such as the good product image 200 or the defective product image 201 shown in FIG. 4, and includes information on the good product label and the defective product label. Thereby, it can be clearly indicated that the defective product image or the good / bad image is a defective product or a good product, respectively.

[0063] On the other hand, the second flag is assigned to verification images for which it is more difficult to determine whether they are defective or non-defective than the defective or non-defective images to which the first flag is assigned. For example, the second flag is assigned to verification images with a lower defect rate than the defective images to which the first flag is assigned. Also, the second flag is assigned to verification images with a lower non-defect rate than the non-defective images to which the first flag is assigned. For any image that any user determines to be a defective image and any image that is determined to be a non-defective image, the first flag may be assigned. On the other hand, for any image that any user is confused about whether it is a defective image (or non-defective image), or an image that one user determines to be a defective image (or non-defective image) while another user determines it to be a non-defective image (or defective image), the second flag is assigned. Also, for images that are not completely non-defective images but meet the criterion that they may be treated as non-defective images, the second flag is assigned. After assigning the first flag, the second flag may be assigned, or conversely, after assigning the second flag, the first flag may be assigned. Step SB3 consists of a first flag assignment step and a second flag assignment step. The first flag assignment step and the second flag assignment step may be separated in the flow.

[0064] The method of assigning the first flag or the second flag is not particularly limited, and an example thereof is shown in FIG. 6. Among the verification images displayed in the list display area 252 of the user interface screen 250 in FIG. 6, when the user right-clicks the mouse 52 or the like on or near the verification image for which the user intends to assign the first flag or the second flag, the input unit 13e detects this and generates a window 254 for assigning the first flag or the second flag. The generated window 254 is displayed on the display device 4 by the display control unit 13c.

[0065] In window 254, “Must” (also referred to as the Must flag), which is an example of the first flag, and “Want” (also referred to as the Want flag), which is an example of the second flag, are displayed. Among “Must” and “Want”, one of them can be selected by the user through the operation of mouse 52. FIG. 6 shows a state where “Must” is selected. It is displayed in an identifiable manner which of “Must” and “Want” is selected. The “Must” or “Want” selected by the user is received by input unit 13e for its input. The received “Must” or “Want” is stored in storage device 19 in a state associated with the corresponding verification image. When “Must” or “Want” is selected, “Must” or “Want” may be displayed on the verification image.

[0066] Since the verification image with “Must” assigned is an image that is easy to judge, it is an image that we want to input all of them into the inference model so that the inference model can make an accurate judgment. On the other hand, the verification image with “Want” assigned is an image that can tolerate the inference model making a misjudgment. In this example, the first flag and the second flag can be assigned in two levels of “Must” and “Want”, but it is not limited to this. For example, as an expression such as “importance”, it may be possible to assign in three or more levels. When it is three or more levels, for example, it is set as 10 levels of “1 to 10”, and “1” is assigned to the verification image that can tolerate the inference model making a misjudgment as the “Want” with the lowest importance. The numerical value corresponding to “Want” can be, for example, “1 to 5”. “6 to 10” is the numerical value corresponding to “Must”, and the larger the numerical value, the more necessary it is for the inference model to accurately judge, and the corresponding numerical value may be assigned. The first flag or the second flag may be assigned with symbols, characters, etc. other than numerical values.

[0067] When the first flag or the second flag is added to the verification image, as shown in FIG. 7, the display control unit 13c displays the second table 261 in the number information display area 251 of the user interface screen 250. In the second table 261, as the addition information of the first flag or the second flag, the number of "no flag" images to which neither the first flag nor the second flag is added, the number of "Must" images to which the Must flag is added, and the number of "Want" images to which the Want flag is added are respectively displayed.

[0068] When the user clicks on the number enclosed by the circle 262 in the column of the second table 261 with the mouse 52, the processor 13a extracts only the verification images to which the non-defective label and the Must flag are added. The display control unit 13c displays the verification images extracted by the processor 13a in the list display area 252 of the user interface screen 250. The circle 262 is an example, and it is not necessary to actually display the circle 262. When the user selects another column of the second table 261 or a number within the column, the verification image corresponding to the column or the number is extracted and displayed in the list display area 252.

[0069] In step SB4 shown in FIG. 5, the inspection unit 13f reads the parameters of the learned inference model from the storage device 19 and constructs the inference model. After constructing the inference model, it proceeds to step SB5 and inputs the verification image into the inference model. Specifically, the inspection unit 13f is a part that inputs the defective or non-defective verification image into the inference model to obtain the pass / fail determination result, and inputs the verification images to which the Must flag is added and the verification images to which the Want flag is added into the inference model. In step SB5, it is also possible to input the verification images to which neither the Must flag nor the Want flag is added into the inference model. Step SB5 is an inspection step.

[0070] In step SB6, the inspection unit 13f obtains the determination result for the verification image input to the inference model in step SB5. Since this pass / fail determination result is for the verification image, it can also be said to be the verification result for each image of the inference model. In step SB5 described above, the inference model was input with the verification image with the Must flag and the verification image with the Want flag. Therefore, in step SB6, both the verification result of the inference model for the verification image with the Must flag and the verification result of the inference model for the verification image with the Want flag can be obtained.

[0071] In step SB7, the verification result is displayed on the display device 4. The display control unit 13c is a part that causes the display device 4 to display the verification result of the pass / fail determination performance of the inference model based on the pass / fail determination result obtained by the inspection unit 13f. Specifically, the display control unit 13c is configured to be able to display on the display device 4 the verification result of the inference model for the image with the Want flag separately from the verification result of the inference model for the image with the Must flag. This step SB7 is a display step.

[0072] The display control unit 13c selects an image to which neither the Must flag nor the Want flag is assigned from the verification images and causes the display device 4 to display it. For example, the display control unit 13c can generate and display on the display device 4 a result display screen 263 as shown in FIG. 8. On the result display screen 263, the pass / fail determination results of the verification images with the good product label and the pass / fail determination results of the verification images with the defective product label are displayed. In this example, among the verification images with the defective product label, there is one image for which the determination was impossible. Impossible to determine means an image for which the inference model could not make a pass / fail determination.

[0073] When neither the Must flag nor the Want flag is assigned to an image for which determination is impossible, the display control unit 13c selects that image. By displaying the selected image on the display device 4 by the display control unit 13c, the user is prompted to assign one of the Must flag or the Want flag. When there are a plurality of images for which determination is impossible and neither the Must flag nor the Want flag is assigned, the display control unit 13c selects and displays those plurality of images on the display device 4. The user can be prompted to assign one of the Must flag or the Want flag to each of these plurality of images one by one. When there is even one image among the images for which determination is impossible that should be assigned the Must flag, the user can be made aware that it is necessary to perform trial and error in adjusting the inference model.

[0074] On the other hand, if it is possible to assign the Want flag to all of the images for which determination is impossible, it means that all of the images assigned the Must flag have been accurately determined. Therefore, the user can judge that the learning of the inference model is completed at that point. Instead of leaving it to the user to judge that the learning of the inference model is completed, for example, the inspection unit 13f detects whether the Want flag has been assigned to all of the images for which determination is impossible, and when this is detected, notifies the user that the learning of the inference model is completed.

[0075] In addition to the Must flag and the Want flag, a hold flag may be assigned to the verification image. A hold flag is assigned to an image for which it is impossible to determine whether it is a good product image or a defective product image, or an image that is a defective product image but for which it is difficult to identify the defective part, and the hold flag and the assigned verification image can be associated and stored in the storage device 19. In this case, it is possible to display on the display device 4 the verification result including the image to which the hold flag is assigned and the verification result excluding the image to which the hold flag is assigned. The user can be made able to select which verification result to display.

[0076] (Specific example of the display form of the verification result) Next, the display forms of the verification results for the images with the Must flag and the verification results for the images with the Want flag will be described. However, the display forms of the verification results are not limited to the forms described below, and various display forms can be used.

[0077] In FIG. 9, as an example of the display form of the verification results, the case where a cumulative histogram is displayed is shown. That is, when the inspection unit 13f obtains the pass / fail determination results for each of a plurality of verification images, the display control unit 13c acquires from the inspection unit 13f the frequency of the images determined to be good products and the frequency of the images determined to be defective products. The display control unit 13c generates a cumulative histogram based on the frequency of the images determined to be good products and the frequency of the images determined to be defective products, and causes the display device 4 to display it. Since the cumulative histogram represents the frequencies of the number of correct answers for the images determined to be good products and the number of correct answers for the images determined to be defective products, information regarding the number of correct answers for the pass / fail determination can be displayed by displaying the cumulative histogram. The information regarding the number of correct answers is not limited to the frequency, and may be, for example, the correct answer rate calculated based on the number of correct answers.

[0078] For example, a histogram display area 270 as shown in FIG. 9 can be provided on the user interface screen, and the cumulative histogram can be displayed within the histogram display area 270. The histogram display area 270 is provided with a selection unit 271 for receiving a selection as to whether to exclude the pass / fail determination results for the images with the Want flag from among the verification images input to the inference model. The selection unit 271 is composed of, for example, a checkbox that can be checked or unchecked by an operation of a mouse 52 or the like by the user, various buttons, and the like.

[0079] FIG. 9A in Fig. 9 shows a case where the user operates the selection unit 271 and checks "Include images with the Want flag in the results". In this case, the pass / fail judgment results for images with the Want flag are not excluded. On the other hand, FIG. 9B in Fig. 9 shows a case where the user does not operate the selection unit 271 and does not check "Include images with the Want flag in the results". In this case, the pass / fail judgment results for images with the Want flag are excluded. The input unit 13e is configured to be able to receive the selection by the user's selection unit 271, that is, the selection of whether to exclude the pass / fail judgment results for images with the Want flag.

[0080] The selection result received by the input unit 13e is output to the display control unit 13c. When the selection unit 271 is not checked, the display control unit 13c excludes the pass / fail judgment results for images with the Want flag and generates a cumulative histogram based on the frequency of the pass / fail judgment for images with the Must flag and displays it on the display device 4 (see FIG. 9B). On the other hand, when the selection unit 271 is checked, the display control unit 13c does not exclude the pass / fail judgment results for images with the Want flag and generates a cumulative histogram based on the frequency of the pass / fail judgment for images with the Want flag and the frequency of the pass / fail judgment for images with the Must flag and displays it on the display device 4 (see FIG. 9A).

[0081] In FIG. 9A, since the cumulative histogram is generated including the pass / fail judgment results for images with the Want flag, the distinction between good product judgment and bad product judgment is not clear. However, in FIG. 9B, since the cumulative histogram is generated without including the pass / fail judgment results for images with the Want flag, the distinction between good product judgment and bad product judgment is clear. Therefore, the user can determine whether the judgment performance of the inference model is sufficient by looking at the cumulative histogram.

[0082] (Operation and effect of the embodiment) As described above, for the verification images that can clearly determine whether they are defective product images or non-defective product images, the user assigns a Must flag, while for the verification images without true correct answers, the user assigns a Want flag. The inspection unit 13f inputs the images with the Must flag into the inference model to obtain a pass / fail determination result, and also inputs the images with the Want flag into the inference model to obtain a pass / fail determination result. Thereafter, since the display device 4 can display the verification results of the images with the Must flag and the verification results of the images with the Want flag separately, the user can grasp the pass / fail determination performance of the inference model when excluding the images without true correct answers. Thereby, the user can easily determine whether the inference model already exhibits sufficient performance or whether trial and error in adjusting the inference model is still necessary.

[0083] The above-described embodiments are merely illustrative in all respects and should not be construed in a limiting sense. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

Industrial Applicability

[0084] As described above, the present invention can be used when inspecting the appearance of a workpiece.

Explanation of Reference Numerals

[0085] 1 Appearance inspection device 4 Display device (display unit) 13a Processor 13b Memory 13c Display control unit 13d Learning unit 13e Input unit 13f Inspection unit

Claims

1. An appearance inspection device that inputs a work image obtained by photographing a work to be inspected into a machine learning network and determines whether the work is good or bad based on the input work image, comprising: a storage unit that stores an inference model after learning processing; an input unit that assigns a first flag indicating a defective product or a non-defective product to a defective product image or a non-defective product image for verification for verifying the pass / fail determination performance of the inference model; an inspection unit that inputs the defective product image or non-defective product image for verification into the inference model to obtain a pass / fail determination result; a display control unit that causes a display unit to display a verification result of the pass / fail determination performance of the inference model based on the pass / fail determination result obtained by the inspection unit. The appearance inspection device is provided with: the input unit is capable of receiving an input of a second flag different from the first flag for an image for verification; the inspection unit is configured to input an image to which the second flag is assigned into the inference model to obtain a pass / fail determination result; the display control unit is configured to be able to display, on the display unit, a verification result for an image to which the second flag is assigned separately from a verification result for an image to which the first flag is assigned. An appearance inspection device.

2. In the appearance inspection device according to Claim 1, the display control unit is configured to be able to display information regarding the number of correct answers for the pass / fail determination of an image to which the first flag is assigned by excluding the verification result for an image to which the second flag is assigned. An appearance inspection device.

3. In the appearance inspection device according to Claim 1 or 2, the input unit is configured to be able to receive a selection as to whether to exclude the pass / fail determination result for an image to which the second flag is assigned; when the input unit receives a selection to exclude the pass / fail determination result for an image to which the second flag is assigned, the display control unit excludes the verification result for an image to which the second flag is assigned and displays information regarding the number of correct answers for the pass / fail determination of an image to which the first flag is assigned, while when the input unit receives a selection not to exclude the pass / fail determination result for an image to which the second flag is assigned, the display control unit is configured to be able to display information regarding the total number of correct answers combining the verification result for an image to which the second flag is assigned and the pass / fail determination for an image to which the first flag is assigned. An appearance inspection device.

4. In the appearance inspection device according to any one of Claims 1 to 3, the display control unit causes a plurality of the images for verification to be displayed in a list on the display unit. The input unit is an appearance inspection device that accepts the assignment of the first flag or the second flag to an image selected from a plurality of the verification images listed on the display unit.

5. In the appearance inspection device according to claim 4, the display control unit is an appearance inspection device that selects an image to which neither the first flag nor the second flag is assigned from the verification images and displays it on the display unit.

6. In the appearance inspection device according to any one of claims 1 to 3, the display control unit displays a list of a plurality of the verification images on the display unit, the input unit is an appearance inspection device that accepts the assignment of the second flag to an image selected from a plurality of the verification images listed on the display unit and to which the first flag is assigned.

7. In the appearance inspection device according to any one of claims 4 to 6, the display control unit is an appearance inspection device that selects an image for which the inference model could not determine good or bad and displays it on the display unit from the verification images.

8. In the appearance inspection device according to any one of claims 1 to 7, the display control unit is an appearance inspection device configured to generate and display on the display unit a cumulative histogram based on the frequency of images determined to be good product images and the frequency of images determined to be defective product images.

9. In the appearance inspection device according to claim 8, the display control unit generates and displays on the display unit a user interface screen that allows selection of whether to include the frequency of images to which the second flag is assigned when generating the cumulative histogram. When it is selected to include the frequency of images to which the second flag is assigned, the cumulative histogram is generated including the frequency of images to which the second flag is assigned, while when it is selected not to include the frequency of images to which the second flag is assigned, the cumulative histogram is generated without including the frequency of images to which the second flag is assigned.

10. In the appearance inspection device according to any one of claims 1 to 9, the appearance inspection device further includes a learning unit that inputs learning data into the machine learning network for learning and generates the inference model.

11. In the appearance inspection device according to any one of claims 1 to 10, An appearance inspection apparatus in which the input unit can receive an input of a second flag to be given to a verification image having a lower defect level than a defective product image to which the first flag is given.

12. In the appearance inspection apparatus according to any one of Claims 1 to 11, An appearance inspection apparatus in which the input unit can receive an input of a second flag to be given to a verification image having a lower non-defective level than a non-defective product image to which the first flag is given.

13. An appearance inspection method for inputting a work image obtained by photographing a work to be inspected into a machine learning network and determining whether the work is good or defective based on the input work image, A storage step of storing an inference model after learning processing; A first flag adding step of adding a first flag indicating that it is a defective product or a non-defective product to a defective product image or a non-defective product image for verification for verifying the good or defective determination performance of the inference model; A second flag adding step of receiving an input of a second flag different from the first flag for a verification image; An inspection step of inputting the image to which the first flag is added and the image to which the second flag is added into the inference model to obtain a good or defective determination result; An appearance inspection method including a display step of displaying a verification result for the image to which the second flag is added on a display unit separately from a verification result for the image to which the first flag is added.

Images