Appearance inspection apparatus and appearance inspection method

Abstract

To quickly present an alternative model without deteriorating usability for a user.SOLUTION: A first inference model 110 for determining the quality of an input image is generated by inputting learning data into a machine learning network to learn the data. Input images successively input to the first inference model 110 and a quality determination result of each input image are stored. A process of generating a second inference model 111 by inputting the stored plurality of input images into the machine learning network to learn the input images is performed in the background of a quality determination process during inspection. A display screen for displaying quality determination performance of the second inference model 111 is displayed on a display screen.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an appearance inspection apparatus and an appearance inspection method for inspecting the appearance of a workpiece.

Background Art

[0002] For example, Patent Document 1 discloses an appearance inspection apparatus that uses machine learning by a computer to determine whether a workpiece is a non-defective product or a defective product. In an appearance inspection apparatus using machine learning as disclosed in Patent Document 1, a large number of images are input into a machine learning network for learning before operation to generate an inference model, and then the operation proceeds to perform an appearance inspection of the workpiece.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, during the operation of the completed inference model, for example, it is conceivable that the surrounding environment of the workpiece changes. Specifically, there are cases where external light hits the workpiece from one direction at a certain time, but from the other direction at another time, or cases where subtle changes occur in the workpiece due to seasonal factors. The inference model cannot follow such changes in the surrounding environment and may make an incorrect determination that a non-defective product is a defective product or that a defective product is a non-defective product.

[0005] As described above, even for an inference model that has been once completed, its pass / fail judgment performance may deteriorate over time. However, there are no immediately available countermeasures at the stage when the deterioration of the pass / fail judgment performance is confirmed. Currently, it is necessary to go through a re-learning process of collecting learning images again, inputting them into a machine learning network, and training to generate a new inference model.

[0006] However, even if the training of the machine learning network is performed using a high-performance computer separate from the appearance inspection device, it takes a long time for the calculation. Thus, during the execution of the re-learning process, the appearance inspection of the work has to be stopped for a long time, which is a major problem for the user and ultimately leads to a deterioration in the usability of the appearance inspection device.

[0007] The present disclosure is made in view of such a point, and its object is to be able to promptly present an alternative model without causing a deterioration in the usability of the user.

Means for Solving the Problem

[0008] To achieve the above object, in one aspect of the present disclosure, it is possible to assume an appearance inspection apparatus that inputs a work image obtained by photographing a work to be inspected into a machine learning network and determines whether the work is good or bad based on the input work image. The appearance inspection apparatus includes a learning unit that inputs learning data into the machine learning network for learning and generates a first inference model for determining whether an input image is good or bad, an inspection unit that sequentially inputs an input image into the first inference model generated by the learning unit and determines whether the input image is good or bad, a storage unit that stores the input images sequentially input into the first inference model generated by the learning unit and the determination results of whether each input image is good or bad, and a display control unit that causes the display unit to display the determination result by the inspection unit. The learning unit can execute a process of inputting a plurality of input images stored in the storage unit into the machine learning network for learning to generate a second inference model in the background of the pass / fail determination process by the inspection unit. The display control unit can cause the display unit to display a display screen for displaying the pass / fail determination performance of the second inference model.

[0009] According to this configuration, when a work image is input into the first inference model that has been learned by inputting learning data, a determination of whether the work is good or bad is made based on the input work image. Further, by causing the machine learning network to learn an input image different from the image used during the learning of the first inference model, a second inference model having parameters different from those of the first inference model is generated. Since the generation of this second inference model is executed in the background of the pass / fail determination process by the inspection unit, it is not necessary to stop the appearance inspection of the work for a long time, and the usability for the user does not deteriorate. Also, when learning the second inference model, for example, by using an image captured at a time different from the time of acquiring the learning data of the first inference model as the input image, it is possible to generate a second inference model with high pass / fail determination performance for a work image when the surrounding environment of the work has changed compared to the time of acquiring the learning data of the first inference model. Note that the learning of the second inference model can be performed, for example, when the pass / fail determination process by the inspection unit is not being performed, such as after the appearance inspection of the work has ended, and is not limited to only during the background of the pass / fail determination process.

[0010] When the second inference model is generated, a display screen for displaying the quality determination performance of the generated second inference model is displayed on the display unit. Therefore, the user can compare the quality determination performance of the first inference model and the second inference model, and select the inference model with higher quality determination performance at that time to continue the appearance inspection of the workpiece.

[0011] Further, a setting unit for setting a first condition regarding the imaging time when the input image used to generate the second inference model is captured, such as the date and time, may be further provided. In this case, the learning unit acquires the first condition set by the setting unit, extracts an input image that satisfies the acquired first condition from among the plurality of input images stored in the storage unit, and inputs the extracted input image into the machine learning network for learning to generate the second inference model.

[0012] That is, for example, in a surrounding environment where the direction of external light on the workpiece changes between morning and afternoon within a day, by setting the time as the first condition for the imaging time, an image captured at a time when the surrounding environment is different from the time when the learning data for the first inference model was acquired can be extracted as the input image. Similarly, in the case of seasonal factors, by setting the date, month, etc. as the first condition, an image captured on a date, month, etc. when the surrounding environment is different from the time when the learning data for the first inference model was acquired can be extracted as the input image. And since the second inference model can be generated using images captured at different times, dates, months, etc. when the surrounding environment is different, a second inference model with high quality determination performance for the workpiece image when the surrounding environment changes can be generated.

[0013] Also, as the trigger condition, it may be possible to set a condition regarding a change in the feature amount of the input image or a statistical change in the quality determination result. In this case, when a change in the feature amount of the input image or a statistical change in the quality determination result satisfies the condition set by the setting unit, the learning of the second inference model can be started.

[0014] In addition, since the learning unit starts learning the second inference model according to a preset trigger condition, the second inference model can be automatically generated in the background of the pass / fail determination process.

[0015] In addition, the setting unit can set a predetermined period as the first condition. In this case, the learning unit extracts input images captured within the predetermined period set as the first condition from among the plurality of input images stored in the storage unit, and inputs the extracted input images into the machine learning network for learning to generate the second inference model.

[0016] In addition, the setting unit can set a second condition regarding the attributes of the input images used to generate the second inference model. In this case, the learning unit acquires the second condition set by the setting unit, extracts input images that satisfy both the acquired second condition and the first condition from among the plurality of input images stored in the storage unit, and inputs the extracted input images into the machine learning network for learning to generate the second inference model. Thereby, input images more suitable for learning the second inference model can be used based on the first condition and the second condition.

[0017] In addition, as an attribute included in the second condition, the setting unit can set whether the input image used to generate the second inference model is a good product image corresponding to a good product. In this case, the learning unit extracts, from among the plurality of input images stored in the storage unit, images that satisfy the first condition and are good product images as input images, and inputs the extracted input images into the machine learning network for learning to generate the second inference model.

[0018] In addition, as an attribute included in the second condition, the setting unit can set whether the input image used to generate the second inference model is a defective product image corresponding to a defective product. In this case, the learning unit extracts, from among the plurality of input images stored in the storage unit, an image that satisfies the first condition and is a defective product image as the input image, and inputs the extracted input image into the machine learning network for learning to generate the second inference model.

[0019] In addition, the learning unit can extract a defective product image in which the feature amount of the defective product image included in the plurality of input images stored in the storage unit is equal to or greater than a predetermined value.

[0020] The setting unit according to another aspect can set a third condition regarding the number of input images used to generate the second inference model. In this case, the learning unit acquires the third condition set by the setting unit, extracts, from among the plurality of input images stored in the storage unit, an input image that satisfies both the acquired third condition and the first condition, and inputs the extracted input image into the machine learning network for learning to generate the second inference model.

[0021] In addition, as the third condition, the setting unit can set the ratio of B to A when the number of the plurality of input images stored in the storage unit is A and the number of input images used to generate the second inference model is B. In this case, the learning unit can extract the number of input images corresponding to the ratio from among the plurality of input images stored in the storage unit, and input the extracted input images into the machine learning network for learning to generate the second inference model.

[0022] In addition, since the learning data used when generating the first inference model can also be used when generating the second inference model, the number of input images used when generating the second inference model can be increased.

[0023] In addition, the learning unit can input verification image data with pre-assigned pass / fail information into the first inference model and the second inference model respectively to perform pass / fail determination on the verification image data. The learning unit calculates a first matching rate, which is the matching rate between the pass / fail information of the verification image data and the pass / fail determination result by the first inference model, and a second matching rate, which is the matching rate between the pass / fail information of the verification image data and the pass / fail determination result by the second inference model. Since the display control unit can provide display areas for the first matching rate and the second matching rate on a display screen for comparing the pass / fail determination performance of the first inference model and the second inference model, and display the display screen on the display unit, the user can quantitatively and easily compare the pass / fail determination performance of the first inference model and the second inference model.

Advantages of the Invention

[0024] As described above, while generating the first inference model with learning data, the second inference model is generated in the background of pass / fail determination processing with a plurality of input images stored in the storage unit, and the pass / fail determination performance of the generated second inference model can be displayed. Therefore, it is possible to select a suitable inference model at that time and perform pass / fail determination processing. As a result, it is possible to quickly present an alternative model without causing deterioration in the user experience.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely illustrative in nature and is not intended to limit the present invention, its applications, or its uses.

[0027] FIG. 1 is a schematic diagram showing the configuration of an appearance inspection apparatus 1 according to an embodiment of the present invention. The appearance inspection apparatus 1 is an apparatus for determining the quality of a work image obtained by imaging a work, which is an inspection target such as various parts and products, and can be used at a production site such as a factory. Specifically, a machine learning network is constructed inside the appearance inspection apparatus 1. A work image obtained by imaging a work to be inspected is input to the generated machine learning network, and the machine learning network can determine the quality of the work image.

[0028] The entire work may be the inspection target, or only a part of the work may be the inspection target. Also, a single work may include a plurality of inspection targets. Further, the work image may include a plurality of works.

[0029] The appearance inspection device 1 includes a control unit 2 serving as the device main body, an imaging unit 3, a display device (display unit) 4, and a personal computer 5. The personal computer 5 is not essential and can also be omitted. Instead of the display device 4, various information and images can be displayed using the personal computer 5, and the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0030] In FIG. 1, as an example of the configuration example of the appearance inspection device 1, the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are described. However, any plurality of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Also, the control unit 2 can be divided into a plurality of units and a part of it can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into a plurality of units and a part of it can be incorporated into other units.

[0031] (Configuration of the imaging unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that executes acquisition of a workpiece image. The camera module 14 includes an AF motor 141 that drives the imaging optical system and an imaging substrate 142. The AF motor 141 is a part that automatically performs focus adjustment by driving the lens of the imaging optical system, and focus adjustment can be performed by a method such as conventional contrast autofocus. The imaging substrate 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to be able to acquire a color image. Instead of the CMOS sensor 143, a light receiving element such as a CCD sensor can also be used.

[0032] The illumination module 15 includes an LED (light emitting diode) 151 as a light emitter that illuminates an imaging area including the workpiece, and an LED driver 152 that controls the LED 151. The light emission timing, light emission time, and light emission amount of the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided as an external illumination unit separately from the imaging unit 3.

[0033] (Configuration of the display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel, an organic EL panel, or the like. The workpiece image, user interface image, etc. output from the control unit 2 are displayed on the display device 4. Further, when the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used as an alternative to the display device 4.

[0034] (Operating device) Examples of the operating device for the user to operate the appearance inspection device 1 include the keyboard 51 and mouse 52 of the personal computer 5, but are not limited thereto, and any device configured to be able to receive various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operating device.

[0035] Operations by the user on the keyboard 51 and mouse 52 can be detected by the control unit 2. Further, the touch panel 41 is a conventionally well-known touch-type operation panel equipped with, for example, a pressure sensor, and the touch operation of the user can be detected by the control unit 2. The same applies when other pointing devices are used.

[0036] (Configuration of the control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. A processor 13a is provided on the main board 13. The processor 13a controls the operations of the connected boards and modules. For example, the processor 13a outputs an illumination control signal for controlling the lighting / extinguishing of the LED 151 to the LED driver 152 of the illumination module 15. The LED driver 152 switches the lighting / extinguishing of the LED 151 and adjusts the lighting time according to the illumination control signal from the processor 13a, and also adjusts the light quantity of the LED 151 and the like.

[0037] In addition, the processor 13a outputs an imaging control signal for controlling the CMOS sensor 143 to the imaging board 142 of the camera module 14. The CMOS sensor 143 starts imaging according to the imaging control signal from the processor 13a, and performs imaging by adjusting the exposure time to an arbitrary time. That is, the imaging unit 3 images within the field of view of the CMOS sensor 143 according to the imaging control signal output from the processor 13a. If there is a workpiece within the field of view, the workpiece will be imaged. However, if there are other objects within the field of view, they can also be imaged. For example, the appearance inspection device 1 can image a good product image corresponding to a good product and a defective product image corresponding to a defective product by the imaging unit 3 as learning images for the machine learning network. The learning images do not have to be the images captured by the imaging unit 3, and can also be the images captured by other cameras or the like. The learning images are learning data for inputting into the machine learning network to train the machine learning network.

[0038] On the other hand, during the operation of the appearance inspection device, the imaging unit 3 can image the workpiece. In addition, the CMOS sensor 143 is configured to be able to output a live image, that is, the currently captured image, at any time with a short frame rate.

[0039] When imaging by the CMOS sensor 143 is completed, the image signal output from the imaging unit 3 is input to and processed by the processor 13a on the main board 13, and is also stored in the memory 13b on the main board 13. Details of the specific processing content by the processor 13a on the main board 13 will be described later. Incidentally, a processing device such as an FPGA or a DSP may be provided on the main board 13. The processor 13a may be integrated with a processing device such as an FPGA or a DSP.

[0040] A display control unit 13c is provided on the main board 13. The display control unit 13c is a part that generates a display screen, controls the display device 4, and causes the display device 4 to display the display screen. Specific operations of the display control unit 13c will be described later.

[0041] The connector board 16 is a part that receives power supply from the outside through a power connector (not shown) provided in the power interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and module, etc. Specifically, it distributes power to the illumination module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 includes an AF motor driver 181. The AF motor driver 181 supplies driving power to the AF motor 141 of the camera module 14 to realize autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a on the main board 13.

[0042] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, communication between the main board 13 and an external control device (not shown), etc. The external control device can be, for example, a programmable logic controller or the like. The communication may be wired or wireless, and any communication form can be realized by a conventionally well-known communication module.

[0043] The control unit 2 is provided with a storage device (storage unit) 19 composed of, for example, a solid state drive, a hard disk drive, etc. The storage device 19 stores a program file 80, a setting file, etc. (software) for enabling the above-mentioned hardware to execute each control and process described later. The program file 80 and the setting file are stored in a storage medium 90 such as an optical disk, for example, and the program file 80 and the setting file stored in this storage medium 90 can be installed in the control unit 2. The program file 80 may be downloaded from an external server using a communication line. Further, the storage device 19 can store, for example, the above-mentioned image data, parameters for constructing the machine learning network of the appearance inspection device 1, etc.

[0044] That is, in the appearance inspection device 1, by using learning data to train a machine learning network, the parameters of the machine learning network are adjusted to generate an inference model. A work image obtained by photographing a work to be inspected is input to the inference model, and it is possible to determine whether the work is good or bad based on the input work image. By using this appearance inspection device 1, an appearance inspection method for determining whether a work is good or bad based on a work image can be executed.

[0045] (Changes in the surrounding environment of the work) Here, changes in the surrounding environment of the work for which appearance inspection is performed by the appearance inspection device 1 and the appearance inspection method will be described. Assume a case where the appearance inspection of the work is performed by the appearance inspection device 1 and the appearance inspection method when the work is being conveyed along a predetermined conveyance path by, for example, a conveyor. In such a case, generally, the work on the conveyance path is imaged by the imaging unit 3 fixed at a predetermined position, so basically all the works are imaged at almost the same position, and even if the work moves within the field of view of the imaging unit 3, it stays within a narrow range in the width direction of the conveyance path.

[0046] If the work is imaged at almost the same position, it is considered that the external light hardly changes. However, for example, at a site where sunlight is captured, the irradiation direction of the external light changes over time, and the intensity of the external light also changes. Also, depending on the site, the lighting state may change between day and night, which also causes changes in the irradiation direction and intensity of the external light.

[0047] FIG. 3 shows a first work image 100 in which a work W is imaged at a first time, and a work image 101 in which a second work W is imaged at a second time different from the first time. As shown in this figure, at the first time, external light is irradiated from the upper left of the work W, but at the second time, external light may be irradiated from the upper right. In the first work image 100 captured at the first time, a shadow is formed at the lower right of the work W, while in the second work image 101 captured at the second time, a shadow is formed at the lower left of the work W. Therefore, even though the work W itself is the same, the first work image 100 and the second work image 101 are different images when viewed as images. Also, members around the work W may be imaged as shadows, and in that case, the positions and shapes of the shadows are different between the work image 100 captured at the first time and the work image 100 captured at the second time. Therefore, the first work image 100 and the second work image 101 are different images.

[0048] Also, a change may occur in the work W due to a change in season. For example, there may be a difference between a work image captured in summer (the first time) and a work image captured in winter (the second time).

[0049] Furthermore, for example, a work W molded by a mold may be the inspection object. There may be a change in the shape of the work W between a work W molded by a new mold and a work W molded by a mold after a period has elapsed since the start of use (assuming this is also a good work). There may be a difference between a work image captured when the mold is new (the first time) and a work image captured when a period has elapsed since the start of use of the mold (the second time).

[0050] Changes in external light, seasonal changes, changes in the molding die, etc. are changes in the surrounding environment of the workpiece W. Even for workpieces W that the user recognizes as being the same, as described above, the workpiece images 100 and 101 may be different due to changes in the surrounding environment of the workpiece W. For example, if only the workpiece image 100 captured at the first time is input to the machine learning network for learning to generate an inference model, when the workpiece image 101 captured at the second time is input to the inference model, although the workpiece W is a non-defective product, there is a risk of misjudging it as a defective product. The opposite misjudgment can also occur. That is, during the operation of an inference model that has once been completed, a deterioration in the pass / fail judgment performance may occur due to changes in the surrounding environment of the workpiece W.

[0051] The appearance inspection apparatus 1 according to the present embodiment has a configuration capable of quickly responding to the surrounding environment of the workpiece W that has changed during operation. Hereinafter, an example of the configuration will be described.

[0052] (Configuration of the processor) As shown in FIG. 2, the processor 13a is provided with a learning unit 13d, an inspection unit 13e, and a setting unit 13f. The learning unit 13d, the inspection unit 13e, and the setting unit 13f may be parts configured by hardware or may be parts configured by executing software. Further, the learning unit 13d, the inspection unit 13e, and the setting unit 13f do not necessarily have to be provided on the main board 13, and a part or all of these may be provided on a board other than the main board 13.

[0053] The learning unit 13d is a part that inputs learning data into a machine learning network for learning and generates a first inference model 110 (shown in FIG. 4) for determining the quality of an input image. The learning unit 13d may be configured by, for example, a learning computer separate from the control unit 2. The learning computer is configured to perform machine learning at high speed. By communicably connecting the learning computer and the control unit 2, parameters for constructing the first inference model 110 generated by the learning computer are transmitted to the control unit 2, and the control unit 2 can construct the first inference model 110. The second inference model 111 will be described later.

[0054] Also, as shown in FIG. 4, the storage device 19 stores a first input image set composed of a plurality of first work images 100. The first work image 100 may be an image captured by the user using the imaging unit 3, or an image captured by a camera separate from the imaging unit 3. In any case, it is prepared in advance and stored in the storage device 19. As the first work images 100 constituting the first input image set, both a good product image captured of a good product and a defective product image captured of a defective product may be included, or only one of them may be included.

[0055] In FIG. 2, the storage device 19 is shown as being integrated with the control unit 2, but the storage device 19 may be separate from the control unit 2. Examples of such a storage device 19 include a network-attached storage (NAS). The NSA and the control unit 2 are connected by a communication line such as a wired LAN or a wireless LAN. The second input image set will be described later.

[0056] FIG. 5 is a flowchart showing an example of the procedure of the appearance inspection method. In step SA1 after starting, the learning unit 13d prepares an unlearned machine learning network. The unlearned machine learning network has, for example, initial values of parameters randomly determined. In step SA2, as shown in FIG. 4, the learning unit 13d inputs the learning dataset into the unlearned machine learning network. In this example, the same dataset as the first input image set stored in the storage device 19 can be used as the learning dataset. In this case, after the learning unit 13d reads the first input image set from the storage device 19, the input images constituting the first input image set are sequentially input into the unlearned machine learning network. Then, in step SA3, the first inference model 110 is generated. At this time, only good product images may be input for good product learning, only defective product images may be input for defective product learning, or both types of learning may be performed. Step SA3 is the first learning step.

[0057] When the input image of the first input image set is a defective product image, annotation is performed on the defective product image before it is input into the machine learning network. That is, the user pre-performs processing such as attaching a label indicating that it is a defective product image to the defective product image, and designating defective parts of the defective product image. The label information given by the annotation and the defective part information designated by the annotation are stored in the storage device 19 in a state associated with the corresponding defective product image. Therefore, during defective product learning, the parameters of the machine learning network are adjusted using the label information and the defective part information, and the obtained parameters are stored in the storage device 19 or the like. The first inference model 110 can be constructed using the obtained parameters.

[0058] After constructing the first inference model 110 in the control unit 2, the process proceeds to step SA4, where the inspection unit 13e captures the work W to be inspected using the imaging unit 3 to obtain a work image. After that, the process proceeds to step SA5, where the inspection unit 13e inputs the work image obtained in step SA4 into the first inference model 110 generated in step SA3. The work image input into the first inference model 110 in step SA5 is the input image. After the input image is input, in step SA6, the inspection unit 13e determines whether the input image is good or bad. For example, if there is a location in the abnormality degree map output from the first inference model 110 that shows a reaction above a predetermined level, it can be determined that the work of the input image is a defective product. On the other hand, if there is no location in the abnormality degree map output from the first inference model 110 that shows a reaction above a predetermined level, it can be determined that the work of the input image is a non-defective product. The pass / fail determination result by the inspection unit 13e can be obtained within the processor 13a. Steps SA5 and SA6 constitute the inspection step.

[0059] After step SA6, the process proceeds to step SA7, where the processor 13a associates the input image input into the first inference model 110 with the pass / fail determination result of the input image and stores them in the storage device 19. Step SA7 is the storage step.

[0060] Steps SA4 to SA7 are repeated while the appearance inspection of the work W is being performed, that is, during the operation of the appearance inspection device 1. That is, by sequentially inputting the input images of the first work W, the second work W, the third work W,... captured into the first inference model 110, the pass / fail determination results of each input image can be sequentially obtained. Once the pass / fail determination result is obtained, the input image and the pass / fail determination result of each input image are stored. When the operation of the appearance inspection device 1 is stopped, the imaging of the work W stops.

[0061] After step SA7, the process proceeds to step SA8, and the display control unit 13c causes the display device 4 to display the pass / fail determination result by the inspection unit 13e. For example, the display control unit 13c generates a result display user interface screen 200 as shown as an example in FIG. 6 and outputs it to the display device 4. The result display user interface screen 200 is provided with an image display area 201 where the input image is displayed and a result display area 202 where the pass / fail determination result is displayed. In the image display area 201, the input images input to the first inference model 110 are displayed in time series. In the image display area 201, a plurality of input images may be displayed in a list format, or only one input image may be displayed.

[0062] In the result display area 202, the pass / fail determination result is displayed for each input image, and one input image and one pass / fail determination result are associated with each other. The display form of the pass / fail determination result is not particularly limited as long as the user can distinguish between a good product and a defective product, and examples thereof include characters, symbols, and the like.

[0063] As shown in FIG. 6, as the number of appearance inspections increases, the number of input images and the number of pass / fail determination results increase correspondingly. That is, the input images and the pass / fail determination results are accumulated in the storage device 19 by repeating steps SA4 to SA7. A plurality of input images accumulated by repeating steps SA4 to SA7 are shown as a second input image set in FIG. 4, and are stored in the storage device 19 as the second input image set.

[0064] The input images constituting the second input image set are images of the second work image 101 taken at a second time after a lapse of time from the time of imaging the first work image 100 constituting the first input image set (the first time shown in FIG. 3). The second input image set is an image set used for background learning described later and is used to generate a second inference model by background learning.

[0065] In step SA9, the trigger condition for background learning to be executed in the subsequent step SA10 and the method for selecting the image set (the second input image set) to be used in the background learning are read. The trigger condition is the condition for starting background learning, which is set by the user in the setting unit 13f shown in FIG. 2. Details will be described later, and an example of the trigger condition is shown in FIG. 7. Also, in the setting unit 13f, as the condition regarding the selection of the above image set (shown in FIG. 8), the first condition regarding the imaging timing when the input image used to generate the second inference model is captured can be set. As the first condition, for example, a predetermined period can be cited. As the predetermined period, as shown in FIG. 7, it can include monthly, weekly, daily. If it is daily, the date and time are specified, if it is weekly, the day of the week and time are specified, and if it is daily, the time is specified. Also, as the predetermined period, for example, yesterday, the last 2 days, the last 3 days, …, the last 1 week, the last 2 weeks, …, the last 1 month, the last 2 months, etc. can be cited. Also, as the predetermined period, it can include between the first time and the second time (the first time is earlier on the time axis), between the first date and the second date (the first date is earlier on the time axis), etc. Also, as a method for setting the predetermined period, for example, a method of setting the number of the most recent input images may be used. For example, by setting it as "the most recent 100 sheets", it means that the period during which those 100 sheets were captured is set.

[0066] When setting the predetermined period, the setting unit 13f causes the display device 4 to display a user interface screen for setting and accepts the setting by the user. When the user operates the keyboard 51, the mouse 52, etc. to input the above-described time, date, number of sheets, etc., the setting unit 13f accepts it as a set value and stores it in the storage device 19 etc. In step SA9, the set value may be read from the storage device 19. The setting by the user can be performed at any time, for example, before step SA1, before step SA4, etc.

[0067] The setting unit 13f can set a second condition regarding the attributes of the input image used to generate the second inference model 111. As shown in FIG. 7, for example, as the attributes included in the second condition, whether the input image used to generate the second inference model 111 is a good product image corresponding to a good product, whether it is a defective product image corresponding to a defective product, the luminance value of the input image, the position of position correction, the edge strength, etc. can be set. In this case, the user may set on the user interface screen for setting that only the good product image is used as the input image. For example, the good product image can be made selectable by a checkbox, a selection button, or the like.

[0068] As a method for discriminating a good product image, for example, there are a method of regarding all the work images captured at a certain time as good product images, a method of regarding those that satisfy specific conditions as good product images, and the like. As the specific conditions, a condition that the feature amount of the image is less than a predetermined value can be cited. That is, those that can be determined to be good product images with a margin by the feature amount are extracted.

[0069] Also, for example, as the attributes included in the second condition, whether the input image used to generate the second inference model 111 is a defective product image corresponding to a defective product can be set. In this case, the user may set on the user interface screen for setting that only the defective product image is used as the input image. For example, the defective product image can be made selectable by a checkbox, a selection button, or the like.

[0070] As a method for discriminating a defective product image, for example, there are a method of regarding all the work images captured at a certain time as defective product images, a method of regarding those that satisfy specific conditions as defective product images, and the like. As the specific conditions, a condition that the feature amount of the image is greater than or equal to a predetermined value can be cited. That is, those that can be determined to be defective product images with a margin by the feature amount are extracted.

[0071] The setting unit 13f can set a third condition regarding the number of input images used to generate the second inference model. As the third condition, when the number of a plurality of input images stored in the storage device 19 is A and the number of input images used to generate the second inference model is B, the ratio C of B to A can be set. For example, when 200 input images are accumulated by repeating steps SA4 to SA6, A becomes 200. If only 100 of them are to be used to generate the second inference model, 50% is set as the ratio C on the user interface screen for setting. The ratio C can be any value, which may be specified by the user or may be a value pre-stored in the storage device 19.

[0072] Also, the third condition may be the accumulated number of pass / fail images, the accumulated number of defective product images, etc. Further, the third condition may be the moving average of the good product scores, etc. That is, the setting unit 13f may be able to set a condition regarding the scores of the input images used to generate the second inference model. When the setting unit 13f receives a specific score setting by the user on the user interface screen for setting, it sets that as the condition. When setting the score, a certain range can also be specified.

[0073] The setting unit 13f can also set whether to execute or not execute annotation for defective product images. That is, when setting the execution of annotation on the user interface screen for setting, the part extracted as the defective part in the input images determined to be defective in step SA6 is treated as the true defective part.

[0074] The setting unit 13f can also set whether to generate the second inference model by good product learning or defective product learning. That is, when setting good product learning on the user interface screen for setting, the input images are set to be only good product images, while when setting defective product learning, the input images are set to be only defective product images.

[0075] The setting unit 13f can also set how to handle the first input image set used for generating the first inference model 110. That is, on the user interface screen for setting, it is possible to set whether to use the first input image set for generating the second inference model 111. When it is set to use the first input image set for generating the second inference model 111, the first input image set is input to the machine learning network to generate the second inference model 111. On the other hand, when it is set not to use the first input image set for generating the second inference model 111, the second inference model 111 is generated only with the second input image set without inputting the first input image set to the machine learning network.

[0076] The user can also set on the user interface screen for setting the trigger conditions for starting background learning. As shown in FIG. 7, as the trigger conditions, for example, settings such as daily, weekly, and monthly are possible. When the set date and time arrive, the setting unit 13f outputs a trigger signal to start background learning.

[0077] The above trigger conditions may be automatically set regardless of the user's setting. For example, the setting unit 13f calculates at least one of the ratio of defective product images and the ratio of non-defective product images among the input images accumulated by repeating steps SA4 to SA6. When the calculated ratio changes by a predetermined amount or more, the setting unit 13f outputs a trigger signal to start background learning. Also, the setting unit 13f acquires at least one of the number of defective product images and the number of non-defective product images, and when one of the numbers reaches a predetermined number or more, the setting unit 13f outputs a trigger signal to start background learning.

[0078] Also, the setting unit 13f acquires information indicating the characteristics of the input images accumulated by repeating steps SA4 to SA6, and when the characteristics of the input images change by a predetermined amount or more, the setting unit 13f outputs a trigger signal to start background learning.

[0079] Further, the setting unit 13f calculates the moving average of the scores of the good product images among the input images accumulated by repeating steps SA4 to SA6, and outputs a trigger signal to start background learning when the calculated moving average changes by a predetermined amount or more.

[0080] Also, the setting unit 13f acquires the luminance value of the input image acquired in step SA4, and when the luminance value changes by a predetermined value or more, outputs a trigger signal to start background learning on the assumption that the illumination condition has changed.

[0081] Also, the setting unit 13f acquires the position of the position correction of the input image acquired in step SA4, and when the position changes by a predetermined amount or more, or acquires the edge intensity of the input image acquired in step SA4, and when the edge intensity changes by a predetermined amount or more, outputs a trigger signal to start background learning.

[0082] In step SA10 of FIG. 5, background learning is executed according to the above-described preset trigger conditions. First, when the setting unit 13f outputs a trigger signal, the learning unit 13d acquires the trigger signal. The learning unit 13d that has acquired the trigger signal executes, in the background of the pass / fail determination process by the inspection unit 13e, a process of inputting the second input image set stored in the storage device 19 into the machine learning network for learning to generate the second inference model 111. Step SA10 is the second learning step.

[0083] The second input image set may be composed of all of the input images accumulated by repeating steps SA4 to SA6, or may be composed only of the input images that satisfy the above conditions set by the setting unit 13f. When no particular conditions are set by the setting unit 13f, all of the input images accumulated by repeating steps SA4 to SA6 are used as the second input image set.

[0084] When the above conditions are set by the setting unit 13f, the input images that make up the second input image set are extracted as follows. That is, when the first condition regarding the imaging time is set by the setting unit 13f, the learning unit 13d acquires the first condition set by the setting unit 13f, and extracts the input images that satisfy the acquired first condition from among the plurality of input images stored in the storage device 19, and configures the second input image set only with the extracted input images. If a predetermined period is set as the first condition, the learning unit 13d extracts the input images captured within the predetermined period set as the first condition from among the input images accumulated by repeating steps SA4 to SA6. Thereby, the second inference model 111 can be generated only with the input images of the period desired by the user.

[0085] When the second condition regarding the attributes of the input images is set by the setting unit 13f, the learning unit 13d also acquires the second condition set by the setting unit 13f, and extracts the input images that satisfy both the acquired second condition and the first condition from among the input images accumulated by repeating steps SA4 to SA6. Thereby, it is possible to generate the second inference model 111 only with the good product images acquired within the period desired by the user, or to generate the second inference model 111 only with the defective product images acquired within that period.

[0086] When the third condition regarding the number of input images is set by the setting unit 13f, the learning unit 13d also acquires the third condition set by the setting unit 13f, and extracts the input images that satisfy both the acquired third condition and the first condition from among the input images accumulated by repeating steps SA4 to SA6. When the above ratio C is set as the condition of the third condition, the number of input images corresponding to the ratio is extracted from among the input images accumulated by repeating steps SA4 to SA6.

[0087] The learning unit 13d inputs the second input image set composed of the input images extracted as described above into the machine learning network for learning. In step SA11, the parameters of the machine learning network are adjusted to generate a second inference model 111 having parameters different from those of the first inference model 110. The obtained parameters are stored in the storage device 19 or the like. Background learning is learning that is performed while the pass / fail determination process by the inspection unit 13e is being executed. In other words, it is to perform the pass / fail determination process by the inspection unit 13e and the learning using the second input image set in parallel. By performing background learning, it is not necessary to stop the appearance inspection of the work W for a long time when generating the second inference model 111, and the usability of the user does not deteriorate. Note that the learning of the second inference model 111 can also be performed, for example, when the pass / fail determination process by the inspection unit 13e is not being performed, for example, after the appearance inspection of the work W is completed, for example, when the appearance inspection device 1 is stopped or set, and is not limited to only during the background of the pass / fail determination process.

[0088] Steps SA10 and SA11 may be executed multiple times. For example, if the trigger condition is daily, steps SA10 and SA11 are executed daily. Therefore, the second inference models 111 increase over time, and their respective parameters are stored in the storage device 19 in a distinguishable manner.

[0089] In step SA12, the display control unit 13c causes the display device 4 to display a display screen for displaying the pass / fail determination performance of the second inference model 111. At this time, a display screen for comparing the pass / fail determination performance of the first inference model 110 and the second inference model 111 may be displayed on the display device 4. Also, a display screen for confirming the pass / fail determination performance of a plurality of second inference models 111 may be displayed on the display device 4.

[0090] Step SA12 is a display step. Before displaying, the learning unit 13d obtains the goodness-of-fit determination performance of the first inference model 110 and the second inference model 111. Specifically, the learning unit 13d inputs verification image data with pre-assigned goodness-of-fit information into the first inference model 110 and executes the goodness-of-fit determination of the verification image data using the first inference model 110. Examples of the goodness-of-fit information include labels indicating that the image is a good product, labels indicating that the image is a defective product, and the like. The learning unit 13d calculates a first matching rate, which is the matching rate between the goodness-of-fit information of the verification image data and the goodness-of-fit determination result by the first inference model 110. If all of the goodness-of-fit determination result by the first inference model 110 and the goodness-of-fit information of the verification image data match, the first matching rate is 100%. Note that the verification image data is pre-stored in the storage device 19.

[0091] On the other hand, the learning unit 13d also inputs verification image data with pre-assigned goodness-of-fit information into the second inference model 111 and executes the goodness-of-fit determination of the verification image data using the second inference model 111. The learning unit 13d calculates a second matching rate, which is the matching rate between the goodness-of-fit information of the verification image data and the goodness-of-fit determination result by the second inference model 111. When a plurality of second inference models 111 are generated, the second matching rate may be calculated for all of the second inference models 111, or the second matching rate may be calculated for some of the second inference models 111.

[0092] The display control unit 13c generates a model selection user interface screen (display screen) 210 as shown in FIG. 9. The model selection user interface screen 210 is provided with a model name display area 211 for displaying the name of the inference model, an information display area 212 for displaying information on the inference model, and a matching rate display area 213 for displaying the matching rate. In the model name display area 211, a name for identifying the inference model generated by the learning unit 13d is displayed. In this example, the A model is displayed as the first inference model 110 in the model name display area 211, and the B model, C model, D model,... are displayed as the second inference model 111. The name of the inference model can be set arbitrarily, and it may be set by the user or automatically set according to a specific rule.

[0093] In the information display area 212, as an example of the information of the inference model displayed in the model name display area 211, information regarding the image data used for learning is displayed in association with the model name. For example, information such as when the acquired image was used for learning, whether a good product image, a defective product image, or all images were used for learning, etc. is displayed in the information display area 212.

[0094] In the matching rate display area 213, the first matching rate (the matching rate by the A model) calculated by the learning unit 13d and the second matching rate (the matching rates by the B model, C model, D model,...) calculated by the learning unit 13d are displayed in association with the model name. Thereby, the user can easily determine which model has high pass / fail judgment performance. Also, it is possible to display in an identifiable manner which model the inference model in operation is. FIG. 9 shows the case where the A model is in operation.

[0095] The user can select a desired inference model from among a plurality of inference models displayed on the model selection user interface screen 210. For example, when operating with the A model and wanting to switch to the C model with a higher matching rate, the user can operate the keyboard 51 or the mouse 52 to align the pointer or cursor with "C model" and perform a selection operation. When the processor 13a detects that the C model has been selected, it switches from the A model to the C model and performs the appearance inspection process of the work W. When switching the inference model, it is necessary to temporarily stop the appearance inspection process of the work W, but since that time is very short, the usability of the user is hardly deteriorated thereby.

[0096] After the user selects an inference model, the process may transition to a pass / fail judgment performance confirmation user interface screen 300 as shown in, for example, FIG. 10. That is, the processor 13a generates the pass / fail judgment performance confirmation user interface screen 300 and causes it to be displayed on the display device 4. The user interface screen 300 is provided with a non-defective product image display area 301. A plurality of non-defective product images can be displayed in the non-defective product image display area 301. Further, the user interface 300 is provided with a defective product image display area 302. The processor 13a causes an image having an area highly likely to be a defective part to be displayed in the defective product image display area 302, assuming it is a defective product image.

[0097] Furthermore, the user interface 300 is provided with a learning result display area 305. The cumulative histogram generated by the processor 13a is displayed in the learning result display area 305. That is, the processor 13a obtains the frequency of being determined as a non-defective product image and the frequency of being determined as a defective product image as the determination results. The cumulative histogram is generated based on the frequency of being determined as a non-defective product image and the frequency of being determined as a defective product image. The user can determine whether the area of the non-defective product image (the area marked as OK in the figure) and the area of the defective product image (the area marked as NG in the figure) are separated in the cumulative histogram. If the area of the non-defective product image and the area of the defective product image are not separated, it is considered that the pass / fail judgment performance of the inference model is insufficient. That is, the user can visually confirm the non-defective product image and the defective product image, and can also confirm the pass / fail judgment performance of the selected inference model.

[0098] (Followability Evaluation Function for Recent Images) When learning with the learning dataset shown in Fig. 4, it is pre - specified for each image constituting the learning dataset whether it is a non - defective image or a defective image. However, for the input images accumulated by repeating steps SA4 to SA6, since they are newly acquired images, whether they are non - defective images or defective images is not specified. Therefore, in order to evaluate how well the inference model follows the most recent input image, it may be better for the user to label the images acquired during operation by themselves. The evaluation of the follow - up performance of the inference model during operation can be performed by calculating the above - mentioned coincidence rate.

[0099] For example, when the user performs a predetermined operation while the model - selection user - interface screen 210 shown in Fig. 9 is displayed, the labeling user - interface screen 220 shown in Fig. 11 is displayed. The labeling user - interface screen 220 is a screen for the user to manually assign one of a label indicating that the input image is a non - defective image (non - defective label) and a label indicating that the input image is a defective image (defective label) while checking the input image.

[0100] The labeling user - interface screen 220 is provided with an image display area 221 and a label setting section 222. The image display area 221 is an area where the input images accumulated by repeating steps SA4 to SA6 are displayed. The plurality of input images may be displayed in a list format or one by one. When displayed in a list format, for example, the user can select one from among the plurality of input images, and the selected input image can be enlarged and displayed in the labeling user - interface screen 220 to facilitate image confirmation.

[0101] The label setting section 222 is provided for each input image displayed in the image display area 221. By operating the label setting section 222 by the user, one of the non - defective label and the defective label can be assigned to the corresponding input image. The input label information is stored in the storage device 19 in a state associated with the corresponding input image.

[0102] (Other functions) For example, when the trigger condition is satisfied multiple times in a short period, it is necessary to generate the second inference model 111 multiple times in a short period. However, since it takes time to generate the second inference model 111, it is conceivable that the next second inference model 111 cannot be generated immediately even if the next trigger condition is satisfied. In such a case, the control unit 2 can be provided with a function of detecting that the generation of the second inference model 111 has been completed. When the learning unit 13d determines that the generation of the second inference model 111 has been completed and the next trigger condition is satisfied, the learning unit 13d starts generating the next second inference model 111.

[0103] Also, although the user determines whether to adopt the generated second inference model 111 during operation, it is not limited to this, and the control unit 2 may make the determination. For example, when the second inference model 111 with the above-mentioned matching rate equal to or higher than a predetermined value is generated, or when the second inference model 111 with a higher matching rate than the currently operating inference model is generated, the control unit 2 is configured to automatically adopt the second inference model 111 and perform the operation. Thereby, the pass / fail determination performance can always be maintained at a high level. Also, when the second inference model 111 with the above-mentioned matching rate equal to or higher than a predetermined value is generated, a function of notifying the user of this may be added.

[0104] Also, there may be a case where the same workpiece W is subjected to appearance inspection on a plurality of lines. In such a case, the input images captured on each line may be used for learning the inference model after being collected.

[0105] (Specific method for learning the machine learning network) The learning method of the above-mentioned machine learning network is not particularly limited. For example, the following method can be used. That is, the machine learning network can be learned by minimizing the Loss function. Although there are various definitions of Loss, Mean Square Error (MSE) can be cited as an example.

[0106]

Number

[0107] Here, T is the target abnormality degree map, 0 is the output image (abnormality degree map), n is the number of pixels in the image T that are 0, and x, y are the pixel positions. Note that loss functions such as Binary Cross Entropy can also be used. The above is merely an example.

[0108] (Operation and Effect of Embodiment) As described above, by causing the machine learning network to learn an input image different from the image used during the learning of the first inference model, a second inference model having parameters different from those of the first inference model can be generated in the background of the pass / fail determination process by the inspection unit 13e. As a result, it is not necessary to stop the appearance inspection of the workpiece for a long time when generating the second inference model, and the usability for the user does not deteriorate.

[0109] Also, during the learning of the second inference model, by using, for example, an image captured at a time different from the acquisition time of the learning data of the first inference model as the input image, a second inference model with high pass / fail determination performance for the workpiece image when the surrounding environment of the workpiece has changed compared to the acquisition time of the learning data of the first inference model can be generated. As a result, an inference model with high pass / fail determination performance can be selected to continue the appearance inspection of the workpiece.

[0110] The above-described embodiment is merely an example in every respect and should not be construed in a limiting manner. Furthermore, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

Industrial Applicability

[0111] As described above, the present invention can be used when inspecting the appearance of a workpiece.

Explanation of Signs

[0112] 1 Appearance inspection device 4 Display device (display unit) 13a Processor 13b Memory 13c Display control unit 13d Learning unit 13e Inspection unit 13f Setting unit 19 Storage device (storage unit)

Claims

1. An appearance inspection device that inputs a work image obtained by photographing a work to be inspected into a machine learning network and determines whether the work is good or bad based on the input work image, a learning unit that inputs learning data into the machine learning network for learning and generates a first inference model for determining whether an input image is good or bad; an inspection unit that sequentially inputs input images into the first inference model generated by the learning unit and determines whether the input images are good or bad; a storage unit that stores the input images sequentially input into the first inference model generated by the learning unit and the determination results of whether each input image is good or bad; a display control unit that causes the display unit to display the determination result of whether the work is good or bad by the inspection unit, and is provided with: The learning unit executes, in the background of the pass / fail determination process by the inspection unit, a process of inputting a plurality of input images stored in the storage unit into a machine learning network for learning to generate a second inference model. The display control unit is an appearance inspection device that causes the display unit to display a display screen for displaying the pass / fail determination performance of the second inference model.

2. In the appearance inspection device according to claim 1, The display control unit is characterized in that it causes the display unit to display a display screen for comparing the pass / fail determination performance of the first inference model and the second inference model.

3. In the appearance inspection device according to claim 1 or 2, Further provided is a setting unit for setting conditions for selecting input images used for generating the second inference model, The learning unit extracts input images that satisfy the conditions set by the setting unit from among the plurality of input images stored in the storage unit, inputs the extracted input images into a machine learning network for learning, and generates the second inference model. The appearance inspection device is characterized by this.

4. In the appearance inspection device according to claim 3, The learning unit inputs input images that satisfy different conditions set by the setting unit into a machine learning network for learning, thereby generating a plurality of second inference models respectively corresponding to each condition. The display control unit causes the display unit to display a display screen for comparing the quality determination performance of the plurality of second inference models. An appearance inspection apparatus characterized by this.

5. In the appearance inspection apparatus according to claim 3 or 4, The setting unit can set a first condition regarding the imaging time when an input image used for generating the second inference model is imaged. The learning unit acquires the first condition set by the setting unit, extracts an input image that satisfies the acquired first condition from among the plurality of input images stored in the storage unit, and inputs the extracted input image into the machine learning network for learning to generate the second inference model. An appearance inspection apparatus.

6. In the appearance inspection apparatus according to claim 3 or 4, The setting unit can set a trigger condition for starting a learning process that is executed in the background for generating the second inference model. The learning unit starts learning of the second inference model according to the trigger condition set by the setting unit. An appearance inspection apparatus.

7. In the appearance inspection apparatus according to claim 6, The setting unit can set, as the trigger condition, a condition regarding the date and time for starting a learning process for generating the second inference model. When the time set by the setting unit arrives, the learning unit starts learning of the second inference model. An appearance inspection apparatus.

8. In the appearance inspection apparatus according to claim 6, The setting unit can set, as the trigger condition, a condition regarding a change in the feature amount of the input image or a statistical change in the quality determination result. The learning unit is an appearance inspection device that starts learning the second inference model when a change in the feature amount of the input image or a statistical change in the pass / fail determination result satisfies the conditions set by the setting unit.

9. In the appearance inspection device according to claim 5, As the first condition, the setting unit can set a predetermined period, The learning unit extracts input images captured within the predetermined period set as the first condition from among the plurality of input images stored in the storage unit, inputs the extracted input images into the machine learning network for learning, and generates the second inference model. An appearance inspection device.

10. In the appearance inspection device according to claim 5 or 9, The setting unit can set a second condition regarding the attributes of the input images used to generate the second inference model, The learning unit acquires the second condition set by the setting unit, extracts input images that satisfy both the acquired second condition and the first condition from among the plurality of input images stored in the storage unit, and inputs the extracted input images into the machine learning network for learning to generate the second inference model. An appearance inspection device.

11. In the appearance inspection device according to claim 10, As an attribute included in the second condition, the setting unit can set whether the input image used to generate the second inference model is a good product image corresponding to a good product, The learning unit extracts, from among the plurality of input images stored in the storage unit, images that satisfy the first condition and are good product images as input images, inputs the extracted input images into the machine learning network for learning, and generates the second inference model. An appearance inspection device.

12. In the appearance inspection device according to claim 10, The setting unit can set, as an attribute included in the second condition, whether the input image used to generate the second inference model is a defective product image corresponding to a defective product. The learning unit extracts, from among a plurality of input images stored in the storage unit, an image that satisfies the first condition and is a defective product image, and inputs the extracted input image into the machine learning network for learning to generate the second inference model. An appearance inspection apparatus.

13. In the appearance inspection apparatus according to claim 10, The learning unit extracts a defective product image having a feature amount of a defective product image included in a plurality of input images stored in the storage unit that is equal to or more than a predetermined value. An appearance inspection apparatus.

14. In the appearance inspection apparatus according to any one of claims 5, 9 to 13, The setting unit can set a third condition regarding the number of input images used to generate the second inference model. The learning unit acquires the third condition set by the setting unit, extracts, from among a plurality of input images stored in the storage unit, an input image that satisfies both the acquired third condition and the first condition, and inputs the extracted input image into the machine learning network for learning to generate the second inference model. An appearance inspection apparatus.

15. In the appearance inspection apparatus according to claim 14, The setting unit can set, as the third condition, the ratio of B to A when the number of a plurality of input images stored in the storage unit is A and the number of input images used to generate the second inference model is B. The learning unit extracts the number of input images corresponding to the ratio from among a plurality of input images stored in the storage unit, and inputs the extracted input image into the machine learning network for learning to generate the second inference model. An appearance inspection apparatus.

16. In the appearance inspection apparatus according to claims 1 to 15, The learning unit is an appearance inspection apparatus that also uses the learning data used when generating the first inference model when generating the second inference model.

17. In the appearance inspection apparatus according to claims 1 to 16, The learning unit inputs verification image data with pre-assigned pass / fail information into the first inference model and the second inference model respectively to perform pass / fail determination on the verification image data, and calculates a first matching rate that is the matching rate between the pass / fail information of the verification image data and the pass / fail determination result by the first inference model, and a second matching rate that is the matching rate between the pass / fail information of the verification image data and the pass / fail determination result by the second inference model. The display control unit provides a display area for the first matching rate and the second matching rate on a display screen for comparing the pass / fail determination performance of the first inference model and the second inference model, and causes the display screen to be displayed on the display unit. The appearance inspection apparatus.

18. An appearance inspection method for inputting a work image obtained by photographing a work to be inspected into a machine learning network and performing pass / fail determination on the work based on the input work image, A first learning step of inputting learning data into the machine learning network for learning and generating a first inference model for performing pass / fail determination on an input image; An inspection step of sequentially inputting an input image into the first inference model generated in the first learning step and performing pass / fail determination on the input image; A storage step of storing the input images sequentially input into the first inference model generated in the first learning step and the pass / fail determination results of each input image; A second learning step of inputting the plurality of input images stored in the storage step into a machine learning network for learning to generate a second inference model, which is executed in the background of the pass / fail determination process in the inspection step; And a display step of causing a display unit to display a display screen for displaying the pass / fail determination performance of the second inference model. The appearance inspection method.

Images