Setting device and setting method

Abstract

To obtain a machine learning network which reduces user's labor at learning time of a machine learning network, excludes dependency on individual skills, and has high detection performance.SOLUTION: A setting device comprises: a display control unit which displays an image to be learned by a machine learning network on a display unit; an input unit which receives defective information inputted by either formula of a first annotation formula which assigns a label indicating that it is a non-defective article image or a defective article image, and a second annotation formula which designates a defective part when the image displayed on the display unit is the defective article image, with respect to the image displayed on the display unit; and a processor which makes the machine learning network learn both of the non-defective article image or the defective article image to which the label was assigned by the first annotation formula, and the defective article image having the defective information inputted by the second annotation formula.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a setting device and setting method for a visual inspection device. [Background technology]

[0002] For example, Patent Document 1 discloses a processing device that uses computer-based machine learning to determine whether a workpiece is a good or defective product. The processing device in Patent Document 1 is configured to perform supervised machine learning on good product data to generate a good product learning model, and to perform supervised machine learning on defective product data to generate a defective product learning model, and then input data on the workpiece to be determined, and determine whether the workpiece is a good or defective product using the good product learning model and the defective product learning model; such a device is also called a workpiece appearance inspection device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-204321 Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, when performing supervised learning on defective product data, the user needs to clearly indicate which parts of the work are defective, which is called annotation. The judgment performance of the defective product learning model generated by defective product learning tends to improve the more detailed the annotation, so it is desirable to perform annotation in as much detail as possible.

[0005] However, defects are not necessarily simple shapes; they often have various shapes and are located in various positions on the workpiece. Accurately specifying the corresponding defect area on the defective product image is difficult. Inaccurate specification of the defect area results in a decline in the judgment performance of the defective product learning model. Furthermore, since annotation is performed by humans, there are cases where a user is unsure whether or not a region should be annotated as a defect on the defective product image, or where annotating a region larger than the corresponding defect area, or conversely, annotating a region smaller than the corresponding defect area. Such differences in user annotations can result in differences in the performance of the final defective product learning model. In other words, while detailed annotations are ideal, the more detailed the annotations, the more subjective the annotations become, which can result in a failure to obtain a defective product learning model with good judgment performance.

[0006] Furthermore, generating a defective product learning model requires, for example, tens or hundreds of images of defective products, and accurately annotating all of these images of defective products would impose a significant time burden on the user, which was also a problem.

[0007] The present disclosure has been made in consideration of these points, and its purpose is to reduce the user's effort when training a machine learning network and to enable a machine learning network with high detection performance to be obtained while eliminating personal dependence. [Means for solving the problem]

[0008] To achieve the above object, one aspect of the present disclosure can be based on a setting device that sets up an appearance inspection device that inputs workpiece images captured of a workpiece to be inspected into a machine learning network generated by learning good product images corresponding to good products and defective product images corresponding to defective products, and determines whether the workpiece images are good or bad. The setting device includes a display control unit that displays images to be learned by the machine learning network on a display unit, an input unit that accepts defect information input using either a first annotation method that assigns a label indicating that the image displayed on the display unit is a good product image or a defective product image, or a second annotation method that specifies the defect location when the image displayed on the display unit is a defective product image, and a processor. The processor can train the machine learning network on both the good product images or defective product images assigned the label using the first annotation method and the defective product images with defect information input using the second annotation method, thereby adjusting parameters of the machine learning network.

[0009] According to this configuration, the first annotation method allows images displayed on the display unit to be sorted into good images and bad images by specifying labels, making it possible to annotate bad images that should be determined to be bad overall. On the other hand, the second annotation method allows defective parts to be specified on the bad image, making it possible to annotate with more detailed specification of the defective parts than with the first annotation method. In other words, there are cases where defective images include images that can be determined to be bad simply by adding a label, and images that cannot be determined to be bad unless the defective parts are clearly specified. In the former case, annotation can be performed easily without any effort on the part of the user, while in the latter case, the user can annotate in detail.

[0010] In another aspect, the input unit is configured to be able to select, as the second annotation method, an area designation method in which the defective part is designated by surrounding the defective part on the defective product image, and a precision designation method in which the defective part is designated in a free shape by tracing the defective part on the defective product image.

[0011] According to this configuration, when the area designation method is selected, the defective part can be easily designated by generating a frame surrounding the defective part using, for example, a pointing device. Furthermore, when the defective part has a free shape, for example, by selecting the precise designation method, the free-form defective part can be precisely designated. In other words, since the user can switch between rough and precise designation of the defective part as needed, the machine learning network can be effectively trained using defective product images while reducing the user's effort.

[0012] The input unit according to another aspect can execute an automatic extraction method for automatically extracting a defective portion within a specified area by surrounding the defective portion in the defective product image and its periphery.

[0013] According to this configuration, the user only needs to roughly encircle the defective part and its surroundings, and the defective part is automatically extracted, thereby further reducing the user's effort.

[0014] In another aspect, the input unit can accept corrections to the shape of the area automatically extracted by the automatic extraction method after the automatic extraction method has been executed.Therefore, for example, if the automatically extracted area deviates from the shape of the defective area, the user can correct the shape of the automatically extracted area so that the shape of the automatically extracted area matches the shape of the defective area, allowing for more precise annotation.

[0015] In another aspect, a processor can input good or defective product images labeled using the first annotation method and defective product images with defect locations specified using the second annotation method into a single machine learning network and adjust the parameters of the machine learning network, thereby enabling a single machine learning network to be trained on multiple defective product images annotated using different methods.

[0016] According to another aspect, a processor can perform a verification process in which verification image data is input to the machine learning network generated by learning images of defective products, and the verification image data is judged to be good or bad. After the verification process, if the verification image data corresponds to a defective product but is not judged to be defective, the processor can perform an update process in which the machine learning network learns the verification image data, in which defective parts are specified using the second annotation method, and updates parameters of the machine learning network.

[0017] In other words, a machine learning network that does not judge verification image data corresponding to defective products as defective has room for further learning. By having such a machine learning network learn verification image data in which the defective parts are precisely specified using the second annotation method, the judgment performance of the machine learning network can be further improved.

[0018] In another aspect, the display control unit can display on the display unit a list of images that have been labeled as defective images using the first annotation method and that have not been determined to be defective. The input unit accepts a selection input of an image for which a defective portion is to be designated using the second annotation method from among the images displayed on the display unit, and accepts the designation of the defective portion using the second annotation method for the selected image.

[0019] In other words, the fact that there are images that were not judged as defective among the images labeled as defective means that there is room for further learning in the machine learning network. In such cases, it is conceivable to have the machine learning network learn from the images that were not judged as defective, but learning in this state will hardly improve the learning effect. In this configuration, images that were not judged as defective despite being labeled as defective can be presented to the user by being displayed on the display unit. From among these images, the user can select an image suitable for precisely specifying the defective part using the second annotation method, and train the machine learning network using the image with the precisely specified defective part, thereby improving the learning effect.

[0020] In another aspect, the processor inputs multiple different verification image data into a machine learning network and performs a pass / fail judgment on each of the verification image data, and the display control unit displays the judgment results obtained by the processor on the display unit and presents them to the user.

[0021] According to this configuration, the processor can obtain the judgment results by having the machine learning network execute a pass / fail judgment on multiple pieces of verification image data. The judgment results are displayed on the display unit, allowing the user to understand whether the images designated as pass / fail can be separated. If the judgment results indicating pass / fail images can be separated from the judgment results indicating fail images, the user can determine that the machine learning network has trained sufficiently. If the separation is not clear, the user can determine that the machine learning network has not trained sufficiently and further training is required.

[0022] In another aspect, a cumulative histogram is generated based on the frequency of images determined to be good quality and the frequency of images determined to be defective, and is displayed on the display unit.

[0023] With this configuration, the machine learning network can determine whether the verification image data is good or bad, and the user can be presented with a histogram based on the frequency of images determined to be good and the frequency of images determined to be bad. This allows the user to easily determine whether the images designated as good and bad have been separated.

[0024] The processor according to another aspect is configured to be able to execute processing for increasing the distance between the abnormality map of an image having non-defective product information and the abnormality map of an image having defective product information by a predetermined amount or more.

[0025] According to this configuration, the learning effect using images having non-defective product information and images having defective product information is further improved.

[0026] A processor according to another aspect is configured to be able to perform at least one of a process of reducing the distance between abnormality maps of images having good product information and a process of reducing the distance between abnormality maps of images having defective product information.

[0027] According to this configuration, the learning effect using a plurality of images having non-defective product information and the learning effect using a plurality of images having defective product information are further improved.

[0028] In another aspect, a processor can adjust the parameters of the machine learning network so that each pixel value in the anomaly map generated by inputting a good product image having good product information input using a first annotation method into the machine learning network becomes 0.

[0029] In another aspect, a processor can adjust the parameters of the machine learning network so that, in an abnormality map generated by inputting a defective product image having defective product information input using the second annotation method into the machine learning network, pixel values ​​of parts corresponding to areas other than the defective parts specified using the second annotation method become 0. [Effects of the Invention]

[0030] As explained above, defect information can be input using either the first annotation method, which assigns a label indicating whether the image is a good or defective product image, or the second annotation method, which specifies the defective parts of the defective product image, and the parameters of the machine learning network can be adjusted based on that defect information.This reduces the user's effort during learning and enables a machine learning network with high detection performance to be obtained while eliminating subjectivity. [Brief explanation of the drawings]

[0031] [Figure 1] FIG. 2 is a schematic diagram showing the configuration of a setting device of the visual inspection apparatus according to the embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the setting device. [Figure 3] 10 is a flowchart illustrating an example of an annotation procedure. [Figure 4] FIG. 4A shows a case where a label is assigned to a defective product image using the first annotation method, and FIG. 4B shows a case where a label is assigned to a non-defective product image using the first annotation method. [Figure 5] FIG. 10 is a diagram showing an example of specifying a defective part in a defective product image using the second annotation method. [Figure 6] FIG. 6A shows an example of surrounding a defective area with a circular frame passing through three points, FIG. 6B shows an example of surrounding a defective area with a circular frame generated by specifying a center point, FIG. 6C shows an example of surrounding a defective area with a polygon, and FIG. 6D shows an example of surrounding a defective area with a frame generated using a freehand tool. [Figure 7] FIG. 10 is a diagram illustrating details of the second annotation method. [Figure 8] FIG. 10 is a diagram illustrating a case where a defective portion according to one example is designated by a first annotation method and a second annotation method. [Figure 9] FIG. 10 is a diagram illustrating another example in which a defective location is designated by the first annotation method and the second annotation method. [Figure 10] 1 is a flowchart showing a first example of a parameter adjustment procedure for a machine learning network. [Figure 11] FIG. 10 is a diagram showing an example of a user interface screen displaying the results of a learning parameter analysis. [Figure 12] FIG. 10 is a diagram showing an example of a user interface screen displaying an image of a defective product that has not been determined to be defective. [Figure 13] FIG. 10 is a diagram showing an example of a user interface screen for executing the second annotation method. [Figure 14] FIG. 10 is a diagram illustrating an example of a user interface screen after a parameter is updated. [Figure 15] 10 is a flowchart showing a second example of a parameter adjustment procedure for a machine learning network. [Figure 16] FIG. 10 is a diagram showing an example of an output image output from a neural network for anomaly extraction. [Figure 17] FIG. 10 is an explanatory diagram of a learning method. [Figure 18] FIG. 10 is an explanatory diagram of a learning method. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0033] FIG. 1 is a schematic diagram showing the configuration of a setting device (hereinafter simply referred to as the setting device) 1 for a visual inspection device according to an embodiment of the present invention. The setting device 1 is a device that enables a user to perform various settings of the visual inspection device (details will be described later). The visual inspection device is a device that performs quality determination of workpiece images acquired by capturing images of workpieces to be inspected, such as various parts and products, and can be used in production sites such as factories. Specifically, a machine learning network is built inside the visual inspection device, and this machine learning network is generated by learning good-product images corresponding to good products and defective-product images corresponding to defective products. A workpiece image captured of the workpiece to be inspected is input to the generated machine learning network, and the machine learning network can determine the quality of the workpiece image.

[0034] The entire workpiece may be the object of inspection, or only a portion of the workpiece may be the object of inspection. Also, one workpiece may contain multiple inspection objects. Also, a workpiece image may contain multiple workpieces.

[0035] The setting device 1 comprises a control unit 2, which is the device main body, an imaging unit 3, a display device (display section) 4, and a personal computer 5. The personal computer 5 is not essential and can be omitted. The personal computer 5 can be used instead of the display device 4 to display various information and images, and the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0036] 1 illustrates a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5 as an example of the configuration of the setting device 1, but any two or more of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Furthermore, the control unit 2 can be divided into multiple units and some of them can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into multiple units and some of them can be incorporated into other units.

[0037] Furthermore, since the hardware of the setting device 1 and the hardware of the appearance inspection device have many parts in common, a program for realizing each function of the appearance inspection device may be incorporated into the setting device 1 to make it a setting device 1 with an appearance inspection function, or a program for realizing each function of the setting device 1 may be incorporated into the appearance inspection device to make it an appearance inspection device with a setting function described below. Furthermore, the setting device 1 and the appearance inspection device may be separate devices, in which case the setting device 1 and the appearance inspection device can be connected to each other so that they can communicate with each other and used.

[0038] (Configuration of imaging unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging section) 14 and an illumination module (illumination section) 15, and is a unit that acquires workpiece images. The camera module 14 includes an AF motor 141 that drives the imaging optical system, and an imaging board 142. The AF motor 141 is a part that automatically adjusts focus by driving the lens of the imaging optical system, and can perform focus adjustment using a conventionally well-known method such as contrast autofocus. The imaging board 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of the CMOS sensor 143, a light receiving element such as a CCD sensor can be used.

[0039] The lighting module 15 includes an LED (light emitting diode) 151 as a light emitting element that illuminates an imaging area including a workpiece, and an LED driver 152 that controls the LED 151. The timing, duration, and amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided separately from the imaging unit 3 as an external lighting unit.

[0040] (Configuration of display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel or an organic EL panel. A work image, a user interface image, etc. output from the control unit 2 are displayed on the display device 4. If the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used in place of the display device 4.

[0041] (operation equipment) Examples of operation devices for the user to operate the setting device 1 include, but are not limited to, the keyboard 51 and mouse 52 of the personal computer 5, and any device configured to be able to accept various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operation devices.

[0042] User operations on the keyboard 51 and mouse 52 can be detected by the control unit 2. The touch panel 41 is a conventionally known touch-type operation panel equipped with, for example, a pressure-sensitive sensor, and user touch operations can be detected by the control unit 2. The same applies when other pointing devices are used.

[0043] (Configuration of control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The main board 13 is provided with a display control unit 13a, an input unit 13b that receives image defect information, and a processor 13c. The display control unit 13a and the input unit 13b can be configured, for example, by an arithmetic processing device mounted on the main board 13. Furthermore, the display control unit 13a, the input unit 13b, and the processor 13c may be configured by a single arithmetic processing device, or the display control unit 13a, the input unit 13b, and the processor 13c may be configured by separate arithmetic processing devices.

[0044] The display control unit 13a, the input unit 13b, and the processor 13c control the operation of each connected board and module. For example, the processor 13c outputs an illumination control signal to the LED driver 152 of the illumination module 15 to control the turning on / off of the LED 151. The LED driver 152 switches the turning on / off of the LED 151 and adjusts the lighting time, and also adjusts the light intensity of the LED 151, in response to the illumination control signal from the processor 13c.

[0045] In addition, the processor 13c outputs an imaging control signal to the imaging board 142 of the camera module 14 to control the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13c and adjusts the exposure time to any desired time to capture the image. That is, the imaging unit 3 captures the image within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13c. If a workpiece is present within the field of view, the image of the workpiece is captured. However, if an object other than the workpiece is present within the field of view, the image of the workpiece can also be captured. For example, the setting device 1 can use the imaging unit 3 to capture images of good products corresponding to good products and images of defective products corresponding to defective products as images for learning the machine learning network. The learning images do not have to be images captured by the imaging unit 3, but may be images captured by another camera, etc.

[0046] On the other hand, when the appearance inspection device is in operation, the workpiece can be imaged by the imaging unit 3. The CMOS sensor 143 is configured to be able to output a live image, i.e., a currently captured image, at a short frame rate at any time.

[0047] When the CMOS sensor 143 has finished capturing an image, the image signal output from the imaging unit 3 is input to the processor 13c of the main board 13 for processing, and is also stored in the memory 13d of the main board 13. Specific details of the processing performed by the processor 13c of the main board 13 will be described later. The main board 13 may be provided with a processing device such as an FPGA or a DSP. The processor 13c may also be an integrated processor such as an FPGA or a DSP.

[0048] The connector board 16 is a part that receives power from an external source via a power connector (not shown) provided on the power interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and module, and specifically distributes power to the illumination module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 is equipped with an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14 to achieve autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 in response to an AF control signal from the processor 13c on the main board 13.

[0049] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, and communication between the main board 13 and an external control device (not shown). An example of the external control device is a programmable logic controller. The communication may be wired or wireless, and either form of communication can be realized by a conventionally known communication module.

[0050] The control unit 2 is provided with a storage device (storage unit) 19, which may be, for example, a solid state drive, a hard disk drive, or the like. The storage device 19 stores program files 80, setting files, and the like (software) that enable the hardware to execute the various controls and processes described below. The program files 80 and setting files can be stored in a storage medium 90, such as an optical disk, and the program files 80 and setting files stored in the storage medium 90 can be installed in the control unit 2. The program files 80 may be downloaded from an external server via a communication line. The storage device 19 can also store, for example, the image data and parameters for constructing a machine learning network for the visual inspection device.

[0051] (annotation) The setting device 1 adjusts the parameters of the machine learning network of the visual inspection device by inputting images of good products corresponding to good products and images of defective products corresponding to defective products into the machine learning network and causing the network to learn. For example, a method can be adopted in which initial values ​​of the parameters of the machine learning network are determined randomly, and the parameters are adjusted while feeding back errors in image recognition output from the machine learning network.

[0052] Prior to learning a machine learning task, the user must specify whether the image to be input to the machine learning network is a good or defective image, and if the image to be input to the machine learning network is a defective image, specify the location of the defect. This is annotation by the user, and in this embodiment, annotation can be performed using multiple methods, including a first annotation method and a second annotation method.

[0053] As shown in FIG. 3, when a user performs an operation to start annotation, the process proceeds to step SA1, where the user selects an image to be annotated. For example, multiple images including a workpiece W as shown in FIG. 4 are stored in the setting device 1, and the user selects a desired image from among them. Then, when the process proceeds to step SA2, the display control unit 13a generates an annotation execution user interface screen 200 and displays it on the display device 4, as shown in FIGS. 4A and 4B of FIG. 4. The annotation execution user interface screen 200 has an image display area 201 for displaying images to be learned by the machine learning network. FIG. 4A shows a case where a defective product image 202 is displayed in the image display area 201, and the black circle indicates the defective portion 202a. FIG. 4B shows a case where a non-defective product image 203 is displayed in the image display area 201. This is the display step. In other words, before the user performs annotation, the display control unit 13a displays on the display device 4 an image for the machine learning network to learn, allowing the user to perform annotation while looking at the image displayed on the display device 4.

[0054] Next, the process proceeds to step SA3, where the user performs annotation using the first annotation method. In the first annotation method, a label indicating whether the image displayed on the display device 4 is a good image or a defective image is assigned to the image. Specifically, in FIG. 4A of FIG. 4, a defective image 202 is displayed, and a defective label A1, such as "NG," is assigned to the defective image 202 as shown on the right side to indicate that it is a defective image. This defective label A1 is the defect information input using the first annotation method.

[0055] On the other hand, in FIG. 4B of FIG. 4, a non-defective image 203 is displayed, and therefore a non-defective label A2, such as "OK," is assigned to the image 203 as a label indicating that the image is a non-defective image, as shown on the right side. This non-defective label A2 is the non-defective information input by the first annotation method. Any method may be used for the user to assign labels. For example, a button labeled "NG" and a button labeled "OK" may be provided on the annotation execution user interface screen 200, and the desired labels A1 and A2 may be assigned by operating these buttons.

[0056] The method of labeling using the first annotation method may involve labeling images one by one, or, for example, storing multiple defective product images in a specific folder and collectively labeling the defective product images in that folder with a defective product label A1. Similarly, for non-defective product images, storing multiple non-defective product images in a specific folder and collectively labeling the non-defective product labels A2 to the non-defective product images in that folder may be similarly performed.

[0057] In the case of the non-defective product image 203, steps SA4 and SA5 in the flowchart shown in Fig. 3 are omitted and the process proceeds to step SA6. In step SA6, the input unit 13b accepts the non-defective product information input using the first annotation method. The input unit 13b can store the non-defective product image 203 displayed on the display device 4 and the non-defective product information (label A2) in the storage device 19 in a state in which they are associated with each other.

[0058] In the case of the defective product image 202, steps SA4 and SA5 in the flowchart shown in Fig. 3 may be omitted and the process may proceed to step SA6, or steps SA4 and SA5 may be executed and the process may proceed to step SA6. If steps SA4 and SA5 are omitted, in step SA6 the input unit 13b accepts the defect information input using the first annotation method. The input unit 13b can store the defective product image 202 displayed on the display device 4 in association with the defect information (defective product label A1) in the storage device 19.

[0059] Steps SA4 and SA5 are examples of a second annotation method for specifying defective locations when the image displayed on the display device 4 is a defective product image 202. Either step SA4 or step SA5 may be performed, or both may be performed. When proceeding from step SA3 to step SA4 or step SA5, the user may perform a predetermined operation, or the process may proceed automatically upon detecting that a defective label A2 has been assigned.

[0060] Step SA4 is an example of an area designation method in which the defective part 202a is designated by surrounding the defective part 202a in the defective product image 202 displayed on the display device 4, and step SA5 is an example of a precise designation method in which the defective part 202a is designated in a free shape by tracing the defective part 202a in the defective product image 202 displayed on the display device 4. The user can select either the area designation method or the precise designation method.

[0061] After step SA3, as shown in FIG. 5, the display control unit 13a causes the display device 4 to display an annotation execution user interface screen 200, and causes the image display area 201 to display a defective product image 202 with a defective product label A1 attached thereto. The annotation execution user interface screen 200 displays a designation method selection window 204. The designation method selection window 204 includes a first button 204a for selecting the area designation method, a second button 204b for selecting the precision designation method, and a third button 204c for selecting the eraser tool. When it detects that the first button 204a has been operated by the user, the process proceeds to step SA4 in the flowchart shown in FIG. 3. When it detects that the second button 204b has been operated by the user, the process proceeds to step SA5. Note that operating the third button 204c can cancel at least a portion of the designation operation performed using the area designation method or the precision designation method, or erase a portion of the designated area.

[0062] In step SA4, the user operates the pointer 205 (shown in FIG. 5) of the mouse 52 displayed on the display device 4 or the click button (not shown) of the mouse 52 to surround the defective portion 202a in the defective product image 202. In the example shown in FIG. 5, the user operates the pointer 205 and click button of the mouse 52 to generate a rectangular frame 206 large enough to surround the defective portion 202a. When generating the rectangular frame 206, for example, the pointer 205 is placed at a position corresponding to the upper left vertex and dragged diagonally downward and to the right, thereby generating a frame 206 of a desired size and shape at a desired position. The size and position of the frame 206 can be changed by operating the mouse 52.

[0063] 6A, three points 207 may be designated on the defective product image 202 with a pointer 205, a circular frame 208 passing through these three points 207 may be generated, and the defective portion 202a may be enclosed by this frame 208. Alternatively, as shown in FIG. 6B, a point 210 may be designated on the defective product image 202 with a pointer 205, a circular frame 211 may be generated with this point 210 as its center, and the defective portion 202a may be enclosed by this frame 211.

[0064] 6C, a plurality of points 212 may be designated on the defective product image 202 with a pointer 205, a frame 213 may be generated by connecting the points 212, and the defective portion 202a may be enclosed by this frame 213. Alternatively, as shown in FIG. 6D, a free-form frame 214 may be generated by moving the pointer 205 so as to enclose the defective portion 202a on the defective product image 202, and the defective portion 202a may be enclosed by this frame 214. In the example shown in FIG. 6D, a method is used in which an area is designated by free-drawing using a so-called freehand tool. The method of designation to be used may be determined depending on the shape and size of the defective portion 202a.

[0065] As described above, in step SA4, the position, shape, and size of the frames 206, 208, 211, 213, and 214 surrounding the defective portion 202a are acquired as defect information. For example, by acquiring the center coordinates and the lengths of the vertical and horizontal sides of the frame 206 shown in Fig. 5, the defective portion 202a in the defective product image 202 can be identified. Similarly, in the case shown in Fig. 6, by acquiring information on the frames 208, 211, 213, and 214, the defective portion 202a in the defective product image 202 can be identified. The information that identifies this defective portion 202a is the defect information.

[0066] When the user designates the defective portion 202a in step SA4, the process proceeds to step SA6. In step SA6, the input unit 13b accepts the defect information input in step SA4. The input unit 13b can store the defective product image 202 displayed on the display device 4 and the defect information input in step SA4 in the storage device 19 in a state in which they are associated with each other.

[0067] When the user operates the second button 204b in the designation method selection window 204 on the annotation execution user interface screen 200 shown in FIG. 5 to proceed to step SA5, a fill tool 220 is displayed on the annotation execution user interface screen 200, as shown in FIG. 7A of FIG. 7. The user can freely move the fill tool 220 by operating the mouse 52. The user can designate the defect area 202a in a free shape by moving the fill tool 220 so as to trace the defect area 202a in the defective product image 202. Tracing the defect area 202a with the fill tool 220 makes it less likely that areas other than the defect area 202a will be included, thereby enabling more precise annotation than the area designation method of step SA4. The position, shape, and size of the area filled with the fill tool 220 can be acquired to identify the defect area 202a in the defective product image 202. Information identifying this defect area 202a is defect information.

[0068] 3, when the user specifies the defective portion 202a using the fill tool 220, the process proceeds to step SA6. In step SA6, the input unit 13b accepts the defect information input in step SA5. The input unit 13b can store the defective product image 202 displayed on the display device 4 and the defect information input in step SA5 in the storage device 19 in a state in which they are associated with each other.

[0069] In step SA5, the defective portion 202a may be designated by a magnet tool 221 as shown in FIG. 7B in addition to the above-mentioned fill tool 220. The magnet tool 221 can be moved by operating the mouse 52, and when it is moved close to the defective portion 202a, it moves so as to be automatically attracted to the defective portion 202a. Since the defective portion 202a is contained within a frame 222 connecting multiple magnet tools 221, the defective portion 202a can be precisely designated while reducing the burden on the user. The position, shape, and size of the frame 222 are defect information.

[0070] Also, in step SA5, the defective portion 202a may be specified using a GrabCut tool 223 as shown in FIG. 7C in FIG. 7. When the GrabCut tool 223 is used to enclose the defective portion 202a and its surroundings, the area is specified, and an automatic extraction method is executed to automatically extract the defective portion 202a from within the specified area. In the automatic extraction method, only the defective portion 202a is specified, and the area around the defective portion 202a is not specified, so that the defective portion 202a is automatically precisely specified, as shown by the white circle on the right. This reduces the burden on the user.

[0071] However, with the GrabCut tool 223, precise designation of the defective area 202a may fail, resulting in the area surrounding the defective area 202a being included. An example of a case where precise designation has failed is shown in the bottom of FIG. 7C. In this example, the circular defective area 202a is mistakenly designated as being nearly rectangular. In such a case, after the automatic extraction method is executed, the user can finely designate the foreground and background by performing stroke correction or click correction, and the input unit 13b will accept the correction of the shape of the area automatically extracted by the GrabCut tool 223.

[0072] The defective portion 202a may also be specified by AI Assisted specification. In the case of AI Assisted specification, the outline of the defective portion 202a is roughly specified and extracted, and then the interior of the extracted portion is specified using a Fill tool or the like. This allows the defective portion 202a to be automatically extracted. After the defective portion 202a is automatically extracted, it is also possible to make fine corrections.

[0073] After step SA6, the process proceeds to step SA7, where it is determined whether the number of annotated images has reached the required number. If the determination is NO and the required number is not reached, the process proceeds to step SA1. On the other hand, if the determination is YES and the required number of annotations have been completed, the annotation ends. Steps SA1 to SA7 are input steps for accepting defect information input using either the first annotation method or the second annotation method.

[0074] (Example of annotation) FIG. 8A in FIG. 8 illustrates a defective product image 202 containing a single linear defect 202a. FIG. 8B illustrates a state in which a defective product label A1 has been assigned using the first annotation method in step SA3. FIG. 8C illustrates a case in which a rectangular frame 206 is generated using the area designation method in step SA4, and the defect 202a is designated by enclosing the defect 202a with the generated frame 206. FIG. 8D illustrates a case in which a frame 213 connecting multiple points is generated using the area designation method in step SA4, and the defect 202a is designated by enclosing the defect 202a with the generated frame 213. FIG. 8E illustrates a case in which a frame 230 surrounding the defect 202a is generated using the freehand tool using the precise designation method in step SA5, and the defect 202a is designated using this frame 230. After the first annotation method, the user may perform annotation using the area designation method or the precise designation method. After annotating using the area designation method, the user may cancel the annotation using the area designation method and then perform annotation using the precise designation method.

[0075] FIG. 9A in FIG. 9 illustrates a defective product image 202 containing two intersecting linear defect portions 202a. FIG. 9B illustrates a state in which a defective product label A1 is assigned using the first annotation method in step SA3. FIG. 9C illustrates a case in which a rectangular frame 206 is generated using the area designation method in step SA4, and the defect portion 202a is designated by enclosing the defect portion 202a with the generated frame 206. FIG. 9D illustrates a case in which a frame 213 connecting multiple points is generated using the area designation method in step SA4, and the defect portion 202a is designated by enclosing the defect portion 202a with the generated frame 213. FIG. 9E illustrates a case in which a frame 230 surrounding the defect portion 202a is generated using the freehand tool using the precise designation method in step SA5, and the defect portion 202a is designated using this frame 230. In the examples shown in FIGS. 9D and 9E, it is possible to specify the defective portion 202a more precisely than in the example shown in FIG. 9C.

[0076] FIG. 9F shows a case where two rectangular frames 206 are generated using the area designation method of step SA4, and the defective portion 202a is designated using the two frames 206. FIG. 9G shows a case where a rectangular frame 206 and a frame 213 connecting multiple points are generated using the area designation method of step SA4, and the defective portion 202a is designated using the frames 206 and 213. In this way, the defective portion 202a may be designated by combining multiple types of frames 206 and 213.

[0077] (First example of parameter tuning procedure for machine learning networks) The processor 13c causes the machine learning network to learn both the defective product image 202 having the defect information input by the first annotation method and the defective product image 202 having the defect information input by the second annotation method. That is, the processor 13c is configured to input the defective product image 202 to which the defective product label A1 indicating that the defective product image 202 has been assigned by the first annotation method and the defective product image 202 to which the defect portion 202a has been designated by the second annotation method into the single machine learning network, and adjust the parameters of the machine learning network based on the defect information of each defective product image 202.

[0078] A first example of the parameter adjustment procedure for the machine learning network will now be described in detail with reference to the flowchart shown in Fig. 10. The parameter adjustment for the machine learning network may be performed by the user, by the manufacturer of the setting device 1, or on the cloud.

[0079] In step SB1 after the start, annotation is performed using the first annotation method. In step SB2, processor 13c causes the machine learning network to learn about good product images 203 to which a good product label A2 has been assigned using the first annotation method and defective product images 202 to which a defective product label A1 has been assigned using the first annotation method. That is, processor 13c adjusts the parameters of the machine learning network based on the good product information of each good product image 203 and the defect information of each defective product image 202. The number of good product images 203 and defective product images 202 to be learned by the machine learning network in step SB2 can be set to any number. Step SB2 is an adjustment step in which the parameters of the machine learning network are adjusted based on the defect information of the defective product images.

[0080] The process then proceeds to step SB3, where the processor 13c executes a verification process. Specifically, the processor 13c inputs multiple different verification image data to a machine learning network generated by learning a defective product image 202 to which a defective product label A1 indicating that the image is a defective product using the first annotation method, and causes the machine learning network to determine whether the verification image data is good or bad. In step SB3, the processor 13c generates a user interface screen 300 as shown in FIG. 11, and the display control unit 13a displays it on the display device 4. Step SB3 is a verification step in which the verification image data is input to the machine learning network after the parameters have been adjusted, and the machine learning network determines whether the verification image data is good or bad.

[0081] The user interface screen 300 has a verification image data display area 301 that displays the verification image data. The verification image data display area 301 displays the verification image data that the processor 13c input to the machine learning network in a list format. The verification image data marked with "OK" is a non-defective image, and the verification image data marked with "NG" is a defective image.

[0082] The user interface screen 300 is provided with an enlarged image display area 302. Of the verification image data displayed in the verification image data display area 301, the verification image data selected by the user is enlarged and displayed in the enlarged image display area 302.

[0083] The user interface screen 300 is provided with a learning parameter analysis area 303. That is, the processor 13c acquires multiple judgment results by having the machine learning network execute pass / fail judgments on multiple pieces of verification image data. The display control unit 13a displays the multiple judgment results acquired by the processor 13c on the display device 4 in the learning parameter analysis area 303, thereby allowing the user to grasp the level of inference performance of the machine learning network.

[0084] In this example, a case where a cumulative histogram is displayed as a display format of the judgment result will be described. In this case, for example, the processor 13c acquires, as the judgment result, the frequency of images judged to be good quality and the frequency of images judged to be defective quality. The processor 13c generates a cumulative histogram based on the frequency of images judged to be good quality and the frequency of images judged to be defective quality. The display control unit 13a causes the display device 4 to display the cumulative histogram generated by the processor 13c. The learning parameter analysis area 303 shown in FIG. 11 displays a frequency distribution of good quality images and a frequency distribution of defective quality images in a graph format. In the graph shown in FIG. 11, the region of good quality images (regions marked "OK" in the figure) and the region of defective quality images (regions marked "NG" in the figure) cannot be separated. In such a case, it is considered that the inference performance of the machine learning network is insufficient, and therefore, in step SB4, it is determined that the desired result has not been obtained by the parameter adjustment in step SB2. On the other hand, if it is determined in step SB4 that a desired result has been obtained, the process proceeds to step SB5. The "case where it is determined that a desired result has been obtained" will be described later. The "operation" in step SB5 means making the visual inspection apparatus operable, and the apparatus may start operation immediately or may be in a standby state.

[0085] In step SB6, which is reached when step SB4 is determined to be NO, the user is prompted to perform second annotation. That is, the user interface screen 300 shown in FIG. 11 is provided with a message display area 304. In this example, the regions of good product images and the regions of defective product images cannot be separated, and the inference performance of the machine learning network is considered to be insufficient. Therefore, in the message display area 304, the processor 13c generates a method for improving the inference performance of the machine learning network, and the display control unit 13a causes the display device 4 to display the method. Examples of methods for improving the inference performance of the machine learning network include "reviewing the feature size," "considering image variations," and "refining the annotation."

[0086] As described above, the second annotation method allows for more detailed annotation than the first annotation method. Therefore, displaying the message "Refine annotation" in the message display area 304 is intended to prompt the user to use the second annotation method or to inform the user that the second annotation method is an option for improving the inference performance of the machine learning network.

[0087] The user interface screen 300 shown in FIG. 11 includes an image count display area 305. The image count display area 305 displays a table with a "Label Designation" column. The "Label Designation" column displays the number of images that were detected as defective and the number of images that were not detected as defective among the defective product images 202 that were assigned the defective product label A1 using the first annotation method. In this example, the number of images that were detected as defective is "80," and the number of images that were not detected as defective is "6." In other words, if the pass / fail determination result in the verification process shows that six images were not determined as defective despite being verification image data corresponding to defective products, it can be determined that the inference performance of the machine learning network is insufficient. Note that if the number of images that were not detected as defective is "0," it can be determined that the inference performance of the machine learning network is sufficient, and operation can be started with that machine learning network.

[0088] The user can use the mouse 52 to click on the number "6" in the table displayed in the image number display area 305, which indicates the number of images that were not detected as defective product images. When the number "6" is clicked with the mouse 52, the display control unit 13a causes the display device 4 to display six images that were not determined to be defective despite being verification image data corresponding to defective products, as shown in FIG. 12. Specifically, the six images are displayed in a list in the verification image data display area 301 of the user interface screen 300. If there are a large number of images, the user can view all of the images by scrolling up and down and left and right.

[0089] When the user wants to perform the second annotation, he or she selects a desired image from the six images displayed in the verification image data display area 301 of the user interface screen 300 shown in Fig. 12. This selection operation can be performed using an operating device such as the mouse 52. That is, the input unit 13b accepts a selection input of an image for which a defective portion is to be specified using the second annotation method from among the images displayed in a list on the display device 4.

[0090] When the input unit 13b accepts an image selection input, as shown in FIG. 13, the display control unit 13a displays the selected image in the enlarged image display area 302 of the user interface screen 300 and also displays the above-mentioned designation method selection window 204. The user uses the tools in the designation method selection window 204 to perform annotation using the area designation method or the precise designation method. In this way, the input unit 13b accepts designation of defect areas for the selected image using the second annotation method, and the accepted result is stored in the storage device 19 together with the image. After designation for one image using the second annotation method is completed, another image may be selected and defect areas may be designated for that other image using the second annotation method. The above is the processing of step SB6 in the flowchart shown in FIG. 10.

[0091] After step SB6, the process proceeds to step SB7. In step SB7, an update process is performed in which the machine learning network learns the verification image data in which defect locations have been specified using the second annotation method in step SB6, thereby updating the parameters of the machine learning network. To perform the learning, the user simply operates the execute button 306 on the user interface screen 300. That is, the parameters of the machine learning network are changed by having the machine learning network learn images in which defect locations have been specified using the second annotation method, which is more detailed than the first annotation method. This improves the inference performance of the machine learning network.

[0092] Whether the inference performance of the machine learning network is sufficient, that is, whether it is determined that the desired result has been obtained, can be confirmed by looking at the graph displayed in the learning parameter analysis area 303 of the user interface screen 300 shown in Fig. 14. In the graph shown in Fig. 14, the areas of good product images and the areas of defective product images are completely separated, indicating that the machine learning network has the inference performance to clearly distinguish between good product images and defective product images.

[0093] As shown in FIG. 14 , the table displayed in the image count display area 305 includes a “Square Area Designation” column and a “Detailed Designation” column. The “Square Area Designation” column shows the number of images detected as defective after annotation using the area designation method and the number of images not detected as defective. Since the number of images detected as defective is “87,” it can be seen that all images that should have been detected as defective were actually detected as defective. The “Detailed Designation” column shows the number of images detected as defective after annotation using the precise designation method and the number of images not detected as defective. For example, if the number of images not detected as defective is not “0” in the “Square Area Designation” column, the user can simply perform annotation using the precise designation method. In the “Detailed Designation” column, the number of images not detected as defective can be set to “0.”

[0094] The above is a description of the setting method using the setting device 1. If sufficient inference performance can be obtained using only the first annotation method, operation can be started without executing the second annotation method, thereby preventing unnecessary burden on the user. Furthermore, if sufficient inference performance cannot be obtained using only the first annotation method, the inference performance can be confirmed by performing annotation using the area designation method, which is a relatively less burdensome option among the second annotation methods. As a result, if sufficient inference performance is not obtained, annotation using the precise designation method can be performed only on necessary images, thereby reducing the burden on the user.

[0095] (Second example of parameter tuning procedure for machine learning networks) A second example of the parameter adjustment procedure for the machine learning network will be specifically described with reference to the flowchart shown in FIG. 15. Step SC1 after the start is the same as step SB1 in the flowchart shown in FIG. 10. Then, in step SC2, annotation is performed using the second annotation method. In step SC3, the processor 13c causes the machine learning network to learn from a non-defective product image 203 assigned a non-defective product label A2 using the first annotation method, a defective product image 202 assigned a defective product label A1 using the first annotation method, and an image in which a defect location is specified using the second annotation method. That is, the processor 13c adjusts the parameters of the machine learning network based on the non-defective product information of each non-defective product image 203 and the defect information of each defective product image 202.

[0096] Then, the process proceeds to step SC4, where the processor 13c executes a verification process. Specifically, the processor 13c inputs multiple different verification image data to the machine learning network generated in step SC3 and causes the machine learning network to determine whether the verification image data is good or bad. Steps SC5 and SC6 are the same as steps SB4 and SB5 in the flowchart shown in FIG. 10.

[0097] In step SC7, which is reached when step SC5 is judged as NO, the user is prompted to perform re-annotation. For example, if only the area designation method has been performed on a certain image as the second annotation method, the user is prompted to perform annotation on that image using the precise designation method. Alternatively, the user may be prompted to perform more precise annotation using the precise designation method.

[0098] After step SC7, the process proceeds to step SC8. In step SC8, an update process is performed in which the machine learning network learns the verification image data re-annotated in step SC7, and the parameters of the machine learning network are updated. The machine learning network learns the images in which defective areas have been specified by the re-annotation, and the parameters of the machine learning network are changed. This improves the inference performance of the machine learning network.

[0099] (More details on neural networks for anomaly detection) The above-mentioned machine learning network is composed of a neural network for anomaly extraction, such as the conceptual diagram shown in Figure 16. The neural network for anomaly extraction outputs the degree of anomaly as a heat map (anomaly map) that indicates whether each pixel constituting the input image is abnormal. This heat map becomes the output image. Therefore, when a good product image is used as the input image, the output image will be an anomaly map made up of pixels all with the same pixel value, whereas when a defective product image is used as the input image, the output image will be an anomaly map in which only the pixels in the defective parts (anomaly parts) have pixel values ​​that differ from the other pixels.

[0100] In this embodiment, a learning method is employed that can extract only defective portions even when precise designation using the second annotation method is not required, and even when multiple annotation methods, such as the first and second annotation methods, are mixed. That is, as shown in FIG. 17, the processor 13c inputs at least three mutually different images, including both non-defective product images having non-defective product information input using the first annotation method, and defective product images having non-defective product information, into a machine learning network to generate an anomaly map. In the example shown in FIG. 17, a first input image, a second input image, and a third input image are input as a group into the same machine learning network. The first input image and the second input image are non-defective product images, and the third input image is a defective product image. Labels A1 and A2 are assigned to the first input image, the second input image, and the third input image using the first annotation method.

[0101] The processor 13c inputs a good-quality product image having good-quality product information input using the first annotation method into the machine learning network and adjusts the parameters of the machine learning network so that each pixel value in the anomaly degree map generated becomes 0. In this example, since the first input image and the second input image are good-quality product images, a restriction is imposed so that each pixel value in the first output image and the second output image becomes 0.

[0102] The processor 13c is also configured to be capable of performing at least one of a process of reducing the distance between the abnormality maps of images having non-defective product information and a process of reducing the distance between the abnormality maps of images having defective product information. In this example, since the first input image and the second input image are non-defective product images, a process of reducing the distance between the abnormality maps of the first output image and the second output image is performed. Note that if the first input image and the second input image are defective product images, a process of reducing the distance between the abnormality maps of the first output image and the second output image is performed.

[0103] The processor 13c is also configured to execute a process for increasing the distance between the abnormality map of the image having non-defective product information and the abnormality map of the image having defective product information by a predetermined distance or more. In this example, since the second input image is a non-defective product image and the third input image is a defective product image, the processor 13c executes a process for increasing the distance between the abnormality maps of the second output image and the third output image by a predetermined distance or more.

[0104] Specifically, the following calculation formulas and concepts (for example, the following calculation methods 1 and 2) can be applied. First, the loss function is defined. The average pixel value of the corresponding anomaly map for each input image can be calculated using the following formula.

[0105]

number

[0106] Here, n is the number of pixels in the abnormality map H, and x and y are pixel coordinate values ​​in the abnormality map H.

[0107] The average pixel value of the anomaly maps is used, and the distance between the anomaly maps is defined as the absolute value of the difference between the average pixel values.

[0108]

number

[0109] Calculation method 1 (directly limiting the distance between anomaly maps)

number

[0110] α is a hyperparameter that represents the distance margin, and λ is a hyperparameter that represents the strength of the loss restriction on the part of the anomaly map that corresponds to a non-defective product.

[0111] Calculation method 2 (method of indirectly limiting the distance between anomaly maps)

number

[0112] The distance between anomaly maps can also be defined as the Euclidean distance between the vectors by converting the anomaly maps into one-dimensional vectors. Learning can also be performed by minimizing the loss function.

[0113] The above describes the case where images labeled using the first annotation method are learned, but images for which defect locations are specified using the second annotation method can also be learned. That is, processor 13c adjusts the parameters of the machine learning network so that, in an anomaly map generated by inputting a defective product image having defect information input using the second annotation method into the machine learning network, pixel values ​​of parts other than those corresponding to defect locations specified using the second annotation method become 0.

[0114] A specific description will be given based on Fig. 18. Fig. 18 shows a case where a defect portion is designated in the third input image by the second annotation method. In this example, a frame 206 surrounding the defect portion is generated by the area designation method, but the defect portion may also be designated by the precise designation method. As shown as a mask image below the third input image, the pixel values ​​of the area corresponding to the outside of the frame 206 are set to 0. In the third output image, the pixel values ​​of the portion other than the area designated as the defect portion are set to 0.

[0115] The definition of the Loss function is as described above, but for the third input image, the average pixel values ​​inside and outside the annotation region are calculated.

[0116]

number

[0117] Furthermore, the average pixel value of the anomaly map is used, and the distance between the maps is defined as the absolute value of the difference between the average pixel values.

[0118]

number

[0119] Calculation method 1 (directly limiting the distance between anomaly maps)

number

[0120] Calculation method 2 (method of indirectly limiting the distance between anomaly maps)

number

[0121] (Effects of the embodiment) As described above, according to this embodiment, the first annotation method allows images displayed on the display device 4 to be classified into good and bad images by specifying labels, making it possible to annotate defective images that should be determined to be generally defective. On the other hand, the second annotation method allows the position and shape of defective areas on defective images to be specifically specified, making it possible to specify defective areas more clearly than with the first annotation method. This allows defective images that should be determined to be generally defective to be annotated without much effort on the part of the user, while allowing the user to annotate in detail defective images where the defective areas should be clearly specified. This reduces the user's effort when training a machine learning network, eliminates personal influences, and enables a machine learning network with high detection performance.

[0122] The above-described embodiments are merely examples in all respects and should not be construed as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention. [Industrial Applicability]

[0123] As described above, the present invention can be used, for example, when setting up an appearance inspection device that inspects the appearance of a workpiece. [Explanation of symbols]

[0124] 1 Setting device 13a Display control unit 13b Input section 13c processor

Claims

1. A setting device for setting an appearance inspection device that inputs a workpiece image obtained by capturing an inspection target workpiece into a machine learning network generated by learning images of good products corresponding to good products and images of defective products corresponding to defective products, and determines whether the workpiece image is good or bad, a display control unit that displays an image to be learned by the machine learning network on a display unit; an input unit that receives defect information input by either a first annotation method that assigns a label indicating whether the image displayed on the display unit is a non-defective image or a defective image, or a second annotation method that specifies a defective location when the image displayed on the display unit is a defective image; a processor capable of executing both a first parameter adjustment process of training the machine learning network on defective product images labeled using the first annotation method and adjusting parameters of the machine learning network, and a second parameter adjustment process of training the machine learning network on defective product images having defect information input using the second annotation method and adjusting parameters of the machine learning network; The processor: Operation of the machine learning network after the first parameter adjustment process has been performed and after the second parameter adjustment process has not been performed; and operating the machine learning network after the second parameter adjustment process.

2. 2. The setting device according to claim 1, The input unit is a setting device for an appearance inspection device that is configured to be able to select, as the second annotation method, an area designation method in which a defective part is designated by surrounding the defective part on the defective product image, and a precision designation method in which the defective part is designated in a free shape by tracing the defective part on the defective product image.

3. 2. The setting device according to claim 1, The input unit is configured to be able to execute, as the second annotation method, an automatic extraction method that automatically extracts the defective part within a specified area by surrounding the defective part in the defective product image and the surrounding area of ​​the defective part.

4. 4. The setting device according to claim 3, The input unit receives, as the second annotation method, a modification of the shape of the area automatically extracted by the automatic extraction method after the automatic extraction method is executed.

5. 5. The setting device according to claim 1, The processor inputs a defective product image labeled using the first annotation method and a defective product image in which defect locations are specified using the second annotation method into a single machine learning network, and adjusts parameters of the machine learning network.

6. 6. The setting device according to claim 1, The processor: a verification process in which verification image data is input to the machine learning network generated by learning defective product images that have been assigned labels indicating that they are defective product images using the first annotation method, and the verification image data is judged to be good or bad; A setting device for an appearance inspection device that is configured to be able to perform an update process in which, if the result of the pass / fail judgment in the verification process is that the verification image data corresponds to a defective product but is not judged to be defective, the machine learning network learns the verification image data in which the defective part is specified by the second annotation method and updates the parameters of the machine learning network.

7. 7. The setting device according to claim 1, the display control unit causes the display unit to display a list of images that have not been determined to be defective among the images labeled as defective images using the first annotation method; The input unit accepts selection input of an image from the list of images displayed on the display unit for specifying defective areas using the second annotation method, and accepts specification of defective areas using the second annotation method for the selected image.

8. 7. The setting device according to claim 1, The processor causes the machine learning network generated by learning defective product images that have been labeled as defective product images using the first annotation method to: A plurality of different verification image data are input, and a quality determination is performed on each of the verification image data to obtain a determination result; The display control unit is a setting device for an appearance inspection device that causes the display unit to display the judgment result acquired by the processor.

9. 9. The setting device according to claim 8, the processor acquires, as the determination result, a frequency of images determined to be good quality images and a frequency of images determined to be defective quality images, and generates a cumulative histogram based on the frequency of images determined to be good quality images and the frequency of images determined to be defective quality images; The display control unit displays the cumulative histogram generated by the processor on the display unit.

10. 10. The setting device according to claim 1, The processor: inputting at least three mutually different images, including both good product images having good product information input using the first annotation method and defective product images having defective product information, into the machine learning network to generate an anomaly map; A setting device for an appearance inspection device configured to be able to execute a process for increasing the distance between an abnormality map of an image having non-defective product information and an abnormality map of an image having defective product information by a predetermined distance or more.

11. 11. The setting device according to claim 10, The processor is configured to perform at least one of a process of reducing the distance between anomaly maps of images having non-defective product information and a process of reducing the distance between anomaly maps of images having defective product information.

12. 12. The setting device according to claim 10, The processor inputs a good-quality product image having good-quality product information input using the first annotation method into the machine learning network, and adjusts the parameters of the machine learning network so that each pixel value in the generated anomaly map becomes 0.

13. 13. The setting device according to claim 12, The processor adjusts the parameters of the machine learning network so that pixel values ​​of parts corresponding to parts other than the defective parts specified by the second annotation method become 0 in an abnormality map generated by inputting a defective product image having defective product information input using the second annotation method into the machine learning network.

14. A method for setting an appearance inspection device in which a workpiece image obtained by capturing an inspection target workpiece is input to a machine learning network generated by learning images of good products corresponding to good products and images of defective products corresponding to defective products, and the method determines whether the workpiece image is good or bad, a display step of displaying an image to be learned by the machine learning network on a display unit; an input step of receiving defect information using a first annotation method that assigns a label indicating whether the image displayed on the display unit in the display step is a non-defective image or a defective image; an input step of accepting defect information inputted for the image by a second annotation method that specifies a defective part when the image displayed on the display unit in the display step is a defective product image; a first adjustment step of training the machine learning network on defective product images labeled using the first annotation method and adjusting parameters of the machine learning network; a second adjustment step of training the machine learning network on defective product images having defect information input using the second annotation method, and adjusting parameters of the machine learning network; A method for configuring an appearance inspection device, comprising a step of accepting a selection between operation of the machine learning network that has been adjusted in the first adjustment step and not adjusted in the second adjustment step, and operation of the machine learning network that has been adjusted in the second adjustment step.

15. The setting method according to claim 14, a verification step of having the machine learning network learn defective product images that have been labeled as defective product images using the first annotation method, and then inputting verification image data into the machine learning network to determine whether the verification image data is good or bad; A method for setting an appearance inspection device, further comprising: if the result of the pass / fail judgment in the verification step is that the verification image data corresponds to a defective product but is not judged to be defective, an update step is performed in which the defective parts are designated using the second annotation method, and then the verification image data in which the defective parts are designated using the second annotation method is trained by the machine learning network to update the parameters of the machine learning network.

Images