Image inspection device and method for setting an image inspection device

Abstract

The present invention relates to an image inspection device and a setting method for an image inspection device. In order to enable easy additional learning of non-defective product images or defective product images in both the setting mode and the operation mode. A designation related to whether to additionally learn the inspection object image displayed on the operation mode screen as a non-defective product image or a defective product image is received. The discriminator generation unit additionally learns the inspection object image designated as the additional learning object as a non-defective product image or a defective product image, and updates the discriminator.

Description

Technical Field

[0001] The present invention relates to an image inspection apparatus and a method for setting the image inspection apparatus, which perform pass / fail determination of an inspection object based on an image obtained by photographing the inspection object. Background Art

[0002] As disclosed, for example, in Japanese Unexamined Patent Application Publication No. 2013-120550, in general image inspection using an image obtained by photographing an inspection object, a main image is first registered, and then various image inspection tools are set on the registered main image. Specifically, a user first makes various imaging settings for obtaining the main image, and uses the set imaging settings to obtain the main image. Next, the user selects an image inspection tool, sets a window for defining a range to which the selected image inspection tool is to be applied on the obtained main image, and makes individual settings for the image inspection tool. After that, the user photographs the inspection object, performs image inspection, and determines whether a desired result of pass / fail determination has been obtained. When the desired result is obtained, the setting is completed. However, when the desired result is not obtained, the user adjusts the imaging settings for obtaining the main image as needed. Since the main image changes when the imaging settings are adjusted, the user adjusts the individual settings of the image inspection tool as needed. The general inspection process described above is a rule-based image inspection in which the user sets characteristic values included in the image on which the inspection is based. Since the image inspection apparatus makes pass / fail determination according to the user's settings in rule-based image inspection, the user can control the criterion for pass / fail determination.

[0003] In the above-described rule-based image inspection, pass / fail determination of the inspection object is performed based on various characteristic values (such as color, edge, and position, etc.) of the inspection object in the image. In an image in which characteristic values such as color and edge are clear, pass / fail determination is easy. However, in an inspection object having a color difference, or an inspection object such as a metal component that is likely to change depending on the surrounding environment, the characteristic values are likely to change depending on imaging conditions, etc., and even when pass / fail determination by the human eye is easy, determination by the image inspection apparatus may be difficult, and the determination result may become unstable.

[0004] As an inspection processing technique capable of supporting such difficult inspections, there is known a technique in which a known learning unit such as a neural network learns the characteristics of a non-defective product image obtained by photographing a non-defective product and a defective product image obtained by photographing a defective product, generates a discriminator, and causes the discriminator to determine whether an image of a newly input inspection object during operation is a non-defective product or a defective product (for example, see Japanese Unexamined Patent Application Publication No. 2018-5640).

[0005] Since the discrimination performance of the discriminator can be improved by inputting non-defective product images and defective product images with various patterns, the requirements for discrimination performance are usually met by inputting many non-defective product images and defective product images when setting the mode.

[0006] However, this method makes the setting mode redundant, thereby increasing the time required to switch from the setting mode to the operating mode for actual inspection.

[0007] In addition, since the judgment performance of the discriminator cannot be confirmed until a new image captured in the operating mode is input into the discriminator, the judgment performance of the discriminator cannot be confirmed until the mode is switched from the setting mode to the operating mode. As a result of the confirmation, if the judgment performance of the discriminator does not meet the requirements, it is necessary to switch from the operating mode to the setting mode and perform additional learning by inputting non-defective product images and defective product images. Therefore, even after switching from the setting mode to the operating mode after the setting mode is completed, it is necessary to switch from the operating mode to the setting mode for additional learning, and then it is necessary to switch from the setting mode to the operating mode, so the operation is complicated. Summary of the Invention

[0008] The present invention solves the above problems, and its purpose is to enable the user to easily perform additional learning of non-defective product images or defective product images in both the setting mode and the operating mode.

[0009] To achieve the above object, according to a first aspect, there is provided an image inspection device for performing pass / fail judgment on an inspection object based on an image of the inspection object captured by an imaging unit. The image inspection device includes: a mode switching unit for switching the mode of the image inspection device between a setting mode for setting the image inspection device and an operation mode for performing inspection; an imaging setting unit for receiving setting of imaging conditions of the imaging unit in the setting mode; a learning image registration unit for registering a non-defective product image with the attributes of a non-defective product given by a user and a defective product image with the attributes of a defective product given by the user in the setting mode; a discriminator generation unit for learning the non-defective product image and the defective product image registered by the learning image registration unit in the setting mode and generating a discriminator for distinguishing the non-defective product image from the defective product image; a pass / fail judgment unit for inputting a newly captured image of the inspection object by the imaging unit to the discriminator and performing pass / fail judgment in the operation mode; a display unit for displaying the inspection object image together with the result of the pass / fail judgment performed by the pass / fail judgment unit on an operation mode screen in the operation mode; and an additional learning image specifying unit for receiving a specification regarding whether to additionally learn the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image, wherein the discriminator generation unit additionally learns the inspection object image specified as an additional learning object by the additional learning image specifying unit as the non-defective product image or the defective product image and updates the discriminator.

[0010] In this structure, in the setting mode, the setting of the imaging conditions of the imaging unit is received, the non-defective product image with the attributes of a non-defective product given by the user and the defective product image with the attributes of a defective product given by the user are registered, the registered non-defective product image and the registered defective product image are learned, and a discriminator for distinguishing the non-defective product image from the defective product image is generated by the discriminator generation unit.

[0011] After that, when the mode switching unit switches to the operation mode, the inspection object image newly captured by the imaging unit is input to the discriminator, and the pass / fail judgment is made through the pass / fail judgment. Since both the inspection object image and the result of the pass / fail judgment are displayed on the operation mode screen of the display unit, the user can grasp the inspection object image and the result of the pass / fail judgment of the image. At this time, when the result of the pass / fail judgment of the inspection object image is correct, the operation mode is continued, or when the result of the pass / fail judgment of the inspection object image is incorrect, additional learning can be performed. That is, in the case where although the inspection object image displayed on the operation mode screen is a non-defective product image, but the inspection object image is also judged as a defective product image, the additional learning image specifying unit receives the specification for additional learning of the inspection object image as a non-defective product image. On the other hand, in the case where although the inspection object image displayed on the operation mode screen is a defective product image, but the inspection object image is also judged as a non-defective product image, the additional learning image specifying unit receives the specification for additional learning of the inspection object image as a defective product image. In the case where the inspection object image designated as the additional learning object is a non-defective product image, the discriminator generation unit additionally learns the inspection object image as a non-defective product image. On the other hand, in the case where the inspection object image designated as the additional learning object is a defective product image, the discriminator generation unit additionally learns the inspection object image as a defective product image. Therefore, the discriminator generation unit designates the image as the additional learning object in the operation mode without switching from the operation mode to the setting mode, performs additional learning, updates the discriminator, and improves the discrimination performance.

[0012] In addition, since the user determines whether to perform additional learning, unnecessary learning such as duplicate input of the same image is prevented, thereby improving the efficiency of additional learning.

[0013] In addition, the discriminator can be generated by inputting non-defective product images and defective product images into, for example, a neural network that has been pre-learned. The neural network can have multiple layers, so that the ability to distinguish non-defective product images from defective product images can be significantly improved.

[0014] In a second aspect, the discriminator generation unit generates a discrimination boundary between the non-defective product image and the defective product image in the eigenvalue space by inputting the non-defective product image and the defective product image into a pre-learned neural network having multiple layers, and updates the discrimination boundary by the additional learning.

[0015] In this structure, since a pre-learned neural network is used, the user does not need to make the neural network learn from scratch, and can generate a discrimination boundary between non-defective product images and defective product images in the eigenvalue space by inputting non-defective product images and defective product images. In addition, since the discrimination boundary is updated by additional learning and the discriminator is also updated, the discrimination performance is improved.

[0016] In a third aspect, the image inspection device further includes: a main image registration unit configured to register, in the setting mode, an image captured under the imaging conditions set by the imaging setting unit as a main image; and an inspection window setting unit configured to, in the setting mode, receive settings of a first learning-based inspection window and a second learning-based inspection window for defining a range of difference detection on the main image registered by the main image registration unit, wherein the operation mode screen displays the result of the pass / fail determination by the pass / fail determination unit on an image within the range defined by the first learning-based inspection window, and the result of the pass / fail determination by the pass / fail determination unit on an image within the range defined by the second learning-based inspection window, and the additional learning image specifying unit receives a specification regarding whether to additionally learn images within the range defined by the first learning-based inspection window and images within the range defined by the second learning-based inspection window separately from each other.

[0017] In this structure, in the case where the first learning-based inspection window and the second learning-based inspection window are set by the inspection window setting unit, one additional learning image includes an image within the range defined by the first learning-based inspection window and an image within the range defined by the second learning-based inspection window. In this case, for example, although the result of the pass / fail determination of the image within the range defined by the first learning-based inspection window is correct, the result of the pass / fail determination of the image within the range defined by the second learning-based inspection window may be incorrect. In this case, learning becomes efficient by additionally learning the image within the range defined by the second learning-based inspection window without the need to additionally learn the image within the range defined by the first learning-based inspection window.

[0018] In a fourth aspect, after receiving a specification regarding whether to additionally learn images within the range defined by the first learning-based inspection window and images within the range defined by the second learning-based inspection window separately from each other, the additional learning image specifying unit receives an operation performed by the user to start additional learning.

[0019] In this structure, after learning whether the images within the range defined by the first learning-based inspection window and the images within the range defined by the second learning-based inspection window are attached for learning in a separated manner from each other, the additional learning can be started.

[0020] In a fifth aspect, after receiving the designation for attaching the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image for additional learning, the additional learning image designation unit receives the designation regarding whether to attach the images within the range defined by the first learning-based inspection window and the images within the range defined by the second learning-based inspection window for additional learning in a separated manner from each other.

[0021] In this structure, when receiving the designation regarding whether to attach the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image for additional learning, the additional learning image designation unit can receive the designation regarding whether the images within the range defined by the first learning-based inspection window and the images within the range defined by the second learning-based inspection window are attached for additional learning in a separated manner from each other.

[0022] In a sixth aspect, when the additional learning image designation unit receives the designation for attaching the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image for additional learning, the pass / fail judgment unit temporarily stops the operation mode, and when the additional learning image designation unit receives the designation for additional learning of the inspection object image displayed on the operation mode screen, the display unit can notify the user that the pass / fail judgment has temporarily stopped the operation mode.

[0023] In this structure, when receiving the designation for attaching the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image for additional learning, the operation mode is temporarily stopped and then the additional learning is performed. Since the display unit notifies the user that the operation mode has been temporarily stopped, the user can be aware of the execution of the additional learning.

[0024] In a seventh aspect, a method for setting an image inspection device is provided. The image inspection device is configured to perform pass / fail determination of an inspection object based on an image of the inspection object captured by an imaging unit. The setting method includes: a shooting condition setting step of setting shooting conditions of the imaging unit in a state where the mode of the image inspection device is set to a setting mode for setting the image inspection device; a learning image registration step of registering a non-defective product image with an attribute of a non-defective product given by a user and a defective product image with an attribute of a defective product given by the user in the setting mode; a discriminator generation step of learning the non-defective product image and the defective product image registered in the learning image registration step and generating a discriminator for distinguishing the non-defective product image from the defective product image in the setting mode; a pass / fail determination step of inputting a newly captured image of the inspection object by the imaging unit to the discriminator and performing pass / fail determination in an operation mode for performing inspection; a display step of displaying the image of the inspection object together with the result of the pass / fail determination performed in the pass / fail determination step on an operation mode screen in the operation mode; and an additional learning setting step of receiving a designation regarding whether to additionally learn the image of the inspection object displayed on the operation mode screen as the non-defective product image or the defective product image. In the discriminator generation step, the image of the inspection object designated as an additional learning object in the additional learning setting step is additionally learned as the non-defective product image or the defective product image, and the discriminator is updated.

[0025] According to the present invention, an image of an inspection object and the result of pass / fail determination are displayed on an operation mode screen, a designation regarding whether to additionally learn the image of the inspection object displayed on the operation mode screen as the non-defective product image or the defective product image is received, the image of the inspection object designated as an additional learning object is additionally learned, and the discriminator is updated. Therefore, in both the setting mode and the operation mode, it is possible to easily additionally learn the non-defective product image or the defective product image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a schematic diagram showing the structure of an image inspection device according to an embodiment of the present invention;

[0027] Figure 2 shows the hardware structure of the image inspection device;

[0028] Figure 3 is a block diagram showing the image inspection device;

[0029] Figure 4 shows the user interface at startup;

[0030] Figure 5 Displays an inspection type selection user interface;

[0031] Figure 6 Displays a camera condition setting user interface;

[0032] Figure 7 Displays a main image registration user interface;

[0033] Figure 8 Displays a learning setting user interface;

[0034] Figure 9 Displays an inspection window setting user interface;

[0035] Figure 10 Displays a learning user interface;

[0036] Figure 11 Displays a learning setting user interface after learning is completed;

[0037] Figure 12 Displays an image processing tool user interface;

[0038] Figure 13 Displays an image processing tool selection user interface for selecting the basic tool tab;

[0039] Figure 14 Displays an image processing tool selection user interface for selecting the extended tab;

[0040] Figure 15 Displays an output assignment user interface;

[0041] Figure 16 Is a flowchart showing the setting process of the image inspection device;

[0042] Figure 17 Displays a running mode user interface;

[0043] Figure 18 Is equivalent to a Figure 17 figure showing an interruption screen for additional learning;

[0044] Figure 19 Displays an additional learning user interface;

[0045] Figure 20 Displays a running mode user interface showing a display menu;

[0046] Figure 21 Is equivalent to a Figure 20 figure showing an interruption screen for historical type selection;

[0047] Figure 22Displays an operation image history display user interface for showing the inspection object image after passing / failing judgment;

[0048] Figure 23 Displays a selection image display user interface;

[0049] Figure 24 Displays a learning image history display user interface for showing the images used in learning;

[0050] Figure 25 Is a diagram corresponding to showing different additional learning images Figure 19 of;

[0051] Figure 26 Is a diagram corresponding to showing an interruption screen for displaying learning candidate images Figure 22 of;

[0052] Figure 27 Is a flowchart during operation in the standard inspection mode;

[0053] Figure 28A 、 28B and 28C show examples of inspection object images including parts like non-defective products and parts like defective products, Figure 28A show the case where the probability of being a non-defective product image is high, Figure 28B show the case where the probability of being a non-defective product image is about half, and Figure 28C show the case where the probability of being a defective product image is high;

[0054] Figure 29A and 29B show examples of inspection object images including characters, Figure 29A show non-defective product images, and Figure 29B show inspection object images with selectively enhanced regions;

[0055] Figure 30A show non-defective product images, Figure 30B show defective product images, Figure 30C show inspection object images in the case of photographing a non-defective product during the operation mode, and Figure 30D show inspection object images in the case of photographing a defective product during the operation mode;

[0056] Figure 31A show defective product images to be additionally learned, and Figure 31B show inspection object images in the case of photographing a defective product during the operation mode after additional learning;

[0057] Figure 32 Is a flowchart during operation in the learning inspection mode;

[0058] Figure 33 is a flowchart in the case of presenting learning candidate images;

[0059] Figure 34 is a flowchart showing the sorting process of historical images;

[0060] Figure 35 Schematically shows an eigenvalue space in which a non-defective product image group and a defective product image group are plotted and a discrimination boundary is generated;

[0061] Figure 36 is a flowchart showing the process for presenting deletion candidate images;

[0062] Figure 37 is a flowchart showing the process of presenting deletion candidate images in the case where the discrimination boundary is changed by additional learning;

[0063] Figure 38 is equivalent to the one in which the discrimination boundary has been changed by additional learning Figure 35 diagram;

[0064] Figure 39 is a flowchart showing the process for presenting deletion candidate images based on the relative relationship between deletion candidate images and adjacent images; and

[0065] Figure 40 is equivalent to the one that illustrates the determination method of the relative relationship between deletion candidate images and adjacent images Figure 35 diagram. Detailed implementation manners

[0066] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. It should be noted here that the following preferred embodiments are substantially only examples and do not limit the present invention, its applications, or its uses.

[0067] Figure 1 is a schematic diagram showing the structure of an image inspection device 1 according to an embodiment of the present invention. The image inspection device 1 makes a pass / fail determination of inspection objects based on images obtained by photographing inspection objects such as various components or products, etc., and can be used in a production site such as a factory. All or part of an inspection object body can be an inspection object. Alternatively, one inspection object body can include multiple inspection objects. Alternatively, an image can include multiple inspection object bodies.

[0068] The image inspection device 1 includes a control unit 2, an imaging unit 3, a display device (display unit) 4, and a personal computer 5 as the device main body. The personal computer 5 is not essential and can be omitted. The personal computer (display unit) 5 can be used to replace the display device 4. Although as an example of the structure of the image inspection device 1, Figure 1 the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are illustrated as separate components, but any of these components can be integrally combined with each other. For example, the control unit 2 can be integrated with the imaging unit 3, or the control unit 2 can be integrated with the display device 4. Alternatively, the control unit 2 can be divided into multiple units such that a part of these units is built into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into multiple units such that a part of these units is built into other units.

[0069] (Structure of the imaging unit 3)

[0070] As Figure 2 shown, the imaging unit 3 includes a camera module (imaging unit) 14 and an illumination module 15. The camera module 14 includes an AF motor 141 for driving an imaging optical system and an imaging board 142. The AF motor 141 automatically focuses by driving the lens of the imaging optical system, and can perform focusing using conventionally known autofocus methods and the like. The imaging board 142 includes a CMOS sensor 143 as a light-receiving element for receiving light incident from the imaging optical system, an FPGA 144, and a DSP 145. The CMOS sensor 143 is an imaging sensor configured to obtain a color image. Instead of the CMOS sensor 143, a light-receiving element such as a CCD sensor can be used. The FPGA 144 and the DSP 145 perform image processing in the imaging unit 3, and the signal output from the CMOS sensor 143 is also input to the FPGA 144 and the DSP 145.

[0071] The illumination module 15 includes an LED (light-emitting diode) 151 as a light-emitting device for illuminating an illumination area including an inspection object, and an LED driver 152 for controlling the LED 151. The light-emitting timing, light-emitting time, and light-emitting amount of the LED 151 can be set to any value by the LED driver 152. The LED 151 can be provided integrally with the imaging unit 3, or can be provided separately from the imaging unit 3 as an external illumination unit. Although not shown, the illumination module 15 includes a reflector for reflecting the light emitted from the LED 151, and a lens through which the light emitted from the LED 151 passes. The illumination range of the LED 151 is set such that the light emitted from the LED 151 irradiates the inspection object and the peripheral area of the inspection object. A light emitter other than a light-emitting diode can also be used.

[0072] (Structure of Control Unit 2)

[0073] The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The FPGA 131, DSP 132, and memory 133 are mounted on the main board 13. The FPGA 131 and DSP 132 constitute the control section 13A, and a main control section formed by integrating these components may also be provided.

[0074] The control section 13A of the main board 13 controls the operations of these connected boards and modules. For example, the control section 13A outputs a lighting control signal for controlling the lighting or extinguishing of the LED 151 to the LED driver 152 of the lighting module 15. The LED driver 152 switches between the lighting and extinguishing of the LED 151 according to the lighting control signal from the control section 13A and controls the light amount of the LED 151.

[0075] In addition, the control section 13A outputs a camera control signal for controlling the CMOS sensor 143 to the camera board 142 of the camera module 14. The CMOS sensor 143 starts imaging according to the camera control signal from the control section 13A and performs imaging by adjusting the exposure time to an arbitrary value. That is, the imaging unit 3 images the inside of the field of view of the CMOS sensor 143 according to the camera control signal output from the control section 13A. The imaging unit 3 images the inspection object when the inspection object exists in the field of view, or images the object when an object other than the inspection object exists in the field of view. For example, when setting up the image inspection device 1, a non-defective product image given the attribute of a non-defective product by the user and a defective product image given the attribute of a defective product by the user can be imaged. During the operation of the image inspection device 1, the inspection object can be imaged. At any time, the CMOS sensor 143 can output a real-time image as the currently captured image at a short frame rate.

[0076] After the imaging using the CMOS sensor 143 is completed, the image signal output from the imaging unit 3 is input to the FPGA 131 of the main board 13, processed by the FPGA 131 and DSP 132, and stored in the memory 133. The specific processing of the control section 13A of the main board 13 will be described later.

[0077] The connector board 16 receives power from the outside via a power connector (not shown) provided in the power interface 161. The power board 18 distributes the power received by the connector board 16 to boards and modules, etc. (specifically, the lighting module 15, the camera module 14, the main board 13, and the communication board 17). The power board 18 has an AF motor driver 181. The AF motor driver 181 supplies driving power to the AF motor 141 of the camera module 14 to achieve autofocus. The AF motor driver 181 adjusts the power to be supplied to the AF motor 141 according to an AF control signal from the control unit 13A of the main board 13.

[0078] The communication board 17 outputs the pass / fail judgment signal and image data of the inspection object output from the control unit 13A of the main board 13, as well as the user interface, etc. to the display device 4, the personal computer 5, and an external control device (not shown), etc. The display device 4 and the personal computer 5 have, for example, a display panel including a liquid crystal panel, etc., and display the image data and the user interface, etc. on the display panel.

[0079] In addition, the communication board 17 receives various operations input by the user via the touch panel 41 of the display device 4 or the keyboard 51 of the personal computer 5, etc. The touch panel 41 of the display device 4 is a conventionally known touch operation panel having, for example, a pressure sensor, detects the touch operation of the user, and outputs the touch operation to the communication board 17. In addition to the keyboard 51, the personal computer 5 has a mouse and a touch panel (not shown) to receive various operations input by the user through these operation devices. Communication can be performed via wired or wireless means, and any communication form can be achieved by a conventionally known communication module.

[0080] The control unit 2 has a storage device (storage unit) 19 such as a hard disk drive. The storage device 19 stores program files 80 and setting files, etc. (software), main images, historical images, and the results of pass / fail judgments, etc. for implementing various types of control and processing described later via the above-mentioned hardware. The program files 80 and the setting files can be stored in a storage medium 90 such as an optical disc, etc., and the program files 80 and the setting files can be installed in the control unit 2.

[0081] (Specific Structure of the Image Inspection Device 1)

[0082] Figure 3 is a block diagram showing the image inspection device 1, and Figure 3Each of the shown parts and units is configured by the control unit 2 installed with the program file 80 and the setting file. That is, the image inspection device 1 includes a shooting setting unit 21, a main image registration unit 23, a learning image registration unit 24, an image processing tool selection unit 25, a parameter adjustment unit 26, an inspection window setting unit 27, an inspection type selection unit 28, a mode switching unit 29, a discriminator generation unit 30, a pass / fail judgment unit 31, an additional learning image specifying unit 32, a stability evaluation calculation unit 33, and a judgment axis extraction unit 34. These parts (units) can be configured by only hardware or a combination of hardware and software. Additionally, Figure 3 Each of the shown parts and units can be independent of each other, or can be one piece of hardware or software with multiple functions. Additionally, Figure 3 The functions of each of the shown parts and units can be realized through the control of the control unit 13A of the main board 13.

[0083] Additionally, in the image inspection device 1, it is possible to switch between a setting mode and an operation mode. The setting mode is used to set various parameters such as shooting settings, register the main image, and generate (learn) a discriminator that can distinguish the internal area of a non-defective product image from a defective product image, and the operation mode is used to perform pass / fail judgment on the inspection object based on the captured image at the actual site. In the setting mode, preparatory work is carried out so that the user can distinguish non-defective products from defective products in the desired product inspection. The switching between the setting mode and the operation mode can be performed via the user interface described later, or the switch to the operation mode can be automatically performed immediately after the completion of the setting mode. In the operation mode, it is possible to modify or change (so-called additional learning) the discrimination boundary of the discriminator.

[0084] Additionally, in the image inspection device 1, it is possible to switch between a rule-based inspection mode (standard inspection mode) and a learning-based inspection mode (learning inspection mode). The rule-based inspection mode performs pass / fail judgment on the inspection object based on various feature values (such as color, edge, and position, etc.) of the inspection object in the image, and the learning-based inspection mode generates a discriminator and uses the generated discriminator to perform pass / fail judgment on the inspection object. In the standard inspection mode, settings representing the feature values included in the image based on which the user's inspection is performed, as well as thresholds, etc. are set, and pass / fail judgment is performed according to this setting. The switching between the standard inspection mode and the learning inspection mode can be performed on the user interface described later.

[0085] (Control at startup)

[0086] When the image inspection device 1 is started up, the control unit 2 generates as Figure 4The startup user interface 100 shown is displayed on the display device 4. The startup user interface 100 can also be generated by a UI generation unit (not shown) or the like. Other user interfaces described below can also be generated by a UI generation unit or the like.

[0087] The startup user interface 100 includes a sensor setting button 100a, an operation start button 100b, and an image display area 100c. Operating the sensor setting button 100a sets the image inspection device 1. Operating the operation start button 100b enters the operation mode for performing inspections. The image display area 100c can display images captured by the imaging unit 3, images stored in the storage device 19, and images read from the outside, etc.

[0088] When it is detected that the Figure 4 sensor setting button 100a of the startup user interface 100 shown is pressed, Figure 3 the mode switching unit 29 shown switches to the setting mode for setting the image inspection device 1. On the other hand, when it is detected that the operation start button 100b of the startup user interface 100 is pressed, the mode switching unit 29 switches to the operation mode. That is, the mode switching unit 29 switches the mode of the operation start button 100b between the setting mode for setting the image inspection device 1 and the operation mode for performing inspections. A method for switching the mode of the image inspection device 1 can be performed, for example, by a signal input from the outside, rather than by the above button operation. It should be noted here that "pressing of the button" is performed by operating the touch panel 41 of the display device 4, or by operating the keyboard 51 or mouse of the personal computer 5. This also applies to the button operations described below.

[0089] (Structure of the inspection type selection unit 28)

[0090] Figure 3 The inspection type selection unit 28 shown receives the selection of the standard inspection mode or the learning inspection mode. The inspection type selection unit 28 generates Figure 5 the inspection type selection user interface 101 shown and displays this interface on the display device 4. The inspection type selection user interface 101 includes a learning inspection mode selection button 101a, a standard inspection mode selection button 101b, a cancel button 101c, and an OK button 101d. When it is detected that the learning inspection mode selection button 101a is pressed, the inspection type selection unit 28 switches to the learning inspection mode. On the other hand, when it is detected that the standard inspection mode selection button 101b is pressed, the inspection type selection unit 28 switches to the standard inspection mode. After that, when the OK button 101d is pressed, the received inspection mode is accepted. When the cancel button 101c is pressed, the Figure 4The startup user interface 100 shown is displayed on the display device 4.

[0091] The part that receives the user's selection of the standard inspection mode or the learning inspection mode is not limited to the above-described inspection type selection part 28, and this part can receive the selection according to, for example, a signal input from the outside.

[0092] Figure 5 The inspection type selection user interface 101 shown has an explanatory text display area 101e, in which an explanatory text for describing the outline of the learning inspection mode and an explanatory text for describing the outline of the standard inspection mode are displayed. When the learning inspection mode selection button 101a is pressed, the explanatory text for describing the outline of the learning inspection mode is displayed in the explanatory text display area 101e. On the other hand, when the standard inspection mode selection button 101b is pressed, the explanatory text for describing the outline of the standard inspection mode is displayed in the explanatory text display area 101e.

[0093] (Structure of the imaging setting unit 21)

[0094] Figure 3 The imaging setting unit 21 shown receives the setting of the imaging conditions of the imaging unit 3 in the setting mode. Although the imaging conditions include, for example, the exposure time, the amount of light to be emitted from the illumination module 15, the gain, the focusing, and the trigger condition for imaging, the imaging conditions are not limited to these and can include conditions that can be set for imaging.

[0095] The imaging setting unit 21 generates Figure 6 The imaging condition setting user interface 102 shown and displays this interface on the display device 4. The imaging condition setting user interface 102 includes an image display area 102a, a trigger condition setting button 102b, a brightness adjustment button 102c, a focusing button 102d, and a next button 102e. The image display area 102a displays the real-time image captured by the CMOS sensor 143. The real-time image represents the current image being captured by the CMOS sensor 143, and when the imaging object or the brightness changes, the image in the image display area 102a changes substantially in real time.

[0096] The reference sign W represents the inspection object.

[0097] The trigger condition setting button 102b sets a imaging trigger signal for causing the imaging unit 3 to start imaging. When it is detected that the trigger condition setting button 102b has been operated, the imaging setting unit 21 generates a user interface (not shown) capable of setting the imaging trigger signal, and displays this interface on the display device 4. The user performs imaging using the imaging unit 3 in a state where the inspection object W exists within the field of view of the CMOS sensor 143 by selecting, for example, an imaging trigger signal input from the outside. The imaging trigger signal can be input from an external PLC or the like, or can be internally generated at a predetermined timing.

[0098] The brightness adjustment button 102c adjusts the exposure time, the amount of light emitted from the illumination module 15, the gain, etc., and can adjust the brightness of the image to be obtained. When it is detected that the brightness adjustment button 102c has been operated, the imaging setting unit 21 generates a user interface (not shown) capable of adjusting the exposure time, etc., and displays this interface on the display device 4. The user can adjust the brightness of the image while viewing the image in the image display area 102a. Since the image inspection device 1 can automatically adjust the brightness of the image to improve the inspection accuracy, the user's adjustment is not necessary.

[0099] Operate the focus adjustment button 102d to perform focusing. When it is detected that the focus adjustment button 102d has been operated, the imaging setting unit 21 generates a user interface (not shown) capable of performing focusing, and displays this interface on the display device 4. Although the above-mentioned autofocus function performs automatic adjustment to focus on the inspection object W, the user can use this function to manually focus on the inspection object W.

[0100] When the imaging conditions are set, operate the next step button 102e. When it is detected that the next step button 102e has been pressed, the imaging setting unit 21 accepts the trigger condition, brightness, focus, etc., and the process proceeds to the next process, i.e., the registration of the main image.

[0101] (Structure of the main image registration unit 23)

[0102] Figure 3 The shown main image registration unit 23 registers, in the setting mode, the image captured under the imaging conditions set by the imaging setting unit 21 as the main image. The main image is an image used as a reference for setting an image inspection frame (i.e., inspection window), and includes the inspection object W of a non-defective product. The main image can be stored in the storage device 19.

[0103] The main image registration unit 23 generates Figure 7The main image registration user interface 103 as shown is presented and displayed on the display device 4. The main image registration user interface 103 includes a main candidate image display area 103a that displays images as candidates for the main image, a live image registration button 103b, a history registration button 103c, a file registration button 103d, and a next button 103e.

[0104] When it is detected that the live image registration button 103b is pressed while the live image is being displayed in the main candidate image display area 103a, the main image registration unit 23 registers the live image displayed in the main candidate image display area 103a as the main image. Additionally, operate the history registration button 103c to set the image held in the image history as the main image. When the main image registration unit 23 detects that the history registration button 103c is pressed, the images taken in the past are displayed in the main candidate image display area 103a. From the images displayed in the main candidate image display area 103a, a desired image can be selected and registered as the main image. The images taken in the past can be automatically stored in the storage device 19 or can be stored through the user's storage operation.

[0105] Operate the file registration button 103d to set the image stored in the storage device 19 as the main image. When it is detected that the file registration button 103d is pressed, the main image registration unit 23 can read the images stored in a specific folder of the storage device 19, display these images in the main candidate image display area 103a, select a desired image from these images as the main image, and register the main image.

[0106] When the main image is registered, operate the next button 103e. When it is detected that the next button 103e is pressed, the main image registration unit 23 transitions to the next process.

[0107] (Learning setting user interface)

[0108] Figure 8 The learning setting user interface 104 is shown. After the main image is registered and the Figure 7 shown next button 103e is pressed, the control unit 2 generates the learning setting user interface 104 and displays it on the display device 4. The learning setting user interface 104 includes a main image display area 104a that displays the main image, an inspection window setting button 104b, a learning button 104c, and a next button 104d. Press the next button 104d to proceed to the next process.

[0109] (Structure of the inspection window setting unit 27)

[0110] Figure 3The inspection window setting unit 27 shown receives the setting of learning inspection windows (learning-based inspection windows) 200A and 200B for defining the range of difference detection on the main image registered in the main image registration unit 23 when setting the learning inspection mode. When it is detected that the Figure 8 inspection window setting button 104b shown is pressed, the inspection window setting unit 27 generates Figure 9 the inspection window setting user interface 105 shown and displays this interface on the display device 4. The setting of the learning inspection windows 200A and 200B is not necessary. The learning inspection window 200A is the first learning-based inspection window, and the learning inspection window 200B is the second learning-based inspection window.

[0111] The inspection window setting user interface 105 includes a main image display area 105a that displays the main image, an inspection window addition button 105b, a position compensation addition button 105c, a position compensation display unit 105d, a first inspection window display unit 105e, a second inspection window display unit 105f, and an OK button 105g. When it is detected that the inspection window addition button 105b is pressed, the inspection window setting unit 27 places the learning inspection windows 200A and 200B in a settable state. The learning inspection windows 200A and 200B can be set to surround the entire inspection object W in the main image or only the characteristic part of the inspection object W. The learning inspection windows 200A and 200B can be, for example, rectangular. In the case of assuming the rectangular ranges of the learning inspection windows 200A and 200B, the learning inspection windows 200A and 200B are superimposed on the main image in the main image display area 105a by dragging from the upper diagonal corner to the lower diagonal corner via a touch operation. The positions, sizes, and shapes of the learning inspection windows 200A and 200B can be corrected.

[0112] Although the first learning inspection window 200A and the second learning inspection window 200B are set in this example, only one of these learning inspection windows can be set, or three or more learning inspection windows can be set. A learning inspection window can be added by pressing the inspection window addition button 105b. The first inspection window display unit 105e corresponds to the first learning inspection window 200A, and the second inspection window display unit 105f corresponds to the second learning inspection window 200B.

[0113] In addition, a position compensation window 201 for identifying the part to be subjected to the position compensation process can be set by pressing the position compensation addition button 105c. The setting method of the position compensation window 201 can be the same as the setting method of the learning inspection windows 200A and 200B. The image in the position compensation window 201 is the object of the position compensation process, and the image in the position compensation window 201 can be compensated to have a predetermined position and a predetermined posture by a conventionally known method.

[0114] When the OK button 105g of the inspection window setting user interface 105 shown in Figure 9 is pressed, the learning setting user interface 104 shown in Figure 8 is displayed on the display device 4. When the learning button 104c of the learning setting user interface 104 is pressed, the process proceeds to learning image registration.

[0115] (Structure of the learning image registration unit 24)

[0116] Figure 3 The learning image registration unit 24 shown in registers a non-defective product image with the attributes of a non-defective product given by the user and a defective product image with the attributes of a defective product given by the user in the setting mode of the learning inspection mode. The non-defective product image is an image obtained by photographing the inspection object W of the non-defective product, and the defective product image is an image other than the non-defective product image. The non-defective product image and the defective product image are registered separately from the main image.

[0117] When it is detected that the learning button 104c of the learning setting user interface 104 shown in Figure 8 is pressed, the learning image registration unit 24 displays the learning user interface 106 shown in Figure 10 on the display device 4. The learning user interface 106 includes an image display area 106a that can display a non-defective product image and a defective product image as learning images, a non-defective product image acquisition operation unit 106b, a defective product image acquisition operation unit 106c, a learning start button 106d, a file read button 106e, and a history read button 106f.

[0118] A live image can be displayed in the image display area 106a. When the non-defective product image acquisition operation unit 106b is operated in a state where a non-defective product image is displayed as a live image in the image display area 106a, the live image is registered as a non-defective product image. Similarly, when the defective product image acquisition operation unit 106c is operated in a state where a defective product image is displayed as a live image in the image display area 106a, the live image is registered as a defective product image. By repeating this operation, a plurality of non-defective product images and defective product images can be registered.

[0119] When the file reading button 106e is pressed, the images stored in a specific folder of the storage device 19 are displayed in the image display area 106a. When the non-defective product image acquisition operation unit 106b is operated while the image displayed in the image display area 106a is in the state of a non-defective product image, this image is registered as a non-defective product image. Similarly, when the defective product image acquisition operation unit 106c is operated while a defective product image read from the storage device 19 is displayed in the image display area 106a, this image is registered as a defective product image.

[0120] When the history reading button 106f is pressed, the images captured in the past are displayed in the image display area 106a. When the non-defective product image acquisition operation unit 106b is operated when the image displayed in the image display area 106a is a non-defective product image, this image is registered as a non-defective product image. Similarly, when the defective product image acquisition operation unit 106c is operated when a defective product image captured in the past is displayed in the image display area 106a, this image is registered as a defective product image.

[0121] The method for registering non-defective product images is not limited to the above method. For example, when the learning inspection mode and the setting mode are selected, the learning image registration unit 24 can register a plurality of images obtained by causing the imaging unit 3 to image the inspection object W multiple times while changing the imaging conditions as non-defective product images. The imaging conditions can include, for example, lighting conditions and exposure time. The non-defective product images obtained by causing the imaging unit 3 to image the inspection object W each time one or both of the lighting conditions and the exposure time are changed can be learning images. The lighting conditions are, for example, the amount of light emitted from the lighting module 15.

[0122] In addition, when the learning inspection mode and the setting mode are selected, the learning image registration unit 24 can also automatically generate and register a plurality of non-defective product images based on the images obtained by imaging the inspection object W with the imaging unit 3. The automatically generated images are images newly determined as non-defective product images (automatically generated non-defective product images), and the number of such images can be one or more than one. For example, the automatically generated non-defective product images can be obtained by reading the non-defective product range (specifically, the brightness range) that contributes to distinguishing the non-defective product images from the defective product images, and automatically generating images having brightness values falling within the non-defective product range. Specifically, the darkest image and the brightest image in the non-defective product range can be generated by changing the brightness of the pre-input non-defective product images via image processing, and images having intermediate brightness values in the non-defective product range can also be generated. This enables the user to generate a plurality of non-defective product images without the need to prepare a plurality of non-defective products. The automatically generated non-defective product images generated as described above can be registered for the learning described later. The non-defective product images captured by the imaging unit 3 and the automatically generated non-defective product images can be used for learning.

[0123] (Structure of the discriminator generation unit 30)

[0124] Figure 3 The discriminator generation unit 30 shown learns the non-defective product images and the defective product images registered by the learning image registration unit 24 in the setting mode of the learning inspection mode, and generates a discriminator that distinguishes the non-defective product images from the defective product images. When it is detected that Figure 10 the learning start button 106d of the learning user interface 106 shown is pressed, the discriminator generation unit 30 reads the non-defective product images and the defective product images registered in the learning image registration unit 24. The discriminator generation unit 30 inputs the read non-defective product images and defective product images into a machine learning device including a plurality of layers.

[0125] Specifically, the discriminator generation unit 30 has a neural network and can use a so-called deep learning method, and can also use well-known genetic algorithms (GA), reinforcement learning, or neuro-fuzzy methods, etc. In this deep learning method, the initial values of the parameters of the neural network are randomly determined in advance, the error of image recognition output from the neural network is fed back, and the parameters are adjusted. The neural network with the adjusted parameters (i.e., the learned neural network) can be provided to the user, or the user can adjust the parameters. The neural network can have a plurality of pre-learning layers.

[0126] Feature values that contribute to distinguishing non-defective product images from defective product images using a discriminator are, for example, brightness, angle, color, shape, and position, etc. However, feature values other than these can also contribute to distinguishing non-defective product images from defective product images, so the feature values are not particularly limited.

[0127] When the learning of the discriminator generation unit 30 is completed, a learned display 104e is displayed on the learning setting user interface 104 as shown in Figure 11 .

[0128] In addition, the discriminator generation unit 30 generates, for example, a discrimination boundary between non-defective product images and defective product images in the feature value space as shown in Figure 35 by inputting non-defective product images and defective product images into a neural network.

[0129] (Structure of the image processing tool selection unit 25)

[0130] The above has described the setting process in the learning inspection mode. The following settings are made in the case of the standard inspection mode. First, the image processing tool selection unit 25 shown in Figure 3 will be described. The image processing tool selection unit 25 functions when the mode switching unit 29 switches to the setting mode and the inspection type selection unit 28 receives the selection of the standard inspection mode, and can receive the selection of an image processing tool. When entering the setting mode and the standard inspection mode, the image processing tool selection unit 25 generates Figure 12 the image processing tool user interface 107 shown in and displays this interface on the display device 4.

[0131] The image processing tool user interface 107 includes a main image display area 107a for displaying a main image, a tool display area 107b, and a next button 107c. One or more selected image processing tools can be displayed in the tool display area 107b. In the present embodiment, a contour tool 107d, a color area tool 107e, and a width tool 107f are displayed in the tool display area 107b.

[0132] The image processing tool user interface 107 includes a tool addition button 107g. When it is detected that the tool addition button 107g is pressed, the image processing tool selection unit 25 generates Figure 13 the image processing tool selection user interface 108 shown in and displays this interface in the display device 4. The image processing tool selection user interface 108 includes a basic tool tab 108a, an extension 1 tab 108b, an extension 2 tab 108c, a cancel button 108d, and an OK button 108e. When the user selects the basic tool tab 108a, as shown in Figure 13The following shows the image processing tools belonging to the basic tool tab 108a. On the other hand, when the user selects the extension 1 tab 108b, as Figure 14 shown, the image processing tools belonging to the extension 1 tab 108b are listed. Although not shown, when the user selects the extension 2 tab 108c, the image processing tools are similarly displayed. The image processing tools are conventionally known, and for example, the edge extraction function extracts the edges of the inspection object W in the image, so that the tilt angle of a specific edge with respect to the X-axis and the length of the specific edge, etc. can be used as feature values.

[0133] Any image processing tool can be selected from the image processing tools belonging to the basic tool tab 108a, the extension 1 tab 108b, and the extension 2 tab 108c. When it is detected that the OK button 108e has been pressed after an image processing tool is selected, the selected image processing tool becomes valid and is displayed in the tool display area 107b of the image processing tool user interface 107 as shown in Figure 12 . On the other hand, when it is detected that the cancel button 108d of the image processing tool selection user interface 108 shown in Figure 13 and Figure 14 has been pressed, the selection operation of the image processing tool is cancelled.

[0134] (Inspection window setting unit 27 in the standard inspection mode)

[0135] Figure 3 The inspection window setting unit 27 shown receives the setting of a standard inspection window (rule-based inspection window) in the setting mode of the standard inspection mode, which is used to define the range to which the image processing tool selected by the image processing tool selection unit 25 is applied on the main image registered in the main image registration unit 23. As shown in Figure 12 , the standard inspection window 202 can be superimposed on the main image. The setting method of the standard inspection window 202 can be the same as the setting method of the inspection window in the learning inspection mode. In addition, multiple standard inspection windows 202 can be set. Furthermore, similar to the learning inspection mode, a position compensation window 203 can be set.

[0136] (Structure of the parameter adjustment unit 26)

[0137] Figure 3 The parameter adjustment unit 26 shown receives the adjustment of the parameters of the image processing tool selected by the image processing tool selection unit 25 in the setting mode of the standard inspection mode. In Figure 12In the tool display area 107b of the image processing tool user interface 107 shown, the image processing tool selected by the image processing tool selection unit 25 is displayed. For each of the contour tool 107d, the color area tool 107e, and the width tool 107f, a slider is provided that can adjust a threshold value as a parameter. The threshold value can be adjusted by the slider or by inputting a value. Parameters other than the threshold value can be adjusted for each image processing tool, and the parameters that can be adjusted by the parameter adjustment unit 26 are not particularly limited.

[0138] (Output assignment operation)

[0139] When the control unit 2 detects that the Figure 8 and Figure 11 Next button 104d of the learning setting user interface 104 shown, or when the Next button 107c of the image processing tool user interface 107 shown is pressed, the process proceeds to the output assignment process. When it is detected that the Next button 104d or the Next button 107c is pressed, the control unit 2 generates Figure 12 the output assignment user interface 109 shown, and displays this interface on the display device 4. Figure 15

[0140] Figure 4 The output assignment user interface 109 includes a setting area 109a for making settings related to output and a completion button 109b. In the setting area 109a, multiple output-related settings such as comprehensive judgment, busy, and error settings can be made. When the output-related settings are made, the completion button 109b is operated. When it is detected that the completion button 109b is pressed, the control unit 2 ends the setting mode, and for example, Figure 4 the startup user interface 100 shown is displayed on the display device 4.

[0141] (Setting method of the image inspection device 1)

[0142] As described above, when the setting mode is selected after startup, the selection of the learning inspection mode or the standard inspection mode setting is received, and the received inspection mode can be set. Since the imaging conditions are set and the main image is registered regardless of whether the learning inspection mode or the standard inspection mode is selected, the setting of the imaging conditions and the registration of the main image are common setting items for the learning inspection mode and the standard inspection mode.

[0143] In the case of setting the learning inspection mode, when the imaging conditions are set and the main image is registered, an inspection window is set, and a learning image is input and registered to generate a discriminator. On the other hand, in the case of the standard inspection mode, when the imaging conditions are set and the main image is registered, an image processing tool is selected and the parameters of the image processing tool are adjusted. After that, regardless of whether the learning inspection mode or the standard inspection mode is selected, the same output assignment is performed. Therefore, the output assignment is also a common setting item for the learning inspection mode and the standard inspection mode.

[0144] Refer to Figure 16 the flowchart shown to illustrate an example of the setting method of the image inspection device 1. The step SA1 after START is a startup operation such as turning on the power. In step SA1, the Figure 4 startup user interface 100 shown is displayed on the display device 4. In step SA2, the sensor setting button 100a of the startup user interface 100 is pressed. When the sensor setting button 100a is pressed, the process proceeds to step SA3, and the Figure 5 inspection type selection user interface 101 shown is displayed on the display device 4. In step SA4, it is judged whether the inspection mode selected in the inspection type selection user interface 101 is the learning inspection mode. Steps SA1 to SA4 are common to the learning inspection mode and the standard inspection mode.

[0145] When pressing the Figure 5 learning inspection mode selection button 101a of the inspection type selection user interface 101 shown, the process proceeds to step SA5. In step SA5, since the Figure 6 imaging condition setting user interface 102 shown is displayed on the display device 4, the user sets the imaging conditions. This is the imaging condition setting step. When step SA5 is completed, the process proceeds to step SA6, and the Figure 7 main image registration user interface 103 shown is displayed on the display device 4. This step is the main image registration step for registering the image captured under the imaging conditions set in step SA5 as the main image. After that, when the Figure 8 learning setting user interface 104 shown is displayed on the display device 4 when the process enters step SA7, the inspection window setting button 104b is pressed to set the inspection window. When the learning button 104c is pressed after the inspection window is set, the process proceeds to step SA8. In step SA8, the Figure 10The learning user interface 106 shown is displayed on the display device 4. This allows the non-defective product image to be registered as a learning image. Furthermore, in step SA9, the defective product image can be registered on the user interface 106 as a learning image. Step SA8 can be interchanged with step SA9. Steps SA8 and SA9 are the learning image registration steps.

[0146] Then, the process proceeds to step SA10. Figure 10 When the learning start button 106d of the learning user interface 106 is pressed, step SA10 is started. Step SA10 is a discriminator generation step for learning non-defective product images and defective product images and generating a discriminator that distinguishes non-defective product images from defective product images.

[0147] Although not required, additional learning can be performed in steps SA11 and SA12. If the user determines that additional learning is required, the user proceeds from step SA11 to step SA12, registers other non-defective product images and other defective product images, and causes the discriminator generation unit 30 to learn these images. When learning is completed, Figure 11 The illustrated learning setting user interface 104 is displayed on the display device 4 .

[0148] After that, the process goes to step SA13. In step SA13, Figure 15 The output assignment user interface 109 shown is displayed on the display device 4 so that output-related settings can be made.

[0149] On the other hand, if step SA4 yields a "No" result and the standard inspection mode is selected, processing proceeds to step SA14. Step SA14 is identical to step SA5. Furthermore, step SA15, which is a subsequent step, is identical to step SA6. In this flowchart, although steps SA5 and SA6 can be described separately from steps SA14 and SA15, since step SA5 is identical to step SA14, these two steps can be described as identical, and since step SA6 is identical to step SA15, these two steps can be described as identical.

[0150] After that, when the process enters step SA16, Figure 12 The image processing tool user interface 107 is shown displayed on the display device 4 so that an image processing tool can be selected. This is the image processing tool selection step.

[0151] After selecting an image processing tool, the process proceeds to step SA17. In step SA17, the user sets a check window that defines the range for applying the image processing tool selected in the image processing tool selection step (step SA16) on the main image registered in the main image registration step (step SA15). This is the check window setting step.

[0152] With the check window set, the process proceeds to step SA18. In step SA18, the parameters of the image processing tool selected in the image processing tool selection step (step SA16) are adjusted. This is the parameter adjustment step. After that, the process proceeds to step SA13, and settings related to output are made. As described above, settings for the learning check mode and the standard check mode can be performed.

[0153] (Structure of the pass / fail judgment unit 31)

[0154] Next, the part related to the operation mode will be described. When the standard check mode is selected by the check type selection unit 28 in the operation mode, Figure 3 the shown pass / fail judgment unit 31 performs a pass / fail judgment on the inspection object W by setting a standard check window on the newly captured image by the imaging unit 3 and applying the image processing tool selected by the image processing tool selection unit 25 to the image in the standard check window. On the other hand, when the learning check mode is selected by the check type selection unit 28, the pass / fail judgment unit 31 performs a pass / fail judgment on the inspection object W by inputting the newly captured image by the imaging unit 3 into the discriminator. When the learning check mode is selected, the range of difference detection can be defined by setting a learning check window.

[0155] In the case where Figure 17 as shown, multiple learning check windows are set, the pass / fail judgment unit 31 performs a pass / fail judgment for each learning check window. That is, the pass / fail judgment unit 31 determines whether the image in the learning check window 200A is a non-defective product image or a defective product image, determines whether the image in the learning check window 200B is a non-defective product image or a defective product image, and separately outputs the judgment results. The pass / fail judgment unit 31 outputs a comprehensive judgment result considering multiple judgment results. When the multiple judgment results are the same, this result can be the comprehensive judgment result. When the judgment results are different from each other, the comprehensive judgment result can be a defective product image.

[0156] The pass / fail judgment unit 31 generates Figure 17The operation mode user interface 120 shown is displayed on the display device 4. The operation mode user interface 120 is included in the operation mode screen. The operation mode user interface 120 includes an image display area 120a for displaying an image of an object to be inspected, a comprehensive judgment result display area 120b, a position compensation result display area 120c, a first inspection window result display area 120d, a second inspection window result display area 120e, an additional learning button 120f, and a threshold adjustment button 120g.

[0157] In the image display area 120a, an image of the object to be inspected newly captured by the imaging unit 3 during the operation mode is displayed. In the comprehensive judgment result display area 120b, it is displayed whether the image of the object to be inspected displayed in the image display area 120a is a non-defective product image or a defective product image. Specifically, when the image of the object to be inspected is a non-defective product image, "OK" is displayed, or when the image of the object to be inspected is a defective product image, "NG" is displayed. However, any display form can be adopted as long as the user can judge whether the image of the object to be inspected is a non-defective product image or a defective product image. Since the image display area 120a and the comprehensive judgment result display area 120b are provided on the same operation mode user interface 120, in the operation mode, the image of the object to be inspected can be displayed together with the pass / fail judgment result on the display device 4.

[0158] The position compensation result display area 120c indicates the result obtained by applying position compensation to the object to be inspected W included in the image of the object to be inspected. The position compensation result display area 120c indicates whether the position compensation was normally performed or the position compensation failed in a manner separate from the comprehensive judgment result. The first inspection window result display area 120d indicates the pass / fail judgment result of the image in the learning inspection window 200A, and the second inspection window result display area 120e indicates the pass / fail judgment result of the image in the learning inspection window 200B. In addition, the threshold adjustment button 120g adjusts the threshold for pass / fail judgment, and when the threshold is adjusted, the pass / fail judgment unit 31 performs pass / fail judgment based on the changed threshold.

[0159] Operate the additional learning button 120f to additionally learn the image of the object to be inspected displayed in the image display area 120a as described later.

[0160] In addition, during the operation mode of the standard inspection mode, the pass / fail judgment unit 31 uses an image processing tool to perform pass / fail judgment on the object to be inspected W. The pass / fail judgment of the object to be inspected W can be performed, for example, by comparing color, edge, position, etc. with a threshold specified by the user. As in the learning inspection mode, the pass / fail judgment result can be displayed on the user interface.

[0161] (Configuration of Additional Learning Image Designation Unit 32)

[0162] Figure 3 The additional learning image designation unit 32 receives a designation regarding whether to additionally learn the inspection object image displayed on the operation mode screen during the learning inspection mode as a non-defective product image or a defective product image. Figure 17 When the additional learning button 120f of the operation mode user interface 120 is clicked, the additional learning image designation unit 32 generates Figure 18 The interruption screen 121 is displayed in a predetermined position on the operation mode user interface 120. Pressing the additional learning button 120f indicates that the inspection target image is designated for additional learning as a non-defective product image or a defective product image. Upon receiving this designation, the pass / fail determination unit 31 temporarily suspends the pass / fail determination process, i.e., the operation mode.

[0163] The interruption screen 121 includes an image display area 121a displaying an image to be additionally learned, a message display area 121b, a cancel button 121c, and an OK button 121d. The image displayed in the image display area 121a is Figure 17 The current inspection object image is displayed in the image display area 120a of the operation mode user interface 120 shown in FIG. The message display area 121b displays a message for notifying the user of the temporary suspension of the operation mode. That is, upon receiving the designation of an additional learning image, the display device 4 can notify the user of the temporary suspension of the operation mode. The notification form described above is not limited to a message, and other forms may be used.

[0164] The cancel button 121c cancels the additional learning. When the cancel button 121c is detected to be pressed, the additional learning image designation unit 32 deletes the interruption screen 121 without learning the image displayed in the image display area 121a, and Figure 17 The illustrated operating mode user interface 120 is displayed on the display device 4 .

[0165] The OK button 121d is used to execute additional learning. Upon detecting that the OK button 121d has been pressed, the additional learning image designation unit 32 instructs the discriminator generation unit 30 to learn the image displayed in the image display area 121a.

[0166] (Additional learning process)

[0167] Figure 3The discriminator generation unit 30 shown adds the inspection object image specified by the additional learning image specifying unit 32 as an additional learning object as a non-defective product image or a defective product image, and updates the discriminator. When it is detected that the additional learning image specifying unit 32 has specified an image of an additional learning object, the discriminator generation unit 30 generates Figure 19 the additional learning user interface 122 shown, and displays this interface on the display device 4. The additional learning user interface 122 includes an image display area 122a that displays the inspection object image specified as the additional learning object, a first inspection window learning instruction area 122b, a second inspection window learning instruction area 122c, a cancel button 122d, and an additional learning start button 122e. The first inspection window learning instruction area 122b is an area for specifying whether to add the images within the range defined in the learning inspection window 200A as non-defective product images or defective product images for additional learning. When "NG" is selected, the images within the range defined in the learning inspection window 200A are learned as defective product images. When "OK" is selected, the images within the range defined in the learning inspection window 200A are learned as non-defective product images. Additionally, the second inspection window learning instruction area 122c is an area for specifying whether to add the images within the range defined in the learning inspection window 200B as non-defective product images or defective product images for additional learning. When "NG" is selected, the images within the range defined in the learning inspection window 200B are learned as defective product images. When "OK" is selected, the images within the range defined in the learning inspection window 200B are learned as non-defective product images. As described above, the images within the range defined in the learning inspection window 200A and the images within the range defined in the learning inspection window 200B can be learned separately. It is also possible to receive an instruction for learning the images within the range defined in the learning inspection window 200A or the images within the range defined in the learning inspection window 200B.

[0168] When it is detected that the cancel button 122d has been pressed, since the discriminator generation unit 30 deletes Figure 19 the additional learning user interface 122 shown without performing additional learning and returns to Figure 17 the operation mode user interface 120 shown, the discriminator is not updated. On the other hand, when it is detected that the additional learning start button 122e (the start operation of additional learning) has been pressed, the discriminator generation unit 30 starts additional learning and updates the discriminator. Performing additional learning, for example, updates Figure 35 the discrimination boundary in the eigenvalue space shown.

[0169] (Storage of historical images)

[0170] Figure 2The storage device 19 shown will utilize Figure 3 A plurality of inspection target images that have undergone pass / fail judgment by the pass / fail judgment unit 31 shown, together with the pass / fail judgment results, are stored as history images. The inspection target images that have undergone pass / fail judgment by the pass / fail judgment unit 31 are stored in the operation image history folder. For example, Figure 17 The inspection target image displayed in the image display area 120a of the operation mode user interface 120 shown and the comprehensive judgment result displayed in the comprehensive judgment result display area 120b can be stored in the storage device 19 in a state where the image and the result are associated with a common ID number or the like. At this time, the inspection target image displayed in the image display area 120a, the result displayed in the position compensation result display area 120c, the result displayed in the first inspection window result display area 120d, and the result displayed in the second inspection window result display area 120e can be stored in the storage device 19 in a state where these four are associated with each other.

[0171] In addition, the storage device 19 may be provided with a learning image history folder. The learning image history folder stores non-defective product images and defective product images used for the discriminator generation unit 30 to perform learning.

[0172] As Figure 20 shown, a menu can be displayed at the lower part of the operation mode user interface 120. The menu of the operation mode user interface 120 includes an operation / learning image history button 120h. When it is detected that the operation / learning image history button 120h is pressed, as Figure 21 shown, the control unit 2 generates a history type selection interruption screen 123 and displays this screen at a predetermined position of the operation mode user interface 120. The history type selection interruption screen 123 includes an operation image history button 123a, a learning image history button 123b, and a cancel button 123c. Operating the operation image history button 123a is to display the inspection target images that have undergone pass / fail judgment stored in the operation image history folder of the storage device 19. When it is detected that the operation image history button 123a is pressed, the control unit 2 generates Figure 22 the operation image history display user interface 124 shown and displays this interface in the display device 4.

[0173] The operation image history display user interface 124 includes an image display area 124a that lists a plurality of inspection object images for pass / fail determination, an additional learning specification button 124b, a return button 124c, an image details button 124d, and a learning candidate image button 124e. The inspection object images and the results of pass / fail determination are stored in the image display area 124a in a state where these images are associated with these results. The images stored in the image display area 124a can be selected, for example, through a user's selection operation. When it is detected that the image details button 124d is pressed while any image in the images displayed in the image display area 124a is selected, the control unit 2 generates Figure 23 the selected image display user interface 125 shown, and displays this interface on the display device 4. The selected image display user interface 125 includes an image display area 125a that displays the image selected through the operation image history display user interface 124, a position compensation result display area 125c, a first inspection window result display area 125d, a second inspection window result display area 125e, an additional learning execution button 125f, and a return button 125g. When it is detected that the additional learning execution button 125f is pressed, the control unit 2 instructs the discriminator generation unit 30 to additionally learn the image displayed in the image display area 125a. When it is detected that the return button 125g is pressed, the control unit 2 will Figure 22 the operation image history display user interface 124 shown be displayed on the display device 4. By repeating this operation, the discriminator generation unit 30 can additionally learn a plurality of historical images.

[0174] When it is detected that the additional learning specification button 124b is pressed, the control unit 2 generates Figure 24 the additional learning user interface 122 shown, and displays this interface on the display device 4. Figure 25 The additional learning user interface 122 shown is different in terms of additional learning images from Figure 19 the additional learning user interface 122 shown.

[0175] When it is detected that the Figure 22 return button 124c of the operation image history display user interface 124 shown is pressed, the control unit 2 will Figure 21 the operation mode user interface 120 shown be displayed on the display device 4. Operate the learning image history button 123b of the operation history type selection interruption screen 123 to display the images used for learning stored in the learning image history folder of the storage device 19. When it is detected that the learning image history button 123b of the operation history type selection interruption screen 123 is pressed, the control unit 2 generates Figure 24The learning image history shown displays the user interface 126 in the display device 4. The learning image history display user interface 126 includes an image display area 126a that lists the images used for learning, a delete button 126b, and a return button 126c. The images displayed in the image display area 126a can be selected, for example, by a user's selection operation. When it is detected that the delete button 126b has been pressed while any of the images displayed in the image display area 126a is selected, the control unit 2 deletes the selected image. When it is detected that a learning image has been deleted, the control unit 2 causes the discriminator generation unit 30 to update the discriminator to prevent the image from being reflected in the learning result. When it is detected that the return button 126c has been pressed, the control unit 2 will Figure 21 the operation mode user interface 120 shown is displayed in the display device 4.

[0176] (Structure of the stability evaluation calculation unit 33)

[0177] Figure 3 The stability evaluation calculation unit 33 shown calculates a stability evaluation value indicating the stability of the pass / fail judgment of a plurality of historical images stored in the storage device 19. The stability evaluation calculation unit 33 can calculate the proximity between a judgment value indicating the similarity to a non-defective product or a defective product and a threshold value for distinguishing between a non-defective product and a defective product, and use this proximity as the stability evaluation value. This stability evaluation value can be displayed in the display device 4 in a form that is easy for the user to understand (for example, by a value), or can be held in the image inspection device 1 without being displayed.

[0178] Since the stability evaluation value indicates the stability of the pass / fail judgment of historical images, the stability evaluation value can be used to determine whether the stability of the pass / fail judgment using the generated discriminator is high or low. In the case where the stability of the pass / fail judgment is high, since historical images can be stably discriminated, it is determined that additional learning images are less needed. On the other hand, in the case where the stability of the pass / fail judgment is low, since historical images cannot be stably discriminated, it is determined that additional learning images are more needed to update the discriminator.

[0179] (Presentation of additional learning candidates)

[0180] The display device 4 can present candidate images for additional learning of the discriminator to the user based on the stability evaluation value calculated by the stability evaluation calculation unit 33. That is, when it is detected that the learning candidate image button 124e of the operation image history display user interface 124 shown has been pressed, as Figure 22 shown, Figure 26As shown, the control unit 2 generates an interruption screen 127 for presenting candidate images to the user and displays the screen at a predetermined position of the operation image history display user interface 124. The interruption screen 127 includes an image display area 127a for displaying candidate images, a cancel button 127b, and an OK button 127c. The image displayed in the image display area 127a is an image that is judged by the generated discriminator to be discriminated unstably based on the stability evaluation value. Specifically, the image can be an image with a high proximity between the judgment value indicating the similarity to a non-defective product or a defective product and the threshold value for distinguishing a non-defective product from a defective product, and an image with a proximity equal to or greater than a predetermined value can be presented to the user as a candidate image by the display device 1.

[0181] The image display area 127a can display the image that is discriminated most unstably by the generated discriminator. In addition, the image display area 127a can also display a plurality of candidate images including the image that is discriminated most unstably by the generated discriminator and the image that is discriminated second most unstably.

[0182] In addition, the display device 4 can present a historical image with a stability evaluation value less than a first predetermined value to the user. Since the discriminator cannot be updated by using a stably discriminated image as a candidate image and additional learning is meaningless, the above first predetermined value can be set to a value that can almost update the discriminator by causing the discriminator generation unit 30 to perform additional learning with this value.

[0183] The display device 4 can adopt a display form for presenting a plurality of candidate images to the user at one time. In this case, a list of candidate images can be presented in ascending or descending order of the stability evaluation value. The specific process will be described later.

[0184] In the case of presenting a plurality of candidate images by the display device 4, the user can receive a designation as an additional learning image via Figure 3 the additional learning image designation unit 32 shown. When the user selects one candidate image from a plurality of candidate images, the additional learning image designation unit 32 receives the designated candidate image as an additional learning image, receives a designation as a non-defective product image or a defective product image, and causes the discriminator generation unit 30 to perform additional learning with the candidate image. The discriminator generation unit 30 performs additional learning on the additional learning image designated by the user as a non-defective product image or a defective product image, and updates the discriminator.

[0185] The display device 4 can switch between a first display form for displaying a plurality of historical images in chronological order and a second display form for presenting candidate images for additional learning to the user. The above first display form is in Figure 22The operation image history shown is implemented in the image display area 124a of the user interface 124. In the image display area 124a, the inspection object images can be displayed in the order in which these inspection object images are taken, that is, in the order in which pass / fail judgments are made. In addition, the above-described second display form is implemented on Figure 26 the interruption screen 127 shown.

[0186] (During operation in the standard inspection mode)

[0187] Next, the Figure 27 flowchart shown will be used to explain the process in the operation mode of the standard inspection mode. In step SB1 after the start, an inspection object image is input. The inspection object image is taken by the imaging unit 3 at a predetermined timing and input from the imaging unit 3 to the pass / fail judgment unit 31.

[0188] Step SB2 is a pass / fail judgment step that inputs the inspection object image newly taken by the imaging unit 3 in step SB1 to the discriminator and makes a pass / fail judgment. The user judges whether it is necessary to adjust the judgment threshold based on the judgment result in step SB2. This is step SB3. When it is necessary to adjust the threshold, the process proceeds to step SB4 to adjust the threshold, and the process proceeds to step SB5. On the other hand, when it is not necessary to adjust the threshold, the process proceeds to step SB5. In step SB5, the judged inspection object image is associated with the judgment result and stored in the storage device 19.

[0189] (Structure of the judgment axis extraction unit 34)

[0190] Next, the Figure 3 judgment axis extraction unit 34 shown will be described. The judgment axis extraction unit 34 extracts a judgment axis indicating the similarity to a non-defective product or the similarity to a defective product based on the non-defective product image and the defective product image registered in the learning image registration unit 24. The judgment axis includes a first judgment axis indicating the similarity to a non-defective product in the non-defective product image and a second judgment axis indicating the similarity to a defective product in the defective product image, but only one of the two can be extracted. Although specific examples of the judgment axis are, for example, color, brightness, and area, etc., the judgment axis can also be other examples.

[0191] An object image can be discriminated by measuring the object image using a specific judgment axis and comparing a preset reference value with the measurement result. For example, when making a judgment using color, the color set as the judgment axis is displayed on the display device 4. Alternatively, when making a judgment using area, the pixels used to calculate the area as the judgment axis are displayed on the display device 4. This enables the user to grasp the judgment axis used by the image inspection device 1 to discriminate images in the learning inspection mode.

[0192] When the discriminator generation unit 30 updates the discriminator by additional learning or the like, the determination axis extraction unit 34 extracts the determination axis again. That is, since the determination axis indicating the similarity to non-defective products or the similarity to defective products can be changed when updating the discriminator, the determination axis extraction unit 34 extracts the determination axis when detecting that the discriminator generation unit 30 has updated the discriminator. This makes the updated discriminator consistent with the determination axis.

[0193] The control unit 2 calculates a determination evaluation value indicating the similarity to non-defective products or the similarity to defective products based on the determination axis extracted by the determination axis extraction unit 34. For example, the determination evaluation value can be calculated to increase when the difference in color from the determination axis set as a non-defective product becomes smaller. On the other hand, the determination evaluation value can be calculated to decrease when the difference in color from the determination axis set as a non-defective product becomes larger. Although the determination evaluation value includes a first determination evaluation value indicating the similarity to non-defective products calculated based on the first determination axis and a second determination evaluation value indicating the similarity to defective products calculated based on the second determination axis, only one of these two values can be calculated.

[0194] The control unit 2 generates an inspection object image that selectively enhances regions having characteristics of non-defective products or defective products, and displays the inspection object image on the display device 4. In the case where the first determination evaluation value and the second determination evaluation value are calculated, the first determination evaluation value and the second determination evaluation value can be used to display on the display device 4 an inspection object image that selectively enhances regions having characteristics of non-defective products and an inspection object image that selectively enhances regions having characteristics of defective products.

[0195] In addition, in the case where the first determination evaluation value and the second determination evaluation value are calculated, as a comparison result between the first determination evaluation value and the second determination evaluation value, when the inspection object image has more characteristics of non-defective products, the control unit 2 displays an inspection object image that selectively enhances regions having characteristics of non-defective products. On the other hand, when the inspection object image has more characteristics of defective products, the control unit 2 generates an inspection object image that selectively enhances regions having characteristics of defective products. Then, the inspection object image is displayed on the display device 4.

[0196] Here, "enhancement" means the difference between a part of an image and other parts, and can be achieved, for example, by coloring only a specific area having characteristics of a non-defective product or characteristics of a defective product, and an image in which the specific area has been colored can be displayed on the display device 4. In particular, when the determination axis extraction unit 34 extracts color as the determination axis, enhancement can be achieved by coloring an area having characteristics of a defective product with the color of the defective product, or coloring an area having characteristics of a non-defective product with the color of the non-defective product. In addition, the color used for the area having characteristics of a defective product can be different from the color used for the area having characteristics of a non-defective product. In addition, the color of the area having characteristics of a non-defective product or characteristics of a defective product can be changed according to the magnitude of the determination evaluation value. The color of an area with a larger determination evaluation value can be deeper, and the color of an area with a smaller determination evaluation value can be lighter.

[0197] Display examples will be described below. Figure 28A 、 28B and 28C show examples of inspection target images including a part like a non-defective product and a part like a defective product, Figure 28A shows a case where the probability of being a non-defective product image is high, Figure 28B shows a case where the probability of being a non-defective product image is about half, and Figure 28C shows a case where the probability of being a defective product image is high. In these figures, the gray part indicated by reference numeral 400 is judged to be like a non-defective product, and the black part indicated by reference numeral 401 is judged to be like a defective product.

[0198] Figure 28A - 28C The upper figure of Figure 28A - 28C is the inspection target image, and Figure 28A the lower figure of

[0199] In Figure 28BIn the case where the area enclosed by the frame 403 includes the gray part 400 and the black part 401, and the area of the gray part 400 is approximately the same as the area of the black part 401, it is difficult to determine whether this area resembles a non-defective product or a defective product. Therefore, the probability of it being an image of a non-defective product is close to 50%. In this case, the area enclosed by the frame 403 is not colored. That is, this area is not enhanced.

[0200] In Figure 28C the case where the area enclosed by the frame 404 is the black part 401, this area is determined to resemble a defective product. Therefore, the probability of it being an image of a non-defective product is a low value such as a very small percentage. The area enclosed by the frame 403 has the color (black) of a defective product and is enhanced.

[0201] Figure 29A and 29B are examples of inspection object images including characters, Figure 29A show non-defective product images, and Figure 29B show inspection object images in which areas are selectively enhanced. In the case where the same characters as those in the non-defective product image also exist in the inspection object image as shown in Figure 29B , the areas including these characters are also determined to resemble non-defective products. The frame 405 encloses the area including the characters, and the area enclosed by the frame 405 is colored for enhancement.

[0202] Figure 30A , 30B , 30C and 30D show other examples of the inspection object W, Figure 30A show non-defective product images, Figure 30B show defective product images, Figure 30C show inspection object images in the case of photographing a non-defective product in the running mode, and Figure 30D show inspection object images in the case of photographing a defective product in the running mode. Although Figure 30A 's non-defective product image includes white circles and characters, Figure 30B 's defective product image does not include white circles and characters. In this case, the judgment axes indicating the similarity to the non-defective product are "existence of white circles" and "existence of characters", and the judgment axes indicating the similarity to the defective product are "non-existence of white circles" and "non-existence of characters". In the case of photographing a non-defective product in the running mode as shown in Figure 30C , the inside of the frame 406 enclosing the non-defective product is colored dark to enhance this area. On the other hand, as shown in Figure 30D , in the case of photographing a defective product in the running mode, the inside of the frame 407 enclosing the defective product is colored white to enhance this area.

[0203] Figure 31A and 31B shows the case where additional learning is performed in the cases shown in Figure 30A , 30B , 30C and 30D, and Figure 31A shows the defective product images used for the additional learning and Figure 31B shows the inspection object image in the case where a defective product is photographed in the operation mode after the additional learning. As Figure 31A shown, the defective product image used for the additional learning includes a white circle in the right part of the inspection object W. That is, the judgment axis indicating the similarity to the defective product is "there is a white circle in the right part". As Figure 31B shown, in the case where a defective product is photographed in the operation mode, the inside of the frame 408 surrounding the defective product is colored white to enhance the area. It should be noted here that the color of the enhanced area is not particularly limited.

[0204] (During the operation in the learning inspection mode)

[0205] Next, the process in the operation mode of the learning inspection mode will be described with reference to the flowchart shown in Figure 32 . In step SC1 after the start, an inspection object image is input. The inspection object image is photographed by the imaging unit 3 at a predetermined timing and input from the imaging unit 3 to the pass / fail judgment unit 31.

[0206] Step SC2 is a pass / fail judgment step that inputs the inspection object image newly photographed by the imaging unit 3 in step SC1 to the discriminator and performs a pass / fail judgment. In this step, the judgment axis extraction unit 34 extracts the judgment axis indicating the similarity to a non-defective product or the similarity to a defective product. The next step SC3 is a display step that is used to display the inspection object image that has been subjected to the pass / fail judgment in the pass / fail judgment step together with the pass / fail judgment result obtained in the pass / fail judgment step on the Figure 17 operation mode user interface 120 shown in

[0207] In step SC4, it is judged whether additional learning of the image displayed on the operation mode user interface 120 is specified. Step SC4 is an additional learning setting step that is used to receive the specification regarding whether the inspection object image displayed on the operation mode user interface 120 in the display step is to be additionally learned as a non-defective product image or a defective product image.

[0208] In the case of specifying additional learning, the process proceeds to step SC5, and additional learning is performed using the image. That is, in step SC5, the inspection object image specified as the additional learning object in the additional learning setting step (step SC4) is additionally learned as a non-defective product image or a defective product image, and the discriminator is updated. After that, in step SC6, the image together with the pass / fail judgment result is stored in the storage device 19 as a history image. On the other hand, in the case where additional learning is not specified in step SC4, the process proceeds to step SC6 without performing additional processing. Step SC6 is a storage step.

[0209] (Process for presenting learning candidate images)

[0210] Figure 33 is a flowchart in the case of presenting learning candidate images. Step SD1 after the start is a stability evaluation value calculation step. Specifically, this step calculates the proximity between the following judgment value and the threshold for distinguishing non-defective products from defective products, and the judgment value represents the similarity to non-defective products or the similarity to defective products of the inspection object image for which pass / fail judgment is made in Figure 32 step SC2 (pass / fail judgment step) of the flowchart described above. After that, the proximity is used as the stability evaluation value indicating the stability of the pass / fail judgment of the inspection object image. In addition, the stability evaluation value calculation step can also calculate the stability evaluation value indicating the stability of the pass / fail judgment of the multiple history images stored in Figure 32 step SC6 (storage step) of the flowchart shown above.

[0211] In step SD2, based on the stability evaluation value calculated in step SD1, candidate images for additional learning by the discriminator are presented to the user. This is a display step. For example, Figure 26 the operation image history display user interface 124 shown above is displayed on the display device 4.

[0212] In step SD3, it is judged whether additional learning of the candidate image is specified. Step SD3 is an additional learning setting step, and this additional learning setting step is used to receive a specification regarding whether the inspection object image displayed on the operation image history display user interface 124 in the display step is additionally learned as a non-defective product image or a defective product image.

[0213] When additional learning is specified, the process proceeds to step SD4, and additional learning is performed using the candidate image. That is, in step SD4, the additional learning image specified as the additional learning object in the additional learning setting step (step SD3) is additionally learned as a non-defective product image or a defective product image, and the discriminator is updated. On the other hand, when additional learning is not specified in step SD3, no additional learning is performed.

[0214] (Sorting of historical images)

[0215] The historical images to be presented to the user as additional learning candidates can be sorted. In step SE1 after the start, the stability evaluation calculation unit 33 calculates the difference between the judgment value indicating the similarity to a non-defective product or a defective product and the threshold value for distinguishing between a non-defective product and a defective product for each historical image. This can obtain the proximity between the judgment value and the threshold value. This difference can be calculated for all historical images stored in the storage device 19 or for the latest multiple historical images.

[0216] In step SE2, the control unit 2 sorts the historical images in ascending order of the difference calculated in step SE1. At this time, all the historical images for which the difference has been calculated can be sorted, or only multiple historical images with a difference less than a predetermined value can be sorted.

[0217] In step SE3, the control unit 2 causes the display device 4 to display the sorting result obtained in step SE2. The control unit 2 can generate a user interface for listing and displaying multiple historical images in ascending order of the difference. The number of images displayed on this user interface can be, for example, 5 to 10, and these images can be displayed in multiple rows. Since it is judged that it is unstable when the difference is small and such images are additional learning candidate images, multiple learning candidate images can be presented to the user.

[0218] In step SE4, it is judged whether additional learning of the candidate image is specified. Step SE4 is an additional learning setting step for receiving a specification regarding whether to additionally learn the inspection object image to be displayed on the operation image history display user interface 124 in the display step as a non-defective product image or a defective product image. When additional learning is specified, the process proceeds to step SE5, and additional learning is performed using the candidate image. That is, in step SE5, the additional learning image specified as the additional learning object in the additional learning setting step (step SE4) is additionally learned as a non-defective product image or a defective product image, and the discriminator is updated. On the other hand, when additional learning is not specified in step SE4, no additional learning is performed.

[0219] (Deletion candidate display)

[0220] The display device 4 presents the historical images whose stability evaluation values are equal to or greater than a predetermined value to the user as deletion candidate images to be deleted from the storage device 19. The predetermined value may be a second predetermined value greater than the first predetermined value, or may be equal to the first predetermined value.

[0221] That is, since the images stably discriminated by the generated discriminator as described above are judged to have high stability evaluation values and require less additional learning, the necessity of storing such images in the storage device 19 is very small, and the memory does not have to be used. The memory of the storage device 19 can be saved by deleting such images from the storage device 19.

[0222] Figure 35 It is the following schematic diagram: The non-defective product image group and the defective product image group registered by the learning image registration unit 24 are plotted in a specific eigenvalue space, and the discrimination boundary between the non-defective product images and the defective product images of the discriminator generated by the discriminator generation unit 30 is shown. In this figure, the non-defective product image group is plotted in the white area, and the defective product image group is plotted in the black area.

[0223] Since the non-defective product images surrounded by the surrounding line 300 in the non-defective product image group are far from the discrimination boundary, even if the non-defective product images surrounded by the surrounding line 300 are additionally learned, the discrimination boundary is not updated, and the learning is almost meaningless. The display device 4 presents the non-defective product images surrounded by the surrounding line 300 to the user as deletion candidate images. In addition, since the defective product images surrounded by the surrounding line 301 in the defective product image group are also far from the discrimination boundary, even if the defective product images surrounded by the surrounding line 301 are additionally learned, the discrimination boundary is not updated, and the learning is almost meaningless. The display device 4 presents the defective product images surrounded by the surrounding line 301 to the user as deletion candidate images. That is, the historical images that are more than a predetermined distance away from the discrimination boundary between the non-defective product images and the defective product images using the discriminator can be presented to the user.

[0224] Refer to Figure 36 The flowchart shown will be used to explain the process for presenting the deletion candidate images. In step SF1 after the start, the discriminator generation unit 30 calculates the eigenvalue of the image (historical image) used for learning. After calculating the eigenvalue, the process proceeds to step SF2, and the historical images are plotted in the eigenvalue space as Figure 35 shown. After that, the process proceeds to step SF3, and the control unit 2 grasps the position of the historical images in the eigenvalue space. This can grasp the positional relationship between the discrimination boundary and each historical image, and separate the historical images close to the discrimination boundary from the historical images that are more than a predetermined distance away from the discrimination boundary.

[0225] In step SF4, historical images that do not contribute to the setting of the discrimination boundary in step SF3 are selected as deletion candidate images. Historical images that do not contribute to the setting of the discrimination boundary are historical images that are at a distance greater than a predetermined distance from the discrimination boundary. In step SF5, the control unit 2 displays the deletion candidate images selected in step SF4 on the display device 4. This can present the deletion candidate images to the user. One deletion candidate image or multiple deletion candidate images can be presented to the user at one time.

[0226] In step SF6, it is determined whether the user has performed a deletion operation on the deletion candidate image. In the case where a deletion operation has been performed, in step SF7, the deletion candidate image is deleted. In the case where no deletion operation has been performed, the process ends without deleting the deletion candidate image. The deletion operation can be performed on one deletion candidate image or on multiple deletion candidate images.

[0227] Figure 37 and Figure 38 shows the case of changing the discrimination boundary through additional learning. Figure 37 Steps SG1 to SG4 in the shown flowchart are respectively the same as Figure 36 steps SF1 to SF4 in the shown flowchart. In Figure 37 step SG5 of the shown flowchart, in the case of changing the discrimination boundary through additional learning as described above, the deletion candidate images for the new discrimination boundary are calculated again.

[0228] For example, when the discrimination boundary in the eigenvalue space shown in the upper part of Figure 38 is changed to the discrimination boundary in the eigenvalue space shown in the lower part of Figure 38 through additional learning, the distances between the discrimination boundary and the non-defective product images and between the discrimination boundary and the defective product images are calculated again. After that, the process proceeds to step SG6, and as new deletion candidate images, historical images (non-defective product images surrounded by the bounding line 302 and defective product images surrounded by the bounding line 303) that are at a distance greater than a predetermined distance from the new discrimination boundary are selected as deletion candidate images. Then, the process proceeds to step SG7, and the non-defective product images surrounded by the bounding line 302 and the defective product images surrounded by the bounding line 303 are displayed as deletion candidate images on the display device 4, and the newly selected deletion candidate images are presented to the user. Steps SG8 and SG9 are respectively the same as Figure 36 steps SF6 and SF7 in the shown flowchart.

[0229] Figure 39 is a flowchart showing the process of presenting deletion candidate images based on the relative relationship between deletion candidate images and adjacent images. Steps SH1 to SH4 are respectively the same asFigure 36 The steps SF1 to SF4 in the flowchart shown are the same. After that, in step SH5, the relative relationship between the deletion candidate image and the adjacent image is calculated. Then, in step SH6, an image that has less influence on the setting of the discrimination boundary in future additional learning is selected as the deletion candidate image.

[0230] That is, this step presents, to the user, as the deletion candidate image, an image that is judged not to contribute to the setting of the discrimination boundary, based on the relative relationship between a historical image that is separated from the discrimination boundary by a predetermined distance or more in the eigenvalue space and an adjacent image plotted adjacent to the historical image. Specifically, in Figure 40 the case of the eigenvalue space shown in the upper part, the non-defective product image surrounded by the enclosing line 300 and the defective product image surrounded by the enclosing line 301 can be deletion candidate images. Among these images, all the non-defective product images surrounded by the enclosing line 300 are separated from the discrimination boundary by a predetermined distance or more, so all the images can be deletion candidate images. However, focusing on Figure 40 the image surrounded by the enclosing line 304 in the lower part image of Figure 36 based on the relative relationship between this image and the images around it, this image is judged not to contribute to the setting of the discrimination boundary. That is, since the historical image is positioned to surround the image surrounded by the enclosing line 304, even when the image surrounded by the enclosing line 304 is deleted, the current discrimination boundary is not affected, and the discrimination boundary to be generated in the future is hardly affected. Therefore, by executing step SH7, an image that hardly affects the setting of the discrimination boundary even when the image is deleted is presented to the user as the deletion candidate image. Steps SH8 and SH9 are the same as steps SF6 and SF7 in the flowchart shown in respectively.

[0231] (Operations and effects of the embodiment)

[0232] As described above, according to this embodiment, by implementing the standard inspection mode and the learning inspection mode in the image inspection device 1 and switching between these inspection modes, inspection can be performed in any inspection mode, and inspection can be performed by taking advantage of the strengths of each inspection mode.

[0233] When capturing a main image in the setting mode of the standard inspection mode, imaging conditions can be set and an image processing tool can be selected. Additionally, the range to which the image processing tool is to be applied can be defined through an inspection window, and the parameters of the image processing tool can be adjusted. That is to say, since the user can set the imaging conditions, the type of the image processing tool, the application range of the image processing tool, and adjust the parameters of the image processing tool in the standard inspection mode, the standard inspection mode can be executed by controlling the pass / fail judgment criteria to obtain a desired result. For example, in the standard inspection mode, pass / fail judgment of the inspection object in the image can be performed based on various characteristic values (such as color, edge, and position, etc.) of the inspection object.

[0234] On the other hand, in the setting mode of the learning inspection mode, a non-defective product image with the attributes of a non-defective product given by the user and a defective product image with the attributes of a defective product given by the user can be registered, and a discriminator for distinguishing the non-defective product image from the defective product image is generated by learning the registered non-defective product image and the registered defective product image. That is to say, in the setting mode of the learning inspection mode, the settings and adjustments that the user makes in the standard inspection mode are not required.

[0235] In addition, a plurality of inspection object images that have undergone pass / fail judgment using the discriminator are stored together with the results of the pass / fail judgment as history images in the storage device 19, so that a stability evaluation value indicating the stability of the pass / fail judgment of the plurality of history images can be calculated. Based on the stability evaluation value, candidate images for additional learning of the discriminator can be presented to the user.

[0236] Therefore, even if the number of history images stored in the storage device 19 is extremely large, the user can select additional learning images from the presented candidate images based on the stability of the pass / fail judgment. When receiving the designation of an additional learning image by the user, the discriminator can be updated by performing additional learning on the additional learning image designated by the user as a non-defective product image or a defective product image.

[0237] In addition, a judgment axis indicating the similarity to a non-defective product or a defective product can be extracted based on the non-defective product image or the defective product image, and based on this judgment axis, a judgment evaluation value indicating the similarity to a non-defective product or a defective product can be calculated. Since an inspection object image in which regions having characteristics of a non-defective product or a defective product are selectively enhanced due to the judgment evaluation value can be displayed, the user can grasp the judgment axis used by the image inspection device for learning inspection to distinguish images.

[0238] In addition, when switching from the setting mode to the operation mode, the newly captured inspection object image is input to the discriminator, and a pass / fail determination is made. Since both the inspection object image and the result of the pass / fail determination are displayed on the operation mode screen of the display device 4, the user can grasp the inspection object image and the result of the pass / fail determination of the image. When the result of the pass / fail determination of the inspection object image is correct, the operation mode continues. When the result of the pass / fail determination of the inspection object image is incorrect, additional learning can be performed. Therefore, without having to switch from the operation mode to the setting mode, the discriminator can be updated by specifying the image to be additionally learned and performing additional learning on the image in the operation mode.

[0239] The above-described embodiments are merely examples in all respects and should not be construed in a limiting manner. Modifications and changes within the scope equivalent to the appended claims are therefore intended to be included.

[0240] As described above, the image inspection device according to the present invention can be used to make a pass / fail determination of an inspection object based on an image obtained by photographing the inspection object.

Claims

1. An image inspection device for performing pass / fail judgment on an inspection object based on an image of the inspection object captured by an imaging unit, the image inspection device comprising: A mode switching unit for switching the mode of the image inspection device between a setting mode for setting the image inspection device and an operation mode for performing inspection; An imaging setting unit for receiving setting of imaging conditions of the imaging unit in the setting mode; A learning image registration unit for registering a non-defective product image with attributes of a non-defective product given by a user and a defective product image with attributes of a defective product given by the user in the setting mode; A discriminator generation unit for learning the non-defective product image and the defective product image registered by the learning image registration unit in the setting mode and generating a discriminator for distinguishing the non-defective product image from the defective product image; A pass / fail judgment unit for inputting a newly captured image of the inspection object by the imaging unit into the discriminator and performing pass / fail judgment in the operation mode; A display unit for displaying the inspection object image together with the result of pass / fail judgment performed by the pass / fail judgment unit on an operation mode screen in the operation mode; And An additional learning image specifying unit for receiving a specification regarding whether to additionally learn the inspection object image displayed on the operation mode screen as the non-defective product image or the defective product image, wherein the discriminator generation unit additionally learns the inspection object image specified as an additional learning object by the additional learning image specifying unit as the non-defective product image or the defective product image and updates the discriminator.

2. The image inspection device according to claim 1, Among them, wherein the discriminator generation unit generates a discrimination boundary between the non-defective product image and the defective product image in a feature value space by inputting the non-defective product image and the defective product image into a pre-learned neural network having a plurality of layers, and updates the discrimination boundary by the additional learning.

3. The image inspection device according to claim 1, further comprising: A main image registration unit for registering an image captured under the imaging conditions set by the imaging setting unit as a main image in the setting mode; And An inspection window setting unit for receiving setting of a first learning-based inspection window and a second learning-based inspection window for defining a range of difference detection on the main image registered by the main image registration unit in the setting mode, wherein the operation mode screen displays the result of pass / fail judgment performed by the pass / fail judgment unit on an image within the range defined by the first learning-based inspection window, and the result of pass / fail judgment performed by the pass / fail judgment unit on an image within the range defined by the second learning-based inspection window, and The additional learning image specifying unit receives a specification regarding whether to additionally learn, in a separated manner, an image within a range defined by the first learning-based inspection window and an image within a range defined by the second learning-based inspection window.

4. The image inspection device according to claim 3, Among them, after receiving the specification regarding whether to additionally learn, in a separated manner, an image within a range defined by the first learning-based inspection window and an image within a range defined by the second learning-based inspection window, the additional learning image specifying unit receives an operation performed by the user to start the additional learning.

5. The image inspection device according to claim 3, Among them, after receiving the specification to additionally learn the inspection target image displayed on the operation mode screen as the non-defective product image or the defective product image, the additional learning image specifying unit receives a specification regarding whether to additionally learn, in a separated manner, an image within a range defined by the first learning-based inspection window and an image within a range defined by the second learning-based inspection window.

6. The image inspection device according to claim 1, wherein when the additional learning image specifying unit receives the specification to additionally learn the inspection target image displayed on the operation mode screen as the non-defective product image or the defective product image, the pass / fail determination unit temporarily stops the operation mode, and when the additional learning image specifying unit receives the specification to additionally learn the inspection target image displayed on the operation mode screen, the display unit can notify the user that the pass / fail determination temporarily stops the operation mode.

7. A setting method for an image inspection device, the image inspection device being configured to perform pass / fail determination of an inspection target based on an image of the inspection target captured by an imaging unit, the setting method including: an imaging condition setting step of setting imaging conditions of the imaging unit in a state where the mode of the image inspection device is set to a setting mode for setting the image inspection device; a learning image registration step of registering, in the setting mode, a non-defective product image given the attribute of a non-defective product by the user and a defective product image given the attribute of a defective product by the user; a discriminator generation step of learning, in the setting mode, the non-defective product image and the defective product image registered in the learning image registration step and generating a discriminator for distinguishing the non-defective product image from the defective product image; a pass / fail determination step of inputting, in an operation mode for performing inspection, a newly captured image of the inspection target by the imaging unit into the discriminator and performing pass / fail determination; a display step of displaying, in the operation mode, the inspection target image together with the result of the pass / fail determination performed in the pass / fail determination step on an operation mode screen; and an additional learning setting step for receiving a designation regarding whether to additionally learn the inspection target image displayed on the operation mode screen in the display step as the non-defective product image or the defective product image wherein, in the discriminator generation step, the inspection target image designated as the additional learning target in the additional learning setting step is additionally learned as the non-defective product image or the defective product image, and the discriminator is updated.

Images