Image inspection device, image processing method, image processing program, computer-readable recording medium, and recorded device

Abstract

To facilitate construction of learning data.SOLUTION: An image inspection apparatus 100 includes: a learned neural network storage unit 19a storing one or more neural networks that previously learned weighting coefficients of layers; and an inference processing unit 20a which determines quality of a workpiece and classifies workpieces based on a work image. The inference processing unit 20a can execute first inference processing that determines quality of the work image based on quality feature quantities obtained by inputting an inspection work image to the neural network and a quality determination boundary, and second inference processing that defines a classification boundary for classifying workpieces on a feature quantity space of the neural network based on multiple classification feature quantities that represent the different types of work images, and classifies the work images based on classification feature quantities obtained by inputting an inspection work image generated by a camera unit 14 to the neural network and a classification boundary.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image inspection device, an image processing method, an image processing program, a computer-readable recording medium, and a device on which the program is recorded. [Background technology]

[0002] In recent years, models equipped with so-called AI learning functions have appeared as image sensors for determining whether a workpiece is good or bad. These models are equipped with a learning-based inspection mode that learns images during operation according to the user environment.

[0003] However, building a neural network to realize such a learning-based inspection mode requires a long time and a large amount of training data, which requires a considerable amount of time and effort, making it not necessarily user-friendly. Furthermore, if you want to perform a different type of inference, you need to switch the neural network itself. For example, applying it to a different production line requires new learning, which poses a problem of considerable time and effort. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-187070 Summary of the Invention [Problem to be solved by the invention]

[0005] An object of the present invention is to provide an image inspection device, an image processing method, an image processing program, a computer-readable recording medium, and a device on which the same is recorded, which facilitate the construction of learning data.

[0006] An image inspection device according to one aspect of the present invention has an illumination unit that irradiates illumination light onto a workpiece to be inspected, a camera unit that receives light irradiated from the illumination unit and reflected by the workpiece to generate a workpiece image, an input layer to which the workpiece image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting feature quantities of the input workpiece image, the image being equipped with a trained neural network storage unit that stores one or more neural networks in which weight coefficients of each layer have been trained in advance, and an inference processing unit that performs a quality judgment of the workpiece or a workpiece sorting judgment based on the workpiece image, and the inference processing unit calculates a quality of the workpiece or a quality of the workpiece on a feature quantity space of the neural network based on a plurality of quality features that characterize each workpiece image obtained by inputting a quality image indicating a quality workpiece and a defective image indicating a defective workpiece generated by the camera unit into a neural network stored in the trained neural network storage unit. and a second inference process in which a sorting boundary for classifying the workpieces is set in the feature space of the neural network based on a plurality of sorting features characterizing each workpiece image obtained by inputting a plurality of different types of workpiece images generated by the camera unit into the neural network stored in the trained neural network storage unit, and the workpiece images are sorted based on the sorting features and the sorting boundary obtained by inputting the workpiece images for inspection generated by the camera unit into the neural network stored in the trained neural network storage unit.With the above configuration, a trained neural network is prepared in advance, and the quality determination and sorting are performed using a common neural network, thereby eliminating the need to construct a neural network for each inference process and realizing a simplified quality determination and sorting processing environment.

[0007] Furthermore, an image inspection method according to another aspect of the present invention is an image inspection method which irradiates illumination light from an illumination unit onto a work to be inspected, receives the light reflected by the work with a camera unit, generates a work image, and performs a pass / fail judgment or a work sorting judgment, and includes a step of preparing one or more trained neural networks having an input layer to which the work image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting feature quantities of the input work image, the weight coefficients of each layer having been trained in advance, and storing the trained neural network in a memory unit; a step of capturing an image of a good work with the camera unit and generating a good image; a step of capturing an image of a defective work with the camera unit and generating a defective image; inputting the good image and the defective image into a neural network stored in the memory unit, obtaining a plurality of good product feature quantities and defective product feature quantities that characterize each of the good product image and the defective product image, respectively, and setting a pass / fail judgment boundary for judging the pass / fail of the work in a feature quantity space of the neural network based on the obtained plurality of good product feature quantities and defective product feature quantities. the step of setting a workpiece image for inspection using the camera unit; the step of capturing images of a plurality of different types of workpieces using the camera unit to generate type-specific workpiece images; the step of inputting the type-specific workpiece images into a neural network stored in the trained neural network storage unit to acquire a plurality of sorting features that characterize each type of workpiece image, and setting a sorting boundary for classifying the workpieces in the feature space of the neural network based on the acquired plurality of sorting features; the step of capturing an image of a workpiece for inspection using the camera unit to generate an inspection workpiece image; the step of inputting the inspection workpiece image into the neural network stored in the trained neural network storage unit to acquire pass / fail features; the step of making a pass / fail judgment of the workpiece image as a first inference process based on the acquired pass / fail features and the pass / fail judgment boundary; the step of inputting the inspection workpiece image into the neural network stored in the trained neural network storage unit to acquire sorting features; and the step of sorting the workpiece image as a second inference process based on the acquired sorting features and the sorting boundary.This allows a trained neural network to be prepared in advance, and pass / fail judgment and sorting to be performed using a common neural network, eliminating the need to construct a neural network for each inference process and realizing a simplified processing environment for pass / fail judgment and sorting.

[0008] Furthermore, an image inspection method according to another aspect of the present invention is an image inspection method in which an illumination unit irradiates a workpiece to be inspected with illumination light, a camera unit receives the light reflected by the workpiece, generates a workpiece image, and performs a pass / fail judgment or a workpiece sorting judgment, the image inspection method including a step of preparing one or more trained neural networks having an input layer to which the workpiece image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting feature quantities of the input workpiece image, the weight coefficients of each layer having been trained in advance, and storing the trained neural network in a trained neural network storage unit; a step of imaging a non-defective workpiece with the camera unit and inputting the generated non-defective image and an image of a defective workpiece with the camera unit and inputting the generated defective image into the neural network stored in the trained neural network storage unit; a step of imaging a plurality of different types of workpiece images with the camera unit, and inputting the generated type workpiece images into the trained neural network in a trained neural network storage unit; The method includes a step of inputting the data into a neural network stored in a neural network storage unit; a step of setting pass / fail judgment criteria for judging the pass / fail of a work in the feature space of the neural network based on a plurality of pass / fail feature quantities characterizing each pass / fail image and each fail / fail image, and a step of setting sorting criteria for classifying the work in the feature space of the neural network based on a plurality of sorting feature quantities characterizing each type of work image; a step of capturing an inspection work image with the camera unit and inputting the generated inspection work image into the neural network stored in the trained neural network storage unit; and a step of judging the pass / fail of the work image as a first inference process based on the inspection feature quantities characterizing the inspection work image and the pass / fail judgment criteria, and if the work image is judged to be pass / fail, sorting the work image as a second inference process based on the inspection feature quantities and the sorting criteria, and outputting the sorting result. This allows a trained neural network to be prepared in advance, and pass / fail judgment and sorting to be performed using a common neural network, eliminating the need to construct a neural network for each inference process and realizing a simplified processing environment for pass / fail judgment and sorting.

[0009] Furthermore, an image inspection program according to another aspect of the present invention is an image inspection program executed by a computer having an illumination unit that irradiates illumination light onto a workpiece to be inspected, a camera unit that receives light irradiated from the illumination unit and reflected by the workpiece, and generates a workpiece image, an input layer to which the workpiece image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting feature quantities of the input workpiece image, the weight coefficients of each layer being trained in advance, and a trained neural network storage unit that stores one or more neural networks, and an inference processing unit that executes a quality judgment or a workpiece sorting judgment based on the workpiece image, the image inspection program having a function of inputting a quality image generated by the camera unit to capture a quality workpiece and a defective workpiece and a defective image generated by the camera unit to a neural network stored in the trained neural network storage unit, and a function of inputting a quality image generated by the camera unit to capture a plurality of different types of workpiece images, a function of inputting an image of a workpiece to the neural network stored in the trained neural network storage unit; a function of setting pass / fail judgment criteria for judging the pass / fail of a workpiece in the feature space of the neural network based on a plurality of pass / fail feature quantities characterizing each of the pass / fail images and the defective feature quantities, and sorting criteria for classifying the workpieces in the feature space of the neural network based on a plurality of sorting feature quantities characterizing each type of workpiece image; a function of capturing an image of a workpiece for inspection with the camera unit and inputting the generated inspection workpiece image to the neural network stored in the trained neural network storage unit; and a function of judging the pass / fail of the workpiece image as a first inference process based on the inspection feature quantities characterizing the inspection workpiece image and the pass / fail judgment criteria, and if the workpiece is judged to be a pass / fail product, sorting the workpiece image as a second inference process based on the inspection feature quantities and the sorting criteria and outputting the sorting results.With the above configuration, a trained neural network is prepared in advance, and pass / fail judgment and sorting are performed using a common neural network, eliminating the need to construct a neural network for each inference process and realizing a simplified processing environment for pass / fail judgment and sorting.

[0010] Furthermore, according to another aspect of the present invention, a computer-readable recording medium or device storing the program stores the program. Examples of the recording medium include magnetic disks such as CD-ROM, CD-R, CD-RW, flexible disk, magnetic tape, MO, DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, Blu-ray, HD DVD (AOD), and UHD (all product names), optical disks, magneto-optical disks, semiconductor memories, and other media capable of storing the program. The program also includes those stored on the recording medium and distributed, as well as those distributed by download over a network such as the Internet. The device storing the program also includes general-purpose or dedicated devices in which the program is implemented in an executable form, such as software or firmware. Furthermore, each process and function included in the program may be executed by computer-executable program software, or each process may be implemented by hardware such as a predetermined gate array (FPGA, ASIC), or by a combination of the program software and partial hardware modules that implement some of the hardware elements. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration of an image inspection device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating a hardware configuration of an image inspection device. [Figure 3] FIG. 1 is a schematic diagram showing a pass / fail judgment of OK or NG. [Figure 4] FIG. 10 is a schematic diagram showing the sorting of non-defective products. [Figure 5]FIG. 10 is a schematic diagram showing how workpieces are classified according to the sorting results. [Figure 6] FIG. 2 is a block diagram showing the functions of a processor unit. [Figure 7] FIG. 10 is a schematic diagram showing a feature space of a neural network in which pass / fail judgment conditions are set. [Figure 8] FIG. 10 is a schematic diagram showing a feature space of a neural network in which sorting conditions are set. [Figure 9] 10 is a flowchart showing a procedure for setting a judgment condition when setting up an image inspection device. [Figure 10] FIG. 1 is a schematic diagram showing a standard sorting tool. [Figure 11] 10 is a flowchart showing a procedure for selecting an inspection mode. [Figure 12] FIG. 1 is a schematic diagram showing how a sorting process is performed using a neural network. [Figure 13] FIG. 13 is a schematic diagram showing how the same neural network as in FIG. 12 is used to perform pass / fail judgment processing. [Figure 14] FIG. 10 is a schematic diagram showing a state in which the "pass / fail determination mode" button is selected on the mode selection screen. [Figure 15] FIG. 10 is a schematic diagram showing a sorting mode screen. [Figure 16] FIG. 10 is a schematic diagram showing a tool setting screen. [Figure 17] FIG. 10 is a schematic diagram showing a detection window setting screen. [Figure 18] FIG. 10 is a schematic diagram showing a variety registration screen. [Figure 19] FIG. 19 is a schematic diagram showing the variety registration screen to which a variety image has been added from FIG. 18. [Figure 20] FIG. 10 is a schematic diagram showing a product type registration screen after learning has been completed; [Figure 21] FIG. 10 is a schematic diagram showing an output allocation screen. [Figure 22] FIG. 10 is a schematic diagram showing a product type registration screen for custom settings. [Figure 23] FIG. 10 is a schematic diagram showing a tool addition screen. [Figure 24]FIG. 10 is a schematic diagram showing a product type registration screen on which a contour tool is set. [Figure 25] FIG. 25 is a schematic diagram showing the variety registration screen to which a color tool is further added in addition to the screen shown in FIG. [Figure 26] FIG. 10 is a schematic diagram showing a product type registration screen on which a width tool is set. DETAILED DESCRIPTION OF THE INVENTION

[0012] Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below exemplify an image inspection apparatus, an image processing method, an image processing program, a computer-readable recording medium, and a device on which the program is recorded, embodying the technical concepts of the present invention. The present invention does not limit the image inspection apparatus, image processing method, image processing program, computer-readable recording medium, and device on which the program is recorded to those described below. Furthermore, this specification does not in any way specify the components described in the claims as components of the embodiments. In particular, the dimensions, materials, shapes, and relative positions of components described in the embodiments are not intended to limit the scope of the present invention, and are merely illustrative examples unless otherwise specified. Note that the size and positional relationships of components shown in each drawing may be exaggerated for clarity. Furthermore, in the following description, the same names and symbols indicate components that are identical or of the same quality, and detailed description will be omitted as appropriate. Furthermore, the elements constituting the present invention may be configured with the same components, so that multiple elements can be served by one component, or conversely, the functions of one component can be shared by multiple components. [Embodiment 1]

[0013] An image inspection device according to a first embodiment of the present invention is shown in the schematic diagram of FIG. 1. The image inspection device is a device for determining whether an object to be inspected, such as various parts or products, is called a workpiece and sorting the object based on an image of the object. The image inspection device is called an image sensor or the like and can be used in production sites such as factories. The object to be inspected may be the entire object, or only a portion of it. Furthermore, one object to be inspected may include multiple objects to be inspected. Furthermore, one image may include multiple objects to be inspected.

[0014] Here, we will explain an example of an image inspection device that captures the appearance of an object under inspection and performs pass / fail judgment and sorting judgment according to predefined inspection conditions. For pass / fail judgment, for example, predetermined pass / fail judgment conditions for judging whether an object is good or bad are set during setup, and then, during operation or operation, an image of the object under inspection is captured and the pass / fail of the object under inspection is judged in light of the pass / fail judgment conditions. Furthermore, for sorting judgment, an object under inspection that has been judged to be good is further judged to belong to one of multiple types according to predefined sorting conditions. For example, this applies to an example in which good objects are sorted into red, blue, or green according to sorting conditions for the color of the object under inspection.

[0015] The image inspection device 100 comprises a control unit 2, which is the device main body, an imaging unit 3, a display unit 4, a personal computer 5, and an operation unit 6. An image inspection program for operating the image inspection device 100 is installed in the personal computer 5. The user interface screen of the image inspection program can be displayed on the monitor of the personal computer 5 or on the display unit 4. The personal computer 5 is not essential and can be omitted. In this case, the control unit 2 executes the image inspection. Alternatively, the control unit 2 may execute the image inspection program.

[0016] Furthermore, a personal computer display can be used instead of the display unit 4. Although FIG. 1 illustrates the control unit 2, imaging unit 3, display unit 4, personal computer 5, and operation unit 6 as separate components in an example configuration of the image inspection device 100, any two or more of these components can be combined and integrated. For example, the control unit 2 and imaging unit 3 can be integrated, or the control unit 2 and display unit 4 can be integrated. Furthermore, the control unit 2 can be divided into multiple units, some of which can be incorporated into the imaging unit 3 or the display unit 4, or the imaging unit 3 can be divided into multiple units, some of which can be incorporated into other units. Furthermore, the operation unit 6 can be provided separately, or it can be integrated with other components, such as by using an input device included in the personal computer or by using a touch panel as the display unit.

[0017] 1, the control unit 2 is connected to the imaging unit 3, the display unit 4, and the personal computer 5 via cables, respectively. However, the present invention does not limit the connections of the components to wired connections, and wireless connections via radio waves such as wireless LAN, public communication lines, and NFC, infrared rays, light, or other media may also be used. Furthermore, standardized general-purpose communication standards such as Ethernet, IEEE802.1x, USB, Bluetooth, and ZigBee (all of which are registered trademarks or product names), as well as dedicated protocols and interfaces, can be used as appropriate.

[0018] The hardware configuration of an image inspection device 100 according to the first embodiment of the present invention is shown in the block diagram of Fig. 2. The image inspection device 100 shown in this figure comprises a housing 1 including a control unit 2 and an imaging unit 3, a display unit 4, and a personal computer 5.

[0019] The housing 1 is a casing that forms the outer shape of the image inspection device 100 and houses the illumination unit 15, the camera unit 14, the trained neural network storage unit 19a, the inference processing unit 20a, etc. The housing 1 is provided with an interface that accepts various settings from the user. Through the interface, a pass / fail judgment mode for judging the pass / fail of workpiece images and a sorting mode for sorting input workpiece images can be selected. The inference processing unit 20a executes either a first inference process or a second inference process depending on the selected mode. This eliminates the need for data communication equipment and realizes inference processing that is resistant to delays and disturbances by providing a trained neural network storage unit 19a in each image inspection device and performing inference processing using locally stored data, rather than using trained neural networks stored in physically different locations such as a cloud service. (Control Unit 2)

[0020] The control unit 2 includes a main board 13, a connector board 16, a communication board 17, a power supply board 18, a memory unit 19, and an output unit 12. The main board 13 is equipped with a processor unit 20 and a memory 133. The memory 133 is configured with RAM, ROM, etc.

[0021] The connector board 16 receives power from an external power source via a power connector provided in the power interface 161. The power supply board 18 distributes the received power to each board. In this embodiment, power is supplied to the camera unit 14 via the main board 13. A motor driver 181 on the power supply board 18 supplies drive power to the motor 141 of the camera unit 14 to achieve autofocus.

[0022] The communication board 17 transmits an OK / NG signal (determination signal) indicating the pass / fail determination result of the object to be inspected, output from the main board 13, as well as image data and the like to the display unit 4. Upon receiving the determination signal, the display unit 4 displays the determination result. Note that, in this embodiment, the determination signal is configured to be output via the communication board 17, but the determination signal may also be output via the connector board 16, for example. (Operation unit 6)

[0023] The image inspection device 100 also includes an operation unit 6 that accepts user operations. The operation unit 6 can use existing input devices such as a keyboard, mouse, or touch panel. In the example of FIG. 2, the communication board 17 is configured to accept various user operations input from the touch panel 41 of the display unit 4 or the keyboard 51 of the personal computer 5. The touch panel 41 of the display unit 4 is, for example, a known touch-type operation panel equipped with a pressure-sensitive sensor, and detects touch operations by the user and outputs them to the communication board 17. In addition to the keyboard 51, the personal computer 5 is also equipped with a mouse and a touch panel, and is configured to accept various user operations input from these operation devices. Communication may be wired or wireless, and either form of communication can be realized by a conventionally known communication module.

[0024] The illumination unit 15 includes a plurality of LEDs 11 that emit illumination light onto an imaging area where an image of an object to be inspected is captured. The LEDs 11 may be provided with a lens or a reflector. The lens may be replaceable as a short-distance or long-distance lens unit. In this specification, illumination light primarily refers to light emitted by the illumination unit 15, but also includes ambient light, such as natural light, that exists regardless of the illumination unit 15.

[0025] The imaging unit 3 includes a camera section 14 and an illumination section 15. The camera section 14 is capable of controlling autofocus operation by being driven by a motor 141. The camera section 14 captures an image of the object to be inspected in response to an imaging instruction signal from the main board 13. In this embodiment, the CMOS board 142 is provided as an imaging element. The captured color image is converted into an HDR image by the CMOS board 142 based on conversion characteristics that widen the dynamic range, and is output to the processor section 20 on the main board 13.

[0026] The main board 13 controls the operation of each board connected to it. For example, for the illumination unit 15, it sends a control signal to an LED driver 151 to control the on / off of the multiple LEDs 11. The LED driver 151 adjusts, for example, the on / off and light intensity of the LEDs 11 in response to a control signal from the processor unit 20. In addition, it sends a control signal to the motor 141 of the camera unit 14 via a motor driver 181 on the power supply board 18 to control autofocus operation. Furthermore, it sends an image capture instruction signal to the CMOS board 142. (Processor unit 20)

[0027] The processor unit 20 on the main board 13 is a control circuit or control element that processes input signals and data, performs various calculations, and outputs the calculation results. The processor unit 20 is not limited to processors such as CPUs, MPUs, GPUs, and TPUs for general-purpose PCs. It can also be configured with gate arrays such as LSIs, FPGAs, and ASICs customized for specific applications, microcomputers, or chipsets or packages such as SoCs. The processor unit 20 performs multiple functions, as described below. Note that the present invention is not limited to an example in which the processor unit is configured with a single physical processor unit; it may also be configured with multiple CPUs. The multiple CPUs may include multiple physical CPUs or a so-called multi-core MPU incorporating multiple CPU cores in a single package. In this case, each function may be performed by multiple CPUs or CPU cores, or different functions may be assigned to each CPU or CPU core. Furthermore, the processor unit may be configured with a combination of a CPU and a GPU. In this case, the GPU not only performs the function of the display control unit described above, but may also be configured to execute some or all of the functions assigned to the processor unit.

[0028] In the example of FIG. 2, the processor unit 20 of the main board 13 is composed of an FPGA and a DSP. The FPGA controls illumination and imaging, and also performs image processing on the acquired image data. The DSP also performs edge detection, pattern search, and other processes on the image data. As a result of the pattern search process, a judgment result indicating the pass / fail of the object to be inspected is output to the communication board 17. The results of the arithmetic processing, etc., are stored in the memory 133. Note that in the above example, the FPGA controls illumination and imaging, etc., but the DSP may also perform these functions. Furthermore, instead of combining the FPGA and the DSP, a single main control circuit or main control unit may be provided. For example, a single CPU serves as the main control unit and performs functions such as sending control signals to the LED driver 151 to control the on / off of the multiple LEDs 11, sending control signals to the motor 141 of the camera unit 14 to control autofocus operation, and sending image capture instruction signals, etc., to the CMOS board 142. (Storage unit 19)

[0029] The control unit 2 is provided with a storage unit 19 such as a hard disk drive. The storage unit 19 stores program files, setting files, etc. (software) for enabling the hardware to execute the various controls and processes described below, as well as master images, pass / fail judgment results, sorting results, etc. The program files and setting files are stored in a portable storage medium such as a USB memory or optical disk, and the program files and setting files stored in this storage medium can be read into the control unit 2.

[0030] The storage unit 19 also functions as a trained neural network storage unit 19a that stores one or more trained neural networks. The trained neural network has an input layer to which a workpiece image is input, an intermediate layer connected to this input layer, and an output layer connected to this intermediate layer and outputting the feature quantities of the input workpiece image. The weight coefficients of each layer have been trained in advance. (Output section 12)

[0031] The output unit 12 is a component for outputting the results of the first inference process and the second inference process performed by the inference processing unit 20a. The output unit 12 has multiple output ports 12a, 12b, ... 12n that output the results of the second inference process performed by the inference processing unit 20a. Each output port 12a, 12b, ... 12n is assigned as an output destination for each type of work sorted in the second inference process. This allows for output of not only a binary result of OK or NG, as in the pass / fail judgment shown in FIG. 3, but also a multi-value result of three or more values, as shown in FIG. 4, making subsequent processing based on the sorting and classification results easier. For example, as shown in FIG. 5, the camera unit 14 can capture images of inspection workpieces WK transported sequentially on a conveyor belt CB, and the images can be used to classify the pass / fail workpieces by color, shape, size, etc., or by ranking the pass / fail workpieces. In this example, depending on the sorting results from the inference processing unit 20a of the main board 13, the output from each output port 12a, 12b, ... 12n of the output unit 12 is input to sorting machines ST1, ST2, ST3 which transport the work WK from the conveyor belt CB to other lines CB1, CB2, CB3, making it possible to allocate the work WK to the appropriate line depending on its type.

[0032] A block diagram of the processor unit 20 is shown in Figure 6. As shown in this figure, the processor unit 20 realizes the functions of an inference processing unit 20a, a mode selection unit 20b, a sorting base selection unit 20c, and a tool setting unit 20d. The inference processing unit 20a is a component for performing a quality judgment of a workpiece or a workpiece sorting judgment based on a workpiece image. The mode selection unit 20b is a component for selecting a quality judgment mode for performing a first inference process and a sorting mode for performing a second inference process. The sorting base selection unit 20c is a component for selecting whether the second inference process performed by the inference processing unit 20a will be performed using learning-based learning sorting or rule-based standard sorting. The tool setting unit 20d is a component for setting a master image and an inspection tool. (Conditions for determining pass / fail)

[0033] The inference processing unit 20a also functions as a judgment condition setting unit that sets the pass / fail judgment conditions for determining the pass / fail quality of a workpiece. Examples of pass / fail judgment conditions include the setting of a pass / fail judgment boundary. Specifically, when setting up the image inspection device 100, images of pass / fail workpieces and images of defective workpieces generated by the camera unit 14 are input to the neural network stored in the trained neural network storage unit 19a. There may be one or more pass / fail images and one or more defective images. Then, for each of the input pass / fail images and defective images, multiple pass / fail feature quantities that characterize each image are acquired from the neural network. Among the features of the neural network, the pass / fail feature quantities used for pass / fail judgment are image parameters that can be effective for pass / fail judgment, such as image color, edge, and position. Furthermore, based on these multiple pass / fail feature quantities, a pass / fail judgment boundary for determining the pass / fail quality of the workpiece is set in the feature space of the neural network. For example, in the neural network feature space FS shown in Fig. 7, the pass / fail feature values ​​of good product images are plotted as ●, and the pass / fail feature values ​​of defective product images are plotted as ○. In this feature space, a pass / fail judgment boundary BD1 is set to distinguish between an area containing a group of good product images and an area containing a group of defective product images. In the example of Fig. 7, items above the pass / fail judgment boundary BD1 are judged to be good products, and items below are judged to be defective products.

[0034] After completing the setup of the image inspection device 100 in this manner, during actual operation, the inference processing unit 20a performs pass / fail judgment in accordance with the judgment conditions set during setup. Specifically, the workpiece to be inspected, which is the target for pass / fail judgment, is imaged with the camera unit 14 to generate an inspection workpiece image. This inspection workpiece image is input to a neural network stored in the trained neural network storage unit 19a to obtain pass / fail feature quantities. The pass / fail feature quantities are then applied to the pass / fail judgment boundary set in the feature space to perform pass / fail judgment on the workpiece image. This pass / fail judgment is called the first inference process. (Sorting conditions)

[0035] Meanwhile, the inference processing unit 20a further classifies workpieces determined to be non-defective according to sorting conditions. The sorting conditions are conditions for determining which of multiple predefined different types each workpiece belongs to. The multiple different types may be three or more. For example, when sorting workpieces by color, they are classified into either red or black, or red, yellow, or black. Workpieces may also be sorted by type, such as shape or size. Therefore, the inference processing unit 20a also functions as a sorting condition setting unit that sets sorting conditions for sorting workpieces. Examples of pass / fail judgment conditions include the setting of a pass / fail judgment boundary. Specifically, when setting up the image inspection device 100, multiple different types of workpiece images generated by the camera unit 14 are input into a neural network stored in the trained neural network storage unit 19a, and a sorting boundary for classifying the workpieces is set in the feature space of the neural network based on multiple sorting features that characterize each workpiece image obtained. The sorting features are image parameters that can be effective for sorting and are set according to the type of workpiece to be sorted. For example, when sorting workpieces according to their color, the chromaticity and brightness of the image are effective, while when sorting according to their shape, the edges of the image are effective. The sorting features may also be used as pass / fail features depending on the type of workpiece. Then, based on the sorting features, a sorting boundary for sorting workpieces is set in the feature space of the neural network. For example, the feature space FS of the neural network shown in Figure 8 shows an example of sorting workpieces according to the shape of their tip: round pins or U-pins. Round pins are represented by a triangle, and U-pins are represented by a circle. In this feature space, a sorting boundary BD2 is set to distinguish between the area containing a group of round pins and the area containing a group of U-pins. In the example of Figure 8, the left side of the sorting boundary BD2 is sorted into round pins, and the right side is sorted into U-pins.

[0036] After the sorting boundaries have been set in this way, in actual operation, the inference processing unit 20a performs sorting according to the sorting conditions. Specifically, the workpiece images that have already been captured of the inspection workpieces to be sorted are input into the neural network stored in the trained neural network storage unit 19a to obtain sorting features. The sorting features are then applied to the sorting boundaries set in the feature space to sort the workpiece images. This sorting process is called the second inference process.

[0037] This configuration allows a trained neural network to be prepared in advance, and a common neural network can be used to perform two different inference processes: quality determination and sorting. As a result, the effort required to build a neural network for each inference process can be eliminated, simplifying the system.

[0038] In particular, by configuring the inference processing unit 20a to classify workpieces determined to be non-defective in the first inference process into multiple values ​​using the second inference process, it becomes possible to go beyond simple non-defective judgment and further classify the non-defective products in a series of processes. In other words, in the past, many image sensors only made a binary pass / fail judgment of the workpiece to be inspected, either OK or NG, and to further classify this, it was necessary to prepare multiple image sensors for sorting according to the number of classifications. However, the image inspection device 100 according to this embodiment can further classify non-defective workpieces into multiple values ​​using a single device. Note that workpieces that do not have a sorting destination are deemed NG (defective).

[0039] In the above example, the inference processing unit 20a is described as also serving as a judgment condition setting unit that sets pass / fail judgment conditions, sorting conditions, etc., but the present invention is not limited to this configuration, and the inference processing unit and the judgment condition setting unit may be prepared separately. (Procedure for setting judgment conditions)

[0040] Next, the procedure for setting judgment conditions in the inference processing unit 20a when setting up the image inspection device 100 will be described with reference to the flowchart in FIG. 9. First, in step S901, the feature quantities of the training image are calculated. Next, in step S902, the training image is plotted in feature space. Furthermore, in step S903, the position of the training image in feature space is grasped. Next, in step S904, it is determined whether or not to set pass / fail judgment. If pass / fail judgment is set, in step S905, a judgment boundary for pass / fail judgment is set, and the process proceeds to step S907. On the other hand, if pass / fail judgment is not set in step S904, the process proceeds to step S906, where a sorting boundary for classification is set, and the process proceeds to step S907. Finally, in step S907, the set judgment boundary is recorded. Here, the judgment boundary is recorded and stored in the memory unit 19. In this manner, the judgment conditions are set. (Inspection tool)

[0041] The image inspection device 100 according to this embodiment may also be equipped with an inspection tool. The inspection tool is a component for setting conditions for learning and sorting. Examples of inspection tools include a learning tool and a sorting tool. The learning tool is a tool for setting conditions for determining pass / fail. The learning tool sets inspection target areas for a master image of a good product and a master image of a defective product, and sets a determination boundary. That is, in the first inference process, settings are made for the master image using the learning tool.

[0042] On the other hand, the sorting tool is a tool for setting sorting conditions. Sorting tools can include learning sorting tools and standard sorting tools. The learning sorting tool sets an area on the master image to be referenced during sorting and registers a type associated with each master image. That is, in the second inference process, learning sorting is set for the master image using the learning sorting tool, and sorting can be performed using a single learning sorting tool, as shown in Figure 4. On the other hand, standard sorting is set using multiple tools for each type. For this reason, the standard sorting tool has multiple standard tools. For example, in the example shown in Figure 10, the standard sorting tool is composed of Standard Tool 1, Standard Tool 2, and Standard Tool 3. Here, the sorting result is assumed to be classified as Work A, Work B, or Work C. In this case, if the inspection image is classified as Work A, the sorting result is output from Standard Tool 1. Similarly, if the inspection image is classified as Work B, the sorting result is output from Standard Tool 2. Furthermore, if the inspection image is classified as Work C, the sorting result is output from Standard Tool 3. If the inspection image cannot be classified into either category, it is deemed NG and is not output.

[0043] In this way, learning sorting uses a single learning sorting tool, whereas standard sorting sets up multiple standard tools for each product type and runs each tool in turn. In the example of Figure 10, standard tools 1 to 3 are run in turn to output the sorting results. This allows for clear management of standard sorting, but it requires the setting up of multiple standard tools. (Select inspection mode)

[0044] The procedure for selecting the sorting mode or the pass / fail judgment mode as the inspection mode will now be described with reference to the flowchart in Fig. 11. First, in step S1101, the mode selection unit 20b of the processor unit 20 shown in Fig. 6 accepts a selection of either the pass / fail judgment mode or the sorting mode from the user. After accepting the selection of the sorting mode from the user, the mode selection unit 20b can further accept a selection of either the learning sorting mode or the standard sorting mode in step S1102.

[0045] Here, the learning sorting mode is a mode that performs learning sorting using machine learning using the neural network described above, and the standard sorting mode is a rule-based sorting mode that does not use a neural network but uses features such as the contours and color of the work.

[0046] When the learning sorting mode is selected in step S1102, the processor unit 20 accepts registration of a master image (step S1103). The master image may be an image captured by the camera unit 14 each time, or may be a past driving history image or a file image stored in the memory unit 19.

[0047] Next, the user sets one or more detection windows as areas to be inspected on the master image (step S1104). The image within the set detection windows is input to the neural network described above, and the feature values ​​within each detection window are output from the neural network.

[0048] Next, the user registers the product type corresponding to the master image for which the detection window has been set (step S1105). For example, if the user wants to register a red ballpoint pen as a product type, the product type corresponding to the master image is registered as a "red ballpoint pen." The registered product type is stored in the storage unit 19 in association with the feature amount in the detection window output from the neural network. Although only one image may correspond to each product type, multiple images corresponding to multiple red ballpoint pens can be registered as master images. For example, if ballpoint pens with slightly different colors are registered as red ballpoint pens, ballpoint pens with a color close to red can be sorted as "red ballpoint pens" even if the color is slightly different during operation.

[0049] To perform sorting, two or more types must be registered. For example, to sort into red ballpoint pens and blue ballpoint pens, a master image corresponding to the blue ballpoint pen must be additionally registered, the features within the detection window must be extracted, and the type must be registered as "blue ballpoint pen."

[0050] The master images corresponding to each variety, the feature values ​​extracted from each master image, and the variety corresponding to each master image are stored in the storage unit 19. In this state, when an instruction to start the learning process is received from the user, the sorting boundaries necessary for sorting each variety are calculated (step S1106). Note that the learning process performed here is a process of calculating the sorting boundaries for sorting each variety based on the feature values ​​output from the neural network that has been learned in advance and stored in the storage unit 19, and is not a process of learning the parameters of the neural network itself. Next, in step S1107, output setting is performed. Details of the output setting will be described later.

[0051] On the other hand, if the rule-based sorting mode is selected in step S1102, the process proceeds to step S1108, where the registration of a master image is accepted, and the setting of sorting rules is accepted in step S1109. When setting sorting rules, a color tool that distinguishes colors within a detection window and a contour tool that detects contours can be set on the registered master image. By selecting the color tool, workpiece sorting rules can be set based on the colors within the detection window, or by using the contour tool, based on information about detected contours. In this way, when the rule-based sorting mode is selected in step S1102, feature values ​​set by the user can be extracted from the image, and workpieces can be sorted based on the extracted feature values, without extracting feature values ​​using the neural network described above.

[0052] On the other hand, if the user selects the pass / fail judgment mode in step S1101, the process proceeds to step S1110, where the user selects whether to perform the pass / fail judgment based on learning or rule. Here, the user can select either the learning pass / fail judgment mode or the rule-based pass / fail judgment mode. When the user selects an inspection tool for learning, the learning pass / fail judgment mode is set. On the other hand, when the user selects a rule-based inspection tool, the rule-based setting mode can be set.

[0053] When the learning pass / fail judgment mode is selected in step S1110, the processor unit 20 accepts the registration of master images corresponding to the good product images and the defective product images (step S1111), and sets a detection window on the master image (step S1112). The image within the detection window set for the master image is then input to a neural network, and the feature values ​​indicating that the image is a good product image and the feature values ​​indicating that the image is a defective product image output from the neural network are mapped onto the feature space, and a judgment boundary for distinguishing between good products and defective products is automatically generated (step S1113). Note that, as in the learning sorting mode described above, there may be one good product image corresponding to a good product and one defective product image corresponding to a defective product, or multiple good product images may be used.

[0054] On the other hand, if the rule-based setting mode is selected in step S1110, the registration of the master image is accepted in step S1114 (step S1115). Here, rule-based inspection tools such as a contour tool and a color tool can be set.

[0055] As described above, in this embodiment, it is possible to set one of the following inspection modes: a learning sorting mode (first sorting mode) in which images of different types are input to a pre-trained neural network to generate a sorting boundary, a rule-based sorting mode (second sorting mode) in which types are sorted based on color and contour information, a learning pass / fail judgment mode (first pass / fail judgment mode) in which images of pass / fail products are input to a pre-trained neural network to generate a pass / fail judgment boundary, and a rule-based pass / fail judgment mode (second pass / fail judgment mode) in which the pass / fail judgment of the workpiece is performed based on color and contour information. The selection of such inspection modes can be performed by the mode selection unit 20b.

[0056] The same neural network can be used to extract features in the learning sorting mode and the learning pass / fail judgment mode.By simply training a single neural network in the image inspection device itself, it is possible to switch between the learning sorting mode, in which three or more product types are registered and sorted, and the pass / fail judgment mode, in which images of good and bad products are learned and pass / fail judgment is performed.

[0057] FIG. 12 shows the sorting process using a neural network, and FIG. 13 shows the pass / fail judgment process using the same neural network. In these figures, A shows the state in which a detection window DW is set in a workpiece image captured of a workpiece WK, B shows the inference process performed by the neural network, and C shows the sorting process or pass / fail judgment process performed in the feature space FS. The neural network NN shown in these figures is the same neural network NN that has been trained in advance and stored in the memory unit 19. The inference process for the sorting process and pass / fail judgment process is performed by inputting the image within the detection window DW shown in FIG. 12 and FIG. 13A into the pre-trained neural network NN. The image inspection device main body stores the pre-trained neural network NN in the trained neural network memory unit 19a of the memory unit 19, and the processor unit 20 is equipped with a dedicated circuit for executing the inference process. The dedicated circuit executes the inference process for sorting and pass / fail judgment. The processor unit 20 does not need to perform the learning process of the neural network NN, but only needs to perform the inference process, so there is no processing load.

[0058] In the learning pass / fail judgment process, an image in a preset detection window DW shown in A of Fig. 13 is input to a pre-trained neural network NN shown in B, and features are extracted. The extracted features are mapped onto a feature space FS shown in C, and compared with a judgment boundary set for the feature space FS, and a pass / fail judgment result is output (first inference process).

[0059] On the other hand, in the learning sorting process, an image within a preset detection window DW shown in A of Fig. 12 is input to a pre-trained neural network NN shown in B of Fig. 12, and features are extracted. The extracted features are compared with the sorting boundaries set by the setting process described above in the feature space FS in C of Fig. 12, and a determination is made as to which of the preset varieties the feature corresponds to, and the result is output (second inference process).

[0060] In these pass / fail judgment and sorting processes, only images within the detection window DW are input to the neural network NN to extract features, reducing the load on inference processing.In addition, images outside the detection window DW do not affect the performance of pass / fail judgment and sorting, allowing for stable inspection.

[0061] In the above embodiment, the same neural network is used for quality determination and sorting, but it goes without saying that separate neural networks may be used. (Sorting GUI)

[0062] The procedure for setting judgment conditions using an inspection tool will be described below with reference to the user interface screens of the image inspection program shown in FIGS. 14 to 21. First, select whether to set pass / fail judgment conditions or sorting conditions. FIG. 14 shows an example of a mode selection screen 210. The mode selection screen 210 includes a "Pass / Fail Judgment Mode" button 211 and a "Sorting Mode" button 212. When the "Pass / Fail Judgment Mode" button 211 is selected, an explanation of the pass / fail judgment mode may be displayed, such as, "Register a pass / fail workpiece as a master image and set tools to distinguish characteristics such as contours, area, and edges. The differences from the master image are determined." On the other hand, when the "Sorting Mode" button 212 is selected, an explanation may be displayed, such as, "Register multiple product type images as master images and distinguish the product type based on the product's characteristics. Sorting can be performed based on the judgment results." In this way, by presenting the user with the items to be selected and explaining each item with text, graphics, etc., even users who are unfamiliar with settings and operations can be guided step-by-step through the setting procedures and meanings, leading them to perform appropriate settings. (Easy setup)

[0063] The sorting condition settings can be set to either easy settings, which simplify the setup procedure, or custom settings, which allow the user to directly specify each setting item. For example, in FIG. 14, pressing the "Sorting Mode" button 212 displays the sorting mode screen 220 shown in FIG. 15, which displays a "Learning Sorting Mode" button 221 and a "Rule-Based Sorting Mode" button 222, allowing the user to select either easy settings or custom settings. Pressing the "Learning Sorting Mode" button 221 starts easy settings and displays the learning sorting tool. Setting sorting conditions using the learning sorting tool includes setting imaging conditions, registering master images, registering product types, and assigning outputs. Setting imaging conditions includes setting the imaging field of view, image brightness, focus, and other imaging conditions when generating inspection images using the camera unit 14.

[0064] The image inspection program also has a navigation function that guides the user through the setup process during simple setup. The navigation function, as shown in Figure 16, for example, illustrates each step in a flow chart at the top of the screen and highlights the item currently being set, allowing the user to see the progress of the setup process at a glance.

[0065] Master images are registered using the master image registration screen. In the master image registration screen, a master image is registered to set the inspection target area for image inspection. The inspection target area is set as a frame-shaped window. If the brightness of the inspection image captured during operation fluctuates, images with varying brightness can be acquired and used as the first registered image to accommodate the brightness fluctuations. Master images can be registered from live images, operating image history, or file images. FIG. 16 shows an example of a tool setting screen 240. In the tool setting screen 240, the inspection target area is set for a registered master image and the type of product to be sorted is registered. The tool setting screen 240 in FIG. 16 has a "Detection Window Setting" button 241 and a "Type Image Registration" button 242 in the operation area 202 on the right. Pressing the "Detection Window Setting" button 241 displays the detection window setting screen 250 in FIG. 17. The inspection target area is set as a frame in the image display area 201 using an input device such as a mouse. Multiple inspection target areas can be specified. The specified inspection target areas are displayed as frames in the image display area 201. In the operation area 202, the specified inspection target areas are assigned individual identification numbers and are displayed in a list. In the example of FIG. 17, three windows are specified as inspection target areas. When all inspection target areas have been specified in this way, the "OK" button 251 at the bottom right is pressed to finish specifying the inspection target areas.

[0066] Meanwhile, sorting variety registration is performed on the variety registration screen. When the "Register variety image" button 242 is pressed on the tool setting screen 240 of FIG. 16, the variety registration screen 260 of FIG. 18 is displayed. On the variety registration screen 260, the variety and image are registered and the sorting boundary is set. Setting the sorting boundary here is also referred to as "learning" in this embodiment. The registered images are displayed in a reduced size list in the variety image display field 261 provided on the left edge of the image display area 201.

[0067] Here, the first variety may be set for an image registered as a master image. In this case, the first variety is set for an image that has already been registered. In the example of FIG. 18, the master image is registered as the first variety image and is registered as number 0 in the variety image display field 261 provided on the left side of the image display area 201. Therefore, images corresponding to the second and subsequent varieties are registered sequentially. Registration of an image corresponding to an additional variety is performed by pressing the "Register Image" button 262 provided in the operation area 202 of FIG. 18. When the "Register Image" button 262 is pressed, the image displayed in the image display area 201 of FIG. 19 is registered as the variety image. For example, in the example of FIG. 19, the workpiece imaged by the camera unit 14 of the image inspection device 100 is changed to a workpiece of the variety to be registered, and while the live image is displayed in the image display area 201 of FIG. 18, the workpiece is registered as the second variety image, and the first variety image is registered in the variety image display field 261 of the image display area 201 of FIG. 19. Furthermore, the variety image is not limited to a live image, and can be registered from an image registered in the past. For example, on the variety registration screen 260 in Fig. 18, by pressing the "From File / History" button 263 provided in the lower part of the operation area 202, a previously saved image can be displayed and registered as the variety image.

[0068] Furthermore, a variety name can be registered along with the variety image. In the example of Fig. 18, the variety name "MASTER_0" is automatically entered, and in the example of Fig. 19, "MASTER_1" is automatically entered. The user can also register any variety name. For example, on the variety registration screen 260 of Fig. 18, for example, pressing the memo-shaped icon 264 provided in the operation area 202 allows the variety name to be edited.

[0069] Once multiple, i.e., two or more, variety images have been registered in this way, learning, in this case, setting of the sorting boundaries, can begin. When the "Start Learning" button 265 located at the bottom right of the variety registration screen 260 in Figure 19 is pressed, the variety images registered at this point are used to set the sorting boundaries in the feature space. Note that learning is not possible after adding a variety name if no variety images have been registered, and the "Start Learning" button 265 is grayed out and cannot be selected.

[0070] After completing the product type registration, the final step is to assign outputs. For example, once learning is complete and the sorting boundaries are set, product type registration is complete, and the "Proceed to STEP 4" button 266 becomes available for selection, as shown in the lower right of FIG. 20. Pressing the "Proceed to STEP 4" button 266 switches to the output assignment screen 270 of FIG. 21. The output assignment screen 270 allows the user to assign output content for each output port. The output that can be assigned to each output port includes information about the type of product that has been set, the pass / fail judgment result (OK or NG), operation, busy, error, no output, etc. The user can select the output content for each output port. In the example of FIG. 21, the setting field 271 for output port 1 allows the user to select the assignable output from a pull-down menu. Once all the judgment conditions have been set, the user presses the "Done" button 272 at the lower right to end the setting process and save the judgment condition settings in the storage unit 19. (Custom Settings)

[0071] The above describes the simple settings. On the other hand, custom settings are used to set up rule-based sorting. Rule-based sorting allows sorting to be performed using features specified by the user, such as color or contour, without using the neural network described above. For example, when rule-based sorting is selected in step S1102 of FIG. 11 by pressing the "rule-based sorting mode" button 222 on the sorting mode screen 220 of FIG. 15, rule-based sorting settings are made in step S1109. In rule-based sorting settings, rule-based sorting tools, such as contour tools and color tools, can be set according to features.

[0072] As an example of setting up rule-based sorting, the product type registration details screen 280 shown in FIG. 22 has an "Add Tool" button 281 in the operation area 202. Pressing this "Add Tool" button 281 displays the add tool screen 290 shown in FIG. 23. The add tool screen 290 allows the user to set which feature to use for sorting. In the example shown in FIG. 23, the operation area 202 provides basic tools: "Outline" 291, "Color Tool" 292, and "Position Correction" 293. By selecting each tool, the user can set the outline, color, position, and other aspects of the workpiece image as feature values ​​in detail. For example, when "Outline" 291 is selected, the product type registration details screen 280 is displayed as shown in FIG. 24. In the product type registration details screen 280, product type image 0 and product type name "MASTER_0" are registered in the operation area 202, and the outline 283 is set as the feature value 01. The outline 283 allows the user to set a threshold value. Here, a threshold value for the degree of match with the contour set on the master image is set using a slider in the range of 0 to 100. As an example of threshold setting, if you want to detect the contour detected within the detection window of the input image as the same product type even if it is not the same length as the contour within the detection window set on the master image or if it is partially missing, set the threshold value low. For example, if you want to determine that the contour of the input image is OK even if it is missing up to one-quarter of the contour of the master image, set the threshold value to "75." By setting the threshold value in this way, if the degree of match with the contour of the input image is equal to or greater than the threshold value, the sorting result will be output as the set product type.

[0073] FIG. 25 shows an example in which a color tool 284 is set as an additional tool. Sorting can be performed based on the degree to which the colors in the detection window in which the color tool is set on the master image match the colors in the detection window of the input image. A threshold for the degree of color match can also be set arbitrarily using a slider for the color tool 284. In this case, an AND setting is used in which the set type 1 is output when the degree of match of the contour detected by the contour tool 283 is equal to or greater than the threshold, and the degree of match of the color detected by the color tool 284 is equal to or greater than the threshold.

[0074] Furthermore, FIG. 26 shows an example in which the width tool 285 has been set as a feature for a different variety image, variety image 1, with the variety name "MASTER_1." The width tool 285 sets the distance between multiple contours on the master image as the width. A threshold can also be set using a slider for the degree of width match, which indicates the degree to which the width detected on the input image matches the width detected on the master image. In this example, input images with a width match equal to or greater than the threshold are output as variety 2.

[0075] In this way, sorting tools can be configured on the product type registration details screen 280 to perform sorting using multiple different feature values. After completing the rule-based sorting configuration, when the sorting process is executed, a detection window corresponding to each sorting tool is set on the master image, and feature values ​​corresponding to each sorting tool within the detection window are extracted. A threshold value, which determines the degree of match required between the feature values ​​detected on the master image and the image to determine the same product type as the master image, can be adjusted during configuration. During operation, when an input image is input, the position of the detection window is specified, for example, by position correction. Then, feature values ​​are extracted within the specified detection window, and a degree of match is calculated, indicating the degree to which the extracted feature values ​​match the feature values ​​of the master image. The calculated degree of match is compared with a threshold value, and a sorting process is performed to determine whether the product type is the same as the product type specified for the master image. Multiple different types of tools can also be configured for the same master image. In this case, sorting can be performed by combining comparison results based on the feature values ​​extracted by each tool. [Industrial Applicability]

[0076] The image inspection device, image inspection method, image inspection program, computer-readable recording medium, and recorded equipment of the present invention can be suitably used for applications such as determining whether an object to be inspected is good or bad or sorting the object based on an image of the workpiece being inspected. [Explanation of symbols]

[0077] 100...Image inspection device 1...Housing 2...Control unit 3...Imaging unit 4...Display section 5...Personal computer 6...Operation unit 11...LED 12...output section; 12a, 12b, ...12n...output ports 13...Main board 14...Camera section 15...Lighting section 16...Connector board 17...Communication board 18...Power supply board 19...Memory section; 19a...Trained neural network memory section 20...processor unit; 20a...inference processing unit; 20b...mode selection unit; 20c...Sorting base selection section; 20d...Tool setting section 41...Touch panel 51...Keyboard 100...Image inspection device 133...Memory 141...Motor 142... Circuit board 151...Driver 161...Power supply interface 181...Motor driver 201...Image display area 202…Operation area 210...Mode selection screen 211... "Good / bad judgment mode" button 212..."Sorting mode" button 220...Sorting mode screen 221... "Study sorting mode" button 222..."Rule-based sorting mode" button 240...Tool setting screen 241... "Detection window settings" button 242... "Register variety image" button 250...Detection window setting screen 251..."OK" button 260...Variety registration screen 261…Type image display field 262..."Register image" button 263..."From File History" button 264...Memo icon 265..."Start learning" button 266... ​​"Proceed to STEP 4" button 270...Output allocation screen 271...Output port 1 setting field 272..."Done" button 280...Variety registration details screen 281..."Add Tool" button 283...Contour 284...Color Tool 285…Width tool 290...Tool Addition Screen 291..."Contour" 292... "Color Tools" 293…“Position correction” CB...conveyor belt; CB1, CB2, CB3...separate lines ST1, ST2, ST3...Sorting machines FS: Feature space BD1: Pass / fail judgment boundary BD2...Sorting boundary WK…Work DW: Detection window NN...Neural network

Claims

1. an illumination unit that irradiates illumination light onto the workpiece to be inspected; a camera unit that receives light irradiated from the lighting unit and reflected by the workpiece, and generates a workpiece image; an input layer to which the workpiece image is input; a hidden layer coupled to the input layer; an output layer connected to the intermediate layer and outputting the feature quantities of the input workpiece image, and a trained neural network storage unit storing one or more neural networks in which the weight coefficients of each layer have been trained in advance; an inference processing unit that executes a quality determination or a work sorting determination based on the work image, The inference processing unit A pass / fail judgment boundary for judging whether a workpiece is pass or fail is set in a feature space of the neural network based on a plurality of pass / fail feature quantities that characterize each workpiece image obtained by inputting a pass / fail image indicating a pass / fail workpiece generated by the camera unit and a fail / fail image indicating a fail / fail workpiece into the neural network stored in the trained neural network storage unit, a first inference process for determining whether the workpiece image is good or bad based on the quality feature amount obtained by inputting the inspection workpiece image generated by the camera unit into the neural network stored in the trained neural network storage unit and the quality determination boundary; a sorting boundary for classifying the workpieces in a feature space of the neural network based on a plurality of sorting features that characterize each of the workpiece images obtained by inputting a plurality of different types of workpiece images generated by the camera unit into the neural network stored in the trained neural network storage unit; a second inference process for sorting the workpiece images for inspection generated by the camera unit based on the sorting feature values ​​and the sorting boundaries obtained by inputting the workpiece images for inspection generated by the camera unit into the neural network stored in the trained neural network storage unit; Make it executable, The apparatus further includes a tool setting unit for setting a master image and an inspection tool, The tool setting unit is capable of setting a learning and sorting tool, which is the inspection tool that executes the second inference process, and in setting the learning and sorting tool, Accepting registration of a first master image as the master image; Accepting setting of a detection window as a region to be referenced on the first master image; After the detection window is set, registration of a second master image is accepted, the second master image being a master image of a type that is classified by the sorting determination and that is different from the first master image; the sorting boundary is calculated based on the first master image, the second master image, and the detection window; The second inference process is performed based on the set detection window and the sorting boundary.

2. 2. The image inspection device according to claim 1, The tool setting unit is an image inspection device that displays a list of the registered master images by type in setting the learning sorting tool.

3. 2. The image inspection device according to claim 1, The tool setting unit is an image inspection device that displays a product type registration screen that can accept the addition of the type and the registration of the master image corresponding to the added type in the setting of the learning sorting tool.

4. The image inspection device according to any one of claims 1 to 3, further comprising: An image inspection device comprising a mode selection unit for selecting a pass / fail judgment mode in which the first inference processing is performed and a sorting mode in which the second inference processing is performed.

5. The image inspection device according to any one of claims 1 to 4, further comprising: a housing that houses the lighting unit, the camera unit, the trained neural network storage unit, and the inference processing unit, The housing is provided with an interface that accepts various settings from a user, via said interface, A quality determination mode for determining the quality of a workpiece image; You can select one of the sorting modes to sort the input workpiece image. The image inspection device is configured such that the inference processing unit executes either the first inference processing or the second inference processing according to a selected mode.

6. The image inspection device according to any one of claims 1 to 5, The inference processing unit is configured to classify workpieces determined to be non-defective in the first inference processing into multiple values ​​in the second inference processing.

7. The image inspection device according to any one of claims 1 to 6, further comprising: an output unit having a plurality of output ports for outputting a result of the second inference processing by the inference processing unit; The plurality of output ports are assigned as output destinations for each type sorted in the second inference process.

8. The image inspection device according to any one of claims 1 to 7, further comprising: a sorting base selection unit for selecting whether the second inference processing performed by the inference processing unit is to be performed using learning-based learning sorting or rule-based standard sorting; The learning sorting can be performed using one tool, The standard sorting is an image inspection device configured to set up multiple tools for each type and execute each tool in sequence.

9. An image inspection method in which an illumination light is irradiated from an illumination unit onto a work to be inspected, the light reflected by the work is received by a camera unit, an image of the work is generated, and a pass / fail judgment or a sorting judgment of the work is performed, a step of preparing one or more trained neural networks each having a weighting coefficient of each layer trained in advance, the trained neural networks including an input layer to which the workpiece image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting the feature quantities of the input workpiece image, and storing the trained neural networks in a trained neural network storage unit; A step of capturing an image of a non-defective workpiece with the camera unit to generate a non-defective image; A step of capturing an image of a defective workpiece with the camera unit to generate a defective workpiece image; a step of inputting the non-defective product images and the defective product images into the neural network stored in the trained neural network storage unit, acquiring a plurality of non-defective product features and defective product features that characterize each of the non-defective product images and the defective product images, and setting a pass / fail judgment boundary for judging the pass / fail of the workpiece in a feature space of the neural network based on the acquired plurality of non-defective product features and defective product features; A process of capturing an image of a first type of workpiece and an image of a second type of workpiece classified as a type different from the first type by the sorting judgment using the camera unit, and generating an image of a first type of workpiece and an image of a second type of workpiece; a learning sorting tool setting process for inputting the first type workpiece image and the second type workpiece image into the neural network stored in the trained neural network storage unit, acquiring a plurality of sorting features that characterize each type of workpiece image, and setting a sorting boundary for classifying the workpieces in the feature space of the neural network based on the acquired plurality of sorting features; A step of capturing an image of a workpiece for inspection with the camera unit to generate an inspection workpiece image; a step of inputting the inspection workpiece image into the neural network stored in the trained neural network storage unit to acquire pass / fail feature quantities; a step of determining whether a workpiece image is good or bad as a first inference process based on the obtained good or bad feature amount and the good or bad determination boundary; a step of inputting the inspection workpiece image into the neural network stored in the trained neural network storage unit to acquire sorting features; a step of sorting the workpiece image as a second inference process based on the obtained sorting feature amount and the sorting boundary; Including, The learning sorting tool setting step includes: a step of accepting registration of a first master image, which is the first type workpiece image to be input to the neural network; a step of accepting setting of a detection window as an area to be referenced on the selected first type workpiece image; a step of accepting registration of a second master image, which is the second type work image to be input to the neural network after the detection window is set; Including, the sorting boundary is input to the neural network using the first master image and the second master image; the classification feature used in the second inference process corresponds to the setting of the detection window, Imaging methods.

10. An image inspection method in which an illumination light is irradiated from an illumination unit onto a work to be inspected, the light reflected by the work is received by a camera unit, an image of the work is generated, and a pass / fail judgment or a sorting judgment of the work is performed, a step of preparing one or more trained neural networks each having a weighting coefficient of each layer trained in advance, the trained neural networks including an input layer to which the workpiece image is input, an intermediate layer connected to the input layer, and an output layer connected to the intermediate layer and outputting the feature quantities of the input workpiece image, and storing the trained neural networks in a trained neural network storage unit; a step of taking an image of a non-defective workpiece with the camera unit and inputting a generated non-defective image and a generated defective image of a defective workpiece with the camera unit into the neural network stored in the trained neural network storage unit; a step of capturing images of a plurality of different types of workpieces with the camera unit, and inputting the generated type workpiece images into the neural network stored in the trained neural network storage unit; a quality determination criterion for determining whether a workpiece is good or bad in the feature space of the neural network based on a plurality of good product features and defective product features that characterize each of the good product images and defective product images; and a step of setting sorting criteria for classifying the workpieces in the feature space of the neural network based on a plurality of sorting feature quantities that characterize each type of workpiece image; a step of capturing an inspection workpiece image with the camera unit and inputting the generated inspection workpiece image into the neural network stored in the trained neural network storage unit; A first inference process is performed to determine whether the workpiece image is good or bad based on an inspection feature that characterizes the inspected workpiece image and the pass / fail determination criterion, and if the workpiece image is determined to be good, a second inference process is performed to sort the workpiece image based on the inspection feature and the sorting criterion; outputting the sorting results; Including, An image inspection method in which the pass / fail judgment criteria are set by inputting a first master image as the good product image and a second master image different from the first master image as the defective product image, and the sorting criteria are set by inputting the first master image as the work image of a first type and the second master image of a second type different from the first type.

11. an illumination unit that irradiates illumination light onto the workpiece to be inspected; a camera unit that receives light irradiated from the lighting unit and reflected by the workpiece, and generates a workpiece image; an input layer to which the workpiece image is input; a hidden layer coupled to the input layer; an output layer connected to the intermediate layer and outputting the feature quantities of the input workpiece image, and a trained neural network storage unit storing one or more neural networks in which the weight coefficients of each layer have been trained in advance; An image inspection program to be executed by a computer including an inference processing unit that performs a quality judgment or a work sorting judgment based on the work image, a function of taking an image of a non-defective workpiece with the camera unit and inputting a generated non-defective image and a generated defective image of a defective workpiece with the camera unit into the neural network stored in the trained neural network storage unit; A function of capturing images of a plurality of different types of workpieces with the camera unit and inputting the generated type workpiece images into the neural network stored in the trained neural network storage unit; a quality determination criterion for determining whether a workpiece is good or bad in the feature space of the neural network based on a plurality of good product features and defective product features that characterize each of the good product images and defective product images; and a function of setting sorting criteria for classifying workpieces in the feature space of the neural network based on a plurality of sorting feature quantities that characterize each type of workpiece image; a function of capturing an image of a workpiece for inspection with the camera unit and inputting the generated inspection workpiece image into the neural network stored in the trained neural network storage unit; A first inference process is performed to determine whether the workpiece image is good or bad based on an inspection feature amount that characterizes the inspected workpiece image and the pass / fail determination criterion, and if the workpiece image is determined to be good, a second inference process is performed to sort the workpiece image based on the inspection feature amount and the sorting criterion; A function to output sorting results, An image inspection program for causing a computer to realize the above, An image inspection program in which the function of setting the pass / fail judgment criteria and the sorting criteria is a function of setting the pass / fail judgment criteria by inputting a first master image as the good product image and a second master image different from the first master image as the defective product image, and setting the sorting criteria by inputting the first master image as the work image of a first type and the second master image of a second type different from the first type.

12. A computer-readable recording medium or storage device on which the program according to claim 11 is recorded.

Images