Image inspection system

Abstract

This facilitates the creation of datasets using scanned images. [Solution] The image inspection system 1 performs inspection on work images, which are images captured by the imaging unit. The image inspection system 1 comprises an image storage device 6 and a UI device 5 connected to the image storage device 6. The image storage device 6 has a storage unit that stores inspected images, which are work images on which inspection has been performed. The UI device 5 has a control unit that displays inspected images and can accept operations from user U. The control unit displays a list of inspected images stored in the storage unit, accepts the selection of an image from the displayed list of inspected images and instructions to add it to a dataset, moves or copies the selected image from the storage unit to the UI device 5 and adds it to the dataset, performs model training based on the first image data included in the dataset, and performs testing of the trained model based on the second image data included in the dataset.

Description

Technical Field

[0007]

[0001] The present invention relates to an image inspection system.

Background Art

[0002] In recent years, apparatuses that perform identification and classification of image data using a machine learning model have been known. For example, Patent Document 1 discloses a configuration in which an inspected image is added as a learning image of a machine learning model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in applications that perform complex image processing or image inspection, it is desirable to evaluate learning system tools, so-called AI [Artificial Intelligence] tools, that constitute the image processing or image inspection, and improve the accuracy of operation.

[0005] For example, consider a case where it is evaluated how accurately a learning system tool can output an inference result for an input work image. In this case, simply adding an inspected image as a learning image is not sufficient, and it is necessary to create a data set necessary for setting and operating the learning system tool using the inspected image. However, there was room for improvement in the procedure for creating a data set using an inspected image.

[0006] An object of the present invention is to provide an image inspection system capable of easily creating a data set from an inspected image.

Means for Solving the Problems

[0007] For example, the image inspection system according to the present invention is An image inspection system that performs inspection on work images, which are images captured by an imaging unit, An image storage device having a storage unit that stores the inspected image, which is the work image on which the inspection has been performed, The system includes a UI [User Interface] device connected to the image storage device, which has a control unit that displays the inspected image and can accept user operations, The control unit, The inspected images stored in the memory unit are displayed in a list. The system accepts the selection of an image from the list of previously inspected images and instructions to add it to the dataset. The selected image is moved or copied from the storage unit to the UI device and added to the dataset. The model is trained based on the first image data included in the aforementioned dataset. This may be an image inspection system that performs a test of the trained model based on a second image data set included in the aforementioned dataset.

[0008] Furthermore, for example, the image inspection system according to the present invention is An image inspection system that performs inspection on work images, which are images captured by an imaging unit, An image storage device having a storage unit that stores the inspected image, which is the work image on which the inspection has been performed, The system comprises a UI device connected to the image storage device, which has a control unit that displays the inspected image and can accept user operations, The control unit, After the inspection is performed, the system transitions to a settings screen that includes a list of the inspected images stored in the memory unit. The system accepts the selection of an image from the list of previously inspected images. After accepting the selection of the aforementioned image, the system accepts instructions to add the image to the dataset. Move or copy the selected image from the storage unit to the UI device and add it to the dataset. It may be an image inspection system.

Advantages of the Invention

[0009] According to the present invention, it becomes possible to easily create a dataset from inspected images.

Brief Description of the Drawings

[0010] [Figure 1] It is a diagram showing a first configuration example (controller type) of an appearance inspection device. [Figure 2] It is a diagram showing a second configuration example (smart camera type) of an appearance inspection device. [Figure 3] It is a diagram showing a first example of dataset addition processing. [Figure 4] It is a diagram showing a second example of dataset addition processing. [Figure 5] It is a diagram showing the GUI [Graphical UI] transition in the first example (Figure 3) of dataset addition processing. [Figure 6] It is a diagram showing the GUI transition in the second example (Figure 4) of dataset addition processing. [Figure 7] It is a diagram showing a first display example of a dataset edit dialog. [Figure 8] It is a diagram showing a second display example of a dataset edit dialog. [Figure 9] It is a diagram showing the usage form of a removable memory in the second configuration example. [Figure 10] It is a diagram showing the GUI transition in the third example of dataset addition processing. [Figure 11] It is a diagram showing the GUI transition in the fourth example of dataset addition processing.

Modes for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail based on the drawings. Note that the following description of the preferred embodiments is merely illustrative in nature and is not intended to limit the present invention, its applications, or its uses.

[0012] <Configuration of Appearance Inspection Device 1 (First Configuration Example)> FIG. 1 is a schematic diagram showing a first configuration example (controller type) of an appearance inspection device 1 according to an embodiment of the present invention. The appearance inspection device 1 is a device for performing a pass / fail determination of a work image obtained by imaging a work that is an inspection target such as various parts and products, and outputting the result of the pass / fail determination to an external device (not shown) connected to the external inspection device 1, and can be used at a production site such as a factory. Specifically, a machine learning network is constructed inside the appearance inspection device 1. A plurality of machine learning networks are constructed in the appearance inspection device 1, including a machine learning network generated by learning a good product image corresponding to a good product and a defective product image corresponding to a defective product, and a machine learning network generated only by a good product image corresponding to a good product. A work image obtained by imaging a work to be inspected is input to the generated machine learning network, and the pass / fail determination of the work image can be performed by the machine learning network. Thus, the appearance inspection device 1 can be understood as one aspect of an image inspection system that executes an inspection on a work image that is an image captured by an imaging unit.

[0013] The entire work may be the inspection target, or only a part of the work may be the inspection target. Also, a single work may include a plurality of inspection targets. Further, a work image may include a plurality of works.

[0014] The visual inspection device 1 comprises a control unit 2, which serves as the main body of the device; an imaging unit 3; a display device (display unit) 4; and a personal computer 5. The control unit 2 can be understood as a controller that controls the visual inspection device 1. The personal computer 5 is not essential and can be omitted. Various information and images can be displayed using the personal computer 5 instead of the display device 4, or the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0015] Figure 1 shows an example of the configuration of the visual inspection device 1, consisting of a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5. However, any combination of these components can be integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. A second configuration example (smart camera type) in which the control unit 2 and the imaging unit 3 are integrated will be described in detail later.

[0016] Furthermore, the control unit 2 can be divided into multiple units, with some parts incorporated into the imaging unit 3 or display device 4, or the imaging unit 3 can be divided into multiple units, with some parts incorporated into other units. Alternatively, the personal computer 5 can be run with the control software for the imaging unit 3, thereby implementing the functionality of the control unit 2 on the personal computer 5. In this case, the imaging unit 3 and the personal computer 5 can be connected without going through the control unit 2.

[0017] <Configuration of imaging unit 3> The imaging unit 3 comprises a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that performs the acquisition of work images. The camera module 14 comprises an AF (Auto Focus) motor 141 that drives the imaging optical system and an imaging substrate 142. The AF motor 141 is the part that automatically adjusts the focus by driving the lens of the imaging optical system, and can perform focus adjustment using conventional methods such as contrast autofocus. The imaging substrate 142 is equipped with a CMOS (Complementary Metal Oxide Semiconductor) sensor 143 as a light-receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of the CMOS sensor 143, a light-receiving element such as a CCD (Charge Coupled Device) sensor can also be used.

[0018] The lighting module 15 includes an LED (Light Emitting Diode) 151 as a light-emitting element that illuminates the imaging area including the workpiece, and an LED driver 152 that controls the LED 151. The timing of light emission, the duration of light emission, and the amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be integrated with the imaging unit 3, or it may be provided as a separate external lighting unit.

[0019] <Configuration of display device 4> The display device 4 has a display panel made of, for example, a liquid crystal panel or an organic EL (Electro-Luminescence) panel. Work images and user interface images output from the control unit 2 are displayed on the display device 4. Furthermore, if the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used as a substitute for the display device 4.

[0020] <Operation equipment> Examples of operating devices for a user to operate the visual inspection device 1 include the keyboard 51 and mouse 52 of a personal computer 5, but are not limited to these; any device configured to accept various operations by the user is acceptable. For example, a pointing device such as the touch panel 41 of a display device 4 is also included as an operating device.

[0021] User operations on the keyboard 51 and mouse 52 can be detected by the control unit 2. The touch panel 41 is a conventional touch-type control panel equipped with, for example, a pressure sensor, and user touch operations can be detected by the control unit 2. The same applies when other pointing devices are used.

[0022] <Configuration of Control Unit 2> The control unit 2 comprises a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The main board 13 is equipped with a processor 13a. The processor 13a controls the operation of each connected board and module. For example, the processor 13a outputs a lighting control signal to the LED driver 152 of the lighting module 15 to control the on / off state of the LED 151. The LED driver 152 switches the on / off state of the LED 151 and adjusts the on / off time, as well as the light intensity of the LED 151, in response to the lighting control signal from the processor 13a.

[0023] Furthermore, the processor 13a outputs an imaging control signal to the imaging substrate 142 of the camera module 14 to control the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13a and adjusts the exposure time to an arbitrary time to perform imaging. In other words, the imaging unit 3 images the area within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13a. If there is a workpiece within the field of view, the workpiece will be imaged, but if there are objects other than workpieces within the field of view, those can also be imaged. For example, the visual inspection device 1 can use the imaging unit 3 to capture images of good products corresponding to good products and images of defective products corresponding to defective products as training images for a machine learning network. The training images do not have to be images captured by the imaging unit 3; they may be images captured by other cameras, etc.

[0024] On the other hand, when the visual inspection device 1 is in operation, the imaging unit 3 can capture images of the workpiece. Furthermore, the CMOS sensor 143 is configured to output live images, i.e., the currently captured images, in real time at a short frame rate.

[0025] Once imaging by the CMOS sensor 143 is complete, the image signal output from the imaging unit 3 is input to the processor 13a on the main board 13 for processing and stored in the memory 13b (storage unit) of the main board 13. Details of the specific processing performed by the processor 13a on the main board 13 will be described later. The main board 13 may also be equipped with processing devices such as an FPGA (Field Programmable Gate Array) or a DSP (Digital Signal Processor). Alternatively, the processor 13a may be an integrated processor with an FPGA or DSP.

[0026] The connector board 16 is the part that receives power from the outside via a power connector (not shown) provided on the power interface 161. The power supply board 18 is the part that distributes the power received by the connector board 16 to each board and module, and specifically distributes power to the lighting module 15, camera module 14, main board 13, and communication board 17. The power supply board 18 is equipped with an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14, thereby realizing autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a of the main board 13.

[0027] The communication board 17 is the part that performs communication between the main board 13 and the display device 4 and the personal computer 5, and communication between the main board 13 and external control equipment (not shown). Examples of external control equipment include programmable logic controllers. Communication may be wired or wireless, and either communication method can be implemented using conventionally known communication modules.

[0028] The control unit 2 is equipped with a storage device 19, which may consist of, for example, a solid-state drive (SSD) or a hard disk drive (HDD). The storage device 19 stores program files 80 and configuration files (software) that enable the execution of the various controls and processes described later by hardware. The program files 80 and configuration files can be stored on a storage medium 90, such as an optical disc, and the program files 80 and configuration files stored on this storage medium 90 can be installed on the control unit 2. The program files 80 may also be downloaded from an external server using a communication line. In addition, the storage device 19 can also store, for example, the image data and parameters for building the machine learning network of the visual inspection device 1.

[0029] In other words, the processor 13a of the visual inspection device 1 is configured to read parameters stored in the memory device 19 to construct a machine learning network, input a work image of the work to be inspected into the constructed machine learning network, and perform a quality determination of the work based on the input work image. By using this visual inspection device 1, a visual inspection method that determines the quality of a work based on a work image can be performed. The machine learning network may also be understood as a machine learning model. In this embodiment, for the sake of explanation, the visual inspection device 1 performs a quality determination, but it may also perform a determination that classifies the work image into an arbitrary class. That is, the configuration may treat "good product" and "defective product" described for the work image as arbitrary classes.

[0030] <Configuration of Visual Inspection Device 1 (Second Configuration Example)> Figure 2 shows a second configuration example (smart camera type) of the visual inspection device 1. In this figure, the visual inspection device 1 is equipped with a smart camera 6 instead of the control unit 2 and imaging unit 3 described above. In addition, the personal computer 5 may be equipped with a display 53 in addition to the keyboard 51 and mouse 52 described above. Also, in this figure, a control unit 54 is explicitly shown as a component of the personal computer 5.

[0031] The personal computer 5 can be understood as an example of a UI device connected to the smart camera 6 that accepts user input. For example, the personal computer 5 accepts user input and configures and instructs the smart camera 6 to operate. In other words, in the second configuration example of the visual inspection device 1, the smart camera 6 configuration function, which was handled by the control unit 2 in the first configuration example (Figure 1), has been transferred to the personal computer 5.

[0032] The display 53 displays inspected images acquired by the smart camera 6, as well as a GUI for configuring various settings of the smart camera 6. The inspected images can be understood as work images on which inspection has been performed by the smart camera 6. In other words, the inspected images can be understood as operation history images acquired when the smart camera 6 is in operation mode.

[0033] The control unit 54 displays the inspected images and GUI on the display 53. The control unit 54 can also accept user input via the keyboard 51 and mouse 52. Furthermore, the control unit 54 has the function of configuring and instructing the smart camera 6 in response to user input.

[0034] The smart camera 6 receives settings and operation instructions from the personal computer 5. The smart camera 6 integrates the aforementioned control unit 2 and imaging unit 3. Specifically, the smart camera 6 comprises the aforementioned main board 13, camera module 14, lighting module 15, connector board 16, communication board 17, power supply board 18, and storage device 19.

[0035] For example, the processor 13a mounted on the main board 13 functions as an inspection unit that performs inspection of the workpiece image. The inspection of the workpiece image may be performed based on setting information set according to setting operations received by the personal computer 5. The setting information may be various parameters of a setting tool.

[0036] The workpiece image that has been inspected by processor 13a, i.e., the inspected image, is stored in memory 13b. In this way, memory 13b functions as a storage unit for storing inspected images.

[0037] The internal configuration of the smart camera 6 is merely an example. For instance, the consolidation or separation of circuit boards is optional.

[0038] <Considerations regarding the creation of datasets> Incidentally, when the visual inspection device 1 is used for applications involving complex image processing or image inspection, it is desirable to evaluate the configured learning tool and improve the accuracy of its operation. In order to improve the learning accuracy of the learning tool and to properly evaluate the configured learning tool, it is necessary to create a dataset using inspected images for both the configuration (training) of the learning tool and its operation for evaluation. Of the image sets included in the dataset, some of the image sets will be used for evaluating the learning tool.

[0039] The following section details the procedure for creating a dataset using examined images, particularly the process of adding examined images to the dataset.

[0040] <Dataset addition process (Example 1)> Figure 3 shows a first example of the dataset addition process. This first example corresponds to a comparative example that will be shown in contrast to the second example later. This figure schematically depicts the operations of user U, personal computer 5, and smart camera 6, as well as the information exchange between them. The entity that controls the operation of personal computer 5 can be understood as the control unit 54 mentioned earlier.

[0041] When a user U sends an operation command to the smart camera 6 via the personal computer 5, the visual inspection device 1 enters operation mode. At this time, the smart camera 6 captures an image of the object to be inspected to acquire a workpiece image (image data) and performs inference processing (inspection processing) on ​​the workpiece image. The workpiece image and the inference results are sequentially stored in the memory 13b. In other words, the operation history of the visual inspection device 1 is stored in the smart camera 6, not the personal computer 5.

[0042] When the visual inspection device 1 switches from operation mode to setting mode, display data is transmitted from the smart camera 6 to the personal computer 5. The personal computer 5 receives the above display data and displays a chronological list of thumbnails (image strips) of the inspected images acquired in operation mode. Thus, by default, the source of the display data is set to the smart camera 6.

[0043] When user U selects one image from the thumbnails of inspected images, the personal computer 5 switches to history retrieval mode. History retrieval mode can be understood as the first operating mode for user U to review previously inspected images and their inference results. In history retrieval mode, the personal computer 5 requests the image data of the inspected image selected by user U from the smart camera 6.

[0044] The smart camera 6, upon receiving a request from the personal computer 5, transmits the image data of the inspected image selected by user U and its inference result. The personal computer 5 downloads the received data from the smart camera 6 and temporarily stores it in a temporary area, and also displays its contents on the display 53. This display allows user U to confirm the selected inspected image and its inference result. The temporary area of ​​the personal computer 5 may be, for example, RAM (Random Access Memory).

[0045] However, the history playback mode is solely for viewing past inspected images and their inference results as a driving history. In other words, the history playback mode is not intended to accept operations such as annotation on inspected images selected by user U, nor is it intended to add inspected images selected by user U to a dataset.

[0046] In other words, only images stored in the non-temporary area of ​​personal computer 5 were included in the dataset; inspected images that were only temporarily stored in the temporary area of ​​personal computer 5 for viewing driving history were not included. The non-temporary area of ​​personal computer 5 could be, for example, an SSD or HDD.

[0047] Therefore, in order to add the inspected images acquired by the smart camera 6 to the dataset, the preliminary work enclosed by the dashed box in the figure, specifically the image export process, is required. Referring to this figure, first, user U must send an image export command to the smart camera 6 via the personal computer 5. The smart camera 6 receives the image export command and moves or copies all the image data of the inspected images and their inference results stored in memory 13b to the personal computer 5. The personal computer 5 downloads the received data from the smart camera 6 and saves it in a non-temporary area.

[0048] Next, user U needs to switch the source of the displayed data from smart camera 6 to personal computer 5. Upon receiving the switch instruction from user U, personal computer 5 displays a chronological list of thumbnails (image strips) of the inspected images stored in the non-temporary area.

[0049] Subsequently, user U can finally select one image to add to the dataset from the thumbnails of the inspected images, perform appropriate labeling, and then issue an instruction to add it to the dataset. Labeling may specify, for example, whether the inspected image should be added to the dataset as a good product image or a defective product image. Upon receiving the instruction from user U, personal computer 5 updates the dataset. Specifically, the addition of the inspected image selected by user U from among the multiple inspected images stored in the non-temporary area is set.

[0050] Thus, in this first example of the dataset addition process, the above series of preparatory steps are required beforehand. Therefore, creating a dataset using inspected images was not necessarily easy for user U. In particular, novice users unfamiliar with handling learning tools were at risk of getting stuck at the stage before actually using the learning tool, that is, at the stage of creating a dataset for setting up the learning tool.

[0051] In light of the above challenges, a second example of a dataset addition process that allows for easy creation of datasets from examined images is proposed below.

[0052] <Dataset addition process (second example)> Figure 4 shows a second example of the dataset addition process. In this figure, as in the first example (Figure 3) above, the operations of user U, personal computer 5, and smart camera 6, and the information exchange between them are schematically depicted. The entity that controls the operation of personal computer 5 can be understood as the control unit 54 mentioned above.

[0053] In the operating mode of the visual inspection device 1, the operating history is saved in the smart camera 6. When the visual inspection device 1 switches from operating mode to setting mode, display data is sent from the smart camera 6 to the personal computer 5, and thumbnails (image strips) of inspected images are displayed in chronological order. Up to this point, the operation is no different from the first example (Figure 3) described above.

[0054] When user U selects one image from the thumbnails of inspected images, the personal computer 5 switches from the previously described history playback mode to the PC image simulation mode. The PC image simulation mode can be understood as a second operating mode for inspecting image data stored in either the temporary or non-temporary area of ​​the personal computer 5 without using the smart camera 6. Even in PC image simulation mode, the personal computer 5 requests the image data of the inspected image selected by user U from the smart camera 6.

[0055] The smart camera 6, upon receiving a request from the personal computer 5, transmits the image data of the inspected image selected by user U and its inference result. The personal computer 5 downloads the received data from the smart camera 6 and temporarily stores it in a temporary area, and also displays its contents on the display 53. This display allows user U to confirm the selected inspected image and its inference result. This display operation is the same as in the first example described above (Figure 3).

[0056] In this PC image simulation mode, unlike the history playback mode described earlier, not only images stored in the non-temporary area but also inspected images temporarily stored in the temporary area can be added to the dataset. Therefore, when adding inspected images acquired by the smart camera 6 to the dataset, the aforementioned preparatory work (image export process shown by the dashed frame in Figure 3) is unnecessary.

[0057] Referring to this diagram, user U can appropriately label the examined image selected from the thumbnails and instruct it to be added to the dataset. Personal computer 5 updates the dataset upon receiving the instruction from user U. Specifically, the addition to the dataset is set for the examined image selected by user U and temporarily stored in the temporary area.

[0058] Thus, in this second example of dataset addition processing, user U can add the selected inspected images to the dataset simply by performing an intuitive operation called "image selection." Therefore, even if user U is a beginner, they can easily create the dataset necessary for setting up and evaluating the learning tool. As a result, high-precision inspection by the visual inspection device 1 becomes possible.

[0059] Next, we will compare the first example (Figure 3) and the second example (Figure 4) of the dataset addition process described above in more detail, illustrating their respective GUI transitions.

[0060] <GUI Transition (First Example)> FIG. 5 is a diagram showing the GUI transition in the first example (FIG. 3) of the dataset addition process. When the control program of the smart camera 6 is executed on the personal computer 5, the GUI 200 is displayed on the display 53. The display content of the GUI 200 changes according to the user's operation. In this figure, screens 200a to 200g are illustrated as the main display content of the GUI 200.

[0061] In the GUI 200, one window may be divided into a plurality of display areas. For example, the GUI 200 may include, as its basic layout, a main display area 210, a list display area 220, a dataset operation area 230, a status display area 241, and an inference result display area 242. The main display area 210 may be arranged, for example, at the center of the window. The list display area 220, the dataset operation area 230, the status display area 241, and the inference result display area 242 may be arranged, for example, so as to surround the main display area 210. In this figure, the list display area 220 has the same width as the main display area 210 and is arranged adjacent to the lower side of the main display area 210. Note that the GUI 200 may include display areas other than those described above.

[0062] In the operating mode of the visual inspection device 1, the display content of the GUI 200 becomes screen 200a (the first row of the left column). The main display area 210 may display the workpiece image W being captured by the smart camera 6 in live format. The list display area 220 may display that the workpiece image W and its inference results are being sequentially stored by the smart camera 6 (for example, "History being stored..."). The list display area 220 may include a reference box 221. The reference box 221 may be a pull-down menu for switching the reference destination of the displayed data. By default, the reference destination of the displayed data may be set to the smart camera 6. The dataset operation area 230 may display various operation buttons, etc. However, dataset operation is not possible in the operating mode of the visual inspection device 1. Therefore, the dataset operation area 230 is in a state that does not accept user operation, a so-called grayed-out state. The status display area 241 may display the operating status of the visual inspection device 1 (for example, "Operating").

[0063] When the visual inspection device 1 is switched from operation mode to setting mode, the display content of the GUI 200 becomes screen 200b (second row from the left). At this time, the list display area 220 displays a list of thumbnails (image strips) of inspected images 222 acquired by the smart camera 6 in operation mode, in chronological order. It is preferable that the display of the list of inspected images be performed automatically without requiring the user to perform any operation such as displaying the operation history. In addition, the operating status of the visual inspection device 1 (for example, "Setting") may be displayed in the status display area 241.

[0064] Screen 200c (third row from the left) shows the state in which one thumbnail of the inspected images 222 displayed in the list on screen 200b has been selected by the user. At this time, the personal computer 5 switches to history playback mode. In history playback mode, the image data of the inspected image selected by the user and its inference result are temporarily saved from the smart camera 6 to the temporary area of ​​the personal computer 5, and the contents are displayed in the inference result display area 242. This display allows the user to confirm the selected inspected image and its inference result. The inference result shows whether the inspected image was judged to be good ("OK") or defective ("NG"). In addition, a heat map showing the abnormal area may be displayed in the main display area 210.

[0065] However, as mentioned earlier, the history playback mode is solely for viewing past driving history. Therefore, in screen 200c, the dataset operation area 230 is grayed out. Consequently, in order to add the inspected images acquired by the smart camera 6 to the dataset, it is necessary to perform the image export process from the smart camera 6 to the personal computer 5 beforehand.

[0066] Screen 200d (first row of the right column) shows the state where the image export dialog 250 is displayed due to user operation. In the image export dialog 250, for example, the destination folder and file format for saving the image data may be specified. When the save button in the image export dialog 250 is clicked, all the image data of the inspected images and their inference results stored in the memory 13b of the smart camera 6 are moved or copied to the non-temporary area (save destination folder) of the personal computer 5. On the other hand, when the cancel button is clicked, the image export dialog 250 closes and the screen returns to screen 200b.

[0067] Next, the user needs to switch the reference destination of the display data from the smart camera 6 to the personal computer 5. Screen 200e (the second row in the right column) shows a state where the personal computer 5 has been selected from the pull-down menu of the reference destination box 221 by the user's operation. By such an operation, thumbnail images (image strips) of the inspected images 222 stored in the non-temporary area of the personal computer 5 are listed in chronological order in the list display area 220.

[0068] Screen 200f (the third row in the right column) shows a state where one thumbnail image of the inspected images 222 listed in screen 200e has been selected by the user's operation. At this time, the image data of the inspected image selected by the user and its inference result are displayed in the main display area 210. With this display, the user can confirm the selected inspected image and its inference result.

[0069] Also, different from the previous screen 200c, in screen 200f, the data set operation area 230 is not in a grayed-out state. Therefore, the user can directly add the inspected image selected in screen 200f to the data set.

[0070] For example, when the add button in the data set operation area 230 is clicked, an image addition dialog 260 is displayed as shown in screen 200g (the fourth row in the right column). The image addition dialog 260 may be provided with, for example, an add button as a learning image (good product / bad product) and an add button as a test image (good product / bad product). Also, the image addition dialog 260 may be provided with a learning start button.

[0071] In this way, in the first example of the data set addition process, it is finally possible to add the inspected image to the data set after going through the prior image writing process (screens 200d and 200e). Therefore, it was not always easy to create a data set using the inspected images.

[0072] <GUI Transition (Second Example)> Figure 6 shows the GUI transitions for the second example of the dataset addition process (Figure 4). In this figure, screens 200a (first row) and 200b (second row) are no different from those in the first example (Figure 5). That is, when the visual inspection device 1 is switched from operation mode to setting mode, the list display area 220 displays a list of thumbnails (image strips) of inspected images 222 acquired by the smart camera 6 in operation mode in chronological order.

[0073] Screen 200h (third row) shows the state where, through user operation, one thumbnail of the inspected images 222 displayed in the list on screen 200b has been selected. At this time, the personal computer 5 switches from the history playback mode described above to the PC image simulation mode. In PC image simulation mode, inspected images temporarily stored in the temporary area of ​​the personal computer 5 from the smart camera 6 can also be added to the dataset.

[0074] Therefore, unlike the previously shown screen 200c (Figure 5), in screen 200h, the dataset manipulation area 230 does not become grayed out even if the reference destination of the displayed image remains the smart camera 6. As a result, the user can add the inspected image selected in screen 200h directly to the dataset.

[0075] For example, when the add button in the dataset manipulation area 230 is clicked, the image add dialog 260 is displayed, as shown in screen 200i (fourth row). The image add dialog 260 may include, for example, buttons to add images as training images (good / bad) and buttons to add images as test images (good / bad). By clicking these add buttons, the user can label any inspected image and add it to the dataset as either a training image or a test image. The image add dialog 260 may also include a start training button. These points are the same as those shown in screen 200g (Figure 5) above.

[0076] Thus, in this second example of dataset addition processing, the prior image export process (screens 200d and 200e in Figure 5) becomes unnecessary. Therefore, users can easily create the dataset necessary for setting up and evaluating learning tools using the inspected images.

[0077] Furthermore, when implementing the second example of the dataset addition process (Figures 4 and 6) in the visual inspection device 1, the operation flow of the control unit 54 can be summarized as follows. First, after performing an inspection on the workpiece image, the control unit 54 transitions the display content of the display 53 to a settings screen (for example, screen 200a → screen 200b) that includes a list of inspected images stored in the memory 13b of the smart camera 6. Next, the control unit 54 accepts the selection of an image from the list of inspected images (for example, screen 200h). After accepting the image selection, the control unit 54 accepts an instruction to add it to the dataset (for example, screen 200i). At this time, as shown in Figure 4, the control unit 54 moves or copies the selected image from the memory 13b of the smart camera 6 to the temporary area of ​​the personal computer 5 and adds the inspected image temporarily stored in the temporary area to the dataset.

[0078] Alternatively, the operation flow of the control unit 54 can also be understood as follows: First, the control unit 54 displays a chronological list of inspected images stored in the smart camera 6's memory 13b on the display 53 (e.g., screen 200b). Next, the control unit 54 receives a request to select an image from the displayed list of inspected images (e.g., screen 200h) and an instruction to add it to the dataset (screen 200i). At this time, as shown in Figure 4, the control unit 54 moves or copies the selected image from the smart camera 6's memory 13b to the temporary area of ​​the personal computer 5 and adds the inspected image temporarily stored in the temporary area to the dataset.

[0079] Thus, in the second example of the dataset addition process (Figures 4 and 6), after the inspection of the work images is completed, the entire dataset addition process, from displaying a list of inspected images to accepting the selection of inspected images and accepting the request to add them to the dataset, is completed within a single screen of GUI200.

[0080] More specifically, after the inspection of the work images is complete, it is possible to add any inspected image to the dataset through a simple and intuitive operation: select the desired inspected image from the list display area 220, which is always displayed in GUI200, and then click the add button in the dataset operation area 203, which is also always displayed in GUI200. Such a GUI200 is particularly convenient for novice users who are unfamiliar with handling learning tools.

[0081] When the edit button in the dataset manipulation area 130 is clicked, the dataset editing dialog 300 is displayed. The dataset editing dialog 300 is described in detail below.

[0082] <Dataset Editing Dialog 300 (First Display Example)> Figure 7 shows a first display example of the dataset editing dialog 300. The dataset editing dialog 300 includes, for example, a learning button 310, a verification button 320, a settings / editing area 330, a toolbar area 340, and a thumbnail area 350.

[0083] When the learning button 310 is clicked, the machine learning model is trained (additional training) based on the first image data specified as the training image from among the multiple images 351 included in the dataset.

[0084] When the validation button 320 is clicked, the trained machine learning model is tested (validated and evaluated) based on the second image data specified as the test image from among the multiple images 351 included in the dataset.

[0085] The settings / editing area 330 may display various information related to the settings and editing of the dataset. Referring to this figure, the labeling table 331 is displayed in the settings / editing area 330.

[0086] Labeling table 331 may display the number of training images and test images for each classification label for all images included in the dataset. In this figure, it is shown that there are 13 training images and 25 test images classified as type A. The same applies to images classified as type B and type C, respectively, as well as unclassified images.

[0087] The toolbar area 340 may display various toolbars 341. Examples of toolbars 341 include "Image Update," "Learning Target," "Result Reflection," "Display Switch," "Zoom In," "Zoom Out," "Report," and "Settings."

[0088] The thumbnail area 350 may display multiple images 351 included in the dataset as thumbnails. The multiple images 351 may include inspected images selected by the user. Among the multiple images 351, the image 351 that the user has set as the training target may be marked with a training image mark 352.

[0089] The control unit 54 may accept an instruction to designate any first image from among the multiple images 351 added to the dataset as the first image data, and may designate the first image as the first image data. Alternatively, the control unit 54 may designate a second image that was not instructed to be the first image data as the second image data. In other words, the control unit 54 may designate the image selected by the user as the training image from among the multiple images 351 included in the dataset, and use the remainder as test images.

[0090] For example, a novice user might be satisfied with simply adding the inspected images displayed as an image strip as training images, and forget to add them as test images. With this configuration, users can effectively utilize images they did not use for training as test images without requiring any special operation.

[0091] <Dataset Editing Dialog 300 (Second Display Example)> Figure 8 shows a second display example of the dataset editing dialog 300. The second display example shown in this figure corresponds to the results of the test performed by clicking the verification button 320. Note that the dataset editing dialog 300 can also display the test results by selecting, for example, only the test images.

[0092] As shown in this figure, the settings / editing area 330 of the dataset editing dialog 300 may display the confusion matrix 332, the evaluation results 333, and the separation degree graph 334 as test results (summary).

[0093] The confusion matrix 332 displays the user-defined classes (type A / B / C / unclassified) and the classes obtained as output results of the machine learning model (type A / B / C) in matrix format.

[0094] Referring to this diagram, the first row, first column displays the number of images where both the setting and the result are type A. The second row, second column displays the number of images where the setting is type A and the result is type B. The third row, third column displays the number of images where the setting is type A and the result is type C. The fourth row, first column displays the total number of images where the setting is type A.

[0095] The second row, first column displays the number of images where the setting is type B and the result is type A. The second row, second column displays the number of images where both the setting and the result are type B. The second row, third column displays the number of images where the setting is type B and the result is type C. The second row, fourth column displays the total number of images where the setting is type B.

[0096] The third row, first column displays the number of images where the setting is type C and the result is type A. The third row, second column displays the number of images where the setting is type C and the result is type B. The third row, third column displays the number of images where both the setting and the result are type C. The third row, fourth column displays the total number of images where the setting is type C.

[0097] The fourth row, first column displays the number of images that have not been categorized (i.e., are unclassified in the settings) and whose result is type A. The fourth row, second column displays the number of images that are unclassified and whose result is type B. The fourth row, third column displays the number of images that are unclassified and whose result is type C. The fourth row, fourth column displays the total number of images that are unclassified. In this way, the confusion matrix 332 shows the number of images that are unclassified in the settings, which can notify the user if they have forgotten to set a category.

[0098] The 5th row, 1st column displays the total number of images whose result is type A. The 5th row, 2nd column displays the total number of images whose result is type B. The 5th row, 3rd column displays the total number of images whose result is type C. The 5th row, 4th column displays the total number of all images.

[0099] As shown above, the confusion matrix 332 contains multiple matrix components. This display makes it easier to visually grasp the inference performance of the machine learning model being validated. For example, the larger the numbers on the diagonal of the confusion matrix 332, i.e., the total number of true images, the higher the inference performance of the machine learning model. A true image refers to an image in which the classification labels for both the setting and the result match.

[0100] The evaluation results 333 may include accuracy and agreement (average, maximum, minimum), as shown in this figure. Accuracy can be calculated as the ratio of true images to all images.

[0101] The separation graph 334 shows the correlation between the degree of agreement (horizontal axis) and the count (vertical axis) for each classification label. In other words, the separation graph 334 displays the frequency distribution for each classification label in graph format.

[0102] In the example shown in this figure, the areas with a high degree of agreement (hatched areas labeled "OK" in the figure) and the areas with a low degree of agreement (hatched areas labeled "NG" in the figure) are clearly separated. In such cases, it can be judged that the inference performance of the machine learning model is sufficient and the desired output result has been obtained. On the other hand, although not shown again in the figure, if there is overlap between the areas with a high degree of agreement and the areas with a low degree of agreement, it is considered that the inference performance of the machine learning model is insufficient and further machine learning (parameter tuning) is required. In this embodiment, a separation graph 334 showing the areas with a high degree of agreement and the areas with a low degree of agreement is displayed as an indicator of inference performance, but it is also possible to have a configuration in which the display is based on numerical values ​​according to the target learning tool. For example, if it is a learning tool that performs anomaly detection, the display may have the anomaly level on the horizontal axis.

[0103] <Usage patterns of removable memory 7> Figure 9 shows the usage configuration of the removable memory 7 in the visual inspection device 1 of the second configuration example (Figure 2). As shown in this figure, the removable memory 7 may be attached to and detached from the memory 13b of the smart camera 6 as a storage unit for storing inspected images and their inference results. The removable memory 7 can be attached to and detached from not only the smart camera 6 but also the personal computer 5. The personal computer 5 can specify the removable memory 7 attached to the smart camera 6 as the storage location for inspected images and their inference results. For example, an SD memory card can be preferably used as the removable memory 7.

[0104] The following describes in detail the procedure for adding the inspected images stored in removable memory 7 to the dataset.

[0105] <Dataset addition process (Example 3)> Figure 10 shows the GUI transitions in the third example of the dataset addition process. This third example corresponds to a comparative example that will be shown later in relation to the fourth example. This figure shows the GUI transitions when there is no function to directly display the inspected images stored in the removable memory 7 in the list display area 220; in other words, when the removable memory 7 is not selected as the reference destination for the display data in the reference box 221.

[0106] In this case, as shown in screens 200j and 200k (the first and second rows in the left column), the user must open the image transfer dialog 270 and transfer the inspected images in advance from the removable memory 7 to the non-temporary area of ​​the personal computer 5.

[0107] The steps after image transfer are as described in screens 200e to 200g in Figure 5 (the first, second, and third rows in the right column of this figure). Specifically, the user switches the data reference destination to personal computer 5, selects one image from the thumbnails of the inspected images 222, and clicks the add button in the dataset manipulation area 230.

[0108] Thus, in this third example of dataset addition processing, the inspected images can only be added to the dataset after prior image transfer processing (screens 200j and 200k). Therefore, creating a dataset using inspected images was not always easy.

[0109] <Dataset addition process (Example 4)> Figure 11 shows the GUI transitions in the fourth example of the dataset addition process. In GUI 200 in this figure, it is possible to directly display the inspected images stored in removable memory 7 in the list display area 220.

[0110] More specifically, the user can switch the reference destination of the displayed data in the reference box 221 to the removable memory 7, as shown in screen 200x~200z, then select any one of the thumbnails of the inspected images 222 displayed in the list display area 220, and click the add button in the dataset operation area 230.

[0111] Thus, in this fourth example of dataset addition processing, the prior image transfer processing (screens 200j and 200k in Figure 10) becomes unnecessary. Therefore, users can easily create the dataset necessary for setting up and evaluating learning tools using the inspected images.

[0112] <Variation> In the above embodiment, a second configuration example (Figure 2) in which the visual inspection device 1 is a smart camera type is given as an example, and a procedure for adding inspected images to a dataset is proposed. However, even in the first configuration example (Figure 1) in which the visual inspection device 1 is a controller type, if the inspected images are stored in the control unit 2, it is necessary to write the inspected images from the control unit 2 to the personal computer 5 as a preliminary step for the dataset addition process.

[0113] In light of this, the dataset addition process proposed in the above embodiment can be understood as applicable to image inspection systems in general, where the device that stores images (image storage device) and the device that creates the dataset (UI device) are different. [Explanation of symbols]

[0114] 1. Visual inspection device (image inspection system) 2 Control Unit 3. Imaging Unit 4 Display device (display section) 41 Touch panel 5. Personal computer (UI device) 51-key keyboard 52 mice 53 displays 54 Control Unit 6. Smart Camera (Image Storage Device) 7 Removable Memory 13 Main board 13a processor 13b Memory (storage unit) 14. Camera module (imaging unit) Motor for 141 AF 142 Imaging substrate 143 CMOS sensor 15. Lighting module (lighting section) 151 LEDs (Light Emitting Diodes) 152 LED Drivers 16 Connector board 161 Power Interface 17 Communication board 18 Power supply board Motor driver for 181 AF 19 Storage device 80 Program Files 90 Storage medium 200 GUI 200a~200k, 200x~200z screen 210 Main display area 220 List display area 221 Reference Box 222 Examined Images 230 Dataset manipulation area 241 Status display area 242 Inference result display area 250 Image Export Dialog 260 Image Addition Dialog 270 Image Transfer Dialog 300 Dataset Editing Dialog 310 Learning Button 320 Verification button 330 Settings / Editing Area 331 Labeling Table 332 confusion matrix 333 Evaluation Results 334 Separation degree graph 340 Toolbar area 341 Toolbar 350 thumbnail area 351 images 352 Learning Image Mark U User W Work Image

Claims

1. An image inspection system that performs inspection on work images, which are images captured by an imaging unit, An image storage device having a storage unit that stores the inspected image, which is the work image on which the inspection has been performed, The system comprises a UI device connected to the image storage device, which has a control unit that displays the inspected image and can accept user operations, The control unit, The inspected images stored in the memory unit are displayed in a list. The system accepts the selection of an image from the list of previously inspected images and instructions to add it to the dataset. The selected image is moved or copied from the storage unit to the UI device and added to the dataset. The model is trained based on the first image data included in the aforementioned dataset. An image inspection system that performs a test of the trained model based on a second image data set contained in the aforementioned dataset.

2. The aforementioned image storage device is The imaging unit captures images of the object to be examined, The device further includes an inspection unit that performs inspection of the work image based on setting information set according to setting operations received by the UI device, The image inspection system according to claim 1.

3. The control unit, The previously examined images are displayed in chronological order. The system receives an instruction to designate the first image as the first image data from among the images added to the dataset, and uses the first image as the first image data, and the second image for which no instruction to designate the first image data was given as the second image data, to perform training and testing of the model. The image inspection system according to claim 1 or 2.

4. An image inspection system that performs inspection on work images, which are images captured by an imaging unit, An image storage device having a storage unit that stores the inspected image, which is the work image on which the inspection has been performed, The system comprises a UI device connected to the image storage device, which has a control unit that displays the inspected image and can accept user operations, The control unit, After the inspection is performed, the system transitions to a settings screen that includes a list of the inspected images stored in the memory unit. The system accepts the selection of an image from the list of previously inspected images. After accepting the selection of the aforementioned image, the system accepts instructions to add the image to the dataset. The selected image is moved or copied from the storage unit to the UI device and added to the dataset. Image inspection system.

5. The image inspection system according to claim 1 or 4, wherein the storage unit is a removable memory that can be attached to and detached from both the image storage device and the UI device.

Images