Image inspection device

Abstract

To detect an object from a picked-up image without setting a ROI.SOLUTION: An image inspection device comprises an imaging unit, and a controlling unit that is separate from the imaging unit. The controlling unit comprises an inspection setting part and an inspection execution part. The inspection setting part extracts a first feature map from a learning image, and extracts a first feature amount corresponding to a position which is included in the first feature map, and which reflects an angle of a window relative to the learning image and is specified by a window. The inspection execution part extracts a third feature map from a picked-up image, specifies a position included in the third feature map and corresponding to the third feature amount similar to the first feature amount, determines a candidate region based on the specified position, and outputs an image showing the candidate region superposed on the picked-up image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image inspection device. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known an image inspection device that includes a "learning tool" that is a tool for setting conditions for determining whether an object is good or bad (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-164146 Summary of the Invention [Problem to be solved by the invention]

[0004] If the position and angle of the inspection target region (ROI: Region of Interest) of the "learning tool" are fixed relative to the imaging field of view, if the object shown in the inspection target image is outside the ROI, it is not possible to determine whether the object is good or bad.

[0005] To solve the above problems, for example, an image inspection device is provided with a "position correction tool" disclosed in Patent Document 1. The "position correction tool" sets a reference position and a reference angle within the imaging field of view, and adjusts the position and angle of the ROI relative to the imaging field of view according to the reference position and the reference angle.

[0006] However, if the image to be inspected does not contain visual features suitable for setting the reference position and reference angle of the "position correction tool," it is difficult to make appropriate adjustments using the "position correction tool." In particular, a separate-type image inspection device, in which the imaging unit and control unit are separate, is more likely to be installed in a position where the imaging unit and the object are close to each other because it is easier to miniaturize the housing containing the imaging unit compared to an integrated image inspection device, in which the imaging unit and the control unit are integrated. If the imaging unit and the object are close to each other, there is a high possibility that the image to be inspected will not contain visual features suitable for setting the reference position and reference angle of the "position correction tool."

[0007] In view of the above-mentioned problems, an object of the present invention is to provide an image inspection device that can detect an object from a captured image without setting an ROI. [Means for solving the problem]

[0008] The image inspection device according to the present invention comprises an imaging unit and a control unit separate from the imaging unit. The imaging unit includes an imaging section that captures an imaging field of view and generates a captured image. The control unit includes an inspection execution section that detects an object from the captured image using a machine learning model and outputs the detection result, and an inspection setting section that configures the inspection execution section. The machine learning model includes a first feature extraction section that extracts features from an input image. The inspection setting section accepts a window setting for a training image, executes the first feature extraction section to extract a first feature map from the training image, and extracts a first feature that is included in the first feature map, reflects the angle of the window relative to the training image, and corresponds to a position identified by the window. The inspection execution section executes the first feature extraction section to extract a third feature map from the captured image, identifies a position that is included in the third feature map and corresponds to a third feature similar to the first feature, determines a candidate area based on the identified position and the window setting, and outputs an image indicating the candidate area superimposed on the captured image.

[0009] Still other features, elements, steps, advantages, and characteristics will become more apparent from the detailed description that follows and the accompanying drawings related thereto. [Effects of the Invention]

[0010] The image inspection device according to the present invention can detect an object from a captured image without setting an ROI. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an image inspection device according to an embodiment of the present invention during operation. FIG. [Figure 2] FIG. 2 is a hardware configuration diagram of the image inspection device. [Figure 3] FIG. 2 is a functional block diagram of the image inspection device. [Figure 4] FIG. 10 is a diagram showing a flow in a setting mode. [Figure 5] FIG. 5 is a diagram showing a GUI (graphical user interface) transition in FIG. 4. [Figure 6] FIG. 1 is a block diagram of a machine learning model. [Figure 7] FIG. 1 is a block diagram of a machine learning model. [Figure 8] FIG. 1 is a block diagram of a machine learning model. [Figure 9] FIG. [Figure 10] FIG. [Figure 11] FIG. [Figure 12] FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0013] FIG. 1 is a diagram illustrating an operation of an image inspection device S according to an embodiment of the present invention. The image inspection device S captures an image of a workpiece W transported by a transport means A according to an imaging setting, detects the workpiece W in the captured image, and outputs the detection result to an external device. An example of the external device is a programmable logic controller (PLC) 5, but a device other than the PLC 5 may also be the external device. The PLC 5 controls the transport means A based on the received detection result, for example, to separate the storage destination of the workpiece W. In the following description, a case where the external device is the PLC 5 will be described. Note that the workpiece W may be a workpiece that is not transported by the transport means A. In the following description, the workpiece will also be referred to as the target object. Note that in the following description, the entire workpiece is referred to as the target object, but the target object may also be a portion of the workpiece.

[0014] The image inspection device S includes an imaging unit 1 for imaging a workpiece W, a control unit 2 to which the image captured by the imaging unit 1 is input, a PC (personal computer) 3 for setting up the image inspection device S, and a display device 4 for displaying a setting screen, a selection screen, workpiece images, detection results, etc. The control unit 2 is capable of executing a trained model for detecting the workpiece W in the input captured image. The control unit 2 outputs to the PLC 5 according to the detection results obtained by the trained model.

[0015] Here, the image inspection device S may be used, for example, to inspect the workpiece W from various angles at various points in a manufacturing device or production line. For this reason, multiple image inspection devices S may be installed in one manufacturing device or one production line, potentially resulting in insufficient installation space and power supply. Therefore, the image inspection device S must be compact to accommodate the installation space and power-efficient to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a GPU (graphics processing unit). The control unit 2 executes a trained model that has undergone machine learning to the extent that it is capable of detecting the workpiece W. However, the image inspection device S is provided to the user by the vendor so that the desired detection accuracy can be achieved without the user having to perform advanced machine learning, for which the use of a GPU is recommended. Details will be described later. Because the user does not have to perform advanced machine learning, the user can execute a trained model capable of detecting the workpiece W without preparing a GPU for training. Furthermore, the time required by the user to prepare a trained model capable of detecting the workpiece W can be reduced. Note that a single image inspection device S may be installed and operated in a manufacturing device or production line. The image inspection device S can also be called an image sensor.

[0016] (Configuration of imaging unit) The imaging unit 1 is separate from the control unit 2 and is installed so that it can capture images of the workpiece W from any desired direction. The workpiece W is sequentially transported by the conveying means A into the imaging field of the imaging unit 1. For example, as shown in FIG. 1 , if equipment B is located above the conveying means A at a short distance from the conveying means A, the imaging unit 1 is disposed between the conveying means A and the equipment B, and the imaging unit 1 captures an image of the workpiece W in close proximity to the conveying means A. Because the imaging unit 1 and the control unit 2 are separate, the imaging unit 1 can be made compact, allowing it to be placed in the narrow space between the conveying means A and the equipment B. Note that in FIG. 1 , the width of the equipment B in the conveying direction of the conveying means A is shown shortened for convenience in order to make the conveying means A and the workpiece W easier to see; however, in reality, the equipment B extends in the conveying direction of the conveying means A.

[0017] Fig. 2 is a hardware configuration diagram of the image inspection device S. As shown in Fig. 2, the imaging unit 1 includes an illumination module 10 for illuminating the workpiece W and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10.

[0018] The lighting module 10 has an LED (light emitting diode) 10a that irradiates light toward the workpiece W, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication section 20 (described later) of the control unit 2, and is controlled by a control section 21 (described later) of the control unit 2.

[0019] The camera module 11 has an AF motor 11a and an imaging board 11b. The AF motor 11a is a member for automatically focusing on the workpiece W by driving a focusing lens of an optical system (not shown). The autofocus method is not particularly limited, and examples include a contrast method and a liquid lens method.

[0020] The imaging board 11b is equipped with a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives light that is irradiated from the LED 10a onto the workpiece W and reflected by the workpiece W. This CMOS sensor 11c is connected to the head communication section 20 of the control unit 2, and is controlled by the control section 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0021] The FPGA 11d is a processing device whose internal processing contents can be changed. The DSP 11e is a signal processing device. A light-receiving amount signal of the light-receiving element of the CMOS sensor 11c is output to the FPGA 11d for processing, and is also output to the DSP 11e for processing. The processing by the FPGA 11d and the DSP 11e is not particularly limited, but examples thereof include various types of filter processing. The image data processed by the FPGA 11d and the DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0022] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Therefore, the control unit 2 can be installed in a location away from the imaging unit 1. In addition, a magnifying lens attachment 8 can be attached to the imaging unit 1. When the magnifying lens attachment 8 is attached to the imaging unit 1, the camera module 11 can focus on the workpiece W even when the imaging unit 1 and the workpiece W are close to each other. Note that when the imaging unit 1 and the workpiece W are far apart and the magnifying lens attachment 8 is not needed, the magnifying lens attachment 8 can be removed from the imaging unit 1.

[0023] (PC configuration) PC3 is configured as a general-purpose personal computer or the like. In this example, the personal computer can be used as PC3 by installing a predetermined program on it. PC3 includes operating devices such as a keyboard 3a and a mouse (not shown). A user of the image inspection device S can perform setting operations and selection operations on the image inspection device S by operating the operating devices on PC3. Specific setting operations and selection operations will be described later.

[0024] The PC 3 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other, and information based on setting operations by the user is sent from the PC 3 to the control unit 2. In addition, the PC 3 is able to receive image data of the workpiece W, inspection results, etc. output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Therefore, the PC 3 can be installed in a location away from where the control unit 2 is installed.

[0025] (Display device configuration) The display device 4 is configured, for example, with a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component capable of detecting operations by the user's finger. The type of the touch panel 4a is not particularly limited, and examples include a capacitive type and an infrared type. The display device 4 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other. Operation information of the touch panel 4a by the user is transmitted from the display device 4 to the control unit 2. In addition, the display device 4 is capable of receiving image data of the workpiece W output from the control unit 2. The display device 4 and the control unit 2 are connected via a communication cable 7. Therefore, the display device 4 can be installed in a location away from the installation location of the control unit 2.

[0026] The PC 3 and the display device 4 may be configured as an integrated unit. For example, the display device 4 may be configured as a display device that the PC 3 has. In this case, the main body of the PC 3 and the display device 4 may be integrated or may be separate. In this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0027] (Control unit configuration) 2, the control unit 2 includes a head communication section 20, a control section 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage section) 26. The head communication section 20 is connected to the control section 21 and is a section that executes mutual communication between the control section 21 and the imaging unit 1. A control signal for the imaging unit 1 output from the control section 21 is transmitted to the imaging unit 1 via the head communication section 20. The control signal for the imaging unit 1 includes a signal that controls the light emission timing and light emission amount of the LED 10a, and a signal that controls the AF motor 11a and the imaging board 11b. Furthermore, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the control section 21 via the head communication section 20.

[0028] The control unit 21 has a DSP 21a and FPGA 21b that perform various signal processing, an accelerator 21c that speeds up the processing, and a memory 21d made up of RAM, ROM, etc. The specific configuration of the control unit 21 will be described later.

[0029] The communication board 22 is connected to the control unit 21, and is a member that executes mutual communication between the control unit 21 and the PC 3, the display device 4, and the PLC 5.

[0030] The connector board 24 includes a power supply interface 24a. A power cable (not shown) for supplying power from an external source is connected to the power supply interface 24a. The connector board 24 is connected to a power supply 23, and power supplied from the outside to the power supply interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the control unit 21. The power supplied to the control unit 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0031] The I / O board 25 is connected to the control unit 21. The inspection results output from the control unit 21 are input to the PLC 5 via the I / O board 25.

[0032] (function block) 3 is a functional block diagram of the image inspection device S. As shown in this diagram, the image inspection device S includes an imaging setting unit 100, an inspection setting unit 200, and an inspection execution unit 300 as its functional blocks.

[0033] The imaging setting section 100 performs various settings related to the imaging operation of the imaging unit 1 (such as imaging field of view, image brightness, focus, and imaging interval (frame rate)).

[0034] The inspection setting section 200 performs various settings related to the inspection of captured images by the inspection execution section 300. Referring to this figure, the inspection setting section 200 includes a tool setting section 210 and an inspection condition setting section 220.

[0035] The tool setting unit 210 sets various tools. Referring to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0036] The tool selection unit 211 selects a tool to be set and used.

[0037] The parameter setting unit 212 sets a rule-based tool. With the rule-based tool, inspection is performed based on various feature amounts (outline, color, position, etc.) of the workpiece W shown in the image.

[0038] The learning tool setting unit 213 sets a learning tool that uses a machine learning model. In the learning tool, a trained model such as a classifier is generated in response to user instructions, and testing is performed based on the output of the trained model. Referring to the figure, the learning tool setting unit 213 includes a training data setting unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0039] The learning data setting unit 213a sets learning data to be input to the machine learning model. The learning data includes a learning image and instruction content. The learning image is an image showing a workpiece W. The instruction content is setting information for a window showing the area in which the workpiece W is shown. Referring to this figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a window setting unit 213a2, and a learning data generation unit 213a3.

[0040] The learning image selection unit 213a1 selects a learning image. The learning image may be an image captured by the imaging unit 1, or may be an image transmitted from the PC 3 and stored in the storage device .

[0041] The window setting unit 213a2 displays the learning image selected by the learning image selection unit 213a1 on the GUI and accepts window settings for the learning image.

[0042] The learning data generation unit 213a3 generates learning data based on the learning images selected by the learning image selection unit 213a1 and the window settings accepted by the window setting unit 213a2.

[0043] The update unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches an expected value in accordance with the teachings. The parameter update can be understood as learning of the machine learning model. However, the user does not necessarily have to perform all of the steps of learning the machine learning model. For example, it is possible to complete relatively computationally intensive learning on the vendor side before shipping the image inspection device S, and then have the user perform only relatively computationally intensive learning before operating the image inspection device S. In this specification, learning performed by the vendor side before shipping is referred to as pre-shipment learning, and learning performed by the user side before operating the image inspection device S is referred to as customer-side learning.

[0044] For example, the machine learning model of the image inspection device S includes a feature extraction unit that is not trained in-house and a judgment unit that is trained in-house. The feature extraction unit extracts features from the image input to the machine learning model. The judgment unit outputs an inspection result based on the features.

[0045] That is, the machine learning model of the image inspection device S may include a parameter-fixed portion. The parameter-fixed portion is a layer in which parameters obtained by pre-shipment training on the vendor side are fixed, in other words, a layer in which on-site training on the user side is not required.

[0046] With this configuration, there is no need for users to prepare equipment with the high processing power required for deep learning, such as a GPU, or for vendors to provide an advanced learning environment using a GPU as a cloud service (such as SaaS), thereby lowering the barrier to introducing the image inspection device S.

[0047] As such, the above-mentioned learning should be broadly interpreted as including not only deep learning, which requires a large amount of computation, but also learning with a small amount of computation, i.e., customer learning in this specification. Note that, since customer learning is learning with a small amount of computation, the machine learning model may be trained using a method that does not involve machine learning techniques.

[0048] The inspection condition setting unit 220 determines the output conditions of the image inspection device S, in other words, the conditions of the sensor output, for example, by combining a plurality of tools.

[0049] The inspection execution unit 300 executes an inspection process on the workpiece W shown in the captured image and outputs the inspection results. Referring to this figure, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0050] The tool execution unit 310 executes the tool selected as the tool to be used by the tool selection unit 211. Referring to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0051] When the tool selection unit 211 selects a rule-based tool as a tool to be used, the rule determination unit 311 executes the rule-based tool.

[0052] The learning tool execution unit 312 executes a learning tool when the tool selection unit 211 selects the learning tool as the tool to be used.

[0053] The inspection result output unit 320 outputs the inspection results in accordance with the output conditions set by the inspection condition setting unit 220. The inspection may include image classification, anomaly detection, and segmentation.

[0054] The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be configured solely with hardware, or may be configured with a combination of hardware and software. The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are realized by a single piece of hardware or software. The software described above can be executed by the control unit 2 (particularly the control unit 21) in which a program file and a setting file are installed.

[0055] The image inspection device S of this configuration example can be switched between a setting mode and an operation mode. In the setting mode, for example, various parameter settings such as imaging settings, registration of a master image, and generation (learning) of a classifier that classifies images based on the feature quantities of the images are performed. Note that the classification here refers to classifying images into arbitrary classes, and includes classifying images into good images and bad images, and into images that show an object and images that do not show an object.

[0056] In the operation mode, the workpiece W is inspected based on the images captured by the imaging unit 1 at the actual site. Switching between the setting mode and the operation mode can be performed on the GUI, which will be described later. It can also be configured to automatically switch to the operation mode immediately after completing the setting mode. In the operation mode, it is also possible to correct (update) the classification boundary in the classifier, that is, to perform so-called additional learning.

[0057] (Learning Search) The image inspection device S is equipped with a learning search tool that performs a "learning search" function as one of the learning tools that uses a machine learning model. The learning search tool is a tool that learns the visual features of the search object, which is the object to be searched, and searches for the search object in the captured image; in other words, it is a tool for detecting the search object in the captured image. In the following description, the term "learning search" may also be used to refer to the learning search tool.

[0058] 4 is a diagram showing a flow in the setting mode (setting flow). It can be understood that the execution subject of the setting flow shown in FIG.

[0059] Fig. 5 is a diagram showing GUI transitions corresponding to the setting flow shown in Fig. 4. When the image inspection program is executed on the PC 3, a GUI 400 is displayed on the display device 4. The GUI 400 (=various screens 400a to 400g) corresponding to the setting flow shown in Fig. 4 includes, as its basic layout, an image display area 410, an operation area 420, and a progress display area 430.

[0060] The image display area 410 displays an image captured by the imaging unit 1, etc. The image display area 410 may also be accompanied by a status display banner 411, a zoom in / out button 412, a maximize button 413, etc. The status display banner 411 clearly displays the operating status (status) of the image inspection program. The zoom in / out button 412 and the maximize button 413 are operated to zoom in / out and maximize the image displayed in the image display area 410, respectively.

[0061] When the image inspection device S is started for the first time, an initial startup screen 400a is displayed. When the image inspection device S is started for the first time, various settings of the image inspection program are incomplete. Therefore, an attention mark a1 (or an attention message) indicating that the master image has not been registered may be displayed in the image display area 410. Furthermore, a banner (e.g., "Master") indicating that the master image has not been registered (or is in the process of being registered) may be displayed in the status display banner 411.

[0062] Also, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, the same applies below), various settings for the image inspection program are started. This corresponds to the start of the setting flow shown in FIG. 4. Switching from the operation mode to the setting mode also corresponds to the start of the setting flow shown in FIG. 4.

[0063] 4, the first process (imaging setting) in step S11, the second process (master registration) in step S12, the third process (tool setting) in step S13, and the fourth process (output allocation) in step S14 are carried out in sequence. In the progress display area 430, each process is displayed in a flow diagram, and the process currently being executed is preferably highlighted. With this configuration, the user can grasp the progress of the setting work at a glance.

[0064] In the first step (imaging setting) of step S11, an imaging setting screen 400b is displayed. In the operation area 420 of the imaging setting screen 400b, for example, a slide bar b1, a back button b2, and a forward button b3 may be displayed.

[0065] When the slide bar b1 is moved by dragging, the brightness of the image is set according to the movement of the slide bar b1. Although not explicitly shown in Fig. 5, in the first step (imaging setting), the imaging field of view, focus, imaging interval (frame rate), etc. may be set.

[0066] Clicking the back button b2 returns to the initial startup screen 400a, whereas clicking the forward button b3 advances to the second step (master registration).

[0067] 5, in the first step (imaging setting), a live image (=moving image being captured) may be displayed in the image display area 410. In this case, the status display banner 411 may display a banner (for example, "Live") indicating that a live image is being displayed.

[0068] In the second process (master registration) of step S12, a master image selection screen 400c is displayed. In the operation area 420 of the master image selection screen 400c, for example, a live button c1, a history button c2, a file button c3, a back button c4, and a forward button c5 may be displayed.

[0069] When the live button c1 is clicked, the live image displayed in the image display area 410 at that time is registered as the master image. By clicking the history button c2, it is possible to register a master image from a driving history image (an inspected image). Also, by clicking the file button c3, it is possible to register a master image from a file image stored in the storage device 26. In this way, the setting methods for registering a master image include a method of registering from a live image, a method of registering from a driving history image, and a method of registering from a file image.

[0070] Clicking the back button c4 returns to the first step (imaging setting) described above, whereas clicking the forward button c5 advances to the third step (tool setting).

[0071] In the third process (tool setting) of step S13, first, a tool selection screen 400d is displayed. For example, the tool selection screen 400d may display a tool selection dialogue d0 as shown in FIG.

[0072] The tool selection dialog d0 displays, for example, a first tool (color area) button d1, a second tool (learning search) button d2, a third tool (learning OCR [Optical Character Recognition]) button d3, an OK button d4, and a cancel button d5. The color area is a rule-based tool for measuring the area of ​​a specified color. The learning search and learning OCR are both learning tools. As mentioned above, the learning search is a tool for learning the visual features of a search object and searching for the search object in a captured image. The learning OCR is a tool for learning the features of characters and identifying characters in a captured image.

[0073] When the first tool (color area) button d1 is clicked, the first tool (color area) is selected. When the second tool (learning search) button d2 is clicked, the second tool (learning search) is selected. When the third tool (learning OCR) button d3 is clicked, the third tool (learning OCR) is selected.

[0074] The tool selection dialog d0 may display guidance that displays an overview and a schematic diagram of the selected tool from the first tool (color area), the second tool (learning search), and the third tool (learning OCR).

[0075] When the OK button d4 is clicked, the tool selection state is confirmed. On the other hand, when the Cancel button d5 is clicked, the tool selection state is canceled and the tool selection dialog d0 is closed. The following explanation assumes that the second tool (learning search) has been selected on the tool selection screen 400d (tool selection dialog d0).

[0076] Once the tool selection state is confirmed, an object selection screen 400e is displayed. An operation area 420 of the object selection screen 400e may display, for example, a rectangular button e1, an oval button e2, guidance e3, a rotation auto-learning ON button e4, a rotation auto-learning OFF button e5, a learning start button e6, and a cancel button e7. Furthermore, a banner indicating that a tool is being set (e.g., "TOOL") may be displayed in the status display banner 411.

[0077] On the object selection screen 400e, a search object shown in a registered master image is specified by setting a window e0. For example, when the rectangle button e1 is clicked, a rectangular window e0 is displayed in the image display area 410. When the oval button e2 is clicked, an oval window (not shown) is displayed in the image display area 410. The shape of the window e0 may be a shape other than a rectangle or oval (for example, a chamfered rectangle) as long as the lengths of the width and height directions, which are orthogonal to each other, are different from each other. Note that the guidance e3 may display a method for specifying the target area, etc.

[0078] The user can specify the search object by following the guidance e3 and adjusting the position, size, and angle of the window e0 so that it surrounds the search object shown in the master image. The position of the window e0 is the position of the window e0 relative to the master image. The angle of the window e0 is the angle of the window e0 relative to the master image.

[0079] The image inspection device S has a function to automatically learn the rotated state of the search object (rotation automatic learning function). When the rotation automatic learning ON button e4 is clicked, the rotation automatic learning function is enabled. When the rotation automatic learning OFF button e5 is clicked, the rotation automatic learning function is disabled.

[0080] After the search object is specified, clicking the Start Learning button e6 starts learning of the machine learning model used in the learning search. On the other hand, clicking the Cancel button e7 cancels the specification of the search object.

[0081] When learning of the machine learning model begins, a learning progress screen 400f is displayed. Note that the learning progress screen 400f may display a learning progress dialog f0, as shown in FIG. 5. The learning progress dialog f0 displays, for example, a learning progress bar f1 indicating the progress level (0% to 100%). When learning of the machine learning model is complete, the process proceeds to the fourth step (output allocation).

[0082] In the fourth step (output allocation) of step S14, an output allocation screen 400g is displayed. In the operation area 420 of the output allocation screen 400g, for example, pull-down menus g1 for the output ports OUT1 to OUT8, a back button g2, and a done button g3 may be displayed.

[0083] The pull-down menu g1 displays multiple candidates for the output content of each of the output ports OUT1 to OUT8. In this diagram, the search result (SEARCH) of work W is selected as the output content of output port OUT1. Furthermore, the busy detection result (BUSY) is selected as the output content of output port OUT2. Furthermore, the error detection result (ERROR) is selected as the output content of output port OUT3. Furthermore, all of the output ports OUT4 to OUT8 are unused (OFF). With this configuration, it is possible to output a variety of information in multiple bits, such as not only the search result of work W, but also the busy detection result and error detection result.

[0084] Clicking the back button g2 returns to the third step (tool setting) described above, while clicking the complete button g3 completes the setting process.

[0085] During the period in which the aforementioned learning progress screen 400f is displayed, the image inspection device S (particularly the inspection setting section 200) executes the following processing as learning of the machine learning model.

[0086] As shown in Figure 6, the image inspection device S (particularly the inspection setting unit 200) inputs a learning image (master image) M1 into a machine learning model MD1. The machine learning model MD1 includes a first feature extraction unit FE1, a second feature extraction unit FE2, and a determination unit D1. Note that the first feature extraction unit FE1 and the second feature extraction unit FE2 may be different feature extraction units or may be the same feature extraction unit.

[0087] The inspection setting unit 200 executes the first feature extraction unit FE1 to extract a first feature map MP1 from the training image (master image) M1. Furthermore, based on the window setting information INF1, the inspection setting unit 200 extracts a first feature F1 from the first feature map MP1 corresponding to a position identified by the window (a predetermined position within the window). The predetermined position within the window may be, for example, the center position of the window, but may be any position uniquely determined within the window. Because the window is set to surround the search object reflected in the training image (master image) M1, a change in the angle (orientation) of the search object also changes the first feature F1 corresponding to the position identified by the window (a predetermined position within the window). In other words, the first feature F1 is a feature that reflects the angle of the window relative to the training image (master image) M1. The parameters of the determination unit D1 are updated according to the first feature F1.

[0088] The inspection setting unit 200 executes the second feature extraction unit FE2 to extract a second feature map MP2 from the training image (master image) M1. Furthermore, based on the window setting information INF1, the inspection setting unit 200 extracts, from the second feature map MP2, a second feature F2 corresponding to a position specified by the window (a predetermined position within the window). Furthermore, based on the window setting information INF1, the inspection setting unit 200 extracts, from the second feature map MP2, a background feature BF1 corresponding to a background position other than the window. The parameters of the determination unit D1 are updated according to the background feature BF1. Since the parameters of the determination unit D1 are updated according to the background feature BF1 without requiring an operation to specify a background, the determination accuracy of the determination unit D1 can be improved without any additional effort.

[0089] Furthermore, when the rotation automatic learning function is enabled, the inspection setting unit 200 generates a rotated image R1 by rotating the training image (master image) M1 around a predetermined position on the training image (master image) M1 as the rotation center. The window in the rotated image R1 is obtained by rotating the window in the training image (master image) M1 around a predetermined position on the training image (master image) M1 as the rotation center. When rotating every α°, 360 / α (however, if the value is not an integer, the decimal point is truncated) rotated images R1 are generated. Then, the image inspection device S (particularly the inspection setting unit 200) inputs the rotated image R1 to the machine learning model MD1, as shown in FIG. 7.

[0090] Then, the inspection setting unit 200 executes the first feature extraction unit FE1 to extract a first feature map MP1 from the rotated image R1. Furthermore, based on the window setting information INF1 and the rotation angle information INF2, the inspection setting unit 200 extracts a first feature F1 from the first feature map MP1 that corresponds to a position specified by the window in the rotated image R1 (a predetermined position within the window), and stores the first feature F1 and the rotation angle information INF2 in association with each other. The parameters of the determination unit D1 when the automatic rotation learning function is enabled are updated according to the first feature F1 associated with the rotation angle information INF2.

[0091] 4 is completed and the mode is switched from the setting mode to the operation mode, a learning search is executed on the captured image captured by the imaging unit 1. The execution entity of the learning search can be basically understood as the inspection execution unit 300.

[0092] As shown in FIG. 8, the image inspection device S (particularly the inspection execution unit 300) inputs a captured image IM1 into the machine learning model MD1.

[0093] The inspection execution unit 300 executes the first feature extraction unit FE1 to extract a third feature map MP3 from the captured image IM1. The inspection execution unit 300 also extracts a third feature F3 similar to the first feature F1 from each feature included in the third feature map MP3. A method for extracting the third feature F3 similar to the first feature F1 includes, but is not limited to, calculating the vector similarity between each feature included in the third feature map MP3 and the first feature F1, and extracting the feature whose vector similarity is equal to or greater than a threshold as the third feature F3. The inspection execution unit 300 identifies a position in the captured image IM1 corresponding to the third feature F3 and determines a candidate area based on the identified position and window setting information INF1. The position identified by the candidate area (a predetermined position within the candidate area) corresponds to the third feature F3. The size of the candidate area matches the size of the window, and the angle of the candidate area relative to the captured image IM1 matches the angle of the window of the training image (master image) M1.

[0094] The inspection execution unit 300 executes the second feature extraction unit FE2 to extract a fourth feature map MP4 from the captured image IM1. The inspection execution unit 300 also extracts, from the fourth feature map MP4, a fourth feature F4 corresponding to the previously identified position (the position corresponding to the third feature F3 in the captured image IM1).

[0095] The inspection execution unit 300 executes the determination unit D1 to classify whether the fourth feature F4 belongs to the same class as the second feature F2, i.e., a class indicating a search object. If the fourth feature F4 is classified as belonging to the same class as the second feature F2, the inspection execution unit 300 outputs an inspection result that identifies the candidate area as the object detection area. When the inspection execution unit 300 executes a learning search, the inspection result includes a detection result including the detection of an object in the captured image IM1 and the location of the object detection area in the captured image IM1, as well as a narrower inspection result indicating that the detected object is a search object. In other words, when a non-defective object is set as the search object, the inspection unit 300 outputs an inspection result equivalent to the result of two operations: detecting whether the object is included in the captured image IM1, and, if the object is included in the captured image IM1, determining whether the object is non-defective. When there are multiple fourth feature values ​​F4 classified as belonging to the same class as the second feature value F2, the inspection execution unit 300 may output a detection result in which each of the multiple candidate regions is the detection region of the object. Furthermore, the inspection range of another tool (e.g., learning OCR) may be determined based on the detection region as the detection result. For example, if the object is a printed portion of a product, the candidate region may be used as the inspection range of the learning OCR.

[0096] The detection area DET1 of the search object is displayed as a frame, for example, as in the detection result screen 400h shown in Figure 9. The object is searched for by extracting the third feature F3 similar to the first feature F1. Therefore, even if the position of the object detected as the search object in the captured image IM1 is far from the position of the window in the learning image (master image) M1, the image inspection device S can detect the object. Furthermore, when the automatic rotation learning function is enabled, the image inspection device S can detect the search object even if the angle of the object in the captured image IM1 is significantly different from the angle of the window in the learning image (master image) M1 (see the detection result screen 400h shown in Figure 10).

[0097] (Additional learning) For example, there may be cases where an object that is not the search target (an object that is different from the search target and lacks a black oval mark) is mistakenly detected, as shown in the detection result screen 400h in Figure 11. By performing additional learning, such mistaken detection can be prevented in the learning search performed after additional learning.

[0098] When the additional learning button h1 displayed in the operation area 420 of the detection result screen 400h is clicked, the inspection setting unit 200 performs additional learning. Specifically, the inspection setting unit 200 sets the object detection area DET1 displayed in the image display area 410 at the time the additional learning button h1 is clicked as the additional learning window, sets the third feature map MP3 of the captured image IM1 displayed in the image display area 410 at the time the additional learning button h1 is clicked as the additional learning feature map of the additional learning image, and extracts an additional learning feature corresponding to the position identified by the additional learning window from the additional learning feature map. The parameters of the determination unit D1 are updated according to the additional learning feature. The parameters of the determination unit D1 are updated so that the background feature FB1 and the additional learning feature belong to the same class (a class indicating that the feature is not an object) and so that the second feature F2 belongs to a different class (a class indicating that the feature is an object) from the background feature FB1 and the additional learning feature.

[0099] In this way, by performing additional learning using the captured image IM1 displayed on the detection result screen 400h, the number of operations related to performing additional learning can be reduced. Note that, instead of using the captured image IM1 displayed on the detection result screen 400h, an object that is not the search target and that appears in the additional learning image may be specified in the additional learning window in the setting mode.

[0100] (Not detected) If the result of the learning search is that the search object is not detected, the inspection execution unit 300 may output an image indicating that there was no search object detection area superimposed on the captured image as the detection result. This allows the user to clearly understand that a learning search was performed even if the search object was not detected. An example of an image indicating that there was no search object detection area could be a display image in which the window format (color, thickness, etc.) set in the learning image (master image) M1 is changed.

[0101] If an object that should be detected as the search object is not displayed in the image display area 410 of the detection result screen 400h, and an object that is not the search object has not been mistakenly detected, it is preferable to display an image h2 indicating that there was no detection area for the search object, as shown in the detection result screen 400h in Figure 12, for example.

[0102] <Other> In addition to the above-described embodiments, the various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above-described embodiments should be considered to be illustrative and not restrictive in all respects. Furthermore, the technical scope of the present invention is defined by the claims, and should be understood to include all modifications that fall within the meaning and scope of the claims. [Explanation of symbols]

[0103] 1 Imaging unit 2. Control Unit 3 PC (Personal Computer) 3a keyboard 4 Display device 4a Touch panel 5 PLC (Programmable Logic Controller) 6 Communication Cable 7. Communication Cable 8 Magnifying Lens Attachment 10 Lighting Module 10a LED (light emitting diode) 10b LED Driver 11 Camera module 11a AF motor 11b Imaging board 11c CMOS sensor 11d FPGA 11e DSP 20 Head communication unit 21 Control section 21a DSP 21b FPGA 21c Accelerator 21d Memory 22 Communication board 23 Power supply 24 Connector board 24a power interface 25 I / O board 26 Storage device (storage unit) 100 Imaging setting section 200 Inspection setting section 210 Tool setting section 211 Tool selection section 212 Parameter setting section 213 Learning Tool Settings 213a Learning data setting unit 213a1 Learning image selection unit 213a2 Label information setting section 213a3 Learning data generation unit 213b Update section 220 Inspection condition setting section 300 Inspection Execution Department 310 Tool Execution Unit 311 Rule Judgment Unit 312 Learning Tool Execution Department 320 Inspection result output unit 400 GUI 400a initial startup screen 400b Image capture setting screen 400c Master image selection screen 400d Tool Selection Screen 400e Object selection screen 400f Learning progress screen 400g Output Allocation Screen 400h Detection result screen 410 Image display area 411 Status Display Banner 412 Zoom In / Out Button 413 Maximum display button 420 Operation area 430 Progress display area a1 Caution mark a2 Setting start button b1 Slide bar b2 Back button b3 Next button c1 Live button c2 History button c3 File button c4 Back button c5 Forward button d0 Tool selection dialog d1 First tool button d2 Second tool button d3 Third tool button d4 OK button d5 Cancel button e0 window e1 Rectangular button e2 oval button e3 guidance e4 Rotation auto learning ON button e5 Rotation auto learning OFF button e6 Start learning button e7 Cancel button f0 Learning progress dialog f1 Learning progress bar g1 pull-down menu g2 Back button g3 Complete button h1 Additional learning button h2 image A. Means of transport (conveyor) B Equipment S Image inspection device W work (object)

Claims

1. An imaging unit; a control unit separate from the imaging unit; Equipped with the imaging unit includes an imaging section that captures an imaging field of view and generates a captured image; The control unit an inspection execution unit that detects an object from the captured image using a machine learning model and outputs a detection result; an inspection setting unit that sets the inspection execution unit, the machine learning model includes a first feature extraction unit that extracts features from an input image; The test setting unit Accepts window settings for training images, extracting a first feature map from the training image by executing the first feature extraction unit, and extracting a first feature included in the first feature map, the first feature reflecting an angle of the window relative to the training image, and corresponding to a position identified by the window; The inspection execution unit executing the first feature extraction unit to extract a third feature map from the captured image; identifying a position included in the third feature map and corresponding to a third feature amount similar to the first feature amount; and determining a candidate area based on the identified position and the window setting; an image inspection device that outputs an image showing the candidate region superimposed on the captured image;

2. the test setting unit executes a second feature extraction unit different from the first feature extraction unit to extract a second feature map from the training image, and extracts a second feature amount included in the second feature map and corresponding to a position specified by the window; The inspection execution unit extracting a fourth feature map from the captured image by executing the second feature extraction unit, and classifying whether a fourth feature amount included in the fourth feature map and corresponding to the specified position belongs to the same class as the second feature amount; 2. The image inspection device according to claim 1, wherein, when the fourth feature amount is classified as belonging to the same class as the second feature amount, the detection result is output, with the candidate region being the detection region of the object.

3. the test setting unit executes the first feature extraction unit to extract the first feature map from each of the training image and a rotated image obtained by rotating the training image, and extracts the first feature amount; the first feature corresponds to a position included in the first feature map, reflecting an angle of the window relative to the training image and identified by the window, or a position included in the first feature map, reflecting an angle of the window relative to the rotated image and identified by the window; 2. The image inspection device according to claim 1, wherein, when the third feature is similar to the first feature extracted from the rotated image, the inspection execution unit determines the candidate area based on the identified position, the window setting, and information regarding the rotation angle of the rotated image.

4. The test setting unit Accepts rotation settings, If the rotation setting is ON, the first feature extraction unit is executed to extract the first feature map from each of the training image and the rotated image; The image inspection device according to claim 3 , wherein if the rotation setting is OFF, the first feature extraction unit is executed to extract the first feature map from the training image.

5. The test setting unit extracting background features included in the first feature map and corresponding to background positions other than the window; The inspection execution unit The image inspection device according to claim 1 , wherein the third feature amount is classified as belonging to the same class as the first feature amount or the same class as the background feature amount.

6. the training image is a first training image, the window is a first window, The test setting unit accepting a second window setting for a second learning image different from the first learning image; In a state where an image indicating the candidate area is superimposed on the captured image and output, the captured image can be set as the second learning image, and the candidate area can be set as the second window, extracting a fifth feature map from the second training image by executing the feature extraction unit, and extracting a fifth feature included in the fifth feature map and corresponding to a position identified by the second window; The image inspection device according to claim 1 , wherein the inspection execution unit classifies whether the third feature amount belongs to the same class as the first feature amount or the same class as the fifth feature amount.

7. The image inspection device according to claim 1 , wherein, when the candidate region is not found, the inspection execution unit outputs an image indicating that the candidate region is not found superimposed on the captured image.

8. 2. The image inspection device according to claim 1, wherein the imaging unit can be fitted with a magnifying lens attachment that can adjust the focus of the imaging unit to the object to be imaged when the distance between the imaging unit and the object to be imaged is equal to or less than a predetermined value.

Images