Image inspection device

Abstract

The invention relates to an image inspection apparatus. An object is detected from a captured image without setting an ROI. The image inspection apparatus includes an imaging unit and a control unit separate from the imaging unit. The control unit includes an inspection setting section and an inspection execution section. The inspection setting section extracts a first feature map from the learning image, and extracts a first feature amount that is included in the first feature map, reflects an angle of the window with respect to the learning image, and corresponds to a position specified by the window. The inspection execution unit extracts a third feature map from the captured image, specifies a position corresponding to a third feature amount that is included in the third feature map and similar to the first feature amount, and determines a candidate region based on the specified position and the window setting. And outputting an image obtained by superimposing an image indicating the candidate region on the captured image.

Description

Technical Field

[0001] The present invention relates to an image inspection device. Background Art

[0002] In the related art, there is known an image inspection apparatus that includes a “learning tool” that is a tool for setting conditions for determining whether an object is a non-defective product (for example, see Japanese Patent Laid-Open No. 2022-164146).

[0003] When the position and angle of the inspection target area (ROI: Region of Interest) of the "learning tool" are fixed relative to the shooting field of view, when an object appearing in the inspection target image is outside the ROI, it is impossible to judge whether the object is a non-defective product.

[0004] In order to solve the above-mentioned problem, for example, the image inspection apparatus includes a “position correction tool” disclosed in Japanese Patent Laid-Open No. 2022-164146.

[0005] The “position correction tool” is used to set a reference position and a reference angle in the field of view, and the position and angle of the ROI are adjusted relative to the field of view based on the reference position and the reference angle.

[0006] However, if the visual characteristics suitable for setting the reference position and reference angle of the "position correction tool" are not included in the inspection target image, it is difficult to make appropriate adjustments using the "position correction tool." In particular, in a separate image inspection device in which the imaging unit is separate from the control unit, the housing including the imaging unit can be easily miniaturized compared to an integrated image inspection device in which the imaging unit and control unit are integrated, making it easier to install the separate image inspection device in a position where the imaging unit and the object are close to each other. When the imaging unit and the object are close to each other, there is a high possibility that the visual characteristics suitable for setting the reference position and reference angle of the "position correction tool" are not included in the inspection target image. Summary of the Invention

[0007] In view of the above problems, an object of the present invention is to provide an image inspection apparatus capable of detecting an object from a captured image without requiring a user to set a ROI.

[0008] An image inspection device according to the present invention includes: an imaging unit; and a control unit separate from the imaging unit. The imaging unit includes an imaging section for capturing a field of view to generate a captured image. The control unit includes an inspection execution section for detecting an object from the captured image using a machine learning model and outputting a detection result; and an inspection setting section for configuring the inspection execution section. The machine learning model includes a first feature extraction section for extracting feature quantities from an input image. The inspection setting section receives a window setting for a learning image and executes the first feature extraction section to extract a first feature map from the learning image and extract a first feature quantity, the first feature quantity included in the first feature map, reflecting an angle of a window relative to the learning image, and corresponding to a position specified by the window. The inspection execution section executes the first feature extraction section to extract a third feature map from the captured image, specifies a position corresponding to a third feature quantity included in the third feature map and similar to the first feature quantity, determines a candidate region based on the specified position and the window setting, and outputs an image obtained by superimposing an image indicating the candidate region on the captured image.

[0009] Note that other characteristics, elements, steps, advantages, and features will become more apparent from the following detailed description and accompanying drawings.

[0010] The image inspection apparatus according to the present invention can detect an object from a captured image without setting a ROI. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram for explaining the operation of the image inspection apparatus according to the embodiment of the present invention;

[0012] Figure 2 It is a hardware structure diagram of the image inspection device;

[0013] Figure 3 It is a functional block diagram of an image inspection device;

[0014] Figure 4 is a diagram illustrating a flow in a setting mode;

[0015] Figure 5 This is an example Figure 4 A graphical user interface [GUI] diagram of the transformation;

[0016] Figure 6 It is a block diagram of a machine learning model;

[0017] Figure 7 It is a block diagram of a machine learning model;

[0018] Figure 8It is a block diagram of a machine learning model;

[0019] Figure 9 is a diagram illustrating an example detection result image;

[0020] Figure 10 is a diagram illustrating an example detection result image;

[0021] Figure 11 is a diagram illustrating an example of a detection result image; and

[0022] Figure 12 exemplifies the detection result image. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following description of the preferred embodiment is merely exemplary in nature and is not intended to limit the invention, its application, or its intended uses.

[0024] Figure 1 : is a diagram for explaining the image inspection device S according to an embodiment of the present invention when in operation. For example, the image inspection device S shoots the workpiece W conveyed by the conveying unit A according to the shooting settings to obtain a shot image, detects the workpiece W in the acquired shot image, and outputs the detection result to an external device. Examples of external devices include a programmable logic controller (PLC) 5, etc., but devices other than the PLC 5 may be external devices. Based on the received detection result, the PLC 5 controls the conveying unit A, for example, to separate the storage destination of the workpiece W. In the following description, the case where the external device is the PLC 5 will be described. Note that the workpiece W may be a workpiece that is not conveyed by the conveying unit A. In addition, in the following description, the workpiece is also referred to as an object. Note that in the following description, the entire workpiece is an object, but the object may be a part of the workpiece.

[0025] The image inspection device S includes an imaging unit 1 for capturing an image of a workpiece W; a control unit 2 to which images captured by the imaging unit 1 are input; a personal computer (PC) 3 for configuring the image inspection device S; and a display device 4 for displaying settings, selection screens, workpiece images, and inspection results. The control unit 2 can execute a trained model for detecting the workpiece W in the input captured images. The control unit 2 outputs detection results corresponding to the trained model to a programmable logic controller 5.

[0026] Here, for example, the image inspection device S can be used when inspecting a workpiece W from various angles at various stages of a manufacturing facility or production line. Consequently, multiple image inspection devices S may be installed in a single manufacturing facility or production line, and it is conceivable that sufficient installation space and power supply may not be secured. Therefore, the image inspection device S needs to be compact enough to accommodate the installation space and energy-efficient enough to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a graphics processing unit (GPU). The control unit 2 executes a trained model (in which machine learning is performed to the extent that the workpiece W can be detected), but the image inspection device S is provided to the user by the supplier so that the desired detection accuracy can be achieved even without performing the advanced machine learning recommended using a GPU. Details will be described later. Since the user does not need to perform advanced machine learning, the user can execute a learned model capable of detecting the workpiece W without having to prepare a GPU suitable for learning. Furthermore, the time required for the user to prepare a trained model capable of detecting the workpiece W can be shortened. Note that a single image inspection device S can be installed and operated in a manufacturing facility or production line. Furthermore, the image inspection device S can also be referred to as an image sensor.

[0027] (Structure of the camera unit)

[0028] The imaging unit 1 is separated from the control unit 2 and is installed so as to be able to photograph the workpiece W from a desired direction. The workpiece W is sequentially transported by the transport unit A to the photographing field of the imaging unit 1. For example, Figure 1 As illustrated, in a case where the facility B is arranged above the conveying unit A at a short distance relative to the conveying unit A, the imaging unit 1 is arranged between the conveying unit A and the facility B, and the imaging unit 1 captures an image of the workpiece W in a state where the imaging unit 1 and the workpiece W are close to each other. Since the imaging unit 1 is separated from the control unit 2 and the imaging unit 1 can be miniaturized, the imaging unit 1 can be arranged in a narrow space between the conveying unit A and the facility B. Note that in Figure 1 , for convenience, the facility B is illustrated in a manner in which the width of the facility B in the conveying direction of the conveying unit A is shortened so that the conveying unit A and the workpiece W can be easily seen, but in fact, the facility B extends in the conveying direction of the conveying unit A.

[0029] Figure 2 This is the hardware structure diagram of the image inspection device S. Figure 2 As illustrated, the imaging unit 1 includes an illumination module 10 for illuminating a workpiece W, and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10 .

[0030] The lighting module 10 includes a light-emitting diode (LED) 10a that illuminates the workpiece W with light, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication unit 20 (described later) of the control unit 2 and is controlled by a controller 21 (described later) of the control unit 2.

[0031] The camera module 11 includes an AF motor 11a and a pickup board 11b. The AF motor 11a drives a focus lens of an optical system (not illustrated) to automatically focus on the workpiece W. The automatic focusing method is not particularly limited, and examples thereof include a contrast method and a liquid lens method.

[0032] The imaging board 11b includes a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives reflected light emitted from the LED 10a toward the workpiece W and reflected by the workpiece W. The CMOS sensor 11c is connected to the head communication unit 20 of the control unit 2 and is controlled by the controller 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0033] The FPGA 11d is a processing device capable of changing the content of internal processing. The DSP 11e is a signal processing device. The light-receiving amount signal from the light-receiving element included in the CMOS sensor 11c is output to and processed by the FPGA 11d, and is also output to and processed by the DSP 11e. The processing performed by the FPGA 11d and DSP 11e is not particularly limited, and examples include various filtering processes. Image data processed by the FPGA 11d and DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0034] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Thus, the control unit 2 can be installed at a location remote from the installation location of the imaging unit 1. In addition, the imaging unit 1 can be attached to the magnifying lens attachment 8. When the magnifying lens attachment 8 is attached to the imaging unit 1, the camera module 11 can focus on the workpiece W even when the distance between the imaging unit 1 and the workpiece W is short. Note that if the imaging unit 1 and the workpiece W are remote from each other and the magnifying lens attachment 8 is not required, the magnifying lens attachment 8 can be detached from the imaging unit 1.

[0035] (PC structure)

[0036] The PC 3 is composed of a general-purpose personal computer or the like. In this example, the PC 3 can be used by installing a predetermined program in the personal computer. The PC 3 includes operating devices such as a keyboard 3a and a mouse (not shown). The user of the image inspection device S can perform setting operations and selection operations for the image inspection device S by operating the operating devices of the PC 3. Specific setting operations and selection operations will be described later.

[0037] The PC 3 is connected to the communication board 22 of the control unit 2 so as to communicate with each other, and information based on setting operations performed by the user is transmitted from the PC 3 to the control unit 2. In addition, the PC 3 can receive image data and inspection results of the workpiece W output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Thus, the PC 3 can be installed at a location far from the installation location of the control unit 2.

[0038] (Structure of Display Device)

[0039] The display device 4 includes, for example, a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component that can detect operations using the user's fingers. The type of touch panel 4a is not particularly limited, and examples thereof include a capacitive type and an infrared type. The display device 4 is connected to the communication board 22 of the control unit 2 in a manner that allows them to communicate with each other. User operation information on the touch panel 4a is sent from the display device 4 to the control unit 2. In addition, the display device 4 can receive image data of the workpiece W output from the control unit 2, etc. The display device 4 and the control unit 2 are connected via a communication cable 7. Thus, the display device 4 can be installed in a place away from the installation place of the control unit 2.

[0040] Note that the PC 3 and the display device 4 may be integrally provided. For example, the display device 4 may be formed by a display device included in the PC 3. In this case, the main body of the PC 3 and the display device 4 may be integrally provided, or may be separate from each other. Furthermore, in this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0041] (Structure of control unit)

[0042] like Figure 2As shown, the control unit 2 includes a head communication unit 20, a controller 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage unit) 26. The head communication unit 20 is connected to the controller 21 and performs communication between the controller 21 and the imaging unit 1. Control signals for the imaging unit 1 output from the controller 21 are transmitted to the imaging unit 1 via the head communication unit 20. The control signals for the imaging unit 1 include signals for controlling the timing and amount of light emission of the LED 10a, as well as signals for controlling the AF motor 11a and the capturing board 11b. In addition, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the controller 21 via the head communication unit 20.

[0043] The controller 21 includes a DSP 21a and an FPGA 21b that perform various signal processing, an accelerator 21c for accelerating processing, and a memory 21d including a RAM and a ROM, etc. The specific structure of the controller 21 will be described later.

[0044] The communication board 22 is a member that is connected to the controller 21 and performs communication among the controller 21 , the PC 3 , the display device 4 , and the PLC 5 .

[0045] The connector plate 24 includes a power interface 24a. A power cable (not shown) for supplying power from the outside is connected to the power interface 24a. The connector plate 24 is connected to the power supply 23, and the power supplied from the outside to the power interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the controller 21. The power supplied to the controller 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0046] The I / O board 25 is connected to the controller 21. The inspection result output from the controller 21 is input to the PLC 5 via the I / O board 25.

[0047] (Function Block)

[0048] Figure 3 2 is a functional block diagram of an image inspection apparatus S. As illustrated in the figure, the image inspection apparatus S includes a photographing setting unit 100 , an inspection setting unit 200 , and an inspection execution unit 300 as its functional blocks.

[0049] The shooting setting section 100 performs various settings related to the shooting operation of the image pickup unit 1 (shooting field of view, brightness of image, focus, shooting interval (frame rate), etc.).

[0050] The inspection setting unit 200 performs various settings related to the inspection of the captured images using the inspection execution unit 300. According to the drawing, the inspection setting unit 200 includes a tool setting unit 210 and an inspection condition setting unit 220.

[0051] The tool setting unit 210 sets various tools. According to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0052] The tool selection unit 211 selects a tool to be set and a tool to be used.

[0053] The parameter setting section 212 sets a rule-based tool. In the rule-based tool, inspection is performed based on various feature quantities (contour, color, position, etc.) of the workpiece W appearing in the image.

[0054] The learning tool setup unit 213 sets up tools for a learning system that uses machine learning models. In the learning system tools, trained models such as discriminators are generated based on user instruction, and checks are performed based on the output of the trained models. As shown in the figure, the learning tool setup unit 213 includes a learning data setup unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0055] The learning data setting unit 213a sets the learning data to be input into the machine learning model. The learning data includes a learning image and teaching content. The learning image is an image on which a workpiece W appears. The teaching content is the setting information of the window indicating the area where the workpiece W appears. As shown in the figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a window setting unit 213a2, and a learning data generation unit 213a3.

[0056] The learning image selection unit 213 a 1 selects a learning image. The learning image may be an image captured by the imaging unit 1 or an image transmitted from the PC 3 and stored in the storage device 26 .

[0057] The window setting unit 213 a 2 displays the learning image selected by the learning image selection unit 213 a 1 on the GUI, and receives window settings for the learning image.

[0058] The learning data generating section 213 a 3 generates learning data based on the learning image selected by the learning image selecting section 213 a 1 and the window setting received by the window setting section 213 a 2 .

[0059] The updating unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches the expected value corresponding to the teaching content. The updating of the parameters can be understood as the learning of the machine learning model. However, the learning of the machine learning model does not necessarily need to be performed by the user in all processes. For example, learning with a relatively large amount of calculation can be completed on the supplier side before the image inspection device S is shipped, and only learning with a relatively small amount of calculation can be performed on the user side before the operation of the image inspection device S. In this specification, the learning performed by the supplier side before shipping is referred to as pre-shipment learning, and the learning performed by the user before the operation of the image inspection device S is referred to as customer learning.

[0060] For example, the machine learning model of the image inspection device S includes a feature extraction unit that does not perform customer learning and a judgment unit that performs customer learning. The feature extraction unit extracts features from the image input to the machine learning model. The judgment unit outputs an inspection result based on the features.

[0061] That is, the machine learning model of the image inspection device S may include a fixed parameter portion. The fixed parameter portion is a layer with fixed parameters obtained through pre-shipment learning on the supplier side, in other words, a layer that does not require customer learning on the user side.

[0062] With this structure, users do not need to prepare facilities such as GPUs with the high processing power required for deep learning, nor do vendors need to provide advanced learning environments using GPUs as cloud services (SaaS, etc.). Therefore, the barriers to introducing image inspection devices S are reduced.

[0063] As described above, learning should be understood in a broad sense to refer not only to computationally intensive deep learning, but also to computationally inexpensive learning (that is, client learning in this specification). Note that since client learning is computationally inexpensive learning, it is possible to train a machine learning model using methods that do not include machine learning methods.

[0064] The inspection condition setting section 220 determines the output conditions of the image inspection apparatus S (in other words, the conditions of the sensor output) by combining a plurality of tools, for example.

[0065] The inspection execution unit 300 executes an inspection process of the workpiece W appearing in the captured image and outputs an inspection result. According to the figure, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0066] The tool execution unit 310 executes the tool selected as the use target by the tool selection unit 211. According to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0067] When the tool selection unit 211 selects a rule-based tool as a usage target, the rule determination unit 311 executes the rule-based tool.

[0068] When the tool selection section 211 selects the tool of the learning system as the use target, the learning tool execution section 312 executes the tool of the learning system.

[0069] The inspection result output unit 320 outputs the inspection result according to the output conditions set by the inspection condition setting unit 220. Inspection may include image classification, abnormality detection, and region segmentation.

[0070] The imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be composed solely of hardware, or may be composed of a combination of hardware and software. Furthermore, the imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are implemented by a single piece of hardware or software. Note that the software may be executed by the control unit 2 (particularly the controller 21) having installed therein program files and setting files.

[0071] The image inspection device S of this structural example can be switched between a setup mode and a drive mode. In the setup mode, for example, various parameter settings such as shooting settings, registration of a master image, and generation (learning) of a classifier for classifying images into classes based on the image's feature quantities are performed. Note that classification, as used herein, refers to the classification of images into any class, and includes the classification of images into images of non-defective products and images of defective products, as well as the classification of images into images in which an object appears and images in which an object does not appear.

[0072] In drive mode, workpieces W are inspected based on images captured by the imaging unit 1 at the actual site. Switching between setup mode and drive mode can be performed using a GUI (described later). Alternatively, the system can be configured to automatically transition to drive mode upon completion of setup mode. In drive mode, the classification boundaries of the classifier can also be corrected (updated), in other words, performing so-called additional learning.

[0073] (Learn to search)

[0074] The image inspection device S includes a learning search tool that implements a "learning search" function as one of the tools of the learning system using a machine learning model. The learning search tool is a tool for learning the visual characteristics of a search object, which is an object to be searched for, and searching for the search object from a captured image (in other words, detecting the search object from the captured image). In the following description, "learning search" may be used as a term to indicate the learning search tool.

[0075] Figure 4This is a diagram illustrating the flow in the setup mode (setup flow). Figure 4 The execution subject of the illustrated setting process can be basically understood as the inspection setting unit 200 .

[0076] Figure 5 is an example of Figure 4 GUI 400 is displayed on the display device 4 when the image inspection program is executed by the PC 3. Figure 4 The GUI 400 (=various screens 400 a to 400 g ) corresponding to the illustrated setting flow includes an image display area 410 , an operation area 420 , and a progress display area 430 as its basic layout.

[0077] Image display area 410 displays images captured by imaging unit 1. Image display area 410 may also include a status display banner 411, zoom-in / out buttons 412, and a maximum display button 413. Status display banner 411 briefly displays the operating status (status) of the image inspection program. Zoom-in / out buttons 412 and maximum display button 413 are operated to zoom in or out and maximize the image displayed in image display area 410, respectively.

[0078] When the image inspection device S is first started, the initial startup screen 400a is displayed. Note that when the image inspection device S is first started, the various settings for the image inspection program are not yet complete. Therefore, an alarm mark a1 (or an alarm message) indicating that the master image is not registered may be displayed in the image display area 410. In addition, a banner (e.g., "Master") indicating that the master image is not registered (or is being registered) may be displayed on the status display banner 411.

[0079] In addition, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, the same applies hereinafter), various settings of the image inspection program are started. Figure 4 The switch from drive mode to setup mode also corresponds to the start of the setup process. Figure 4 This corresponds to the start of the illustrated setup flow.

[0080] exist Figure 4In the illustrated setup process, the first step (shooting setup) is shown in step S11, the second step (master registration) is shown in step S12, the third step (tool setup) is shown in step S13, and the fourth step (output allocation) is shown in step S14. In progress display area 430, each step can be displayed as a flowchart, and the currently executing step can be highlighted. This structure allows the user to understand the progress of the setup work at a glance.

[0081] In the first process (shooting setting) of step S11, the shooting setting screen 400b is displayed. In the operation area 420 of the shooting setting screen 400b, for example, a slide bar b1, a back button b2, and a forward button b3 can be displayed.

[0082] When the slide bar b1 is moved by dragging, the brightness of the image corresponding to the movement of the slide bar b1 is set. Figure 5 Although not explicitly illustrated in the figure, in the first step (shooting setting), the shooting field of view, focus, shooting interval (frame rate), etc. can be set.

[0083] When the back button b2 is clicked, the screen returns to the initial startup screen 400a described above. On the other hand, when the forward button b3 is clicked, the process proceeds to the second process (main registration).

[0084] Note that Figure 5 As illustrated, in the first process (shooting setting), a live image (=moving image being shot) may be displayed in the image display area 410. In this case, the status display banner 411 may display a banner (eg, "Live") indicating that a live image is being displayed.

[0085] In the second step (master registration) of step S12, a master image selection screen 400c is displayed. In the operation area 420 of the master image selection screen 400c, for example, a live button c1, a history button c2, a file button c3, a back button c4, and a forward button c5 may be displayed.

[0086] When the real-time button c1 is clicked, the real-time image displayed in the image display area 410 at that time is registered as the main image. Note that the main image can be registered from the drive history image (checked image) by clicking the history button c2. In addition, the main image can be registered from the file image stored in the storage device 26 by clicking the file button c3. As described above, examples of the setting method for registering the main image include a method for registering from a real-time image, a method for registering from a drive history image, and a method for registering from a file image.

[0087] When the back button c4 is clicked, the process returns to the first process (shooting setting) described above. On the other hand, when the forward button c5 is clicked, the process proceeds to the third process (tool setting).

[0088] In the third process (tool setting) of step S13, first, the tool selection screen 400d is displayed. Figure 5 As illustrated, the tool selection dialog d0 may be displayed on the tool selection screen 400d.

[0089] In the tool selection dialog box d0, for example, a first tool (color area) button d1, a second tool (learning search) button d2, a third tool (learning optical character recognition [OCR]) button d3, an OK button d4, and a cancel button d5 are displayed. Color area is a rule-based tool for measuring the area of ​​a predetermined color. Learning search and learning OCR are each tools for learning the system. As described above, learning search is a tool for learning the visual characteristics of a search object and searching for the search object from a captured image. Learning OCR is a tool for learning the characteristics of characters and recognizing characters from a captured image.

[0090] When the first tool (color area) button d1 is clicked, the first tool (color area) is selected. When the second tool (learning search) button d2 is clicked, the second tool (learning search) is selected. When the third tool (learning OCR) button d3 is clicked, the third tool (learning OCR) is selected.

[0091] In the tool selection dialog d0 , a guide for displaying an outline and an illustration of a tool selected from the first tool (color area), the second tool (learning search), and the third tool (learning OCR) may be displayed.

[0092] Clicking the OK button d4 confirms the tool selection. Clicking the Cancel button d5 cancels the tool selection and closes the tool selection dialog box d0. The following description assumes that the second tool (Learning Search) has been selected on the tool selection screen 400d (tool selection dialog box d0).

[0093] When the tool selection status is confirmed, the object selection screen 400e is displayed. In the operation area 420 of the object selection screen 400e, for example, a rectangular button e1, an oval button e2, a guide e3, a rotational automatic learning ON button e4, a rotational automatic learning OFF button e5, a learning start button e6, and a cancel button e7 may be displayed. In addition, a banner indicating that the tool is being set (e.g., "TOOL") may be displayed on the status display banner 411.

[0094] On the object selection screen 400e, the search object that appears in the registered main image is specified by setting window e0. For example, clicking the rectangular button e1 displays the rectangular window e0 in the image display area 410. Clicking the elliptical button e2 displays an elliptical window (not shown) in the image display area 410. The shape of window e0 can be other than rectangular or elliptical (for example, a chamfered rectangle), as long as the lengths in the orthogonal width and height directions are different. Note that the guide e3 may display methods for specifying the target area, etc.

[0095] The user can specify a search object by adjusting the position, size, and angle of window e0 according to the guide e3 to surround the search object appearing in the main image. The position of window e0 is the position of window e0 relative to the main image. The angle of window e0 is the angle of window e0 relative to the main image.

[0096] The image inspection device S has a function of automatically learning the rotation state of the search object (rotation automatic learning function). When the rotation automatic learning ON button e4 is clicked, the rotation automatic learning function is enabled. When the rotation automatic learning OFF button e5 is clicked, the rotation automatic learning function is disabled.

[0097] When the learning start button e6 is clicked after specifying the search object, the learning of the machine learning model used for the learning search is started. On the other hand, when the cancel button e7 is clicked, the designation of the search object is canceled.

[0098] When the learning of the machine learning model starts, the learning progress screen 400f is displayed. Figure 5 As illustrated, a learning progress dialog box f0 may be displayed on the learning progress screen 400f. In the learning progress dialog box f0, for example, a learning progress bar f1 indicating the degree of progress (0% to 100%) is displayed. When the learning of the machine learning model is completed, the process proceeds to the fourth process (output allocation).

[0099] In the fourth step (output allocation) of step S14, an output allocation screen 400g is displayed. In the operation area 420 of the output allocation screen 400g, for example, a pull-down menu g1 for each of the output ports OUT1 to OUT8, a back button g2, and a finish button g3 may be displayed.

[0100] In the pull-down menu g1, multiple candidates are displayed as the output content of each of the output ports OUT1 to OUT8. In this figure, the search result (SEARCH) of workpiece W is selected as the output content of output port OUT1. In addition, the busy detection result (BUSY) is selected as the output content of output port OUT2. In addition, the error detection result (ERROR) is selected as the output content of output port OUT3. In addition, all output ports OUT4 to OUT8 are unused (OFF). With this structure, not only the search result of workpiece W can be output in multiple bits, but also various information such as busy detection results and error detection results can be output in multiple bits.

[0101] When the back button g2 is clicked, the process returns to the third process (tool setting) described above. On the other hand, when the finish button g3 is clicked, the setting work is completed.

[0102] While the learning progress screen 400f is displayed, the image inspection apparatus S (particularly, the inspection setting unit 200) performs the following processing as learning of the machine learning model.

[0103] like Figure 6 As shown, the image inspection device S (particularly the inspection setup unit 200) inputs a learning image (main image) M1 into a machine learning model MD1. The machine learning model MD1 includes a first feature extraction unit FE1, a second feature extraction unit FE2, and a judgment unit D1. Note that the first feature extraction unit FE1 and the second feature extraction unit FE2 may be different feature extraction units or may be the same feature extraction unit.

[0104] The inspection setting unit 200 executes the first feature extraction unit FE1 to extract the first feature map MP1 from the learning image (main image) M1. In addition, the inspection setting unit 200 extracts the first feature quantity F1 corresponding to the position specified by the window (predetermined position inside the window) from the first feature map MP1 based on the window setting information INF1. The predetermined position inside the window is, for example, the center position of the window, but even if the position is not the center position, it can be a position uniquely determined inside the window. Since the window is set to surround the search object appearing in the learning image (main image) M1, when the angle (orientation) of the search object changes, the first feature quantity F1 corresponding to the position specified by the window (predetermined position inside the window) also changes. That is, the first feature quantity F1 is a feature quantity that reflects the angle of the window relative to the learning image (main image) M1. The parameters of the judgment unit D1 are updated based on the first feature quantity F1.

[0105] The inspection setup unit 200 executes the second feature extraction unit FE2 to extract a second feature map MP2 from the learning image (main image) M1. Furthermore, based on the window setting information INF1, the inspection setup unit 200 extracts a second feature quantity F2 corresponding to the position specified by the window (a predetermined position within the window) from the second feature map MP2. Furthermore, based on the window setting information INF1, the inspection setup unit 200 extracts a background feature quantity BF1 corresponding to a background position other than the window from the second feature map MP2. The parameters of the judgment unit D1 are updated based on the background feature quantity BF1. Since the parameters of the judgment unit D1 are updated based on the background feature quantity BF1 without performing an operation to specify the background in this manner, the judgment accuracy of the judgment unit D1 can be improved without increasing time and effort.

[0106] In addition, when the rotation automatic learning function is valid, the inspection setting unit 200 generates a rotated image R1 obtained by rotating the learning image (main image) M1 with the predetermined position of the learning image (main image) M1 as the rotation center. Note that the window in the rotated image R1 is obtained by rotating the window in the learning image (main image) M1 with the predetermined position of the learning image (main image) M1 as the rotation center. When the learning image is rotated α° each time, 360 / α (however, if it is not an integer, it is rounded down to the nearest integer) rotated images R1 are generated. Then, as Figure 7 As illustrated, the image inspection apparatus S (particularly the inspection setting unit 200 ) inputs the rotated image R1 into the machine learning model MD1 .

[0107] The inspection setup unit 200 then executes the first feature extraction unit FE1 to extract a first feature map MP1 from the rotated image R1. Furthermore, based on the window setting information INF1 and the rotation angle information INF2, the inspection setup unit 200 extracts a first feature quantity F1 corresponding to the position specified by the window in the rotated image R1 (a predetermined position within the window) from the first feature map MP1 and stores the first feature quantity F1 in association with the rotation angle information INF2. The parameters of the determination unit D1, when the automatic rotation learning function is enabled, are updated based on the first feature quantity F1 associated with the rotation angle information INF2.

[0108] exist Figure 4 After the illustrated setting flow is completed and the setting mode is switched to the driving mode, a learning search is performed on the captured image captured by the imaging unit 1. The learning search can be basically understood as being performed by the inspection execution unit 300.

[0109] like Figure 8 As illustrated, the image inspection apparatus S (particularly the inspection execution unit 300 ) inputs the captured image IM1 into the machine learning model MD1 .

[0110] The inspection execution unit 300 executes the first feature extraction unit FE1 to extract a third feature map MP3 from the captured image IM1. In addition, the inspection execution unit 300 extracts a third feature value F3 similar to the first feature value F1 from the feature values ​​included in the third feature map MP3. As a method for extracting the third feature value F3 similar to the first feature value F1, there is a method for obtaining the vector similarity between each feature value included in the third feature map MP3 and the first feature value F1, and extracting a feature value having a vector similarity equal to or greater than a threshold as the third feature value F3, but the method is not limited to this. The inspection execution unit 300 specifies a position in the captured image IM1 corresponding to the third feature value F3, and determines a candidate area based on the specified position and the window setting information INF1. The position specified by the candidate area (a predetermined position within the candidate area) becomes the position corresponding to the third feature value F3, the size of the candidate area is consistent with the size of the window, and the angle of the candidate area relative to the captured image IM1 is consistent with the angle of the window of the learning image (main image) M1.

[0111] The inspection execution unit 300 executes the second feature extraction unit FE2 to extract the fourth feature map MP4 from the captured image IM1. In addition, the inspection execution unit 300 extracts the fourth feature quantity F4 corresponding to the above-mentioned specified position (the position corresponding to the third feature quantity F3 in the captured image IM1) from the fourth feature map MP4.

[0112] The inspection execution unit 300 executes the judgment unit D1 to perform classification based on whether the fourth feature quantity F4 belongs to the same class as the second feature quantity F2 (that is, the class indicating the search object). If the fourth feature quantity F4 is classified as belonging to the same class as the second feature quantity F2, the inspection execution unit 300 outputs an inspection result in which the candidate region is set as the detection region for the object. The inspection result when the inspection execution unit 300 performs a learning search includes a detection result including, for example, the detection of the object in the captured image IM1 and the location of the detection region for the object in the captured image IM1, as well as an inspection result narrowly indicating that the detected object is the search object. In other words, if a non-defective object is set as the search object, the inspection output is equivalent to the result of detecting whether the object is included in the captured image IM1 and, if so, determining whether the object is a non-defective product. If there are multiple fourth feature quantities F4 classified as belonging to the same class as the second feature quantity F2, the inspection execution unit 300 may output a detection result in which each of the multiple candidate regions is set as the detection region for the object. In addition, the inspection range of other tools (such as learning OCR) can be determined based on the detection area as the detection result. For example, if the object is a printed part of a product, the candidate area can be set as the inspection range of learning OCR.

[0113] For example, Figure 9 As shown in the illustrated detection result screen 400h, the detection area DET of the search object is displayed as a frame. Since the object is searched by extracting the third feature quantity F3 similar to the first feature quantity F1, the image inspection device S can detect the object detected as the search object even if the position of the object detected as the search object appearing in the captured image IM1 is far away from the position of the window of the learning image (main image) M1. In addition, when the rotation automatic learning function is effective, the image inspection device S can detect the search object even if the angle of the object appearing in the captured image IM1 is greatly different from the angle of the window of the learning image (main image) M1 (see Figure 10 The detection result screen 400h is shown as an example).

[0114] (Additional Learning)

[0115] For example, Figure 11 As shown in the detection result screen 400h, an object not being the search target (an object lacking a black oval mark other than the search target) is mistakenly detected. By performing additional learning, it is possible to prevent false detection in a learning search performed after the additional learning.

[0116] When the additional learning button h1 displayed in the operation area 420 of the detection result screen 400h is clicked, the inspection setting unit 200 performs additional learning. Specifically, the inspection setting unit 200 sets the detection area DET of the object displayed in the image display area 410 at the time when the additional learning button h1 is clicked as the additional learning window, sets the third feature map MP3 of the captured image IM1 displayed in the image display area 410 at the time when the additional learning button h1 is clicked as the additional learning feature map of the additional learning image, and extracts the additional learning feature quantity corresponding to the position specified by the additional learning window from the additional learning feature map. The parameters of the judgment unit D1 are updated according to the additional learning feature quantity. The parameters of the judgment unit D1 are updated so that the background feature quantity FB1 and the additional learning feature quantity belong to the same class (indicating that the background feature quantity FB1 is not a class of the object), and the second feature quantity F2 belongs to a class different from the additional learning feature quantity (indicating that the background feature quantity FB1 is a class of the object).

[0117] By using the captured image IM1 displayed on the detection result screen 400h for additional learning in this manner, the number of operations associated with performing additional learning can be reduced. Note that, instead of using the captured image IM1 displayed on the detection result screen 400h, an object that is not the search object and appears in the additional learning image can be specified in the additional learning window in the setup mode.

[0118] (Not detected)

[0119] If the search object is not detected as a result of a learning search, the inspection execution unit 300 can output an image indicating the absence of the search object's detection area superimposed on the captured image as the detection result. As a result, even if the search object is not detected, the user can clearly understand that a learning search has been performed. For example, a display image that changes the form (color, thickness, etc.) of the window provided in the learning image (main image) M1 can be considered as an image indicating the absence of the search object's detection area.

[0120] In the case where the object to be detected as the search target is not displayed in the image display area 410 of the detection result screen 400h and an object other than the search target is not erroneously detected, for example, Figure 12 As in the illustrated detection result screen 400h, an image h2 indicating a detection area where the search object does not exist may be displayed.

[0121] <Other>

[0122] Note that, in addition to this embodiment, various modifications may be made to the various technical features disclosed in this specification without departing from the spirit of the technical innovation. In other words, it should be considered that this embodiment is illustrative in all respects and not restrictive. In addition, the technical scope of the present invention is defined by the claims and should be understood to include all modifications that fall within the meaning and scope equivalent to the claims.

Claims

1. An image inspection device, comprising: Camera unit; as well as a control unit, which is separate from the camera unit, The camera unit includes a camera unit, which is used to capture a field of view to generate a captured image. The control unit comprises: an inspection execution unit configured to detect an object from the captured image using a machine learning model to output a detection result; and An inspection setting unit, configured to set up the inspection execution unit, The machine learning model includes a first feature extraction unit for extracting a feature quantity from an input image, and the inspection setting unit: Receives a window setting for the window to learn the image, and executing the first feature extraction unit to extract a first feature map from the learning image, and extracting a first feature quantity, the first feature quantity being included in the first feature map, reflecting the angle of the window relative to the learning image, and corresponding to the position specified by the window, The inspection execution unit: executing the first feature extraction section to extract a third feature map from the captured image, specifying a position corresponding to a third feature amount included in the third feature map and similar to the first feature amount, and determining a candidate region based on the specified position and the window setting, and An image obtained by superimposing an image indicating the candidate area on the captured image is output.

2. The image inspection device according to claim 1, in, The inspection setting section executes a second feature extraction section different from the first feature extraction section to extract a second feature map from the learning image, and extracts a second feature amount included in the second feature map and corresponding to a position specified by the window, and The inspection execution unit: executing the second feature extraction unit to extract a fourth feature map from the captured image and perform classification based on whether a fourth feature quantity included in the fourth feature map and corresponding to the specified position belongs to the same class as the second feature quantity, and In a case where the fourth feature amount is classified as belonging to the same class as the second feature amount, a detection result in which the candidate area is set as the detection area of ​​the object is output.

3. The image inspection device according to claim 1, in, the inspection setting section executes the first feature extraction section to extract the first feature map from each of the learning image and a rotation image obtained by rotating the learning image, and extract the first feature amount, The first feature quantity is included in the first feature map, reflects the angle of the window relative to the learning image, and corresponds to the position specified by the window, or is included in the first feature map, reflects the angle of the window relative to the rotated image, and corresponds to the position specified by the window, and When the third feature amount is similar to the first feature amount extracted from the rotational image, the inspection execution section determines the candidate region based on the designated position, the window setting, and information on a rotation angle of the rotational image.

4. The image inspection device according to claim 3, in, The inspection setting part: Receive rotation settings, When the rotation setting is on, executing the first feature extraction unit to extract the first feature map from each of the learning image and the rotated image, and When the rotation setting is off, the first feature extraction section is executed to extract the first feature map from the learning image.

5. The image inspection device according to claim 1, in, The inspection setting section extracts a background feature amount included in the first feature map and corresponding to a background position other than the window, and The inspection execution unit performs classification based on whether the third feature quantity belongs to the same class as the first feature quantity or the same class as the background feature quantity.

6. The image inspection device according to claim 2, in, The learning image is a first learning image, The window is a first window, The inspection setting part: receiving a second window setting for a second window of a second study image different from the first study image, In a state where an image obtained by superimposing an image indicating the candidate area on the captured image is output, the captured image can be set as the second learning image and the candidate area can be set as the second window, and executing the second feature extraction section to extract a fifth feature map from the second learning image, and extracting a fifth feature quantity included in the fifth feature map and corresponding to a position specified by the second window, and The inspection execution unit performs classification based on whether the third feature quantity belongs to the same class as the first feature quantity or the same class as the fifth feature quantity.

7. The image inspection apparatus according to claim 1, wherein: In a case where the candidate region does not exist, the inspection execution section outputs an image obtained by superimposing an image indicating that the candidate region does not exist on the captured image.

8. The image inspection apparatus according to claim 1, wherein: The imaging unit may be attached with a magnifying lens attachment capable of focusing the imaging unit on the imaging target when a distance between the imaging unit and the imaging target is equal to or smaller than a predetermined value.

Images