Image inspection device

Abstract

To count an object with high accuracy.SOLUTION: An image inspection device includes: an image capturing section that continuously captures a capturing field of view to generate a plurality of frame images aligned in time series; an inspection execution section that executes inspection processing of an object appearing in the plurality of frame images to output an inspection result; and an inspection setting section that performs setting of the inspection execution section. The inspection setting section receives setting of a window for a learning image on which the object appears, and the inspection execution section respectively detects target regions from the plurality of frame images on the basis of the learning image and the window set for the learning image, respectively acquires first positional information of the target region detected in a first frame image and second positional information of the target region detected in a second frame image, performs matching processing of the target region between the first frame image and the second frame image on the basis of the first positional information, the second positional information, and an overall moving direction of the object, and counts the object on the basis of a result of the matching processing.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an image inspection device. [Background technology]

[0002] Conventional image sensors generally perform object detection processing for one frame image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-164146 Summary of the Invention [Problem to be solved by the invention]

[0004] Image sensors can be used to robustly detect objects presented by a user against changes in the image features of the objects and count the objects passing through the imaging field of view. When counting objects passing through the imaging field of view using an image sensor that performs object verification processing on a single frame image, as in Patent Document 1, the output from the image sensor must be input to a programmable logic controller (PLC), which then executes a predetermined process. This requires time and effort to program the PLC.

[0005] Furthermore, when counting the passing of objects detected by an image sensor based on the output (e.g., ON / OFF) of the image sensor, it is necessary to increase the frame rate so that no images capable of detecting passing objects are missed.

[0006] For this reason, even when a single object passes, there will be multiple frame images that show that object. In other words, if you simply count the number of times the image sensor outputs ON, or the number of times the image sensor output is ON in the PLC scan cycle, you may end up counting the same object multiple times.

[0007] Furthermore, even if such circumstances are taken into consideration and it is determined that one count occurs when the output of the image sensor transitions from OFF to ON to OFF, the occurrence of double counting and missed counts cannot be completely eliminated.

[0008] For example, if the image sensor output is correctly determined as OFF → ON → ON → ON → ON → OFF for five consecutive frame images that show one object, the count will be "1." However, if the image sensor output is incorrectly determined as OFF → ON → "OFF" → ON → OFF, the count will be "2." This is a state in which duplicate counting occurs.

[0009] Also, for example, if the image sensor output is correctly determined as "OFF → ON → OFF → ON → OFF" for five consecutive frame images showing two objects, the count will be "2." However, if the image sensor output is incorrectly determined as "OFF → ON → "ON" → ON → OFF," the count will be "1." This is a state in which an error has occurred. For example, if multiple objects are lined up in a direction (up and down) that intersects with the direction of travel (left and right), or if multiple objects are close to each other, they may be counted as a single object.

[0010] In view of the above-mentioned problems, an object of the present invention is to provide an image inspection device that can count objects with high accuracy. [Means for solving the problem]

[0011] For example, an image inspection device according to the present invention comprises an imaging unit that continuously captures an imaging field of view to generate a plurality of frame images arranged in chronological order, an inspection execution unit that executes an inspection process for objects shown in the plurality of frame images and outputs the inspection results, and an inspection setting unit that configures the inspection execution unit, wherein the inspection setting unit accepts a window setting for a learning image in which the object is shown, and the inspection execution unit detects an object area from each of the plurality of frame images based on the learning image and the window set for the learning image, acquires first position information of the object area detected in a first frame image and second position information of the object area detected in a second frame image, performs a matching process for the object area between the first frame image and the second frame image based on the first position information, the second position information and the overall direction of movement of the object, and counts the object based on the results of the matching process.

[0012] Still other features, elements, steps, advantages, and characteristics will become more apparent from the detailed description that follows and the accompanying drawings related thereto. [Effects of the Invention]

[0013] The image inspection device according to the present invention can count objects with high accuracy. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an image inspection device according to an embodiment of the present invention during operation. FIG. [Figure 2] FIG. 2 is a hardware configuration diagram of the image inspection device. [Figure 3] FIG. 2 is a functional block diagram of the image inspection device. [Figure 4] FIG. 10 is a diagram showing a setting flow of a passage counting tool. [Figure 5] FIG. 5 is a diagram showing a GUI (graphical user interface) transition in FIG. 4. [Figure 6]FIG. 10 is a diagram showing an execution flow of a passage counting tool. [Figure 7] FIG. 10 is a conceptual diagram relating to a matching process between frame images. [Figure 8] FIG. 10 is a diagram showing GUI transitions for additional learning. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0016] FIG. 1 is a diagram illustrating an image inspection device S according to an embodiment of the present invention during operation. The image inspection device S captures an image of a workpiece W transported by a transport means A according to an imaging setting to acquire inference image data, detects the workpiece W in the image of the acquired inference image data, and outputs the detection result to an external device. An example of the external device is a programmable logic controller (PLC) 5, but a device other than the PLC 5 may also be the external device. The PLC 5 controls the transport means A based on the received detection result, for example, to separate the storage destination of the workpiece W. In the following description, a case will be described in which the external device is the PLC 5. Note that the workpiece W may be a workpiece that is not transported by the transport means A. In the following description, the workpiece is also referred to as the target object.

[0017] The image inspection device S includes an imaging unit 1 for imaging the workpiece W, a control unit 2 to which inference image data captured by the imaging unit 1 is input, a PC (personal computer) 3 for configuring the image inspection device S, and a display device 4 for displaying a setting screen, a selection screen, workpiece images, detection results, etc. The control unit 2 is capable of executing a trained model for detecting the workpiece W in the image of the input inference image data. The control unit 2 outputs to the PLC 5 according to the detection results obtained by the trained model.

[0018] Here, the image inspection device S may be used, for example, to inspect the workpiece W from various angles at various points in a manufacturing device or production line. For this reason, multiple image inspection devices S may be installed in one manufacturing device or one production line, potentially resulting in insufficient installation space and power supply. Therefore, the image inspection device S must be compact to accommodate the installation space and power-efficient to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a GPU (graphics processing unit). The control unit 2 executes a trained model that has undergone machine learning to the extent that it is capable of detecting the workpiece W. However, the image inspection device S is provided to the user by the vendor so that the desired detection accuracy can be achieved without the user having to perform advanced machine learning, for which the use of a GPU is recommended. Details will be described later. Because the user does not have to perform advanced machine learning, the user can execute a trained model capable of detecting the workpiece W without preparing a GPU for the training. Furthermore, the time required by the user to prepare a trained model capable of detecting the workpiece W can be reduced. Note that a single image inspection device S may be installed and operated in a manufacturing device or production line. The image inspection device S can also be called an image sensor.

[0019] (Configuration of imaging unit) The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to capture an image of the workpiece W from a desired direction. The workpieces W are sequentially transported by the transport means A into the imaging field of view of the imaging unit 1.

[0020] Fig. 2 is a hardware configuration diagram of the image inspection device S. As shown in Fig. 2, the imaging unit 1 includes an illumination module 10 for illuminating the workpiece W and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10.

[0021] The lighting module 10 has an LED (light emitting diode) 10a that irradiates light toward the workpiece W, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication section 20 (described later) of the control unit 2, and is controlled by a control section 21 (described later) of the control unit 2.

[0022] The camera module 11 has an AF motor 11a and an imaging board 11b. The AF motor 11a is a member for automatically focusing on the workpiece W by driving a focusing lens of an optical system (not shown). The autofocus method is not particularly limited, and examples include a contrast method.

[0023] The imaging board 11b is equipped with a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives light that is irradiated from the LED 10a onto the workpiece W and reflected by the workpiece W. This CMOS sensor 11c is connected to the head communication section 20 of the control unit 2, and is controlled by the control section 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0024] The FPGA 11d is a processing device whose internal processing contents can be changed. The DSP 11e is a signal processing device. A light-receiving amount signal of the light-receiving element of the CMOS sensor 11c is output to the FPGA 11d for processing, and is also output to the DSP 11e for processing. The processing by the FPGA 11d and the DSP 11e is not particularly limited, but examples thereof include various types of filter processing. The image data processed by the FPGA 11d and the DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0025] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Therefore, the control unit 2 can be installed in a location away from the location where the imaging unit 1 is installed.

[0026] (PC configuration) PC3 is configured as a general-purpose personal computer or the like. In this example, the personal computer can be used as PC3 by installing a predetermined program on it. PC3 includes operating devices such as a keyboard 3a and a mouse (not shown). A user of the image inspection device S can perform setting operations and selection operations on the image inspection device S by operating the operating devices on PC3. Specific setting operations and selection operations will be described later.

[0027] The PC 3 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other, and information based on setting operations by the user is sent from the PC 3 to the control unit 2. In addition, the PC 3 is able to receive image data of the workpiece W, inspection results, etc. output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Therefore, the PC 3 can be installed in a location away from where the control unit 2 is installed.

[0028] (Configuration of display device 4) The display device 4 is configured, for example, with a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component capable of detecting operations by the user's finger. The type of the touch panel 4a is not particularly limited, and examples include a capacitive type and an infrared type. The display device 4 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other. Operation information of the touch panel 4a by the user is transmitted from the display device 4 to the control unit 2. In addition, the display device 4 is capable of receiving image data of the workpiece W output from the control unit 2. The display device 4 and the control unit 2 are connected via a communication cable 7. Therefore, the display device 4 can be installed in a location away from the installation location of the control unit 2.

[0029] The PC 3 and the display device 4 may be configured as an integrated unit. For example, the display device 4 may be configured as a display device that the PC 3 has. In this case, the main body of the PC 3 and the display device 4 may be integrated or may be separate. In this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0030] (Configuration of control unit 2) 2, the control unit 2 includes a head communication section 20, a control section 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage section) 26. The head communication section 20 is connected to the control section 21 and is a section that executes mutual communication between the control section 21 and the imaging unit 1. A control signal for the imaging unit 1 output from the control section 21 is transmitted to the imaging unit 1 via the head communication section 20. The control signal for the imaging unit 1 includes a signal that controls the light emission timing and light emission amount of the LED 10a, and a signal that controls the AF motor 11a and the imaging board 11b. Furthermore, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the control section 21 via the head communication section 20.

[0031] The control unit 21 has a DSP 21a and FPGA 21b that perform various signal processing, an accelerator 21c that speeds up the processing, and a memory 21d made up of RAM, ROM, etc. The specific configuration of the control unit 21 will be described later.

[0032] The communication board 22 is connected to the control unit 21, and is a member that executes mutual communication between the control unit 21 and the PC 3, the display device 4, and the PLC 5.

[0033] The connector board 24 includes a power supply interface 24a. A power cable (not shown) for supplying power from an external source is connected to the power supply interface 24a. The connector board 24 is connected to a power supply 23, and power supplied from the outside to the power supply interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the control unit 21. The power supplied to the control unit 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0034] The I / O board 25 is connected to the control unit 21. The inspection results output from the control unit 21 are input to the PLC 5 via the I / O board 25.

[0035] (function block) 3 is a functional block diagram of the image inspection device S. As shown in this diagram, the image inspection device S includes an imaging setting unit 100, an inspection setting unit 200, and an inspection execution unit 300 as its functional blocks.

[0036] The imaging setting section 100 performs various settings (such as imaging field of view, image brightness, focus, and imaging interval (frame rate)) related to the imaging operation of the imaging unit 1. The imaging unit 1 can be understood as an imaging section that continuously captures the imaging field of view and generates a plurality of frame images FR arranged in chronological order.

[0037] The inspection setting section 200 performs various settings related to the inspection of the frame image FR by the inspection execution section 300. Referring to this figure, the inspection setting section 200 includes a tool setting section 210 and an inspection condition setting section 220.

[0038] The tool setting unit 210 sets various tools. Referring to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0039] The tool selection unit 211 selects a tool to be set and used.

[0040] The parameter setting unit 212 sets a rule-based tool. With the rule-based tool, inspection is performed based on various feature amounts (outline, color, position, etc.) of the workpiece W shown in the image.

[0041] The learning tool setting unit 213 sets a learning tool that uses a machine learning model. In the learning tool, a trained model such as a classifier is generated in response to user instructions, and testing is performed based on the output of the trained model. Referring to the figure, the learning tool setting unit 213 includes a training data setting unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0042] The learning data setting unit 213a sets learning data to be input to the machine learning model. The learning data includes learning images and instruction content. The learning images include, for example, at least one of images of good products and images of defective products. The instruction content includes label information such as "this image is a good product," "this image is a defective product," or "this part is defective." The label information includes information corresponding to the class into which the workpiece W should be classified. Referring to this figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a label information setting unit 213a2, and a learning data generation unit 213a3.

[0043] The learning image selection unit 213a1 selects learning images. The learning image selection unit 213a1 may have a function of presenting recommended learning images when setting up a passage counting tool, for example. Details will be described later.

[0044] The label information setting unit 213a2 receives label information for displaying the learning image selected by the learning image selection unit 213a1 on the GUI as learning data. For example, when setting a passage count tool, the label information setting unit 213a2 receives window settings for the learning image in which the workpiece W is displayed.

[0045] The learning data generation unit 213a3 generates learning data based on the learning images selected by the learning image selection unit 213a1 and the window settings accepted by the label information setting unit 213a2. The learning data generation unit 213a3 stores image features of the workpiece W, for example, based on the window settings.

[0046] The update unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches an expected value in accordance with the teachings. The parameter update can be understood as learning of the machine learning model. However, the user does not necessarily have to perform all of the steps of learning the machine learning model. For example, it is possible to complete relatively computationally intensive learning on the vendor side before shipping the image inspection device S, and then have the user perform only relatively computationally intensive learning before operating the image inspection device S. In this specification, learning performed by the vendor side before shipping is referred to as pre-shipment learning, and learning performed by the user side before operating the image inspection device S is referred to as customer-side learning.

[0047] For example, the machine learning model of the image inspection device S may include a feature extraction unit that is not customer-trained and a judgment unit that is customer-trained. The feature extraction unit extracts features from the image. The judgment unit outputs an inspection result based on the features.

[0048] That is, the machine learning model of the image inspection device S may include a parameter-fixed portion. The parameter-fixed portion is a layer in which parameters obtained by pre-shipment training on the vendor side are fixed, in other words, a layer in which on-site training on the user side is not required. Furthermore, the machine learning model of the image inspection device S may include a segmentation model that facilitates on-site training on the user side.

[0049] With this configuration, there is no need for users to prepare equipment with the high processing power required for deep learning, such as a GPU, or for vendors to provide an advanced learning environment using a GPU as a cloud service (such as SaaS), thereby lowering the barrier to introducing the image inspection device S.

[0050] As such, the above-mentioned learning should be broadly interpreted as including not only deep learning, which requires a large amount of computation, but also learning with a small amount of computation, i.e., customer learning in this specification. Note that, since customer learning is learning with a small amount of computation, the machine learning model may be trained using a method that does not involve machine learning techniques.

[0051] The inspection condition setting unit 220 determines the output conditions of the image inspection device S, in other words, the conditions of the sensor output, for example, by combining a plurality of tools.

[0052] The inspection execution unit 300 executes an inspection process on the workpiece W shown in the multiple frame images FR and outputs the inspection results. Referring to this figure, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0053] The tool execution unit 310 executes the tool selected as the tool to be used by the tool selection unit 211. Referring to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0054] When the tool selection unit 211 selects a rule-based tool as a tool to be used, the rule determination unit 311 executes the rule-based tool.

[0055] The learning tool execution unit 312 executes a learning tool when the tool selection unit 211 selects the learning tool as the tool to be used.

[0056] The inspection result output unit 320 outputs the inspection results in accordance with the output conditions set by the inspection condition setting unit 220. The inspection may include image classification, anomaly detection, and segmentation.

[0057] The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be configured solely with hardware, or may be configured with a combination of hardware and software. The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are realized by a single piece of hardware or software. The software described above can be executed by the control unit 2 (particularly the control unit 21) in which a program file and a setting file are installed.

[0058] The image inspection device S of this configuration example can be switched between a setting mode and an operation mode. In the setting mode, for example, various parameter settings such as imaging settings, registration of a master image, and generation (learning) of a classifier that classifies images are performed. Image classification here includes classification as to whether an image is a good or defective image, and the image inspection device S may be configured to determine whether the workpiece W is good or defective based on the classification results. Also, in the setting mode, for example, preliminary work is performed so that the user can distinguish between good and defective products in the desired product inspection.

[0059] In operation mode, the workpieces W are inspected based on the frame images FR captured at the actual site. Inspection of the workpieces W includes not only the pass / fail judgment described above, but also a piece count to count the number of workpieces W. Switching between setting mode and operation mode can be performed on the GUI, which will be described later. It is also possible to configure the system to automatically switch to operation mode immediately after completing setting mode. In operation mode, it is also possible to correct or change the classification boundary used by the classifier, i.e., to perform so-called additional learning.

[0060] (pass mode) Incidentally, the image inspection device S has, as one of its various modes, a passing mode in which workpieces W sequentially passing through the imaging field of the imaging unit 1 are detected. That is, the image inspection device S has a mode for performing time series processing independent of a mode in which one judgment is made for one frame image. In the passing mode, it is possible to set and operate a passing count tool as a tool for detecting multiple workpieces W sequentially passing through the imaging field of the imaging unit 1 and counting the number of workpieces W.

[0061] 4 is a diagram showing a setting flow of the passage counting tool. Note that it can be understood that the execution body of this flow is basically the test setting unit 200.

[0062] When this flow starts, in step S11, one mode is selected from multiple modes (e.g., standard mode, sorting mode, pass mode). In this embodiment, since the only tool that can be set in the pass mode is the pass count tool, a configuration in which the pass count tool is selected instead of the pass mode may be used. The following explanation assumes that the pass mode is selected in this step. In step S12, an overall movement direction Wd of the workpiece W is set. The overall movement direction Wd corresponds to the conveying direction of the conveying means A (e.g., a conveyor). In step S13, at least one training image is selected from multiple frame images FR. In step S14, a target region is determined from the training image. Specifically, the workpiece W shown in the training image is specified as a target region (ROI [region of interest]) by setting a window. In step S15, an object detection model, which is one of the machine learning models, is updated so that the target region can be detected from the training image. In step S16, a pass line for the workpiece W is set. The pass line can be understood as a dividing line that divides the detection range of the workpiece W into a pre-pass region and a post-pass region. In step S17, the test result outputs are assigned to a plurality of output ports.

[0063] Note that the passage counting tool in this embodiment detects the target area using an object detection model obtained by machine learning, and therefore this flow (particularly steps S12 to S14) includes the selection of a training image (step S12), etc. However, the target area is not limited to detection using an object detection model, and may be detected on a rule-based basis. If the image inspection device S detects the target area on a rule-based basis, for example, instead of steps S12 to S14, the image inspection device S may be configured to accept input of feature amounts (contour, color, position, etc.) required for detecting the workpiece W, and store the feature amounts.

[0064] Fig. 5 is a diagram showing GUI transitions corresponding to the setting flow of Fig. 4. When the image inspection program is executed on the PC 3, a GUI 400 (=various screens 400a to 400i) is displayed on the display device 4. The GUI 400 includes, as its basic layout, an image display area 410, an operation area 420, and a progress display area 430.

[0065] The image display area 410 displays a frame image FR captured by the imaging unit 1, etc. The image display area 410 may also be accompanied by a status display banner 411, a zoom in / out button 412, a maximize button 413, etc. The status display banner 411 clearly displays the operating status (status) of the image inspection program. The zoom in / out button 412 and the maximize button 413 are operated to zoom in / out and maximize the image displayed in the image display area 410, respectively.

[0066] When the image inspection device S is started for the first time, an initial startup screen 400a (first row in the left column) is displayed. When the image inspection device S is started for the first time, the image inspection program has not yet been set. Therefore, a warning mark a1 (or a warning message) indicating that the master image has not been registered may be displayed in the image display area 410. Furthermore, a banner (e.g., "Master") indicating that the master image has not been registered (or is in the process of being registered) may be displayed in the status display banner 411.

[0067] Additionally, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, the same applies below), setting of the image inspection program is started. This corresponds to the start of the setting flow shown in FIG.

[0068] When setting up an image inspection program, a mode selection screen 400b (second row on the left) is first displayed. The mode selection screen 400b corresponds to step S11 in Fig. 4. For example, the mode selection screen 400b may display a mode selection dialogue b0, as shown in this figure.

[0069] The mode selection dialogue b0 displays, for example, a standard mode button b1, a sorting mode button b2, a passing mode button b3, guidance b4, an OK button b5, and a cancel button b6.

[0070] When the standard mode button b1 is clicked, the standard mode is selected. In the standard mode, the workpieces W are distinguished as good or bad. When the sorting mode button b2 is clicked, the sorting mode is selected. In the sorting mode, the workpieces W are sorted (classified) into multiple classes. When the passing mode button b3 is clicked, the passing mode is selected. In the passing mode, the number of workpieces W that pass sequentially through the imaging field of the imaging unit 1 is counted. The guidance b4 may display an overview and a schematic diagram of the mode selected by clicking the standard mode button b1, the sorting mode button b2, or the passing mode button b3.

[0071] When the OK button b5 is clicked, the mode selection state is confirmed. On the other hand, when the Cancel button b6 is clicked, the mode selection state is canceled and the mode selection dialog b0 is closed. The following explanation assumes that the passing mode has been selected on the mode selection screen 400b (mode selection dialog b0).

[0072] When the pass mode is selected on the mode selection screen 400b, the pass mode setting work proceeds in the following order: first step (imaging setting), second step (master registration), third step (tool setting), and fourth step (output allocation). From this point on, each step is displayed in a flow diagram in the progress display area 430, and the currently executing step is preferably highlighted. This configuration allows the user to grasp the progress of the setting work at a glance.

[0073] First, in the first step (imaging setting), a movement direction setting screen 400c (third row in the left column) is displayed. The movement direction setting screen 400c corresponds to step S12 in Fig. 4. For example, a pull-down menu c1, a back button c2, and a forward button c3 may be displayed in the operation area 420 of the movement direction setting screen 400c.

[0074] The pull-down menu c1 displays multiple candidates for the overall movement direction Wd of the workpiece W (for example, four directions: "left to right," "right to left," "top to bottom," and "bottom to top"). In this figure, "right to left" has been selected as the overall movement direction Wd of the workpiece W. The overall movement direction Wd of the workpiece W may be displayed in the image display area 410. In this way, the image inspection device S (particularly the inspection setting unit 200) accepts the setting of the overall movement direction Wd when setting the pass mode. With this configuration, a matching result that meets the user's intentions can be obtained (details will be described later).

[0075] The primary target of surveillance cameras is people. Therefore, rather than assuming that all people move in the same direction, surveillance cameras distinguish between targets as "person a" and "person b" rather than simply "people" in order to predict the direction in which each person will move and track people who leave the field of view and then return.

[0076] On the other hand, in the image inspection device S mainly used in the factory automation field, it is assumed that all workpieces W to be inspected move in the same direction. Therefore, by setting the overall movement direction Wd of the workpieces W in advance, the matching accuracy of the workpieces W between frame images can be improved.

[0077] However, the image inspection device S is not necessarily limited to a configuration in which the user manually sets the overall movement direction Wd of the workpiece W. For example, the image inspection device S may be configured to estimate the movement direction of the workpiece W by repurposing existing surveillance camera technology and automatically set the estimated result as the overall movement direction Wd of the workpiece W.

[0078] Clicking the back button c2 returns to the mode selection screen 400b, whereas clicking the forward button c3 advances to the second step (master registration).

[0079] As shown in the figure, in the first step (imaging setting), a live image (=moving image being captured) may be displayed in the image display area 410. In this case, the status display banner 411 may display a banner (for example, "Live") indicating that a live image is being displayed.

[0080] Although not shown in the figure, in the first step (imaging setting), the imaging field of view, image brightness, focus, imaging interval (frame rate), and the like may be set.

[0081] In the second step (master registration), a master image selection screen 400d (fourth row in the left column) is displayed. The master image selection screen 400d corresponds to step S13 in Fig. 4. For example, a live button d1, a history button d2, a file button d3, a back button d4, and a forward button d5 may be displayed in the operation area 420 of the master image selection screen 400d.

[0082] When the live button d1 is clicked, the live image displayed in the image display area 410 at that time is registered as the master image. By clicking the history button d2, it is possible to register a master image from a driving history image (an inspected image). Also, by clicking the file button d3, it is possible to register a master image from a file image stored in storage. In this way, the setting methods for registering a master image include a method of registering from a live image, a method of registering from a driving history image, and a method of registering from a file image.

[0083] Clicking the back button d4 returns to the first step (imaging setting) mentioned above, whereas clicking the forward button d5 advances to the third step (tool setting).

[0084] In the third step (tool setting), a tool setting screen 400e (first row in the right column) is displayed. The tool setting screen 400e corresponds to step S14 in FIG. 4. For example, a rectangular button e1, a circular button e2, guidance e3, a start learning button e4, and a cancel button e5 may be displayed in the operation area 420 of the tool setting screen 400e. Furthermore, a banner indicating that the tool is being set (for example, "TOOL") may be displayed in the status display banner 411.

[0085] On the tool setting screen 400e, the workpiece W shown in the registered master image is designated as a region of interest (so-called ROI) by setting a window e0. For example, when the rectangular button e1 is clicked, a rectangular window e0 is displayed in the image display area 410. When the circular button e2 is clicked, a circular window (not shown) is displayed in the image display area 410. Note that the guidance e3 may also display information such as how to designate the region of interest.

[0086] The user can specify the target area by following the guidance e3 and adjusting the position, size, and angle of the window e0 so that it surrounds the workpiece W shown in the master image. Label information (class) may be added to the specified target area. Multiple target areas may be specified. However, it is not necessary for the user to specify all target areas, and some may be specified automatically. The image inspection device S may also have a function to automatically learn the state of the target area when rotated.

[0087] After the target area is specified, clicking the Start Learning button e4 starts learning of the machine learning model (object detection model) used in the pass mode. On the other hand, clicking the Cancel button e5 cancels the specification of the target area.

[0088] When learning of the machine learning model begins, a learning progress screen 400f (second row in the right column) is displayed. The learning progress screen 400f corresponds to step S15 in FIG. 4. The image inspection device S (particularly the inspection setting unit 200) inputs a learning image (master image) into the machine learning model as part of learning the machine learning model, thereby extracting and storing a first feature amount (e.g., a feature vector of the target area) corresponding to the image features of the work W from the learning image. In this embodiment, the first feature amount corresponding to the image features of the work W is extracted and stored as part of learning the machine learning model, but the machine learning model may also be configured to update parameters based on the image features of the work W so that it can detect the work W.

[0089] Note that the learning progress screen 400f may display a learning progress dialog f0 as shown in this drawing. The learning progress dialog f0 displays, for example, a learning progress bar f1 indicating the progress level (0% to 100%).

[0090] When the learning of the machine learning model is completed, a passing line setting screen 400g (third row in the right column) is displayed. The passing line setting screen 400g corresponds to step S16 in Fig. 4. For example, a valid button g1, an invalid button g2, guidance g3, a back button g4, and a forward button g5 may be displayed in the operation area 420 of the passing line setting screen 400g.

[0091] The enable button g1 and the disable button g2 are used to enable / disable the conveyor end mode (drop mode), respectively. The conveyor end mode may be enabled, for example, when counting workpieces W that drop from the conveyor. When the enable button g1 is clicked, the conveyor end mode is enabled. On the other hand, when the disable button g2 is clicked, the conveyor end mode is disabled. The guidance g3 may also display an overview and a schematic diagram of the conveyor end mode.

[0092] A passing line g0 that separates the pre-passing area X1 and post-passing area X2 of the workpiece W is displayed in the image display area 410. In other words, the detection range of the workpiece W is divided into the pre-passing area X1 and the post-passing area X2. When the conveyor end mode is disabled, the passing line g0 is fixed at the center of the image display area 410. On the other hand, when the conveyor end mode is enabled, the user's setting of the passing line g0 is accepted. For example, the passing line g0 may be set to align with the end of the conveyor in accordance with the guidance g3. With this configuration, the pre-passing area X1 and the post-passing area X2 can be arbitrarily determined in accordance with the user's intentions.

[0093] When the conveyor terminal mode is disabled, if movement of a workpiece W across the passing line g0 is detected, that is, if a workpiece W detected in the pre-pass area X1 in a frame image is detected in the post-pass area X2 in the subsequent frame image, a count is performed. On the other hand, if the conveyor terminal mode is enabled, a count is performed upon detecting the disappearance of a workpiece W detected in the pre-pass area X1 in the post-pass area X2. That is, in the conveyor terminal mode, the number of workpieces W that have fallen from the end of the conveyor (workpieces W in the lower basket) is counted.

[0094] Therefore, if the conveyor end mode is enabled, even if the passing line g0 is not set to the end of the conveyor and a work W that has crossed the passing line g0 has not yet fallen off the end of the conveyor, there will be no discrepancy between the count value of the work W and the number of work W actually placed in the basket.

[0095] Clicking the back button g4 returns to the second step (master registration) mentioned above, whereas clicking the forward button g5 advances to the fourth step (output allocation).

[0096] In the fourth step (output allocation), an output allocation screen 400h (fourth row in the right column) is displayed. The output allocation screen 400h corresponds to step S17 in Fig. 4. In the operation area 420 of the output allocation screen 400h, for example, a pull-down menu h1 for each of the output ports OUT1 to OUT8, a back button h2, and a done button h3 may be displayed.

[0097] The pull-down menu h1 displays multiple candidates for the output content of each of the output ports OUT1 to OUT8. In this diagram, the work W count result (COUNT UP) is selected as the output content of output port OUT1. Furthermore, the busy detection result (BUSY) is selected as the output content of output port OUT2. Furthermore, the error detection result (ERROR) is selected as the output content of output port OUT3. Furthermore, all of the output ports OUT4 to OUT8 are unused (OFF). With this configuration, it is possible to output a variety of information in multiple bits, such as not only the work W count result but also the busy detection result and error detection result.

[0098] Clicking the back button h2 returns to the third step (tool setting) described above. On the other hand, clicking the done button h3 completes the setting work for the pass mode.

[0099] 6 is a diagram showing the execution flow of the passage counting tool. Note that the execution body of this flow can be understood as basically being the inspection execution unit 300. Furthermore, the multiple frame images FR sequentially generated by the imaging unit 1 include frame images FR1 and FR2.

[0100] When this flow starts, in step S21, a target area is detected from the frame image FR1 based on the image features (first feature amount) of the workpiece W. For example, the inspection execution unit 300 may input the frame image FR1 into a machine learning model (object detection model) to detect a target area from the frame image FR1 from which a second feature amount closest to the first feature amount can be obtained. In this way, a configuration in which the target area is detected using a machine learning model improves robustness against changes in image features.

[0101] In step S22, position information Dxy(FR1) of the target region detected in the frame image FR1 is acquired. If multiple target regions are detected in the frame image FR1, position information Dxy1(FR1), Dxy2(FR1), ..., Dxy*(FR1) corresponding to each of them is acquired. For example, the inspection execution unit 300 may acquire the position information Dxy(FR1) of the target region by comparing the first feature amount of the workpiece W with the second feature amount of each region in the frame image FR1.

[0102] In step S23, a target area is detected from the frame image FR2 based on the image features (first feature amount) of the workpiece W. For example, the inspection execution unit 300 may input the frame image FR2 into a machine learning model (object detection model) to detect a target area from the frame image FR2 from which a second feature amount closest to the first feature amount can be obtained. This is the same as in step S21 described above.

[0103] In step S24, position information Dxy(FR2) of the target area detected in frame image FR2 is acquired. If multiple target areas are detected in frame image FR2, position information Dxy1(FR2), Dxy2(FR2), ..., Dxy*(FR2) corresponding to each of them is acquired. For example, the inspection execution unit 300 may acquire the position information Dxy(FR2) of the target area by comparing a first feature amount of the workpiece W with a second feature amount of the target area in frame image FR2. This is the same as in step S21 described above.

[0104] In step S25, matching processing of the target area is performed between the frame images FR1 and FR2 based on the position information Dxy(FR1) and Dxy(FR2) and the overall movement direction Wd of the workpiece W. This matching processing is performed as a pre-process of processing to detect how the workpiece W is moving (so-called tracking processing).

[0105] In step S26, the target regions are counted based on the results of the matching process. For example, the inspection execution unit 300 may count only the target regions that are determined to be located in the pre-passage region X1 in the frame image FR1 and the post-passage region X2 in the frame image FR2 based on the results of the matching process.

[0106] FIG. 7 is a conceptual diagram relating to the matching process carried out between the frame image FR(t) and the frame image FR(t+1).

[0107] In the frame image FR(t), four workpieces W are detected as the aforementioned region of interest (ROI), and are assigned ID1 to ID4 in order from left to right. Next, four workpieces W are also detected in the frame image FR(t+1), which was captured one frame later.

[0108] In conventional tracking processing, a method is generally used in which the same ID is assigned to the detection results with the closest relative positions between frame image FR(t) and frame image FR(t+1). When this method is used, as shown in the matching result M1, ID1 to ID4 are assigned sequentially from left to right to the four workpieces W shown in frame image FR(t+1).

[0109] However, at a site where the image inspection device S is installed, all workpieces W placed on the conveyor move uniformly along the overall movement direction Wd (from right to left in this figure). In other words, the frame image FR(t+1) shows the workpiece W located at the leftmost position in frame FR(t) moving out of the left edge of the imaging field of view, and a new workpiece W moving in from the right edge of the imaging field of view.

[0110] Therefore, for the four workpieces W shown in the frame image FR(t+1), the matching result M1 is incorrect, and they should be assigned ID2 to ID5 in order from left to right, as shown in the matching result M2. However, unless the overall movement direction Wd of the workpieces W is given, it is difficult to determine which of the matching results M1 and M2 is correct.

[0111] On the other hand, if the user has set the overall movement direction Wd of the workpiece W in advance, a correct matching result M2 can be obtained based on the position information Dxy of the workpiece W and the overall movement direction Wd shown in each of the frame images FR(t) and FR(t+1). As a result, the position information Dxy of each of the multiple workpieces W can be accurately obtained based on the matching result M2, making it possible to reduce duplicate counting and omissions in counting of the workpieces W.

[0112] Furthermore, at sites where the image inspection device S is installed, the conveyor often stops while in operation. At such times, the workpiece W may move significantly differently before and after the conveyor stops. Conventional tracking processes cannot handle such cases. On the other hand, as described above, if the overall movement direction Wd of the workpiece W is set, it is possible to determine whether the conveyor has stopped by comparing the detection results of the region of interest (ROI) in each of the multiple frame images FR. Therefore, it is possible to correctly count the workpiece W even if the conveyor has stopped.

[0113] The inspection execution unit 300 may also detect an abnormal condition and output a warning by comparing the detection results of the region of interest (ROI) in each of the multiple frame images FR. Examples of such abnormal conditions include the sudden appearance (overdetection) or disappearance of a target region outside the boundaries of the imaging field of view (the upstream and downstream ends of the overall movement direction Wd). The inspection execution unit 300 may output a warning without delay when an abnormal condition is detected in even one frame. Alternatively, the inspection execution unit 300 may wait to output a warning unless an abnormal condition is detected in multiple frames. Furthermore, when the conveyor end mode is enabled, the inspection execution unit 300 may allow the target region to disappear in the post-passage area X2. More specifically, when the conveyor end mode is enabled, the inspection execution unit 300 does not output a warning even if a workpiece W detected in the pre-passage area X1 disappears in the post-passage area X2, but instead counts up when the workpiece W disappears from the post-passage area X2.

[0114] In the conveyor end mode in this embodiment, the condition for counting up is the disappearance of a workpiece W from the post-pass area X2, but it may also be configured to count up when a workpiece W disappears from the pre-pass area X1. For example, even if a workpiece W that has fallen from the conveyor end is included in the imaging field of view, by locating the passing line g0 at the conveyor end, it is possible to count the workpiece W that has fallen from the conveyor end with high accuracy.

[0115] (Additional learning) 8 is a diagram showing the GUI transition for additional learning in the operation mode (or during test operation in the setting mode, the same applies below). When the image inspection device S is in the operation mode, an operation screen 400i is displayed on the display device 4. The operation screen 400i may include, for example, a thumbnail display area 440 in addition to the image display area 410 and operation area 420 mentioned above.

[0116] For example, an image operation button i1 may be displayed in the image display area 410. The image operation button i1 is operated to play / pause, fast-rewind / fast-forward, and frame-rewind / frame-forward a frame image (for example, a driving history image including an inspected frame image) displayed in the image display area 410.

[0117] In the operation area 420, for example, an additional learning button i2 and a recommended learning image button i3 are displayed. When the additional learning button i2 is clicked, the screen transitions to a tool setting screen (see, for example, the above-mentioned tool setting screen 400e or the below-mentioned tool setting screen 400k) for additional learning of the frame image displayed in the image display area 410 at that time. When the recommended learning image button i3 is clicked, the screen transitions to an image selection screen 400j.

[0118] A recommended learning image j1 is displayed in the thumbnail display area 440 of the image selection screen 400j. As the recommended learning image j1, for example, a frame image FR from a section in which an abnormal state was detected during the tracking process described above may be picked out from among multiple frame images FR. In the recommended learning image j1, there is a high possibility that the background is detected as the target area or that the workpiece W is not detected as the target area. Therefore, the recommended learning image j1 can be said to be suitable as a learning image for additional training of a machine learning model (object detection model). Note that the recommended learning image j1 selected in the thumbnail display area 440 may be displayed in the image display area 410.

[0119] The operation area 420 of the image selection screen 400j displays, for example, an OK button j2 and a Cancel button j3. Clicking the OK button j2 transitions to a tool setting screen 400k for performing additional learning on the recommended learning image j1 currently displayed in the image display area 410. That is, the test setting unit 200 accepts the recommended learning image j1 as a learning image for additional learning. On the other hand, clicking the Cancel button j3 cancels the selection of the recommended learning image j1.

[0120] The operation area 420 of the tool setting screen 400k displays, for example, an add button k1, a delete button k2, a tool selection box k3, a start learning button k4, and a cancel button k5. When the add button k1 is clicked, a window k0 is added to the image display area 410. The user can specify the target area by adjusting the position, size, and angle of the window k0 so that it surrounds the workpiece W. On the other hand, when the delete button k2 is clicked, the selected window k0 is deleted. In this way, the inspection setting unit 200 can select a learning image for additional learning from the inspection result screen that displays the frame image FR and accept the setting of the window k0.

[0121] The tool selection box k3 is used to select the tool for which additional learning is desired when there are multiple tools. In other words, multiple tools can be additionally learned from a single frame image.

[0122] After the target area is specified, clicking the Start Learning button k4 starts additional learning of the machine learning model (object detection model) used in the pass mode. On the other hand, clicking the Cancel button k5 cancels the specification of the target area.

[0123] <Other> In addition to the above-described embodiments, the various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above-described embodiments should be considered to be illustrative and not restrictive in all respects. Furthermore, the technical scope of the present invention is defined by the claims, and should be understood to include all modifications that fall within the meaning and scope of the claims. [Explanation of symbols]

[0124] 1 Imaging unit 2. Control Unit 3 PC (Personal Computer) 3a keyboard 4 Display device 4a Touch panel 5 PLC (Programmable Logic Controller) 6 Communication Cable 7. Communication Cable 10 Lighting Module 10a LED (light emitting diode) 10b LED Driver 11 Camera module 11a AF motor 11b Imaging board 11c CMOS sensor 11d FPGA 11e DSP 20 Head communication unit 21 Control section 21a DSP 21b FPGA 21c Accelerator 21d Memory 22 Communication board 23 Power supply 24 Connector board 24a power interface 25 I / O board 26 Storage device (storage unit) 100 Imaging setting section 200 Inspection setting section 210 Tool setting section 211 Tool selection section 212 Parameter setting section 213 Learning Tool Settings 213a Learning data setting unit 213a1 Learning image selection unit 213a2 Label information setting section 213a3 Learning data generation unit 213b Update section 220 Inspection condition setting section 300 Inspection Execution Department 310 Tool Execution Unit 311 Rule Judgment Unit 312 Learning Tool Execution Department 320 Inspection result output unit 400 GUI 400a initial startup screen 400b Mode selection screen 400c Movement direction setting screen 400d Master Image Selection Screen 400e tool setting screen 400f Learning progress screen 400g passing line setting screen 400h Output Allocation Screen 400i driving screen 400j Image selection screen 400k tool setting screen 410 Image display area 411 Status Display Banner 412 Zoom In / Out Button 413 Maximum display button 420 Operation area 430 Progress display area 440 Thumbnail display area a1 Caution mark a2 Setting start button b0 Mode selection dialog b1 Standard mode button b2 Sorting mode button b3 Passing mode button b4 Guidance b5 OK button b6 Cancel button c1 Drop-down menu c2 Back button c3 Next button d1 Live button d2 History button d3 file button d4 Back button d5 Next button e0 window e1 Rectangular button e2 circular button e3 guidance e4 Start learning button e5 Cancel button f0 Learning progress dialog f1 Learning progress bar g0 passing line g1 Enabled Button g2 Disabled button g3 Guidance g4 Back button g5 Forward button h1 Drop-down menu h2 Back button h3 Complete button i1 Image Operation Buttons i2 Additional Learning Button j1 Recommended learning images j2 OK button j3 Cancel button k0 window k1 Add button k2 Delete button k3 Tool Selection Box k4 Start learning button k5 Cancel button A. Means of transport (conveyor) Dxy(FR1), Dxy(FR2) location information FR, FR1, FR2, FR(t), FR(t+1) frame images M1, M2 matching results S Image inspection device X1 Pre-pass area X2 area after passing W work (object) Wd Overall movement direction

Claims

1. an imaging unit that continuously captures an imaging field of view to generate a plurality of frame images arranged in time series; an inspection execution unit that executes an inspection process on the object shown in the plurality of frame images and outputs an inspection result; an inspection setting unit that sets the inspection execution unit; Equipped with the inspection setting unit accepts a window setting for a learning image in which the object is shown; The inspection execution unit detects a target area from each of the multiple frame images based on the learning image and the window set for the learning image, obtains first position information of the target area detected in a first frame image and second position information of the target area detected in a second frame image, performs a matching process for the target area between the first frame image and the second frame image based on the first position information, the second position information and the overall movement direction of the object, and counts the object based on the result of the matching process.

2. the inspection setting unit stores image features of the object based on the setting of the window; The image inspection device according to claim 1 , wherein the inspection execution unit detects a target region from each of the plurality of frame images based on the image features of the target object.

3. the inspection setting unit inputs the training images into a machine learning model to extract and store first feature amounts corresponding to the image features of the object from the training images; 3. The image inspection device according to claim 2, wherein the inspection execution unit detects the target area from which a second feature closest to the first feature can be obtained by inputting the frame image into the machine learning model.

4. The image inspection device according to claim 1 , wherein the inspection setting unit accepts a setting of the overall movement direction.

5. The detection range of the object is divided into a pre-passage area and a post-passage area, 2. The image inspection device of claim 1, wherein the inspection execution unit counts only the target areas that are determined to be located in the pre-passage area in the first frame image and that are determined to be located in the post-passage area in the second frame image based on the results of the matching process.

6. The image inspection device according to claim 5 , wherein the inspection setting unit accepts the setting of a passing line that separates the pre-passage area and the post-passage area.

7. the inspection setting unit accepts a setting that allows the target region to disappear in the post-passage region; 6. The image inspection device according to claim 5, wherein when the inspection setting unit receives a setting that allows the target area to disappear, the inspection execution unit counts the target area even if the target area disappears in the post-pass area.

8. 2. The image inspection device according to claim 1, wherein the inspection execution unit detects an abnormal state by comparing the detection results of the target region in each of the plurality of frame images and outputs a warning.

9. The image inspection device according to claim 8 , wherein the inspection setting unit accepts, as a recommended learning image, a frame image in a section in which the abnormal state is detected, from among the plurality of frame images.

10. The image inspection device according to claim 1 , wherein the inspection setting unit accepts the window setting by selecting the learning image for additional learning from an inspection result screen that displays the frame image.

Images