Image inspection device

Abstract

The invention relates to an image inspection apparatus. The image inspection apparatus includes: an imaging section configured to generate a plurality of frame images arranged in a time sequence; an inspection execution unit for outputting an inspection result; and an inspection setting unit for setting the inspection execution unit. The inspection setting section receives a setting of a window for a learning image in which an object appears. The inspection execution unit detects a target region from the first frame image and the second frame image on the basis of the learning image and a window set for the learning image. A matching process of a target region is performed between the first frame image and the second frame image based on the target region detected from the first frame image and the second frame image and an overall moving direction of the object, and the object is counted based on a result of the matching process.

Description

Technical Field

[0001] The present disclosure relates to an image inspection device. Background Art

[0002] An image sensor in the related art generally performs detection processing of an object on one frame image (for example, see Japanese Patent Laid-Open No. 2022-164146).

[0003] Incidentally, the image sensor can be used to robustly detect an object presented by a user with respect to changes in image characteristics of the object, and count the objects passing through the photographing field of view.

[0004] When counting objects passing through a field of view using an image sensor that verifies the objects for a single frame image, as in Japanese Patent Application Laid-Open No. 2022-164146, the output from the image sensor is input to a programmable logic controller (PLC), which then executes a predetermined process. Consequently, programming the PLC requires time and effort.

[0005] In addition, when attempting to count the passage of an object detected by an image sensor based on its output (eg, ON or OFF), the frame rate needs to be increased so that there is no omission in capturing an image capable of detecting a passing object.

[0006] Therefore, even when an object passes through, there are multiple frame images in which the object appears. In other words, when simply counting the number of times the image sensor output is on or the number of times the image sensor output is on during a PLC scan cycle, the same object is redundantly counted.

[0007] In addition, considering such a case, for example, even in the case where the output of the image sensor transitions from OFF→ON→OFF to determine one count, count duplication and count omission cannot be completely eliminated.

[0008] For example, if the image sensor output is correctly determined to be OFF→ON→ON→ON→OFF for five consecutive frame images of a single object, the count becomes "1." However, if the image sensor output is incorrectly determined to be OFF→ON→OFF→ON→OFF, the count becomes "2." This is a state where count overlap occurs.

[0009] Furthermore, for example, if the image sensor output is correctly determined to be "OFF→ON→OFF→ON→OFF" for five consecutive frame images in which two objects appear, the count number is "2." However, if the image sensor output is incorrectly determined to be OFF→ON→ON→ON→OFF, the count number is "1," indicating a state where a count omission occurs. For example, if multiple objects are arranged in a direction (vertical direction) intersecting the direction of travel (horizontal direction) or if multiple objects are close to each other, multiple objects may be counted as one object. Summary of the Invention

[0010] In view of the above-mentioned problems, an object of the present invention is to provide an image inspection apparatus capable of counting objects with high accuracy.

[0011] For example, an image inspection device according to the present invention includes: an imaging unit for continuously photographing a photographing field of view to generate a plurality of frame images arranged in a time series; an inspection execution unit for executing an inspection process for an object appearing in the plurality of frame images to output an inspection result; and an inspection setting unit for performing settings for the inspection execution unit. The inspection setting unit receives a setting of a window for a learning image in which the object appears, and the inspection execution unit detects a target area from the plurality of frame images based on the learning image and the window set for the learning image, obtains first position information of the target area detected in the first frame image and second position information of the target area detected in the second frame image, performs matching processing of the target area between the first frame image and the second frame image based on the first position information, the second position information, and the overall movement direction of the object, and counts the objects based on the result of the matching processing.

[0012] Note that other characteristics, elements, steps, advantages, and features will become more apparent from the following detailed description and accompanying drawings.

[0013] The image inspection apparatus according to the present invention can count objects with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram for explaining the operation of the image inspection apparatus according to the embodiment of the present invention;

[0015] Figure 2 It is a hardware structure diagram of the image inspection device;

[0016] Figure 3 It is a functional block diagram of an image inspection device;

[0017] Figure 4is a diagram illustrating a setting flow of a pass counting tool;

[0018] Figure 5 This is an example Figure 4 A diagram of a graphical user interface [GUI] transition;

[0019] Figure 6 is a diagram illustrating the execution flow of a pass counting tool;

[0020] Figure 7 is a conceptual diagram related to the matching process between frame images; and

[0021] Figure 8 1 is a diagram illustrating GUI transitions during additional learning. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following description of the preferred embodiment is merely exemplary in nature and is not intended to limit the invention, its application, or its intended uses.

[0023] Figure 1 : is a diagram for explaining the image inspection device S according to an embodiment of the present invention when in operation. For example, the image inspection device S shoots the workpiece W conveyed by the conveying unit A according to the shooting settings to obtain inference image data, detects the workpiece W in the image of the acquired inference image data, and outputs the detection result to an external device. Examples of external devices include a programmable logic controller (PLC) 5, etc., but a device other than the PLC 5 may be an external device. Based on the received detection result, the PLC 5 controls the conveying unit A, for example, to separate the storage destination of the workpiece W. In the following description, the case where the external device is the PLC 5 will be described. Note that the workpiece W may be a workpiece that is not conveyed by the conveying unit A. In addition, in the following description, the workpiece is also referred to as an object.

[0024] The image inspection device S includes an imaging unit 1 for capturing images of a workpiece W; a control unit 2 to which the inference image data captured by the imaging unit 1 is input; a personal computer (PC) 3 for configuring the image inspection device S; and a display device 4 for displaying settings, selection screens, workpiece images, and inspection results. The control unit 2 executes a trained model for detecting the workpiece W in the input inference image data. The control unit 2 outputs the inspection results corresponding to the trained model to a programmable logic controller 5.

[0025] Here, for example, when inspecting a workpiece W from various angles at various stages of a manufacturing facility or production line, an image inspection device S can be used. Consequently, multiple image inspection devices S may be installed in a single manufacturing facility or production line, and it is conceivable that sufficient installation space and power supply may not be secured. Therefore, the image inspection device S needs to be compact enough to accommodate the installation space and energy-efficient enough to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a graphics processing unit (GPU). The control unit 2 executes a trained model (in which machine learning is performed to the extent that the workpiece W can be detected), but the image inspection device S is provided to the user from a supplier so that the desired detection accuracy can be achieved even if the user does not perform the advanced machine learning recommended using a GPU. Details will be described later. Since the user does not need to perform advanced machine learning, the user can execute a trained model capable of detecting the workpiece W without having to prepare a GPU suitable for learning. Furthermore, the time it takes for the user to prepare a trained model capable of detecting the workpiece W can be shortened. Note that a single image inspection device S can be installed and operated in a manufacturing facility or production line. Furthermore, the image inspection device S can also be referred to as an image sensor.

[0026] (Structure of the camera unit)

[0027] The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to image the workpiece W from a desired direction. The workpiece W is sequentially transported by the transport unit A to the imaging field of the imaging unit 1 .

[0028] Figure 2 This is the hardware structure diagram of the image inspection device S. Figure 2 As illustrated, the imaging unit 1 includes an illumination module 10 for illuminating a workpiece W, and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10 .

[0029] The lighting module 10 includes a light-emitting diode (LED) 10a that illuminates the workpiece W with light, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication unit 20 (described later) of the control unit 2 and is controlled by a controller 21 (described later) of the control unit 2.

[0030] The camera module 11 includes an AF motor 11a and a pickup board 11b. The AF motor 11a drives a focus lens of an optical system (not illustrated) to automatically focus on the workpiece W. The automatic focusing method is not particularly limited, and examples thereof include a contrast method and the like.

[0031] The imaging board 11b includes a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives reflected light emitted from the LED 10a toward the workpiece W and reflected by the workpiece W. The CMOS sensor 11c is connected to the head communication unit 20 of the control unit 2 and is controlled by the controller 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0032] The FPGA 11d is a processing device capable of changing the content of internal processing. The DSP 11e is a signal processing device. The light-receiving amount signal from the light-receiving element included in the CMOS sensor 11c is output to and processed by the FPGA 11d, and is also output to and processed by the DSP 11e. The processing performed by the FPGA 11d and DSP 11e is not particularly limited, and examples include various filtering processes. Image data processed by the FPGA 11d and DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0033] The camera unit 1 and the control unit 2 are connected via a communication cable 6. Thus, the control unit 2 can be installed at a location far away from the installation location of the camera unit 1.

[0034] (PC structure)

[0035] The PC 3 is composed of a general-purpose personal computer or the like. In this example, the PC 3 can be used by installing a predetermined program in the personal computer. The PC 3 includes operating devices such as a keyboard 3a and a mouse (not shown). The user of the image inspection device S can perform setting operations and selection operations for the image inspection device S by operating the operating devices of the PC 3. Specific setting operations and selection operations will be described later.

[0036] The PC 3 is connected to the communication board 22 of the control unit 2 so as to communicate with each other, and information based on setting operations performed by the user is transmitted from the PC 3 to the control unit 2. In addition, the PC 3 can receive image data and inspection results of the workpiece W output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Thus, the PC 3 can be installed at a location far from the installation location of the control unit 2.

[0037] (Structure of Display Device 4)

[0038] The display device 4 includes, for example, a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component that can detect operations using the user's fingers. The type of touch panel 4a is not particularly limited, and examples thereof include a capacitive type and an infrared type. The display device 4 is connected to the communication board 22 of the control unit 2 in a manner that allows them to communicate with each other. User operation information on the touch panel 4a is sent from the display device 4 to the control unit 2. In addition, the display device 4 can receive image data of the workpiece W output from the control unit 2, etc. The display device 4 and the control unit 2 are connected via a communication cable 7. Thus, the display device 4 can be installed in a place away from the installation place of the control unit 2.

[0039] Note that the PC 3 and the display device 4 may be integrally provided. For example, the display device 4 may be formed by a display device included in the PC 3. In this case, the main body of the PC 3 and the display device 4 may be integrally provided, or may be separate from each other. Furthermore, in this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0040] (Structure of Control Unit 2)

[0041] like Figure 2 As shown, the control unit 2 includes a head communication unit 20, a controller 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage unit) 26. The head communication unit 20 is connected to the controller 21 and performs communication between the controller 21 and the imaging unit 1. Control signals for the imaging unit 1 output from the controller 21 are transmitted to the imaging unit 1 via the head communication unit 20. The control signals for the imaging unit 1 include signals for controlling the timing and amount of light emission of the LED 10a, as well as signals for controlling the AF motor 11a and the capturing board 11b. In addition, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the controller 21 via the head communication unit 20.

[0042] The controller 21 includes a DSP 21a and an FPGA 21b that perform various signal processing, an accelerator 21c for accelerating processing, and a memory 21d including a RAM and a ROM, etc. The specific structure of the controller 21 will be described later.

[0043] The communication board 22 is a member that is connected to the controller 21 and performs communication among the controller 21 , the PC 3 , the display device 4 , and the PLC 5 .

[0044] The connector plate 24 includes a power interface 24a. A power cable (not shown) for supplying power from the outside is connected to the power interface 24a. The connector plate 24 is connected to the power supply 23, and the power supplied from the outside to the power interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the controller 21. The power supplied to the controller 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0045] The I / O board 25 is connected to the controller 21. The inspection result output from the controller 21 is input to the PLC 5 via the I / O board 25.

[0046] (Function Block)

[0047] Figure 3 2 is a functional block diagram of an image inspection apparatus S. As illustrated in the figure, the image inspection apparatus S includes a photographing setting unit 100 , an inspection setting unit 200 , and an inspection execution unit 300 as its functional blocks.

[0048] The imaging setting section 100 makes various settings related to the imaging operation of the imaging unit 1 (such as the imaging field of view, image brightness, focus, and imaging interval (frame rate)). Note that the imaging unit 1 can be understood as an imaging section that continuously captures the imaging field of view to generate a plurality of frame images FR arranged in a time series.

[0049] The inspection setting section 200 performs various settings related to the inspection of the frame image FR using the inspection execution section 300. According to the drawing, the inspection setting section 200 includes a tool setting section 210 and an inspection condition setting section 220.

[0050] The tool setting unit 210 sets various tools. According to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0051] The tool selection unit 211 selects a tool to be set and a tool to be used.

[0052] The parameter setting section 212 sets a rule-based tool. In the rule-based tool, inspection is performed based on various feature quantities (contour, color, position, etc.) of the workpiece W appearing in the image.

[0053] The learning tool setup unit 213 sets up tools for a learning system that uses machine learning models. In the learning system tools, trained models such as discriminators are generated based on user instruction, and checks are performed based on the output of the trained models. As shown in the figure, the learning tool setup unit 213 includes a learning data setup unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0054] The learning data setting unit 213a sets the learning data to be input into the machine learning model. The learning data includes learning images and teaching content. For example, the learning images include at least one of an image of a non-defective product and an image of a defective product. The teaching content includes label information such as "This image is a non-defective product," "This image is a defective product," and "This part is defective." The label information includes information corresponding to the class into which the workpiece W is classified. As shown in the figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a label information setting unit 213a2, and a learning data generation unit 213a3.

[0055] The learning image selection unit 213a1 selects a learning image. For example, the learning image selection unit 213a1 may have a function of presenting a learning-recommended image when setting the pass count tool. Details will be described later.

[0056] The label information setting unit 213a2 receives label information for displaying the learning image selected by the learning image selection unit 213a1 on the GUI and using the learning image as learning data. For example, when setting the pass counting tool, the label information setting unit 213a2 receives window settings for the learning image on which the workpiece W appears.

[0057] The learning data generating unit 213a3 generates learning data based on the learning image selected by the learning image selecting unit 213a1 and the window setting received by the label information setting unit 213a2. For example, the learning data generating unit 213a3 stores the image characteristics of the workpiece W based on the window setting.

[0058] The updating unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches the expected value corresponding to the teaching content. The updating of the parameters can be understood as the learning of the machine learning model. However, the learning of the machine learning model does not necessarily need to be performed by the user in all processes. For example, learning with a relatively large amount of calculation can be completed on the supplier side before the image inspection device S is shipped, and only learning with a relatively small amount of calculation can be performed on the user side before the operation of the image inspection device S. In this specification, the learning performed by the supplier side before shipping is referred to as pre-shipment learning, and the learning performed by the user before the operation of the image inspection device S is referred to as customer learning.

[0059] For example, the machine learning model of the image inspection device S may include a feature extraction unit that does not perform customer learning and a judgment unit that performs customer learning. Note that the feature extraction unit extracts feature quantities from the image. In addition, the judgment unit outputs an inspection result based on the feature quantities.

[0060] In other words, the machine learning model of the image inspection device S may include a fixed parameter portion. This fixed parameter portion is a layer with fixed parameters obtained through pre-shipment learning on the supplier's side. In other words, it is a layer that does not require user-side customer learning. Furthermore, the machine learning model of the image inspection device S may include a segmentation model that facilitates user-side customer learning.

[0061] With this structure, users do not need to prepare, for example, a GPU as a facility with the high processing power required for deep learning, nor do vendors need to provide an advanced learning environment using GPUs as a cloud service (SaaS, etc.). Therefore, the barriers to introducing the image inspection device S are reduced.

[0062] As described above, learning should be understood in a broad sense to refer not only to computationally intensive deep learning, but also to computationally inexpensive learning (that is, client learning in this specification). Note that since client learning is computationally inexpensive learning, it is possible to train a machine learning model using methods that do not include machine learning methods.

[0063] The inspection condition setting section 220 determines the output conditions of the image inspection apparatus S (in other words, the conditions of the sensor output) by combining a plurality of tools, for example.

[0064] The inspection execution unit 300 executes an inspection process of the workpiece W appearing in the plurality of frame images FR and outputs an inspection result. According to the drawing, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0065] The tool execution unit 310 executes the tool selected as the use target by the tool selection unit 211. According to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0066] When the tool selection unit 211 selects a rule-based tool as a usage target, the rule determination unit 311 executes the rule-based tool.

[0067] When the tool selection section 211 selects the tool of the learning system as the use target, the learning tool execution section 312 executes the tool of the learning system.

[0068] The inspection result output unit 320 outputs the inspection result according to the output conditions set by the inspection condition setting unit 220. Inspection may include image classification, abnormality detection, and region segmentation.

[0069] The imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be composed solely of hardware, or may be composed of a combination of hardware and software. Furthermore, the imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are implemented by a single piece of hardware or software. Note that the software may be executed by the control unit 2 (particularly the controller 21) having installed therein program files and setting files.

[0070] The image inspection device S of this structural example can be switched between a setting mode and a driving mode. In the setting mode, for example, various parameter settings such as shooting settings, registration of a main image, and generation (learning) of a discriminator for identifying an image are performed. The recognition of an image as used in this article includes recognition related to whether the image is an image of a non-defective product or an image of a defective product, and the image inspection device S can be configured to make a quality judgment related to whether the workpiece W is a non-defective product or a defective product based on the recognition result. In addition, in the setting mode, for example, preparatory work is performed to enable the user to separate non-defective products from defective products in the desired product inspection.

[0071] In the driving mode, the workpiece W is inspected based on the frame image FR captured at the actual site. In addition to the quality judgment described above, the inspection of the workpiece W also includes a quantity count for counting the number of workpieces W. Switching between the setup mode and the driving mode can be performed on the GUI to be described later. In addition, it can be configured to automatically transition to the driving mode upon completion of the setup mode. In the driving mode, correction or change of the recognition boundary using the discriminator (that is, so-called additional learning) can also be performed.

[0072] (via mode)

[0073] Incidentally, the image inspection device S includes a passing mode, one of various modes, for detecting workpieces W that sequentially pass through the field of view of the imaging unit 1. Specifically, the image inspection device S has a mode for performing time-series processing independently of the mode for performing a single judgment on a single frame image. In the passing mode, a passing counting tool can be provided and driven as a tool for detecting and counting a plurality of workpieces W that sequentially pass through the field of view of the imaging unit 1.

[0074] Figure 4 2 is a diagram illustrating a setting flow by a counting tool. Note that the execution subject of this flow can be basically understood as the inspection setting unit 200.

[0075] When the process starts, in step S11, a mode is selected from a plurality of modes (e.g., standard mode, sorting mode, and pass mode). In the present embodiment, since the pass counting tool can be set only in the pass mode, the pass counting tool can be selected instead of the pass mode. The following description is based on the assumption that the pass mode is selected in this step. In step S12, the overall moving direction Wd of the workpiece W is set. The overall moving direction Wd corresponds to the conveying direction of the conveying unit A (e.g., conveyor). In step S13, at least one learning image is selected from a plurality of frame images FR. In step S14, the target area is determined based on the learning image. Specifically, the workpiece W appearing in the learning image is designated as the target area (ROI [region of interest]) by setting a window. In step S15, the object detection model, which is one of the machine learning models, is updated so that the target area is detected from the learning image. In step S16, the pass line of the workpiece W is set. The pass line can be understood as a dividing line for dividing the detection range of the workpiece W into a pre-pass area and a post-pass area. In step S17 , the output of the inspection result is distributed to a plurality of output ports.

[0076] Note that since the counting tool according to this embodiment detects the target area using the object detection model obtained by machine learning, the process (particularly steps S12 to S14) includes the selection of the learning image (step S12), etc. However, the target area is not limited to detection using the object detection model and can be detected based on rules. In the case where the image inspection device S detects the target area based on rules, for example, instead of steps S12 to S14, the image inspection device S can be configured to receive input of the feature quantity (contour, color, or position, etc.) required for detecting the workpiece W and store the feature quantity.

[0077] Figure 5 is an example of Figure 4 When the image inspection program is executed by the PC 3, a GUI 400 (=various screens 400a to 400i) is displayed on the display device 4. The GUI 400 includes an image display area 410, an operation area 420, and a progress display area 430 as a basic layout.

[0078] Image display area 410 displays the frame image FR captured by imaging unit 1. Image display area 410 may also include a status display banner 411, zoom-in and zoom-out buttons 412, and a maximum display button 413. Status display banner 411 briefly displays the operating status (status) of the image inspection program. Zoom-in and zoom-out buttons 412 and maximum display button 413 are operated to zoom in or out and maximize the image displayed in image display area 410, respectively.

[0079] When the image inspection device S is first started, the initial startup screen 400a (first row in the left column) is displayed. Note that the image inspection program cannot be set when the image inspection device S is first started. Therefore, an alarm mark a1 (or an alarm message) indicating that the master image is not registered can be displayed in the image display area 410. In addition, a banner (e.g., "Master") indicating that the master image is not registered (or is being registered) can be displayed on the status display banner 411.

[0080] In addition, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, and the same applies below), the setting of the image inspection program is started. Figure 4 This corresponds to the start of the illustrated setup flow.

[0081] In the setting of the image inspection program, first, the mode selection screen 400b (the second row in the left column) is displayed. Figure 4 For example, as illustrated in the figure, a mode selection dialog box b0 may be displayed on the mode selection screen 400b.

[0082] In the mode selection dialog box b0 , for example, a standard mode button b1 , a sorting mode button b2 , a pass mode button b3 , a guide b4 , an OK button b5 , and a cancel button b6 are displayed.

[0083] When the standard mode button b1 is clicked, the standard mode is selected. In the standard mode, the workpieces W are identified as either non-defective products or defective products. When the sorting mode button b2 is clicked, the sorting mode is selected. In the sorting mode, the workpieces W are sorted (classified) into a plurality of categories. When the through mode button b3 is clicked, the through mode is selected. In the through mode, the number of workpieces W that sequentially pass through the field of view of the camera unit 1 is counted. In the guide b4, an overview and schematic diagram of the modes selected by clicking the standard mode button b1, the sorting mode button b2, and the through mode button b3 can be displayed.

[0084] Clicking the OK button b5 confirms the selected mode. Clicking the Cancel button b6 cancels the selected mode and closes the mode selection dialog box b0. The following description assumes that the pass-through mode has been selected on the mode selection screen 400b (mode selection dialog box b0).

[0085] When the through mode is selected on the mode selection screen 400b, the through mode setup process proceeds sequentially through the first step (shooting setup), the second step (master registration), the third step (tool setup), and the fourth step (output allocation). Each step is then displayed as a flowchart in the progress display area 430, with the currently executing step highlighted. This allows the user to easily understand the progress of the setup process.

[0086] First, in the first process (shooting setting), the moving direction setting screen 400c (the third row in the left column) is displayed. Figure 4 For example, a pull-down menu c1, a back button c2, and a forward button c3 may be displayed in the operation area 420 of the moving direction setting screen 400c.

[0087] In the drop-down menu c1, multiple candidates (for example, the four directions of "from left to right," "from right to left," "from top to bottom," and "from bottom to top") are displayed as the overall movement direction Wd of the workpiece W. In this figure, "from right to left" is selected as the overall movement direction Wd of the workpiece W. The overall movement direction Wd of the workpiece W can be displayed in the image display area 410. As described above, the image inspection device S (particularly the inspection setting unit 200) receives the setting of the overall movement direction Wd when setting the through mode. Using this structure, a matching result corresponding to the user's intention is obtained (details will be described later).

[0088] Note that in surveillance cameras, the primary surveillance target is people. Therefore, instead of assuming that all people move in the same direction, in order to estimate the direction in which each person moves or to track a person who exits the field of view and then returns, the surveillance target is distinguished into "person a" and "person b" rather than "person."

[0089] On the other hand, in image inspection apparatuses S used primarily in the field of FA, there is an assumption that all workpieces W to be inspected move in the same direction. Therefore, the overall movement direction Wd of the workpieces W is set in advance, thereby improving the matching accuracy of the workpieces W between frame images.

[0090] However, the image inspection device S is not necessarily limited to a configuration in which a user manually sets the overall moving direction Wd of the workpiece W. For example, the image inspection device S may be configured to estimate the moving direction of the workpiece W by reusing existing technology of a surveillance camera and automatically set the estimated result as the overall moving direction Wd of the workpiece W.

[0091] When the back button c2 is clicked, the screen returns to the above-mentioned mode selection screen 400b. On the other hand, when the forward button c3 is clicked, the process proceeds to the second process (main registration).

[0092] Note that, as illustrated in the figure, in the first step (shooting setting), a live image (=moving image being shot) can be displayed in the image display area 410. In this case, the status display banner 411 can display a banner indicating that a live image is being displayed (e.g., "Live").

[0093] In addition, although not explicitly shown in the figure, in the first step (shooting setting), the shooting field of view, image brightness, focus, shooting interval (frame rate), etc. can be set.

[0094] In the second step (master registration), the master image selection screen 400d (the fourth row in the left column) is displayed. Figure 4 In the operation area 420 of the main image selection screen 400d, for example, a live button d1, a history button d2, a file button d3, a back button d4, and a forward button d5 may be displayed.

[0095] When the Live button d1 is clicked, the live image currently displayed in the image display area 410 is registered as the main image. Note that the main image can be registered from the drive history image (checked image) by clicking the History button d2. Furthermore, the main image can be registered from the file image stored in the storage area by clicking the File button d3. As described above, examples of the setting method for registering the main image include a method for registering from a live image, a method for registering from a drive history image, and a method for registering from a file image.

[0096] When the back button d4 is clicked, the process returns to the first process (shooting setting) described above. On the other hand, when the forward button d5 is clicked, the process proceeds to the third process (tool setting).

[0097] In the third step (tool setting), the tool setting screen 400e (first row in the right column) is displayed. Figure 4 4. In the operation area 420 of the tool setting screen 400e, for example, a rectangular button e1, a circular button e2, a guide button e3, a learning start button e4, and a cancel button e5 may be displayed. In addition, a banner indicating that the tool is being set (e.g., "TOOL") may be displayed on the status display banner 411.

[0098] In the tool setting screen 400e, a workpiece W appearing in the registered main image is designated as a target region (so-called ROI) by setting a window e0. For example, clicking the rectangular button e1 displays the rectangular window e0 in the image display area 410. Clicking the circular button e2 displays a circular window (not shown) in the image display area 410. Note that the guide e3 may display methods for designating the target region, etc.

[0099] The user can specify the target area by adjusting the position, size and angle of the window e0 to surround the workpiece W appearing in the main image according to the guide e3. Label information (class) can be added to the specified target area. Multiple target areas can be specified. However, the user does not necessarily need to specify all target areas, and a portion of the target area can be automatically specified. The image inspection device S can have a function for automatically learning the state of the target area rotation.

[0100] When the learning start button e4 is clicked after designating the target area, learning of the machine learning model (object detection model) used in the pass mode is started. On the other hand, when the cancel button e5 is clicked, the designation of the target area is canceled.

[0101] When the learning of the machine learning model starts, the learning progress screen 400f (the second row in the right column) is displayed. Figure 4 This corresponds to step S15 in . The image inspection device S (particularly the inspection setup unit 200) inputs the learning image (primary image) to the machine learning model, extracting and storing a first feature quantity corresponding to the image characteristics of the workpiece W (e.g., a feature vector of the target area) from the learning image as learning for the machine learning model. In this embodiment, the first feature quantity corresponding to the image characteristics of the workpiece W is extracted and stored as learning for the machine learning model. However, the machine learning model can be configured to update parameters based on the image characteristics of the workpiece W to detect the workpiece W.

[0102] Note that, as illustrated in the figure, a learning progress dialog f0 may be displayed on the learning progress screen 400f. In the learning progress dialog f0, for example, a learning progress bar f1 indicating a degree of progress (0% to 100%) is displayed.

[0103] When the learning of the machine learning model is completed, the through line setting screen 400g (the third row in the right column) is displayed. Figure 4 In the operation area 420 of the through-line setting screen 400g, for example, a valid button g1, a invalid button g2, a guide g3, a back button g4, and a forward button g5 may be displayed.

[0104] The enable button g1 and disable button g2 are used to enable and disable the conveyor end mode (drop mode), respectively. For example, if it is desired to count the workpieces W dropped from the conveyor, the conveyor end mode can be enabled. Clicking the enable button g1 enables the conveyor end mode. On the other hand, clicking the disable button g2 disables the conveyor end mode. Note that the guide g3 can display an overview and schematic diagram of the conveyor end mode.

[0105] In the image display area 410, a pass line g0 is displayed, which divides the workpiece W into a front pass area X1 and a rear pass area X2. In other words, the detection range of the workpiece W is divided into the front pass area X1 and the rear pass area X2. When the conveyor terminal end mode is disabled, the pass line g0 is fixed in the center of the image display area 410. On the other hand, when the conveyor terminal end mode is enabled, the user's setting of the pass line g0 is accepted. For example, the pass line g0 can be set according to the conveyor terminal end according to the guide g3. With this structure, the front pass area X1 and the rear pass area X2 can be arbitrarily determined according to the user's intention.

[0106] When the conveyor end mode is disabled, an upward count is performed when a workpiece W is detected moving across the pass line g0—that is, when a workpiece W detected in the pre-pass area X1 in a certain frame image is detected in the post-pass area X2 in a subsequent frame image. On the other hand, when the conveyor end mode is enabled, an upward count is performed when a workpiece W detected in the pre-pass area X1 disappears from the post-pass area X2. In other words, in the conveyor end mode, the number of workpieces W that fall from the conveyor end (the workpieces W in the lower holder) is counted.

[0107] Therefore, when the conveyor terminal end mode is valid, even if the passing line g0 is not set at the terminal end of the conveyor and the workpiece W beyond the passing line g0 has not yet fallen from the terminal end of the conveyor, no deviation will occur between the count value of the workpiece W and the number of workpieces W actually entering the retaining frame.

[0108] When the back button g4 is clicked, the process returns to the second process (main registration) described above. On the other hand, when the forward button g5 is clicked, the process proceeds to the fourth process (output allocation).

[0109] In the fourth step (output allocation), the output allocation screen 400h (the fourth row in the right column) is displayed. Figure 4In the operation area 420 of the output allocation screen 400h, for example, a pull-down menu h1 for each of the output ports OUT1 to OUT8, a back button h2, and a finish button h3 may be displayed.

[0110] In the pull-down menu h1, multiple candidates are displayed as the output content of each of the output ports OUT1 to OUT8. In this figure, the counting result of workpiece W (COUNT UP) is selected as the output content of output port OUT1. In addition, the busy detection result (BUSY) is selected as the output content of output port OUT2. In addition, the error detection result (ERROR) is selected as the output content of output port OUT3. In addition, all output ports OUT4 to OUT8 are unused (OFF). With this structure, not only the counting result of workpiece W can be output in multiple bits, but also various information such as busy detection results and error detection results can be output in multiple bits.

[0111] When the back button h2 is clicked, the process returns to the third process (tool setting) described above. On the other hand, when the finish button h3 is clicked, the setting work of the pass mode is completed.

[0112] Figure 6 3 is a diagram illustrating an execution flow of a pass counting tool. Note that the execution subject of this flow can be basically understood as the inspection execution unit 300. In addition, the plurality of frame images FR sequentially generated by the imaging unit 1 include frame images FR1 and FR2.

[0113] At the start of this process, in step S21, a target area is detected from the frame image FR1 based on the image characteristics (first feature quantity) of the workpiece W. For example, the inspection execution unit 300 can input the frame image FR1 into a machine learning model (object detection model) to detect the target area from the frame image FR1, which has a second feature quantity that is closest to the first feature quantity. As described above, the structure of detecting the target area using the machine learning model improves robustness against changes in image characteristics.

[0114] In step S22, position information Dxy(FR1) of the target area detected in the frame image FR1 is acquired. If multiple target areas are detected in the frame image FR1, position information Dxy1(FR1), Dxy2(FR1), ..., and Dxy*(FR1) corresponding to each target area are acquired. For example, the inspection execution unit 300 can acquire the position information Dxy(FR1) of the target area by comparing the first feature quantity of the workpiece W with the second feature quantity of each area in the frame image FR1.

[0115] In step S23, a target area is detected from the frame image FR2 based on the image characteristics (first feature quantity) of the workpiece W. For example, the inspection execution unit 300 may input the frame image FR2 into a machine learning model (object detection model) to detect the target area from the frame image FR2 for which a second feature quantity closest to the first feature quantity is obtained. This is similar to step S21 described above.

[0116] In step S24, position information Dxy(FR2) of the target area detected in frame image FR2 is acquired. If multiple target areas are detected in frame image FR2, position information Dxy1(FR2), Dxy2(FR2), ..., and Dxy*(FR2) corresponding to the target areas are acquired. For example, the inspection unit 300 can acquire the position information Dxy(FR2) of the target area by comparing the first feature quantity of the workpiece W with the second feature quantity of the target area in frame image FR2. This is similar to step S22 described above.

[0117] In step S25 , a matching process of the target area is performed between the frame images FR1 and FR2 based on the position information Dxy(FR1) and Dxy(FR2) and the overall movement direction Wd of the workpiece W. The matching process, as used herein, is performed as a preliminary step to the process of detecting how the workpiece W is moving (so-called tracking process).

[0118] In step S26, the target areas are counted based on the result of the matching process. For example, the inspection execution unit 300 may count only the target areas that are determined to be located in the pre-passage area X1 in the frame image FR1 and in the post-passage area X2 in the frame image FR2 based on the result of the matching process.

[0119] Figure 7 This is a conceptual diagram related to the matching process performed between the frame image FR(t) and the frame image FR(t+1).

[0120] In the frame image FR(t), four workpieces W are detected as the target region (ROI) described above and are sequentially assigned ID1 to ID4 from left to right. Subsequently, in the frame image FR(t+1) captured one frame later, four workpieces W are also detected.

[0121] In conventional tracking processing, a method is generally employed for associating the same ID with the detection result having the closest relative position between frame image FR(t) and frame image FR(t+1). Using this method, as indicated by matching result M1, ID1 to ID4 are also assigned sequentially from left to right to the four workpieces W appearing in frame image FR(t+1).

[0122] However, at the site where the image inspection device S is installed, all workpieces W mounted on the conveyor move uniformly along the general movement direction Wd (from right to left in the figure). In other words, in the frame image FR(t+1), the following state occurs: the workpiece W located on the far left in the frame image FR(t) exits the image from the left end of the field of view, and a new workpiece W enters the image from the right end of the field of view.

[0123] Therefore, for the four workpieces W appearing in frame image FR(t+1), matching result M1 is incorrect, and as indicated by matching result M2, ID2 to ID5 should be assigned sequentially from left to right. However, without the overall movement direction Wd of the workpiece W, it is difficult to determine which of matching results M1 and M2 is correct.

[0124] On the other hand, if the user has previously set the overall movement direction Wd of the workpiece W, a correct matching result M2 is obtained based on the position information Dxy of the workpiece W and the overall movement direction Wd shown in each of the frame images FR(t) and FR(t+1). As a result, the position information Dxy of each of the multiple workpieces W is accurately acquired based on the matching result M2, thereby reducing duplicate counting and missed counting of the workpieces W.

[0125] Furthermore, at sites where image inspection devices S have been introduced, conveyors often stop during operation. In such cases, workpieces W may move in significantly different ways before and after the conveyor stops. Conventional tracking processing cannot handle such situations. On the other hand, by setting the overall movement direction Wd of the workpieces W as described above, it is possible to determine whether the conveyor has stopped by comparing the detection results of the target region (ROI) in each of the multiple frame images FR. Therefore, even when the conveyor stops, workpieces W can be accurately counted.

[0126] In addition, the inspection execution unit 300 can detect an abnormal state and output a warning by comparing the detection results of the target region (ROI) in each of the multiple frame images FR. Examples of abnormal states may include the sudden appearance (overdetection) or disappearance of the target region at a position other than the boundary of the shooting field of view (the upstream and downstream ends of the overall moving direction Wd). Note that the inspection execution unit 300 can output a warning without delay at a time point when an abnormal state is detected even in one frame. Alternatively, the inspection execution unit 300 can wait for the warning output unless an abnormal state is detected in multiple frames. In addition, when the conveyor terminal end mode is valid, the inspection execution unit 300 can allow the disappearance of the target region in the through-rear area X2. More specifically, when the conveyor terminal end mode is valid, the inspection execution unit 300 does not output a warning even if the workpiece W detected in the through-front area X1 disappears in the through-rear area X2, and counts up when the workpiece W disappears from the through-rear area X2.

[0127] In the conveyor terminal end mode according to this embodiment, the disappearance of the workpiece W from the post-passage area X2 is set as a condition for upward counting, but upward counting can be performed by the disappearance of the workpiece W from the pre-passage area X1. For example, even if the workpiece W falling from the conveyor terminal end is included in the imaging field of view, the workpiece W falling from the conveyor terminal end can be counted with high accuracy by arranging the passing line g0 at the conveyor terminal end.

[0128] (Additional Learning)

[0129] Figure 8 4 is a diagram illustrating the GUI transition for additional learning in drive mode (or during a test drive in setup mode, and the same applies hereinafter). When the image inspection device S is in drive mode, a drive screen 400i is displayed on the display device 4. In addition to the image display area 410 and operation area 420 described above, the drive screen 400i may also include a thumbnail display area 440.

[0130] For example, an image operation button i1 may be displayed in the image display area 410. The image operation button i1 is operated to cause the frame image (e.g., a drive history image including a checked frame image) displayed in the image display area 410 to be reproduced or paused, rewound or fast-forwarded, or rewound or forwarded frame by frame.

[0131] In the operation area 420, for example, an additional learning button i2 and a learning recommended image button i3 are displayed. When the additional learning button i2 is clicked, the screen changes to a tool setting screen (for example, see the aforementioned tool setting screen 400e or the tool setting screen 400k described below) for performing additional learning on the frame image currently displayed in the image display area 410. When the learning recommended image button i3 is clicked, the screen changes to an image selection screen 400j.

[0132] The recommended learning image j1 is displayed in the thumbnail display area 440 of the image selection screen 400j. As the recommended learning image j1, for example, a frame image FR in the interval in which an abnormal state is detected in the above-mentioned tracking process can be found from the plurality of frame images FR. In this recommended learning image j1, there is a high possibility that the background is detected as the target area or the workpiece W is not detected as the target area. Therefore, it can be considered that the recommended learning image j1 is suitable as a learning image for additionally learning the machine learning model (object detection model). Note that the recommended learning image j1 selected in the thumbnail display area 440 can be displayed in the image display area 410.

[0133] In the operation area 420 of the image selection screen 400j, for example, an OK button j2 and a Cancel button j3 are displayed. Clicking the OK button j2 causes the screen to transition to the tool setting screen 400k for performing additional learning on the recommended learning image j1 currently displayed in the image display area 410. This means that the examination setup unit 200 receives the recommended learning image j1 as a learning image for additional learning. On the other hand, clicking the Cancel button j3 cancels the selection of the recommended learning image j1.

[0134] The operation area 420 of the tool setting screen 400k displays, for example, an Add button k1, a Delete button k2, a tool selection box k3, a Learning Start button k4, and a Cancel button k5. Clicking the Add button k1 adds a window k0 to the image display area 410. The user can specify a target area by adjusting the position, size, and angle of the window k0 to surround the workpiece W. Clicking the Delete button k2 deletes the selected window k0. As described above, the inspection setting unit 200 can select a learning image for additional learning from the inspection result screen displaying the frame image FR and receive settings for the window k0.

[0135] The tool selection box k3 is used to select a tool for additional learning when there are multiple tools. In other words, multiple tools can be additionally learned from one frame image.

[0136] When the learning start button k4 is clicked after designating the target area, additional learning by the machine learning model (object detection model) used in the mode is started. On the other hand, when the cancel button k5 is clicked, the designation of the target area is canceled.

[0137] <Other>

[0138] Note that, in addition to this embodiment, various modifications may be made to the various technical features disclosed in this specification without departing from the spirit of the technical innovation. In other words, it should be considered that this embodiment is illustrative in all respects and not restrictive. In addition, the technical scope of the present invention is defined by the claims and should be understood to include all modifications that fall within the meaning and scope equivalent to the claims.

Claims

1. An image inspection device, comprising: The camera unit is used to continuously shoot the shooting field of view to generate a plurality of frame images arranged in a time series; an inspection execution unit configured to execute an inspection process of the objects appearing in the plurality of frame images to output an inspection result; as well as An inspection setting unit, configured to set up the inspection execution unit, wherein the inspection setting unit receives a setting of a window for a learning image in which the object appears, and The inspection execution unit detects a target area from the multiple frame images based on the learning image and the window set for the learning image, obtains first position information of the target area detected in the first frame image and second position information of the target area detected in the second frame image, performs matching processing of the target area between the first frame image and the second frame image based on the first position information, the second position information and the overall movement direction of the object, and counts the objects based on the result of the matching processing.

2. The image inspection device according to claim 1, in, The inspection setting section stores image characteristics of the object based on the setting of the window, and The inspection execution section detects the target area from the plurality of frame images based on image characteristics of the object.

3. The image inspection device according to claim 2, in, The inspection setting section extracts and stores a first feature amount corresponding to an image characteristic of the object from the learning image by inputting the learning image to a machine learning model, and The inspection execution unit detects a target area having a second feature value closest to the first feature value from the frame image by inputting the frame image to the machine learning model.

4. The image inspection device according to claim 1, wherein The inspection setting section receives a setting of the general moving direction.

5. The image inspection device according to claim 1, in, The detection range of the object is divided into a pre-passing area and a post-passing area, and The inspection execution unit counts only target areas determined to be located in the pre-passage area in the first frame image and determined to be located in the post-passage area in the second frame image based on a result of the matching process.

6. The image inspection device according to claim 5, wherein: The inspection setting section receives a setting of a pass line for dividing the pre-passage area and the post-passage area.

7. The image inspection device according to claim 5, in, The inspection setting section receives a setting for allowing disappearance of the target area in the post-passage area, and When the inspection setting section receives a setting for allowing disappearance of the target area, the inspection execution section counts the target area even if the target area disappears in the post-passage area.

8. The image inspection apparatus according to claim 1, wherein: The inspection execution section detects an abnormal state by comparing detection results of the target area in the plurality of frame images and outputs a warning.

9. The image inspection device according to claim 8, wherein: The inspection setting section receives, as learning recommended images, frame images in a section in which the abnormal state is detected among the plurality of frame images.

10. The image inspection apparatus according to claim 1, wherein: The inspection setting section selects the learning image for additional learning from an inspection result screen for displaying the frame image, and receives setting of the window.

Images