Image inspection device

Abstract

To set a flexible trigger condition.SOLUTION: An image inspection device includes: an image capturing section that continuously captures a capturing field of view to generate a plurality of frame images aligned in time series; an inspection execution section that inspects an object appearing in the frame images by a machine learning model to output a result; and an inspection setting section that performs setting of the inspection execution section. The machine learning model includes: a feature extraction section that extracts a feature amount from the frame image; and a determination section that outputs an inspection result from the feature amount. The inspection setting section receives selection of a first image, and determines a score calculation method on the basis of a first feature amount extracted from the first image such that a score based on the first feature amount satisfies a predetermined relative relationship with respect to a threshold. The inspection execution section extracts each feature amount from continuous first and second frame images and outputs an inspection trigger when the threshold is present between a first score based on a feature amount extracted from the first frame image and a second score based on a feature amount extracted from the second frame image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image inspection device. [Background technology]

[0002] Conventional image sensors generally perform object detection processing for one frame image (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-164146 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when performing image inspection on a moving object, it is necessary to acquire a frame image as the inspection object image, captured at the timing when the object is in a desired state (position, angle, etc.) within the imaging field of view. If the position or angle of the object shifts within the imaging field of view, the position or angle of the inspection object area can be adjusted using the "position correction tool" disclosed in Patent Document 1. However, adjustment is not possible unless the reference position of the "position correction tool" and the inspection object area are included in the captured frame image of the inspection object in the first place.

[0005] Therefore, when performing an image inspection of a moving object, the accuracy of the image inspection depends on whether or not an appropriate trigger timing is given to the image sensor, but it is time-consuming for the user to set up an external device to give the image sensor the appropriate trigger timing. For this reason, a configuration is known in which an inspection trigger is output when frame images sequentially acquired by the image sensor satisfy a predetermined trigger condition, and an inspection is performed on the captured frame images (=images to be inspected) at a timing specified based on the inspection trigger.

[0006] However, if the trigger conditions are too strict, there is a risk that the inspection of images that should be inspected will be missed. For example, if the trigger conditions are so strict that only good objects can be satisfied, a problem may occur in which the inspection trigger is not activated when the object is defective. Conversely, if the trigger conditions are too lenient, there is a risk that inspection will be performed on images that should not be inspected, resulting in unnecessary image inspection results. For example, if an inspection is performed as a triggered inspection that outputs an OK result as a pass / fail judgment for the object when the target image is a good image, an image that should not be inspected may be judged as a defective image because it does not contain an object, and a NG result may be output even though no defective object was detected.

[0007] As described above, with conventional image sensors, it was difficult to set trigger conditions so that an inspection trigger would be output at the timing desired by the user. Whether the configuration is to set a judgment area in part of the imaging field of view or to search for an object within the imaging field of view, as long as it is rule-based, the same problem can arise because it is difficult to set the speed of the trigger conditions.

[0008] In view of the above-mentioned problems, an object of the present invention is to provide an image inspection device that allows flexible setting of trigger conditions. [Means for solving the problem]

[0009] The image inspection device of the present invention comprises an imaging unit that continuously captures an imaging field of view to generate multiple frame images arranged in chronological order, an inspection execution unit that performs inspection processing of objects shown in the multiple frame images using a machine learning model and outputs inspection results, and an inspection setting unit that configures the inspection execution unit, wherein the machine learning model includes a feature extraction unit that extracts features from the frame images and a determination unit that outputs the inspection results from the features, and the inspection setting unit accepts selection of a first image and determines a score calculation method based on the first feature extracted from the selected first image so that the score based on the first feature satisfies a predetermined relative relationship with a threshold, and the inspection execution unit extracts the features from the first frame image and a second frame image subsequent to the first frame image as the multiple frame images, and outputs an inspection trigger when it is determined that the threshold is between the first score based on the feature extracted from the first frame image and the second score based on the feature extracted from the second frame image.

[0010] Still other features, elements, steps, advantages, and characteristics will become more apparent from the detailed description that follows and the accompanying drawings related thereto. [Effects of the Invention]

[0011] The image inspection device according to the present invention allows flexible setting of trigger conditions. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an image inspection device according to an embodiment of the present invention during operation. FIG. [Figure 2] FIG. 2 is a hardware configuration diagram of the image inspection device. [Figure 3] FIG. 2 is a functional block diagram of the image inspection device. [Figure 4] FIG. 10 is a diagram showing the setting flow of the AI ​​trigger tool. [Figure 5] FIG. 5 is a diagram showing a GUI (graphical user interface) transition in FIG. 4. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0014] FIG. 1 is a diagram illustrating an image inspection device S according to an embodiment of the present invention during operation. The image inspection device S captures an image of a workpiece W transported by a transport means A according to an imaging setting to acquire inference image data, detects the workpiece W in the image of the acquired inference image data, and outputs the detection result to an external device. An example of the external device is a programmable logic controller (PLC) 5, but a device other than the PLC 5 may also be the external device. The PLC 5 controls the transport means A based on the received detection result, for example, to separate the storage destination of the workpiece W. In the following description, a case will be described in which the external device is the PLC 5. Note that the workpiece W may be a workpiece that is not transported by the transport means A. In the following description, the workpiece is also referred to as the target object.

[0015] The image inspection device S includes an imaging unit 1 for imaging the workpiece W, a control unit 2 to which inference image data captured by the imaging unit 1 is input, a PC (personal computer) 3 for configuring the image inspection device S, and a display device 4 for displaying a setting screen, a selection screen, workpiece images, detection results, etc. The control unit 2 is capable of executing a trained model for detecting the workpiece W in the image of the input inference image data. The control unit 2 outputs to the PLC 5 according to the detection results obtained by the trained model.

[0016] Here, the image inspection device S may be used, for example, to inspect the workpiece W from various angles at various points in a manufacturing device or production line. For this reason, multiple image inspection devices S may be installed in one manufacturing device or one production line, potentially resulting in insufficient installation space and power supply. Therefore, the image inspection device S needs to be compact to accommodate the installation space and power-efficient to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a GPU (graphics processing unit). The control unit 2 can execute a trained model that has been trained to the extent that it can detect the workpiece W. However, the control unit 2 is provided to the user by the vendor so that the desired detection accuracy can be achieved without the user performing advanced machine learning, for which the use of a GPU is recommended. Since the user does not need to perform advanced machine learning, the user can execute a trained model capable of detecting the workpiece W without preparing a GPU for training. Furthermore, the time required by the user to prepare a trained model capable of detecting the workpiece W can be reduced. Note that a single image inspection device S may be installed and operated in a manufacturing device or production line. The image inspection device S may also be referred to as an image sensor.

[0017] (Configuration of imaging unit) The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to capture an image of the workpiece W from a desired direction. The workpieces W are sequentially transported by the transport means A into the imaging field of view of the imaging unit 1.

[0018] Fig. 2 is a hardware configuration diagram of the image inspection device S. As shown in Fig. 2, the imaging unit 1 includes an illumination module 10 for illuminating the workpiece W and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10.

[0019] The lighting module 10 has an LED (light emitting diode) 10a that irradiates light toward the workpiece W, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication section 20 (described later) of the control unit 2, and is controlled by a control section 21 (described later) of the control unit 2.

[0020] The camera module 11 has an AF motor 11a and an imaging board 11b. The AF motor 11a is a member for automatically focusing on the workpiece W by driving a focusing lens of an optical system (not shown). The autofocus method is not particularly limited, and examples include a contrast method.

[0021] The imaging board 11b is equipped with a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives light that is irradiated from the LED 10a onto the workpiece W and reflected by the workpiece W. This CMOS sensor 11c is connected to the head communication section 20 of the control unit 2, and is controlled by the control section 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0022] The FPGA 11d is a processing device whose internal processing contents can be changed. The DSP 11e is a signal processing device. A light-receiving amount signal of the light-receiving element of the CMOS sensor 11c is output to the FPGA 11d for processing, and is also output to the DSP 11e for processing. The processing by the FPGA 11d and the DSP 11e is not particularly limited, but examples thereof include various types of filter processing. The image data processed by the FPGA 11d and the DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0023] The imaging unit 1 and the control unit 2 are connected via a communication cable 6. Therefore, the control unit 2 can be installed in a location away from the location where the imaging unit 1 is installed.

[0024] (PC configuration) PC3 is configured as a general-purpose personal computer or the like. In this example, the personal computer can be used as PC3 by installing a predetermined program on it. PC3 includes operating devices such as a keyboard 3a and a mouse (not shown). A user of the image inspection device S can perform setting operations and selection operations on the image inspection device S by operating the operating devices on PC3. Specific setting operations and selection operations will be described later.

[0025] The PC 3 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other, and information based on setting operations by the user is sent from the PC 3 to the control unit 2. In addition, the PC 3 is able to receive image data of the workpiece W, inspection results, etc. output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Therefore, the PC 3 can be installed in a location away from where the control unit 2 is installed.

[0026] (Configuration of display device 4) The display device 4 is configured, for example, with a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component capable of detecting operations by the user's finger. The type of the touch panel 4a is not particularly limited, and examples include a capacitive type and an infrared type. The display device 4 and the communication board 22 of the control unit 2 are connected to be able to communicate with each other. Operation information of the touch panel 4a by the user is transmitted from the display device 4 to the control unit 2. In addition, the display device 4 is capable of receiving image data of the workpiece W output from the control unit 2. The display device 4 and the control unit 2 are connected via a communication cable 7. Therefore, the display device 4 can be installed in a location away from the installation location of the control unit 2.

[0027] The PC 3 and the display device 4 may be configured as an integrated unit. For example, the display device 4 may be configured as a display device that the PC 3 has. In this case, the main body of the PC 3 and the display device 4 may be integrated or may be separate. In this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0028] (Configuration of control unit 2) 2, the control unit 2 includes a head communication section 20, a control section 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage section) 26. The head communication section 20 is connected to the control section 21 and is a section that executes mutual communication between the control section 21 and the imaging unit 1. A control signal for the imaging unit 1 output from the control section 21 is transmitted to the imaging unit 1 via the head communication section 20. The control signal for the imaging unit 1 includes a signal that controls the light emission timing and light emission amount of the LED 10a, and a signal that controls the AF motor 11a and the imaging board 11b. Furthermore, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the control section 21 via the head communication section 20.

[0029] The control unit 21 has a DSP 21a and FPGA 21b that perform various signal processing, an accelerator 21c that speeds up the processing, and a memory 21d made up of RAM, ROM, etc. The specific configuration of the control unit 21 will be described later.

[0030] The communication board 22 is connected to the control unit 21, and is a member that executes mutual communication between the control unit 21 and the PC 3, the display device 4, and the PLC 5.

[0031] The connector board 24 includes a power supply interface 24a. A power cable (not shown) for supplying power from an external source is connected to the power supply interface 24a. The connector board 24 is connected to a power supply 23, and power supplied from the outside to the power supply interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the control unit 21. The power supplied to the control unit 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0032] The I / O board 25 is connected to the control unit 21. The inspection results output from the control unit 21 are input to the PLC 5 via the I / O board 25.

[0033] (function block) 3 is a functional block diagram of the image inspection device S. As shown in this diagram, the image inspection device S includes an imaging setting unit 100, an inspection setting unit 200, and an inspection execution unit 300 as its functional blocks.

[0034] The imaging setting section 100 performs various settings (such as imaging field of view, image brightness, focus, and imaging interval (frame rate)) related to the imaging operation of the imaging unit 1. The imaging unit 1 can be understood as an imaging section that continuously captures the imaging field of view and generates a plurality of frame images FR arranged in chronological order.

[0035] The inspection setting section 200 performs various settings related to the inspection of the frame image FR by the inspection execution section 300. Referring to this figure, the inspection setting section 200 includes a tool setting section 210 and an inspection condition setting section 220.

[0036] The tool setting unit 210 sets various tools. Referring to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0037] The tool selection unit 211 selects a tool to be set and used.

[0038] The parameter setting unit 212 sets a rule-based tool. With the rule-based tool, inspection is performed based on various feature amounts (outline, color, position, etc.) of the workpiece W shown in the image.

[0039] The learning tool setting unit 213 sets a learning tool that uses a machine learning model. In the learning tool, a trained model such as a classifier is generated in response to user instructions, and testing is performed based on the output of the trained model. Referring to the figure, the learning tool setting unit 213 includes a training data setting unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0040] The learning data setting unit 213a sets learning data to be input to the machine learning model. The learning data includes learning images and instruction content. The learning images include, for example, at least one of images of good products and images of defective products. The instruction content includes label information such as "this image is a good product," "this image is a defective product," or "this part is defective." The label information includes information corresponding to the class into which the workpiece W should be classified. Referring to this figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a label information setting unit 213a2, and a learning data generation unit 213a3.

[0041] The learning image selection unit 213a1 selects learning images. The learning image selection unit 213a1 may have a function of presenting recommended learning images when setting up a passage counting tool, for example. Details will be described later.

[0042] The label information setting unit 213a2 receives label information for displaying the training images selected by the training image selection unit 213a1 on a GUI and using them as training data. The label information setting unit 213a2 receives ON registration and OFF registration, for example, when setting an AI trigger tool.

[0043] The learning data generation unit 213a3 generates learning data based on the learning images selected by the learning image selection unit 213a1 and the window settings accepted by the label information setting unit 213a2. The learning data generation unit 213a3 stores image features of the workpiece W, for example, based on the window settings.

[0044] The update unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches an expected value in accordance with the teachings. The parameter update can be understood as learning of the machine learning model. However, the user does not necessarily have to perform all of the steps of learning the machine learning model. For example, it is possible to complete relatively computationally intensive learning on the vendor side before shipping the image inspection device S, and then have the user perform only relatively computationally intensive learning before operating the image inspection device S. In this specification, learning performed by the vendor side before shipping is referred to as pre-shipment learning, and learning performed by the user side before operating the image inspection device S is referred to as customer-side learning.

[0045] For example, the machine learning model of the image inspection device S may include a feature extraction unit that is not customer-trained and a judgment unit that is customer-trained. The feature extraction unit extracts features from the image. The judgment unit outputs an inspection result based on the features.

[0046] That is, the machine learning model of the image inspection device S may include a parameter-fixed portion. The parameter-fixed portion is a layer in which parameters obtained by pre-shipment training on the vendor side are fixed, in other words, a layer in which on-site training on the user side is not required. Furthermore, the machine learning model of the image inspection device S may include a segmentation model that facilitates on-site training on the user side.

[0047] With this configuration, there is no need for users to prepare equipment with the high processing power required for deep learning, such as a GPU, or for vendors to provide an advanced learning environment using a GPU or similar device as a cloud service (such as SaaS), thereby lowering the barrier to introducing the image inspection device S.

[0048] As such, the above-mentioned learning should be broadly interpreted as including not only deep learning, which requires a large amount of computation, but also learning with a small amount of computation, i.e., customer learning in this specification. Note that, since customer learning is learning with a small amount of computation, the machine learning model may be trained using a method that does not involve machine learning techniques.

[0049] The inspection condition setting unit 220 determines the output conditions of the image inspection device S, in other words, the conditions of the sensor output, for example, by combining a plurality of tools.

[0050] The inspection execution unit 300 executes an inspection process on the workpiece W shown in the multiple frame images FR and outputs the inspection results. Referring to this figure, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0051] The tool execution unit 310 executes the tool selected as the tool to be used by the tool selection unit 211. Referring to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0052] When the tool selection unit 211 selects a rule-based tool as a tool to be used, the rule determination unit 311 executes the rule-based tool.

[0053] The learning tool execution unit 312 executes a learning tool when the tool selection unit 211 selects the learning tool as the tool to be used.

[0054] The inspection result output unit 320 outputs the inspection results in accordance with the output conditions set by the inspection condition setting unit 220. The inspection may include image classification, anomaly detection, and segmentation.

[0055] The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be configured solely with hardware, or may be configured with a combination of hardware and software. The imaging setting unit 100, the inspection setting unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are realized by a single piece of hardware or software. The software described above can be executed by the control unit 2 (particularly the control unit 21) in which a program file and a setting file are installed.

[0056] The image inspection device S of this configuration example can be switched between a setting mode and an operation mode. In the setting mode, for example, various parameter settings such as imaging settings, registration of a master image, and generation (learning) of a classifier that classifies images are performed. Image classification here includes classification as to whether an image is a good or defective image, and the image inspection device S may be configured to determine whether the workpiece W is good or defective based on the classification results. Also, in the setting mode, for example, preliminary work is performed so that the user can distinguish between good and defective products in the desired product inspection.

[0057] In operation mode, the workpieces W are inspected based on the frame images FR captured at the actual site. Inspection of the workpieces W includes not only the pass / fail judgment described above, but also a piece count to count the number of workpieces W. Switching between setting mode and operation mode can be performed on the GUI, which will be described later. It is also possible to configure the system to automatically switch to operation mode immediately after completing setting mode. In operation mode, it is also possible to correct or change the classification boundary used by the classifier, i.e., to perform so-called additional learning.

[0058] (AI trigger tool) Incidentally, when performing image inspection on a moving workpiece W, it is necessary to acquire a frame image FR as the inspection target image, captured at the timing when the workpiece W reaches the desired state (position, angle, etc.) within the imaging field of view.

[0059] Photoelectric sensors, which are commonly used as trigger sensors that output triggers, have difficulty activating the trigger on thin workpieces. Furthermore, when trigger conditions are set to activate on workpieces with a specific outline, size, or color, it is also difficult to activate the trigger on multiple types of workpieces with different outlines, sizes, or colors. It is also difficult to activate the trigger on multiple workpieces lined up without any gaps. There is also a risk of the trigger being activated twice on circular workpieces. Some of the above issues can be resolved by using a color sensor, etc. However, this requires careful selection and skilled techniques.

[0060] In recent years, there have also been cases where image sensors are used to activate triggers. In this case, even for workpieces that are difficult to trigger using a photoelectric sensor, such as thin workpieces, the trigger can be activated if the trigger conditions can be defined using visual features in the captured image. However, since there are many workpieces whose visual features in the captured image are unstable, setting the trigger conditions is difficult. For example, in the case of a process that searches for the outline of a workpiece to activate a trigger, if the outline of the workpiece in the captured image becomes unstable due to halation, etc., it is difficult to stably activate the trigger. Furthermore, for example, if the outlines of OK workpieces and NG workpieces are different, it is difficult to activate the trigger. For this reason, the cases in which image sensors can be used as a trigger generation method have been extremely limited.

[0061] Meanwhile, the image inspection device S is equipped with an AI trigger tool as one of various tools. The AI ​​trigger tool is a tool that uses a machine learning model to determine whether each of multiple sequentially acquired frame images FR is an image for which the inspection trigger should be turned on.

[0062] The AI ​​trigger tool that generates an inspection trigger for the workpiece W shown in the frame image FR digitizes the frame image FR and compares it with a threshold value TH for determining whether the inspection trigger should be turned on (details will be described later). In contrast, the AI ​​tool that determines whether the workpiece W shown in the frame image FR is pass / fail based on the frame image FR digitizes the frame image FR and compares it with a threshold value for determining whether the workpiece W is pass / fail. In other words, when viewed internally within the image inspection device S, the AI ​​trigger tool can be understood as a type of AI tool.

[0063] By using AI to make trigger decisions, it is possible to achieve trigger decisions that are robust against halation and other factors.

[0064] Furthermore, the image inspection device S allows the user to flexibly set trigger conditions so that an inspection trigger is output at a timing desired by the user. This point will be described in detail below.

[0065] 4 is a diagram showing the setting flow of the AI ​​trigger tool. It can be understood that the execution body of this flow is basically the test setting unit 200.

[0066] When this flow starts, in step S11, one mode is selected from multiple modes (e.g., standard mode, sorting mode, passing mode). The following explanation assumes that standard mode is selected in this step. In step S12, an ON image F [Foreground] in which the work W is in the detection area is selected. On the other hand, in step S13, an OFF image B [Background] in which the work W is not in the detection area is selected. Note that at least one of the ON image F and the OFF image B may be cut out from a frame image FR during live shooting or replay playback. The order of steps S12 and S13 may also be reversed.

[0067] In step S14, the ON image F and the OFF image B are each input to a feature extraction unit of the machine learning model. The feature extraction unit acquires a feature amount Ff of the ON image F and a feature amount Fb of the OFF image B. Also in step S14, a threshold value TH relative to the score SC to be compared with the score SC of the frame image FR generated by the imaging unit 1 is determined.

[0068] The score SC is a numerical value that indicates whether the frame image FR is classified as an ON image F or an OFF image B. For example, the score SC increases as the frame image FR approaches the ON image F, and decreases as the frame image FR approaches the OFF image B. In other words, the score SC increases as the feature amount Fq acquired when the frame image FR is input to the feature extraction unit resembles the feature amount Ff of the ON image F, and decreases as the feature amount Fb of the OFF image B resembles it. The feature amounts Ff, Fb, and Fq may all be feature vectors of the detection area.

[0069] Thus, the score SC may be a value that increases as the frame image FR approaches the ON image F and decreases as the frame image FR approaches the OFF image B. Therefore, the threshold value TH determined in step S14 of this embodiment may be a threshold value TH relative to the score SC of the frame image FR. For example, a fixed value may be set as the threshold value TH, and the method of calculating the score SC based on the feature amount Ff may be changed to essentially change the threshold value TH so that the score SC based on the feature amount Ff is greater than the threshold value TH and the score SC based on the feature amount Fb is smaller than the threshold value TH. In other words, the method of calculating the score SC may be determined so that a relative relationship is satisfied in which the score SC based on the feature amount Ff is greater than a predetermined threshold value TH and the score SC based on the feature amount Fb is smaller than the threshold value TH. Alternatively, the value of the threshold value TH itself may be changed based on the feature amount.

[0070] In this embodiment, two images, an ON image F and an OFF image B, are selected in steps S13 and S14, but the images may be selected so that a relative threshold value TH can be determined. For example, a configuration may be adopted in which the threshold value TH defining the range of scores SC deemed similar to the ON image F is determined based on the feature value Ff extracted from the ON image F. Also, a configuration may be adopted in which the threshold value TH defining the range of scores SC deemed to be images in a state different from the OFF image B, i.e., an image containing a workpiece W, is determined based on the feature value Fb extracted from the OFF image B.

[0071] In step S15, additional selection (re-learning) of at least one of the ON image F and the OFF image B is accepted. In step S16, a delay is set from when the inspection trigger is output until the inspection is actually performed. However, neither step S15 nor S16 is an essential step for setting up the AI ​​trigger tool.

[0072] Fig. 5 is a diagram showing GUI transitions corresponding to the setting flow of Fig. 4. When the image inspection program is executed on the PC 3, a GUI 400 (= various screens 400a to 400g) is displayed on the display device 4. The GUI 400 includes, as its basic layout, an image display area 410, an operation area 420, and a progress display area 430.

[0073] The image display area 410 displays a frame image FR captured by the imaging unit 1, etc. The image display area 410 may also be accompanied by a status display banner 411, a zoom in / out button 412, a maximize button 413, etc. The status display banner 411 clearly displays the operating status (status) of the image inspection program. The zoom in / out button 412 and the maximize button 413 are operated to zoom in / out and maximize the image displayed in the image display area 410, respectively.

[0074] When the image inspection device S is started for the first time, an initial startup screen 400a (first row in the left column) is displayed. When the image inspection device S is started for the first time, the image inspection program has not yet been set. Therefore, a warning mark a1 (or a warning message) indicating that the master image has not been registered may be displayed in the image display area 410. Furthermore, a banner (e.g., "Master") indicating that the master image has not been registered (or is in the process of being registered) may be displayed in the status display banner 411.

[0075] Additionally, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, the same applies below), setting of the image inspection program is started. This corresponds to the start of the setting flow shown in FIG.

[0076] When setting up an image inspection program, a mode selection screen 400b (second row on the left) is first displayed. The mode selection screen 400b corresponds to step S11 in Fig. 4. For example, the mode selection screen 400b may display a mode selection dialogue b0, as shown in this figure.

[0077] The mode selection dialogue b0 displays, for example, a standard mode button b1, a sorting mode button b2, a passing mode button b3, guidance b4, an OK button b5, and a cancel button b6.

[0078] When the standard mode button b1 is clicked, the standard mode is selected. In the standard mode, the workpieces W are distinguished as good or bad. When the sorting mode button b2 is clicked, the sorting mode is selected. In the sorting mode, the workpieces W are sorted (classified) into multiple classes. When the passing mode button b3 is clicked, the passing mode is selected. In the passing mode, the number of workpieces W that pass sequentially through the imaging field of the imaging unit 1 is counted. The guidance b4 may display an overview and a schematic diagram of the mode selected by clicking the standard mode button b1, the sorting mode button b2, or the passing mode button b3.

[0079] Clicking the OK button b5 confirms the mode selection. On the other hand, clicking the Cancel button b6 cancels the mode selection and closes the mode selection dialog b0. The following explanation assumes that the standard mode has been selected on the mode selection screen 400b (mode selection dialog b0).

[0080] When the standard mode is selected on the mode selection screen 400b, the standard mode setting work proceeds in the following order: first step (image capture setting), second step (master registration), third step (tool setting), and fourth step (output allocation). From this point on, each step is displayed in a flow diagram in the progress display area 430, and the currently executing step is preferably highlighted. This configuration allows the user to grasp the progress of the setting work at a glance.

[0081] First, in the first step (imaging setting), on / off setting screens 400c1 and 400c2 (third and fourth rows in the left column) are displayed. The on / off setting screens 400c1 and 400c2 correspond to steps S12 and S13 in FIG. 4. The operation area 420 of the on / off setting screen 400c may display, for example, an on-image display area c1, an off-image display area c2, an on-image registration button c3, an off-image registration button c4, a start learning button c5, a back button c6, and a cancel button c7.

[0082] Also, as shown in the figure, in the first step (imaging setting), a live image (=video being captured) or a replay playback image (=recorded video) may be displayed in the image display area 410. In accordance with the figure, the status display banner 411 may display a banner (for example, "Replay") indicating that a replay playback image is being displayed.

[0083] At this time, image operation buttons c8 may be displayed in the image display area 410. The image operation buttons c8 are operated to play / pause the replay playback image, rewind / fast-forward, and rewind / fast-forward frame by frame.

[0084] In the process of selecting the ON image F and the OFF image B, a window w (=detection area) is first set for the replay playback image. The window w may be, for example, rectangular. The user can specify the detection area by adjusting the position, size, and angle of the window w so that it surrounds the workpiece W shown in the replay playback image.

[0085] Next, the image operation button c8 is used to search for images to be extracted from the replay image as the ON image F and the OFF image B. For example, the image display area 410 of the ON / OFF setting screen 400c1 displays an image captured when a workpiece W is present inside the window w (= the state in which the imaging trigger should be activated). When the ON image registration button c3 is clicked in this state, the image displayed in the image display area 410 is selected as the ON image F. The selected ON image F is displayed in the ON image display area c1. The workpiece W shown in the ON image F may be either a good or defective product. The OFF image display area c2 may display a message indicating that the OFF image B has not been selected, as shown in this figure.

[0086] Additionally, the image display area 410 of the ON / OFF setting screen 400c2 displays an image captured in a state where no workpiece W is present inside the window w (=a state where an inspection trigger should not be applied). When the OFF image registration button c4 is clicked in this state, the image being displayed in the image display area 410 is selected as the OFF image B. The selected OFF image B is displayed in the OFF image display area c2.

[0087] After the ON image F and the OFF image B are selected, clicking the Start Learning button c5 starts learning of the machine learning model used in the AI ​​trigger tool. Clicking the Back button c6 returns to the mode selection screen 400b. Clicking the Cancel button c7 cancels the selection of the ON image F and the OFF image B.

[0088] When learning of the machine learning model begins, a learning progress screen 400d (first row in the right column) is displayed. The learning progress screen 400d corresponds to step S14 in Figure 4. As part of learning of the machine learning model, the image inspection device S (particularly the inspection setting unit 200) inputs training images (ON image F and OFF image B) into the machine learning model to acquire the feature amount Ff of the ON image F and the feature amount Fb of the OFF image B, respectively.

[0089] In addition, a threshold value TH relative to the score SC to be compared with the score SC of the frame image FR generated by the imaging unit 1 is determined. The threshold value TH is determined so as to be able to determine whether or not the workpiece W is reflected inside the window w. Whether the workpiece W is good or bad does not matter. In other words, an inspection trigger can be activated even if the workpiece W is defective. In other words, it is possible to perform a series of inspection outputs, in which an inspection trigger is activated regardless of whether the workpiece W reflected in the frame image FR is good or bad, and then the workpiece is judged to be good or bad through image inspection. Hysteresis may be added to the threshold value TH in order to prevent chattering of the inspection trigger.

[0090] Note that the learning progress screen 400d may display a learning progress dialog d0 as shown in this drawing. The learning progress dialog d0 displays, for example, a learning progress bar d1 indicating the progress level (0% to 100%).

[0091] In this way, machine learning using live images or replay images of the workpiece W allows the user to set an appropriate trigger condition (threshold value TH) without having to abstract the image features of the ON image F and the OFF image B. This improves the flexibility of user settings.

[0092] The score graph e1 displays the relationship between the score SC of the image displayed in the image display area 410 (particularly the detection area partitioned by the window wd) and the threshold value TH as a graph. This visualization makes it possible to visually convey to the user the stability of the inspection trigger. For example, the user can check how the inspection trigger is activated by tracking the change in the score SC over time and comparing the image state before and after the inspection trigger is activated.

[0093] The score graph e1 displays the relationship between the score SC of the image displayed in the image display area 410 (particularly the detection area partitioned by the window wd) and the threshold value TH as a graph. This visualization makes it possible to visually convey to the user the stability of the inspection trigger. For example, the user can check how the inspection trigger is activated by tracking the change in the score SC over time and comparing the image state before and after the inspection trigger is activated.

[0094] The score graph e1 may also be referenced by the user as an auxiliary tool for selecting images suitable for additional learning. For example, in the additional image selection screen 400e shown in this figure, most of the work W is inside the window w, but the score SC does not reach the threshold TH. Therefore, the number of triggers e2 remains "0 times." If an inspection trigger should be applied to this image, it is desirable to perform additional learning on this image as an ON image F. Conversely, if an inspection trigger should not be applied to this image, this image may be performed as an OFF image B for additional learning.

[0095] Furthermore, although not explicitly shown in this figure, the score graph e1 may be provided with a mark indicating, for example, the peak (inflection point) of the score SC, which fluctuates up and down. In many cases, the score SC reaches its peak value when the workpiece W enters or leaves the window w. Therefore, a configuration that provides the above-mentioned marks makes it possible to efficiently search for images in which an inspection trigger output was omitted. In particular, it is preferable that when a mark provided to the score SC is clicked, the corresponding image is jumped to and displayed in the image display area 410. Furthermore, if there are multiple images recommended for learning, each may be displayed as a thumbnail as a candidate for additional learning.

[0096] When the additional learning button e3 is clicked, the screen transitions to a confirmation screen 400f for confirming whether or not to additionally learn the image currently displayed in the image display area 410. As shown in this figure, the confirmation screen 400f may display an additional learning dialog f0. The additional learning dialog f0 may display, for example, an additional image display area f1, an ON image registration button f2, an OFF image registration button f3, and a cancel button f4.

[0097] The additional image display area f1 displays the image that was displayed in the image display area 410 at the time the additional learning button e3 was clicked. When the ON image registration button f2 is clicked in this state, the image being displayed in the additional image display area f1 is additionally learned as the ON image F. As a result, an inspection trigger is activated when the workpiece W enters the window w to the same extent as the added image.

[0098] On the other hand, when the OFF image registration button f3 is clicked, the image displayed in the additional image display area f1 is additionally learned as the OFF image B. As a result, even if the workpiece W enters the window w to the same extent as the added image, the inspection trigger will no longer be activated.

[0099] The image used for the additional learning may be cut out from a frame image FR during live shooting or replay playback.

[0100] Furthermore, when a delay DL from when the score SC exceeds the threshold value TH until the inspection trigger is actually output is set, a delay setting screen 400g is displayed. That is, when a frame image FR a predetermined number of frames after the frame image FR in which the score SC exceeds the threshold value TH is set as the detection target image, the delay setting screen 400g is displayed. The operation area 420 of the delay setting screen 400g may display, for example, a score graph g1, a delay setting tool g2, an OK button g3, and a cancel button g4.

[0101] As described above, the score graph g1 displays, as a graph, the relationship between the score SC and the threshold value TH of the image displayed in the image display area 410. The score graph g1 may also clearly show the delay DL.

[0102] The delay setting tool g2 is operated when setting the delay DL. The delay setting tool g2 may include a box for accepting direct input of the delay DL, a + button for accepting an increment of the delay DL, and a − button for accepting a decrement of the delay DL.

[0103] When DL=0, an inspection trigger is output without delay when the score SC exceeds the threshold value TH, and image inspection (such as determining whether the product is good or bad) is performed. On the other hand, when DL>0, an inspection trigger is output several frames after the score SC exceeds the threshold value TH, and the frame image FR several frames later is treated as the image to be detected. In this way, by setting the delay DL, it is possible to perform image inspection on an image different from the image obtained when the score SC exceeds the threshold value TH. For example, setting the delay DL can be effective when a portion of the workpiece W is to be inspected.

[0104] Clicking the OK button g3 confirms the delayed download, while clicking the Cancel button g4 cancels the delayed download.

[0105] The above is an overview of the GUI transition in the first step (imaging setting). Although not explicitly shown in this figure, in the first step (imaging setting), the imaging field of view, image brightness, focus, and imaging interval (frame rate), etc. may be set.

[0106] After the completion of the first step (imaging setting), in the second step (master registration), a master image of the work W to be judged as good or bad is registered. Note that the master image may be registered from a live image, an operation history image, or a file image. In the third step (tool setting), various settings regarding rule-based or learning-based tools are performed. In the fourth step (output assignment), output contents (such as inspection results of good / bad products, busy results, and error results) are assigned to the output ports of the inspection result output unit 320.

[0107] (Inspection trigger) Finally, the output operation of the inspection trigger will be described. The image inspection device S in the operation mode (especially the inspection execution unit 300) acquires the respective feature amounts Fq1 and Fq2 from the frame images FR1 and FR2 sequentially generated by the imaging unit 1. And when the score SC of the frame image FR1 calculated based on the feature amount Fq1 is lower than the threshold TH, and the score SC of the frame image FR2 calculated based on the feature amount Fq2 is not less than the threshold TH, that is, when Fq1 < TH < Fq2, an inspection trigger is output.

[0108] Note that the image inspection device S (especially the inspection execution unit 300) may execute inspection processing on the frame image FR2 when the score SC is not less than the threshold TH. With such a configuration, for example, even when the work W is out of the imaging field of view in the frame image FR3 generated immediately after the frame image FR2, image inspection can be performed without trouble.

[0109] <Others> In addition to the above-described embodiments, the various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation. In other words, the above-described embodiments should be considered to be illustrative and not restrictive in all respects. Furthermore, the technical scope of the present invention is defined by the claims, and should be understood to include all modifications that fall within the meaning and scope of the claims. [Explanation of symbols]

[0110] 1 Imaging unit 2. Control Unit 3 PC (Personal Computer) 3a keyboard 4 Display device 4a Touch panel 5 PLC (Programmable Logic Controller) 6 Communication Cable 7. Communication Cable 10 Lighting Module 10a LED (light emitting diode) 10b LED Driver 11 Camera module 11a AF motor 11b Imaging board 11c CMOS sensor 11d FPGA 11e DSP 20 Head communication unit 21 Control section 21a DSP 21b FPGA 21c Accelerator 21d Memory 22 Communication board 23 Power supply 24 Connector board 24a power interface 25 I / O board 26 Storage device (storage unit) 100 Imaging setting section 200 Inspection setting section 210 Tool setting section 211 Tool selection section 212 Parameter setting section 213 Learning Tool Settings 213a Learning data setting unit 213a1 Learning image selection unit 213a2 Label information setting section 213a3 Learning data generation unit 213b Update section 220 Inspection condition setting section 300 Inspection Execution Department 310 Tool Execution Unit 311 Rule Judgment Unit 312 Learning Tool Execution Department 320 Inspection result output unit 400 GUI 400a initial startup screen 400b Mode selection screen 400c1, 400c2 On / Off setting screen 400d Learning progress screen 400e additional image selection screen 400f confirmation screen 400g delay setting screen 410 Image display area 411 Status Display Banner 412 Zoom In / Out Button 413 Maximum display button 420 Operation area 430 Progress display area a1 Caution mark a2 Setting start button b0 Mode selection dialog b1 Standard mode button b2 Sorting mode button b3 Passing mode button b4 Guidance b5 OK button b6 Cancel button c1 On-image display area c2 Off-image display area c3 On image registration button c4 Off image registration button c5 Start learning button c6 Back button c7 Cancel button c8 Image operation buttons d0 Learning progress dialog d1 Learning progress bar e0 window e1 score graph e2 Trigger count e3 Additional learning button f0 Additional learning dialog f1 Additional image display area f2 On image registration button f3 Off image registration button f4 Cancel button g1 score graph g2 delay setting tool g3 OK button g4 Cancel button w Window (detection area) A. Means of transport (conveyor) B Off image (background image) F On Image FR Frame Image S Image inspection device SC Score W work (object) Wd Overall movement direction

Claims

1. an imaging unit that continuously captures an imaging field of view to generate a plurality of frame images arranged in time series; an inspection execution unit that executes an inspection process for an object shown in the plurality of frame images using a machine learning model and outputs an inspection result; an inspection setting unit that sets the inspection execution unit; Equipped with the machine learning model includes a feature extraction unit that extracts feature amounts from the frame images, and a determination unit that outputs the inspection result from the feature amounts; The test setting unit Accepting a selection of a first image; determining a score calculation method based on a first feature amount extracted from the selected first image such that the score based on the first feature amount satisfies a predetermined relative relationship with a threshold value; The inspection execution unit extracting the feature amounts from a first frame image and a second frame image consecutive to the first frame image as the plurality of frame images; An image inspection device that outputs an inspection trigger when it is determined that the threshold value is between a first score based on a feature extracted from the first frame image and a second score based on a feature extracted from the second frame image.

2. The test setting unit Accepting a selection of an image in which the object is in a detection area as the first image and a second image in which the object is not in the detection area; 2. The image inspection device according to claim 1, wherein the relative threshold value to be compared with the score indicating whether the frame image is classified as the first image or the second image is determined based on the first feature amount and a second feature amount extracted from the second image.

3. the test setting unit determines a relative threshold for the score so that the score of the frame image similar to the first image is higher than the score of the frame image similar to the second image; 3. The image inspection device of claim 2, wherein the inspection execution unit acquires a third feature and a fourth feature from each of a first frame image and a second frame image generated as the plurality of frame images, and outputs an inspection trigger when a first score of the first frame image calculated based on the third feature extracted from the first frame image is lower than the threshold value and a second score of the second frame image calculated based on the fourth feature extracted from the second frame image subsequent to the first frame image is equal to or greater than the threshold value.

4. The image inspection device according to claim 1 , wherein the inspection execution unit executes the inspection process on the second frame image.

5. The test setting unit Display images during live shooting or replay, The image inspection device according to claim 1 , wherein the image to be displayed is selected as at least one of the first image and the second image.

6. The image inspection device according to claim 1 , further comprising a GUI that displays the scores as a graph.

7. an imaging unit that continuously captures an imaging field of view to generate a plurality of frame images arranged in time series; an inspection execution unit that executes an inspection process for an object shown in the plurality of frame images using a machine learning model and outputs an inspection result; an inspection setting unit that sets the inspection execution unit; Equipped with the machine learning model includes a feature extraction unit that extracts feature amounts from the frame images, and a determination unit that outputs the inspection result from the feature amounts; The test setting unit Accepting a selection of a first image; determining a threshold value for the score based on a first feature amount extracted from the selected first image; The inspection execution unit extracting the feature amounts from a first frame image and a second frame image consecutive to the first frame image as the plurality of frame images; An image inspection device that outputs an inspection trigger when it is determined that the threshold value is between a first score based on a feature extracted from the first frame image and a second score based on a feature extracted from the second frame image.

Images