Image inspection device

Abstract

The invention relates to an image inspection apparatus. The image inspection apparatus includes: an imaging section configured to generate a plurality of frame images arranged in a time sequence; an inspection execution section configured to execute inspection processing of an object appearing in the plurality of frame images using a machine learning model to output an inspection result; and an inspection setting unit for setting the inspection execution unit. The machine learning model includes a feature extraction unit and a determination unit that outputs an inspection result on the basis of the feature amount. The inspection setting section receives a selection of the first image and determines a threshold value. When it is determined that the threshold value exists between a first score based on the feature amount extracted from the first frame image and a second score based on the feature amount extracted from the second frame image, the inspection execution unit outputs an inspection trigger.

Description

Technical Field

[0001] The present invention relates to an image inspection device. Background Art

[0002] An image sensor in the related art generally performs detection processing of an object on one frame image (for example, see Japanese Patent Laid-Open No. 2022-164146).

[0003] Incidentally, in the case of performing image inspection on a moving object, it is necessary to acquire a frame image captured at a timing when the object is in a desired state (position or angle, etc.) in the photographing field of view as an inspection target image.

[0004] If the position or angle of the object in the photographic field of view deviates, the position or angle of the inspection target area can be adjusted using the "position correction tool" disclosed in Japanese Patent Application Laid-Open No. 2022-164146. However, adjustment cannot be performed unless the reference position of the "position correction tool" and the inspection target area are initially included in the photographic frame image of the inspection target.

[0005] Therefore, when performing image inspections on moving objects, the accuracy of the inspection depends on whether the image sensor is properly triggered. However, setting up an external device to properly trigger the image sensor requires time and effort from the user. Therefore, a known system outputs an inspection trigger when frame images sequentially captured by the image sensor meet predetermined trigger conditions, and then performs an inspection on the captured frame images (the inspection target image) at the timing specified by the inspection trigger.

[0006] However, if the trigger conditions are strict, the inspection of the image to be inspected may be missed. For example, if the trigger conditions are strict enough to only meet the requirements for non-defective products as the target, there may be a problem in that the inspection trigger is not applied when the target is a defective product. Conversely, if the trigger conditions are loose, an inspection is performed on an image that should not be the target of inspection, and unnecessary image inspection results may be obtained. For example, in the case of an inspection that outputs an OK judgment as a quality judgment for the target when the target image is an image of a non-defective product, as an inspection performed by triggering, there may be a problem in that an image that should not be the target of inspection is an image without an object and is therefore judged to be an image of a defective product, and an NG judgment is output even if the target of a defective product is not detected.

[0007] As described above, conventional image sensors have difficulty setting trigger conditions so that the inspection trigger is output at the desired timing. Note that even with a configuration that sets a judgment area within a portion of the field of view or searches for an object within the field of view, as long as the settings are rule-based, it is difficult to set loose or strict trigger conditions, and similar problems may occur. Summary of the Invention

[0008] In view of the above problems, an object of the present invention is to provide an image inspection device capable of setting flexible trigger conditions.

[0009] For example, an image inspection apparatus according to the present invention includes: an imaging unit for continuously capturing a field of view to generate a plurality of frame images arranged in a time series; an inspection execution unit for executing an inspection process for an object appearing in the plurality of frame images using a machine learning model to output an inspection result; and an inspection setting unit for configuring the inspection execution unit. The machine learning model includes: a feature extraction unit for extracting a feature quantity from the frame images; and a determination unit for outputting the inspection result based on the feature quantity. The inspection setting unit receives a selection of a first image and determines a score calculation method based on a first feature quantity extracted from the selected first image so that a score based on the first feature quantity satisfies a predetermined relative relationship with respect to a threshold value. The inspection execution unit extracts the feature quantity from a first frame image and a second frame image that is continuous with the first frame image, and outputs an inspection trigger when determining that the threshold value is between a first score based on the feature quantity extracted from the first frame image and a second score based on the feature quantity extracted from the second frame image.

[0010] Note that other characteristics, elements, steps, advantages, and features will become more apparent from the following detailed description and accompanying drawings.

[0011] The image inspection device according to the present invention can set flexible trigger conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram for explaining the operation of the image inspection apparatus according to the embodiment of the present invention;

[0013] Figure 2 It is a hardware structure diagram of the image inspection device;

[0014] Figure 3 It is a functional block diagram of an image inspection device;

[0015] Figure 4 is a diagram illustrating the setup flow of the AI ​​trigger tool; and

[0016] Figure 5 This is an example Figure 4 Figure 1. Graphical user interface [GUI] transition diagram. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following description of the preferred embodiment is merely exemplary in nature and is not intended to limit the invention, its application, or its intended uses.

[0018] Figure 1 : is a diagram for explaining the image inspection device S according to an embodiment of the present invention when in operation. For example, the image inspection device S shoots the workpiece W conveyed by the conveying unit A according to the shooting settings to obtain inference image data, detects the workpiece W in the image of the acquired inference image data, and outputs the detection result to an external device. Examples of external devices include a programmable logic controller (PLC) 5, etc., but a device other than the PLC 5 may be an external device. Based on the received detection result, the PLC 5 controls the conveying unit A, for example, to separate the storage destination of the workpiece W. In the following description, the case where the external device is the PLC 5 will be described. Note that the workpiece W may be a workpiece that is not conveyed by the conveying unit A. In addition, in the following description, the workpiece is also referred to as an object.

[0019] The image inspection device S includes an imaging unit 1 for capturing images of a workpiece W; a control unit 2 to which the inference image data captured by the imaging unit 1 is input; a personal computer (PC) 3 for configuring the image inspection device S; and a display device 4 for displaying settings, selection screens, workpiece images, and inspection results. The control unit 2 executes a trained model for detecting the workpiece W in the input inference image data. The control unit 2 outputs the inspection results corresponding to the trained model to a programmable logic controller 5.

[0020] Here, for example, the image inspection device S can be used when inspecting a workpiece W from various angles at various stages of a manufacturing facility or production line. Consequently, multiple image inspection devices S may be installed in a single manufacturing facility or production line, and it is conceivable that sufficient installation space and power supply may not be secured. Therefore, the image inspection device S needs to be compact enough to accommodate the installation space and energy-efficient enough to accommodate the power supply. To meet these requirements, the image inspection device S according to this embodiment does not include a graphics processing unit (GPU). The control unit 2 can execute a trained model (in which learning is performed to the extent that the workpiece W can be detected), but the vendor provides the user with an image inspection device that allows the user to achieve the desired detection accuracy even without performing the advanced machine learning recommended using a GPU. Since the user does not need to perform advanced machine learning, the user can execute a trained model capable of detecting the workpiece W without having to prepare a GPU suitable for learning. Furthermore, the time it takes for the user to prepare a trained model capable of detecting the workpiece W can be shortened. Note that a single image inspection device S can be installed and operated in a manufacturing facility or production line. Furthermore, the image inspection device S can also be referred to as an image sensor.

[0021] (Structure of the camera unit)

[0022] The imaging unit 1 is separate from the control unit 2 and is installed so as to be able to image the workpiece W from a desired direction. The workpiece W is sequentially transported by the transport unit A to the imaging field of the imaging unit 1 .

[0023] Figure 2 This is the hardware structure diagram of the image inspection device S. Figure 2 As illustrated, the imaging unit 1 includes an illumination module 10 for illuminating a workpiece W, and a camera module 11 for capturing an image of the workpiece W illuminated by the illumination module 10 .

[0024] The lighting module 10 includes a light-emitting diode (LED) 10a that illuminates the workpiece W with light, and an LED driver 10b that controls the light intensity and light emission timing of the LED 10a. The LED driver 10b is connected to a head communication unit 20 (described later) of the control unit 2 and is controlled by a controller 21 (described later) of the control unit 2.

[0025] The camera module 11 includes an AF motor 11a and a pickup board 11b. The AF motor 11a drives a focus lens of an optical system (not illustrated) to automatically focus on the workpiece W. The automatic focusing method is not particularly limited, and examples thereof include a contrast method and the like.

[0026] The imaging board 11b includes a CMOS sensor 11c, an FPGA 11d, and a DSP 11e. The CMOS sensor 11c is an image sensor that receives reflected light emitted from the LED 10a toward the workpiece W and reflected by the workpiece W. The CMOS sensor 11c is connected to the head communication unit 20 of the control unit 2 and is controlled by the controller 21 of the control unit 2 to perform exposure processing at a predetermined timing for a predetermined time.

[0027] The FPGA 11d is a processing device capable of changing the content of internal processing. The DSP 11e is a signal processing device. The light-receiving amount signal from the light-receiving element included in the CMOS sensor 11c is output to and processed by the FPGA 11d, and is also output to and processed by the DSP 11e. The processing performed by the FPGA 11d and DSP 11e is not particularly limited, and examples include various filtering processes. Image data processed by the FPGA 11d and DSP 11e is transmitted from the imaging unit 1 to the control unit 2.

[0028] The camera unit 1 and the control unit 2 are connected via a communication cable 6. Thus, the control unit 2 can be installed at a location far away from the installation location of the camera unit 1.

[0029] (PC structure)

[0030] The PC 3 is composed of a general-purpose personal computer or the like. In this example, the PC 3 can be used by installing a predetermined program in the personal computer. The PC 3 includes operating devices such as a keyboard 3a and a mouse (not shown). The user of the image inspection device S can perform setting operations and selection operations for the image inspection device S by operating the operating devices of the PC 3. Specific setting operations and selection operations will be described later.

[0031] The PC 3 is connected to the communication board 22 of the control unit 2 so as to communicate with each other, and information based on setting operations performed by the user is transmitted from the PC 3 to the control unit 2. In addition, the PC 3 can receive image data and inspection results of the workpiece W output from the control unit 2. The PC 3 and the control unit 2 are connected via a communication cable 7. Thus, the PC 3 can be installed at a location far from the installation location of the control unit 2.

[0032] (Structure of Display Device 4)

[0033] The display device 4 includes, for example, a liquid crystal display or an organic EL display. In this example, the display device 4 includes a touch panel 4a. The touch panel 4a is a component that can detect operations using the user's fingers. The type of touch panel 4a is not particularly limited, and examples thereof include a capacitive type and an infrared type. The display device 4 is connected to the communication board 22 of the control unit 2 in a manner that allows them to communicate with each other. User operation information on the touch panel 4a is sent from the display device 4 to the control unit 2. In addition, the display device 4 can receive image data of the workpiece W output from the control unit 2, etc. The display device 4 and the control unit 2 are connected via a communication cable 7. Thus, the display device 4 can be installed in a place away from the installation place of the control unit 2.

[0034] Note that the PC 3 and the display device 4 may be integrally provided. For example, the display device 4 may be formed by a display device included in the PC 3. In this case, the main body of the PC 3 and the display device 4 may be integrally provided, or may be separate from each other. Furthermore, in this example, the communication board 22 and the PLC 5 are connected via a communication cable 7.

[0035] (Structure of Control Unit 2)

[0036] like Figure 2 As shown, the control unit 2 includes a head communication unit 20, a controller 21, a communication board 22, a power supply 23, a connector board 24, an I / O board 25, and a storage device (storage unit) 26. The head communication unit 20 is connected to the controller 21 and performs communication between the controller 21 and the imaging unit 1. Control signals for the imaging unit 1 output from the controller 21 are transmitted to the imaging unit 1 via the head communication unit 20. The control signals for the imaging unit 1 include signals for controlling the timing and amount of light emission of the LED 10a, as well as signals for controlling the AF motor 11a and the capturing board 11b. In addition, image data acquired by the imaging unit 1 is output from the imaging unit 1 and then transmitted to the controller 21 via the head communication unit 20.

[0037] The controller 21 includes a DSP 21a and an FPGA 21b that perform various signal processing, an accelerator 21c for accelerating processing, and a memory 21d including a RAM and a ROM, etc. The specific structure of the controller 21 will be described later.

[0038] The communication board 22 is a member that is connected to the controller 21 and performs communication among the controller 21 , the PC 3 , the display device 4 , and the PLC 5 .

[0039] The connector plate 24 includes a power interface 24a. A power cable (not shown) for supplying power from the outside is connected to the power interface 24a. The connector plate 24 is connected to the power supply 23, and the power supplied from the outside to the power interface 24a is adjusted to a predetermined voltage by the power supply 23 and then supplied to the controller 21. The power supplied to the controller 21 is supplied to the imaging unit 1 via the head communication unit 20.

[0040] The I / O board 25 is connected to the controller 21. The inspection result output from the controller 21 is input to the PLC 5 via the I / O board 25.

[0041] (Function Block)

[0042] Figure 3 2 is a functional block diagram of an image inspection apparatus S. As illustrated in the figure, the image inspection apparatus S includes a photographing setting unit 100 , an inspection setting unit 200 , and an inspection execution unit 300 as its functional blocks.

[0043] The imaging setting section 100 makes various settings related to the imaging operation of the imaging unit 1 (such as the imaging field of view, image brightness, focus, and imaging interval (frame rate)). Note that the imaging unit 1 can be understood as an imaging section that continuously captures the imaging field of view to generate a plurality of frame images FR arranged in a time series.

[0044] The inspection setting section 200 performs various settings related to the inspection of the frame image FR using the inspection execution section 300. According to the drawing, the inspection setting section 200 includes a tool setting section 210 and an inspection condition setting section 220.

[0045] The tool setting unit 210 sets various tools. According to the figure, the tool setting unit 210 includes a tool selection unit 211, a parameter setting unit 212, and a learning tool setting unit 213.

[0046] The tool selection unit 211 selects a tool to be set and a tool to be used.

[0047] The parameter setting section 212 sets a rule-based tool. In the rule-based tool, inspection is performed based on various feature quantities (contour, color, position, etc.) of the workpiece W appearing in the image.

[0048] The learning tool setup unit 213 sets up tools for a learning system that uses machine learning models. In the learning system tools, trained models such as discriminators are generated based on user instruction, and checks are performed based on the output of the trained models. As shown in the figure, the learning tool setup unit 213 includes a learning data setup unit 213a and an update unit 213b. The machine learning model may include a neural network.

[0049] The learning data setting unit 213a sets the learning data to be input into the machine learning model. The learning data includes learning images and teaching content. For example, the learning images include at least one of an image of a non-defective product and an image of a defective product. The teaching content includes label information such as "This image is a non-defective product," "This image is a defective product," and "This part is defective." The label information includes information corresponding to the class into which the workpiece W is classified. As shown in the figure, the learning data setting unit 213a includes a learning image selection unit 213a1, a label information setting unit 213a2, and a learning data generation unit 213a3.

[0050] The learning image selection unit 213a1 selects a learning image. For example, the learning image selection unit 213a1 may have a function of presenting a learning-recommended image when setting the pass count tool. Details will be described later.

[0051] The label information setting unit 213a2 receives label information for displaying the learning image selected by the learning image selection unit 213a1 on the GUI and using the learning image as learning data. For example, when setting the AI ​​trigger tool, the label information setting unit 213a2 receives ON registration and OFF registration.

[0052] The learning data generating unit 213a3 generates learning data based on the learning image selected by the learning image selecting unit 213a1 and the window setting received by the label information setting unit 213a2. For example, the learning data generating unit 213a3 stores the image characteristics of the workpiece W based on the window setting.

[0053] The updating unit 213b updates the parameters of the machine learning model so that the output of the machine learning model approaches the expected value corresponding to the teaching content. The updating of the parameters can be understood as the learning of the machine learning model. However, the learning of the machine learning model does not necessarily need to be performed by the user in all processes. For example, learning with a relatively large amount of calculation can be completed on the supplier side before the image inspection device S is shipped, and only learning with a relatively small amount of calculation can be performed on the user side before the operation of the image inspection device S. In this specification, the learning performed by the supplier side before shipping is referred to as pre-shipment learning, and the learning performed by the user before the operation of the image inspection device S is referred to as customer learning.

[0054] For example, the machine learning model of the image inspection device S may include a feature extraction unit that does not perform customer learning and a judgment unit that performs customer learning. Note that the feature extraction unit extracts feature quantities from the image. In addition, the judgment unit outputs an inspection result based on the feature quantities.

[0055] In other words, the machine learning model of the image inspection device S may include a fixed parameter portion. This fixed parameter portion is a layer with fixed parameters obtained through pre-shipment learning on the supplier's side. In other words, it is a layer that does not require user-side customer learning. Furthermore, the machine learning model of the image inspection device S may include a segmentation model that facilitates user-side customer learning.

[0056] With this structure, users do not need to prepare, for example, a GPU as a facility with the high processing power required for deep learning, nor do vendors need to provide an advanced learning environment using GPUs as a cloud service (SaaS, etc.). Therefore, the barriers to introducing the image inspection device S are reduced.

[0057] As described above, learning should be understood in a broad sense to refer not only to computationally intensive deep learning, but also to computationally inexpensive learning (that is, client learning in this specification). Note that since client learning is computationally inexpensive learning, it is possible to train a machine learning model using methods that do not include machine learning methods.

[0058] The inspection condition setting section 220 determines the output conditions of the image inspection apparatus S (in other words, the conditions of the sensor output) by combining a plurality of tools, for example.

[0059] The inspection execution unit 300 executes an inspection process of the workpiece W appearing in the plurality of frame images FR and outputs an inspection result. According to the drawing, the inspection execution unit 300 includes a tool execution unit 310 and an inspection result output unit 320.

[0060] The tool execution unit 310 executes the tool selected as the use target by the tool selection unit 211. According to the figure, the tool execution unit 310 includes a rule determination unit 311 and a learning tool execution unit 312.

[0061] When the tool selection unit 211 selects a rule-based tool as a usage target, the rule determination unit 311 executes the rule-based tool.

[0062] When the tool selection section 211 selects the tool of the learning system as the use target, the learning tool execution section 312 executes the tool of the learning system.

[0063] The inspection result output unit 320 outputs the inspection result according to the output conditions set by the inspection condition setting unit 220. Inspection may include image classification, abnormality detection, and region segmentation.

[0064] The imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be composed solely of hardware, or may be composed of a combination of hardware and software. Furthermore, the imaging setup unit 100, the inspection setup unit 200, and the inspection execution unit 300 may each be independent, or may be configured so that multiple functions are implemented by a single piece of hardware or software. Note that the software may be executed by the control unit 2 (particularly the controller 21) having installed therein program files and setting files.

[0065] The image inspection device S of this structural example can be switched between a setting mode and a driving mode. In the setting mode, for example, various parameter settings such as shooting settings, registration of a main image, and generation (learning) of a discriminator for identifying an image are performed. The recognition of an image as used in this article includes recognition related to whether the image is an image of a non-defective product or an image of a defective product, and the image inspection device S can be configured to make a quality judgment related to whether the workpiece W is a non-defective product or a defective product based on the recognition result. In addition, in the setting mode, for example, preparatory work is performed to enable the user to separate non-defective products from defective products in the desired product inspection.

[0066] In the driving mode, the workpiece W is inspected based on the frame image FR captured at the actual site. In addition to the quality judgment described above, the inspection of the workpiece W also includes a quantity count for counting the number of workpieces W. Switching between the setup mode and the driving mode can be performed on the GUI to be described later. In addition, it can be configured to automatically transition to the driving mode upon completion of the setup mode. In the driving mode, correction or change of the recognition boundary using the discriminator (that is, so-called additional learning) can also be performed.

[0067] (AI trigger tool)

[0068] Incidentally, when performing image inspection on the moving workpiece W, it is necessary to acquire, as an inspection target image, a frame image FR captured at a timing when the workpiece W is in a desired state (position and angle, etc.) in the imaging field.

[0069] Photoelectric sensors, commonly used as trigger sensors for output triggering, are difficult to trigger with thin workpieces. Furthermore, even when triggering conditions are set so that the trigger is applied to a workpiece with a predetermined outline, size, or color, it is difficult to trigger each of multiple types of workpieces with different outlines, sizes, or colors. Furthermore, it is difficult to trigger with multiple workpieces arranged without gaps. Alternatively, it is possible to trigger twice with a ring-shaped workpiece. Note that some of the above issues can be resolved by using a color sensor, etc. However, appropriate selection and skilled techniques are required.

[0070] In addition, in recent years, there are also examples of using image sensors to apply triggers. In this case, for example, even in workpieces such as thin workpieces that are difficult to generate triggers using photoelectric sensors, triggers can be applied when the trigger conditions can be defined using visual characteristics on the captured image. However, since there are many workpieces whose visual characteristics on the captured image are unstable, it is difficult to set the trigger conditions. For example, in the case of a process of searching for the outline of a workpiece and applying a trigger to the outline of the workpiece, it is difficult to stably apply the trigger when the outline of the workpiece on the captured image becomes unstable due to halo or the like. In addition, for example, even in the case where the outline is different between an OK (qualified) workpiece and an NG (unqualified) workpiece, it is difficult to apply the trigger. Therefore, the cases where an image sensor can be used as a trigger generation unit are extremely limited.

[0071] Meanwhile, the image inspection apparatus S includes an AI trigger tool as one of various tools. The AI ​​trigger tool uses a machine learning model to determine whether each of a plurality of sequentially acquired frame images FR is an image for initiating an inspection trigger.

[0072] Note that the AI ​​trigger tool that generates an inspection trigger for the workpiece W appearing in the frame image FR digitizes the frame image FR and compares the digitized value with a threshold value TH used to determine whether to activate the inspection trigger (details will be described later). On the other hand, the AI ​​tool that determines whether the workpiece W shown in the frame image FR is a non-defective product or a defective product based on the frame image FR digitizes the frame image FR and compares the frame image FR with the threshold value used to determine whether the workpiece W is a non-defective product or a defective product. In other words, the AI ​​trigger tool can be understood as a type of AI tool, as viewed from within the image inspection device S.

[0073] By using AI to make trigger judgments, it is possible to achieve trigger judgments that are robust to halo and other factors.

[0074] In addition, the image inspection apparatus S can flexibly set trigger conditions so that an inspection trigger is output at a timing desired by the user. This will be described in detail below.

[0075] Figure 4 2 is a diagram illustrating an example of the setup flow of the AI ​​trigger tool. The execution subject of this flow can be basically understood as the inspection setup unit 200.

[0076] At the start of this process, in step S11, a mode is selected from a plurality of modes (e.g., standard mode, sorting mode, and pass-through mode). The following description assumes that the standard mode is selected in this step. In step S12, an ON image F (foreground) is selected, in which the workpiece W is within the detection area. On the other hand, in step S13, an OFF image B (background) is selected, in which the workpiece W is not within the detection area. Note that at least one of the ON image F and the OFF image B can be cut out from the frame image FR during real-time imaging or playback. Furthermore, the order of steps S12 and S13 can be reversed.

[0077] In step S14, the ON image F and the OFF image B are input to the feature extraction unit of the machine learning model. The feature extraction unit obtains the feature quantity Ff of the ON image F and the feature quantity Fb of the OFF image B. Furthermore, in step S14, a threshold value TH relative to the score SC of the frame image FR generated by the imaging unit 1 is determined for comparison.

[0078] The score SC is a numerical value indicating whether the frame image FR is classified as an ON image F or an OFF image B. For example, the score SC increases as the frame image FR becomes closer to the ON image F, and decreases as the frame image FR becomes closer to the OFF image B. In other words, the score SC increases as the feature quantity Fq acquired when the frame image FR is input to the feature extraction unit becomes similar to the feature quantity Ff of the ON image F, and decreases as the feature quantity Fq becomes similar to the feature quantity Fb of the OFF image B. The feature quantities Ff, Fb, and Fq can each be a feature vector of the detection area.

[0079] As described above, since the score SC can be a value that increases as the frame image FR approaches the ON image F and decreases as the frame image FR approaches the OFF image B, the threshold TH determined in step S14 of this embodiment can be determined as a threshold TH relative to the score SC of the frame image FR. For example, while the threshold TH is set to a fixed value, the method used to calculate the score SC based on the feature quantity can be changed so that the score SC based on the feature quantity Ff becomes greater than the threshold TH and the score SC based on the feature quantity Fb becomes less than the threshold TH, thereby substantially changing the threshold TH. In other words, the method used to calculate the score SC can be determined so that the score SC based on the feature quantity Ff is relatively large relative to the predetermined threshold TH, while the score SC based on the feature quantity Fb is relatively small. Furthermore, the value of the threshold TH itself can be changed based on the feature quantity.

[0080] In this embodiment, two images, the ON image F and the OFF image B, are selected in steps S13 and S14. However, the images may be selected so that a relative threshold value TH can be determined. For example, the threshold value TH defining the range of scores SC considered similar to the ON image F may be determined based on the feature quantity Ff extracted from the ON image F. Alternatively, the threshold value TH defining the range of scores SC for images considered to be in a state different from the OFF image B (that is, a state including the workpiece W) may be determined based on the feature quantity Fb extracted from the OFF image B.

[0081] In step S15, an additional selection (relearning) of at least one of the ON image F and the OFF image B is received. In step S16, a delay is set from the output of the inspection trigger until the actual execution of the inspection. However, steps S15 and S16 are not necessary steps for setting the AI ​​trigger tool.

[0082] Figure 5 is an example of Figure 4 When the image inspection program is executed by PC 3, GUI 400 (=various screens 400a to 400g) is displayed on display device 4. GUI 400 includes an image display area 410, an operation area 420, and a progress display area 430 as a basic layout.

[0083] Image display area 410 displays the frame image FR captured by imaging unit 1. Image display area 410 may also include a status display banner 411, zoom-in and zoom-out buttons 412, and a maximum display button 413. Status display banner 411 briefly displays the operating status (status) of the image inspection program. Zoom-in and zoom-out buttons 412 and maximum display button 413 are operated to zoom in or out and maximize the image displayed in image display area 410, respectively.

[0084] When the image inspection device S is first started, the initial startup screen 400a (first row in the left column) is displayed. Note that the image inspection program cannot be set when the image inspection device S is first started. Therefore, an alarm mark a1 (or an alarm message) indicating that the master image is not registered can be displayed in the image display area 410. In addition, a banner (e.g., "Master") indicating that the master image is not registered (or is being registered) can be displayed on the status display banner 411.

[0085] In addition, for example, a setting start button a2 is displayed in the operation area 420 of the initial startup screen 400a. When the setting start button a2 is clicked (or tapped, and the same applies below), the setting of the image inspection program is started. Figure 4This corresponds to the start of the illustrated setup flow.

[0086] In the setting of the image inspection program, first, the mode selection screen 400b (the second row in the left column) is displayed. Figure 4 For example, as illustrated in the figure, a mode selection dialog box b0 may be displayed on the mode selection screen 400b.

[0087] In the mode selection dialog box b0 , for example, a standard mode button b1 , a sorting mode button b2 , a pass mode button b3 , a guide b4 , an OK button b5 , and a cancel button b6 are displayed.

[0088] When the standard mode button b1 is clicked, the standard mode is selected. In the standard mode, the workpieces W are identified as either non-defective products or defective products. When the sorting mode button b2 is clicked, the sorting mode is selected. In the sorting mode, the workpieces W are sorted (classified) into a plurality of categories. When the through mode button b3 is clicked, the through mode is selected. In the through mode, the number of workpieces W that sequentially pass through the field of view of the camera unit 1 is counted. In the guide b4, an overview and schematic diagram of the modes selected by clicking the standard mode button b1, the sorting mode button b2, and the through mode button b3 can be displayed.

[0089] Clicking the OK button b5 confirms the selected mode. Clicking the Cancel button b6 cancels the selected mode and closes the mode selection dialog box b0. The following description assumes that the standard mode has been selected on the mode selection screen 400b (mode selection dialog box b0).

[0090] When standard mode is selected on mode selection screen 400b, the standard mode setup process proceeds sequentially through the first step (shooting setup), the second step (master registration), the third step (tool setup), and the fourth step (output allocation). Each step is then displayed as a flowchart in the progress display area 430, with the currently executing step highlighted. This allows the user to easily understand the progress of the setup process.

[0091] First, in the first process (shooting setting), ON and OFF setting screens 400c1 and 400c2 (the third and fourth rows in the left column) are displayed. Figure 4In the operation area 420 of the ON and OFF setting screens 400c1 and 400c2, for example, an ON image display area c1, an OFF image display area c2, an ON image registration button c3, an OFF image registration button c4, a learning start button c5, a back button c6, and a cancel button c7 can be displayed.

[0092] As shown in the figure, in the first step (shooting setting), a real-time image (=moving image being shot) or a replayed image (=recorded moving image) can be displayed in the image display area 410. According to the figure, the status display banner 411 can display a banner indicating that the replayed image is being displayed (for example, "Replay").

[0093] At this time, for example, an image operation button c8 may be displayed in the image display area 410. The image operation button c8 is operated to play and pause, rewind and fast forward, and rewind or forward frame by frame.

[0094] In selecting the ON image F and the OFF image B, a window w (=detection area) is first set for the replay image. Window w can be, for example, rectangular. The user can specify the detection area by adjusting the position, size, and angle of window w to surround the workpiece W appearing in the replay image.

[0095] Subsequently, by using the image operation button c8, images to be cut out of the replayed image as ON image F and OFF image B are searched. For example, in the image display area 410 of the ON and OFF setting screen 400c1, an image obtained by capturing a state in which a workpiece W is present inside the window W (the state in which a capture trigger is to be applied) is displayed. When the ON image registration button c3 is clicked in this state, the image currently displayed in the image display area 410 is selected as the ON image F. The selected ON image F is displayed in the ON image display area c1. The workpiece W appearing in the ON image F can be a non-defective product or a defective product. In the OFF image display area c2, as illustrated in the figure, information indicating that the OFF image B is not selected can be displayed.

[0096] In addition, the image display area 410 of the ON / OFF setting screen 400c2 displays an image obtained by capturing a state in which no workpiece W exists within the inspection window w (a state in which the inspection trigger is not to be applied). When the OFF image registration button c4 is clicked in this state, the image currently displayed in the image display area 410 is selected as the OFF image B. The selected OFF image B is displayed in the OFF image display area c2.

[0097] After selecting ON image F and OFF image B, clicking the Learning Start button c5 starts learning the machine learning model used in the AI ​​trigger tool. Clicking the Back button c6 returns the screen to the aforementioned mode selection screen 400b. Clicking the Cancel button c7 cancels the selection of ON image F and OFF image B.

[0098] When the learning of the machine learning model starts, the learning progress screen 400d (the first row in the right column) is displayed. Figure 4 The image inspection device S (particularly the inspection setting unit 200) obtains the feature quantity Ff of the ON image F and the feature quantity Fb of the OFF image B by inputting the learning images (ON image F and OFF image B) into the machine learning model as learning for the machine learning model.

[0099] Furthermore, a threshold value TH relative to the score SC of the frame image FR generated by the imaging unit 1 is determined for comparison. This threshold value TH is determined to allow for discrimination of whether the workpiece W appears inside the window w. Whether the workpiece W is a non-defective product or a defective product is acceptable. In other words, an inspection trigger can be applied even if the workpiece W is a defective product. In other words, a series of inspection outputs can be performed: regardless of whether the workpiece W appears in the frame image FR as a non-defective product or a defective product, the inspection trigger is applied, and the non-defective or defective product is determined through image inspection. Hysteresis can be applied to the threshold value TH to prevent jitter in the inspection trigger.

[0100] Note that, as illustrated in the figure, a learning progress dialog d0 may be displayed on the learning progress screen 400d. In the learning progress dialog d0, for example, a learning progress bar d1 indicating a degree of progress (0% to 100%) is displayed.

[0101] As described above, based on machine learning using real-time images or replayed images of the workpiece W, the user can set an appropriate trigger condition (threshold value TH) without having to abstract the image characteristics of the ON image F and the OFF image B and appropriately set the trigger condition. This improves the flexibility of user settings.

[0102] In the score curve graph e1, the relationship between the score SC and the threshold TH for the image displayed in the image display area 410 (specifically, the detection area demarcated by the window wd) is displayed as a graph. This visualization can visually convey the stability of the inspection trigger to the user. For example, the user can verify how the inspection trigger is being applied by comparing the image morphology before and after the inspection trigger is applied while tracking the transition of the score SC over time.

[0103] In the score curve graph e1, the relationship between the score SC and the threshold TH for the image displayed in the image display area 410 (specifically, the detection area demarcated by the window wd) is displayed as a graph. This visualization can visually convey the stability of the inspection trigger to the user. For example, the user can verify how the inspection trigger is being applied by comparing the image morphology before and after the inspection trigger is applied while tracking the transition of the score SC over time.

[0104] The score curve graph e1 can also be used by the user as a tool to assist in selecting images suitable for additional learning. For example, in the additional image selection screen 400e shown in the figure, the majority of workpiece W is inside window w, but the score SC does not reach the threshold TH. Therefore, the trigger count e2 remains "0." If an inspection trigger is to be applied to this image, it is desirable to perform additional learning on this image as an ON image F. Conversely, if an inspection trigger is not to be applied to this image, it can be performed additional learning on this image as an OFF image B.

[0105] In addition, although not explicitly illustrated in the figure, for example, a mark indicating a peak (inflection point) of the vertically varying score SC can be given to the score curve graph e1. The score SC generally takes a peak at the timing when the workpiece W enters or leaves the window w. Thus, using the structure given a mark, it is possible to efficiently search for images in which output omissions caused by inspection triggers have occurred. In particular, when a mark given to the score SC is clicked, the screen can jump to the corresponding image and display it in the image display area 410. In addition, in the case where there are multiple learning recommendation images, each learning recommendation image can be displayed as a thumbnail as an additional learning candidate.

[0106] When the additional learning button e3 is clicked, the screen changes to a confirmation screen 400f for confirming whether to perform additional learning on the image currently displayed in the image display area 410. As illustrated in the figure, an additional learning dialog box f0 may be displayed on confirmation screen 400f. Additional learning dialog box f0 may include, for example, an additional image display area f1, an ON image registration button f2, an OFF image registration button f3, and a cancel button f4.

[0107] The image displayed in the image display area 410 at the time the additional learning button e3 was clicked is displayed in the additional image display area f1. If the ON image registration button f2 is clicked in this state, the image currently displayed in the additional image display area f1 is additionally learned as the ON image F. As a result, the inspection trigger is applied at the time when the workpiece W enters the window w to the same extent as the added image.

[0108] On the other hand, when the OFF image registration button f3 is clicked, the image currently displayed in the additional image display area f1 is additionally learned as the OFF image B. As a result, even if the workpiece W enters the window w to the same extent as the added image, the inspection trigger is not triggered.

[0109] Note that an image to be used for additional learning may be cut out from the frame image FR during real-time imaging or playback reproduction.

[0110] Furthermore, if a delay DL is set from the time the score SC exceeds the threshold TH until the actual output check is triggered, the delay setting screen 400g is displayed. Specifically, the delay setting screen 400g is displayed when a frame image FR that is a predetermined number of frames later than the frame image FR in which the score SC exceeds the threshold TH is set as the detection target image. For example, the operation area 420 of the delay setting screen 400g may display a score curve graph g1, a delay setting tool g2, an OK button g3, and a Cancel button g4.

[0111] In the score graph g1, as described above, the relationship between the score SC and the threshold value TH of the image displayed in the image display area 410 is displayed as a graph. In addition, the delay DL can be clearly indicated in the score graph g1.

[0112] When setting the delay DL, the delay setting tool g2 is operated. The delay setting tool g2 may include a box for receiving a direct input of the delay DL, a + button for receiving an increment of the delay DL, and a - button for receiving a decrement of the delay DL.

[0113] Note that when DL = 0, an inspection trigger is output without delay at the point in time when the score SC exceeds the threshold TH, and image inspection is performed (e.g., determining whether a product is defective or not). On the other hand, when DL > 0, an inspection trigger is output several frames after the score SC exceeds the threshold TH, and the frame image FR several frames later is processed as the detection target image. As described above, by setting a delay DL, it is possible to perform image inspection on an image different from the image obtained at the point in time when the score SC exceeds the threshold TH. For example, setting a delay DL can be effective when inspecting a portion of a workpiece W.

[0114] When the OK button g3 is clicked, the delay DL is confirmed. On the other hand, when the cancel button g4 is clicked, the delay DL is canceled.

[0115] The above description is an overview of the GUI transition in the first step (shooting settings). Note that although not explicitly shown in the figure, in the first step (shooting settings), the shooting field of view, image brightness, focus, shooting interval (frame rate), etc. can be set.

[0116] After the first step (shooting setup) is completed, the second step (master registration) registers a master image of the workpiece W to be judged as a non-defective product or a defective product. Note that the master image can be registered from a real-time image, a drive history image, or a file image. In the third step (tool setup), various settings related to rule-based or learning-based tools are made. In the fourth step (output allocation), the output content (non-defective or defective inspection results, busy results, error results, etc.) is allocated to the output ports of the inspection result output unit 320.

[0117] (Check trigger)

[0118] Finally, the output operation of the inspection trigger will be described. The image inspection device S (specifically, the inspection execution unit 300) in the driving mode acquires feature quantities Fq1 and Fq2 from the frame images FR1 and FR2 sequentially generated by the imaging unit 1. A detection trigger is output when the score SC of the frame image FR1 calculated based on the feature quantity Fq1 is lower than the threshold value TH, and the score SC of the frame image FR2 calculated based on the feature quantity Fq2 is equal to or greater than the threshold value TH (that is, when Fq1 < TH < Fq2).

[0119] Note that the image inspection device S (particularly the inspection execution unit 300) preferably performs inspection processing on the frame image FR2 at a point in time when the score SC is equal to or greater than the threshold value TH. With such a configuration, for example, even if the workpiece W is outside the field of view in the frame image FR3 generated immediately after the frame image FR2, image inspection can be performed without any trouble.

[0120] <Other>

[0121] Note that, in addition to this embodiment, various modifications may be made to the various technical features disclosed in this specification without departing from the spirit of the technical innovation. In other words, it should be considered that this embodiment is illustrative in all respects and not restrictive. In addition, the technical scope of the present invention is defined by the claims and should be understood to include all modifications that fall within the meaning and scope equivalent to the claims.

Claims

1. An image inspection device, comprising: The camera unit is used to continuously shoot the shooting field of view to generate a plurality of frame images arranged in a time series; an inspection execution unit configured to execute an inspection process of the objects appearing in the plurality of frame images using a machine learning model to output an inspection result; as well as An inspection setting unit, configured to set up the inspection execution unit, The machine learning model includes: a feature extraction unit, configured to extract feature quantities from the frame image; and a judgment unit, configured to output the inspection result according to the feature quantity, The inspection setting part: receiving a selection of a first image, and determining a score calculation method based on a first feature amount extracted from the selected first image so that a score based on the first feature amount satisfies a predetermined relative relationship with respect to a threshold value, and The inspection execution unit: extracting the feature amount from a first frame image as the plurality of frame images and a second frame image continuous with the first frame image, and When it is determined that the threshold value exists between a first score based on the feature amount extracted from the first frame image and a second score based on the feature amount extracted from the second frame image, a check trigger is output.

2. The image inspection device according to claim 1, in, The inspection setting part: receiving a selection of an image in which the object is in a detection area as the first image and a second image in which the object is not in the detection area, and A threshold value to be compared with a score indicating whether the frame image is classified as the first image or the second image is determined based on the first feature amount and a second feature amount extracted from the second image.

3. The image inspection device according to claim 2, in, The inspection setting section determines the threshold value for the score so that the score of a frame image similar to the first image is higher than the score of a frame image similar to the second image, and The inspection execution unit obtains a third feature quantity and a fourth feature quantity from the first frame image and the second frame image generated as the multiple frame images, respectively, and outputs an inspection trigger at a time point when the first score of the first frame image calculated based on the third feature quantity extracted from the first frame image is lower than the threshold, and the second score of the second frame image calculated based on the fourth feature quantity extracted from the second frame image continuous with the first frame image is equal to or greater than the threshold.

4. The image inspection device according to claim 1, wherein The inspection execution unit executes the inspection process on the second frame image.

5. The image inspection device according to claim 2, in, The inspection setting part: Display images during live recording or playback, and A selection of an image to be displayed as at least one of the first image and the second image is received. 6 . The image inspection apparatus according to claim 1 , further comprising a GUI for displaying the score as a graph.

7. An image inspection device comprising: The camera unit is used to continuously shoot the shooting field of view to generate a plurality of frame images arranged in a time series; an inspection execution unit configured to execute an inspection process of the objects appearing in the plurality of frame images using a machine learning model to output an inspection result; as well as An inspection setting unit, configured to set up the inspection execution unit, The machine learning model includes: a feature extraction unit, configured to extract feature quantities from the frame image; and a judgment unit, configured to output the inspection result according to the feature quantity, The inspection setting part: receiving a selection of a first image, and determining a threshold value for the score based on a first feature amount extracted from the selected first image, and The inspection execution unit: extracting the feature amount from a first frame image as the plurality of frame images and a second frame image continuous with the first frame image, and When it is determined that the threshold value exists between a first score based on the feature amount extracted from the first frame image and a second score based on the feature amount extracted from the second frame image, a check trigger is output.

Images