Image inspection apparatus

Abstract

To provide an image inspection apparatus that inspects inspection image data using a model capable of reducing the amount of calculation required for learning at a work production site.SOLUTION: An image inspection apparatus (1) comprises an imaging unit (14) and a control unit (13a) that executes a model to which inspection image data obtained by imaging by the imaging unit is input. The model is a segmentation model. The model comprises an encoder unit (101) that includes a plurality of convolution layers and extracts a first feature from the inspection image data; a connection unit (102) that receives a second feature different from the first feature from at least one of the plurality of convolution layers, converts the second feature into a third feature, and supplies the third feature; and a decoder unit (103) that upsamples the first feature while using the third feature. The control unit updates the parameters of the connection unit and the parameters of the decoder unit when performing machine learning of the model based on training image data.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image inspection device. [Background technology]

[0002] There is known an apparatus that uses images of non-defective products at a work production site to train a machine learning model so that defective parts of defective products are extracted (see, for example, Patent Document 1).

[0003] In order to improve the accuracy of the boundaries of defective areas, it is possible to use U-Net (see, for example, Non-Patent Document 1), which is an example of a segmentation model that classifies image data on a pixel-by-pixel basis, instead of the machine learning model disclosed in Patent Document 1. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-077054 [Non-patent literature]

[0005] [Non-Patent Document 1] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation” [online], May 18, 2015, arXiv, [searched on December 8, 2020], Internet<URL: https: / / arxiv.org / pdf / 1505.04597.pdf > [Non-patent document 2] Systems Planning Research Institute, Inc. “gLupe” [online], [Retrieved October 17, 2024], Internet <URL:https: / / glupe.jp / glupe / index.php?utm_source=bing&utm_medium=cpc&msclkid=ba67502dfd3e1dc02ec5372a8db534be > Summary of the Invention [Problem to be solved by the invention]

[0006] However, U-Net requires a large amount of calculation to train the encoder that makes up the model, making it difficult to train on equipment with low processing power (such as a central processing unit (CPU)). For this reason, U-Net is difficult to train on production sites. Note that "difficult to train" also includes cases where training takes an extremely long time.

[0007] In view of the above problems, the present invention aims to provide an image inspection device that inspects inspection image data using a model that can reduce the amount of calculation required for learning at the work production site. [Means for solving the problem]

[0008] For example, an image inspection device according to the present invention is an image inspection device that inspects inspection image data using a model whose parameters are updated through machine learning based on training image data presented by a user. The image inspection device includes an imaging unit and a control unit that executes the model to which the inspection image data obtained by imaging by the imaging unit is input. The model outputs image data that can distinguish, based on a label indicating a first class assigned to the training image data, image regions of the input inspection image data that belong to the first class from regions that do not belong to the first class. The model includes multiple convolutional layers and includes an encoder unit that extracts first features from the inspection image data, a connection unit that receives second features different from the first features from at least one of the multiple convolutional layers, converts the second features into third features, and supplies the third features, and a decoder unit that upsamples the first features extracted by the encoder unit using the third features supplied by the connection unit. The control unit updates parameters of the connection unit and the decoder unit when executing machine learning of the model based on the training image data.

[0009] Still other features, elements, steps, advantages, and characteristics will become more apparent from the detailed description that follows and the accompanying drawings related thereto. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide an image inspection device that inspects inspection image data using a model that can reduce the amount of calculation required for learning at the work production site. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram showing a configuration of a visual inspection apparatus according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram showing a hardware configuration (controller type) of a visual inspection device. [Figure 3] 10A and 10B are diagrams illustrating input / output processing in the learning stage and the operation stage. [Figure 4] FIG. 1 is a diagram illustrating a schematic configuration of a machine learning model. [Figure 5] FIG. 2 is a diagram illustrating an example of the structure of an encoder unit. [Figure 6] 10A and 10B are diagrams illustrating an example of the structure of a connection portion. [Figure 7] FIG. 2 is a diagram illustrating an example of the structure of a decoder unit. [Figure 8] FIG. 10 is a diagram showing a schematic flow of annotation processing. [Figure 9] FIG. 2 is a diagram showing a display screen of a display device. [Figure 10] FIG. 2 is a diagram showing a display screen of a display device. [Figure 11] FIG. 2 is a diagram showing a display screen of a display device. [Figure 12] FIG. 2 is a diagram showing a display screen of a display device. [Figure 13] FIG. 10 is a diagram showing a schematic flow of a test result display process. [Figure 14] FIG. 2 is a diagram showing a display screen of a display device. [Figure 15] FIG. 10 is a diagram showing a schematic flow of simple annotation processing. [Figure 16] FIG. 10 is a diagram showing an example of a non-defective product image. [Figure 17] FIG. 10 is a diagram showing an example of a defective product image. [Figure 18] FIG. 2 is a diagram showing a display screen of a display device. [Figure 19] FIG. 2 is a diagram showing a display screen of a display device. [Figure 20] FIG. 2 is a diagram showing a display screen of a display device. [Figure 21] FIG. 10 is a block diagram showing another hardware configuration (smart camera type) of the appearance inspection device. [Figure 22] FIG. 10 is a diagram showing how a removable memory is used in a smart camera type visual inspection device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0013] (Configuration of visual inspection device 1) FIG. 1 is a schematic diagram showing the configuration of an appearance inspection device 1 according to an embodiment of the present invention. The appearance inspection device 1 is a device that captures an image of a workpiece to be inspected, such as various parts or products, and determines whether the image is good or bad. The result of the determination is output to an external device (not shown) connected to the external inspection device 1. The appearance inspection device 1 can be used in production sites such as factories. Specifically, a machine learning network is built inside the appearance inspection device 1, and this machine learning network is generated by learning at least one of good product images corresponding to good products and defective product images corresponding to defective products. A workpiece image captured from the workpiece to be inspected is input to the generated machine learning network, and the machine learning network can determine whether the workpiece image is good or bad. The appearance inspection device 1 can be understood as one aspect of an image inspection device.

[0014] The entire workpiece may be the object of inspection, or only a portion of the workpiece may be the object of inspection. Also, one workpiece may contain multiple inspection objects. Also, a workpiece image may contain multiple workpieces.

[0015] The appearance inspection device 1 comprises a control unit 2, which is the device main body, an imaging unit 3, a display device (display section) 4, and a personal computer 5. The personal computer 5 is not essential and can be omitted. The personal computer 5 can be used instead of the display device 4 to display various information and images, and the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0016] 1 illustrates a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5 as an example of the configuration of the visual inspection device 1, but any two or more of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Furthermore, the control unit 2 can be divided into multiple units and some of them can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into multiple units and some of them can be incorporated into other units.

[0017] (Configuration of imaging unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging section) 14 and an illumination module (illumination section) 15, and is a unit that acquires workpiece images. The camera module 14 includes an AF (autofocus) motor 141 that drives the imaging optical system, and an imaging board 142. The AF motor 141 automatically adjusts focus by driving the lens of the imaging optical system, and can perform focus adjustment using a conventionally known method such as contrast autofocus. The imaging board 142 includes a CMOS (complementary metal oxide semiconductor) sensor 143 as a light-receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of the CMOS sensor 143, a light-receiving element such as a CCD (charge coupled device) sensor can also be used.

[0018] The lighting module 15 includes an LED (light emitting diode) 151 as a light emitting element that illuminates an imaging area including a workpiece, and an LED driver 152 that controls the LED 151. The timing, duration, and amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided separately from the imaging unit 3 as an external lighting unit.

[0019] (Configuration of display device 4) The display device 4 has a display panel made up of, for example, a liquid crystal panel or an organic EL (electroluminescence) panel. A work image, a user interface image, etc. output from the control unit 2 are displayed on the display device 4. If the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used in place of the display device 4.

[0020] (operation equipment) Examples of operation devices for a user to operate the visual inspection apparatus 1 include, but are not limited to, the keyboard 51 and mouse 52 of the personal computer 5, and any device configured to be able to accept various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operation devices.

[0021] User operations on the keyboard 51 and mouse 52 can be detected by the control unit 2. The touch panel 41 is a conventionally known touch-type operation panel equipped with, for example, a pressure-sensitive sensor, and user touch operations can be detected by the control unit 2. The same applies when other pointing devices are used.

[0022] (Configuration of control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The main board 13 is provided with a processor 13a. The processor 13a controls the operation of each connected board and module. For example, the processor 13a outputs a lighting control signal to an LED driver 152 of the lighting module 15 to control the turning on / off of the LED 151. In response to the lighting control signal from the processor 13a, the LED driver 152 switches the turning on / off of the LED 151 and adjusts the lighting time, and also adjusts the light intensity of the LED 151.

[0023] In addition, the processor 13a outputs an imaging control signal to the imaging board 142 of the camera module 14 to control the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13a and adjusts the exposure time to any desired time to perform imaging. That is, the imaging unit 3 captures an image within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13a. If a workpiece is present within the field of view, the image of the workpiece is captured. However, if an object other than the workpiece is present within the field of view, the image of the object can also be captured. For example, the visual inspection device 1 can use the imaging unit 3 to capture non-defective product images corresponding to non-defective products and defective product images corresponding to defective products as learning images for the machine learning network. The learning images do not have to be images captured by the imaging unit 3, but may be images captured by another camera, etc.

[0024] On the other hand, when the visual inspection device 1 is in operation, the workpiece can be imaged by the imaging unit 3. The CMOS sensor 143 is configured to be able to output a live image, i.e., a currently captured image, at a short frame rate at any time.

[0025] When the CMOS sensor 143 has finished capturing an image, the image signal output from the imaging unit 3 is input to the processor 13a of the main board 13 for processing, and is also stored in the memory 13b of the main board 13. Specific details of the processing performed by the processor 13a of the main board 13 will be described later. The main board 13 may be provided with a processing device such as an FPGA (field programmable gate array) or a DSP (digital signal processor). The processor 13a may also be an integrated processor such as an FPGA or a DSP.

[0026] The connector board 16 is a part that receives power from an external source via a power connector (not shown) provided on the power supply interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and module, specifically, to the illumination module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 is equipped with an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14 to achieve autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 in response to an AF control signal from the processor 13a on the main board 13. The connector board 16 also outputs inspection results to an external device via an I / O terminal provided on the I / O interface 162.

[0027] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, and communication between the main board 13 and an external control device (not shown). An example of the external control device is a programmable logic controller. The communication may be wired or wireless, and either form of communication can be realized by a conventionally known communication module.

[0028] The control unit 2 is provided with a storage device 19, which may be, for example, a solid state drive, a hard disk drive, or the like. The storage device 19 stores a program file 80, a setting file, and the like (software) for enabling the hardware to execute the various controls and processes described below. The program file 80 and the setting file may be stored in a storage medium 90, such as an optical disk, and the program file 80 and the setting file stored in the storage medium 90 may be installed in the control unit 2. The program file 80 may be downloaded from an external server via a communication line. The storage device 19 may also store, for example, the image data and parameters for constructing a machine learning network for the visual inspection apparatus 1.

[0029] That is, the processor 13a of the visual inspection device 1 is configured to read parameters and the like stored in the storage device 19 to construct a machine learning network, input a workpiece image obtained by photographing the workpiece to be inspected into the constructed machine learning network, execute the constructed machine learning network, and determine whether the workpiece is good or bad based on the input workpiece image. By using this visual inspection device 1, it is possible to execute a visual inspection method for determining whether the workpiece is good or bad based on the workpiece image. The machine learning network may be understood as a machine learning model (a model in which parameters are updated by machine learning).

[0030] (Input / output processing) 3 is a diagram showing input / output processing in the learning stage and the operation stage of the visual inspection device 1. As shown in this diagram, in the learning stage of the visual inspection device 1, learning of a machine learning model is performed based on learning data presented by a user (a customer from the vendor's perspective).

[0031] The training data includes training image data and instruction content. The training image data includes at least one of image data of a good product and image data of a defective product. The instruction content includes labels indicating classes such as "this image data is a good product," "this image data is a defective product," or "this part is abnormal."

[0032] In the above learning, the parameters of the machine learning model are updated (adjusted) so that the output of the machine learning model approaches an expected value according to the teaching content. Multiple machine learning models may be prepared (model 1 to model 3). With this configuration, it becomes possible to arbitrarily select a learning target or an operation target depending on the application of the visual inspection device 1.

[0033] It should be noted that the user does not necessarily have to perform all of the above learning processes. For example, learning that requires a relatively large amount of calculation may be completed by the vendor before shipping the visual inspection apparatus 1, and only learning that requires a relatively small amount of calculation may be performed by the user before operating the visual inspection apparatus 1. In this specification, learning performed by the vendor before shipping is referred to as pre-shipment learning, and learning performed by the user before operating the visual inspection apparatus 1 is referred to as customer learning.

[0034] That is, the machine learning model of the visual inspection apparatus 1 may include a parameter-fixed portion. The parameter-fixed portion is a layer in which parameters obtained by pre-shipment learning on the vendor side are fixed, in other words, a layer in which on-site learning on the user side is not required.

[0035] Since the machine learning model of the visual inspection device 1 includes a parameter fixed portion, it is no longer necessary for the user to prepare high-performance equipment (such as a GPU [graphics processing unit]) or for the vendor to provide an advanced learning environment using a GPU or the like as a cloud service (such as SaaS). This lowers the barrier to introducing the visual inspection device 1.

[0036] As such, the above-mentioned learning should be understood broadly to include not only learning that requires a large amount of computation, such as deep learning, but also learning that requires a small amount of computation (customer-based learning).

[0037] Meanwhile, in the operational stage of the visual inspection device 1, inspection image data is inspected using the trained machine learning model. The inspection includes segmentation. In addition to segmentation, the inspection may also include image classification, anomaly detection, and the like.

[0038] In image region segmentation, each pixel forming the image is classified, and each region is divided by classification. In pass / fail judgment by image inspection, the visual inspection device 1 classifies each pixel forming the image as abnormal / normal. If the area consisting of pixels classified as abnormal is equal to or greater than a certain area, the object (workpiece) appearing in the image is judged as defective. In image classification, the image or each region specified in the image is classified. In pass / fail judgment by image inspection, the visual inspection device 1 classifies images showing the object (workpiece) into good images and defective images, and judges the object (workpiece) appearing in the good image as good, and the object (workpiece) appearing in the defective image as defective. In image anomaly detection, abnormal parts are extracted from the image. For example, anomaly detection using an autoencoder is well known. An autoencoder can be understood as a machine learning model that is trained (parameters adjusted) so that, when normal and abnormal images are input, abnormal parts in the abnormal image are more likely to stand out. The visual inspection device 1 determines whether the image is a non-defective image or a defective image based on the degree of abnormality and area of ​​the abnormality detected by the abnormality detection, thereby determining whether the object (workpiece) shown in the image is good or bad.

[0039] The visual inspection device 1 also includes a report output unit (model evaluation result generating unit) that outputs a report display based on the output result of the machine learning model in the learning stage or the operation stage. In other words, the target image data input to the machine learning model when displaying the report may be at least one of the learning image data and the inspection image data.

[0040] The report output unit can be understood as, for example, one function of editor software executed on the personal computer 5. In other words, the personal computer 5 functions as the report output unit by executing the editor software.

[0041] (Segmentation model) The machine learning model model1 used in the visual inspection device 1 is a segmentation model that classifies image data on a pixel-by-pixel basis. Based on a label indicating a first class (abnormal) assigned to the training image data, the machine learning model model1 outputs image data that can distinguish areas belonging to the first class from areas not belonging to the first class among the image areas of the input inspection image data.

[0042] 4 is a diagram showing a schematic configuration of the machine learning model model 1. The machine learning model model 1 includes an encoder unit 101, a connection unit 102, and a decoder unit 103.

[0043] The encoder unit 101 is a neural network having an FCN (fully convolution network) structure and includes multiple convolution layers. The encoder unit 101 receives input image data IN1. The encoder unit 101 extracts a first feature value FT1 from the input image data IN1. The encoder unit 101 supplies the first feature value FT1 to the decoder unit 103. In the learning stage of the visual inspection device 1, the input image data IN1 is training image data. In the operation stage of the visual inspection device 1, the input image data IN1 is inspection image data.

[0044] The connection unit 102 receives second features FT2, which are different from the first features FT1, from at least one of the multiple convolutional layers included in the encoder unit 101. As will be described in detail later, the second features FT2 have information that is not included in the first features FT1 and are information that improves the processing accuracy in the decoder unit 103. The connection unit 102 converts the second features FT2 into third features FT3. The connection unit 102 supplies the third features FT3 to the decoder unit 103.

[0045] The decoder unit 103 upsamples the first feature FT1 while using the third feature FT3. The decoder unit 103 outputs the output image data OUT1 obtained by upsampling the first feature FT1. The output image data OUT1 has the same size (number of pixels in the width direction × number of pixels in the height direction) as the input image data IN1. The number of channels of the output image data OUT1 is 1, regardless of the number of channels of the input image data IN1.

[0046] The parameters of the encoder unit 101 are fixed by learning at the time of shipment by the vendor. On the other hand, the parameters of the connection unit 102 and the decoder unit 103 are adjusted based on learning data presented by the user during the learning stage of the visual inspection device 1.

[0047] As described above, the parameters of the encoder unit 101 are fixed by learning at the time of shipment by the vendor. Therefore, the machine learning model model1 can reduce the amount of calculation required for learning based on learning data presented by the user (learning at the work production site). Note that some of the parameters of the encoder unit 101 may be adjusted based on the learning data presented by the user, as long as the amount of calculation required for learning based on the learning data presented by the user does not exceed the allowable upper limit.

[0048] As described above, the connection unit 102 does not directly supply the second feature FT2 to the decoder unit 103, but converts the second feature FT2 into a third feature FT3 and supplies the third feature FT3 to the decoder unit 103. By tuning the feature (converting the second feature FT2 into the third feature FT3), the decoder unit 103 can acquire a feature suitable for upsampling processing of the feature in the decoder unit 103 as the feature (third feature FT3) supplied from the connection unit 102 to the decoder unit 103. As described above, the parameters of the connection unit 102 are adjusted based on training data presented by the user. Therefore, even if the parameters of the encoder unit 101 are fixed by training at the time of shipment by the vendor as described above, a certain level of accuracy can be ensured in the inspection of inspection image data using the trained machine learning model model1. More specifically, if the parameters of the encoder unit 101 are determined by learning performed by the vendor before shipping, and the parameters of the decoder unit 103 are adjusted using learning data presented by the user, the second feature FT2 supplied from the encoder unit 101 may become unsuitable for the decoder unit 103 whose parameters have been adjusted. By adjusting the parameters of the connection unit 102 in accordance with the adjustment of the parameters of the decoder unit 103, the connection unit 102 can output a third feature FT3 that is suitable for the decoder unit 103.

[0049] (Example of encoder structure) FIG. 5 is a diagram illustrating an exemplary structure of the encoder unit 101. The encoder unit 101 of the exemplary structure illustrated in FIG. 5 includes convolutional layers C1 to C13 and pooling layers P1 to P5. In each of the convolutional layers C1 to C13, a nonlinear transformation process using an activation function is performed after the convolution process. The activation function is typically a ReLU function. However, the activation function is not limited to the ReLU function and may be, for example, a sigmoid function, a softmax function, a leaky ReLU function, a GELU function, a hyperbolic tangent function, or the like. The pooling layer is typically a maximum value pooling layer. However, the pooling layer is not limited to a maximum value pooling layer and may be, for example, an average pooling layer.

[0050] 5 shows feature amounts F1 to F18 generated in the encoder unit 101 when input image data IN1 having 224 data pieces in the width direction, 224 data pieces in the height direction, and 3 channels (corresponding to the three colors R, G, and B) is input to the encoder unit 101. In the following description, the fact that the number of data pieces in the width direction is a, the number of data pieces in the height direction is b, and the number of channels is c will be expressed as [a×b]×c.

[0051] The convolutional layer C1 performs convolution processing and nonlinear transformation processing on the [224×224]×3 input image data IN1 to convert the [224×224]×3 input image data IN1 into [224×224]×64 feature values ​​F1. The [224×224]×64 feature values ​​F1 are supplied to the convolutional layer C2.

[0052] The convolutional layer C2 performs convolution processing and nonlinear transformation processing on the [224×224]×64 features F1 to convert the [224×224]×64 features F1 into [224×224]×64 features F2. The [224×224]×64 features F2 are supplied to the connection unit 102 (see FIG. 4) and the pooling layer P1.

[0053] The pooling layer P1 performs pooling on the [224 × 224] × 64 features F2, converting them into [112 × 112] × 128 features F3, which are then supplied to the convolutional layer C3.

[0054] The convolutional layer C3 performs convolution processing and nonlinear transformation processing on the [112×112]×128 feature F3 to convert the [112×112]×128 feature F3 into a [112×112]×128 feature F4. The [112×112]×128 feature F4 is supplied to the convolutional layer C4.

[0055] The convolutional layer C4 performs convolution processing and nonlinear transformation processing on the [112×112]×128 feature F4 to convert the [112×112]×128 feature F4 into the [112×112]×128 feature F5. The [112×112]×128 feature F5 is supplied to the connection unit 102 (see FIG. 4) and the pooling layer P2.

[0056] The pooling layer P2 performs pooling on the [112 × 112] × 128 features F5, converting them into [56 × 56] × 256 features F6, which are then supplied to the convolutional layer C5.

[0057] The convolutional layer C5 performs convolution processing and nonlinear transformation processing on the [56×56]×256 feature F6 to convert the [56×56]×256 feature F6 into a [56×56]×256 feature F7. The [56×56]×256 feature F7 is supplied to the convolutional layer C6.

[0058] The convolutional layer C6 performs convolution processing and nonlinear transformation processing on the [56×56]×256 feature F7 to convert the [56×56]×256 feature F7 into a [56×56]×256 feature F8. The [56×56]×256 feature F8 is supplied to the convolutional layer C7.

[0059] The convolutional layer C7 performs convolution processing and nonlinear transformation processing on the [56×56]×256 features F8 to convert them into [56×56]×256 features F9. The [56×56]×256 features F9 are supplied to the connection unit 102 (see FIG. 4) and the pooling layer P3.

[0060] The pooling layer P3 performs pooling on the [56×56]×256 features F9, converting them into [28×28]×512 features F10, which are then supplied to the convolutional layer C8.

[0061] The convolutional layer C8 performs convolution processing and nonlinear transformation processing on the [28×28]×512 feature F10 to convert the [28×28]×512 feature F10 into a [28×28]×512 feature F11. The [28×28]×512 feature F11 is supplied to the convolutional layer C9.

[0062] The convolutional layer C9 performs convolution processing and nonlinear transformation processing on the [28×28]×512 feature F11 to convert the [28×28]×512 feature F11 into a [28×28]×512 feature F12. The [28×28]×512 feature F12 is supplied to the convolutional layer C10.

[0063] The convolutional layer C10 performs convolution processing and nonlinear transformation processing on the [28×28]×512 features F12 to convert the [28×28]×512 features F12 into [28×28]×512 features F13. The [28×28]×512 features F13 are supplied to the connection unit 102 (see FIG. 4) and the pooling layer P4.

[0064] The pooling layer P4 performs pooling on the [28×28]×512 features F13 to convert them into [14×14]×512 features F14. The [14×14]×512 features F14 are supplied to the convolutional layer C11.

[0065] The convolutional layer C11 performs convolution processing and nonlinear transformation processing on the [14×14]×512 feature F14 to convert the [14×14]×512 feature F14 into a [14×14]×512 feature F15. The [14×14]×512 feature F15 is supplied to the convolutional layer C12.

[0066] The convolutional layer C12 performs convolution processing and nonlinear transformation processing on the [14×14]×512 feature F15 to convert the [14×14]×512 feature F15 into a [14×14]×512 feature F16. The [14×14]×512 feature F16 is supplied to the convolutional layer C13.

[0067] The convolutional layer C13 performs convolution processing and nonlinear transformation processing on the [14×14]×512 feature F16 to convert the [14×14]×512 feature F16 into a [14×14]×512 feature F17. The [14×14]×512 feature F17 is supplied to the decoder unit 103 (see FIG. 4) and the pooling layer P5.

[0068] The pooling layer P4 performs pooling processing on the [14×14]×512 feature F17, converting the [14×14]×512 feature F17 into a [7×7]×512 feature F18. In this embodiment, the feature F17 is supplied to the decoder unit 103, but a configuration in which the feature F18 is supplied may also be used. Alternatively, a configuration in which a feature obtained from any layer included in the encoder unit 101 is supplied to the decoder unit 103 may also be used.

[0069] The feature F17 of [14×14]×512 is the first feature FT1 described above.

[0070] The feature F2 of [224 × 224] × 64, the feature F5 of [112 × 112] × 128, the feature F9 of [56 × 56] × 256, and the feature F13 of [28 × 28] × 512 are the second feature FT2 mentioned above. Note that the number of second feature FT2 is not limited to four, and may be at least one.

[0071] The smaller the size of a feature, the less spatial information there is. Therefore, the second feature FT2, which is larger in size than the first feature FT1, contains spatial information that is not included in the first feature FT1.

[0072] (Example of connection structure) Fig. 6 is a diagram showing an example structure of the connection unit 102. The connection unit 102 in the example structure shown in Fig. 6 includes convolutional layers C21 to C24. Convolutional processing is performed in each of the convolutional layers C21 to C24.

[0073] The convolutional layer C21 performs convolution processing on the [224×224]×64 feature F2 to convert the [224×224]×64 feature F2 into a [224×224]×64 feature F21. The [224×224]×64 feature F21 is supplied to the decoder unit 103 (see FIG. 4).

[0074] The convolutional layer C22 performs convolution processing on the [112×112]×128 feature F5 to convert the [112×112]×128 feature F5 into a [112×112]×64 feature F22. The [112×112]×64 feature F22 is supplied to the decoder unit 103 (see FIG. 4).

[0075] The convolutional layer C23 performs convolution processing on the [56×56]×256 feature F9 to convert the [56×56]×256 feature F9 into a [56×56]×64 feature F23. The [56×56]×64 feature F23 is supplied to the decoder unit 103 (see FIG. 4).

[0076] The convolutional layer C24 performs convolution processing on the [28×28]×512 features F13 to convert the [28×28]×512 features F13 into [28×28]×64 features F24. The [28×28]×64 features F24 are supplied to the decoder unit 103 (see FIG. 4).

[0077] The [224 x 224] x 64 feature F21, the [112 x 112] x 64 feature F22, the [56 x 56] x 64 feature F23, and the [28 x 28] x 64 feature F24 are the third feature FT3 mentioned above. Note that the number of third feature FT3 is not limited to four, as long as there is at least one. However, the number of second feature FT2 and third feature FT3 is the same.

[0078] (Example of decoder structure) FIG. 7 is a diagram illustrating an example structure of the decoder unit 103. The decoder unit 103 of the example structure illustrated in FIG. 7 includes convolutional layers C31 to C36, deconvolutional layers D31 to D34, and connected layers A31 to A34. Convolutional processing is performed in each of the convolutional layers C31 to C36. Deconvolutional processing (the inverse of convolution processing) is performed in the deconvolutional layers D31 to D34. Note that a layer that combines upsampling processing and convolution processing may be used instead of the deconvolutional layers. Connecting processing is performed in the connected layers A31 to A34. Note that the convolutional layers C31 to C35 may be omitted.

[0079] The convolutional layer C31 performs convolution processing on the [14×14]×512 feature F17 to convert the [14×14]×512 feature F17 into [14×14]×64 feature F31. The [14×14]×64 feature F31 is supplied to the deconvolutional layer D31.

[0080] The deconvolution layer D31 performs deconvolution processing on the [14×14]×64 features F31 to convert them into [28×28]×64 features F32. The [28×28]×64 features F32 are supplied to the combination layer A31.

[0081] The combined layer A31 adds the [28×28]×64 feature F24 to the [28×28]×64 feature F32, converting the [28×28]×64 feature F32 into the [28×28]×64 feature F33. The [28×28]×64 feature F33 is supplied to the convolutional layer C32.

[0082] The aforementioned addition is an example of a combining process, which adds together corresponding data of two feature amounts (data at the same widthwise position, same heightwise position, and same channelwise position). The two feature amounts to be added have the same widthwise size, heightwise size, and number of channels.

[0083] Another example of a combining process is concatenation. Concatenation is a process in which the width and height positions of two feature amounts are aligned and the two feature amounts are arranged along the channel direction. The two feature amounts to be concatenated have the same width and height sizes, and the number of channels may be the same or different.

[0084] By adopting addition rather than concatenation as the combining process, it is possible to reduce the number of feature channels supplied to the layer provided after the combined layer, and therefore the amount of calculation in the layer provided after the combined layer.In addition, adopting concatenation as the combining process makes it possible to combine features with different numbers of channels, thereby improving design flexibility.

[0085] The convolutional layer C32 performs convolution processing on the [28×28]×64 features F33 to convert the [28×28]×64 features F33 into [28×28]×64 features F34. The [28×28]×64 features F34 are supplied to the deconvolutional layer D32.

[0086] The deconvolution layer D32 performs deconvolution processing on the [28×28]×64 features F34 to convert them into [56×56]×64 features F35. The [56×56]×64 features F35 are supplied to the combination layer A32.

[0087] The combined layer A32 adds the [56×56]×64 feature F23 to the [56×56]×64 feature F35, converting it into the [56×56]×64 feature F36. The [56×56]×64 feature F36 is supplied to the convolutional layer C33.

[0088] The convolutional layer C33 performs convolution processing on the [56×56]×64 feature F36 to convert the [56×56]×64 feature F36 into a [56×56]×64 feature F37. The [56×56]×64 feature F37 is supplied to the deconvolutional layer D33.

[0089] The deconvolution layer D33 performs deconvolution processing on the [56×56]×64 features F37 to convert them into [112×112]×64 features F38. The [112×112]×64 features F38 are supplied to the combination layer A33.

[0090] The combined layer A33 adds the [112×112]×64 feature F22 to the [112×112]×64 feature F38, converting it into a [112×112]×64 feature F39. The [112×112]×64 feature F39 is supplied to the convolutional layer C34.

[0091] The convolutional layer C34 performs convolution processing on the [112×112]×64 feature F39 to convert the [112×112]×64 feature F39 into a [112×112]×64 feature F40. The [112×112]×64 feature F40 is supplied to the deconvolutional layer D34.

[0092] The deconvolution layer D34 performs deconvolution processing on the [112×112]×64 features F40 to convert the [112×112]×64 features F40 into [224×224]×64 features F41. The [224×224]×64 features F41 are supplied to the combination layer A34.

[0093] The combined layer A34 adds the [224×224]×64 feature F21 to the [224×224]×64 feature F41, converting the [224×224]×64 feature F41 into the [224×224]×64 feature F42. The [224×224]×64 feature F42 is supplied to the convolutional layer C35.

[0094] The convolutional layer C35 performs convolution processing on the [224×224]×64 feature F42 to convert the [224×224]×64 feature F42 into a [224×224]×64 feature F43. The [224×224]×64 feature F43 is supplied to the convolutional layer C36.

[0095] The convolution layer C36 performs convolution processing on the [224×224]×64 feature F43 to convert the [224×224]×64 feature F43 into output image data OUT1.

[0096] (Number of dimensions of input image data) The input image data IN1 may be image data obtained by one imaging session, or may be a collection of image data obtained by multiple imaging sessions.

[0097] When the input image data IN1 is image data obtained by a single imaging operation, the input image data IN1 becomes three-dimensional image data, and the features generated by the encoder unit 101, the connection unit 102, and the decoder unit 103 become three-dimensional features, and the output image data OUT1 becomes three-dimensional image data.

[0098] If the input image data IN1 is a compilation of image data obtained by capturing images multiple times, the input image data IN1 becomes four-dimensional image data, the features generated by the encoder unit 101, the connection unit 102, and the decoder unit 103 each become four-dimensional features, and the output image data OUT1 becomes four-dimensional image data.

[0099] (annotation) The processor 13a performs annotation processing for the machine learning model model 1 on the training image data, which is image data of a defective product, to generate training data for the machine learning model model 1. The training image data may be image data captured by the camera module 14, or may be image data captured by an imaging unit other than the camera module 14 and stored in the storage device 19.

[0100] Fig. 8 is a diagram showing a schematic flow of annotation processing for the machine learning model model 1. When learning image data to be annotated is selected by a user operation on the keyboard 51 or the mouse 52, the flow shown in Fig. 8 starts.

[0101] In step S1, the processor 13a displays an image (learning image) based on the learning image data to be annotated on the display device 4. At this time, as shown in Fig. 9, on the display screen 42 of the display device 4, icons 105 to 107 for annotation processing, an add button 108, a delete button 109, an OK button 110, a cancel button 111, a pointer 112, and a message are displayed together with the learning image 104.

[0102] In step S2 following step S1, the processor 13a designates an abnormal region. The pointer 112 can be moved to any position on the display screen 42 by a user operation on the mouse 52. When the user clicks on the mouse 52 while the pointer 112 is overlapping with a specific icon or button, the specific icon or button is selected. Note that any one of the icons 105 to 107 may be selected from the beginning as a default setting.

[0103] When the Add button 108 is selected, it becomes possible to specify one abnormal area. When the Delete button 109 is selected, it becomes possible to delete an abnormal area that was mistakenly specified by selecting an abnormal area that was mistakenly specified. When the OK button 110 is selected, the specification process in step S2 is completed, and the process proceeds to step S3. When the Cancel button 111 is selected, the specification process in step S2 is invalidated, and the flow shown in FIG. 8 is forcibly terminated without proceeding to step S3.

[0104] When the icon 105 is selected as shown in FIG. 10, the processor 13a specifies an abnormal area in the learning image 104 with a free-form paint image 113 formed in response to a user operation (a user dragging operation on the mouse 52).

[0105] 11, when the icon 106 is selected, the processor 13a specifies an abnormal region in the training image 104 as a region 115 whose boundary is a free curve 114 formed in response to a user operation (a user dragging operation on the mouse 52). If the free curve 114 is closed, the region 115 is the region surrounded by the free curve 114. If the free curve 114 is open, the region 115 is the region surrounded by the free curve 114 and a line segment connecting the start point and end point of the free curve 114.

[0106] Instead of or in addition to the icon 106, an icon may be provided for designating an abnormal region in the training image 104 as a region whose boundary is a geometric curve formed in response to a user operation. Similarly, instead of or in addition to the icon 106, an icon may be provided for designating an abnormal region in the training image 104 as a region whose boundary is a Bezier curve formed in response to a user operation.

[0107] 12, when icon 107 is selected, processor 13a specifies an abnormal region in learning image 104 as region 117 whose boundary is a polygonal line 116 formed in response to a user operation (a user click operation on mouse 52). If polygonal line 116 is closed, region 117 is the region surrounded by polygonal line 116. If polygonal line 116 is open, region 117 is the region surrounded by polygonal line 116 and a line segment connecting the start point and end point of polygonal line 116.

[0108] As described above, specifying an abnormal area in the training image 104 as an area whose boundary is a line formed in response to a user operation or as a free-form painted image formed in response to a user operation enables more precise annotation than specifying an abnormal area with a predetermined shape such as a rectangle. Because the machine learning model model1 is a segmentation model that classifies image data on a pixel-by-pixel basis, precise annotation is particularly useful.

[0109] In step S3, the processor 13a generates learning data by adding the instruction content (teaching content) in step S2 as metadata to the learning image data, and stores the generated learning data in the storage device 19. When the processing in step S3 is completed, the flow shown in FIG. 8 ends.

[0110] (Display of test results) During the operation stage of the appearance inspection device 1, the appearance inspection device 1 performs a segmentation inspection of the inspection image data using the trained machine learning model model 1. Fig. 13 is a diagram showing a schematic flow of an inspection result display process that displays the results of the segmentation inspection of the inspection image data. When the inspection image data is input as input image data IN1 to the trained machine learning model model 1 and output image data OUT1 is output from the trained machine learning model model 1, completing the segmentation inspection of the inspection image data, the flow shown in Fig. 13 starts.

[0111] In step S11, the processor 13a generates an inspection result of the inspection image data based on the output image data OUT1. For example, the inspection result of the inspection image data based on the output image data OUT1 may be "NG" if the number of pixels in the abnormal region of the output image data OUT1 is equal to or greater than a threshold, and may be "OK" if the number of pixels in the abnormal region of the output image data OUT1 is less than the threshold.

[0112] In step S12 following step S11, the processor 13a causes the display device 4 to display the output image data OUT1 together with the inspection results generated in step S11. At this time, as shown in FIG. 14, an image 119 based on the output image data OUT1 is also displayed on the display screen 42 of the display device 4 together with the inspection results 118 generated in step S11. Black areas in the image 119 based on the output image data OUT1 are abnormal areas, and non-black areas in the image 119 based on the output image data OUT1 are non-abnormal areas. When the processing of step S12 is completed, the flow shown in FIG. 13 ends.

[0113] Inspection by the appearance inspection device 1 is a substitute for inspection that was originally performed visually by a person. Therefore, if a discrepancy occurs between the inspection result ("OK" or "NG") by the appearance inspection device 1 and the inspection result ("OK" or "NG") by a person visually, it is preferable to be able to immediately check the image 119 based on the output image data OUT1, which is the cause of the discrepancy. Therefore, as in the display process in step S12, it is preferable that the image 119 based on the output image data OUT1 is displayed together with the inspection result 118 generated in step S11. In other words, it is preferable that the inspection result 118 generated in step S11 and the image 119 based on the output image data OUT1 are displayed on a single screen.

[0114] As described above, the connection unit 102 converts the second feature value FT2 into the third feature value FT3 and supplies the third feature value FT3 to the decoder unit 103, thereby ensuring a certain level of accuracy in the inspection of the inspection image data using the trained machine learning model model 1. Therefore, the appearance inspection device 1 can reduce discrepancies between the inspection results of the appearance inspection device 1 and the inspection results obtained by human visual inspection.

[0115] (simple annotation) In the annotation process described above, ideally, the user is required to specify all abnormal regions. In the annotation process described above, the more abnormal regions are omitted, the lower the accuracy of the annotation. However, if the user is forced to perform annotation work that minimizes the omission of abnormal regions, i.e., precise annotation work, the difficulty of the user's operation increases.

[0116] Therefore, it is desirable to equip the appearance inspection device 1 with a function (a function for performing simple annotation processing) that outputs detection results that are closer to the actual defective part than the user's rough specification, even if the user roughly specifies the abnormal area. By equipping the appearance inspection device 1 with a function for performing simple annotation processing, user operation of the appearance inspection device 1 becomes simple.

[0117] Conventionally, a simple annotation process is known, as disclosed in Non-Patent Document 2. However, the simple annotation process disclosed in Non-Patent Document 2 has a problem in that it requires not only rough designation of an abnormal region but also rough designation of regions other than the abnormal region.

[0118] A simple annotation process that can solve this problem, that is, a simple annotation process that does not require rough designation of areas other than the abnormal area, will be described below.

[0119] Processor 13a receives a simple annotation specifying a part of a defective portion (abnormal region) in a defective product image and a non-defective product image, and performs simple annotation processing for machine learning model model 1 to generate learning data for machine learning model model 1. The data on the defective product image and the data on the non-defective product image may each be imaging data captured by camera module 14, or may be imaging data captured by an imaging unit other than camera module 14 and stored in storage device 19.

[0120] The processor 13a can display an annotation processing icon for starting the above-mentioned annotation processing and an annotation processing icon for starting the simplified annotation processing on the display screen 42 of the display device 4. When the annotation processing icon is selected by a user operation on the keyboard 51 or the mouse 52, and further when the learning image data (defective product image data) to be annotated is selected, the flow shown in Fig. 8 is started. On the other hand, when the simplified annotation processing icon is selected by a user operation on the keyboard 51 or the mouse 52, the flow shown in Fig. 15 is started.

[0121] FIG. 15 is a diagram showing an outline flow of the simple annotation process for the machine learning model model1.

[0122] In step S21, the processor 13a accepts non-defective product image data and defective product image data selected by a user operation. The non-defective product image data and defective product image data selected by a user operation are mutually related image data, that is, image data of the same type of workpiece. For example, the processor 13a accepts data of a non-defective product image 201 shown in FIG. 16 and data of a defective product image 202 shown in FIG. 17.

[0123] In step S22 following step S21, the processor 13a displays the defective product image 202 to be subjected to simple annotation on the display device 4. At this time, as shown in Fig. 18, on the display screen 42 of the display device 4, icons 203 to 204 for simple annotation processing, an add button 205, a delete button 206, an OK button 207, a cancel button 208, a pointer 209, and a message are also displayed together with the defective product image 202.

[0124] In step S23 following step S22, the processor 13a designates a part of the defective portion. The pointer 209 can be moved to any position on the display screen 42 by the user's operation of the mouse 52. When the user clicks the mouse 52 while the pointer 209 is overlapping with a specific icon or button, the specific icon or button is selected. Note that either the icon 203 or 204 may be selected from the beginning as a default setting.

[0125] When the Add button 205 is selected, it becomes possible to specify a defective part. When the Delete button 206 is selected, it becomes possible to delete a defective part that was specified by mistake by selecting the defective part that was specified by mistake. When the OK button 207 is selected, the specification process in step S23 is completed, and the process proceeds to step S24. When the Cancel button 208 is selected, the specification process in step S23 is invalidated, and the flow shown in FIG. 15 is forcibly terminated without proceeding to the process in step S23.

[0126] 19, when the icon 205 is selected, the processor 13a specifies the defective part in the defective product image 202 by a free curve 210 formed in response to a user operation (a user dragging the mouse 52). The free curve 210 may be either a closed curve or an open curve.

[0127] Instead of or in addition to the icon 205, an icon may be provided for specifying a defective portion of the defective product image 202 using a geometric curve formed in response to a user operation. Similarly, instead of or in addition to the icon 205, an icon may be provided for specifying a defective portion of the defective product image 202 using a Bezier curve formed in response to a user operation.

[0128] When the icon 204 is selected as shown in FIG. 20, the processor 13a designates the defective portion in the defective product image 202 with a dot 211 formed in response to a user operation (a user click operation on the mouse 52).

[0129] In step S24 following step S23, the processor 13a generates differential image data by calculating the difference between the data of the non-defective product image 201 and the data of the defective product image 202. Since the positions, brightness, etc. of the workpieces in the non-defective product image 201 and the defective product image 202 do not completely match, the processor 13a may calculate the difference using a known method such as matching based on normalized correlation.

[0130] In step S25 following step S24, the processor 13a identifies a group of pixels where the difference in the differential image data generated in step S24 is equal to or greater than a certain value as a candidate for an abnormal area, and identifies a group of pixels where the difference in the differential image data generated in step S24 is less than a certain value as a candidate for a background area (an area other than an abnormal area).

[0131] In step S26 following step S25, the processor 13a identifies an abnormal area based on the candidate abnormal area identified in step S24 and part of the defective portion designated in step S23.

[0132] The processing content of step S26 may be, for example, the following first processing or second processing.

[0133] In the first process, the processor 13a identifies an abnormal area based on the boundary between the candidate abnormal area and the candidate background area identified in step S25 and the part of the defective area specified in step S23. For example, the processor 13a calculates the midpoint between each point (each pixel) constituting the boundary and the part of the defective area specified in step S23 that is closest to each point, calculates a midpoint that is a collection of the midpoints, and identifies the part surrounded by the midpoint as an abnormal area. The ratio of the distance between the point constituting the boundary and the midpoint to the distance between the part of the defective area specified in step S23 that is closest to the point constituting the boundary and the midpoint may be fixed or may be arbitrarily changeable by a setting change performed by a user operation.

[0134] In the second process, the processor 13a identifies an abnormal region based on the feature amounts of each microregion constituting the candidate abnormal region identified in step S25 and the feature amounts of each microregion constituting part of the defective portion specified in step S23. For example, the processor 13a extracts feature amounts similar to the feature amounts of each microregion constituting part of the defective portion specified in step S23 from the feature amounts of each microregion constituting the candidate abnormal region identified in step S25, and identifies a collection of microregions having the extracted feature amounts as an abnormal region. The processor 13a determines the similarity of the feature amounts based on the direction and magnitude of the feature vector.

[0135] In either the first process or the second process described above, the processor 13a identifies an area that is larger than the part of the defective portion designated in step S23 as the abnormal area. Then, the processor 13a generates learning data by adding the abnormal area (instruction content) as metadata to the data of the defective product image 202, and stores the generated learning data in the storage device 19. When the process of step S26 is completed, the flow shown in FIG. 15 ends.

[0136] Since an area extending beyond the portion of the defective area specified in step S23 is identified as an abnormal area, the machine learning model model1 trained using the training data generated by executing the flow shown in Figure 15 can output a detection result that is closer to the actual defective area than the simple annotation (the portion of the defective area specified in step S23).

[0137] <Other> In addition to the above-described embodiments, the various technical features disclosed in this specification can be modified in various ways without departing from the spirit of the technical creation.

[0138] In other words, the above-described embodiments should be considered to be illustrative in all respects and not restrictive. The technical scope of the present invention is defined by the claims, and it should be understood that all modifications within the meaning and scope of the claims are included.

[0139] In the above embodiment, the visual inspection device 1, i.e., the image inspection device that inspects inspection image data using a model that can reduce the amount of calculation required for learning at the work production site, is equipped with a function for performing simple annotation processing that does not require rough designation of areas other than the abnormal area. However, in order to realize an image inspection device that can solve the problem that not only rough designation of the abnormal area but also rough designation of the areas other than the abnormal area is required in the simple annotation processing, for example, an image inspection device having a U-Net may be equipped with a function for performing simple annotation processing that does not require rough designation of areas other than the abnormal area.

[0140] The hardware configuration of the appearance inspection device 1 may be other than the configuration (controller type) shown in the above embodiment. FIG. 21 is a diagram showing another hardware configuration (smart camera type) of the appearance inspection device 1. The appearance inspection device 1 in this figure includes a smart camera 6 instead of the previously described control unit 2 and imaging unit 3. The personal computer 5 may also include a display 53 in addition to the previously described keyboard 51 and mouse 52. In this figure, a control unit 54 is clearly shown as a component of the personal computer 5.

[0141] The personal computer 5 can be understood as an example of a UI device that is connected to the smart camera 6 and accepts user operations. For example, the personal computer 5 accepts user operations to set up and operate the smart camera 6. That is, in the smart camera type appearance inspection device 1, among the various functions that were previously handled by the controller type control unit 2, the setting function of the smart camera 6 has been transferred to the personal computer 5.

[0142] The display 53 displays inspected images acquired by the smart camera 6 and displays a GUI for configuring various settings of the smart camera 6. The inspected images can be understood as workpiece images inspected by the smart camera 6.

[0143] The control unit 54 displays the inspected image and GUI on the display 53. The control unit 54 can also accept user operations via the keyboard 51 and mouse 52. Furthermore, the control unit 54 also has a function of setting and operating the smart camera 6 in response to user operations.

[0144] The smart camera 6 receives setting and operation instructions from the personal computer 5. The smart camera 6 integrates the aforementioned control unit 2 and imaging unit 3. That is, the smart camera 6 includes the aforementioned main board 13, camera module 14, lighting module 15, connector board 16, communication board 17, power supply board 18, and storage device 19.

[0145] For example, the processor 13a mounted on the main board 13 functions as an inspection unit that inspects the workpiece image. The inspection of the workpiece image may be performed based on setting information set in response to a setting operation received by the personal computer 5. The setting information may be various parameters of a setting tool.

[0146] The workpiece image inspected by the processor 13a, i.e., the inspected image, is stored in the memory 13b, but may also be written to the storage device 19. In this way, the memory 13b or the storage device 19 functions as a storage unit that stores the inspected image.

[0147] The internal configuration of the smart camera 6 is merely an example. For example, the boards may be integrated or divided as desired.

[0148] FIG. 22 is a diagram showing how the removable memory 7 is used in the smart camera-type appearance inspection device 1. As shown in this diagram, the removable memory 7 may be attached to and detached from the memory 13b of the smart camera 6 as a storage unit for storing inspected images and their inference results. The removable memory 7 can be attached and detached not only to the smart camera 6 but also to the personal computer 5. The personal computer 5 can specify the removable memory 7 attached to the smart camera 6 as the storage destination for the inspected images and their inference results. An SD memory card, for example, can be suitably used as the removable memory 7. [Explanation of symbols]

[0149] 1. Visual inspection equipment (image inspection equipment) 2. Control Unit 3 Imaging unit 4 Display device (display section) 41 Touch Panel 42 Display screen 5. Personal Computers 51 keyboard 52 Mouse 6. Smart Camera 7. Removable Memory 13 Main board 13a Processor (control unit) 13b Memory 14 Camera module (imaging unit) 141 AF motor 142 Imaging board 143 CMOS sensor 15 Lighting module (lighting section) 151 LED (light emitting diode) 152 LED drivers 16 Connector board 161 Power Interface 162 I / O interfaces 17 Communication board 18 Power supply board 181 AF motor driver 19 Storage device (storage unit) 80 Program Files 90 Storage medium 101 Encoder section 102 Connection 103 Decoder section 104 training images 105~107, 203, 204 Icons 108, 205 Add button 109, 206 Delete button 110, 207 OK button 111, 208 Cancel button 112, 209 pointer 113 Paint Images 114 Free curve 115, 117 area 116 Line 118 Test Results 119 Image based on output image data 201 good quality images 202 Defective Product Image A31~A34 Bonding layer C1~C13, C21~C24, C31~C36 convolution layers D31~D34 deconvolution layers P1~P5 pooling layers model1~model3 machine learning models

Claims

1. An image inspection device that inspects inspection image data using a model in which parameters are updated by machine learning based on training image data presented by a user, An imaging unit; a control unit that executes the model to which the inspection image data obtained by imaging by the imaging unit is input, The model outputs image data that can distinguish areas of the image area of ​​the input inspection image data that belong to the first class from areas that do not belong to the first class, based on a label indicating a first class assigned to the training image data, an encoder unit having a plurality of intermediate layers including a convolutional layer and configured to extract a first feature amount from the inspection image data; a connection unit that receives a second feature amount different from the first feature amount from at least one of the plurality of intermediate layers, converts the second feature amount into a third feature amount, and supplies the third feature amount; a decoder unit that upsamples the first feature extracted by the encoder unit while using the third feature supplied by the connection unit, An image inspection device, wherein the control unit updates parameters of the connection unit and parameters of the decoder unit when performing machine learning of the model based on the training image data.

2. The image inspection device according to claim 1 , wherein the second feature amount has spatial information different from that of the first feature amount.

3. 2. The image inspection device according to claim 1, wherein the decoder unit generates a fourth feature having the same size and the same number of channels as the third feature, and generates a fifth feature having the same size and the same number of channels as the third feature by adding the third feature and the fourth feature together.

4. 2. The image inspection device according to claim 1, wherein the decoder unit generates a fourth feature having the same size as the third feature, and concatenates the third feature and the fourth feature to generate a fifth feature having a larger number of channels than the third feature.

5. The image inspection device according to claim 1 , wherein the connection unit receives a plurality of the second feature amounts each having a different size.

6. A display unit is provided, the control unit generates an inspection result of the inspection image data based on an output of the model; The image inspection device according to claim 1 , wherein the display unit is capable of displaying an output of the model together with an inspection result of the inspection image data.

7. a reception unit that receives user operations; 2. The image inspection device according to claim 1, further comprising a designation unit that designates an area corresponding to the first class for an image based on the training image data as an area whose boundary is a line formed in response to the user operation.

8. The image inspection device according to claim 7 , wherein the line formed in response to the user's operation is a geometric curve, a Bezier curve, or a free curve.

9. a reception unit that receives user operations; 2. The image inspection device according to claim 1, further comprising a designation unit that designates an area corresponding to the first class in an image based on the training image data by using a free-form paint image formed in response to the user's operation.

10. An imaging unit; an execution control unit that executes the model; a setting control unit that sets the model, The model is An encoder unit; A decoder unit; a connection between the encoder unit and the decoder unit, The setting control unit Defective product images and receiving a simple annotation specifying a part of a defective portion of the defective product image; Accepting good quality images, updating the parameters of the connection unit and the decoder unit based on the simple annotations and the non-defective product image; The model in which the parameters of the connection section and the parameters of the decoder section have been updated outputs a detection result that is closer to the actual defect location than the simple annotation.

11. An imaging unit; an execution control unit that executes the model; a setting control unit that sets the model, the model is a model that divides an input image into a first region and a second region; The setting control unit Defective product images and receiving a simple annotation that specifies a part of the first region of the defective product image; Accepting good quality images, The image inspection device updates parameters of the model based on the simple annotations and the image of the non-defective product.

12. a reception unit that receives a user operation, The setting control unit The image inspection device according to claim 10 or 11, wherein the simple annotation is accepted by dots formed on the defective product image in response to the user operation.

13. a reception unit that receives a user operation, The setting control unit The image inspection device according to claim 10 or 11, wherein the simplified annotation is accepted by a curve formed on the defective product image in response to the user operation.

Images