Appearance inspection apparatus and appearance inspection method

Abstract

To stably exert a high ability to detect a defective image having an unknown defect, while shorten tact time during operation by using a machine learning network having learned both a non-defective image and a defective image.SOLUTION: A processor executes first learning processing of applying noise to a non-defective image and causes a machine learning network to learn the non-defective image; second learning processing of causing the machine learning network to learn a defective image; and detection processing of detecting both an unknown defect having different characteristics from the non-defective image and a known defect having characteristics designated as a defective part by inputting a workpiece image to the machine learning network with a parameter adjusted through the first learning processing and the second learning processing.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an appearance inspection apparatus and an appearance inspection method for inspecting the appearance of a workpiece.

Background Art

[0002] For example, Patent Document 1 discloses a processing apparatus that uses machine learning by a computer to determine whether a workpiece is a good product or a defective product. The processing apparatus of Patent Document 1 performs supervised machine learning on good product data to generate a good product learning model, and performs supervised machine learning on defective product data to generate a defective product learning model. Then, after inputting the data of the workpiece to be determined, it is configured to be able to determine whether the workpiece is a good product or a defective product by the good product learning model and the defective product learning model. Such an apparatus is also called a workpiece appearance inspection apparatus.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the production site of workpieces, defective products rarely occur. Therefore, it is easy to collect a large amount of good product data, while it is difficult to collect defective product data. Therefore, good product learning targeting good product data is assumed as a solution. However, in the case of a machine learning network trained only with good product data, the ability to detect defective products is insufficient, and in inspections with high difficulty, the performance is inferior to defective product learning.

[0005] Even if a large number of defective product data can be collected, a machine learning network trained with defective product data will exhibit high detection ability for the defective data taught during training, but the detection of unknown defective data will be unstable and prone to detection omissions.

[0006] Therefore, as disclosed in Patent Document 1, a non-defective product learning model trained with non-defective product data and a defective product learning model trained with defective product data are generated, and at the time of operation, inference processing is performed on the work image using the non-defective product learning model and the defective product learning model respectively, and a method of synthesizing the obtained inference results can be considered.

[0007] However, when using the non-defective product learning model and the defective product learning model, tuning during learning is required for each of the two models, and tuning is also required for the process of synthesizing the two inference results. As a result, the learning difficulty increases and the labor during learning also increases.

[0008] Also, when performing appearance inspection of a work using the non-defective product learning model and the defective product learning model, since inference processing is performed for each of the two models, the processing time becomes long, and moreover, time is also required for the process of synthesizing the two inference results, and an increase in tact may become a problem.

[0009] Furthermore, originally, since non-defective product learning and defective product learning have different natures, it may be difficult to configure a logic that absorbs the difference in nature between non-defective product learning and defective product learning.

[0010] The present disclosure has been made in view of such points, and the object is to use a machine learning network that has learned both non-defective product images and defective product images to stably exhibit high detection ability for defective product images having unknown defects while shortening the tact during operation.

Means for Solving the Problem

[0011] In order to achieve the above object, in one aspect of the present disclosure, it is possible to assume an appearance inspection apparatus including a storage unit that stores a machine learning network, and a processor that inputs a work image obtained by photographing a work to be inspected into the machine learning network and determines whether the work is good or defective based on the input work image. The processor is configured to be capable of executing a first learning process of adjusting parameters of the machine learning network so that a part corresponding to the noise is extracted by adding noise to a good product image corresponding to a good product and causing the machine learning network to learn. Further, the processor is configured to be capable of executing a second learning process of adjusting parameters of the machine learning network so that the defective part specified in advance by the user is extracted on the defective product image by causing the machine learning network to learn a defective product image corresponding to a defective product having a defective part. Furthermore, the processor is configured to be capable of executing a detection process for both an unknown defect having characteristics different from those of the good product image and a known defect having characteristics specified as the defective part by inputting the work image into the machine learning network whose parameters have been adjusted by the first learning process and the second learning process.

[0012] According to this configuration, instead of performing learning of the machine learning network only with defective product images, learning of the machine learning network is also performed using good product images with noise added. Therefore, a machine learning network with high detection ability not only for known defects included in the defective product images used for learning but also for unknown defects is generated. As a result, compared with the case of performing inference processing with a good product learning model and a defective product learning model as in the prior art, the learning difficulty is reduced, and the labor required for learning can be reduced. In addition, since the synthesis process of the inference results becomes unnecessary during appearance inspection, the tact time during operation is shortened.

[0013] A processor according to another aspect, at the time of setting the appearance inspection device, inputs an input image with noise added to the good product image into the machine learning network, and adjusts the parameters of the machine learning network so that an abnormality degree map indicating the position of the noise becomes a first output image, and the first learning process, and inputs a defective product image that has received a designation of a defective part by the user into the machine learning network, and adjusts the parameters of the machine learning network so that an abnormality degree map indicating the position of the defective part designated by the user becomes a second output image, and the second learning process can be executed.

[0014] According to this configuration, in both good product learning and defective product learning, defective parts can be directly extracted as abnormalities and an abnormality degree map can be output.

[0015] A processor according to another aspect, at the time of the first learning process, generates a target abnormality degree map image based on the difference in pixel values between corresponding portions of the good product image with noise added and the good product image without noise added, and since the parameters of the machine learning network can be adjusted so that the first output image matches the target abnormality degree map image, the learning effect using the good product image with noise added is improved.

[0016] A processor according to another aspect randomly adds a plurality of the noises having a predetermined size or more to the good product image, so that while suppressing the erroneous detection of a portion such as near the edge of the work as a defective part, the detection performance of fine defective parts is improved.

[0017] A processor according to another aspect, when the good product image is a color image, adds color noise to the good product image, so that the detection performance for color abnormalities is improved.

[0018] The processor according to another aspect increases the amount of noise added to the good product image as the good product image is larger, so that an appropriate amount of noise suitable for the size of the good product image can be automatically added, thereby enhancing the learning effect while reducing the user's effort.

[0019] The processor according to another aspect adds a plurality of types of noise with different shapes to a single good product image, so that the detection performance of unknown defects with various shapes can be enhanced.

[0020] When the work image is an image of a defective product but is not determined to be a defective product as a result of the inspection process, the processor according to another aspect executes an update process of causing the machine learning network to learn a source dataset in which a defective product image with annotation information specifying a defective part is added, and updating the parameters of the machine learning network, so that detection omission of defective parts can be suppressed.

[0021] When the work image is determined to be a defective product even though it is an image of a good product as a result of the inspection process, the processor according to another aspect executes an update process of causing the machine learning network to learn a source dataset in which the image is added as a good product image, and updating the parameters of the machine learning network, so that over-detection can be suppressed.

Advantages of the Invention

[0022] As described above, by causing the machine learning network to learn both an image with noise added to the good product image and a defective product image, it is possible to detect both unknown defects having characteristics different from those of the good product image and known defects specified as defective parts. Thereby, while shortening the cycle time during operation, a high detection ability can be stably exhibited for defective product images having unknown defects.

Brief Description of the Drawings

[0023]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0025] FIG. 1 is a schematic diagram showing the configuration of an appearance inspection apparatus 1 according to an embodiment of the present invention. The appearance inspection apparatus 1 is an apparatus for determining the quality of a work image obtained by imaging a work, which is an inspection target such as various parts and products, and can be used at a production site such as a factory. Specifically, a machine learning network is constructed inside the appearance inspection apparatus 1, and this machine learning network is generated by learning a good product image corresponding to a good product and a defective product image corresponding to a defective product. A work image obtained by imaging a work to be inspected is input to the generated machine learning network, and the machine learning network can determine the quality of the work image.

[0026] The workpiece may be entirely subject to inspection, or only a part of the workpiece may be subject to inspection. Also, a single workpiece may contain multiple inspection targets. Further, the workpiece image may contain multiple workpieces.

[0027] The appearance inspection apparatus 1 includes a control unit 2 that serves as the apparatus main body, an imaging unit 3, a display device (display unit) 4, and a personal computer 5. The personal computer 5 is not essential and can be omitted. Instead of the display device 4, the personal computer 5 can also be used to display various information and images, or the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0028] In FIG. 1, as an example of the configuration example of the appearance inspection apparatus 1, the control unit 2, the imaging unit 3, the display device 4, and the personal computer 5 are described. However, any plurality of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Also, the control unit 2 can be divided into a plurality of units and a part of it can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into a plurality of units and a part of it can be incorporated into other units.

[0029] (Configuration of Imaging Unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that executes acquisition of a work image. The camera module 14 includes an AF motor 141 that drives an imaging optical system and an imaging substrate 142. The AF motor 141 is a part that automatically performs focusing by driving the lens of the imaging optical system, and can perform focusing by a method such as conventional contrast autofocus. The imaging substrate 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to be able to acquire a color image. Instead of the CMOS sensor 143, a light receiving element such as a CCD sensor can also be used.

[0030] The illumination module 15 includes an LED (light emitting diode) 151 as a light emitter that illuminates an imaging region including a work, and an LED driver 152 that controls the LED 151. The light emission timing, light emission time, and light emission amount of the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided as an external illumination unit separately from the imaging unit 3.

[0031] (Configuration of the display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel or an organic EL panel. The work image, user interface image, etc. output from the control unit 2 are displayed on the display device 4. Also, when the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used instead of the display device 4.

[0032] (Operating device) Examples of the operating device for the user to operate the appearance inspection device 1 include, for example, the keyboard 51 and the mouse 52 of the personal computer 5, but are not limited thereto, and any device configured to be able to receive various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operating device.

[0033] Operations by the user on the keyboard 51 and the mouse 52 can be detected by the control unit 2. Also, the touch panel 41 is a conventionally well-known touch-type operation panel equipped with, for example, a pressure-sensitive sensor, and the touch operation by the user can be detected by the control unit 2. The same applies when other pointing devices are used.

[0034] (Configuration of the control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. A processor 13a is provided on the main board 13. The processor 13a controls the operations of the connected boards and modules. For example, the processor 13a outputs an illumination control signal for controlling the lighting / extinguishing of the LED 151 to the LED driver 152 of the illumination module 15. The LED driver 152 switches the lighting / extinguishing of the LED 151 and adjusts the lighting time according to the illumination control signal from the processor 13a, and also adjusts the light amount and the like of the LED 151.

[0035] In addition, the processor 13a outputs an imaging control signal for controlling the CMOS sensor 143 to the imaging substrate 142 of the camera module 14. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13a and performs imaging while adjusting the exposure time to an arbitrary time. That is, the imaging unit 3 images within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13a. If there is a workpiece within the field of view, the workpiece will be imaged. However, if there are other objects within the field of view, they can also be imaged. For example, the appearance inspection device 1 can image a good product image corresponding to a good product and a defective product image corresponding to a defective product by the imaging unit 3 as learning images for the machine learning network. The learning images do not have to be the images captured by the imaging unit 3 and can be the images captured by other cameras or the like.

[0036] On the other hand, during the operation of the appearance inspection device, the imaging unit 3 can image the workpiece. In addition, the CMOS sensor 143 is configured to be able to output a live image, that is, the currently captured image, at any time with a short frame rate.

[0037] When the imaging by the CMOS sensor 143 is completed, the image signal output from the imaging unit 3 is input to and processed by the processor 13a on the main board 13 and is stored in the memory 13b on the main board 13. Details of the specific processing content by the processor 13a on the main board 13 will be described later. Note that a processing device such as an FPGA or a DSP may be provided on the main board 13. The processor 13a may be integrated with a processing device such as an FPGA or a DSP.

[0038] The connector board 16 is a part that receives power supply from the outside through a power connector (not shown) provided on the power interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board, module, etc. Specifically, it distributes power to the lighting module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 is provided with an AF motor driver 181. The AF motor driver 181 supplies driving power to the AF motor 141 of the camera module 14 to realize autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a of the main board 13.

[0039] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, communication between the main board 13 and an external control device (not shown), etc. Examples of the external control device include a programmable logic controller and the like. The communication may be wired or wireless, and any communication form can be realized by a conventionally well-known communication module.

[0040] The control unit 2 is provided with a storage device (storage unit) 19 composed of, for example, a solid state drive, a hard disk drive, etc. The storage device 19 stores a program file 80, a setting file, etc. (software) for enabling each control and process described later to be executed by the above hardware. The program file 80 and the setting file can be stored in a storage medium 90 such as an optical disk, for example, and the program file 80 and the setting file stored in this storage medium 90 can be installed in the control unit 2. The program file 80 may be downloaded from an external server using a communication line. Also, the storage device 19 can store, for example, the above image data, parameters for constructing the machine learning network of the appearance inspection device 1, etc.

[0041] That is, the processor 13a of the appearance inspection device 1 reads parameters and the like stored in the storage device 19 to construct a machine learning network, inputs a work image obtained by photographing a work to be inspected into the constructed machine learning network, and determines whether the work is good or bad based on the input work image. By using this appearance inspection device 1, an appearance inspection method for determining whether a work is good or bad based on a work image can be executed.

[0042] (Learning process of machine learning network) Next, based on the flowchart shown in FIG. 3, the learning process of the machine learning network performed when setting the appearance inspection device 1 will be described. The learning process of the machine learning network is to adjust the parameters of the machine learning network by inputting a good product image corresponding to a good product and a defective product image corresponding to a defective product into the machine learning network for learning.

[0043] In step SA1 after starting, an unlearned machine learning network is prepared. The unlearned machine learning network has, for example, initial values of parameters randomly determined. Alternatively, a machine learning network that has been learned to some extent for appearance inspection may be prepared in advance.

[0044] In step SA2, a good product image corresponding to a good product is acquired. The good product image acquired here is the good product image for learning shown in FIG. 4, and it may be a color image or a black-and-white image. For example, a good product image can be acquired by imaging a good product work with the camera module 14 of the imaging unit 3. Only one good product image may be acquired, or a plurality of good product images may be acquired by imaging different good products. The acquired good product images are stored, for example, in the storage device 19.

[0045] Also, in step SA3, a defective product image corresponding to the defective product is acquired. The defective product image acquired here is the learning defective product image shown in FIG. 4, and it may be a color image or a black-and-white image. For example, the defective product image can be acquired by imaging the defective product work with the camera module 14 of the imaging unit 3. Only one defective product image may be acquired, or a plurality of defective product images may be acquired by imaging different defective products. The acquired defective product image is stored in the storage device 19, for example. The order of steps SA1 to SA3 does not have to be as described above.

[0046] In step SA4, noise is added to the non-defective product image. FIG. 4 shows a first input image input as a learning image to the machine learning network, and this first input image is an image generated by adding noise to the learning non-defective product image. Conventionally, as a method of adding noise to an image, a method of adding single-pixel noise following a Gaussian distribution has been common. However, with this method, in a non-defective area such as near the edge of the work, it was likely to be over-detected as a defective area. In the present embodiment, a method completely different from the conventional noise addition method is adopted. That is, the processor 13a randomly adds a plurality of noises having a predetermined size equal to or larger than a single pixel, rather than a single pixel, to the learning non-defective product image. The shape of the noise may be circular, elliptical, polygonal such as square, or free-form. Also, a plurality of types of noises with different shapes may be added to a single learning non-defective product image. At this time, the processor 13a increases the amount of noise added to the learning non-defective product image as the learning non-defective product image is larger.

[0047] Conventionally, when adding noise to an image, it is common to add gray noise. However, when gray noise is added, it is difficult to detect color abnormalities. In this embodiment, when the good image for learning is a color image, the processor 13a adds color noise to the good image for learning. Color noise refers to chromatic noise, which is noise of colors other than white, black, and gray. When adding a plurality of noises to the good image for learning, the color may be changed for each noise, or the same color may be used. When the good image for learning is a black-and-white image, gray noise may be added. The locations where noise is added on the good image for learning become abnormal locations.

[0048] After adding noise, proceed to step SA5. In step SA5, a first target abnormality degree map image (shown in FIG. 4) is generated based on the difference in pixel values between corresponding locations of the good image for learning (the first input image shown in FIG. 4) with noise added in step SA4 and the good image for learning without noise added. Specifically, the average of the absolute values of the differences between the added noise (abnormality) and the locations corresponding to the noise in the original good image for learning is calculated, and the calculated average value is multiplied by a predetermined gain. As a result, a first target abnormality degree map image in which the pixel values of the portions other than the locations where noise is added are 0 can be obtained. In this example, since a large number of small circular noises are added, in the first target abnormality degree map image, the locations corresponding to the noise are white and the other portions are black (pixel value is 0).

[0049] Also, in step SA6, annotation is performed on the defective product image obtained in step SA3. That is, the user designates that the defective product image obtained in step SA3 is an image corresponding to the defective product. The method of this designation is not particularly limited, and for example, a method of attaching a label indicating that it is a defective product image can be cited. As a method of attaching a label, it may be attached one by one to the defective product images, or for example, a plurality of defective product images may be stored in a specific folder, and labels may be attached to the defective product images in the folder all at once. The processor 13a can store the defective product image and the label, which is the defective information input by the user, in the storage device 19 in an associated state.

[0050] Also, annotation includes the user designating where the defective part of the defective product image is. For example, it may be annotation of the area designation method in which the user surrounds the defective part of the defective product image displayed on the display device 4 to designate the defective part, or it may be annotation of the precise designation method in which the user traces the defective part of the defective product image displayed on the display device 4 to designate the defective part in a free form. The user can select either the area designation method or the precise designation method.

[0051] In the area designation method, the user operates the mouse 52 to generate a frame surrounding the defective part of the defective product image. For example, the defective part on the defective product image can be designated by generating a rectangular, circular, or free-form frame of the size surrounding the defective part by the user.

[0052] In the precise designation method, when the user moves the filling tool along the defective part 202a, the parts other than the defective part are less likely to be included in the designated area, so more precise annotation becomes possible compared to the above area designation method. In addition to the filling tool, the defective part may also be designated with a magnet tool. The magnet tool can be moved by operating the mouse 52. When it is moved close to the defective part, it moves so as to be automatically attracted to the defective part. By placing the defective part within a frame connecting multiple magnet tools, precise designation of the defective part can be performed while reducing the user's burden.

[0053] Also, the defective part may be designated with the GrabCut tool. When the defective part and the area around the defective part are surrounded by the GrabCut tool, that area is designated, and an automatic extraction method for automatically extracting the defective part within the designated area is executed. In the automatic extraction method, only the defective part is designated and the area around the defective part is not designated, so precise designation of the defective part is automatically performed as shown by the white circle on the right. Thereby, the user's burden can be reduced.

[0054] However, in the case of the GrabCut tool, precise designation of the defective part may fail. In this case, it will include the area around the defective part. In such a case, after executing the automatic extraction method, the user finely designates the foreground and background by performing stroke correction or click correction.

[0055] Also, the defective part may be designated by AI Assisted designation. In the case of AI Assisted designation, after roughly designating and extracting the contour of the defective part, the inside of the extracted part is designated with a Fill tool or the like. Thereby, the defective part can be automatically extracted. After automatically extracting the defective part, fine correction is also possible.

[0056] Step SA7 generates a second target abnormality degree map image (shown in FIG. 4) based on the defective product image on which the annotation was executed in step SA6. In this example, since the defective part is a linear scratch, in the second target abnormality degree map image, the defective part that appears linearly is white, and the part other than the defective part is black (pixel value is 0).

[0057] In step SA8, the parameters of the machine learning network are adjusted. Specifically, the processor 13a inputs the learning good product image with noise added in step SA4, the first target abnormality degree map image generated in step SA5, the defective product image on which the annotation was executed in step SA6, and the second target abnormality degree map image generated in step SA7 into the machine learning network. A dataset is composed of the learning good product image and the defective product image. The adjustment of the parameters of the machine learning network may be performed by the user, the manufacturer that manufactures the appearance inspection device 1, or on the cloud.

[0058] As shown in FIG. 4, when the learning good product image with noise added is input into the machine learning network, a first output image corresponding to the learning good product image with noise added is output from the machine learning network. The first output image is an abnormality degree map indicating the position of the noise. The processor 13a adjusts the parameters of the machine learning network so that the first output image matches the first target abnormality degree map image. That is, the processor 13a executes a first learning process of adding noise to the good product image, causing the machine learning network to learn, and adjusting the parameters of the machine learning network so that the part corresponding to the noise is extracted.

[0059] Further, when the defective product image with annotation is input into the machine learning network, the machine learning network outputs a second output image corresponding to the defective product image. The second output image is an abnormality map indicating the position of the defective part specified by the user. The processor 13a adjusts the parameters of the machine learning network so that the second output image matches the second target abnormality map image. That is, the processor 13a causes the machine learning network to learn the defective product images corresponding to the defective products having defective parts, and executes a second learning process for adjusting the parameters of the machine learning network so that the defective parts previously specified by the user are extracted on the defective product images.

[0060] The first learning process can be performed multiple times using a plurality of learning good product images with noise added thereto and a plurality of first target abnormality map images respectively corresponding to them. Also, the second learning process can be performed multiple times using a plurality of defective product images with annotation and a plurality of second target abnormality map images respectively corresponding to them.

[0061] By adjusting the parameters in step SA8, a trained machine learning network is generated. Then, in step SA9, information for constructing the machine learning network, such as the parameters adjusted in step SA8, is stored in the storage device 19 or the like.

[0062] (Startup procedure of the appearance inspection device 1) As described above, when setting the appearance inspection device 1, a trained machine learning network can be generated and stored in the storage device 19. However, as in the flowchart shown in FIG. 5, the startup process of the appearance inspection device 1 may be executed.

[0063] Steps SB1 to SB5 of the flowchart shown in FIG. 5 are the same as steps SA1 to SA5 of the flowchart shown in FIG. 3. Also, steps SB6 and SB7 of the flowchart shown in FIG. 5 are the same as steps SA8 and SA9 of the flowchart shown in FIG. 3. In step SB6, when adjusting the parameters of the machine learning network, the processor 13a inputs the learned good product image with noise added in step SB4 and the first target abnormality degree map image generated in step SB5 into the machine learning network. Therefore, in the first step SB6, learning using defective product images is not performed.

[0064] In step SB8 of the flowchart shown in FIG. 5, verification processing is executed. In this verification processing, the detection ability of the machine learning network whose parameters were adjusted in step SA8 of the flowchart shown in FIG. 3 is verified. That is, although the machine learning network whose parameters were adjusted in step SA8 is a learned one, it is possible that the degree of learning is low. If operation is started with a machine learning network in a state of low learning degree, there is a risk of causing detection omission of defective product images or over-detection of good product images as being defective. Therefore, before operating the learned machine learning network, the detection ability of the machine learning network is verified so that the detection ability can be enhanced if it is insufficient.

[0065] In the verification processing, a work image of a defective product is prepared. This work image may be an image acquired before the verification processing, an image newly acquired for the verification processing, or an image acquired during the operation of the appearance inspection device 1. Since it is a work image used for verification, it can also be called a test image. As shown in FIG. 6, the processor 13a inputs the work image into the learned machine learning network. An output image (abnormality degree map) corresponding to the work image is output from the machine learning network.

[0066] In the example shown in FIG. 6, a case where the workpiece has a first defective portion B1 and a second defective portion B2 is shown. That is, in the good product image with noise shown in FIG. 4, a large number of small circular noises are added, whereas the first defective portion B1 in the workpiece image shown in FIG. 6 is a single circular shape. Therefore, since the first defective portion B1 has characteristics different from those of the good product image with noise shown in FIG. 4, it corresponds to an unknown defect that has not been learned even by a learned machine learning network. However, since a good product image with noise including the shape of the first defective portion B1 was input during learning, even if the first defective portion B1 is an unknown defect for the machine learning network, the first defective portion B1 can be detected as shown in the output image in FIG. 6.

[0067] Also, since the second defective portion B2 is almost the same as the defective portion specified by annotation in the second input image shown in FIG. 4, it corresponds to a known defect having the characteristics specified as the defective portion. Since the second defective portion B2 is a known defect for the learned machine learning network, the second defective portion B2 can be detected by the machine learning network. That is, the processor 13a inputs the workpiece image to the machine learning network whose parameters have been adjusted by the first learning process and the second learning process described above, thereby performing the detection processes for both an unknown defect having characteristics different from those of the good product image with noise and a known defect having the characteristics specified as the defective portion. The verification process may be executed using a single workpiece image, or may be executed by sequentially inputting a plurality of mutually different workpiece images to the machine learning network.

[0068] When at least one of the unknown defect and the known defect is detected as a result of the above detection process, the processor 13a is configured to be able to execute an inspection process of determining that the workpiece in the workpiece image is a defective product, while when neither the unknown defect nor the known defect is detected, it is configured to determine that the workpiece in the workpiece image is a good product. Note that at the time of setting, instead of determining defective and good products, it may be determined whether or not at least one of the unknown defect and the known defect is detected.

[0069] After step SB8, proceed to step SB9. In step SB9, it is determined whether there is a detection omission based on the result of the verification process in step SB8. If there is a detection omission in the work image input to the machine learning network, proceed to step SB10. On the other hand, if there is no detection omission in the work image input to the machine learning network, it is determined as NO in step SB9 and proceed to step SB11.

[0070] As a case where it is determined as YES in step SB9, for example, a case where, although the work image input to the machine learning network is an image capturing a defective product, it is not determined as a defective product can be cited. In this case, in step SB10, the user performs annotation on the work image corresponding to the defective product input to the machine learning network in step SB8. The annotation can be executed in the same manner as step SA6 of the flowchart shown in FIG. 3. By going through step SB10, a defective product image with annotation information can be obtained. The defective product image with annotation information can be added to the dataset and stored in the storage device 19 or the like.

[0071] In step SB12, in the same manner as step SA7 of the flowchart shown in FIG. 3, a second target abnormality degree map image is generated based on the defective product image with annotation information. Then, proceed to step SB6, and the processor 13a inputs the defective product image with annotation information and the second target abnormality degree map image generated in step SB12 to the machine learning network. At this time, the machine learning network is made to learn the original dataset added with the defective product image with annotation information. Then, an abnormality degree map indicating the position of the defective part designated by the user is output from the machine learning network. The processor 13a re-adjusts the parameters of the machine learning network so that the abnormality degree map output from the machine learning network matches the second target abnormality degree map image. That is, the processor 13a executes an update process of making the machine learning network learn the defective product image with annotation information in which the defective part is designated by annotation and updating the parameters of the machine learning network.

[0072] Also, in step SB11, it is determined whether over-detection has occurred. If there is over-detection in the work image input to the machine learning network, the process proceeds to step SB13. On the other hand, if there is no over-detection in the work image input to the machine learning network, it is determined as NO in step SB11 and the process proceeds to step SB14. In step SB14, the result of the verification process is output and presented to the user.

[0073] As a case where it is determined as YES in step SB11, for example, a case where the work image input to the machine learning network is an image of a good product but is determined as a defective product can be cited. In this case, in step SB13, an image (over-detected good product image) that is determined as a defective product despite being a good product image is acquired, and the process proceeds to step SB6. In step SB6 that has advanced through step SB13, the processor 13a inputs the good product image acquired in step SB13 to the machine learning network for learning. At this time, the machine learning network is made to learn the original dataset with the good product image added. Thereby, an update process for updating the parameters of the machine learning network can be executed.

[0074] (Specific method of learning of the machine learning network) Next, an example of a specific method of learning of the machine learning network will be described. For example, the machine learning network can be learned by minimizing the Loss function. Although the definition of Loss is various, Mean Square Error (MSE) can be cited as an example.

[0075]

Number

[0076] Here, T is the target abnormality map, 0 is the output image (abnormality map), n is the number of pixels in the image T that are 0, and x and y are pixel positions. Note that loss functions such as Binary Cross Entropy can also be used. The above is merely an example, and the learning method of the machine learning network is not limited to these methods.

[0077] (During operation of the appearance inspection device 1) Next, regarding the operation of the appearance inspection device 1, it will be described based on the flowchart shown in FIG. 7. In step SC1 after starting, the processor 13a reads parameters and the like stored in the storage device 19 to prepare a learned machine learning network. In step SC2, the imaging unit 3 images the work to be inspected to obtain a work image. Then, it proceeds to step SC3, and the work image obtained in step SC2 is input into the machine learning network prepared in step SC1.

[0078] Next, in step SC4, the machine learning network executes an inference process on the work image input in step SC3. Then, in step SC5, the machine learning network outputs an abnormality map as a result of the inference process. The abnormality map shows the presence or absence of unknown defects having characteristics different from those of a good product image with noise added, and the presence or absence of known defects having characteristics designated as defective portions.

[0079] After that, in step SC6, a pass / fail determination of the work is made based on the abnormality map output in step SC5. That is, when at least one of an unknown defect having characteristics different from those of a good product image with noise added and a known defect having characteristics designated as a defective portion is detected, it is determined that the work is a defective product, but when neither the unknown defect nor the known defect is detected, it is determined that the work is a good product. This pass / fail determination is made by the processor 13a. The pass / fail determination result of the work can be output to, for example, the display device 4 or the like and presented to the user, and can also be stored in the storage device 19. Note that steps SC2 to SC6 can be executed each time the work changes.

[0080] (Advantages and effects of the embodiment) As described above, according to this embodiment, the machine learning network is not trained only with defective product images on which annotation has been performed, but can also be trained using non-defective product images to which noise has been added. Therefore, it is possible to generate a machine learning network with high detection ability not only for known defects included in the defective product images used for learning, but also for unknown defects. Compared with the case of performing inference processing using a non-defective product learning model and a defective product learning model as in the prior art, the learning difficulty is reduced, the labor during learning can be reduced, and the synthesis processing of the inference results becomes unnecessary during appearance inspection, so the tact time during operation is shortened.

[0081] The above-described embodiment is merely illustrative in every respect and should not be construed in a limiting sense. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

Industrial applicability

[0082] As described above, the present invention can be used when inspecting the appearance of a workpiece.

Explanation of reference numerals

[0083] 1 Appearance inspection device 13a Processor 19 Storage device (storage unit)

Claims

1. An appearance inspection device comprising a storage unit for storing a machine learning network, and a processor for inputting a work image obtained by photographing a work to be inspected into the machine learning network and performing a pass / fail determination of the work based on the input work image, wherein the processor, performs a first learning process of adding noise to a good product image corresponding to a good product and causing the machine learning network to learn, and adjusting parameters of the machine learning network so that a portion corresponding to the noise is extracted, performs a second learning process of causing the machine learning network to learn a defective product image corresponding to a defective product having a defective portion, and adjusting parameters of the machine learning network so that the defective portion previously specified by a user is extracted on the defective product image, and is configured to be capable of executing detection processes for both an unknown defect having features different from those of the good product image and a known defect having features specified as the defective portion by inputting the work image into the machine learning network whose parameters have been adjusted by the first learning process and the second learning process.

2. In the appearance inspection device according to claim 1, the processor, at the time of setting, performs the first learning process of inputting an input image obtained by adding noise to the good product image into the machine learning network and adjusting parameters of the machine learning network so that an abnormality map indicating the position of the noise becomes a first output image, and performs the second learning process of inputting a defective product image for which a user has accepted designation of a defective portion into the machine learning network and adjusting parameters of the machine learning network so that an abnormality map indicating the position of the defective portion designated by the user becomes a second output image.

3. In the appearance inspection device according to claim 2, the processor, During the first learning process, a target abnormality degree map image is generated based on the difference in pixel values between corresponding locations of the good product image with the noise added and the good product image without the noise added, and the parameters of the machine learning network are adjusted so that the first output image matches the target abnormality degree map image. An appearance inspection device.

4. In the appearance inspection device according to any one of claims 1 to 3, The processor is An appearance inspection device that randomly adds a plurality of the noises of a predetermined size or more to the good product image.

5. In the appearance inspection device according to any one of claims 1 to 4, The processor is An appearance inspection device that, when the good product image is a color image, adds the color noise to the good product image.

6. In the appearance inspection device according to any one of claims 1 to 5, The processor is An appearance inspection device that increases the amount of the noise added to the good product image as the good product image is larger.

7. In the appearance inspection device according to any one of claims 1 to 6, The processor is An appearance inspection device that adds a plurality of types of noises with different shapes to a single good product image.

8. In the appearance inspection device according to any one of claims 1 to 7, The processor is Even though the work image is an image of a defective product, if it is not determined as a defective product, an update process is executed in which the machine learning network is made to learn a raw dataset obtained by adding a defective product image with annotation information in which defective parts are specified by annotation, and the parameters of the machine learning network are updated. An appearance inspection device.

9. In the appearance inspection apparatus according to any one of claims 1 to 8, the processor In the case where, as a result of the detection process, the work image is determined to be a defective product even though it is an image of a non-defective product, an update process is executed to cause the machine learning network to learn the original dataset obtained by adding the image as a non-defective product image and update the parameters of the machine learning network. An appearance inspection apparatus.

10. An appearance inspection method for inputting a work image obtained by photographing a work to be inspected into a machine learning network and determining whether the work is defective or non-defective based on the input work image, a first learning process of adding noise to a non-defective product image corresponding to a non-defective product and causing the machine learning network to learn, and adjusting the parameters of the machine learning network so that a portion corresponding to the noise is extracted; a second learning process of causing the machine learning network to learn a defective product image corresponding to a defective product having a defective portion, and adjusting the parameters of the machine learning network so that the defective portion specified in advance by a user is extracted on the defective product image; An appearance inspection method for executing a detection process for both an unknown defect having a feature different from that of the non-defective product image and a known defect having a feature specified as the defective portion by inputting the work image into the machine learning network whose parameters have been adjusted by the first learning process and the second learning process.

Images