Appearance inspection device and appearance inspection method

Abstract

To reduce a time required to learn a defective product by naturally creating a defective product image for learning, which has various defective states that may be actually assumed in a workpiece.SOLUTION: A new defective product image is generated by performing an extraction process for extracting a defective point of a defective product image, a pasting process for pasting a defective point, extracted in the extraction process, to a reference image, and a process for making a boundary between the defective point, pasted in the pasting process, and the reference image inconspicuous. An inference model is generated by causing a machine learning network to learn a good product image and defective product image stored in a storage unit in advance and a new defective product image.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an appearance inspection device and an appearance inspection method for inspecting the appearance of a workpiece. [Background technology]

[0002] For example, Patent Document 1 discloses a processing device that uses computer-based machine learning to determine whether a workpiece is a good or defective product. The processing device in Patent Document 1 is configured to perform supervised machine learning on good product data to generate a good product learning model, and to perform supervised machine learning on defective product data to generate a defective product learning model, and then input data on the workpiece to be determined, and determine whether the workpiece is a good or defective product using the good product learning model and the defective product learning model; such a device is also called a workpiece appearance inspection device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-204321 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, when performing supervised learning on defective product data, dozens or even hundreds of images of defective products are required. However, since defective products rarely occur at work production sites, collecting such a large number of images of defective products would take a long time. Therefore, learning on good product data is considered a solution, but machine learning networks trained only on good product data lack the ability to detect defective products, and their performance is inferior to that of defective product learning in more difficult inspections, so defective product learning is desirable.

[0005] In response to this, if defective product images for learning can be generated inside the visual inspection device, a large number of defective product images can be acquired in a relatively short time. However, generating defective product images inside the visual inspection device has been limited to simple transformations such as image contrast conversion, simple rotation, and aspect ratio change, making it difficult to generate a wide range of defective product images. Specifically, for example, when generating an inference model for detecting "cracks" in a workpiece, a wide range of defective product images is required because the shape, position, and size of "cracks" in actual workpieces can vary widely. However, it has been difficult to achieve this with images that simply have their contrast converted, rotated, or aspect ratio changed.

[0006] The present disclosure has been made in consideration of such points, and its purpose is to shorten the time required for defective product learning by making it possible to naturally create defective product images for learning that have various defect states that are actually expected in workpieces. [Means for solving the problem]

[0007] To achieve the above object, one aspect of the present disclosure provides a visual inspection device that inputs workpiece images captured of a workpiece to be inspected into a machine learning network and determines whether the workpiece is good or bad based on the input workpiece images. The visual inspection device includes: a memory unit that stores good product images corresponding to good products and defective product images corresponding to defective products; an image processing unit that generates new defective product images by performing an extraction process that extracts defective areas from the defective product images pre-stored in the memory unit, a pasting process that pastes the defective areas extracted by the extraction process onto a reference image including the workpiece, and a process that makes the boundary between the pasted defective areas and the reference image less noticeable; and a learning unit that trains a machine learning network on the good product images and defective product images pre-stored in the memory unit and the new defective product image generated by the image processing unit to generate an inference model. The visual inspection device also includes an inspection unit that inputs workpiece images captured of the workpiece to be inspected into the inference model generated by the learning unit and determines whether the workpiece is good or bad based on the input workpiece images.

[0008] According to this configuration, when a defective portion in a defective product image is extracted, the extracted defective portion is pasted onto a reference image. The reference image onto which the defective portion is pasted may be a non-defective product image or a defective product image. As a result, a new defective product image having the defective portion is generated. In the newly generated defective product image, the boundary between the pasted defective portion and the reference image is inconspicuous, resulting in a natural defective product image. By repeating this process, multiple learning images of defective products having various defect states that are actually expected in workpieces can be obtained.

[0009] By having the machine learning network learn new defective product images and images of good and defective products previously stored in the storage unit, an inference model with high defect detection capabilities is generated, thereby improving visual inspection performance.

[0010] In another aspect, the image processing unit can acquire the features of the defective product image stored in the memory unit during the extraction process, and extract the area having features different from the features of the surrounding area as the defective area.

[0011] According to this configuration, defective parts of the defective product images stored in the storage unit are automatically extracted, thereby saving the user time and effort.

[0012] The image processing unit according to another aspect can accept a correction of the range of the extracted defective portion in the extraction process.

[0013] In other words, it is assumed that the range of the defective part automatically extracted based on the feature amount of the defective product image is incorrect. In this case, if the user corrects the range of the defective part, the correction is accepted, and the range of the defective part can be set appropriately.

[0014] In another aspect, the image processing unit can accept a user's selection operation of the defective part from among the defective product images stored in the memory unit during the extraction process, and extract the defective part based on the accepted selection operation.

[0015] That is, it is expected that there may be cases where extraction based on feature quantities is difficult depending on the defective product image or defective portion. In such cases, if the user selects the defective portion from the defective product image, the selected defective portion can be extracted, making it less likely that the defective portion will be extracted erroneously.

[0016] The image processing unit according to another aspect can paste the defect portion extracted in the extraction process onto the reference image in a state in which the defect portion has been subjected to image conversion.

[0017] According to this configuration, instead of changing the contrast or rotating the entire defective product image, image conversion is performed only on the defective portion, so that the variety of new defective product images that are generated can be significantly increased.

[0018] The image processing unit according to another aspect can identify a work part included in the reference image as a target area for pasting the defective part, and paste the defective part into the identified target area for pasting, thereby generating a defective product image suitable for learning. Furthermore, after the paste processing of the defective part, a trimming process can be performed to trim the part of the defective part that protrudes from the target area for pasting.

[0019] According to this configuration, unnatural defective product images in which the defective part is located outside the work part are not generated, and therefore the accuracy of learning can be improved.

[0020] The image processing unit according to another aspect further includes a display control unit that displays the new defective product image generated by the image processing unit on a display unit, so that the user can visually confirm what kind of defective product image has been generated by pasting and blurring the defective portion. As a result, if the image looks strange, the user can decide whether or not to use it for learning, for example, by not using it for learning.

[0021] The image processing unit according to another aspect can assign identification information to the new defective product image generated by the image processing unit to distinguish it from defective product images obtained by photographing a defective workpiece. According to this configuration, since it is possible to distinguish between defective product images generated by the image processing unit and defective product images obtained by photographing defective workpieces, for example, when a large number of defective product images obtained by photographing defective workpieces have been accumulated, it is possible to exclude defective product images generated by the image processing unit from the learning targets. Also, when learning defective product images generated by the image processing unit, it is possible to lighten the weight of learning compared to defective product images obtained by photographing defective workpieces. [Effects of the Invention]

[0022] As explained above, by pasting defective parts extracted from a defective product image onto a reference image and then making the boundary between the defective parts and the reference image less noticeable, new defective product images with various defect states that are actually expected in workpieces can be naturally created. Then, by having a machine learning network learn the new defective product images and the good product images and defective product images pre-stored in the storage unit, an inference model with high defect detection capabilities can be generated in a short time. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a schematic diagram showing a configuration of a visual inspection apparatus according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing a hardware configuration of the appearance inspection device. [Figure 3] 10 is a flowchart illustrating an example of a procedure for a learning process. [Figure 4] 10 is a flowchart illustrating an example of a procedure for internally generating a defective product image. [Figure 5] FIG. 5A shows an image of an actual defective product, and FIG. 5B shows an image of the defective part extracted from the defective product image. [Figure 6] FIG. 6A shows a reference image, and FIG. 6B shows an image in which the region to be pasted has been identified. [Figure 7] FIG. 7A shows an example of an image of a doughnut-shaped defective product, and FIGS. 7B and 7C show how the defective part is pasted. [Figure 8] FIG. 8A shows an example of an image of a long defective product, and FIGS. 8B and 8C show how the defective part is pasted. [Figure 9] FIG. 9A shows an example of an image of a defective product with a complex shape, and FIGS. 9B and 9C show how the defective part is pasted. [Figure 10] FIG. 10A shows a state in which the defective portion is pasted within the paste target area, and FIG. 10B shows a state in which the defective portion is pasted while protruding from the paste target area. [Figure 11]10 is a flowchart illustrating an example of a procedure for operating the visual inspection device. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or its uses.

[0025] FIG. 1 is a schematic diagram showing the configuration of an appearance inspection device 1 according to an embodiment of the present invention. The appearance inspection device 1 is a device for determining the quality of a workpiece image acquired by capturing an image of a workpiece to be inspected, such as various parts or products, and can be used in production sites such as factories. Specifically, a machine learning network is built inside the appearance inspection device 1. A workpiece image captured of the workpiece to be inspected is input to the generated machine learning network, and the machine learning network can determine the quality of the workpiece image.

[0026] The entire workpiece may be the object of inspection, or only a portion of the workpiece may be the object of inspection. Also, one workpiece may contain multiple inspection objects. Also, a workpiece image may contain multiple workpieces.

[0027] The appearance inspection device 1 comprises a control unit 2, which is the device main body, an imaging unit 3, a display device (display section) 4, and a personal computer 5. The personal computer 5 is not essential and can be omitted. The personal computer 5 can be used instead of the display device 4 to display various information and images, and the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0028] 1 illustrates a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5 as an example of the configuration of the visual inspection device 1, but any two or more of these can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Furthermore, the control unit 2 can be divided into multiple units and some of them can be incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into multiple units and some of them can be incorporated into other units.

[0029] (Configuration of imaging unit 3) As shown in FIG. 2, the imaging unit 3 includes a camera module (imaging section) 14 and an illumination module (illumination section) 15, and is a unit that acquires workpiece images. The camera module 14 includes an AF motor 141 that drives the imaging optical system, and an imaging board 142. The AF motor 141 is a part that automatically adjusts focus by driving the lens of the imaging optical system, and can perform focus adjustment using a conventionally well-known method such as contrast autofocus. The imaging board 142 includes a CMOS sensor 143 as a light receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of the CMOS sensor 143, a light receiving element such as a CCD sensor can be used.

[0030] The lighting module 15 includes an LED (light emitting diode) 151 as a light emitting element that illuminates an imaging area including a workpiece, and an LED driver 152 that controls the LED 151. The timing, duration, and amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or may be provided separately from the imaging unit 3 as an external lighting unit.

[0031] (Configuration of display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel or an organic EL panel. A work image, a user interface image, etc. output from the control unit 2 are displayed on the display device 4. If the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used in place of the display device 4.

[0032] (operation equipment) Examples of operation devices for a user to operate the visual inspection apparatus 1 include, but are not limited to, the keyboard 51 and mouse 52 of the personal computer 5, and any device configured to be able to accept various operations by the user may be used. For example, a pointing device such as the touch panel 41 of the display device 4 is also included in the operation devices.

[0033] User operations on the keyboard 51 and mouse 52 can be detected by the control unit 2. The touch panel 41 is a conventionally known touch-type operation panel equipped with, for example, a pressure-sensitive sensor, and user touch operations can be detected by the control unit 2. The same applies when other pointing devices are used.

[0034] (Configuration of control unit 2) The control unit 2 includes a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The main board 13 is provided with a processor 13a. The processor 13a controls the operation of each connected board and module. For example, the processor 13a outputs a lighting control signal to an LED driver 152 of the lighting module 15 to control the turning on / off of the LED 151. In response to the lighting control signal from the processor 13a, the LED driver 152 switches the turning on / off of the LED 151 and adjusts the lighting time, and also adjusts the light intensity of the LED 151.

[0035] In addition, the processor 13a outputs an imaging control signal to the imaging board 142 of the camera module 14 to control the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13a and adjusts the exposure time to any desired time to perform imaging. That is, the imaging unit 3 captures an image within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13a. If a workpiece is present within the field of view, the image of the workpiece is captured. However, if an object other than the workpiece is present within the field of view, the image of the object can also be captured. For example, the visual inspection device 1 can use the imaging unit 3 to capture non-defective product images corresponding to non-defective products and defective product images corresponding to defective products as learning images for the machine learning network. The learning images do not have to be images captured by the imaging unit 3, but may be images captured by another camera, etc.

[0036] On the other hand, when the appearance inspection device is in operation, the workpiece can be imaged by the imaging unit 3. The CMOS sensor 143 is configured to be able to output a live image, i.e., a currently captured image, at a short frame rate at any time.

[0037] When the CMOS sensor 143 has finished capturing an image, the image signal output from the imaging unit 3 is input to the processor 13a of the main board 13 for processing, and is also stored in the memory 13b of the main board 13. Specific details of the processing performed by the processor 13a of the main board 13 will be described later. The main board 13 may be provided with a processing device such as an FPGA or a DSP. The processor 13a may also be an integrated processor such as an FPGA or a DSP.

[0038] A display control unit 13c is provided on the main substrate 13. The display control unit 13c is a part that generates a display screen, controls the display device 4, and causes the display screen to be displayed on the display device 4. The specific operation of the display control unit 13c will be described later.

[0039] The connector board 16 is a part that receives power from an external source via a power connector (not shown) provided on the power interface 161. The power supply board 18 is a part that distributes the power received by the connector board 16 to each board and module, and specifically distributes power to the illumination module 15, the camera module 14, the main board 13, and the communication board 17. The power supply board 18 is equipped with an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14 to achieve autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 in response to an AF control signal from the processor 13a on the main board 13.

[0040] The communication board 17 is a part that executes communication between the main board 13 and the display device 4 and the personal computer 5, and communication between the main board 13 and an external control device (not shown). An example of the external control device is a programmable logic controller. The communication may be wired or wireless, and either form of communication can be realized by a conventionally known communication module.

[0041] The control unit 2 is provided with a storage device (storage unit) 19, which may be, for example, a solid state drive, a hard disk drive, or the like. The storage device 19 stores program files 80, setting files, and the like (software) that enable the hardware to execute the various controls and processes described below. The program files 80 and setting files can be stored in a storage medium 90, such as an optical disk, and the program files 80 and setting files stored in the storage medium 90 can be installed in the control unit 2. The program files 80 may be downloaded from an external server via a communication line. The storage device 19 can also store, for example, the image data and the like, as well as parameters for constructing an inference model obtained after the learning process described below.

[0042] 2, the storage device 19 is shown as being integrated with the control unit 2, but the storage device 19 may be separate from the control unit 2. An example of such a storage device 19 is a network-attached storage (NAS). The NAS and the control unit 2 are connected by a communication line such as a wired LAN or a wireless LAN.

[0043] That is, in the visual inspection device 1, the machine learning network is trained using training images, and the parameters of the machine learning network are adjusted to generate an inference model. Workpiece images of the workpiece to be inspected are input to the inference model, and the quality of the workpiece can be determined based on the input workpiece images. By using this visual inspection device 1, a visual inspection method can be performed in which the quality of the workpiece is determined based on the workpiece images.

[0044] (Processor configuration) 2, the processor 13a is provided with an image processing unit 13d, a learning unit 13e, and an inspection unit 13f. The image processing unit 13d, the learning unit 13e, and the inspection unit 13f may be configured as hardware or may be configured by executing software. Furthermore, the image processing unit 13d, the learning unit 13e, and the inspection unit 13f do not necessarily have to be provided on the main board 13; some or all of these may be provided on a board other than the main board 13.

[0045] The details of each part will be described later, but the outline is as follows. The image processing unit 13d is a part that internally generates defective product images, which will be described later. The learning unit 13e is a part that inputs learning data into a machine learning network, causes it to learn, and generates an inference model for determining whether the input image is good or bad. The learning unit 13e may be composed of, for example, a learning computer separate from the control unit 2. The learning computer is configured to be able to perform machine learning at high speed. By communicatively connecting the learning computer and the control unit 2, parameters for constructing the inference model generated by the learning computer can be transmitted to and stored in the storage device 19. This allows the inference model to be constructed in the control unit 2. The inspection unit 13f is a part that inputs workpiece images into the inference model and determines whether the workpiece is good or bad.

[0046] (Learning process) Next, we will explain the learning process of the machine learning network in detail. There are two types of learning: defective product learning, which involves learning from defective product images, and good product learning, which involves learning from good product images. Since the majority of products at work production sites are good, it is easy to collect a large number of good product images for good product learning. However, machine learning networks trained only with good product images lack the ability to detect defective products and perform worse than defective product learning in more difficult inspections. Therefore, there is a demand for defective product learning. However, since the number of defective products occurring at work production sites is extremely low, it is difficult to collect defective product images for learning in a short period of time. In this embodiment, the visual inspection device 1 is configured to naturally generate a large number of defective product images with a wide variety. This will be described in detail below.

[0047] The flowchart in FIG. 3 shows an example of the procedure of the learning process. In step SA1 after the start, the learning unit 13e prepares an untrained machine learning network. The untrained machine learning network has, for example, initial parameter values ​​determined randomly. In step SA2, learning images of non-defective products and defective product images to be used for learning the machine learning network are stored in the storage device 19. This step SA2 is a storage step. The storage step may be executed by a user before step SA1. The learning images of non-defective products and defective product images may be images that the user has previously photographed and stored, images that have been newly photographed for learning purposes, or images that have been captured by the imaging unit 3 during operation of the visual inspection apparatus 1. In addition, non-defective product images and defective product images stored in a storage device (not shown) other than the storage device 19 may also be used as the learning images of non-defective products and defective product images.

[0048] Annotations are performed on defective product images in advance. That is, the user performs processing in advance to clearly indicate that the defective product image is a defective product image, to specify the defective part of the defective product image, etc. The information added by the annotations is stored in the storage device 19 in association with the corresponding defective product image.

[0049] The process then proceeds to step SA3. In step SA3, it is determined whether internal generation of defective product images is necessary. Internal generation of defective product images, which will be described in detail later, refers to a process of creating another defective product image within the visual inspection device 1 using an existing defective product image. The determination in step SA3 can be made by the user. For example, if only one or a few defective product images have been collected, the number of defective product images for defective product learning is insufficient, and the user determines that internal generation of defective product images is necessary. On the other hand, if several tens or hundreds of defective product images have been collected for defective product learning, the number of defective product images for defective product learning is sufficient, and the user determines that internal generation of defective product images is not necessary. The user can input whether internal generation of defective product images is necessary using an operating device such as the mouse 52. For example, the display control unit 13c can generate a user interface screen that allows the user to select whether internal generation of defective product images is necessary and display the screen on the display device 4, allowing the user to easily input whether internal generation of defective product images is necessary. The input by the user is received by the learning unit 13e, and therefore, inside the visual inspection device 1, it can be treated as if the learning unit 13e is making a judgment in step SA3.

[0050] If the determination in step SA3 is YES and internal generation processing of defective product images is necessary, the process proceeds to step SA4, whereas if the determination in step SA3 is NO and internal generation processing of defective product images is not necessary, the process proceeds to step SA5.

[0051] In step SA4, an internal generation process of defective product images is executed to increase the number of defective product images for learning. An example of the procedure of the internal generation process is shown in the flowchart in FIG. 4. The flowchart in FIG. 4 starts when a YES decision is made in step SA3 of the flowchart in FIG. 3. In step SB1 after the start of the flowchart in FIG. 4, the learning unit 13e reads out the non-defective product images and the defective product images stored in the storage device 19, and the display control unit 13c displays the non-defective product images and the defective product images read out by the learning unit 13e on the display device 4. Any number of images may be displayed on the display device 4, but it is possible to display all of the defective product images. In this case, the defective product images can be displayed in a list format. This makes it possible to present various defect locations to the user.

[0052] In step SB2, the learning unit 13e accepts the user's selection of a defective product image. For example, in the previous step SB1, the display control unit 13c generates a user interface screen capable of displaying defective product images in a list format, and displays it on the display device 4. Selection buttons and the like are incorporated into the user interface screen, allowing the user to select a desired defective product image by operating the mouse 52 or the like. When the learning unit 13e detects the user's selection operation, it temporarily stores the defective product image corresponding to the selection operation as the selected image.

[0053] In step SB3, the learning unit 13e accepts the user's selection of a reference image. A reference image is an original image used for internal generation, and in many cases, a non-defective product image is suitable, but a defective product image may also be used. For example, a selection button or the like may be incorporated into the user interface screen displayed in step SB1, allowing the user to select a desired image by operating the mouse 52 or the like. When the learning unit 13e detects the user's selection operation, it temporarily stores the image corresponding to the selection operation as the selected image. The order of steps SB2 and SB3 may be reversed.

[0054] In step SB4, the image processing unit 13d reads the defective product image selected in step SB2 and executes an extraction process to extract defective parts from the defective image. The extraction of defective parts may be performed using a rule-based algorithm or manually by the user. In the extraction process of defective parts, the image processing unit 13d acquires feature amounts of the defective product image and executes an algorithm to extract parts having feature amounts different from those of the surrounding areas as defective parts.

[0055] A specific example of the process for extracting defective parts will be described below with reference to Fig. 5. Fig. 5A in Fig. 5 shows a defective product image 200 selected in step SB2. The defective product image 200 is acquired by imaging a workpiece W having a defective part W1. Fig. 5B in Fig. 5 shows an extracted image 201 in which the defective part W1 is extracted from the defective product image 200 using a rule-based algorithm. Examples of algorithms for extracting only the defective part W1 include, but are not limited to, a color extraction algorithm and a GrabCut algorithm, and other algorithms can also be used.

[0056] In the case of the color extraction algorithm, when a user specifies a color corresponding to the defective portion W1 on the defective product image 200, an area having that color is automatically extracted as the defective portion W1. In the case of the GrabCut algorithm, when a user surrounds the defective portion W1 and its surroundings on the defective product image 200, that area is specified, and the defective portion W1 is automatically extracted from within the specified area. That is, in the extraction process, the image processing unit 13d accepts a selection operation of the defective portion W1 from the defective product image 200 by the user, and extracts the defective portion W1 based on the accepted selection operation.

[0057] Then, in step SB5, the image processing unit 13d accepts corrections to the range of the defective area W1 extracted in step SB4. If the area color-extracted by the color extraction algorithm is significantly larger or smaller than the defective area W1, the user may re-specify the color or make corrections such as changing the threshold value. When the image processing unit 13d accepts this correction, the defective area W1 is re-extracted. In the case of the GrabCut algorithm, the user can specify the foreground and background in detail by performing stroke correction or click correction, and the defective area W1 is re-extracted each time.

[0058] Furthermore, when the user manually extracts the defective portion W1 on the defective product image 200, the user may, for example, enclose the defective portion W1 with the mouse 52, paint the defective portion W1, or the like. For example, a freehand tool or the like may be used to generate a frame surrounding the defective portion W1, and the defective portion W1 can be extracted using this frame. In this case, the image processing unit 13d extracts the enclosed or painted-in area as the defective portion W1.

[0059] The defective area W1 may also be extracted using AI Assisted specification. In the case of AI Assisted specification, the outline of the defective area W1 is roughly specified and extracted, and then the interior of the extracted area is specified using a Fill tool or the like. This allows the defective area W1 to be extracted automatically. After the defective area W1 is automatically extracted, it is also possible to make minor corrections.

[0060] When the user confirms that the defect location W1 has been extracted as intended and performs an operation to proceed to the next step, the process proceeds to step SB6. In step SB6, the image processing unit 13d determines whether or not to apply image conversion processing, which will be described later, to the defect location W1 extracted in steps SB4 and SB5. For example, the image processing unit 13d generates a user interface screen for selecting whether or not to apply the image conversion processing, and the display control unit 13c causes the generated user interface screen to be displayed on the display device 4. The user selects whether or not to apply the image conversion processing on the user interface screen by operating the mouse 52 or the like. When the image processing unit 13d detects that "apply" has been selected, the process proceeds to step SB7, whereas when it detects that "do not apply" has been selected, the process proceeds to step SB8.

[0061] In step SB7, the image processing unit 13d performs image conversion processing on the defective portion W1 extracted in steps SB4 and SB5. Examples of image conversion processing include aspect ratio change processing for changing the aspect ratio of the defective portion W1, rotation processing for rotating the defective portion W1, movement processing for moving the defective portion W1 in the horizontal direction (X direction) or vertical direction (Y direction), contrast conversion processing for changing the contrast of the defective portion W1, and color change processing for changing the color of the defective portion W1. However, the image conversion processing is not limited to these, and other image conversion processing may also be applied. Furthermore, any multiple image conversion processing may be performed on the defective portion W1 in any order. By performing step SB7, the defective portion W1 in the image-converted state can be pasted onto the reference image.

[0062] The user may be allowed to select which image conversion process to apply. For example, the image processing unit 13d generates a user interface screen for selecting the image conversion process, and the display control unit 13c causes the display device 4 to display the generated user interface screen. When the user selects the desired image conversion process on the user interface screen by operating the mouse 52 or the like, the image processing unit 13d detects this and applies the detected image conversion process to the defect location W1. The appearance inspection device 1 may also automatically set which image conversion process to apply. Note that steps SB6 and SB7 may be omitted.

[0063] After step SB7 or after a negative judgment in step SB6, the process proceeds to step SB8. In step SB8, a paste target area for pasting the extracted defective portion W1 is identified in the reference image. For example, assume that a reference image 202 as shown in FIG. 6A in FIG. 6 is selected. This reference image 202 is a non-defective image including a non-defective workpiece W. FIG. 6B in FIG. 6 is an area-identified image 204 after the paste target area 203 has been identified. The paste target area 203 is identified by extracting the workpiece W portion included in the reference image 202 shown in FIG. 6A. A method conventionally used in the field of image processing can be applied to extract the workpiece W portion. In FIG. 6B, a binary mask image is used to distinguish the paste target area 203 from other areas, with white portions representing the paste target area 203 and black portions representing other areas.

[0064] Thereafter, the process proceeds to step SB9, where it is determined whether or not to apply a rule, which will be described later, when pasting the defective portion W1 to the pasting target area 203. For example, the image processing unit 13d generates a user interface screen for selecting whether or not to apply a rule, and the display control unit 13c causes the generated user interface screen to be displayed on the display device 4. The user selects whether or not to apply a rule on the user interface screen by operating the mouse 52 or the like. If the image processing unit 13d detects that "apply" has been selected, the process proceeds to step SB10, whereas if it detects that "not apply" has been selected, the process proceeds to step SB11.

[0065] In step SB10, a pasting process is executed in which a rule is applied when pasting the defective portion W1 in the pasting target area 203. The rule can be, for example, a rule that pastes the defective portion W1 in the pasting target area 203. For example, suppose a defective product image 200 as shown in FIG. 7A of FIG. 7 is selected and a doughnut-shaped workpiece W is the inspection target. In this case, it is known that the defective portion W1 exists somewhere in the circumferential direction of the workpiece W. Therefore, as shown in FIGS. 7B and 7C of FIG. 7, on the image 204 after the pasting target area 203 has been specified, the defective portion W1 is pasted by moving it in a direction corresponding to the circumferential direction of the workpiece W. In other words, a circumferential constraint (an example of a predetermined constraint condition) is imposed on the pasting position of the defective portion W1.

[0066] Also, for example, suppose a defective product image 200 such as that shown in FIG. 8A in FIG. 8 is selected, and a linear workpiece W is the object of inspection. In this case, it is known that the defective portion W1 exists somewhere in the longitudinal direction of the workpiece W. Therefore, as shown in FIGS. 8B and 8C in FIG. 8, after the paste target area 203 has been specified, the defective portion W1 is moved in a direction corresponding to the longitudinal direction of the workpiece W and pasted on the image 204. In other words, a longitudinal constraint (an example of a predetermined constraint condition) is imposed on the paste position of the defective portion W1.

[0067] Also, suppose a defective product image 200 such as that shown in FIG. 9A in FIG. 9 is selected, and a ring-shaped workpiece W with a complex shape is the object of inspection. In this case, the defective portion W1 is located somewhere on the workpiece W, but it is difficult to simply identify it as in the cases shown in FIGS. 7 and 8. Therefore, for example, the outer and inner edges of the workpiece W are detected to identify the workpiece W, and the normal direction of the detected edges is determined, and the defective portion W1 is moved in the normal direction and pasted.

[0068] For example, as shown in FIG. 10A in Fig. 10, the defective part W1 may be pasted randomly in a portion corresponding to the workpiece W, which is the pasting target area 203. If the defective part W1 is pasted randomly, it is conceivable that a part of the defective part W1 may protrude from the portion corresponding to the workpiece W, which is the pasting target area 203, as shown in FIG. 10B in Fig. 10, but the protruding part may be trimmed by a trimming process described later.

[0069] In step SB11, which is reached when it is determined in step SB9 that the rule is not to be applied, the user pastes the defective part W1 into the paste target area 203. For example, the user operates the mouse 52 or the like to select the paste position of the defective part W1, and then performs a paste confirmation operation, thereby pasting the defective part W1 at the desired position. In step SB11, the defective part W1 may be pasted randomly into the paste target area 203.

[0070] After the paste processing of the defective portion, in step SB12, a trimming process is performed to trim the portion of the defective portion W1 that protrudes from the paste target area 203 (shown in FIG. 10B of FIG. 10). Specifically, the image processing unit 13d acquires information indicating the size, shape, and position of the paste target area 203, as well as information indicating the size, shape, and position of the pasted defective portion W1. Based on the acquired information, it is determined whether the entire defective portion W1 is located inside the paste target area 203. If the entire defective portion W1 is located inside the paste target area 203, it is determined that trimming is unnecessary. On the other hand, if a part of the defective portion W1 is located outside the paste target area 203, the part located outside is determined to be a protruding part, and this part is trimmed. As a result, the internally generated defective product image reflects the defective state of the actual workpiece, thereby improving learning efficiency.

[0071] In step SB13, the image processing unit 13d executes a process to make the boundary between the defective portion W1 pasted in the pasting process of steps SB10 and SB11 and the reference image 202 less noticeable. That is, since the image from which the defective portion W1 is extracted is different from the reference image 202, simply pasting the defective portion W1 onto the reference image 202 may make the boundary between the defective portion W1 and the reference image 202 more noticeable. Furthermore, since the position from which the defective portion W1 is extracted is different from the position to which the defective portion W1 is pasted, this may also make the boundary between the defective portion W1 and the reference image 202 more noticeable. If the boundary is more noticeable, it is not desirable for the image to be used for learning. Therefore, a blurring process is executed in step SB13. Examples of the process to make the boundary less noticeable include blurring.

[0072] The blurring process may be any blurring process, such as a GAN (Generative Adversarial Network), Deep Image Harmonization, or other rule-based processes. The blurring process is also called deep learning harmonization processing. By performing the blurring process, the edge of the defective area W1 blends neatly into the reference image 202. In this manner, the image processing unit 13d internally generates a new defective product image. This internal generation process is the image processing step. The internally generated defective product image can be stored in the storage device 19, for example.

[0073] Steps SB10 to SB13 are executed multiple times. At this time, the pasting position of the defect portion W1 is made different for each image. This allows multiple defective product images to be internally generated in which the pasting positions of the defect portion W1 are different from each other. Also, defect portions W1 with different image conversions may be pasted onto the reference image. This allows multiple defective product images with different defect portions W1 to be internally generated.

[0074] In step SB14, the display control unit 13c acquires the internally generated defective product images and displays them on the display device 4. Specifically, the display control unit 13c generates a user interface screen for displaying the internally generated defective product images. The user interface screen is provided with a display area that can display one or more internally generated defective product images in a list. This allows the user to view and confirm the internally generated defective product images. As a result, defective product images that are determined not to be used for learning can be set not to be used for learning. This setting can be set for each defective product image on the user interface screen. The information "not to be used for learning" is also stored in the storage device 19.

[0075] In step SB15, the image processing unit 13d adds identification information to the new defective product image generated to distinguish it from defective product images obtained by photographing defective workpieces. An identification flag is added to the internally generated defective product image, and the defective product image is stored in the storage device 19 together with the added information.

[0076] This completes step SA4 in the flowchart of Fig. 3. In step SA6, a setting is made so that the internally generated defective product images are not used for learning (exclusion setting), and weighting is performed if they are to be used for learning.

[0077] For example, the learning unit 13e generates a user interface screen for selecting whether or not to use the internally generated defective product images for learning, and the display control unit 13c causes the generated user interface screen to be displayed on the display device 4. The user selects whether or not to use the images on the user interface screen by operating the mouse 52 or the like. This selection information is stored in, for example, the memory 13b.

[0078] This user interface screen also allows the user to set weighting when using internally generated defective product images for learning. Internally generated defective product images differ from actually captured defective product images, and may therefore be less suitable for learning. In such cases, it may be better to lighten the weighting of learning using defective product images to which identification information has been assigned in step SB15 shown in FIG. 4 compared with the weighting of learning using defective product images captured by photographing defective workpieces. The user can freely set the weighting, and the weighting of learning using internally generated defective product images may be the same as the weighting of learning using defective product images captured by photographing defective workpieces.

[0079] In step SA7, the learning unit 13e inputs the non-defective and defective product images pre-stored in the storage device 19 and the defective product images internally generated by the image processing unit 13d to the machine learning network. Then, the process proceeds to step SA8, where the machine learning network learns the input images to generate an inference model. Steps SA7 and SA8 are learning steps.

[0080] At this time, learning is performed based on the information set in step SA6. For example, if a setting is made to use internally generated defective product images for learning, the machine learning network is also made to learn the internally generated defective product images as described above. However, if a setting is made to not use internally generated defective product images for learning, the internally generated defective product images are excluded from the learning objects and are not made to learn the machine learning network. For example, at a stage when a large number of defective product images obtained by photographing defective workpieces have been accumulated, the internally generated defective product images can be excluded from the learning objects.

[0081] In addition, the weighting set in step SA6 is also reflected in the learning of the machine learning network, so that the weighting of learning using internally generated defective product images can be made lighter than the weighting of learning using defective product images taken of defective workpieces.

[0082] If step SA3 returns NO and defective product images are not internally generated, the process proceeds to step SA5. In step SA5, images of non-defective products and images of defective products pre-stored in storage device 19 are input to the machine learning network. Thereafter, the process proceeds to step SA8, where the machine learning network learns the input images and generates an inference model.

[0083] Step SA9 stores the parameters for constructing the learned inference model in the storage device 19.

[0084] The training method for the machine learning network is not particularly limited, but the following method can be used, for example. That is, the machine learning network can be trained by minimizing the loss function. There are various definitions of loss, but the mean square error (MSE) can be given as an example.

[0085]

number

[0086] Here, T is the target anomaly map, 0 is the output image (anomaly map), n is the number of pixels in image T that are 0, and x and y are the pixel positions. Note that loss functions such as binary cross entropy can also be used. The above is merely an example.

[0087] (When visual inspection device 1 is in operation) Next, the operation of the visual inspection device 1 will be described based on the flowchart shown in Fig. 11. In step SC1 after starting, the processor 13a reads parameters and the like stored in the storage device 19 to prepare a trained machine learning network. This is the inference model generated by the learning unit 13e. In step SC2, the workpiece to be inspected is imaged by the imaging unit 3 to obtain a workpiece image. Thereafter, the process proceeds to step SC3, where the workpiece image obtained in step SC2 is input into the inference model.

[0088] Next, in step SC4, the inference model performs inference processing on the workpiece image input in step SC3. After that, in step SC5, the inference model outputs a map image as a result of the inference processing.

[0089] Thereafter, in step SC6, the quality of the workpiece is determined based on the map image output in step SC5. Steps SC2 to SC6 can be executed each time a different workpiece is used. The processing shown in this flowchart corresponds to the inspection steps executed by inspection unit 13f.

[0090] (Effects of the embodiment) As described above, according to this embodiment, defective parts can be extracted from defective product images acquired by photographing defective workpieces. The extracted defective parts are pasted onto the reference image, thereby generating new defective product images containing the defective parts. Multiple defective product images with different pasting positions of the defective parts or multiple defective product images with different image transformations for the defective parts can be automatically and quickly generated internally. In the newly generated defective product images, blurring is performed so that the boundary between the pasted defective parts and the reference image is not noticeable, resulting in natural defective product images.

[0091] By having the machine learning network learn the new defective product image and the images of good and defective products pre-stored in the storage device 19, an inference model with high defective product detection capability is generated, thereby improving the visual inspection performance. The present invention is intended to be illustrative in all respects and should not be construed as limiting. Furthermore, all modifications and variations within the scope of the claims are within the scope of the present invention. [Industrial Applicability]

[0092] As described above, the present invention can be used when inspecting the appearance of a workpiece. [Explanation of symbols]

[0093] 1. Visual inspection equipment 4 Display device (display section) 13a processor 13b Memory 13c Display control unit 13d Image processing unit 13e Learning Department 13F Inspection Department

Claims

1. An appearance inspection device that inputs a workpiece image obtained by photographing a workpiece to be inspected into a machine learning network and determines whether the workpiece is good or bad based on the input workpiece image, a storage unit that stores images of non-defective products corresponding to non-defective products and images of defective products corresponding to defective products; an image processing unit that generates a new defective product image by executing an extraction process that extracts defective parts from the defective product image stored in advance in the storage unit, an attachment process that attaches the defective parts extracted in the extraction process to a reference image including a workpiece based on a user's selection of an attachment position for the defective parts, and a process that makes the boundary between the defective parts attached in the attachment process and the reference image inconspicuous; a learning unit that generates an inference model by training a machine learning network on the non-defective product images and defective product images pre-stored in the storage unit and the new defective product images generated by the image processing unit; an inspection unit that inputs a workpiece image obtained by photographing a workpiece to be inspected into the inference model generated by the learning unit and determines whether the workpiece is good or bad based on the input workpiece image; The image processing unit identifies a work portion included in the reference image as a target area for pasting the defective portion, and after the pasting process for the defective portion, performs a trimming process to trim the portion of the defective portion that protrudes from the target area for pasting.

2. 2. The visual inspection apparatus according to claim 1, In the extraction process, the image processing unit acquires the features of the defective product image stored in the memory unit, and extracts the area having features different from the features of the surrounding area as the defective area.

3. 3. The visual inspection apparatus according to claim 2, The image processing unit is an appearance inspection device that accepts correction of the range of the extracted defective portion in the extraction process.

4. 4. The visual inspection apparatus according to claim 1, The image processing unit is an appearance inspection device that, during the extraction process, accepts a user's selection operation of the defective part from among the defective product images stored in the memory unit, and extracts the defective part based on the accepted selection operation.

5. 5. The visual inspection apparatus according to claim 1, The image processing unit is an appearance inspection device that applies image conversion to the defective portion extracted by the extraction processing and pastes the defective portion onto the reference image.

6. 6. The visual inspection apparatus according to claim 1, The visual inspection device further comprises a display control unit that causes the new defective product image generated by the image processing unit to be displayed on a display unit.

7. 7. The visual inspection apparatus according to claim 1, The image processing unit is an appearance inspection device that assigns identification information to the new defective product image generated by the image processing unit to distinguish it from defective product images taken of defective workpieces.

8. 8. The visual inspection apparatus according to claim 7, The learning unit is configured to be able to learn about defective products by excluding the defective product images to which the identification information has been assigned from learning targets.

9. 9. The visual inspection apparatus according to claim 7, The learning unit is an appearance inspection device configured to be able to lighten the weight of learning using the defective product images to which the identification information has been assigned compared to the weight of learning using defective product images photographed of defective workpieces.

10. 6. The visual inspection apparatus according to claim 1, The image processing unit is an appearance inspection device that marks defective areas on the workpiece portion included in the reference image in accordance with predetermined constraints.

11. A visual inspection method in which a workpiece image taken of an inspection target workpiece is input to a machine learning network and a quality determination of the workpiece is performed based on the input workpiece image, a storage step of storing a non-defective product image corresponding to a non-defective product and a defective product image corresponding to a defective product; an image processing step of generating a new defective product image by executing an extraction process of extracting defective parts from the defective product image previously stored in the storage step, an attachment process of attaching the defective parts extracted in the extraction process to a reference image including the workpiece based on a user's selection of an attachment position for the defective parts, and a blurring process of making the boundary between the defective parts attached in the attachment process and the reference image less noticeable; a learning step of generating an inference model by training a machine learning network on the non-defective product images and defective product images pre-stored in the storing step and the new defective product images generated in the image processing step; an inspection step of inputting a workpiece image of the workpiece to be inspected into the inference model generated in the learning step, and determining whether the workpiece is good or bad based on the input workpiece image; The image processing step includes: Identifying a work portion included in the reference image as a region to be pasted with the defective portion; after the pasting process of the defective portion, performing a trimming process of trimming a portion of the defective portion that protrudes from the pasting target area.

Images