Inspection device and inspection method

Abstract

This invention relates to an inspection apparatus and an inspection method. The inspection apparatus includes: an image acquisition unit that acquires an image of an object to be inspected; a storage unit that stores a learning model learned using a normal image representing a normal object, the learning model being used to generate a comparison image that approximates the normal image; and an inspection unit that uses the acquired image and the comparison image to determine whether the object to be inspected has any abnormalities. The learning model includes a generator that generates the comparison image from inputs of first noise and second noise, the generator comprising: a mapping network layer that performs a nonlinear transformation on the first noise to generate third noise; and a synthesis network layer that uses the second noise and the generated third noise to generate an image, and repeatedly upsamples the generated image to generate the comparison image.

Description

Technical Field

[0001] This disclosure relates to inspection devices and inspection methods. Background Technology

[0002] A device for judging superiority or inferiority is known. Based on the judgment results of the superiority or inferiority of the sample images, it calculates feature quantities from each sample image. For the calculated feature quantities, it performs learning based on neural networks to generate a benchmark for judging superiority or inferiority (e.g., Japanese Patent Application Laid-Open No. 2007-114843). Summary of the Invention

[0003] In inspection devices that use learning models, it is desirable to improve inspection accuracy.

[0004] This disclosure can be implemented in the following ways.

[0005] (1) According to one aspect of the present disclosure, an inspection apparatus is provided. The inspection apparatus includes: an image acquisition unit that acquires an image of an object to be inspected; a storage unit that stores a learning model learned using a normal image representing the normal object, the learning model being used to generate a comparison image approximating the normal image; and an inspection unit that uses the captured image and the comparison image to determine whether the object to be inspected has any abnormalities. The learning model includes a generator that generates the comparison image from inputs of first noise and second noise, the generator comprising: a mapping network layer that performs a nonlinear transformation on the first noise to generate third noise; and a synthesis network layer that uses the second noise and the generated third noise to generate an image, and repeatedly upsamples the generated image to generate the comparison image.

[0006] The inspection device based on this method can use high-resolution image data to determine whether there are any abnormalities, thereby improving the inspection accuracy of the device.

[0007] (2) Based on the inspection device described above, the learning model may also be provided without a discriminator for distinguishing between the captured image and the comparison image.

[0008] The inspection device according to this method can compare captured images with comparison images with a simple configuration, thereby improving the productivity of the inspection device.

[0009] (3) Based on the inspection device described above, the learning model may be a generative adversarial network that also has a discriminator that outputs a value representing the degree of difference between the comparison image and the captured image.

[0010] According to this inspection device, it is possible to determine whether there is an abnormality by using known configurations, captured images, and comparison images.

[0011] (4) Based on the inspection device described above, if the inspection unit determines that the captured image is not similar to the comparison image, it will re-determine and repeat the determination a predetermined number of times. The re-determination is to determine whether the inspection object has any abnormality by using a new comparison image generated by changing at least one of the first noise and the second noise.

[0012] The inspection device based on this method can suppress the omission of anomalies and improve inspection accuracy by repeatedly changing the features of the comparison images.

[0013] (5) Based on the inspection device described above, the re-determination may include: a first re-determination, which involves performing a predetermined number of re-determinations on the comparison image generated by fixing the second noise and changing the first noise; and a second re-determination, which involves performing a predetermined number of re-determinations on the comparison image generated by fixing the first noise and changing the second noise after the first re-determination has been performed.

[0014] According to the inspection device of this method, by first performing a first re-determination of first noise that alters the features in the image with a large change, compared with the case where a second re-determination is performed first, the anomaly determination can be terminated earlier.

[0015] (6) Based on the inspection device described above, the generator may further include a weight adjustment layer that changes the first noise input to the mapping network layer in such a way that the difference between the captured image and the comparison image is reduced.

[0016] According to this inspection device, by using the simple method of changing the first noise, a comparison image that is similar to the captured image can be generated without the need for generator learning during inspection.

[0017] (7) In the inspection device described above, the inspection unit may determine that the inspection object is abnormal if the cumulative difference between the gray value of the first pixel contained in the captured image and the gray value of the second pixel at the position corresponding to the first pixel in the comparison image is greater than a predetermined threshold.

[0018] According to the inspection device of this method, compared with the case of calculating the likelihood using mean square error, it is able to suppress the case where the difference value of RGB grayscale values ​​that are easily recognized by humans becomes too large.

[0019] This disclosure can also be implemented in various other ways, such as inspection apparatus, inspection method, etc. For example, it can be implemented in the form of an inspection program, a learned model, a learning method for the learned model, a manufacturing method for the part to be inspected, a manufacturing method for the inspection apparatus, a control method for the inspection apparatus, a learning program for a machine learning model, a non-transitory recording medium on which these computer programs are recorded, etc. Attached Figure Description

[0020] Hereinafter, with reference to the accompanying drawings, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described, in which the same reference numerals denote the same elements, wherein:

[0021] Figure 1 This is an illustrative diagram that schematically shows the overall structure of the inspection system.

[0022] Figure 2 This is a block diagram illustrating the internal functional structure of the inspection device.

[0023] Figure 3 It is an illustrative diagram that conceptually shows the structure of a machine learning model.

[0024] Figure 4 This is an explanatory diagram showing the detailed structure of a machine learning model.

[0025] Figure 5 This is a flowchart illustrating the inspection method performed by the inspection device.

[0026] Figure 6 This is an explanatory diagram showing a connector as an example of an object to be inspected.

[0027] Figure 7 This is an explanatory diagram showing an example of a captured image representing a normal connector.

[0028] Figure 8 This is an explanatory diagram showing an example of a captured image representing an abnormality in a connector.

[0029] Figure 9 This is an explanatory diagram showing a comparison image generated using a captured image as a normal image.

[0030] Figure 10 This is an illustrative diagram showing a comparison image generated using a captured image as an anomalous image.

[0031] Figure 11 This is an explanatory diagram showing a difference image generated by subtracting the RGB grayscale values ​​of the captured image and the comparison image.

[0032] Figure 12This is a schematic diagram illustrating the method for calculating the difference.

[0033] Figure 13 This is an explanatory diagram showing the general configuration of the machine learning model provided by the inspection device as a second embodiment. Detailed Implementation

[0034] A. First implementation method:

[0035] Figure 1 This is an explanatory diagram schematically showing the overall configuration of the inspection system 100. The inspection system 100 includes an inspection device 60, as described in the first embodiment of this disclosure, and a camera 70. The inspection device 60, in cooperation with the camera 70, uses images of the inspection object processed on the production line Ln to inspect the object for any abnormalities. The inspection device 60 utilizes a learned machine learning model. The inspection object is, for example, an automotive part WK manufactured in the previous process PR1. In this embodiment, the automotive part WK will be described using a connector as an example (described later). A connector is a component used to connect pipes that allow the flow of fuel, refrigerant, etc., in a vehicle to each other. Abnormalities to be detected may include, for example, poor connector connection, connector breakage, poor engagement of the connector's locking part, pipe breakage, etc., and may encompass many types of abnormalities. In this embodiment, because it is difficult to prepare images of inspection objects with abnormalities separately, an unsupervised learning machine learning model is used.

[0036] Camera 70 takes an image of the object to be inspected and sends the image to inspection device 60. Camera 70 includes an imaging unit 72 and a communication unit 74. Imaging unit 72 captures images of part WK during transport from previous process PR1 to subsequent process PR2. In this embodiment, imaging unit 72 captures images of part WK mounted on a vehicle. The image captured by imaging unit 72 is output to communication unit 74. Camera 70 is not limited to being independent of inspection device 60; it can also be integrated with inspection device 60.

[0037] In this embodiment, the captured image obtained by the imaging unit 72 is constructed using an RGB input image signal, which is composed of image signal components represented by R (red), G (green), and B (blue). For example, each image signal is allocated 8 bits, totaling 24 bits of information. In this embodiment, the imaging unit 72 acquires a so-called 12-megapixel image of the entire part WK, defined as 4000 pixels wide × 3000 pixels wide. The resolution of the captured image acquired by the imaging unit 72 is preferably determined by considering the inspection range required by the inspection device 60, the accuracy of defect detection, the size of the defect to be detected, and the image size that can be processed using the learning model. Furthermore, to improve the inspection accuracy and expand the inspection range of the inspection device 60, it is preferable that the captured image of the inspection object has a high resolution. For example, to inspect a part WK mounted in a car, for example, to move the camera 70 away from the car to avoid contact between the camera 70 and the car, sometimes the inspection object is photographed from a position far from the inspection object. Additionally, sometimes the shooting range is small, or sometimes small defects are detected relative to a large shooting range. In these cases, it is also preferable to examine images of the object with high resolution.

[0038] The communication unit 74 transmits captured images to the inspection device 60 wirelessly according to any communication protocol. The communication unit 74 can also receive execution commands from the inspection device 60.

[0039] In this embodiment, the inspection device 60 and the camera 70 can transmit and receive data from each other wirelessly, and can also communicate with each other. Figure 1 The information processing device PC and terminal devices PD1 and PD2 shown transmit and receive data. For example, the inspection device 60 can send inspection results to the information processing device PC and terminal devices PD1 and PD2 according to requests from them. Furthermore, the inspection device 60 can be configured as a local server in an area near the production line Ln, or as a remote server in a location far from the production line Ln. When the inspection device 60 is configured as a remote server, images captured by the camera 70 and inspection results sent to the terminal devices PD1 and PD2 and the information processing device PC are transmitted and received by the inspection device 60 via a wireless access point within the factory and a network such as an intranet or the Internet.

[0040] Figure 2 This is a block diagram showing the internal functional configuration of the inspection device 60. The inspection device 60 includes a CPU 62 as a central processing unit, a storage device 64, and a communication unit 66. The CPU 62, storage device 64, and communication unit 66 are connected to each other via a bus 61, enabling bidirectional communication. The inspection device 60 may also include a display unit, such as a liquid crystal display or a touch panel, for displaying inspection results.

[0041] The communication unit 66 is an interface for communication control between the inspection device 60 and the camera 70 via a network, used for receiving captured images for inspection purposes. In this embodiment, the communication unit 66 functions as an image acquisition unit that acquires captured images of the inspection object via the camera 70. In this embodiment, the communication unit 66 also transmits inspection results and data for displaying inspection results to the terminal devices PD1, PD2, and the information processing device PC. The communication unit 66 may also receive instruction signals from the terminal devices PD1, PD2, and the information processing device PC requesting the execution of various processes. Furthermore, when the inspection device 60 and the camera 70 are integrated, or when the inspection device 60 includes an image acquisition unit 72, the image acquisition unit 72 functions as an image acquisition unit.

[0042] Storage device 64 is, for example, RAM, ROM, or hard disk drive (HDD). Various programs for implementing the functions provided in this embodiment are stored in the HDD or ROM. Various programs read from the HDD or ROM are loaded into RAM and executed by the CPU 62. Storage device 64 includes a learning model storage unit 642 for storing machine learning models. Furthermore, image data acquired by the image acquisition unit, various images generated by the inspection unit 622, calculation results, etc., are temporarily stored in storage device 64. Optical discs, SSDs (Solid State Drives), flash memory, etc., can also be used for storage device 64.

[0043] CPU 62 functions as inspection unit 622 by executing a program stored in storage device 64. Inspection unit 622 uses a machine learning model stored in learning model storage unit 642 to check for anomalies on the inspection object.

[0044] Figure 3 This is an explanatory diagram conceptually illustrating the configuration of the machine learning model 80 stored in the learning model storage unit 642. In this embodiment, the machine learning model 80, which has been learned using normal images representing the normal inspected object, is stored in the learning model storage unit 642. The machine learning model 80 includes a weight adjustment layer 82 and a generator 84.

[0045] The generator 84 is in a state of having been sufficiently learned using captured images of normal inspection objects (hereinafter also referred to as "normal images"). The generator 84 generates image data (hereinafter also referred to as "comparison images") that approximates the captured images, using first noise Z1 and second noise Z2 as inputs as latent variables. In this embodiment, a machine learning model also known as StyleGAN is applied to the generator 84, as described later.

[0046] The first noise Z1 and the second noise Z2 are, for example, latent variables consisting of approximately 100 elements, ranging from 0 to 1. The first noise Z1 and the second noise Z2 are determined by the inspection unit 622, for example, based on uniform distribution and normal distribution through random sampling, and are input to the generator 84.

[0047] After acquiring the comparison image PR and the captured image PT acquired by the imaging unit 72, the inspection unit 622 determines whether the inspected object has any abnormalities by comparing the captured image PT with the comparison image PR. If the captured image PT and the comparison image PR are similar, the inspection unit 622 determines that the inspected object has no abnormalities. If the inspection unit 622 determines that the captured image PT and the comparison image PR are not similar, it performs a re-determination, that is, it modifies at least one of the first noise Z1 and the second noise Z2 to generate a new comparison image PR, and then determines whether the inspected object has any abnormalities again.

[0048] The weight adjustment layer 82 adjusts the weights of the first noise Z1 input to the generator 84. In this embodiment, the weight adjustment layer 82 applies a fully connected layer utilizing an activation function. During re-determination, the weight adjustment layer 82 modifies the first noise Z1 in a manner that reduces the difference between the captured image PT and the comparison image PR. As a result, the generator 84 does not need to relearn during inspection, and can generate a comparison image PR that is more similar to the captured image PT each time a re-determination is performed.

[0049] Figure 4 This is an explanatory diagram showing the detailed structure of machine learning model 80. (As shown...) Figure 4 As shown, the StyleGAN model is used in generator 84. Therefore, compared to DCGAN, which generates image data at a resolution of 64 pixels x 64 pixels, a high-resolution comparison image PR can be generated.

[0050] The generator 84 includes a mapping network unit 842 and multiple synthesis network units 844. The mapping network layer 842, for example, has multiple fully connected layers (FClayers) of up to eight layers. The mapping network layer 842 generates a third noise W1 as a latent space by performing a nonlinear transformation on the first noise Z1, which is a latent variable, using the eight fully connected layers. The generated third noise W1 is then input to the synthesis network layer 844.

[0051] The synthesis network layer 844 generates a comparison image PR based on a fixed value CV, which is a 4x4x512 tensor. The synthesis network layer 844 upsamples the image multiple times, progressively increasing the resolution to generate a high-resolution comparison image PR. Within the synthesis network layer 844, convolutional processing layers and a normalization layer (AdaIN: Adaptive Instance Normalization) AL are provided for each upsampling resolution. Each time the synthesis network layer 844 performs upsampling, style information and noise information are introduced. Specifically, after the convolutional processing layer, a second noise Z2 is input as probabilistically variable noise information, and a third noise W1 generated by the mapping network layer 842 as the latent space is introduced via the normalization layer AL as multidimensional style information. The third noise W1 as the latent space and the first noise Z1 used to generate the third noise W1 are factors that may affect large features in the image, while the second noise Z2, as noise information, is a factor that may affect detailed features.

[0052] Figure 5 This is a flowchart illustrating the inspection method performed by the inspection device 60. This process begins when part WK, delivered from the previous process PR1, arrives at the inspection process. Whether part WK has arrived at the inspection process can be determined, for example, by using image recognition of part WK that has arrived at the inspection process using a camera 70 or the like.

[0053] In step S10, the inspection device 60 acquires the captured image PT. In this embodiment, the captured image PT is acquired by the imaging unit 72 of the camera 70 capturing an image of the part WK, which is the object of inspection. The captured image PT acquired by the imaging unit 72 is transmitted to the inspection device 60 via the communication unit 74. The inspection device 60 acquires the captured image PT via the communication unit 66.

[0054] Figure 6 This is an explanatory diagram showing connector 30 as an example of the object of inspection. (See diagram below.) Figure 6 As shown, the connector 30 includes a main body 36. A fitting portion 38 is provided in the main body 36. By pressing the fitting portion 38 toward the connection portion of the first conduit 32 and the second conduit 34 with the first conduit 32 and the second conduit 34 inserted into the main body 36, the first conduit 32 and the second conduit 34 are fixed in a connected state.

[0055] Figure 7 This is an explanatory diagram showing an example of a captured image PTg representing a normal image of a normal connector 30. Figure 8 This is an explanatory diagram showing an example of a captured image PTn representing an abnormality in connector 30. (See diagram below.) Figure 7As shown in the captured image PTg, the connector 30 is shown in its normal state, with the fitting portion 38 fully inserted into the connection portion between the first conduit 32 and the second conduit 34, and the first conduit 32 and the second conduit 34 fixed. Figure 8 As shown in the captured image PTn, the connector 30 is shown in a state where the mating portion 38 is not fully inserted and protrudes from the body 36. The captured image PTn shows the protruding portion P1 of the mating portion 38 protruding from the body 36 and the notch portion P2 at a position different from that of the connector 30 in its normal state. In the following description, an example will be used where the inspection device 60 determines that the insufficient insertion of the mating portion 38 of the connector 30 is abnormal.

[0056] In step S20, the inspection unit 622 inputs randomly sampled first noise Z1 and second noise Z2 into the generator 84 of the machine learning model 80 to generate a comparison image PR. The first noise Z1 is input into the generator 84 via the weight adjustment layer 82, and then... Figure 4 The mapping network layer 842 shown is nonlinearly transformed into a third noise W1 as the latent space. The synthesis network layer 844 generates an image based on a fixed value CV, while importing the third noise W1 as style information and the second noise Z2 as noise information, and repeatedly upsampling the image to generate a 1024 pixel × 1024 pixel comparison image PR.

[0057] Figure 9 This is an explanatory diagram showing a comparison image PRg generated using the captured image PTg as a normal image. Figure 10 This is an explanatory diagram showing a comparison image PRn generated using the captured image PTn as an anomalous image. (As shown...) Figure 9 As shown, the comparison image PRg can be understood as being generated in a state that approximates the normal image. Furthermore, Figure 10 The comparison image PRn shown is an example of a comparison image generated after performing re-decision until the upper limit number of times is reached. Figure 10 As shown, this can be understood as generating regions P3 and P4 in the comparison image PRn. Region P3 is used to... Figure 8 The protruding portion P1 in the captured image PTn shown is generated in a manner that approximates a normal image, and the generated portion P4 is generated in a manner that approximates the notched portion P2 in a normal image.

[0058] In step S30, the inspection unit 622 compares the captured image PT with the generated comparison image PR. In this embodiment, the inspection unit 622 compares the cumulative difference between the total RGB grayscale values ​​of the pixels in the captured image PT and the total RGB grayscale values ​​of the pixels in the comparison image PR at the same position as the pixels in the captured image PT. The pixels in the captured image PT are also referred to as "first pixels," and the pixels in the comparison image PR are also referred to as "second pixels."

[0059] Figure 11 This is an explanatory diagram showing a difference image CP generated using the difference between the RGB grayscale values ​​of the captured image PTn and the comparison image PRn. According to... Figure 11 This can be understood as follows: in the differential image CP, the parts with large differences in RGB gray values ​​include, for example, the difference part P5 which is the difference between the protruding part P1 and the generating part P3, and the difference part P6 which is the difference between the gap part P2 and the generating part P4.

[0060] Figure 12 This is a schematic diagram illustrating the method for calculating the difference between the captured image PTn and the comparison image PRn. Figure 12 In order to make the technology easier to understand, the number of pixels is shown as less than the actual number of pixels in the image. Figure 12 Table TB1 conceptually shows the RGB grayscale values ​​of each pixel in the captured image PTn. Table TB2 conceptually shows the RGB grayscale values ​​of each pixel in the comparison image PRn, and Table TB3 conceptually shows the RGB grayscale values ​​of each pixel in the difference image CP. The values ​​in Table TB1 represent the sum of the RGB grayscale values ​​of each pixel in the captured image PTn. Similarly, the values ​​in Table TB2 represent the sum of the RGB grayscale values ​​of each pixel in the comparison image PRn. The values ​​in Table TB3 represent the difference between the sum of the RGB grayscale values ​​of each pixel in Table TB1 and Table TB2.

[0061] As shown in Table TB3, the inspection unit 622 calculates the difference between the total RGB grayscale values ​​of the captured image PTn and the total RGB grayscale values ​​of the generated comparison image PRn for each pixel. If the captured image PTn is an anomalous image, the comparison image is repeatedly re-evaluated after being generated by the generator 84, thereby being regenerated in a manner approximating the captured image PTn, which is an anomalous image. Therefore, as shown in Table TB3, for example, the difference between the captured image and the comparison image remains as a difference in RGB grayscale values. The inspection unit 622 accumulates the calculated RGB grayscale values ​​of each pixel across all pixels. Figure 12 In the example of Table TB3, the cumulative value of the RGB grayscale is 11.

[0062] return Figure 5 In step S40, the inspection unit 622 uses the cumulative value of the calculated RGB grayscale values ​​of all pixels to determine whether the object being inspected is abnormal. In this embodiment, the inspection unit 622 determines the abnormality of the object by comparing the cumulative value of the calculated RGB grayscale values ​​of all pixels with a predetermined threshold stored in the storage device 64. If the cumulative value is lower than the threshold (S40: Yes), the process moves to step S42, and the inspection unit 622 determines that the object being inspected is not abnormal and ends the process. If the cumulative value is higher than the threshold (S40: No), the process moves to step S50.

[0063] In step S50, the inspection unit 622 determines whether the number of inspections N has reached a predetermined upper limit. The number of inspections is equivalent to the number obtained by adding 1 to the number of re-judgments. The upper limit of the number of inspections N can be arbitrarily set based on the cycle time requested by the inspection device 60, etc., and in this embodiment, it is set to 100 times. If the number of inspections N reaches 100 times (S50: Yes), the process moves to step S52, where the inspection unit 622 determines that the inspected object is abnormal and ends the process. The case where the number of inspections N reaches 100 times is equivalent to determining that none of the 100 comparison images PR generated by changing the first noise Z1 and the second noise Z2 are similar to the captured image PT. If the number of inspections N is less than 100 times (S50: No), the process moves to step S60.

[0064] In step S60, the inspection unit 622 determines whether the number of inspections N has reached a predetermined number of switching operations. The number of switching operations is a threshold used to determine whether to switch the combination method of the first noise Z1 and the second noise Z2. The number of switching operations can be arbitrarily set; in this embodiment, the number of switching operations is set to half of the upper limit, i.e., 50 times. If the number of inspections N is less than 50 times (S60: Yes), the process moves to step S62, where the inspection unit 622 fixes the second noise Z2 to the value input in step S20 and re-determines the first noise Z1 through random sampling. If the number of inspections N is 50 times or more (S60: No), the process moves to step S64, where the inspection unit 622 fixes the first noise Z1 to the value at the time point when the number of inspections N is 50 and re-determines the second noise Z2 through random sampling. In the re-determination, the re-determination of the newly generated comparison image PR using the fixed second noise Z2 and the change of the first noise Z1 is also called the "first re-determination," and the re-determination of the newly generated comparison image PR using the fixed first noise Z1 and the change of the second noise Z2 is also called the "second re-determination." After the first noise Z1 and the second noise Z2 are newly determined in step S62 or step S64, the inspection unit 622 increases the inspection count N by 1 and returns to step S20.

[0065] As described above, the inspection apparatus 60 of this embodiment includes: an image acquisition unit that acquires an image PT of an inspection object; a storage device 64 that stores a machine learning model 80 learned using a normal image representing a normal inspection object, the machine learning model 80 being used to generate a comparison image PR that approximates the normal image; and an inspection unit 622 that uses the image PT and the comparison image PR to determine whether the inspection object has any abnormalities. The machine learning model 80 includes a generator 84 that generates the comparison image PR using first noise Z1 and second noise Z2 as inputs. The generator 84 includes: a mapping network layer 842 that performs a nonlinear transformation on the first noise Z1 to generate a third noise W1; and a synthesis network layer 844 that uses the second noise Z2 and the generated third noise W1 to generate an image, and repeatedly upsamples the generated image to generate the comparison image PR. The generator 84 of the machine learning model 80 includes: a mapping network layer 842, which uses StyleGAN to perform a nonlinear transformation on the input first noise Z1 to generate a third noise W1; and a synthesis network layer 844, which uses the second noise Z2 and the third noise W1 to generate an image and repeatedly performs upsampling to increase the resolution of the generated image, thereby generating a comparison image PR. Therefore, in the inspection device 60 utilizing the machine learning model 80, by employing a StyleGAN network construction on the generator 84, it is possible to determine the presence or absence of anomalies using high-resolution image data.

[0066] According to the inspection apparatus 60 of this embodiment, the machine learning model 80 does not have a discriminator 86 for distinguishing between the captured image PT and the comparison image PR. Therefore, the captured image PT and the comparison image PR can be compared with a simple configuration, which can improve the productivity of the inspection apparatus 60.

[0067] According to the inspection apparatus 60 of this embodiment, when the inspection unit 622 determines that the inspection object is abnormal, it repeatedly performs a re-determination a predetermined number of times. This re-determination uses a new comparison image PR generated by changing at least one of the first noise Z1 and the second noise Z2 to determine whether the inspection object is abnormal. By repeatedly changing the characteristics of the comparison image PR, erroneous determination of abnormalities can be suppressed, and inspection accuracy can be improved.

[0068] According to the inspection apparatus 60 of this embodiment, the re-determination performed by the inspection unit 622 includes: a first re-determination, which involves performing a predetermined number of switching operations on a comparison image PR generated by using a fixed first noise Z1 and changing a second noise Z2; and a second re-determination, which involves performing a re-determination on a comparison image PR generated by using a fixed second noise Z2 and changing a first noise Z1 after the first re-determination, until a predetermined upper limit number of re-determinations is reached. By performing re-determination by generating a new comparison image PR by changing only one of the first noise Z1 and the second noise Z2, it is possible to suppress the situation where the features of the captured image PT deviate significantly from the features of the comparison image PR during re-determination. In addition, by performing a first re-determination after changing the first noise Z1, which changes the large features in the changed image, and then performing a second re-determination after changing the second noise Z2, which changes the detailed features, the probability of determining that there are no abnormalities within the number of switching operations is increased, the abnormality determination can be ended earlier, and the productivity of the inspection process can be improved.

[0069] According to the inspection apparatus 60 of this embodiment, the generator 84 further includes a weight adjustment layer 82, which modifies the first noise Z1 input to the mapping network layer 842 in a manner that reduces the difference between the captured image PT and the comparison image PR. Therefore, during the inspection process based on the inspection apparatus 60, by utilizing this simple method of modifying the first noise Z1, a comparison image PR that is similar to the captured image PT can be generated without the need for learning the generator 84 during the inspection.

[0070] According to the inspection apparatus 60 of this embodiment, the inspection unit 622 calculates the difference between the RGB grayscale value of the first pixel contained in the captured image PT and the RGB grayscale value of the second pixel at the position corresponding to the first pixel in the comparison image PR. If the cumulative value of the calculated difference in RGB grayscale values ​​is greater than a predetermined threshold, the inspection object is determined to be abnormal. Therefore, for example, compared to calculating the difference value using the mean squared error (MSE) of the RGB grayscale values ​​at each pixel of the captured image PT and the comparison image PR, it is possible to suppress the situation where the difference in RGB grayscale values, which is easily recognized by humans, becomes too large. By appropriately taking into account the tonal variations of the RGB grayscale values, anomalies are determined, and inspection results that are approximately equivalent to those of human inspection can be obtained.

[0071] B. Second implementation method:

[0072] Figure 13 This is an explanatory diagram showing a schematic configuration of the machine learning model 80b included in the inspection device 60 as a second embodiment. (See diagram below.) Figure 13As shown, machine learning model 80b is a model that can be used as an anomaly detection method utilizing Generative Adversarial Networks (GANs). Machine learning model 80b can also be called AnoGAN (Anomaly Detection with Generative Adversarial Networks). Machine learning model 80b differs from machine learning model 80 shown in the first embodiment in that it also includes a discriminator 86 and a decision 88. Furthermore, in this embodiment, the inspection unit 622 uses the output from the decision 88 to determine whether an anomaly exists.

[0073] The discriminator 86 is in a state where it has been sufficiently learned to distinguish between a normal image representing a normal inspection object and the generated comparison image PR. The discriminator 86 outputs a likelihood representing the degree of difference between the comparison image PR and the captured image PT. After acquiring the comparison image PR and the captured image PT acquired by the imaging unit 72, the discriminator 86 compares the captured image PT with the comparison image PR, and outputs the likelihood obtained by quantifying the degree of difference between the comparison image PR and the captured image PT to the determiner 88. In this embodiment, the likelihood is a value representing the probability that the captured image PT is abnormal; for example, a high likelihood value indicates that the inspection object is abnormal. In this embodiment, the discriminator 86, for example, calculates the sum of a first difference value and a second difference value as the likelihood output. The first difference value is the difference between the captured image PT and the comparison image PR, and the second difference value is the difference between the output value of the intermediate layer when the comparison image PR is input to the discriminator 86 and the output value of the intermediate layer when the captured image PT is input to the discriminator 86. However, when the likelihood represents the probability value of a normal product, it can also be detected as an anomaly if the value is low.

[0074] The determiner 88 uses the likelihood output from the discriminator 86 to determine whether the inspected object has an anomaly. In step S40, the determiner 88 may, if the likelihood is lower than a predetermined threshold, consider the captured image PT and the comparison image PR to be similar and move to step S42; if the likelihood is higher than the threshold, consider the captured image PT and the comparison image PR to be dissimilar and move to step S50. The anomaly determination result obtained by the determiner 88 of the machine learning model 80b is output to the inspection unit 622, which determines whether the inspected object has an anomaly based on the result of the determiner 88. Furthermore, in the machine learning model 80b, the generator 84 and the discriminator 86 may not perform learning during the inspection process.

[0075] According to the inspection apparatus 60 of this embodiment, the machine learning model 80b is a generative adversarial network that also includes a discriminator 86 that outputs a value representing the degree of difference between the comparison image PR and the captured image PT. Therefore, it is possible to determine whether there is an anomaly using the captured image PT and the comparison image PR using a known configuration.

[0076] C. Other implementation methods:

[0077] (C1) In the first embodiment described above, the object of inspection was described using a connector, which is a part WK used in automobiles, as an example. In contrast, the object of inspection is not limited to connectors used in automobiles, but can also be various parts included in automobiles, such as the automobile body and other components. The object of inspection is not limited to automobile parts, but can be parts used in various moving bodies such as rail vehicles, aircraft, and ships, or various manufactured items other than moving bodies.

[0078] (C2) In the first embodiment described above, an example was shown where the inspection unit 622 calculates the difference between the total RGB grayscale values ​​of the captured image PTn and the total RGB grayscale values ​​of the generated comparison image PRn for each pixel. Alternatively, the inspection unit 622 can compare the difference between the total RGB grayscale values ​​of the captured image PTn and the total RGB grayscale values ​​of the generated comparison image PRn with a predetermined threshold, accumulate the number of pixels whose difference exceeds the threshold, and determine an anomaly if the number of such pixels is above a predetermined value, and if it is below the predetermined value, determine no anomaly. Furthermore, the inspection unit 622 can also use the difference between the R, G, and B grayscale values ​​calculated for each pixel in the captured image PT and the comparison image PR, and then accumulate the calculated differences between the R, G, and B grayscale values. The inspection unit 622 can also use the mean squared error (MSE) of the RGB grayscale values ​​in each pixel of the captured image PT and the comparison image PR to calculate the likelihood.

[0079] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, technical features in embodiments corresponding to the technical features in the various methods described in the Summary of the Invention section can be appropriately replaced or combined to solve some or all of the above-mentioned problems, or to achieve some or all of the above-mentioned effects. In addition, technical features that are not described as essential features in this specification can be appropriately deleted.

Claims

1.An inspection apparatus comprising: an image-captured-image acquisition unit that acquires an image captured of an inspection target; a storage device that stores a learning model learned using a normal image representing a normal inspection target, the learning model being used to generate a comparison image approximating the normal image; and an inspection unit that determines whether or not the inspection target is abnormal using the image captured and the comparison image, the learning model comprising a generator that generates the comparison image through input of first noise and second noise, the generator comprising: a mapping network layer that generates third noise by performing a non-linear transformation on the first noise; and a synthesis network layer that generates an image using the second noise and the generated third noise, and repeatedly performs upsampling of the generated image a plurality of times, thereby generating the comparison image. 2.The inspection apparatus according to claim 1, wherein the learning model does not include a discriminator used to discriminate between the image captured and the comparison image. 3.The inspection apparatus according to claim 1, wherein the learning model is a generative adversarial network further including a discriminator that outputs a likelihood representing a degree of difference of the comparison image with respect to the image captured. 4.The inspection apparatus according to claim 1, wherein the inspection unit repeatedly performs re-determination a predetermined number of times in a case where it is determined that the image captured and the comparison image are not approximated, the re-determination being determination of whether or not the inspection target is abnormal using a new comparison image generated by changing at least one of the first noise and the second noise. 5.The inspection apparatus according to claim 4, wherein the re-determination includes: first re-determination of the re-determination using the comparison image generated by fixing the second noise and changing the first noise a predetermined number of times; and second re-determination of the re-determination using the comparison image generated by fixing the first noise and changing the second noise a predetermined number of times after the first re-determination. 6.The inspection apparatus according to claim 1, wherein the generator further includes a weight adjustment layer that changes the first noise input to the mapping network layer in such a manner that a difference between the image captured and the comparison image becomes smaller. 7.The inspection apparatus according to claim 1, wherein the inspection unit determines that the inspection target is abnormal in a case where a cumulative value of differences between a gray value of a first pixel included in the image captured and a gray value of a second pixel corresponding to the first pixel in the comparison image is greater than a predetermined threshold value. 8.An inspection method in which an image captured of an inspection target is acquired, the generator includes a mapping network layer that generates third noise by performing a non-linear transformation on the first noise and a synthesis network layer that generates an image using the second noise and the generated third noise, and repeatedly performs upsampling of the generated image a plurality of times, thereby generating the comparison image, and the inspection unit determines whether or not the inspection target is abnormal using the image captured and the comparison image. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The first noise and the second noise are input to the generator to generate a comparison image, wherein ​ The captured image and the comparison image are used to determine whether the inspection object is abnormal or not.

Images