Anomaly detection system and anomaly detection method

Abstract

This invention is an anomaly detection system, comprising: an acquisition unit that acquires an image of an object to be inspected; a restoration image generation unit that inputs the image of the object to be inspected into a learning model and generates a restoration image, wherein the learning model is formed by using and learning from images of normal objects to be inspected as teacher data; and a detection unit that uses the restoration image and a normal restoration image to determine whether the object to be inspected in the image of the object to be inspected is abnormal, wherein the normal restoration image is a restoration image generated by inputting an image of a normal object to the learning model.

Description

Technical Field

[0001] This invention relates to an anomaly detection system and anomaly detection method. Background Technology

[0002] Previously, as a method for detecting anomalies in inspection objects through image analysis, a known approach involves learning from images of acceptable products without anomalies and images of non-accepting products containing anomalies to generate a model, and then using this model to detect anomalies in the inspection object. However, depending on the inspection object, sometimes the frequency of non-accepting products is low, making it impossible to collect sufficient images of non-accepting products.

[0003] Therefore, for example, Japanese Patent Application Publication No. 2020-160997 discloses a technique that uses a neural network learned by using an image of a qualified product as a learning image to generate a restored image (simulated image) based on an image of an inspection object, and detects abnormalities in the inspection object based on the difference between the image of the inspection object and the restored image. Summary of the Invention

[0004] Here, the inventors discovered that in the method of detecting anomalies by using the difference between such an inspection object image and the restored image, it is easy to produce a problem such as detecting an inspection object that is originally a qualified product as an anomaly, i.e., so-called over-detection.

[0005] For example, when inspecting images of objects that deviate from the shape of qualified products, such as weld marks formed by welding, or images of objects whose position or orientation deviates from the image due to the shooting environment of the object, there is a problem that over-inspection may occur because of these deviations, which may result in qualified objects being detected as abnormal.

[0006] The present invention was made in view of the following circumstances, and provides an anomaly detection system and anomaly detection method that suppress the generation of over-detection and improve the accuracy of anomaly detection.

[0007] An anomaly detection system according to one aspect of the present invention has:

[0008] The acquisition department acquires images of the objects to be inspected.

[0009] The image restoration generation unit inputs the image of the object under inspection into the learning model and generates a restored image. The learning model is formed by using and learning from images of normal objects under inspection as teacher data.

[0010] The detection unit uses the restored image and the normal restored image to determine whether the object to be inspected in the image is abnormal. The normal restored image is a restored image generated by inputting an image of a normal object to the learning model.

[0011] With this structure, even if an over-detection occurs due to deviations such as brightness, the over-detection can be suppressed in previous methods that used the difference between the image of the object being inspected and the restored image. This is because the restored image generated from the image of the object being inspected will be similar to the normal restored image generated from the image of the normal object being inspected, thus improving the accuracy of anomaly detection.

[0012] In the anomaly detection system described above, the detection unit uses the similarity between the restored image and the normal restored image to determine whether the object to be inspected in the image is abnormal. The normal restored image is a restored image generated by inputting an image of a normal object to the learning model.

[0013] This structure enables the determination of whether an anomaly is present with high accuracy, thus suppressing over-detection and improving the accuracy of anomaly detection.

[0014] In the anomaly detection system described above, there is also a pre-detection unit. The pre-detection unit uses the difference between the image of the object to be inspected and the restored image to determine whether the object to be inspected in the image of the object to be inspected is abnormal. The detection unit then makes a further determination on whether the object to be inspected that was determined to be abnormal by the pre-detection unit is indeed abnormal.

[0015] With this structure, the pre-detection unit performs differential judgments using minimal computational processing, and the detection unit only re-judges the objects judged as abnormal, thus shortening the time required for abnormality handling.

[0016] In the anomaly detection system described above, the learning model is a learning model that uses a generative adversarial network.

[0017] This structure improves the reproducibility of the restored image generated by the restored image generation unit, thereby increasing the accuracy of anomaly detection.

[0018] In the anomaly detection system described above, the object of inspection is weld marks formed on the component through welding.

[0019] With this structure, in particular, in the anomaly detection of weld marks, which are prone to over-detection due to deviations in shape, over-detection can be suppressed, thereby improving the accuracy of anomaly detection.

[0020] In the anomaly detection system described above, the image of the object to be inspected is an image captured during production line operations.

[0021] With this structure, in particular, in anomaly detection using images taken in production lines where over-detection is prone to occur due to deviations or biases in the position or orientation of the inspected object, over-detection can be suppressed, thereby improving the accuracy of anomaly detection.

[0022] An anomaly detection method according to one aspect of the present invention includes: an acquisition step, acquiring an image of an object to be inspected; a restoration image generation step, inputting the image of the object to be inspected into a learning model and generating a restoration image, wherein the learning model is learned by using and learning from images of normal objects to be inspected as teacher data; and a detection step, using the restoration image and a normal restoration image to determine whether the object to be inspected in the image of the object to be inspected is abnormal, wherein the normal restoration image is a restoration image generated by inputting images of normal objects to the learning model.

[0023] According to the present invention, an anomaly detection system and anomaly detection method are provided that can suppress the generation of over-detection and improve the detection accuracy of anomalies. Attached Figure Description

[0024] Hereinafter, with reference to the accompanying drawings, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described, in which the same reference numerals denote the same elements.

[0025] Figure 1 This is a block diagram showing an overview of the hardware configuration of the anomaly detection system according to the first embodiment.

[0026] Figure 2 This is a block diagram illustrating the functional configuration of the anomaly detection system according to the first embodiment.

[0027] Figure 3 This is a flowchart illustrating the anomaly detection process of the anomaly detection system according to the first embodiment.

[0028] Figure 4 It is an illustrative diagram representing the captured image, restored image, and differential image of a normal inspection object with no deviations in brightness, etc.

[0029] Figure 5 It is an illustrative diagram representing the captured image, restored image, and differential image of an inspection object that has deviations in brightness, etc., but is within the normal range.

[0030] Figure 6 It is an explanatory diagram that schematically represents the captured images, restored images, and differential images of an abnormal inspection object containing non-conforming parts.

[0031] Figure 7 It is an explanatory diagram that schematically represents the restored image of the object being inspected. Detailed Implementation

[0032] Hereinafter, specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments. Furthermore, for clarity, the following description and drawings have been appropriately simplified.

[0033] <First Implementation Method>

[0034] First, the anomaly detection system of the first embodiment will be described. Figure 1 This is a schematic block diagram illustrating the hardware structure of the anomaly detection system according to this embodiment.

[0035] like Figure 1 As shown, the anomaly detection system 100 has computer resources typical of information processing devices such as personal computers. Specifically, it includes a CPU (Central Processing Unit) 101, ROM (Read Only Memory) 102, RAM (Random Access Memory) 103, HDD (Hard Disk Drive) 104, a communication interface (I / F) 105, and an input / output interface (I / F) 106. Furthermore, these components are communicatively connected to each other via a data bus 107. While this description uses an example of anomaly detection system 100 implemented by a typical information processing device, some or all of these functions can also be implemented in external devices such as edge computing or cloud computing.

[0036] CPU 101 is the microprocessor of the integrated control anomaly detection system 100. Specifically, it reads various control programs stored in ROM 102 and HDD 104 that are executed in this embodiment, and executes these programs loaded on RAM 103. Here, HDD 104 may be used as a storage device instead of the storage device, or an SSD (Solid State Drive) may be further provided on top of it.

[0037] The communication interface 105 enables communication between the anomaly detection system 100 and external devices. In this embodiment, an image of the object being inspected is received from an external device such as an image capturing device. Here, communication can be achieved using various communication technologies, whether wireless or wired.

[0038] The input / output interface 106 allows the anomaly detection system 100 to input and output to external devices. For example, it may include an output device such as a display showing anomaly detection information, or an input device for operator use.

[0039] Next, the functional structure of the anomaly detection system of this embodiment will be explained. Figure 2 This is a block diagram illustrating the functional structure of the anomaly detection system in this embodiment.

[0040] like Figure 2 As shown, the anomaly detection system 100 includes, as a functional structure, an acquisition unit 201, a restored image generation unit 202, a pre-detection unit 203, and a detection unit 204. These functions are executed by the CPU 101 executing various control programs stored in the ROM 102, etc. Furthermore, some or all of the functions of the anomaly detection system 100 can be implemented using hardware circuitry.

[0041] The acquisition unit 201 acquires images from an external device. Specifically, when detecting an anomaly in an object to be inspected, it acquires an image of the object to be inspected and a normal image of a normal object to be inspected. Here, a normal image refers to an image of an object to be inspected that is confirmed to be free of an anomaly. The acquired image of the object to be inspected is output to the restoration image generation unit 202 and the pre-detection unit 203, while the acquired normal image is output to the restoration image generation unit 202. Furthermore, while an example of acquiring images from an imaging device that is an external device is described here, it can also be implemented by including the imaging device in the anomaly detection system.

[0042] The restored image generation unit 202 inputs the inspection object image and the normal image output from the acquisition unit 201 into a learning model that has been pre-learned using the normal image as teacher data, thereby generating a restored image that restores these captured images. The learning method of this learning model will be described in detail later. The restored image that restores the inspection object image is output to the pre-detection unit, and the normal restored image that restores the normal image is output to the detection unit.

[0043] The pre-detection unit 203 uses the difference between the inspection object image output from the acquisition unit 201 and the restored image output from the restored image generation unit 202 to determine whether the inspection object is abnormal. Specifically, firstly, these difference images are generated by subtracting the pixel values ​​of the restored image from the pixel values ​​of the inspection object image. Next, it is determined whether the total value of the pixel values ​​contained in the generated difference images is greater than a predetermined threshold; if it is greater, it is determined to be abnormal. Then, the restored image of the inspection object determined to be abnormal is output to the detection unit. Furthermore, here, the difference image is generated by subtracting the pixels of the restored image at the corresponding position from each pixel of the inspection object image, but the difference image can also be generated using the difference of each group containing multiple pixels.

[0044] The detection unit 204 uses the restored image of the object to be inspected, which was determined to be abnormal and output by the pre-detection unit 203, and the restored image (i.e., the normal restored image) restored from the normal image output by the restored image generation unit 202, to re-determine whether the object to be inspected is abnormal. Specifically, firstly, the similarity of these images is calculated. For example, the similarity can be calculated using various similarity calculation methods, such as normalized correlation (NCC) or shape pattern matching based on edge features. It is determined whether the calculated similarity is lower than a predetermined threshold, and if it is, it is determined to be abnormal. Furthermore, here, the similarity between the restored image of the object to be inspected and the normal restored image (which is a restored image of a normal image) is calculated to determine whether it is abnormal. However, similar to the method based on the pre-detection unit, a difference image can be generated based on these images, and whether it is abnormal can be determined based on the pixel values ​​contained in the difference image. In this way, the detection unit re-determines whether it is abnormal and outputs the abnormality detection information to the input / output interface 106.

[0045] Here, when comparing the anomaly detection methods of the pre-detection unit 203 and the detection unit 204 in this embodiment, the anomaly detection method based on the pre-detection unit 203 can make a judgment simply by obtaining the difference between the compared images. Therefore, compared to the method of calculating the similarity of the detection unit 204, the judgment accuracy is lower, but less computation is required, and the processing time is shorter. Therefore, in this embodiment, by using the pre-detection unit 203 to pre-determine whether the inspection object is abnormal, and by using the detection unit 204 to re-determine only the inspection objects that have been determined to be abnormal, over-detection can be suppressed, and the time required for overall inspection can be shortened.

[0046] Furthermore, here, an example with a pre-detection unit 203 is described as the functional structure of the anomaly detection system 100, but a functional structure without a pre-detection unit 203 is also possible. In this case, the restored image of the object to be inspected and the normal restored image of the restored normal image generated by the restored image generation unit 202 are directly output to the detection unit 204. Then, the detection unit 204 uses the restored image of the object to be inspected and the normal restored image of the restored normal image to determine whether the object to be inspected is an anomaly.

[0047] Next, the learning method of the learning model in this embodiment will be explained. In this embodiment, as a pre-stage of performing anomaly detection processing, the learning model of the neural network provided by the restoration image generation unit 202 is learned.

[0048] When implementing neural network learning, the acquisition unit 201 acquires normal images of the inspection objects, i.e., normal images, as teacher data. Alternatively, the acquisition unit 201 may acquire a small number of images of inspection objects containing anomalies, which are less than a predetermined proportion relative to the number of normal images acquired as teacher data.

[0049] Using the acquired teacher data, a generative adversarial network (GAN) is implemented as the learning model of the restored image generation unit 202. Specifically, the restored image generation unit 202 has a generator and a discriminator. The generator has a neural network that generates simulated data based on the acquired teacher data, and the discriminator has a neural network that judges the authenticity of the simulated data. The restored image generation unit 202 uses the teacher data and the simulated data generated by the generator to enable the generator and discriminator to learn. In this way, by using a generative adversarial network, the reproducibility of the restored image can be improved. In this embodiment, a restored image is generated by inputting an image of the object to be inspected into the learning model that has learned in this way.

[0050] Next, the anomaly detection process of the anomaly detection system 100 of this embodiment will be described. Figure 3 This is a flowchart of the anomaly detection process in this embodiment. This process begins after the learning model described above has completed its learning. Furthermore, it is preferable to pre-store the restored normal image (which has been restored to a normal image) in the anomaly detection system 100.

[0051] In this instruction manual, Figures 4 to 7 The illustrated diagram (hereinafter referred to as the image) is used as an example to illustrate the object of inspection, and weld marks formed on various components such as parts by welding are used as an example. Specifically, Figure 4 (a) is an image of a normal weld mark without any deviations in brightness or other characteristics. Figure 5 (a) is an image of the weld mark on part 5A, which appears normal but has a different color due to variations in brightness, etc. Figure 6 (a) is an image of an abnormal weld mark containing the defective part 6A.

[0052] First, in S11, the acquisition unit 201 acquires images captured by an external device. Figure 4 (a) Figure 5 (a) Figure 6 Image of the object to be inspected shown in (a).

[0053] Next, in S12, the restored image generation unit 202 inputs the inspection object image output from the acquisition unit 201 into the pre-learned learning model, thereby generating a restored image that restores these captured images. Specifically, Figure 4 (b) Figure 5 (b) Figure 6 (b) are respectively the Figure 4 (a) Figure 5 (a) Figure 6 (a) The image is input into the learning model to generate the restored image. Here, as mentioned above, this learning model is developed by learning from normal images as teacher data; therefore, the input captured image is restored to an image close to a normal image. Thus, in Figure 5 (a) Figure 6 In image (a), the areas with different colors (5A) and the defective areas (6A) are identified as being used in the reconstructed image. Figure 5 (b) Figure 6 (b) were not restored. On the other hand, Figure 5 The image in (a) contains, except for the region 5A with a different local color, the same as... Figure 4 Image (a) is also an image of a normal weld mark; therefore, the restored image... Figure 5 (b) The restored image becomes the same as the restored Figure 4 Image (a) Figure 4 (b) is a more similar image to the restored image, in contrast, Figure 6 Image (a) contains the defective part 6A, therefore the characteristic values ​​of the image change, and the restoration Figure 6 (b) The restored image becomes a blurred image.

[0054] Next, in S13, the pre-detection unit 203 uses the difference between the captured image and the restored image to determine whether the object to be inspected is abnormal. Specifically, firstly, by... Figure 4 (a) Figure 5 (a) Figure 6 The captured image shown in (a) is similar to Figure 4 (b) Figure 5 (b) Figure 6 The difference between the restored image shown in (b) generates... Figure 4 (c) Figure 5 (c) Figure 6 (c) shows the difference image. Here, it is a difference image of a normal weld mark without deviations such as brightness. Figure 4 (c) Become Figure 4 (a) and Figure 4 (b) The images are close, therefore the difference becomes smaller. On the other hand, Figure 5 (c) Figure 6 (c) shows the difference image as described above, which serves as the restored image. Figure 5 (b) Figure 6 In (b), because the color-different areas 5A and the defective area 6A were not restored, the difference increased. Next, it is determined whether the difference exceeds a predetermined threshold. Figure 4 As shown in the differential image (c), if the difference is smaller than a predetermined threshold, the pre-detection unit 203 determines that the object being inspected has no abnormalities and ends the process. On the other hand, as... Figure 5 (c) Figure 6 As shown in (c), when the difference is larger than a predetermined threshold, proceed to S14.

[0055] Here, although Figure 5 The image shown in (a) is a normal weld mark image, but in S13, which uses the difference between the image of the inspected object and the restored image for judgment, the inspected object is not judged as having no abnormality. In other words, when there are deviations in the brightness or shape of the image of the inspected object, the normal images used in the learning model cannot cover all normal images. Therefore, even normal images cannot be properly restored, resulting in an over-detection problem where the inspected object is detected as abnormal due to the difference. On the other hand, Figure 4 The captured image shown in (a) is determined to be normal in S13, so the detection unit 204 does not need to make a second judgment, which can shorten the processing time required for abnormality detection processing.

[0056] Next, in S14, the detection unit 204 uses the restored image of the inspection object that was detected as abnormal by the pre-detection unit 203, and the normal restored image that was generated by the restored image generation unit 202 or pre-stored in the anomaly detection system 100 and restored to normal, to determine again whether the inspection object is abnormal. Specifically, first, the similarity of these images is calculated. Figure 7 (a) represents the restored image after the original image of a normal weld mark was captured. Figure 7 (b) indicates Figure 5 (b) shows the restored image of the object under inspection. Figure 7 (c) indicates Figure 6 (b) shows the restored image of the inspected object. Here, as previously stated, the restored image refers to an image captured of the weld mark at location 5A, where the color differs locally due to variations in brightness, etc. Figure 7 (b) is a restored image of the weld mark, except for the part 5A with a local color difference, which is a normal weld mark. Therefore, it is comparable to the normal restored image. Figure 7 (a) shows a high degree of similarity. On the other hand, the reconstructed image is an image of an abnormal weld mark containing the defective part 6A. Figure 7 (c) becomes a blurred image, and therefore differs from the normally restored image. Figure 7 (a) shows a lower similarity compared to the previous example. Next, it is determined whether the calculated similarity is below a predetermined threshold. In the case of... Figure 7If the similarity to the restored image of the object being inspected, as shown in (b), is high, the detection unit 204 determines that the object being inspected has no abnormalities and ends the process. On the other hand, in cases such as Figure 7 If the similarity to the restored image of the object being inspected, as shown in (c), is low, the detection unit 204 determines that the object being inspected is abnormal and ends the process. Furthermore, although omitted in this process, a step of outputting this abnormality detection information to the input / output interface 106 can be added.

[0057] Furthermore, here, as an anomaly detection processing flow of the anomaly detection system 100, an example with S13 is described, but an anomaly detection processing flow without S13 is also possible. In this case, after generating the restored image in S12, it directly proceeds to S14. Then, in S14, the detection unit 204 uses the restored image of the object to be inspected and a normal restored image that has been restored to a normal image to determine whether the object to be inspected is an anomaly. In this anomaly detection processing flow, similarly, when implementing... Figure 4 (a) Figure 5 (a) Figure 6 (a) During the anomaly detection processing of the object being inspected in the captured image, since S13 is not present, S14 is used instead. Figure 4 (a) Figure 5 The object being inspected in the image shown in (a) was determined to be without abnormalities. Figure 6 The object being inspected in the captured image shown in (a) was determined to be abnormal.

[0058] As described above, the anomaly detection processing flow of the anomaly detection system 100 in this embodiment is as follows: Figure 4 (a) shows normal weld marks with no deviation in brightness, etc. Figure 5 (a) The inspected object, although normal, contained weld marks with localized color differences due to variations in brightness, etc., and was judged to have no abnormalities. On the other hand... Figure 6 (a) The inspection object with abnormal weld marks containing defective parts shown is judged as abnormal. That is, even if the above-mentioned over-detection problem occurs in the abnormality judgment (S13) based on the pre-detection unit 203 due to the deviation of brightness and shape, the abnormality judgment (S14) based on the detection unit 204 can suppress the occurrence of over-detection and improve the detection accuracy of abnormality. Here, the abnormality detection processing of the abnormality detection system 100 of this embodiment is not limited to the inspection object. In particular, it is preferred when the inspection object is a weld mark whose shape and brightness are prone to deviation, or when the image of the inspection object is taken in a production line where the position and direction of the inspection object in the image of the inspection object are prone to deviation due to the shooting environment.

[0059] <Other Implementation Methods>

[0060] In the first embodiment, a generative adversarial network, specifically a GAN, was described as the learning model for the restored image generation unit 202. However, in addition to GAN, various other generative adversarial networks such as CGAN, DCGAN, SRGAN, CycleGAN, and VAEGAN can also be used. Furthermore, in addition to GAN, various autoencoders such as AE, VAE, and CVAE can also be used.

[0061] Furthermore, the present invention is not limited to the embodiments described above, and can be appropriately modified within the scope of the spirit. For example, in addition to an anomaly detection system, it can also be implemented by various means such as an anomaly detection device, an anomaly detection method, and a computer program implementing the method.

Claims

1. An abnormality detection system, comprising: an acquisition unit that acquires an inspection target image of an inspection target; a restored image generation unit that inputs the inspection target image to a learning model and generates a restored image, the learning model being learned using an image of a normal inspection target as teacher data; a pre-detection unit that determines whether or not the inspection target in the inspection target image is abnormal using a difference between the inspection target image and the restored image; a detection unit that determines whether or not the inspection target in the inspection target image is abnormal using a similarity between the restored image and a normal restored image in a case where the inspection target in the inspection target image is determined to be abnormal by the pre-detection unit, the normal restored image being a restored image generated by inputting an image of a normal inspection target to the learning model.

2. The abnormality detection system according to claim 1, wherein the learning model is a learning model using a generative adversarial network.

3. The abnormality detection system according to claim 1 or 2, wherein the inspection target is a weld mark formed on a member by welding.

4. The abnormality detection system according to claim 1 or 2, wherein the inspection target image is an image captured in a production line.

5. The abnormality detection system according to claim 3, wherein the inspection target image is an image captured in a production line.

6. An abnormality detection method, comprising: an acquisition step of acquiring an inspection target image of an inspection target; a restored image generation step of inputting the inspection target image to a learning model and generating a restored image, the learning model being learned using an image of a normal inspection target as teacher data; a pre-detection step of determining whether or not the inspection target in the inspection target image is abnormal using a difference between the inspection target image and the restored image; a detection step of determining whether or not the inspection target in the inspection target image is abnormal using a similarity between the restored image and a normal restored image in a case where the inspection target in the inspection target image is determined to be abnormal in the pre-detection step, the normal restored image being a restored image generated by inputting an image of a normal inspection target to the learning model.

Images