Anomaly detection device and anomaly detection method

Abstract

To provide a technique of automatically detecting an abnormality from an image even if the abnormality is a small scratch.SOLUTION: The present invention relates to an abnormality detector 1 for detecting an abnormality of an inspection target object on the basis of an inspection image 2 of the inspection target object by using a neural network, the abnormality detector including: a missing provision image generation unit 5 for generating a plurality of missing provision images with a missing region from the inspection image 2; an interpolation image generation unit 8, using a learned model 7 having learned to output an interpolation image formed by interpolating the missing region when the missing provision image is input; a re-configured image generation unit 10 for generating a re-configured image by inputting a plurality of missing provision images generated by the missing provision image generation unit 5 into the interpolation image generation unit 8, thereby synthesizing a plurality of interpolation images output from the interpolation image generation unit 8; and an abnormality determination unit 12 for detecting an abnormal region in the inspection image on the basis of the difference between the re-configured image and the inspection image, thereby determining an abnormality of the inspection target object.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an abnormality detection apparatus and an abnormality detection method for detecting an abnormality in an inspection target based on an image.

Background Art

[0002] In order to detect an abnormality included in an image by utilizing deep learning technology, a large amount of training images for supervised learning are required. However, images containing abnormalities are not actively collected very much. On the other hand, socially, the detection of abnormalities is more important than the detection of normal conditions, and there is a need for a method of detecting abnormalities without a small number of abnormal data examples or, extremely, teacher data regarding abnormalities. In recent years, image conversion using a generative adversarial network (GAN) has been actively studied, and a technique called GLCIC has also attracted attention for realizing extremely natural interpolation for the problem of interpolating a missing region of an image. By learning the process of interpolating a deliberately given defect for an image not containing an abnormality using this GLCIC, since the interpolation accuracy significantly decreases in a region containing an abnormality, it becomes possible to specify an abnormal region without using an abnormal image example.

Prior Art Documents

Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In quality control in a manufacturing line of precision equipment, not only abnormalities that can be immediately recognized visually but also small scratches must be detected with high accuracy. However, when small scratches are taken into consideration, the possible appearances of abnormalities are very diverse. On the other hand, it is difficult to deliberately create various abnormalities in a manufacturing line.

[0005] Therefore, the present invention aims to provide a technology for automatically detecting anomalies, including small scratches, from images without using prior knowledge about anomalies. [Means for solving the problem]

[0006] To achieve this objective, the anomaly detection device of the present invention uses a neural network to detect anomalies in an object under inspection based on an inspection image taken of the object under inspection, and from the inspection image The size and shape of the inspection image are predetermined according to the size of the image, the accuracy of anomaly detection, and the required processing speed. The system includes: a missing image generation unit that generates multiple missing image files having missing regions; an interpolation image generation unit that uses a trained model trained to output an interpolated image in which the missing region is interpolated when a missing image file is input; a reconstructed image generation unit that generates a reconstructed image by synthesizing multiple interpolated images output from the interpolation image generation unit when multiple missing images generated by the missing image generation unit are input to the interpolation image generation unit; and an abnormality determination unit that detects abnormal regions in the inspection image based on the difference between the reconstructed image and the inspection image and performs abnormality determination of the object to be inspected.

[0007] Furthermore, in the anomaly detection device of the present invention, the multiple images with missing data generated by the missing data image generation unit have different positions for the missing data regions. Furthermore, the abnormality detection device of the present invention is The multiple images with missing data generated by the missing data generation unit have the missing data regions in randomly shifted positions. It's okay to be there. Furthermore, the abnormality detection device of the present invention is Multiple images with missing data generated by the missing data generation unit have partially overlapping locations of the missing data regions. You can. Furthermore, the abnormality detection device of the present invention may also include an abnormality determination unit comprising a difference image generation unit that generates a difference image between a reconstructed image and an inspection image, and a region size determination unit that determines that there is an abnormality in the inspection target when the size of the abnormal region in the difference image exceeds a predetermined threshold.

[0008] Furthermore, the anomaly detection method of the present invention uses a neural network to detect anomalies in an object to be inspected based on an inspection image taken of the object to be inspected, and comprises: a missing image generation step of generating multiple missing images from the inspection image, each having a missing region of a predetermined size and shape according to the size of the inspection image, the accuracy of anomaly detection, and the required processing speed; an interpolation image generation step of generating multiple interpolated images using a trained model that has been trained to output an interpolated image in which the missing region has been interpolated when a missing image is input; a reconstructed image generation step of generating a reconstructed image by synthesizing the multiple interpolated images; and an anomaly determination step of detecting an anomaly in the inspection image based on the difference between the reconstructed image and the inspection image and determining the anomaly of the object to be inspected. 。

[0009] In other words, by using GLCIC to interpolate defects intentionally introduced into normal images, it is possible to reconstruct the original image with very high accuracy for images that do not contain abnormalities. On the other hand, in regions that contain some kind of abnormality, a significant difference in reconstruction accuracy occurs, allowing the presence of the abnormality to be detected. Because GLCIC can re-identify even finer structures than conventional deep learning, it can detect any abnormality, including small scratches, even without training images that contain abnormalities. By applying the GLCIC method to image data that does not contain anomalies, it becomes possible to automatically detect even small scratches without having to prepare example images containing anomalies. This reduces the cost of data collection while enabling anomaly detection that can find even smaller scratches. [Effects of the Invention]

[0010] According to the present invention, the accuracy of anomaly detection of an object under inspection based on images taken of the object can be improved. Furthermore, anomaly data can be eliminated from the creation of a trained model. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing an example of an anomaly detection device of the present invention. [Figure 2] Figure 1 is a block diagram showing the creation of a trained model. [Figure 3] This is a conceptual diagram illustrating the flow of anomaly detection using the anomaly detection device of the present invention. [Figure 4] These diagrams illustrate image processing. (A) shows the inspection image when there is no abnormality in the subject being inspected, (B) shows the missing region when there is no abnormality in the subject being inspected, (C) shows the interpolated image when there is no abnormality in the subject being inspected, (D) is a magnified view of the part corresponding to the missing region in the interpolated image when there is no abnormality in the subject being inspected, (E) is a magnified view of the part corresponding to the missing region in the inspection image when there is no abnormality in the subject being inspected, (F) shows the inspection image when there is an abnormality in the subject being inspected, (G) shows the missing region when there is an abnormality in the subject being inspected, (H) shows the interpolated image when there is an abnormality in the subject being inspected, (I) is a magnified view of the part corresponding to the missing region in the interpolated image when there is an abnormality in the subject being inspected, and (J) is a magnified view of the part corresponding to the missing region in the inspection image when there is an abnormality in the subject being inspected. [Figure 5] The diagrams illustrate image processing, with (A) showing the initial image when there is no abnormality in the subject, (B) showing the reconstructed image when there is no abnormality in the subject, (C) showing the difference image when there is no abnormality in the subject, (D) showing the initial image when there is an abnormality in the subject, (E) showing the reconstructed image when there is an abnormality in the subject, and (F) showing the difference image when there is an abnormality in the subject. [Figure 6] This flowchart shows the processing steps for the anomaly detection method of the present invention. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the abnormality detection device according to the present invention will be described with reference to the drawings. Figures 1 to 5 show an anomaly detection device according to the present invention. The anomaly detection device 1 uses a neural network to detect anomalies in an inspection target based on an inspection image 2 taken of the inspection target. It comprises a missing image generation unit 5 that generates multiple missing image 4 having missing regions 3 from the inspection image 2, an interpolation image generation unit 8 that uses a trained model 7 which has been trained to output an interpolation image 6 in which the missing regions 3 are interpolated when a missing image 4 is input, a reconstructed image generation unit 10 that generates a reconstructed image 9 by synthesizing multiple interpolation images 6 output from the interpolation image generation unit 8 when multiple missing image 4 generated by the missing image generation unit 5 are input to the interpolation image generation unit 8, and an anomaly determination unit 12 that detects anomaly regions 11 in the inspection image 2 based on the difference between the reconstructed image 9 and the inspection image 2 and determines an anomaly in the inspection target. In this embodiment, a smoothing processing unit 13 is provided that performs smoothing processing on the inspection image 2 and the reconstructed image 9 generated by the reconstructed image generation unit 10.

[0013] The inspection image 2 is an image of an object to be inspected (not shown), and in this embodiment, pre-taken images are stored in the database 17. The database 17 stores a large number of inspection images 2, and they are selected one by one in order and input to the anomaly detection device 1. The inspection images 2 input to the anomaly detection device 1 are supplied to the missing image generation unit 5 and the difference image generation unit 15. The inspection images 2 stored in the database 17 are from before the inspection, and there is a possibility that abnormal images may be included among those that are normal.

[0014] The missing image generation unit generates multiple missing images 4 based on the inspection image 2. The position of the missing region 3 in each missing image 4 is different. In this embodiment, the position of the missing region 3 in each missing image 4 is shifted slightly vertically and horizontally, for example by 1 pixel at a time, from the top left to the bottom right of the image, similar to a raster scan. However, the amount of shift is not limited to 1 pixel. The positions of the missing regions 3 in each missing image 4 may partially overlap, or the missing regions 3 may be provided so as not to overlap. Furthermore, the positions of the missing regions 3 in each missing image 4 may be shifted regularly or randomly.

[0015] The size and shape of the defect area 3 are appropriately determined according to the size of the inspection image 2, the detection accuracy of abnormalities, the required processing speed, and the like. Also, the number of defect-added images 4 generated for one inspection image 2: n is appropriately determined according to the size of the inspection image 2, the detection accuracy of abnormalities, the required processing speed, and the like.

[0016] The interpolation image generation unit 8 is configured by a learned model 7 using a neural network. Preferably, the learned model 7 is configured by a generative adversarial network (GAN, Generative Adversarial Networks). By using a generative adversarial network, it becomes possible to accurately interpolate the defect area 3. In the present embodiment, GLCIC (Globally and Locally Consistent Image Completion) is adopted as the generative adversarial network, and it becomes possible to perform interpolation of the defect area 3 with even higher accuracy.

[0017] As shown in FIG. 2, the learned model 7 is created by performing unsupervised learning using a large number of training images 18. The training images 18 are normal images (good product images) of the inspection target, and are stored in, for example, a database 19.

[0018] The learned model 7 using GLCIC includes three networks, that is, an interpolation network (generator, Generator) that performs image interpolation, a global discriminator network (global discriminator, Global Discriminator) that evaluates whether the input image is real or fake, and a local discriminator network (local discriminator, Local Discriminator).

[0019] The interpolation network receives the missing image generated by the missing image generation unit 20 and the training image 18 as input, and is trained to output an interpolated image 21 by interpolating the missing region of the missing image. Meanwhile, the global and local discriminant networks receive the discriminant image 21 interpolated by the interpolation network or the training image 18 as input, and are trained to output the discriminant result of the input image. By having the interpolation network compete with the global and local discriminant networks, it becomes possible to output an interpolated image 21 that is consistent throughout the entire scene and also natural locally.

[0020] Thus, the trained model 7 is trained to perform image loss interpolation using only normal data (normal images). While it can generate normal data, it cannot reliably generate abnormal data (abnormal regions in an image). The anomaly detection device 1 utilizes this to detect abnormal regions 3 in the inspection image 2.

[0021] The missing image generation unit 5 and the interpolation image generation unit 8 process images one by one. That is, when the missing image generation unit 5 generates one missing image 4, the interpolation image generation unit 8 generates an interpolation image 6. The generated interpolation image 6 is temporarily stored in a memory (not shown). Then, the missing image generation unit 5 generates the next missing image 4, and the same process is repeated thereafter. The generation of missing image 4 and interpolation image 6 is repeated until a predetermined number of interpolation images 6 (n images) have been generated.

[0022] The reconstructed image generation unit 10 reads all the interpolated images 6 from memory and combines them to generate a single reconstructed image 9.

[0023] The smoothing processing unit 13 smooths the reconstructed image 9 generated by the reconstructed image generation unit 10. In this embodiment, a Gaussian filter is used for smoothing, but it is not limited to this. For example, an average filter, a weighted average filter, a median filter, etc., can also be used. Furthermore, the smoothing processing unit 13 also smooths the inspection image 2 supplied to the difference image generation unit 15. In this embodiment, a Gaussian filter is used for smoothing, but it is not limited to this. For example, an average filter, a weighted average filter, a median filter, etc., can also be used.

[0024] The abnormality detection unit 12 includes a difference image generation unit 15 that generates a difference image 14 between the reconstructed image 9 and the inspection image 2, and an abnormality area size determination unit 16 that determines that there is an abnormality in the inspection target if the size of the abnormal area 11 in the difference image 14 exceeds a predetermined threshold T1.

[0025] The difference image generation unit 15 generates a difference image 14 between the reconstructed image 9, which has been smoothed by the smoothing processing unit 13, and the inspection image 2. The difference image generation unit 15 compares the smoothed reconstructed image 9 and the inspection image 2 pixel by pixel at the same coordinates to find the difference d, and represents this difference d as the difference image 14. The abnormal area size determination unit 16 then determines that an abnormal area 11 is the area where a group of pixels have a difference d greater than a predetermined threshold T2, and determines that there is an abnormality in the inspection target if the size of the abnormal area 11, or in other words, the number of pixels included in the abnormal area 11 (pixels whose difference d is greater than the threshold T2), exceeds a predetermined threshold T1.

[0026] The thresholds T1 and T2 are set appropriately according to the state of the inspection image 2 and the accuracy required for anomaly detection.

[0027] Figures 4(A) to (E) show images where the abnormal region 11 is not visible in the examination image 2, i.e., when there is no abnormality in the subject being examined. Figure (A) is the examination image 2, Figure (B) is the missing region 4 with the missing region 3 added, Figure (C) is the interpolated image 6, Figure (D) is a magnified view of the part of the interpolated image 6 corresponding to the missing region 3, and Figure (E) is a magnified view of the part of the examination image 2 corresponding to the missing region 3. Since the trained model 7 is trained to perform image missing region interpolation using only normal data (normal images), it can successfully generate the region corresponding to the missing region 3 for normal data (examination image 2 in which the abnormal region 11 is not visible) (Figure (D)).

[0028] Furthermore, Figures 4(F) to (J) show images where the abnormal region 11 is visible in the examination image 2, that is, when there is an abnormality in the subject being examined. Figure 4(F) is the examination image 2, Figure 4(G) is the missing region 4 with a missing region 3 added at the position corresponding to the abnormal region 11, Figure 4(H) is the interpolated image 6, Figure 4(I) is a magnified view of the part of the interpolated image 6 corresponding to the missing region 3, and Figure 4(J) is a magnified view of the part of the examination image 2 corresponding to the missing region 3 (i.e., the region corresponding to the abnormal region 11). Since the trained model 7 is trained to perform image missing region interpolation using only normal data (normal images), it cannot successfully reproduce the missing region 3 of the examination image 2 (Figure 4(I)).

[0029] Figures 5(A) to (C) show the difference images 14 when the abnormal region 11 is not visible in the examination image 2, i.e., when there is no abnormality in the subject being examined. Figure (A) is the examination image 2, Figure (B) is the reconstructed image 9, and Figure (C) is the difference image 14. Figures 5(D) to (F) also show the difference images 14 when the abnormal region 11 is visible in the examination image 2, i.e., when there is an abnormality in the subject being examined. Figure (D) is the examination image 2, Figure (E) is the reconstructed image 9, and Figure (F) is the difference image 14.

[0030] Since the trained model 7 is trained to perform image loss interpolation using only normal data (normal images), it can generate a reconstructed image 9 for normal data (inspection image 2 that does not show the abnormal region 11) (Figure (B)), but it cannot successfully generate abnormal data (inspection image 2 that shows the abnormal region 11) (Figure (E)). Therefore, the reconstructed image 9 in Figure (B) is similar to the inspection image 2, and none of the pixels in the difference image 14 exceed the threshold T2 (Figure (C)). On the other hand, the reconstructed image 9 in Figure (E) does not reproduce the abnormal region 11, and the pixels corresponding to the abnormal region 11 are far from the pixels in inspection image 2. Therefore, the pixels corresponding to the abnormal region 11 in the difference image 14 in Figure (F) are greater than the threshold T2, making it possible to detect the abnormal region 11.

[0031] The missing image generation unit 5, interpolation image generation unit 8, smoothing processing unit 13, reconstructed image generation unit 10, and anomaly detection unit 12 are realized by having a computer execute a predetermined program.

[0032] Next, the anomaly detection method of the present invention will be described. Figure 6 shows the anomaly detection method. The anomaly detection method uses a neural network to detect anomalies in an object to be inspected based on an inspection image 2 taken of the object to be inspected. It includes a missing image generation step S51 which generates multiple missing image 4 having missing regions 3 from the inspection image 2, an interpolation image generation step S52 which generates multiple interpolation images 6 using a trained model 7 that has been trained to output an interpolated image 6 in which the missing regions 3 have been interpolated when a missing image 4 is input, a reconstructed image generation step S55 which generates a reconstructed image 9 by synthesizing the multiple interpolation images 6, and an anomaly determination step S56 which detects anomaly regions 11 in the inspection image 2 based on the difference between the reconstructed image 9 and the inspection image 2 and determines that the object to be inspected is abnormal. In this embodiment, there is a smoothing process step S53 which performs smoothing on the reconstructed image 9 and the inspection image 2.

[0033] Furthermore, the abnormality determination step S56 includes a difference image generation step S57 that generates a difference image 14 between the reconstructed image 9 and the examination image 2, and an abnormality area size determination step S58 that determines that there is an abnormality in the examination target if the size of the abnormal area 11 in the difference image 14 exceeds a predetermined threshold T1.

[0034] In the anomaly detection method of the present invention, a trained model 7 is first created as preparation for performing anomaly detection based on the inspection image 2. The trained model 7 is created by unsupervised learning using training images 18. This trained model 7 is incorporated into the anomaly detection device 1 as an interpolation image generation unit 8.

[0035] Furthermore, images of the subject to be examined (examination image 2) are pre-stored in database 17.

[0036] Then, when the inspection image 2 from database 17 is input to the anomaly detection device 1, anomaly detection is started. The inspection image 2 is supplied to the missing image generation unit 5 and the difference image generation unit 15. In the image generation unit 5, a missing image 4 is generated from the inspection image 2 (step S51). First, for example, an image 4 with a missing region 3 added to the upper left of the image is generated. This first image 4 with a missing region is input to the interpolation image generation unit 8.

[0037] The interpolation image generation unit 8 generates an interpolated image 6 by interpolating the missing region 3, and saves it to a memory (not shown).

[0038] Then, the process returns to step S51 (step S54: No), and the second missing image 4 is generated (step S51). Then, the process in step S52 is executed, and the second interpolated image 6 is generated and stored in memory. These processes are performed for n missing images 4.

[0039] Then, once processing is complete for the n missing image 4 (step S54: Yes), the process proceeds from step S54 to step S55, and the reconstructed image 9 is generated by the reconstructed image generation unit 10. The reconstructed image generation unit 10 reads n interpolated images 6 from memory, combines them, and generates a single reconstructed image 9.

[0040] The reconstructed image 9 is output to the smoothing processing unit 13. In the smoothing processing unit 13, smoothing is applied to the reconstructed image 9 and the examination image 2 (step S53). The reconstructed image 9 and the inspection image 2 are supplied to the difference image generation unit 15 of the abnormality determination unit 12 (step S56), and the difference image generation unit 15 generates a difference image 14 (step S57). Then, the abnormality area size determination unit 16 compares the abnormality area 11 in the difference image 14 with a threshold T1, and if the size of the abnormality area 11 exceeds the threshold T1 (size of abnormality area 11 > threshold T1), it is determined that there is an abnormality in the object being inspected (step S58). On the other hand, if the size of the abnormality area 11 is less than or equal to the threshold T1 (size of abnormality area 11 ≤ threshold T1), it is determined that there is no abnormality in the object being inspected (step S58). The determination result is supplied and displayed on a display device (not shown), for example.

[0041] The embodiments described above are merely examples of preferred implementations of the present invention, and are not limited thereto. Various modifications are possible without departing from the spirit of the present invention. For example, in the above explanation, the inspection images 2 were pre-stored in the database 17, and one by one were selected from them and input into the anomaly detection device 1. However, the system is not necessarily limited to this configuration; for example, images could be input directly from a camera or other imaging device.

[0042] The image recognition device 1 and image recognition method of the present invention are particularly suitable for image recognition of industrial robots, household robots, etc., but are not limited to these applications. [Industrial applicability]

[0043] For example, it can be applied to fields in industries such as automotive, structural engineering, and healthcare that require anomaly detection that was previously judged visually. [Explanation of symbols]

[0044] 1. Anomaly detection device 2. Images for examination 3. Missing region 4. Images with missing parts 5. Image generation unit with missing data. 6 Interpolated images 7. Pre-trained models 8. Interpolation Image Generation Unit 9 Reconstructed image 10 Reconstructed image generation section 11 Abnormal area 12 Abnormality determination section 13 Smoothing Processing Unit 14 difference images 15. Difference Image Generation Unit 16 Abnormal area size determination unit 17 Databases 18 Training images 19 Databases 20 Image generation unit with missing image 21 Interpolated image

Claims

1. An anomaly detection device that uses a neural network to detect an anomaly in an object being inspected based on an inspection image taken of the object being inspected, A defect-adding image generation unit generates multiple defect-adding images from the aforementioned inspection image, each having a defect region of a predetermined size and shape according to the size of the inspection image, the accuracy of anomaly detection, and the required processing speed. An interpolation image generation unit using a trained model that has been trained to output an interpolated image in which the missing region has been interpolated when the aforementioned missing image is input, A reconstruction image generation unit generates a reconstructed image by inputting a plurality of missing image projections generated by the missing image generation unit into the interpolation image generation unit, and synthesizing the plurality of interpolation images output from the interpolation image generation unit. An abnormality determination unit that detects abnormal regions in the inspection image based on the difference between the reconstructed image and the inspection image and performs an abnormality determination of the object to be inspected, An anomaly detection device characterized by comprising:

2. The anomaly detection device according to claim 1, characterized in that the multiple images with missing data generated by the missing data generation unit have different positions for the missing data regions.

3. The anomaly detection device according to claim 2, characterized in that the multiple images with missing regions generated by the missing region image generation unit are randomly shifted in the position of the missing region.

4. The anomaly detection device according to claim 2, characterized in that the multiple images with missing data generated by the missing data generation unit partially overlap in the positions of the missing data regions.

5. The abnormality determination unit, A difference image generation unit generates a difference image between the reconstructed image and the inspection image, A region size determination unit that determines that there is an abnormality in the inspection target when the size of the abnormal region in the difference image exceeds a predetermined threshold, An abnormality detection device according to any one of claims 1 to 4, characterized by comprising:

6. An anomaly detection method that uses a neural network to detect abnormalities in an object under inspection based on an image of the object under inspection, A defect-adding image generation step of generating multiple defect-adding images from the inspection image, each having a defect region of a predetermined size and shape according to the size of the inspection image, the accuracy of anomaly detection, and the required processing speed, An interpolation image generation step in which multiple interpolation images are generated using a trained model that has been trained to output an interpolated image in which the missing region has been interpolated when the missing image is input, A reconstructed image generation step involves synthesizing multiple interpolated images to generate a reconstructed image, An abnormality determination step involves detecting abnormal regions in the inspection image based on the difference between the reconstructed image and the inspection image, and performing an abnormality determination of the object to be inspected. An anomaly detection method characterized by comprising:

Images