Information processing device, information processing method, and program

Abstract

To provide an abnormality location detection technique capable of coping with both unknown abnormalities and known abnormalities.SOLUTION: An information processing device for performing abnormality location detection processing on an input image, reconstructs a normal image from the input image, outputs a determination basis of an abnormality class on the input image and the reconstructed image, and detects an abnormality location on the input image based on a difference between the input image and the reconstructed image and the determination basis of the output abnormality class.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, deep learning has been used for visual inspection in the manufacturing process of industrial parts, food, etc. Since it is difficult to predict all possible abnormal patterns in the manufacturing process in advance, anomaly detection technology that can handle unexpected and unknown anomalies is desired. However, deep learning anomaly detection methods that can handle unknown anomalies have the disadvantage that they cannot utilize information on known abnormality patterns.

[0003] Therefore, Patent Document 1 discloses an anomaly detection technique that can handle both unknown and known anomalies. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Spatially-weighted Anomaly Detection(https: / / arxiv.org / abs / 1810.02607) DISCLOSURE OF THEINVENTION [Problem to be solved by the invention]

[0005] Patent Document 1 describes a technology for detecting anomalies on an image-by-image basis. However, there is a problem in that it is not suitable for cases where it is necessary to detect anomalies by detecting where anomalies exist in an image. In addition, since it is assumed that the output of AI (a trained model trained by a deep learning method) is always correct, it is not a method that takes into account AI errors.

[0006] Therefore, an object of the present invention is to provide a technique capable of detecting an abnormality with higher accuracy. [Means for solving the problem]

[0007] The information processing system of the present invention is characterized by comprising a reconstruction means for reconstructing a normal image from an input image, an output means for outputting a basis for determining an abnormality class between the input image and the image reconstructed by the reconstruction means, and a detection means for detecting an abnormal portion in the input image based on a difference between the input image and the reconstructed image and the basis for determining the abnormality class output by the output means. Effect of the Invention

[0008] According to the present invention, it is possible to provide a technology that can detect abnormalities with higher accuracy. Also, even in cases where the output result of AI is incorrect, it is possible to detect abnormalities with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration for detecting an abnormal portion according to an embodiment of the present invention. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an information processing device 101 according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device 101 according to an embodiment of the present invention. [Figure 4] 10 is a flowchart showing an example of a process for detecting an abnormal portion in an input image according to an embodiment of the present invention. [Diagram 5] A comparison of the reasons for determining normal and abnormal classes DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0011] FIG. 1 is a diagram showing an example of a system configuration for detecting an abnormal portion according to an embodiment of the present invention.

[0012] The abnormal point detection system is a system in which an information processing device 101 and an external device 102 are communicably connected via a network 110. The information processing device 101 is a device operated by a user, and receives an image to be processed, executes the process shown in the flowchart of FIG. 4, displays the processing result (the abnormal part detection result), and so on.

[0013] The external device 102 performs management of images to be processed.

[0014] 4 will be described, the external device 102 may execute the process. Also, the process described as being performed by the external device 102, such as management of images to be processed, may be performed by the information processing device 101. FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing device 101 and the external device 102 of the present invention.

[0015] As shown in FIG. 2, the information processing device 101 and the external device 102 are connected to a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a storage device 204, an input controller 205, an audio controller 206, a video controller 207, a memory controller 208, and a communication I / F controller 209 via a system bus 200.

[0016] The CPU 201 comprehensively controls each device and controller connected to the system bus 200 . ROM 202 or external memory 213 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by CPU 201, computer-readable and executable programs for realizing this information processing method, and various necessary data (including data tables). The RAM 203 functions as a main memory, a work area, etc., for the CPU 201. The CPU 201 loads programs and the like required for executing processes from the ROM 202 or the external memory 213 into the RAM 203, and executes the loaded programs to realize various operations. The input controller 205 controls input from an input device such as a keyboard 210 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can give various instructions by pressing (touching with a finger or the like) icons, cursors, or buttons displayed on the touch panel.

[0017] The touch panel may be a touch panel capable of detecting positions touched by multiple fingers, such as a multi-touch screen. The video controller 207 controls display on an external output device such as a display 212. The display also includes the display of a notebook computer integrated with the main body. Note that the external output device is not limited to a display, and may be, for example, a projector. In addition, for the device capable of receiving the above-mentioned touch operation, an input device is also provided. The video controller 207 is capable of controlling a video memory (VRAM) for display control, and can use part of the RAM 203 as the video memory area, or can provide a separate dedicated video memory. The memory controller 208 controls access to the external memory 213. As the external memory, an external storage device (hard disk) for storing a boot program, various applications, font data, user files, edit files, and various data, a flexible disk (FD), or a compact flash (registered trademark) memory connected to a PCMCIA card slot via an adapter, etc., can be used. The communication I / F controller 209 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, communication using TCP / IP, telephone lines such as ISDN, and 4G and 5G lines for mobile phones are possible. The CPU 201 enables display on the display 212 by, for example, executing a process of expanding (rasterizing) an outline font in a display information area in the RAM 203. The CPU 201 also enables a user to give instructions using a mouse cursor (not shown) or the like on the display 212. Next, an example of the functional configuration of each device in the embodiment of the present invention will be described with reference to FIG. The information processing device 101 includes an image acquisition unit 401, an image reconstruction unit 402, an error image generation unit 403, a heat map generation unit (Abs) 404, a heat map generation unit (ReLU) 405, a heat map integration unit 406, a weighting unit 407, and an output unit 408.

[0018] An image acquisition unit 401 acquires an input image to be processed, and passes it to an image reconstruction unit 402 , an error image generation unit 403 , and a heat map generation unit (ReLU) 405 .

[0019] The image reconstruction unit 402 receives an image from the image acquisition unit 401, and reconstructs a normal image of the image. Then, the reconstructed image is passed to an error image generation unit 403 and a heat map generation unit (Abs) 404.

[0020] The error image generating unit 403 generates an error image by taking the difference between the images received from the image acquiring unit 401 and the image reconstructing unit 402 , and passes the generated image to a weighting unit 407 .

[0021] The heat map generation unit (Abs) 404 receives the reconstructed image from the image reconstruction unit 402 as input, outputs the basis for the classifier's determination of the abnormal class as a heat map, and passes it to the heat map integration unit 406 .

[0022] The heat map generation unit (ReLU) 405 receives the image from the image acquisition unit 401 as input, outputs the basis for the classifier's determination that the image is an abnormal class as a heat map, and passes it to the heat map integration unit 406.

[0023] The heat map integration unit 406 adds up the heat maps received from the heat map generation unit (Abs) 404 and the heat map generation unit (ReLU) 405 for each pixel to create a single heat map, normalizes the values ​​from 0 to 1, and then passes the result to the weighting unit 407.

[0024] The weighting unit 407 weights the error image received from the error image generation unit 403 with the heat map received from the heat map integration unit 406 and passes the weighted image to an output unit 408 .

[0025] The output unit 408 performs threshold processing on the error image received from the weighting unit 407 and outputs an anomaly detection result.

[0026] Next, a process executed by the abnormal point detection in the embodiment of the present invention will be described with reference to the flowchart of FIG.

[0027] The flowchart in Figure 4 is a flowchart showing processing in which the CPU 201 of the information processing device 101 reads out and executes a predetermined control program, receives an input image, detects abnormal areas (areas where scratches, stains, etc. exist) in an object contained in the image, and outputs the results.

[0028] In step S401, the CPU 201 of the information processing device 101 acquires image data stored in the external device 102 as an input image.

[0029] In step S402, the CPU 201 of the information processing device 101 reconstructs a normal image (an image of an object without any abnormalities such as scratches or stains) from the input image acquired in step S401. For the reconstruction, a known technique such as an autoencoder that performs learning using normal images is used. By using an autoencoder, a normal image can be reconstructed even when an abnormal image is input, and by acquiring the difference, it becomes possible to detect unknown abnormalities. In step S403, the CPU 201 of the information processing device 101 generates an error image by calculating the difference between the input image acquired in step S401 and the image reconstructed in step S402.

[0030] This process makes it possible to obtain the difference between an input image, which is an abnormal image (an image containing an abnormal area), and a reconstructed image, which is a normal image, and thereby generate an image showing the abnormal area in the input image.

[0031] However, if the object contained in the input image has large scratches or dirt, or if the input image is unclear, the reconstructed image of the object is not necessarily a normal image (i.e., even an autoencoder that has been trained using normal images cannot necessarily generate a completely normal image), so the processing from step S404 onwards is performed. In step S404, the CPU 201 of the information processing device 101 outputs data indicating the basis for the classifier's decision that the input image is in the abnormal class. In this embodiment, the data is output as a heat map, making it easy to recognize visually. For example, when it is desired to detect scratches or stains on a metal plate as shown in FIG. 5, if the basis for deciding the normal class is output, the entire metal plate will be displayed because the parts other than the scratches are normal metal plate (FIG. 5A). This makes it difficult to identify the abnormal part. In contrast, by outputting the basis for deciding the abnormal class, only the abnormal part (501) in the metal plate will be displayed (FIG. 5B), making it possible to pinpoint the abnormal part.

[0032] The classifier uses a known technique such as a Convolutional Neural Network that is trained in advance using an image and answer data indicating whether the image is a normal image or an abnormal image (the same applies to step S405). The heat map output uses Grad-CAM, a known technique, and uses "Relu (Rectified Linear Unit)" as the activation function. Relu is a function that outputs only values ​​greater than 0.

[0033] In step S405, the CPU 201 of the information processing device 101 outputs a heat map showing the reason why the classifier judged the reconstructed image to be an abnormal class. The activation function of Grad-CAM changed from Relu to an absolute value is used for outputting the heat map. By changing to an absolute value, it becomes possible to capture the reason for judging the normal class (as described above, the heat map showing the reason for judging the normal class often reacts to the entire object). This makes it possible to suppress excessive reaction to a failure in reconstruction. For example, when reconstruction fails in a place other than the object, in the case of Relu, a reaction is shown in a place other than the object. This reaction is a reaction to the part where the normal image failed when reconstructing the normal image, so it is not a very strong reaction. In contrast, in the case of absolute values, there is a strong reaction not only to the failed part but also to the part judged to be normal (i.e., the entire object). Then, when Grad-CAM is output, the value is normalized from 0 to 1, so as a result, the reaction to the failed part is weakened and the reaction to the entire object is output. In other words, by using the absolute value as the activation function, it is possible to suppress reactions to places other than the target object and highlight reactions to the target object, thereby making it possible to suppress excessive reactions to reconstruction failures by autoencoders, etc. In step S406, the CPU 201 of the information processing device 101 adds up the heat maps output in steps S404 and S405 for each pixel, integrates them into one, and normalizes the value from 0 to 1.

[0034] In step S407, the error image generated in step S403 is weighted by the heat map generated in step S406. In step S408, threshold processing is performed on the weighted error image, and the abnormality detection result is output.

[0035] The above process makes it possible to obtain pixel-by-pixel output simply by training an autoencoder and a classifier that require correct labels on an image-by-image basis. In this way, it is possible to pinpoint abnormalities in objects contained in an image without creating correct labels that specify abnormalities on a pixel-by-pixel basis, which eliminates the need to generate training data and enables efficient and accurate detection of abnormalities. Although the embodiment in which processing is executed in the information processing device 101 has been described above, the present invention can be embodied, for example, as a system, an apparatus, a method, a program, or a recording medium, etc. Specifically, the present invention may be applied to a system composed of multiple devices, or may be applied to an apparatus composed of a single device.

[0036] The program of the present invention is a program that enables a computer to execute the processing method of the flowchart shown in FIG. 4, and the storage medium of the present invention stores a program that enables a computer to execute the processing method of FIG. As described above, it goes without saying that the object of the present invention can be achieved by supplying a recording medium on which a program that realizes the functions of the above-mentioned embodiments is recorded to a system or device, and having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0037] In this case, the program itself read out from the recording medium will realize the novel functions of the present invention, and the recording medium on which the program is recorded will constitute the present invention.

[0038] Examples of recording media for supplying the program include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, silicon disks, etc.

[0039] Furthermore, it goes without saying that the functions of the above-mentioned embodiments are not only realized by the computer executing a program it has read, but also includes cases where an OS (operating system) running on the computer performs some or all of the actual processing based on the instructions of the program, and the functions of the above-mentioned embodiments are realized through that processing.

[0040] Furthermore, it goes without saying that this also includes cases where a program read from a recording medium is written into a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to a computer, and then a CPU or the like provided on the function expansion board or function expansion unit performs some or all of the actual processing based on the instructions of the program code, thereby realizing the functions of the above-mentioned embodiments.

[0041] The present invention may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device. Needless to say, the present invention can also be applied to a case where the object is achieved by supplying a program to a system or apparatus. In this case, the effect of the present invention can be enjoyed by the system or apparatus by reading a recording medium storing a program for achieving the present invention into the system or apparatus.

[0042] Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that the present invention also includes configurations that combine the above-mentioned embodiments and their modified examples. [Explanation of symbols]

[0043] 101 Information processing device 102 External device 110 Network

Claims

1. A reconstruction means for reconstructing a normal image from an input image; a first output means for outputting data indicating a reason why the input image and the image reconstructed by the reconstruction means are determined to be in an abnormal class; a second output means for outputting an image showing an abnormality in the input image in a identifiable manner based on a difference between the input image and the reconstructed image and the data output by the first output means; Equipped with The first output means outputs data indicating a reason why the input image is determined to be an abnormal class using Relu as an activation function, and outputs data indicating a reason why the reconstructed image is determined to be an abnormal class using an absolute value as an activation function. An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the second output means outputs an image showing identifiable abnormal areas in the input image based on data obtained by adding together the data output by the first output means, which indicates a basis for determining that the input image is in the abnormal class, and the data output by the first output means, which indicates a basis for determining that the reconstructed image is in the abnormal class, and a difference between the input image and the reconstructed image.

3. 3. The information processing system according to claim 1, wherein the reconstruction means reconstructs a normal image from an input image by inputting the input image to an autoencoder that has been trained using a normal image.

4. 4. The information processing system according to claim 1, wherein the first output means outputs data indicating a reason why the class is determined to be an abnormal class by the Convolutional Neural Network.

5. First data indicating a difference between an input image and a reconstructed image reconstructed from the input image using an autoencoder; second data obtained by adding together a heat map output using Relu as an activation function for an abnormal area in the input image identified by a Convolutional Neural Network and a heat map output using an absolute value as an activation function for an abnormal area in the reconstructed image identified by the Convolutional Neural Network; The information processing system further comprises an output unit that outputs an image showing an abnormal portion in the input image in a identifiable manner based on the input image.

6. A reconstruction step in which a reconstruction means of the information processing system reconstructs a normal image from an input image; a first output step in which an output means of the information processing system outputs data indicating a reason why the input image and the image reconstructed by the reconstruction step are determined to be in an abnormal class; a second output step in which the detection means of the information processing system outputs an image showing an abnormality in the input image in a identifiable manner based on a difference between the input image and the reconstructed image and the data output by the output means; Equipped with The first output step outputs data indicating a reason why the input image is determined to be an abnormal class using Relu as an activation function, and outputs data indicating a reason why the reconstructed image is determined to be an abnormal class using an absolute value as an activation function. An information processing method comprising:

7. A program for causing a computer to function as each of the means according to any one of claims 1 to 5.

Images