Abnormality detection system and abnormality detection method

Abstract

[Abstract] [Problem] To provide an abnormality detection system and an abnormality detection method that can reduce discrepancy between a normal / abnormal determination by a simple comparison of pixel values and a normal / abnormaldetermination by human observation when an input image and an image that was outputted by AI are compared. [Solution] An abnormality detection system and an abnormality detection method for the same, said abnormality detection system comprising: a first AI model 213 that has been trained using normal images of a target object as training data; a reconstruction unit 203 that uses the first AI model 213 to generate reconstructed image data from captured image data; a second AI model 216 that has been trained so as to extract, from two items of image data having a difference, a difference that is close to perception by human observation; a difference extraction unit 205 that uses the second AI model 216 to extract difference image data from the captured image data and the reconstructed image data; and an abnormality extraction unit 206 that extracts an abnormal portion from the difference image data.

Description

Anomaly detection system and anomaly detection method

[0001] The present invention relates to an anomaly detection system that detects anomalies in captured video using artificial intelligence (AI), and in particular to an anomaly detection system and an anomaly detection method that can eliminate the discrepancy between normal / abnormal judgments detected by AI and normal / abnormal judgments made by humans.

[0002] [Prior Art] Anomaly detection systems that automatically determine whether manufactured products are good or bad (normal or abnormal) using imaging devices such as industrial cameras are becoming widespread. In recent years, anomaly detection systems using AI have become mainstream. In particular, in systems that determine whether manufactured products are good or bad, it is difficult to collect large amounts of abnormal data, so methods such as AE (Auto Encoder) and inpainting, which can build AI using only good product data, are used.

[0003] In these methods, the model is trained using only normal (good) image data, and when it is put into operation, any data is input; if it is restored correctly, it is judged to be normal, and if it is not restored correctly, it is judged to be abnormal.

[0004] Conventional anomaly detection systems mainly calculate and compare the difference between an input image and an image output by AI, and determine an anomaly if this difference value exceeds a threshold. However, for example, in areas where the human eye cannot perceive a clear difference, such as differences in image color tone or brightness, a simple comparison of pixel values ​​may determine that there is a sufficient difference. Conversely, in areas where the human eye can perceive a clear difference, a simple comparison of pixel values ​​may determine that there is not a sufficient difference. In other words, in conventional anomaly detection systems, when comparing an input image with an image output by AI, a discrepancy may occur between the normal / abnormal judgment made by the human eye and the normal / abnormal judgment made by a simple comparison of pixel values.

[0005] [Related Art] Incidentally, a related prior art is International Publication No. 2022 / 201451 "Detection Apparatus and Detection Method" (Patent Document 1). Patent Document 1 discloses a detection apparatus that prevents a detection target from being overlooked and determining whether the target is normal or abnormal, even when the target is shown in an extremely small area in an image.

[0006] International Publication No. 2022 / 201451

[0007] As mentioned above, conventional anomaly detection systems had a problem in that when comparing an input image with an image output by AI, there was a discrepancy between the normal / abnormal judgment made by simply comparing pixel values ​​and the normal / abnormal judgment made by a human eye.

[0008] Furthermore, conventional anomaly detection systems have had the problem that it is not easy to collect abnormal data from the learning data for AI.

[0009] Incidentally, Patent Document 1 does not describe a configuration for using AI to perform abnormality detection in a manner similar to the normal / abnormal judgment made by the human eye.

[0010] The present invention has been made in consideration of the above-mentioned circumstances, and aims to provide an anomaly detection system and an anomaly detection method that can train AI using easily collected learning data, and that, when comparing an input image with an image output by the AI, can reduce the discrepancy between the normal / abnormal judgment made by the human eye and the normal / abnormal judgment made by simply comparing pixel values.

[0011] The present invention, which aims to solve the problems of the above-mentioned conventional examples, is an anomaly detection system that detects abnormalities in photographed image data of an object, and includes a first AI model that has been trained using normal images of the object as learning data, a reconstruction unit that generates reconstructed image data from the photographed image data using the first AI model, a second AI model that has been trained to extract the difference between two pieces of image data that have a difference, a difference extraction unit that uses the second AI model to extract differential image data from the photographed image data and the reconstructed image data, and an anomaly extraction unit that extracts abnormal parts from the differential image data.

[0012] In the anomaly detection system according to the present invention, the two image data sets having a difference may not include abnormal data.

[0013] Furthermore, in the anomaly detection system of the present invention, differential true value image data obtained by extracting the difference between two pieces of image data having a difference, and differential image data generated from the two pieces of image data using a second AI model, are used to train the second AI model so that the differential image data becomes equal to the differential true value image data.

[0014] In the anomaly detection system according to the present invention, the differential true value image data is corrected based on two image data having a difference so as to reflect a difference observed by a human eye.

[0015] The present invention also provides an anomaly detection method for detecting anomalies in photographed image data of an object, in which a reconstruction unit generates reconstructed image data from the photographed image data using a first AI model that has been trained using normal images of the object as training data, a difference extraction unit extracts differential image data from the photographed image data and the reconstructed image data using a second AI model that has been trained to extract the difference between two pieces of image data that have a difference, and an anomaly extraction unit extracts an abnormal portion from the differential image data.

[0016] According to the present invention, there is provided an anomaly detection system for detecting abnormalities in image data of an object, the anomaly detection system comprising: a first AI model trained on normal images of the object as learning data; a reconstruction unit that generates reconstructed image data from the image data of the object using the first AI model; a second AI model trained to extract differences between two pieces of image data that have differences; a difference extraction unit that extracts differential image data from the image data of the object and the reconstructed image data using the second AI model; and an anomaly extraction unit that extracts abnormal parts from the differential image data. Therefore, differences can be extracted with high accuracy using AI for difference extraction, and by training the second AI model to reflect differences that can be recognized by the human eye, anomaly detection can be performed with reduced deviation from human judgment.

[0017] Furthermore, according to the present invention, the anomaly detection system does not require the two image data with a difference to contain abnormal data, which has the effect of making it easy to collect learning data for the second AI model.

[0018] It is an explanatory diagram showing the schematic configuration of the anomaly detection system. It is a configuration block diagram of an anomaly detection device. It is an explanatory diagram showing the outline of the operation of a first AI learning unit. It is an explanatory diagram showing the outline of the operation of a second AI learning unit.

[0019] An embodiment of the present invention will be described with reference to the drawings. [Outline of the embodiment] An anomaly detection system according to an embodiment of the present invention (the present anomaly detection system) is an anomaly detection system that detects anomalies in photographed image data of an object, and includes: a first AI model (reconstructed AI model) that has been trained using normal images of the object as training data; a reconstruction unit that inputs the photographed image data to the first AI model to generate reconstructed image data; a second AI model (difference extraction AI model) that has been trained to extract differences between two pieces of image data that have differences; a difference extraction unit that inputs the photographed image data and the reconstructed image data to the second AI model to extract difference image data; and an anomaly extraction unit that extracts abnormal parts from the difference image data. The difference extraction can be performed using AI to extract differences with high accuracy, and the second AI is trained to reflect differences that are recognizable to the human eye, thereby suppressing the influence of color, brightness, etc., and enabling anomaly detection that reduces discrepancies with human judgment.

[0020] Moreover, the anomaly detection method according to the embodiment of the present invention is an anomaly detection method in the anomaly detection system.

[0021] [Schematic configuration of the present anomaly detection system: Figure 1] The schematic configuration of the present anomaly detection system will be described using Figure 1. Figure 1 is an explanatory diagram showing the schematic configuration of the present anomaly detection system. As shown in Figure 1, the present anomaly detection system basically includes an image acquisition device 101, an anomaly detection device 102, and a display output device 103.

[0022] The video capture device 101 is a device that captures and outputs real-time or past video or still images. Examples of the video capture device 101 include an industrial camera, a surveillance camera, a USB (Universal Serial Bus) camera, a smartphone, or a recorder or player that can play a CD (Compact Disc), an HDD (Hard Disc Drive), or a Blu-ray Disc (registered trademark), or other cameras or video recording devices.

[0023] The anomaly detection device 102 is a characteristic part of this anomaly detection system, and is equipped with AI to detect whether the image input from the video acquisition device 101 is normal or abnormal. The anomaly detection device 102 is configured with hardware such as a central processing unit (CPU), digital signal processor (DSP), field-programmable gate array (FPGA), graphics processing unit (GPU), etc., as well as memory, a recording device, etc. Note that the anomaly detection device 102 may also include other electronic devices and hardware. The configuration and operation of the anomaly detection device 102 will be described later.

[0024] The display output device 103 outputs the results (anomalies) detected by the anomaly detection device 102 in some form, such as display or sound. The display output device 103 is, for example, a display with a personal computer (PC), a smartphone, a mobile phone, an alarm device such as a commercial radio or an alarm, or other devices.

[0025] [Configuration of anomaly detection device 102: Figure 2] Next, the configuration of the anomaly detection device 102 will be described using Figure 2. Figure 2 is a configuration block diagram of the anomaly detection device. As shown in Figure 2, the anomaly detection device 102 basically includes a still image acquisition unit 201, an input still image storage unit 211, a first AI learning data storage unit 212, a first AI learning unit 202, a first AI model 213, a reconstruction unit 203, a reconstructed image storage unit 214, a second AI learning data storage unit 215, a second AI learning unit 204, a second AI model 216, a difference extraction unit 205, a difference image storage unit 217, an anomaly extraction unit 206, and an information transmission unit 207.

[0026] The individual components of the anomaly detection device 102 will now be described. The still image acquisition unit 201 inputs an image from the video acquisition device 101 and stores the still image in the input still image storage unit 211. Specifically, when video (moving images) is input from the video acquisition device 101, the still image acquisition unit 201 divides the video into frames, acquires each frame as an input still image, and stores the input still image in the input still image storage unit 211. When dividing the video into single frames, the division may be performed by thinning out every several frames. Furthermore, when a still image is input from the video acquisition device 101, the still image acquisition unit 201 stores the input still image in the input still image storage unit 211 as is.

[0027] Here, the color space of the input still image is expressed in RGB (Red, Green, Blue), HSV (Hue, Saturation, Value), HLS (Hue, Luminance, Saturation), or other color spaces. Furthermore, in order to reduce the effects of noise, flicker, and the like, preprocessing such as smoothing filters, edge enhancement filters, brightness conversion, and histogram equalization may be performed. Furthermore, in order to improve the accuracy of anomaly detection and reduce processing costs, the image may be enlarged or reduced to a predetermined size. The input still image storage unit 211 stores the still image input from the still image acquisition unit 201.

[0028] The first AI learning data storage unit 212 stores learning data (first AI learning data) for the first AI learning unit 202 to learn the first AI model 213. The first AI learning data is a normal image of the target for anomaly detection. The first AI learning unit 202 uses the first AI learning data to learn the first AI model 213. The operation of the first AI learning unit 202 will be described later; the first AI learning data 212 is input to the first AI model 213, and various parameters are adjusted and learned so that the input correct image and the image generated by the first AI model 213 are identical.

[0029] Since the learning algorithm of the first AI is well known, a description of the specific configuration will be omitted. Also, if the first AI learning unit 202 learns the first AI model 213 before the anomaly detection system is put into operation, it does not need to be executed thereafter (after the start of operation), and the first AI model 213 may be re-trained irregularly or periodically.

[0030] The first AI model 213 is a trained AI model (reconstructed AI model) that has been trained to restore (reconstruct) and output an image identical to the input image when a normal image is input, and to generate and output an image different from the input image when an abnormal image is input. In other words, if the input still image from the input still image storage unit 211 is normal, the first AI model 213 reconstructs and outputs an image identical to the input image. The first AI model 213 may be, for example, an AI model such as an autoencoder, a CAE (Convolutional Autoencoder), a VAE (Variational Autoencoder), or inpainting after superimposing a mask region, or may be an AI model capable of achieving other reconstruction. The reconstruction unit 203 stores the reconstructed image generated from the input still image from the input still image storage unit 211 using the trained first AI model 213 in the reconstructed image storage unit 214. Specifically, the reconstruction unit 203 inputs an input still image into the first AI model 213 and stores the generated reconstructed image in the reconstructed image storage unit 214.

[0031] The second AI learning data storage unit 215 stores, as learning data, many sets of two images with differences and images (difference true value images) that represent the differences between those images detected by the human eye. As will be described later, the second AI learning data may be any set of images with clear differences, and may be images unrelated to the object for anomaly detection when the anomaly detection system is actually operated.

[0032] The second AI learning unit 204 uses the second AI learning data to train the second AI model 216. The operation of the second AI learning unit 204 will be described later. A feature of this anomaly detection system is that, when two images having a difference are input, the second AI model 216 is trained so that a difference image equivalent to an image reflecting the difference recognized by the human eye (a true difference image) is generated, regardless of differences in brightness or color tone. In the true difference image, areas in which a difference is perceived by the human eye are represented as white, and areas in which a difference is not perceived as black. Note that if the second AI model 216 is trained before operation, it does not need to be executed after operation begins, and the second AI model 216 may be retrained irregularly or periodically.

[0033] The second AI model 216 is a trained AI model (difference extraction AI model) that extracts the difference between two input images and outputs a difference image, and is realized, for example, by an AI model for segmentation such as an FCN (Fully Convolutional Network) with twice the input channels, SegNet, or U-Net, or may be any other AI model that can realize difference extraction.

[0034] The difference extraction unit 205 generates a difference image between the input still image from the input still image storage unit 211 and the reconstructed image from the reconstructed image storage unit 214 using the second AI model 216, and stores the difference image in the difference image storage unit 217. Specifically, the difference extraction unit 205 inputs the input still image and the reconstructed image to the second AI model 216, obtains the difference image output from the second AI model 216, and stores it in the difference image storage unit 217.

[0035] The difference image storage unit 217 stores the difference image. The anomaly extraction unit 206 extracts abnormal areas using the difference image input from the difference image storage unit 217. The anomaly extraction unit 206 binarizes (0 and 1) the difference image using a threshold value (T2), performs an opening process that repeats contraction and expansion, and a closing process that repeats expansion and contraction to remove fine noise, performs a labeling process, and extracts areas within the labeled areas (white areas) that have an area larger than a threshold value (T3) as abnormal. The labeling process makes it possible to determine the position, size, and number of areas (white areas) where differences have been extracted.

[0036] The order of the opening and closing processes and the number of times each process is performed can be changed for each abnormal image, and the comparison target for the threshold T3 in the labeling process may be the vertical or horizontal length of the region. Note that the algorithms for the opening, closing, and labeling processes are well known, so a description of their specific configurations will be omitted.

[0037] The information transmission unit 207 transmits the anomaly extracted by the anomaly extraction unit 206 in a format suitable for the display output device 103, such as an image, text, audio, compressed data, or other information format.

[0038] [Learning of the first AI model 213 and the second AI model 216] Before explaining the operation of the anomaly detection device 102 as a whole, we will explain the operation of the first AI learning unit 202 that learns the first AI model 213 and the second AI learning unit 204 that learns the second AI model 216.

[0039] [Outline of Operation of First AI Learning Unit: Figure 3] First, an outline of the operation of the first AI learning unit 202 will be explained using Figure 3. Figure 3 is an explanatory diagram showing an outline of the operation of the first AI learning unit. As shown in Figure 3, the first AI learning unit 202 inputs first AI learning data 300 stored in the first AI learning data storage unit 212 into the first AI model 213 to generate a reconstructed image 301, and compares the input image and the output image to adjust and learn the parameters of the first AI model 213 so that the input image and the output image are equal.

[0040] Here, the first AI learning data are multiple normal images of the target object for anomaly detection, and may be subjected to augmentation such as inversion, rotation, brightness conversion, noise removal, and noise addition, with the color space and image size being the same as the input still image acquired by the still image acquisition unit 201 during operation. Normal images of the target object for anomaly detection are easy to obtain, so the first AI learning data can be easily collected. Through learning, the first AI model 213 becomes a model that, when a normal image is input, outputs a restored image equivalent to the normal image.

[0041] [Outline of Operation of Second AI Learning Unit: FIG. 4] Next, an outline of the operation of the second AI learning unit 204 will be described using FIG. 4. FIG. 4 is an explanatory diagram showing an outline of the operation of the second AI learning unit. As shown in FIG. 4, the second AI learning unit 204 inputs two images with a difference (a set of difference extraction AI learning data) into the second AI model 216, and compares a difference image 404 generated by the second AI model 216 with an image of the difference judged by a human eye (a difference true value image) 403, thereby learning the second AI model 216 so that the two become equal.

[0042] As the second AI learning data, a data set for verifying moving object detection accuracy or the like is used, but is represented by a schematic diagram for ease of understanding in Fig. 4. Here, as shown in Fig. 4, the second AI learning data is a plurality of stored sets of corresponding background image 401, foreground image 402 in which another object is superimposed on background image 401, and true difference image 403 in which the difference between background image 401 and foreground image 402 is recognized by human eyes.

[0043] Foreground image 402 is an image at the same position and angle of view as background image 401, and contains people, cars, motorcycles, trains, airplanes, desks, chairs, tableware, and other living things and objects that are not included in background image 401. Background image 401 and foreground image 402 may be subjected to augmentation such as inversion, rotation, brightness conversion, noise removal, and noise addition, and the color space and image size are made equal to those of the input still image input from still image acquisition unit 201 during operation. The second AI learning data may be easily obtained images unrelated to the object to be detected for anomaly during operation, and there is no need to use images of defective products, which are difficult to collect, making it easy to prepare learning data.

[0044] The second AI learning data storage unit 215 has stored in advance, as second AI learning data, a plurality of pairs of correspondences between known background image 401 and foreground image 402 and difference true image 403. The difference true image 403 is not simply constituted by the difference in numerical values ​​between background image 401 and foreground image 402, but is an image generated from known background image 401 and foreground image 402. In the difference true image 403, areas where the difference can be perceived by the human eye are configured as white (numeric value "1"), and areas where the difference cannot be perceived by the human eye are configured as black (numeric value "0"). In other words, the difference true image 403 is an image that reflects the recognition of the difference by the human eye, suppressing the influence of differences due to factors other than the presence or shape of an object, such as brightness and color.

[0045] Then, the second AI learning unit 204 reads a set of background image 401 and foreground image 402 from the learning data stored in the second AI learning data memory unit 215, inputs them into the second AI model 216, and compares the generated difference image 404 with the difference true value image 403 corresponding to the input set, thereby learning the second AI model 216 to make them equal.

[0046] For example, for a certain region of the background image 401 and the foreground image 402, the difference in the output numerical value from the second AI model 216 may be greater than a specific threshold (T1), but the same region may be black in the difference true value image 403. Similarly, for another region, the difference in the output numerical value from the second AI model 216 may be smaller than the threshold, but the corresponding region may be white in the difference true value image 403. In other words, there may be a discrepancy between the difference detection by the second AI model 216 and the difference detection by the human eye. In this anomaly detection system, the second AI model 216 is trained to reduce this discrepancy.

[0047] 4 shows a difference image 404 in an insufficiently learned state, with white dots in black areas and no match with the true difference image 403. The second AI learning unit 204 adjusts and learns the parameters of the second AI model 216 so that the difference image 404 matches the true difference image 403, and the second AI model 216 is thereby able to perform difference detection that is close to judgment by the human eye.

[0048] [Operation of anomaly detection device 102: Figure 2] The operation of the anomaly detection device 102 will be explained using Figure 2. Note that here, it is assumed that the first AI model 213 and the second AI model 216 have already been trained, and the operation during operation will be explained. The still image acquisition unit 201 appropriately processes the input image to generate a still image, which is stored in the input still image storage unit 211. The input still image is input to the reconstruction unit 203, and a reconstructed image is generated and output using the first AI model 213 and stored in the reconstructed image storage unit 214.

[0049] Then, the reconstructed image from the reconstructed image memory unit 214 and the input still image from the input still image memory unit 211 are input to the difference extraction unit 205, and a difference image in which the difference is extracted using a second AI model 216 is generated, and the difference image is stored in the difference image memory unit 217.

[0050] If the object in the input still image is normal, there will be almost no difference from the reconstructed image, and the difference image will be solid black, but if there is an abnormality in the object in the input still image, the difference from the reconstructed image will be large, and a white area will appear. In particular, in this anomaly detection system, as described above, the difference image generated by the second AI model 216 reflects the presence or absence of differences that can be recognized by the human eye, making it possible to detect anomalies in a manner close to that achieved by the human eye.

[0051] The differential image is input to the anomaly extraction unit 206, and when the anomaly extraction unit 206 extracts an anomaly, a notification of the anomaly detection is output to the information transmission unit 207, which then transmits information notifying the anomaly in a predetermined form, such as display data or audio data. In this manner, the anomaly detection device 102 operates.

[0052] As a result, this anomaly detection system can learn without using defective products as learning data for the second AI model 216, and can detect anomalies in a manner similar to the senses of the human eye.This is expected to be applicable to a variety of industrial fields, such as the sorting of agricultural products and anomaly detection systems on production lines for processed food products and parts.

[0053] [Effects of the embodiment] According to the present anomaly detection system, which is an anomaly detection system that detects anomalies in captured image data of an object, the system includes a first AI model 213 that has been trained to use normal images of the object as learning data, a reconstruction unit 203 that inputs the captured image data into the first AI model 213 and generates reconstructed image data, a second AI model 216 that has been trained to extract a difference from two pieces of image data that have a difference, a difference extraction unit 205 that inputs the captured image data and the reconstructed image data into the second AI model 216 and extracts differential image data, and an anomaly extraction unit 206 that extracts an abnormal portion from the differential image data. Therefore, differences can be extracted with high accuracy using AI for difference extraction, and by training the second AI model 216 to reflect differences that can be recognized by the human eye, the effects of color, brightness, etc. can be suppressed, and anomaly detection can be performed that reduces the discrepancy with human judgment.

[0054] In addition, according to this anomaly detection system, images of good products that are easy to collect or images unrelated to the target object can be used as learning data for the second AI model 216, making it easier to collect learning data and enabling the second AI model 216 to be trained efficiently.

[0055] This application claims the benefit of priority from Japanese Patent Application No. 2023-045392, filed on March 22, 2023, the entire disclosure of which is incorporated herein by reference.

[0056] The present invention is suitable for an anomaly detection system and an anomaly detection method that can train AI using easily collected learning data and can use AI to perform anomaly detection that is close to the normal / abnormal judgment made by humans.

[0057] DESCRIPTION OF SYMBOLS 101...Video acquisition device, 102...Abnormality detection device, 103...Display output device, 201...Still image acquisition unit, 202...First AI learning unit, 203...Reconstruction unit, 204...Second AI learning unit, 205...Difference extraction unit, 206...Abnormality extraction unit, 211...Input still image storage unit, 212...First AI learning data storage unit, 213...First AI model, 214...Reconstructed image storage unit, 215...Second AI learning data storage unit, 216...Second AI model, 217...Difference image storage unit, 300...First AI learning data, 301...Reconstructed image, 401...Background image, 402...Foreground image, 403...Difference true value image, 404...Difference image

Claims

1. An anomaly detection system that detects anomalies in photographed image data of an object, A first AI model trained using normal images of the object as training data; A reconstruction unit that generates reconstructed image data from the captured image data using the first AI model; A second AI model trained to extract differences from two image data having differences; A difference extraction unit that extracts difference image data from the captured image data and the reconstructed image data using the second AI model; an abnormality extraction unit that extracts an abnormal portion from the differential image data; and an AI learning unit that uses differential true value image data obtained by extracting a difference from the two pieces of image data having a difference, and differential image data generated from the two pieces of image data using the second AI model, and trains the second AI model so that the differential image data becomes equal to the differential true value image data.

2. The anomaly detection system according to claim 1 , wherein the two image data having the difference may not include abnormal data.

3. 2. The anomaly detection system according to claim 1, wherein the differential true value image data is corrected so as to reflect a difference observed by a human eye based on the two image data having the difference.

4. An anomaly detection method for detecting an anomaly in photographed image data of an object, comprising: A reconstruction unit generates reconstructed image data from the captured image data using a first AI model that has been trained using normal images of the object as training data, A difference extraction unit extracts difference image data from the captured image data and the reconstructed image data using a second AI model that has been trained to extract a difference from two pieces of image data having a difference; An abnormality extraction unit extracts an abnormal portion from the difference image data, An anomaly detection method in which an AI learning unit uses differential true value image data obtained by extracting the difference between two pieces of image data having a difference, and differential image data generated from the two pieces of image data using the second AI model, and trains the second AI model so that the differential image data becomes equal to the differential true value image data.

5. An anomaly detection method as described in claim 4, wherein the two image data having the difference may not contain abnormal data.

6. The anomaly detection method as described in Claim 4, wherein the differential true value image data is corrected to reflect the difference observed by human eyes based on the two image data having the difference.