Anomaly Detection System

Abstract

The purpose of the present invention is to provide an abnormality detection system that can detect an abnormal portion with higher accuracy using a DL model. The abnormality detection system comprises: a signal acquisition unit (201) for acquiring an input signal from a sensor; a mask generation unit (202) for generating a mask to be superimposed on the input signal; a mask superimposition unit (203) for generating a masked signal by superimposing, on the input signal, the mask generated by the mask generation unit (202); a signal reconstruction unit (204) for generating a reconstructed signal by reconstructing the masked signal generated by the mask superimposition unit (203); and an abnormality determination unit (205) for determining whether or not the input signal includes an abnormal portion on the basis of an error, in a mask region, between the input signal and the reconstructed signal. The signal reconstruction unit (204) reconstructs a region that is masked by using a deep learning model obtained by performing training using normal input signals.

Description

[Technical Field]

[0001] The present invention relates to an anomaly detection system, and more particularly to an anomaly detection system that uses AI technology such as deep learning. [Background technology]

[0002] Deep learning (DL) and other AI (artificial intelligence) technologies are increasingly being applied to tasks such as anomaly detection, prediction, and attribute classification using time-series signals and images as input. Incorporating such functions into products and systems can replace human work, reducing labor costs and increasing the added value of products, thereby contributing to sales promotion of products and systems. For this reason, many companies are showing a strong interest in AI.

[0003] One well-known anomaly detection method using DL is the autoencoder. The autoencoder uses a neural network to extract features from an input signal and then uses the extracted features to restore (reconstruct) a signal so that the error between the input signal and the reconstructed signal is zero. Learning is performed unsupervised, with parameters trained so that the error between an arbitrary normal input signal and a signal reconstructed using the autoencoder is zero. During operation, the input signal is judged to contain an abnormal portion based on the error between the input signal and a signal whose features are extracted and reconstructed using a trained DL model (a neural network and a set of parameters). If the input signal contains an abnormal portion, the abnormal portion will not be reconstructed correctly, resulting in a large error between the input signal and the reconstructed signal, and the input signal is determined to contain an abnormal portion.

[0004] In practice, generalization performance is required to reconstruct signals with zero error even for unknown input signals that contain only normal portions. For example, in the case of a system that uses cameras to check outdoor public infrastructure facilities for anomaly detection, the shooting environment may vary due to differences in weather, time of day, etc. In this case, the shooting environment when the input signal (learning data) is collected during learning may differ significantly from the shooting environment when the actual image is captured. This increases the possibility of false positives, where non-anomalous conditions are judged to be abnormal, resulting in reduced accuracy. For this reason, it is generally desirable to obtain training data in a variety of environments to ensure diversity.

[0005] On the other hand, Patent Document 1 describes an obstacle detection system that extracts the area and pixel values ​​of an object that has occurred in a monitored area from an image captured by a photographing means, divides the acquired object area and pixel values ​​into blocks based on criteria set for each angle of view and position in the image, identifies the type of object from local features, and detects the presence or absence of an obstacle that has occurred in the image from information on the type of object. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-124986 A [Non-patent literature]

[0007] [Non-Patent Document 1] G. Liu et.al., Image Inpainting for Irregular Holes Using Partial Convolutions, arXiv:1804.07723 Summary of the Invention [Problem to be solved by the invention]

[0008] However, if a DL model has generalization capabilities after training with diverse training data, the abnormal parts will be reconstructed without error when input signals containing abnormal parts are used in operation. This reduces the error between the input signal and the reconstructed signal, causing the problem of abnormalities not being detected. This problem is thought to occur because the presence of features similar to those of the abnormal parts in the diverse training data makes it possible to reconstruct the abnormal parts as normal parts.

[0009] Furthermore, in Patent Document 1, local features are used, and these local features need to be extracted by prior learning or the like, so there is a possibility that they may be affected by the environment during learning.

[0010] In view of the above-mentioned problems, an object of the present invention is to provide an anomaly detection system that can detect an abnormal part with higher accuracy using a DL model. [Means for solving the problem]

[0011] To achieve the above object, one representative anomaly detection system of the present invention includes a signal acquisition unit that acquires an input signal from a sensor, a mask generation unit that generates a mask to be superimposed on the input signal, a mask superimposition unit that generates a masked signal by superimposing the mask generated by the mask generation unit on the input signal, a signal reconstruction unit that reconstructs the masked signal generated by the mask superimposition unit to generate a reconstructed signal, and an anomaly determination unit that determines whether the input signal includes an abnormal portion based on an error (e.g., squared error, SNR, PSNR, SSIM, or other known feature) within a mask region between the input signal and the reconstructed signal, and the signal reconstruction unit reconstructs the masked region using a deep learning model trained using a normal input signal. The mask generation unit generates a mesh pattern mask in which a plurality of locations are arranged in a predetermined pattern, and generates a mask by sequentially changing the position of the mesh pattern for each image to which the mask is applied, so that when these mesh patterns are combined, all locations in the image can be covered. It is characterized by: [Effects of the Invention]

[0012] According to the present invention, an anomaly detection system can use a DL model to detect an abnormal portion with higher accuracy. Problems, configurations, and effects other than those described above will become apparent from the following embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of a computer system for implementing aspects according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an embodiment of the anomaly detection system of the present invention. [Figure 3] FIG. 3 is a functional block diagram showing an example of the analysis server of the anomaly detection system of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an example of the anomaly detection method of the anomaly detection system of the present invention during learning. [Figure 5] FIG. 5 is a diagram illustrating an example of the anomaly detection method of the anomaly detection system of the present invention during operation. [Figure 6] FIG. 6 is a diagram illustrating an example of the processing of the anomaly detection system of the present invention. [Figure 7] FIG. 7 is a diagram illustrating an example of a mask that reduces the number of masked signals in the anomaly detection system of the present invention. [Figure 8] FIG. 8 is a diagram showing a model for explaining an example of a method for selecting a mask using background subtraction in the anomaly detection system of the present invention. [Figure 9] FIG. 9 is a diagram illustrating an example of a method for selecting a mask using background subtraction in the anomaly detection system of the present invention. [Figure 10] FIG. 10 is a diagram illustrating an example of a method for adjusting a mask using the object detection result in the anomaly detection system of the present invention. [Figure 11] FIG. 11 is a diagram showing a model for explaining a first specific example of the anomaly detection system of the present invention. [Figure 12] FIG. 12 is a diagram illustrating a first specific example of the anomaly detection system of the present invention. [Figure 13] FIG. 13 is a diagram showing a model for explaining a second specific example of the anomaly detection system of the present invention. [Figure 14] FIG. 14 is a diagram illustrating a second specific example of the anomaly detection system of the present invention. [Figure 15] FIG. 15 is a diagram illustrating an example of application of the anomaly detection system of the present invention to time-series data. [Figure 16] FIG. 16 is an example of a processing flowchart of the anomaly detection system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described with reference to the drawings.

[0015] <Examples of hardware for implementing the embodiment> 1 is a block diagram of a computer system 1 for implementing aspects according to embodiments of the present invention. The mechanisms and apparatus of various embodiments disclosed herein may be applied to any suitable computing system.

[0016] The major components of computer system 1 include one or more processors 2, memory 4, terminal interface unit 12, storage interface unit 14, I / O (input / output) device interface unit 16, and network interface 18. These components may be interconnected via memory bus 6, I / O bus 8, bus interface unit 9, and I / O bus interface unit 10.

[0017] Computer system 1 may include one or more processing units 2A and 2B, collectively referred to as processors 2. Each processor 2 executes instructions stored in memory 4 and may include an on-board cache. In some embodiments, computer system 1 may include multiple processors, while in other embodiments, computer system 1 may be a single processing unit. Examples of processing units include a central processing unit (CPU), a field-programmable gate array (FPGA), a graphics processing unit (GPU), and a digital signal processor (DSP).

[0018] In some embodiments, memory 4 may include random-access semiconductor memory, storage devices, or storage media (either volatile or nonvolatile) for storing data and programs. In some embodiments, memory 4 represents the entire virtual memory of computer system 1 and may include virtual memories of other computer systems connected to computer system 1 via a network. While memory 4 may be conceptually considered a single entity, in other embodiments, memory 4 may be a more complex organization, such as a hierarchy of caches and other memory devices. For example, memory may exist as multiple levels of caches, and these caches may be divided by function. As a result, one cache may hold instructions, while other caches hold non-instruction data used by the processor. Memory may also be distributed and associated with various different processing units, such as in a so-called NUMA (Non-Uniform Memory Access) computer architecture.

[0019] Memory 4 may store all or part of the programs, modules, and data structures that implement the functions described herein. For example, memory 4 may store latent factor identification application 50. In some embodiments, latent factor identification application 50 may include instructions or descriptions that execute the functions described below on processor 2, or may include instructions or descriptions that are interpreted by other instructions or descriptions. In some embodiments, latent factor identification application 50 may be implemented in hardware via semiconductor devices, chips, logic gates, circuits, circuit cards, and / or other physical hardware devices instead of or in addition to a processor-based system. In some embodiments, latent factor identification application 50 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 9, processor 2, or other hardware in computer system 1. Such a configuration may reduce the need for processor 2 to access memory 4 and the latent factor identification application.

[0020] Computer system 1 may include a bus interface unit 9 that facilitates communication between processor 2, memory 4, display system 24, and I / O bus interface unit 10. I / O bus interface unit 10 may couple to I / O bus 8 for transferring data to and from various I / O units. I / O bus interface unit 10 may communicate with multiple I / O interface units 12, 14, 16, and 18, also known as I / O processors (IOPs) or I / O adapters (IOAs), via I / O bus 8. Display system 24 may include a display controller, display memory, or both. The display controller may provide video, audio, or both data to display device 26. Computer system 1 may also include one or more sensors or other devices configured to collect data and provide the data to processor 2. For example, computer system 1 may include environmental sensors that collect humidity data, temperature data, pressure data, etc., and motion sensors that collect acceleration data, movement data, etc. Other types of sensors may also be used. The functions provided by the bus interface unit 9 may be realized by an integrated circuit including the processor 2 .

[0021] The I / O interface unit provides functionality for communicating with various storage or I / O devices. For example, the terminal interface unit 12 may be equipped with user I / O devices 20, such as user output devices such as a video display, a television with speakers, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, buttons, a light pen, or other pointing device. A user may use a user interface to enter input data or instructions into the user I / O devices 20 and the computer system 1 and receive output data from the computer system 1 by operating the user input devices. The user interface may be displayed on a display, played through speakers, or printed via a printer via the user I / O devices 20, for example.

[0022] The storage interface unit 14 may be attached to one or more disk drives or direct access storage devices 22 (typically magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In some embodiments, the storage device 22 may be implemented as any secondary storage device. The contents of the memory 4 may be stored in the storage device 22 and retrieved from the storage device 22 as needed. The I / O device interface unit 16 may provide an interface to other I / O devices, such as printers, fax machines, etc. The network interface 18 may provide a communications path that allows the computer system 1 and other devices to communicate with each other. This communications path may be, for example, a network 30.

[0023] While computer system 1 shown in FIG. 1 includes a bus structure providing direct communication paths between processor 2, memory 4, bus interface unit 9, display system 24, and I / O bus interface unit 10, in other embodiments, computer system 1 may include point-to-point links, multiple hierarchical buses, parallel or redundant communication paths in a hierarchical, star, or web configuration. Furthermore, while I / O bus interface unit 10 and I / O bus 8 are shown as a single unit, computer system 1 may actually include multiple I / O bus interface units 10 or multiple I / O buses 8. Additionally, while multiple I / O interface units are shown isolating I / O bus 8 from various communication paths leading to various I / O devices, in other embodiments, some or all of the I / O devices may be directly connected to a single system I / O bus.

[0024] In some embodiments, computer system 1 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 1 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

[0025] <Overall configuration example> FIG. 2 is a block diagram showing an embodiment of the anomaly detection system of the present invention.

[0026] 2 includes an analysis server 101, a sensor 102, and a database server 103. The analysis server 101, the sensor 102, and the database server 103 are connected to each other via a network 104.

[0027] The analysis server 101 is configured by an electronic computer system equipped with a processor such as a CPU. In addition to the CPU, the processor may also be equipped with a DSP, FPGA, GPU, etc. The analysis server 101 performs the processing described below.

[0028] The sensor 102 is a device that continuously acquires signal data, such as a camera, an acceleration sensor, or a temperature sensor. A signal may be acquired by combining multiple devices. In the case of a camera, for example, a camera configuration can be applied in which incident light is focused on an image sensor via a lens or an aperture to acquire information. Examples of the image sensor here include a CCD (Charge-Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera can capture images and videos, for example, at 3 frames per second (3 fps) or more, and the information is sent to the analysis server 101 and the database server 103. Multiple cameras can be installed depending on the situation.

[0029] The database server 103 is a database server equipped with a storage device. It can record information required for the analysis by the analysis server 101 and information acquired by the sensor 102. It can also record the results of the analysis by the analysis server 101.

[0030] The network 104 is a line capable of data communication that connects each server, and can be of any type, such as a dedicated line, an intranet, or an IP network such as the Internet.

[0031] Signal data acquired by the sensor 102 is analyzed by the analysis server 101, and the results of anomaly detection are stored in the database server 103. Note that the configuration in FIG. 2 is an example, and other configurations are also applicable. For example, the functions of the analysis server 101 may be provided integrally with the sensor 102, and the processing of the anomaly detection system may be performed there. Furthermore, the storage device of the database server 103 may also be integrated with the sensor 102 or the analysis server 101.

[0032] <An example of an analysis server> 3 is a functional block diagram showing an example of the analysis server of the anomaly detection system of the present invention. The functional blocks of the analysis server 101 will be described with reference to FIG.

[0033] The analysis server 101 includes a signal acquisition unit 201 , a mask generation unit 202 , a mask superposition unit 203 , a signal reconstruction unit 204 , an abnormality determination unit 205 , an output control unit 206 , and an auxiliary storage unit 207 .

[0034] The auxiliary storage unit 207 stores signals input from the sensor 102. The auxiliary storage unit 207 also holds necessary information such as setting parameters. The auxiliary storage unit 207 is usually configured with a non-volatile memory such as an HDD (Hard Disk Drive) or flash memory, and stores programs executed by the analysis server 101 and data to be processed by the programs. The signals from the auxiliary storage unit 207 are output to the signal acquisition unit 201, the mask generation unit 202, and the output control unit 206.

[0035] The signal acquisition unit 201 acquires a signal from the auxiliary storage unit 207. The acquired signal is a signal from the sensor 102, and may be pre-processed to reduce the effects of noise, flicker, and the like. Examples of pre-processing here include processing using a smoothing filter or an edge enhancement filter. In the case of image data, a data format such as RGB color, YUV, or monochrome may be selected depending on the application. Furthermore, the signal may be reduced to a predetermined size to reduce processing costs. The signal that has undergone these processes is output to the mask superposition unit 203.

[0036] The mask generation unit 202 acquires mask setting parameters from the auxiliary storage unit 207 and generates multiple masks. At this time, the mask to be used may be determined by acquiring a signal from the auxiliary storage unit 207 and performing preprocessing such as background subtraction, which will be described later. The generated mask signal is output to the mask superimposition unit 203.

[0037] The mask superimposition unit 203 superimposes the input signal obtained by the signal acquisition unit 201 and the multiple masks generated by the mask generation unit 202 to generate a masked signal. Here, the masked portion is a portion that does not retain information about the original input signal. The masked signal is output to the signal reconstruction unit 204.

[0038] The signal reconstruction unit 204 reconstructs the masked signal generated by the mask superposition unit 203. This reconstruction generates a reconstructed signal in which the signal of the masked portion is reconstructed. The reconstruction is performed by inputting the signal into a deep learning (DL) model of AI (artificial intelligence) that can restore (interpolate) signals using techniques such as inpainting. Deep learning can apply techniques using neural networks and parameter sets. Here, inpainting refers to a technique of masking an image and restoring it, and a model for this purpose is called an inpainting model. Image inpainting is a specific example of inpainting. For example, the technique described in Non-Patent Document 1 may be used for image inpainting. The reconstructed signal is sent to the anomaly determination unit 205.

[0039] The abnormality determination unit 205 calculates the error between the signal acquired by the signal acquisition unit 201 and the signal reconstructed by the signal reconstruction unit 204, and determines whether or not there is an abnormal portion using a preset threshold. In particular, if the error in the masked region is equal to or greater than the threshold, it can be determined to be abnormal. The threshold here can be a threshold that indicates the extent to which the signal difference is greater than a predetermined value. Furthermore, the error can be evaluated using, for example, square error, SNR (Signal-to-Noise Ratio), PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), or other known feature quantities. The determination result is output to the output control unit 206.

[0040] The output control unit 206 outputs the output result of the abnormality determination obtained from the abnormality determination unit 205, the signal information stored in the auxiliary storage unit 207, and the like to the database server 103.

[0041] In the following, an example will be described in which the sensor 102 is a camera, the signal acquired from the sensor 102 is image data, and image inpainting is used as the DL model capable of restoring (interpolating) the signal.

[0042] <Anomaly detection overview> Fig. 4 is a diagram illustrating an example of the anomaly detection method of the anomaly detection system of the present invention during learning. Fig. 5 is a diagram illustrating an example of the anomaly detection method of the anomaly detection system of the present invention during operation. An overview of the anomaly detection method using image inpainting will be described using Figs. 4 and 5.

[0043] As shown in Figure 4, during training, an input image 310, which is a normal input signal (an image showing a normal test object 301), is used. A mask 305 is superimposed on this input image 310 to generate a masked image 320, which is a masked signal. Here, the position of the mask 305 on the masked image 320 is randomly superimposed. A monochromatic mask, such as black or white, can be applied to the mask 305 so that information from the original image in this area is not retained. Then, image inpainting 330 is used to reconstruct (restore) the masked image 320 to an unmasked state, creating a reconstructed image 340. This reconstructed image 340 is compared with the unmasked input image 310. In particular, comparison is made between the input image 310 and the area covered by the mask 305, and training is performed to reconstruct a normal input image. At this time, a loss function is calculated, and feature extraction and signal reconstruction parameters for error-free reconstruction are learned. In this way, machine learning is performed in advance to generate an inpainting model, which is a DL model.

[0044] Figure 5 explains anomaly detection during operation. The upper part explains the case where there is no abnormality in the inspection object 401, and the lower part explains the case where there is an abnormality 502 in the inspection object 501. During operation, a mask is superimposed on the signal acquired from the camera (sensor 102) and input into the inpainting model. Whether the input signal contains an abnormal part is determined based on the error in the mask area between the signal reconstructed by the inpainting model and the input signal without the mask. If the mask is superimposed on an abnormal part, the abnormal part will be reconstructed as if it were a normal part, resulting in a large error between the input signal and the reconstructed signal.

[0045] A case will be described where there is no abnormality in the inspection object 401 in the upper part of Fig. 5. A masked image 420 is generated by superimposing a mask 405 on an input image 410, and image inpainting 430 processing is performed to generate a reconstructed image 440. Then, the input image 410 and the reconstructed image 440 are compared, and the error in the area of ​​the mask 405 is calculated. Here, since there is no abnormality in the area of ​​the mask 405, the error in the area of ​​the mask 405 is small.

[0046] A case will be described where an anomaly 502 exists in the inspection object 501 shown in the lower part of Figure 5. A masked image 520 is generated by superimposing a mask 505 on an input image 510, and image inpainting 530 is performed to generate a reconstructed image 540. The input image 510 and the reconstructed image 540 are then compared, and the error in the area of ​​the mask 505 is calculated. Here, since the anomaly 502 is located in the area of ​​the mask 505, this anomaly is not reflected in the reconstructed image 540. As a result, the error in the area of ​​the mask 505 becomes large.

[0047] <Example of anomaly detection system processing> Fig. 6 is a diagram illustrating an example of the processing of the anomaly detection system of the present invention. Fig. 6 illustrates the operations of the sensor 102, signal acquisition unit 201, mask generation unit 202, mask superposition unit 203, signal reconstruction unit 204, anomaly determination unit 205, and auxiliary storage unit 207 shown in Fig. 3 when the input signal is an image.

[0048] The signal acquisition unit 201 acquires an image transmitted from the camera, which is the sensor 102, and stored in the auxiliary storage unit 207, and outputs it as an input image 610. Here, an example is shown in which the input image 610 shows an inspection object 601 with an abnormality 602.

[0049] The mask generation unit 202 reads parameters such as the size, shape, number, and slide amount of the masks stored in the auxiliary storage unit 207, and generates a mask 605. The position of the mask 605 varies for each image to which it is applied. In FIG. 6, one rectangular mask is formed for each image to which it is applied, with the position changed in sequence. This generates n mask patterns 620-1 to 620-n. It is preferable that these patterns can be combined to cover all positions within the image. Note that the shape of the mask 605 is rectangular, which is a shape suitable for sequentially masking the entire image, but is not limited to this and any specific shape can also be used.

[0050] Next, the mask superimposing unit 203 superimposes the mask generated by the mask generating unit 202 onto the input image 610 to create masked images 630-1 to 630-n. That is, n images are created by superimposing masks 605 at different positions on the same input image 610. In Fig. 6, the position is changed by sliding the mask area.

[0051] Next, the signal reconstruction unit 204 inputs the masked images generated by the mask superimposition unit 203 into an inpainting model (image inpainting 635). Then, as an output result, n reconstructed images 640-1 to 640-n are obtained. At this time, the masked images 630-1 to 630-n may be input to the inpainting model one by one, or multiple images may be batch processed (parallel processed). The reconstructed images 640-1 to 640-n are each reconstructed from the mask 605 portion of the masked images 630-1 to 630-n.

[0052] Next, the anomaly determination unit 205 compares the reconstructed images 640-1 to 640-n reconstructed from the masked images 630-1 to 630-n with the input image 610. This comparison calculates the error in each mask 605 region, and if there is an error equal to or greater than a certain threshold, it is determined to be "abnormal." If all errors are less than the threshold, it is determined to be "no abnormality." The threshold value used here can be a threshold value that indicates the extent of the difference between the binarized images and a reference value. If the difference is greater than a predetermined range (number of pixels), it can be determined to be an abnormal portion. Note that the reference value used for binarization can be a difference in pixel value greater than a predetermined value (e.g., greater than a predetermined number of pixels), and any appropriate value can be used. Figure 6 shows binarized error images 650-1 to 650-n, and error image 650-m shows a difference in the anomaly 602.

[0053] The size, shape, number and sliding amount of the mask 605 can be set arbitrarily.

[0054] <Example of reducing the number of masked signals> 7 is a diagram illustrating an example of a mask that reduces the number of masked signals in the anomaly detection system of the present invention. Here, an example is shown in which an input image 710 shows an inspection object 701 with an anomaly 702.

[0055] When the number of masked signals input to the inpainting model is large, the processing time increases and real-time performance is impaired. For this reason, one way to reduce the number of masked signals input to the inpainting model is to use a mesh pattern mask as shown in Figure 7. Unlike Figure 6 above, a mask that covers a single area is not used, but rather a mesh pattern mask that covers multiple areas arranged in a predetermined pattern.

[0056] The mask generation unit 202 forms a mesh pattern mask 705 for each image to be applied by sequentially shifting its position. This generates four mask patterns 720-1 to 720-4. It is preferable that these patterns can be combined to cover all positions within the image. Note that a rectangular shape for one mesh pattern mask is suitable for covering the entire image while shifting its position, but the shape is not limited to this and any specific shape can also be applied.

[0057] Next, mask superimposition section 203 creates masked images 730-1 to 730-4 by superimposing mask patterns 720-1 to 720-4 on input image 710. That is, four images are created by superimposing masks 705 at different positions on the same input image 710.

[0058] The location of the abnormality can be identified by the abnormality determination unit 205 in the same manner as in FIG.

[0059] In the example shown in Figure 7, using a mesh pattern reduces the number of masked signals, making it possible to reduce processing. Specifically, the 48 masks can be reduced to four mask patterns, allowing analysis to be performed using the masked signals. This significantly reduces the time required for analysis. However, if a large area is hidden at once, it may be difficult to restore the image during reconstruction, so in this case, it is more effective to use a method in which each area is covered with a mask one by one, as shown in Figure 6.

[0060] <Example of mask selection method using background subtraction> Fig. 8 is a diagram showing a model for explaining an example of a method for selecting a mask using background subtraction in the anomaly detection system of the present invention. Fig. 9 is a diagram for explaining an example of a method for selecting a mask using background subtraction in the anomaly detection system of the present invention.

[0061] A mask pattern is generated by the mask generation unit 202 shown in Fig. 3 and Fig. 6, but in order to minimize the number of masks used at this time, preprocessing for selecting a mask may be added. This preprocessing can reduce the range in which the mask is used by, for example, limiting the area in which the mask is selected in advance. As a specific example, a method for selecting a mask using background subtraction will be described with reference to Figs. 8 and 9.

[0062] Here, as shown in Fig. 8, an example is shown in which an inspection object 802 flowing on a conveyor 803 is photographed from above using a fixed camera 801. The camera 801 corresponds to the sensor 102 shown in Figs. 2, 3, and 6, and photographs the inspection object 802 from above. The inspection object 802 moves on the conveyor 803 in the direction of the arrow. At this time, the angle of view of the camera 801 is fixed, and the inspection object 802 passes laterally within the angle of view of the camera 801.

[0063] As shown in FIG. 9, a background image 920 that does not show the object to be inspected is prepared. Specifically, this is the image in FIG. 8 when the object to be inspected 802 is not present. Furthermore, an input image 910 is an image in which the object to be inspected 802 is shown. Then, the difference between the background image 920 and the input image 910 is taken, and binarized using a threshold to generate a difference image 930. The threshold here is selected to be appropriate for the difference. In the example of FIG. 9, the white part of the difference image 930 is determined to be the area of ​​the object to be inspected 802.

[0064] The mask 905 is selected in the white area of ​​the generated difference image 930 (the area corresponding to the inspection target 802). The upper left and lower right coordinates of the white area of ​​the difference image 930 are identified, and the mask is selected based on the size and slide amount of the mask 905. Specifically, a pattern such as the mask slide image 940 is determined in advance. In the mask slide image 940, the mask is slid from the upper left to the right edge in parallel, and then moved below that from the left edge to the right edge. In this way, a mask sufficient to mask the white area of ​​the difference image 930 is selected.

[0065] 9 shows an example in which the mask generation unit 202 selects a mask from the difference image 930. Here, it is determined that the mask 905 selection area can be covered with a single vertical mask 905. For this reason, four mask patterns 950-1 to 950-4 are generated in which the position of the mask 905 is sequentially changed from left to right.

[0066] This allows us to significantly reduce the number of masked signals input to the inpainting model.

[0067] <Example of a mask selection method other than the background subtraction method> There are other methods for selecting a mask besides the background subtraction method described in Figures 8 and 9. For example, template matching with an image that does not show the object to be inspected, inter-frame subtraction when the object to be inspected is moving, and motion detection using optical flow may be used.

[0068] In the case of template matching, a method can be used in which an image is compared with a template image prepared in advance, and an image that is similar to the template is excluded from the area selected for the mask. In addition, in the case of optical flow, the range of a moving object such as a person is detected, and by masking that part, the rest of the area can be excluded from the area selected for the mask.

[0069] <Example of how to adjust the mask using object detection results> FIG. 10 is a diagram illustrating an example of a method for adjusting a mask using the object detection result in the anomaly detection system of the present invention.

[0070] If the size of the mask is larger than the size of the abnormal part, information about the abnormal part will not be input into the inpainting model. This will improve the accuracy of detecting abnormal parts because the abnormal part will not be included in the image reconstruction. On the other hand, if the mask is superimposed so that most of the object of inspection is hidden, the area to be reconstructed will be larger. This will reduce the accuracy of reconstruction of the object of inspection, which will be a factor in deteriorating detection accuracy. Therefore, by adding a mask size adjustment process, it is possible to further improve detection accuracy.

[0071] Figure 10 shows an example of adjusting the mask size using the object detection results of a DL model that can detect the object to be inspected. To ensure that the object to be inspected can be detected, the DL model is trained in advance using normal detection objects, etc.

[0072] First, an input image 1010 is input to a trained object detection DL model. The object detection DL model predicts the area of ​​the inspection object 1001 and calculates the object detection result 1020. In the object detection result 1020, the range of the inspection object 1001 is specified. In FIG. 10, the ranges of three inspection objects 1001 are specified. Then, a mask 1005 is created that covers part of the specified inspection object 1001 area. Multiple masks are used to cover one of the inspection objects 1001. In FIG. 10, masks are created so that each mask covers 1 / 4 of the area.

[0073] The mask superimposing unit 203 superimposes the created mask on the input image 1010. In Fig. 10, each of the three inspection objects 1001 is covered with 1 / 4 of the area, so a total of 12 masked images 1030-1 to 1030-12 are created. The size of the area covered by the mask may be adjusted appropriately to 1 / n (n is a natural number, and can be an integer of 2 or more, an integer of 4 or more, etc.) of the area of ​​the detection object.

[0074] This configuration restores the signal without concealing most of the object to be inspected, thereby improving the accuracy of the inpainting model and reducing the processing load.

[0075] Alternatively, the size of the inspection target containing the abnormality within the screen may be determined using the background subtraction method as shown in Figures 8 and 9, and then the size of the mask may be adjusted using the method shown in Figure 10.

[0076] <First specific example> Fig. 11 is a diagram showing a model for explaining a first specific example of the anomaly detection system of the present invention. Fig. 12 is a diagram for explaining the first specific example of the anomaly detection system of the present invention.

[0077] In the first specific example, an example is shown in which an anomaly detection is performed by fixing a mask to a moving inspection target. As shown in Fig. 11, an inspection target 1102 moving on a belt conveyor 1103 is photographed from above using a fixed camera 1101. The camera 1101 corresponds to the sensor 102 shown in Figs. 2, 3, and 6, and photographs the inspection target 1102 from above. The multiple inspection targets 1102 move on the belt conveyor 1103 in the direction of the arrow (from right to left) at intervals. At this time, the camera 1101 has a fixed angle of view, and the inspection target 1102 passes laterally within the angle of view of the camera 1101.

[0078] As shown in FIG. 12, the inspection object 1102 moves from the right side to the left side of the image captured by the camera. FIG. 12 shows consecutive input image frames n-1, n, n+1, and n+2. As shown in FIG. 12, the mask 1205 area in the input image 1210 is fixed to a width A (width perpendicular to the direction of travel) through which the inspection object 1102 passes or a width wider than that. In other words, the position and size of the mask 1205 area in the input image 1210 are always constant. Furthermore, the width of the mask 1205 in the direction of travel of the inspection object 1102 is set to be equal to or greater than the length of travel of the inspection object 1102 in one frame. This makes it possible to inspect the entire area of ​​the inspection object.

[0079] Examples of the inspection object 1102 include industrial products, food, transported items such as cardboard boxes, and various other objects that are expected to move.

[0080] In this way, in the first specific example, by fixing the position of the mask 1205, the number of masked images is reduced, and therefore a system that enables real-time analysis can be realized.

[0081] <Second specific example> Fig. 13 is a diagram showing a model for explaining a second specific example of the anomaly detection system of the present invention. Fig. 14 is a diagram for explaining the second specific example of the anomaly detection system of the present invention. The second specific example differs from the first specific example in that the camera moves, but is common in that the inspection target moves relative to the camera.

[0082] In the second specific example, as shown in Fig. 13, a fixed inspection object 1302 is photographed using a camera 1301 that moves in the direction of the arrow (from left to right). The camera 1301 corresponds to the sensor 102 shown in Figs. 2, 3, and 6, and photographs the inspection object 1302 from above. The camera 1301 is equipped with a movement mechanism that enables it to move parallel to the inspection object 1302. At this time, the angle of view of the camera 1301 is fixed, and the inspection object 1302 passes laterally within the angle of view of the camera 1301.

[0083] As shown in Fig. 14, the mask 1405 area in the input image 1410 is fixed to a width equal to or wider than the width B (width perpendicular to the direction of travel) of the inspection object 1302. In other words, the position and size of the mask 1405 area in the input image 1410 are constant. Also, the width of the mask 1405 in the direction of travel of the camera 1301 is set to be equal to or greater than the length that the camera 1301 moves in one frame. This makes it possible to inspect the entire area of ​​the inspection object by capturing images while moving the camera 1301.

[0084] In this way, in the second specific example, it is possible to continuously inspect the inspection object 1302 while moving the camera 1301. This is particularly effective for the inspection object 1302, which is a fixed object such as a long object like an electric wire or an object with a large surface area. Also, when it is desired to inspect the entire circumference of a cross section perpendicular to the direction of travel, it is possible to use multiple cameras arranged in the circumferential direction of the cross section or a mirror that captures the circumferential direction.

[0085] <Example of application to time series data> FIG. 15 is a diagram illustrating an example of application of the anomaly detection system of the present invention to time-series data.

[0086] The input signal used for anomaly detection is not limited to a single signal. For example, in the case of a time-series continuous signal, anomaly detection can be performed by inputting a part of the time-series continuous signal as a mask region into the inpainting model. Figure 15 explains an example of application to time-series data. Here, video data is used as an example of time-series data.

[0087] First, several consecutive frames are extracted from the video data. In Fig. 15, n frames 1510-1 to 1510-n are extracted. Next, a mask is superimposed on some of the frames. In Fig. 15, a mask is superimposed on the entire area of ​​1510-2 to create masked data 1520-2. Because a mask is not superimposed on the other frames 1510-1, 1510-3 to 1510-n, the masked data 1520-1, 1520-3 to 1520-n are the same image as the original frames 1510-1, 1510-3 to 1510-n.

[0088] The mask may be superimposed on the entire area of ​​the frame, or on a part of the area that is the detection area. The frame on which the mask is superimposed may be fixed to one, i.e., the mth frame from the first frame of the extracted frames. Alternatively, the mask may be superimposed on multiple frames.

[0089] Next, a group of images in which a mask is superimposed on some of the frames are input to an inpainting model 1530. The inpainting model 1530 performs reconstruction according to an inpainting technique to generate a group of images of the same number of frames as the number of input frames. In FIG. 15, reconstructed frames 1540-1 to 1540-n are generated. In particular, a masked reconstructed frame 1540-2 is reconstructed.

[0090] Then, the error between the reconstructed frame 1540-2 corresponding to the frame with the mask superimposed and the original frame 1510-2 is calculated to determine whether the frame contains an abnormal part. A threshold is set for the determination, and if there is an error above a certain level, it is determined that there is an abnormal part. In this case, the threshold can be applied in the same way as in Figure 6. Figure 15 shows a binarized error image 1550 showing the range of difference.

[0091] As another alternative, only the frames with the mask superimposed may be reconstructed. As another alternative, instead of superimposing any mask as described above, any frame may be deleted and the remaining frames may be input into the inpainting model, and the deleted frame may be reconstructed.

[0092] Examples of applications to time series data include detecting people or vehicles that are moving unsteadily (detecting abnormal behavior), and detecting stagnation or deviant behavior by people working on a production line.

[0093] Another application example is skeletal detection. For example, a person's skeletal coordinates are estimated for each frame of a video. Then, part of the results are masked, and skeletal coordinates are reconstructed for the frame at the masked time. In this way, abnormal behavior can be detected from the error between the estimated skeletal coordinates and the reconstructed skeletal coordinates.

[0094] In Figure 15, video is used as an example of time-series data, but sensor data such as vibration, voltage, and sound can also be used. In this case, since it is waveform data, part of the waveform data at a certain time is masked and that part is reconstructed. Anomalies can then be detected from the error between the input signal and the reconstructed signal. For example, an anomaly can be detected when there is a peak.

[0095] Furthermore, anomaly detection may be performed after any transformation such as power spectrum or spectrogram is performed on the signal acquired from the sensor 102. For example, any frequency component of the power spectrum converted into the frequency domain using FFT (Fast Fourier Transform) or the like may be masked and input. Then, anomaly detection may be performed by treating the spectrogram as an image and inputting it with a portion masked.

[0096] <Flowchart> FIG. 16 is an example of a processing flowchart of the anomaly detection system of the present invention.

[0097] First, the processor unit of the analysis server 101 shown in Fig. 3 executes a program loaded from the auxiliary storage unit 207 to the main storage unit to start the anomaly detection system. The anomaly detection system may allow the user to check the results using a GUI (Graphical User Interface), or may simply notify the user of the presence or absence of a determined anomaly.

[0098] After the anomaly detection system is started, in step 1601, parameters such as the sensor 102 that acquires the input signal, the frequency at which the input signal is acquired, the size of the input signal, the size and shape of the mask, the number of masks, the amount of sliding, and the error threshold are determined. One mask may be analyzed simultaneously, or multiple masks such as a mesh pattern may be analyzed simultaneously. These parameters may be set by loading a pre-prepared configuration file, or the user may be able to select them using a GUI. The number of sensors 102, such as cameras, that acquire the input signal may be one or more. In the following, an example will be described in which the device that acquires the input signal is a single camera as the sensor 102, and images are acquired sequentially in real time.

[0099] Next, in step 1602, the signal acquisition unit 201 reads the input signal acquired from the camera (sensor 102).

[0100] Next, in step 1603, it is determined whether an end command has been executed. If the end command has been executed, the image analysis system is terminated. If the end command has not been executed, the process proceeds to step 1604. Here, the end command may be entered by operating the keyboard or by operating on the GUI.

[0101] In step 1604, the mask generation unit 202 adjusts and selects a mask. Note that the mask size may be fixed and all masks may be set to be used in step 1601, or the size of an inspection target having an abnormality may be predicted using object detection using a DL model, and the mask may be adjusted to cover part of the predicted inspection target area. Also, a region sufficient to mask the portion where the inspection target object is expected to be located may be selected using background subtraction, template matching, optical flow, etc.

[0102] Next, in step 1605, the mask superimposing unit 203 superimposes one of the masks determined in step 1604 onto the input signal.

[0103] Next, in step 1606, the signal on which the mask has been superimposed is input to the inpaint model in the signal reconstruction unit 204, and the signal is reconstructed.

[0104] Next, in step 1607 , the error between the input signal and the signal reconstructed in step 1606 is calculated in the abnormality determination unit 205 .

[0105] In step 1608, the abnormality determination unit 205 determines whether the error calculated in step 1607 is larger than the error threshold determined in step 1601. If the condition is met, the process proceeds to step 1609;

[0106] In step 1609, the abnormality determination unit 205 determines that the area where the mask is superimposed is an abnormal portion.

[0107] In step 1610, the abnormality determination unit 205 determines whether determination has been performed for all masks selected in step 1604. If the condition is met, the process proceeds to step 1612;

[0108] In step 1611, the mask to be used is changed in the mask superimposing unit 203. Based on the setting in step 1601 and the results of the adjustment and selection in step 1604, a mask that has not been used so far is determined to be used next, and the process returns to step 1605.

[0109] In step 1612, the output control unit 206 notifies the user of the abnormal part. The occurrence notification may be made on the GUI, or an event occurrence notification may be sent to a small terminal. Once the event occurrence notification is complete, the process proceeds to step 1602 and the next input signal is read.

[0110] As described above, by masking the abnormal portion, only the normal signal portion is input, and the signal is reconstructed from the masked normal signal. This allows for highly accurate anomaly detection by comparing the reconstructed signal with the original signal containing the abnormal portion. Furthermore, by optimizing the analysis by selecting the mask to be used and adjusting its size, high-speed and highly accurate anomaly detection can be achieved.

[0111] As described above, the embodiments of the present invention have been described, but the present invention is not limited to the above examples and includes various modifications. For example, the present invention is not limited to those having all of the configurations provided in the above examples. Furthermore, it is possible to delete part of the configuration of one example, replace it with the configuration of another example, or add the configuration of another example to the configuration of one example. [Explanation of symbols]

[0112] 1...computer system, 2...processor, 2A, 2B...processing device, 4...memory, 6...memory bus, 8...I / O bus, 9...bus interface unit, 10...I / O bus interface unit, 12...terminal interface unit, 14...storage interface unit, 16...device interface unit, 18...network interface, 20...user I / O device, 22...storage device, 24...display system, 26...display device, 30...network, 50...latent factor identification application, 1 01...analysis server, 102...sensor, 103...database server, 104...network, 201...signal acquisition unit, 202...mask generation unit, 203...mask superposition unit, 204...signal reconstruction unit, 205...abnormality determination unit, 206...output control unit, 207...auxiliary memory unit, 301...inspection object, 305...mask, 310...input image, 320...masked image, 330...image inpainting, 340...reconstructed image, 401...inspection object, 405...mask, 410...input image, 420...masked image, 430...image inpainting, 440... Reconstructed image, 501...inspection object, 502...anomaly, 505...mask, 510...input image, 520...masked image, 530...image inpainting, 540...reconstructed image, 601...inspection object, 602...anomaly, 605...mask, 610...input image, 620...mask pattern, 630...masked image, 640...reconstructed image, 650...error image, 705...mask, 720...mask pattern, 730...masked image, 801...camera, 802...inspection object, 803...conveyor, 905...mask, 910...input image, 920...background image, 920 ...Mask pattern, 930...Difference image, 940...Mask slide image, 1001...Inspection object, 1005...Mask, 1010...Input image, 1020...Object detection result, 1030...Masked image, 1101...Camera, 1102...Inspection object, 1103...Conveyor belt, 1205...Mask, 1210...Input image, 1301...Camera, 1302...Inspection object, 1405...Mask, 1410...Input image, 1510...Frame, 1520...Masked data, 1530...Inpainting model, 1540...Reconstructed frame, 1550...Error image

Claims

1. a signal acquisition unit that acquires an input signal from a sensor; a mask generator that generates a mask to be superimposed on the input signal; a mask superimposition unit that superimposes the mask generated by the mask generation unit on the input signal to generate a masked signal; a signal reconstruction unit that reconstructs the masked signal generated by the mask superposition unit to generate a reconstructed signal; an abnormality determination unit that determines whether the input signal includes an abnormal portion based on an error between the input signal and the reconstructed signal within a mask region; The signal reconstruction unit reconstructs the masked region using a deep learning model trained using a normal input signal; The anomaly detection system is characterized in that the mask generation unit generates a mesh pattern mask in which multiple locations are arranged in a predetermined pattern, and generates a mask by changing the position of the mesh pattern in sequence for each image to which it is applied, so that when these mesh patterns are combined, they can cover all locations in the image.

2. a signal acquisition unit that acquires an input signal from a sensor; a mask generator that generates a mask to be superimposed on the input signal; a mask superimposition unit that superimposes the mask generated by the mask generation unit on the input signal to generate a masked signal; a signal reconstruction unit that reconstructs the masked signal generated by the mask superposition unit to generate a reconstructed signal; an abnormality determination unit that determines whether the input signal includes an abnormal portion based on an error between the input signal and the reconstructed signal within a mask region; The signal reconstruction unit reconstructs the masked region using a deep learning model trained using a normal input signal; the sensor is a camera and the input signal is an image; The camera captures an image of an inspection object that moves relative to the camera, The anomaly detection system is characterized in that the width of the mask in the direction perpendicular to the direction of movement is fixed to be equal to or wider than the width of the object to be inspected, and the width of the mask in the direction of movement is set to be equal to or greater than the length that the camera moves in one frame.

3. In the anomaly detection system according to claim 1 or claim 2, The anomaly detection system is characterized in that the anomaly judgment unit judges that there is an "anomaly" if the error within the mask area between the input signal and the reconstructed signal is equal to or greater than a predetermined threshold, and judges that there is no anomaly if the error is less than the threshold.

4. The anomaly detection system according to claim 3, The anomaly detection system is characterized in that the judgment using the threshold value involves binarizing the input signal and the reconstructed signal using a predetermined reference value, and judging whether the range of difference due to the binarization is greater than or equal to a predetermined range using a threshold value.

Images