Abnormality detection system, abnormality detection method, and program

Abstract

To easily check a detected state by comparing it with a normal state when detecting an abnormality by picking up an image of an object.SOLUTION: An abnormality detection system has: imaging means that picks up an image of an object; a control unit that detects an abnormality in the object from picked-up image data picked up by the imaging means, and performs image processing on the picked-up image data; information processing means that includes a storage unit, acquires the picked-up image data and stores it in the storage unit, and outputs the picked-up image data to the control unit; and a display unit that displays detected abnormality information. The control unit compares a plurality of pieces of picked-up image data obtained by the imaging means picking up images of the object at a predetermined interval, sequentially along the time series of imaging times, when a change area of the compared pieces of picked-up image data is present, sets a marking surrounding the area to the extracted change area and displays it superimposed on the picked-up image data, and when determining that an abnormality occurs in the object on the basis of the change area, displays information on the abnormality in the object on the display unit in association with the imaging time.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to an anomaly detection system, an anomaly detection method, and a program. [Background technology]

[0002] Conventionally, various techniques have been proposed for detecting abnormalities occurring in factories or facilities (Patent Documents 1 and 2).

[0003] Patent Document 1 discloses a mechanism for detecting anomalies using background subtraction, in which a binary image obtained from a difference image, which is the brightness difference between the monitoring image of the leak monitoring area and a reference image, is divided into multiple sections, a counter is provided for storing the degree of continuous detection for each section, the current degree of continuous detection for a section in which detection occurred in the binary image is calculated by adding 1 to the previous degree of continuous detection stored in the counter corresponding to that section, the current degree of continuous detection for sections in which no detection occurred is set to zero, the current degree of continuous detection for each section is stored in the corresponding counter, and a leak is detected by detecting a section in which the current degree of continuous detection is above a predetermined upper limit.

[0004] Patent Document 2 discloses a system including a solid-liquid separation device that separates water to be treated into solids and liquids to produce a separated product, an imaging device that generates image data, an image analysis system that analyzes an abnormal state of the separated product based on the image data, an edge server, and a cloud server, the cloud server having a web application that displays the analysis results of the abnormal state and the image data on the web application, the edge server having an inference model constructed by machine learning using training data that includes at least the image data, the image data being input into the inference model, and the system infers an abnormal state of the separated product according to the inference model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 8-122197 [Patent Document 2] Japanese Patent Publication No. 2022-137973 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technology described in Patent Document 1 does not consider at all a method for displaying a section when an abnormality is determined. The technology described in Patent Document 2 discloses a mechanism for displaying an image when an abnormality occurs, but it is difficult to compare normal and abnormal states or to compare them over time. Therefore, there has been a demand for technology that, when detecting abnormalities in the operating state or operational state of an object such as equipment in a factory or facility by capturing an image of the object, the detected state can be easily compared with the normal state and confirmed.

[0007] The present invention has been made in consideration of such problems, and its purpose is to provide an anomaly detection system, an anomaly detection method, and a program that, when detecting an abnormality by imaging an object, can easily compare the detected state with a normal state and confirm it. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the above-mentioned object, an anomaly detection system according to one aspect of the present invention comprises: an imaging means configured to be able to capture an image of an object at predetermined time intervals; a control unit configured to be able to detect an abnormality that has occurred in the object based on captured image data generated by the imaging means and to be able to perform predetermined image processing on the captured image data; an information processing means including a memory unit for storing information, configured to acquire captured image data of the object captured by the imaging means and store it in the memory unit, and to be able to output the captured image data to the control unit; and a display unit configured to be able to display information regarding the detected abnormality, wherein the control unit is configured to The imaging device sequentially compares a plurality of pieces of image data of the object taken at predetermined intervals in chronological order of the image capturing times, and if a change area in which the difference in luminance is equal to or greater than a predetermined difference in luminance is present in at least two of the compared image data, a marking is set to surround the change area extracted in the image data associated with the image capturing time, and the marking is superimposed on the image data and the image data, and if it is determined that an abnormality has occurred in the object based on the change area, abnormality information including details of the abnormality occurring in the object at the image capturing time is displayed on the display device, associated with the image capturing time.

[0009] In one aspect of the anomaly detection system of the present invention, in the above invention, the control unit determines that an abnormality has occurred in the object when the number of change areas in the captured image data continues along the time series and exceeds a predetermined threshold.

[0010] In one aspect of the anomaly detection system of the present invention, in the above invention, the captured image data and the anomaly information are associated based on the image capture time, and when the display unit selects either the captured image data displayed on the display unit or the anomaly information displayed on the display unit, it displays the other anomaly information or captured image data associated by the image capture time, corresponding to the selected anomaly information or captured image data.

[0011] An anomaly detection method according to one aspect of the present invention is an anomaly detection method executed by an anomaly detection system including: an imaging means configured to be able to capture an image of an object at predetermined time intervals; a control unit configured to be able to detect an abnormality occurring in the object based on captured image data generated by the imaging means and to perform predetermined image processing on the captured image data; an information processing unit including a storage unit for storing information, configured to acquire captured image data of the object captured by the imaging means and store it in the storage unit, and to output the captured image data to the control unit; and a display unit configured to display information regarding the detected abnormality, wherein the control unit The imaging device sequentially compares a plurality of pieces of image data of the object taken at predetermined intervals in the chronological order of the image capturing times, and if a change area is present in at least two of the compared image data, a marking is set to surround the change area extracted in the image data associated with the image capturing time, and the display unit displays the marking superimposed on the image data and the image data, and if it is determined that an abnormality has occurred in the object based on the change area, abnormality information including details of the abnormality occurring in the object at the image capturing time is displayed on the display unit in association with the image capturing time.

[0012] A program according to one aspect of the present invention is an anomaly detection system including: an imaging means configured to be able to image an object at predetermined time intervals; a control unit configured to be able to detect an abnormality that has occurred in the object based on image data generated by the imaging means and to be able to perform predetermined image processing on the image data; a storage unit for storing information, information processing means configured to acquire image data of the object imaged by the imaging means and store it in the storage unit and output the image data to the control unit; and a display unit configured to be able to display information about the detected abnormality, the program including: an imaging means configured to be able to image an object at predetermined time intervals based on image data generated by the imaging means; The computer sequentially compares the plurality of pieces of image data obtained by capturing images of the object at each image capture time in chronological order of the image capture times, and when a change area is present in at least two of the compared image data, a marking is set to surround the change area extracted in the image data associated with the image capture time, and the marking is superimposed on the image data and the image data on the display unit, and when it is determined that an abnormality has occurred in the object based on the change area, abnormality information including details of the abnormality occurring in the object at the image capture time is displayed on the display unit in association with the image capture time. [Effects of the Invention]

[0013] According to the anomaly detection system, anomaly detection method, and program of the present invention, when detecting an anomaly by imaging an object, it becomes possible to easily compare the detected state with a normal state and confirm it. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram that schematically shows an anomaly detection system to which an information processing device according to an embodiment of the present invention can be applied. [Figure 2] FIG. 2 is a block diagram schematically showing the configuration of the information processing device shown in FIG. [Figure 3]FIG. 3 is a block diagram schematically showing the configuration of the plant control device shown in FIG. [Figure 4] FIG. 4 is a block diagram schematically showing the configuration of the image processing server shown in FIG. [Figure 5] FIG. 5 is a block diagram schematically showing the configuration of the imaging device shown in FIG. [Figure 6] FIG. 6 is a block diagram schematically showing the configuration of the user terminal shown in FIG. [Figure 7] FIG. 7 is a diagram illustrating an example of a display method displayed on the output unit of the information processing device according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings of the embodiment below, the same or corresponding parts are designated by the same reference numerals. Furthermore, the present invention is not limited to the embodiment described below.

[0016] First, before describing the embodiments of the present invention, the problems that the inventors have found and the intensive studies that have been carried out to solve the problems will be described.

[0017] First, the present inventor conducted various studies on technologies for detecting abnormalities occurring in factories or facilities by imaging. The present inventor examined the technologies described in Patent Documents 1 and 2 and studied a method for deriving a difference image, which serves as a criterion for determining whether an abnormality exists, by capturing images of the inside of a factory or facility using a predetermined imaging unit such as a camera. The present inventor then found that there was room for improvement in the visibility when visualizing a change area obtained from the difference image compared to the technologies described in Patent Documents 1 and 2. According to the present inventor's study, the technology described in Patent Document 2 in particular needed to improve the visibility when comparing a normal image with an abnormal image.

[0018] The inventors have further investigated ways to improve visibility, and have devised a method for deriving changed areas that serve as criteria for determining whether or not an abnormality exists, by displaying a graph showing the presence or absence of changed areas and the difference amount calculated as the area of ​​the part where a change has occurred between multiple images, for each time series at a fixed time interval, and displaying this time series of images at this fixed time interval on the same screen.

[0019] Furthermore, the present inventors have devised a method for displaying a change area determined to be abnormal as a closed area of ​​a predetermined shape using, for example, a predetermined color or pattern. The color may be any color that has high visibility when displayed on a display unit, and in this embodiment, red, for example, may be used. The predetermined shape may be any shape, such as a rectangle or a circle, that makes it easy to find the location of the abnormality.

[0020] The inventors have also devised a method of displaying a time-series graph along with an image on a display unit in different colors for normal and abnormal conditions. This makes it possible to clarify the timing at which an abnormality is detected and improve visibility. The inventors have also devised a method of allowing a user to arbitrarily set a detection exclusion area on a display unit that displays a detected abnormality, so that even if a change area is detected on the screen, the area is not determined to be an abnormal area. As a result of the inventors' research, it has become possible to clarify on the screen of the display unit in which part of a captured image an abnormality was detected, thereby improving visibility.

[0021] Specifically, the calculated difference amount is displayed on the display screen as a graph, such as a line graph, at predetermined time intervals, for example, every five minutes. The time interval can be changed as needed. A preset threshold is displayed as a line of a predetermined color, for example, a red line. When the difference amount exceeds the threshold, the color of the bar graph displayed on the display screen changes from a safety-inducing color, such as green, to a cautionary color, such as red. This makes it clear that an abnormality has been determined in the captured image and improves visibility. Furthermore, captured images, for example, every five minutes, are displayed side by side on the display screen, and if an abnormality is detected, a red box is automatically added to the detected portion. Furthermore, captured images from previous capture times can be viewed by scrolling the images horizontally or tapping the graph at the top, as needed. Displaying the graph and image on the same screen of the display screen makes it easy to compare the captured image at the time the abnormality was detected, or the captured image during normal and abnormal conditions. Furthermore, since the parts detected as differences can be marked and displayed, visibility is improved, and the user can quickly check whether the abnormality determination is correct or whether it is a false detection. The embodiment described below was devised by the inventors through the above-mentioned intensive studies.

[0022] (Anomaly detection system) First, an anomaly detection system to which an information processing device according to one embodiment of the present invention can be applied will be described. Fig. 1 is a schematic diagram showing an anomaly detection system 1 to which an information processing device according to this embodiment can be applied. As shown in Fig. 1, the anomaly detection system 1 according to this embodiment includes an information processing device 10, a plant control device 20, an image processing server 30, an imaging device 40, and a user terminal 50, which are capable of communicating with each other via a network 2. In the following description, information is transmitted and received between the respective components via the network 2, but a detailed description of this point will be omitted.

[0023] The network 2 is configured from an internet network, a mobile phone network, etc. The network 2 is, for example, a public communication network such as the internet, and may also include other communication networks such as a wide area network (WAN), a telephone communication network such as a mobile phone, or a wireless communication network such as Wi-Fi.

[0024] (Information processing device) 2 is a block diagram showing a schematic configuration of the information processing device 10. As shown in FIG. 2, the information processing device 10 has the configuration of a general computer that is capable of communicating via a network 2. The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an output unit 14, and an input unit 15.

[0025] Specifically, the control unit 11 includes a processor such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field-Programmable Gate Array), and a main memory unit such as a RAM (Random Access Memory) or a ROM (Read Only Memory) (none of which are shown).

[0026] The storage unit 12 is configured with a storage medium selected from an EPROM (Erasable Programmable ROM), a hard disk drive (HDD), and removable media. The removable media is, for example, a Universal Serial Bus (USB) memory or a disc storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark) Disc (BD). The storage unit 12 can store an operating system (OS), various programs, various tables, various databases, and the like.

[0027] The control unit 11 loads a program stored in the storage unit 12 into a working area of ​​the main storage unit, executes it, and controls each component through the execution of the program, thereby realizing a function that matches a predetermined purpose. The control unit 11 functions as an image processing unit 111 through the execution of the program.

[0028] The storage unit 12 stores an image database 121 in which various types of data are stored in a searchable manner. The image database 121 is, for example, a relational database (RDB). The database (DB) described below is constructed by a database management system (DBMS) program executed by the above-mentioned processor controlling the data stored in the storage unit 12. The image database 121 stores searchable image information including captured images of equipment installed in factories, facilities, etc. When the information processing device 10 communicates with a user terminal 50, it is also possible to associate unique user identification information for identifying the user terminal 50 with user input information input by a user to the user terminal 50 and store the associated information in the image database 121.

[0029] The communication unit 13 is, for example, a LAN (Local Area Network) interface board or a wireless communication circuit for wireless communication. The LAN interface board or the wireless communication circuit is connected to a network 2 such as the Internet, which is a public communication network. The communication unit 13 is connected to the network 2 and communicates with the plant control device 20, the image processing server 30, the imaging device 40, and the user terminal 50. The communication unit 13 receives image identification information and image information unique to each image processing server 30 from each image processing server 30, and transmits various instruction signals and confirmation signals to the image processing server 30.

[0030] Furthermore, the communication unit 13 transmits information to the user terminal 50 owned by the user when the user uses the image processing server 30, and receives user identification information and various information for identifying the user from the user terminal 50. The user identification information includes various information for identifying individual users from each other. The user identification information is, for example, a user ID that can identify each individual user terminal 50, and includes information necessary for accessing the information processing device 10 when transmitting information related to the user terminal 50. When the user terminal 50 transmits predetermined information such as user input information together with the user identification information to the information processing device 10, the information processing device 10 associates the predetermined information with the user identification information and stores it in a searchable state in the image database 121 of the storage unit 12.

[0031] The output unit 14 as an output means is configured to notify the outside of predetermined information by displaying characters, figures, etc. on the screen of a display such as a liquid crystal display or a plasma display, or by outputting sound from a speaker, under the control of the control unit 11. Furthermore, the output unit 14 includes a printer that outputs predetermined information by printing it on printing paper, etc. The various information stored in the memory unit 12 can be confirmed, for example, on a monitor of the output unit 14 installed in a predetermined office, etc.

[0032] The input unit 15 as an input means is configured, for example, with a keyboard or a touch panel keyboard that is incorporated inside the output unit 14 and detects touch operations on the display panel, or a voice input device that enables calls with the outside, etc. The output unit 14 and the input unit 15 may be integrated into one unit to form an input / output unit such as a touch panel display or a speaker / microphone.

[0033] (Plant control device) FIG. 3 is a block diagram showing a schematic configuration of a plant control device 20. As shown in FIG. 3, the plant control device 20 has a configuration capable of communicating via a network 2. The plant control device 20 is configured, for example, as a distributed control system (DCS) made up of at least one computer. The plant control device 20 is installed in a plant such as a factory or facility, and is configured to be able to control equipment installed in the plant. The plant control device 20 includes a plant control unit 21, a memory unit 22, a communication unit 23, and an input / output unit 24. The plant control unit 21, the memory unit 22, and the communication unit 23 have physical configurations similar to those of the control unit 11, the memory unit 12, and the communication unit 13 described above, respectively.

[0034] The storage unit 22 is configured to be able to store various types of information (plant information) supplied from equipment installed in the plant. The plant control unit 21 may further collect various types of information from the imaging device 40 and individual user terminals 50 and store the information in the storage unit 22. The communication unit 23 is connected to the network 2 and is capable of communicating with the information processing device 10, the image processing server 30, the imaging device 40, and the user terminal 50. The input / output unit 24 has a configuration similar to the above-mentioned configuration in which the output unit 14 and the input unit 15 are integrated together, but the input unit and the output unit may be configured separately.

[0035] (Image processing server) Fig. 4 is a block diagram showing a schematic configuration of the image processing server 30. As shown in Fig. 4, the image processing server 30, which is at least a part of the information processing means, has a configuration capable of communicating via the network 2. The image processing server 30 is provided in a cloud connected to the network 2. The image processing server 30 is an image processing system server that performs predetermined image processing on image information received via the network 2. Here, the image processing system is a series of system configurations that process, synthesize, and read the characteristics of two-dimensional or three-dimensional images or data.

[0036] The image processing server 30 includes a control unit 31, a storage unit 32, and a communication unit 33. The control unit 31, the storage unit 32, and the communication unit 33 have the same physical configuration as the above-described control unit 11, the storage unit 12, and the communication unit 13, respectively. The control unit 31 of the image processing server 30 includes an image processing unit 311 and a difference processing unit 312. The communication unit 33 is connected to the network 2 and is capable of communicating with the information processing device 10, the plant control device 20, the imaging device 40, and the user terminal 50.

[0037] The storage unit 32 is configured to be able to store image information relating to captured images transmitted from the information processing device 10 or an imaging device 40 (described later). The storage unit 32 is able to store various programs, various tables, various databases, etc. The storage unit 32 of the image processing server 30 stores databases including an image database 321, etc., and programs including an image processing program 322 and a difference processing program 323, etc.

[0038] The control unit 31 loads the program stored in the storage unit 32 into the working area of ​​the main storage unit, executes it, and controls each component unit through the execution of the program, thereby realizing functions that meet a predetermined purpose. The control unit 31 functions as an image processing unit 311 through the execution of an image processing program 322, and as a difference processing unit 312 through the execution of a difference processing program 323.

[0039] The image processing unit 311 of the control unit 31 performs predetermined processing on image information such as captured image data stored in the image database 321. The difference processing unit 312 is configured to be able to perform difference processing, which compares multiple pieces of image data and derives and identifies differences between the compared pieces of image data. As described above, various data stored in the image database 121 is stored appropriately in corresponding databases in the image processing server 30. Specifically, for example, image information including various pieces of captured image data acquired from the camera group 45 and stored in the image database 121 of the information processing device 10 is transmitted to the image processing server 30 and stored sequentially or appropriately in the corresponding image database 321.

[0040] (imaging device) Fig. 5 is a block diagram showing a schematic configuration of the imaging device 40. As shown in Fig. 5, the imaging device 40 is configured to include a group of cameras that can communicate via a network 2. The imaging device 40 includes a camera control unit 41, a storage unit 42, and a communication unit 43. The camera group 45 is provided in a plant 44. The imaging device 40 captures images of equipment provided in the plant 44 using the camera group 45 and transmits the images to the communication unit 43 of the imaging device 40. Various information may be supplied to the imaging device 40 from the information processing device 10 or the plant control device 20.

[0041] The camera control unit 41 and the storage unit 42 have the same physical configuration as the above-described control unit 11 and storage unit 12, respectively. The communication unit 43 has the same configuration as the above-described communication unit 13, is connected to the network 2, and transmits images captured at predetermined time intervals, such as one-minute intervals, to the information processing device 10 and the plant control device 20 sequentially or as needed. Note that the time intervals and frequency of capturing images are not limited to these, and the number of image data to be acquired can also be set appropriately depending on the time intervals and frequency of capturing images.

[0042] The communication unit 43 transmits and receives captured image information including image data captured by the camera group 45 and at least a part of the above-mentioned image identification information to and from at least the information processing device 10. The communication unit 43 may also communicate with the image processing server 30 or the user terminal 50 as necessary. The communication unit 43 may also transmit and receive captured image information and image identification information between the plant control device 20 and the imaging device 40.

[0043] The plant 44 may be, for example, a facility that separates recyclable resources, incinerates combustible waste, or detoxifies residue, or may be a facility for composting food waste or a sludge treatment facility for a sewage treatment plant. That is, examples of the plant 44 include factories such as waste disposal plants and manufacturing plants, and facilities such as waste incineration plants, but are not necessarily limited to these facilities.

[0044] Examples of conditions under which the anomaly detection system 1 according to this embodiment can be applied include indoor locations where the lights are not turned on or off for 24 hours, locations where sunlight does not enter from outdoors, and so-called locations where changes in brightness are small. Other applicable conditions include locations where there is very little change in appearance under normal conditions and detection is required when an irregular phenomenon occurs, locations where detection is required when an animal or person remains in a restricted area, including but not limited to liquid leakage and parts falling off, and locations where detection is required when an insect or other insect enters a location where the intrusion of insects or other insects is undesirable.

[0045] The camera group 45 is a group of cameras installed inside these plants 44 to capture images of the equipment inside various facilities and factories, as well as the conditions inside the facilities and factories. The camera group 45 is composed of, for example, approximately 30 cameras serving as imaging means, arranged in various locations. The camera group 45 is composed of, for example, multiple cameras, such as CCD cameras or CMOS cameras, but is not necessarily limited to these. The individual cameras constituting the camera group 45 are preferably installed near pumps, tanks, conveyors, etc. to detect leaks such as liquid leaks and ash leaks, but are not necessarily limited to these. Furthermore, when detecting flames, smoke, or fallen parts in a plant, each camera constituting the camera group 45 is installed near a location where fire, smoke, or fallen parts are expected.

[0046] (user terminal) The user terminal 50, which serves as a user terminal, is operated by a worker who is a user. The user terminal 50 transmits various information, such as user information including user identification information and user input information, to the information processing device 10, for example, by a call using various data and voice via a communication application. The user terminal 50 is configured to be able to receive various information, such as captured image information, information related to background difference (background difference information), and time-series graphs, from the image processing server 30. FIG. 6 is a block diagram showing a schematic configuration of the user terminal 50.

[0047] 6, user terminal 50 includes electronic control unit 51, memory unit 52, communication unit 53, and input / output unit 54, which are communicatively connected to one another. Electronic control unit 51, memory unit 52, and communication unit 53 have the same physical configurations as control unit 11, memory unit 12, and communication unit 13 described above, respectively.

[0048] The electronic control unit 51 executes an operating system (OS) and various application programs stored in the storage unit 52 to comprehensively control the operations of the storage unit 52, the communication unit 53, and the input / output unit 54. The communication unit 53 transmits and receives various information, such as user input information, to and from an external server such as the information processing device 10 via the network 2.

[0049] The input / output unit 54 has a display unit 541 capable of displaying images. The display unit 541 of the input / output unit 54 serving as output means is configured to be capable of displaying captured images and graphs accompanied by figures on the display screen under control of the electronic control unit 51. The input / output unit 54 serving as input means is configured with a touch panel display that allows touch operation of the display panel and a speaker / microphone that allows input and output by voice. Here, in the user terminal 50, communication with the outside world also includes communication with an operator resident in the information processing device 10 and an artificial intelligence system connected to the network 2. The input / output unit 54 may be configured as a separate entity, similar to the output unit 14 and input unit 15 described above.

[0050] Specifically, the above-mentioned user terminal 50 may be a mobile phone such as a smartphone, a notebook or tablet type information terminal, or a notebook or desktop type personal computer.

[0051] (Information processing method) Next, an anomaly detection method executed by the information processing device 10 of the above-described anomaly detection system 1 will be described.

[0052] First, the information processing device 10, which constitutes a part of the information processing means, acquires captured image data from the imaging device 40 and stores it in the image database 121 of the storage unit 12. The information processing device 10 transmits the acquired captured images to the image processing server 30. Here, each camera constituting the camera group 45 in the imaging device 40 is installed near, for example, a pump, a tank, or a conveyor. The camera group 45 captures images of these objects at intervals of, for example, one minute to generate captured images every minute and transmit the images to the information processing device 10. The information processing device 10 stores the acquired captured images in the image database 121 of the storage unit 12 and transmits them to the image processing server 30.

[0053] In the image processing server 30, the control unit 31 loads the difference processing program 323 and executes difference processing by the difference processing unit 312. That is, the difference processing unit 312 first acquires, as still image data, images captured by the camera group 45 over a predetermined period of time, such as the most recent five minutes, from the information processing device 10. Here, if the imaging device 40 captures images of an object at predetermined intervals of one minute, five still image data sets will be acquired for five minutes of still image data.

[0054] The differential processor 312 of the image processing server 30 extracts the median luminance value of the pixels in the acquired image data for each pixel coordinate (x, y) of each image data. The differential processor 312 then assigns the median luminance information to the corresponding pixel coordinate (x, y). This generates an image (hereinafter, a median image) that aggregates the median luminance values ​​for each pixel. The generated median image is stored in the image database 321 of the storage unit 32. Here, by generating a median image from multiple still image data captured at predetermined time intervals, such as one minute (e.g., five still image data captured over a five-minute period), even if a momentary luminance change occurs in part of the still image data, such as when a user or other worker passes in front of one of the cameras 45 in a factory or facility, the influence of the luminance change can be eliminated from the median image. Using this median image as the image to be monitored reduces the likelihood of falsely detecting a change in the situation due to a user or other worker passing through a facility or factory, compared to using the still image data as is. In contrast, liquid leaks and ash leaks from equipment often occur continuously, and when generating a median image, it is highly likely that the majority of still images will contain images of these liquid leaks and ash leaks.Therefore, by using the median image as a monitoring image, it is possible to detect continuous abnormalities such as liquid leaks and ash leaks.

[0055] Once the median image is generated, the subtraction processing unit 312 compares the generated median image with a still image of a normal state (hereinafter referred to as the normal image). Here, the normal image is typically composed of a median image of image data captured under normal conditions. However, it is also possible to register a median image generated at any timing as the normal image. Furthermore, it is also possible to register a median image at a predetermined time after the normal state is restored after an abnormality has been detected as the normal image. Figure 7 shows an example display for explaining a display method executed by an information processing method according to an embodiment of the present invention. As shown in Figure 7, the subtraction processing unit 312 compares the normal image with the generated median image, and marks, for example, a rectangular area, a change area, which is a pixel area where the difference in brightness is greater than or equal to a predetermined difference in brightness.

[0056] In the example shown in FIG. 7 , on the display screen 100 of the output unit 14 of the information processing device 10 or the display unit 541 of the user terminal 50, the subtraction processing unit 312 detects an abnormal portion of the median image 102 at the time "YYYY-MM-DD 16:25:00" in the screen area, relative to the median image 101, which is a normal image at the time "YYYY-MM-DD 16:20:00." Note that "YYYY-MM-DD" refers to the "year-month-day" format, which indicates the date and time of image capture. Images (median images) every 5 minutes, for example, are displayed horizontally on the display unit 541 of the output unit 14 or the user terminal 50. Here, by tapping a predetermined arrow 107a (the left and right circled arrows in FIG. 7 ), the displayed image can be scrolled horizontally to view the image at any desired time.

[0057] The subtraction processing unit 312 generates, for example, rectangular regions in pixel regions where the difference in brightness is greater than or equal to a predetermined difference in brightness, generates markings 104a and 104b on the median image 102, and displays them superimposed on the median image 102. Note that the markings 104a and 104b are not limited to rectangular shapes, and the types of the markings 104a and 104b are not limited. Thereafter, five minutes later, at "YYYY-MM-DD 16:30:00," the subtraction processing unit 312 extracts pixel regions where the difference in brightness from the normal image included in the median image 103 is greater than or equal to a predetermined difference in brightness, and generates marking 105 as a changed region and displays it superimposed.

[0058] Here, the difference processing unit 312 generates markings 104a, 104b, and 105 by extracting changed areas from the median images 101, 102, and 103 on the display screen 100 to be displayed on the output unit 14 or the display unit 541. The image processing unit 311 of the image processing server 30 superimposes the markings 104a, 104b, and 105 on the median images 101, 102, and 103 displayed on the display unit 541 or the like. A toolbar 106 for adjusting the median images 101 to 103 and the screen is displayed on the screen. The user can zoom in or out on an image by tapping a specific part of the toolbar 106 according to the desired operation.

[0059] Furthermore, if the area of ​​the change region, i.e., the sum of the areas of the markings 104a and 104b or the area of ​​the marking 105, is equal to or greater than a predetermined value, the subtraction processor 312 determines that an abnormality has occurred at the location where the image is captured. Information about the image capture locations of the median images 101, 102, and 103 may be included in the image identification information. The subtraction processor 312 continuously generates the markings 104 and 105 and determines whether an abnormality has occurred. If the subtraction processor 312 determines an abnormality a predetermined number of times, for example, three consecutive times for a total of 15 minutes, it transmits an alarm signal to the plant control device 20. In this case, the median images 102 and 103 have been determined to be abnormal for, for example, three images over a 15-minute period. Various time periods can be set for determining the duration of an abnormality before transmitting an alarm signal, based on the predetermined number of times and the time interval between the median images 101 to 103.

[0060] 3, upon receiving an alarm signal, the plant control unit 21 of the plant control device 20 outputs abnormality information indicating that an abnormality has occurred from the input / output unit 24. Here, the input / output unit 24 can notify the outside of the occurrence of the abnormality by outputting audio, images, or videos. Upon receiving the alarm from the plant control device 20, the user checks the images captured by the camera group 45 using the user terminal 50 or the like, and performs processing to check the site and eliminate the abnormality as necessary.

[0061] As shown in FIG. 7 , the difference processing unit 312 of the image processing server 30 generates time-series data 107 based on the calculated difference amount at predetermined intervals, for example, every five minutes, and outputs the time-series data 107 to the output unit 14 or the display unit 541. The time interval of the time-series data 107 can be arbitrarily set by the user. In the example shown in FIG. 7 , the time-series data 107 is output as anomaly information using a predetermined graph. However, various display methods can be set, and the difference amount can be output as a line graph 107b. The time-series data 107 is associated with median images 101 to 103 based on the imaging time. Thus, by tapping any position on the time-series data 107, the median images 101 to 103 corresponding to the time of the tapped portion are displayed below the time-series data 107. Various arrangements can be set for the median images 101 to 103 and the time-series data, and are not limited to the example shown in FIG. 7 .

[0062] In the example of the time-series data 107 shown in FIG. 7, a preset threshold is indicated by a red line (dotted line in FIG. 7). When the difference amount calculated for the median images 101 to 103 exceeds the threshold for determining an "abnormality," the difference processing unit 312 changes the color of the bar graph of the time-series data 107, for example, from green (normal part: lightly hatched part in FIG. 7) to red (abnormal part: darkly hatched part in FIG. 7). When an operator visits the location where the abnormality occurred and takes measures to correct the abnormality ("taking measures"), the graph corresponding to that time is changed to, for example, yellow (diagonal lines in FIG. 7). This indicates that the "taking measures" period indicates that the operator is taking measures at the location where the abnormality occurred. In this case, a user such as an operator can arbitrarily change the status to "taking measures." Furthermore, when the status is set to "taking measures," the abnormality detection process may be stopped.

[0063] According to the embodiment described above, the output unit 14 of the information processing device 10 and the display unit 541 of the user terminal 50 display multiple median images of multiple images taken at predetermined positions within the plant 44, and median images 101 to 103 are displayed by extracting change areas that are indicators of the occurrence of an abnormality and superimposing them using markings 104, 105, and time-series data 107 showing the calculated difference amount at regular intervals are displayed simultaneously on the same display screen 100, thereby making it possible to clarify in which part an abnormality has been detected, and to easily compare and confirm the abnormal part with the normal part from the median images 101 to 103 obtained by imaging the equipment.

[0064] (Recording medium) In the above-described embodiment, a program capable of executing the image display method can be recorded on a computer-readable recording medium for use in a computer or other machine or device (hereinafter, referred to as a computer, etc.). By loading and executing the program from the recording medium into a computer, etc., the computer functions as the information processing device 10 or the user terminal 50. Here, a computer-readable recording medium refers to a non-transitory recording medium that stores information such as data and programs through electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc. Examples of such recording media that are removable from a computer, etc. include flexible disks, magneto-optical disks, CD-ROMs, CD-R / Ws, DVDs, BDs, DATs, magnetic tapes, and memory cards such as flash memory. Furthermore, examples of recording media that are fixed to a computer, etc. include hard disks and ROMs. Furthermore, SSDs can be used as both recording media that are removable from a computer, etc. and recording media that are fixed to a computer, etc.

[0065] Furthermore, the program executed by the driving assistance device according to an embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0066] (Other embodiments) Furthermore, in the information processing device 10, the plant control device 20, the image processing server 30, the imaging device 40, and the user terminal 50 according to an embodiment, the above-mentioned "unit" can be read as a "circuit" or the like. For example, the communication unit can be read as a communication circuit.

[0067] Further effects and modifications can be easily derived by those skilled in the art. The broader aspects of the present invention are not limited to the specific details and representative embodiments shown and described above. Therefore, various changes are possible without departing from the spirit or scope of the general inventive concept defined by the appended claims and their equivalents. For example, the numerical values ​​and types of information listed in the above embodiment are merely examples, and different numerical values ​​and types of information may be used as necessary. The present invention is not limited by the description and drawings that form part of the disclosure of the above embodiment of the present invention.

[0068] In the embodiment described above, the information processing device 10, the plant control device 20, and the imaging device 40 are configured as separate entities, but they can also be configured as a single information processing device. Furthermore, it is also possible to configure multiple of these as a single server, with at least one other server configured as a separate entity. For example, the information processing device 10 and the plant control device 20 can be configured as a single cloud server, or the information processing device 10, the plant control device 20, and the imaging device 40 can be configured as a single server. [Explanation of symbols]

[0069] 1. Anomaly detection system 2 Network 10. Information processing equipment 11,31 Control unit 12,22,32,42,52 Storage section 13, 23, 33, 43, 53 Communications Department 14 Output section 15 Input section 20 Plant control device 21 Plant Control Section 24 Input / output section 30 Image Processing Server 40 Imaging device 41 Camera control unit 44 Plant 45 cameras 50 User Terminals 51 Electronic control unit 54 Input / output section 100 display screens 101,102,103 Median image 104, 104a, 104b, 105 marking 106 Toolbar 107 Time Series Data 107a Arrow 107b Line graph 111,311 Image processing unit 121,321 image database 312 Differential Processing Unit 322 Image Processing Program 323 Differential Processing Program 541 Display section

Claims

1. an imaging means configured to be able to capture images of an object at predetermined time intervals; a control unit configured to be able to detect an abnormality occurring in the object based on the captured image data generated by the imaging means, and to be able to perform predetermined image processing on the captured image data; an information processing means configured to acquire captured image data of the object captured by the imaging means, store the captured image data in the storage means, and output the captured image data to the control means; a display unit configured to be able to display information about the detected abnormality, The control unit a plurality of pieces of image data of the object captured at predetermined intervals by the imaging means are compared in a time series of the image capturing times, and when a change area in which the difference in luminance is equal to or greater than a predetermined difference in luminance exists in at least two of the compared image data, a marking is set to surround the change area extracted in the image data associated with the image capturing times, and the image data and the marking are superimposed on the image data and displayed on the display unit; When it is determined that an abnormality has occurred in the object based on the changed region, abnormality information including details of the abnormality occurring in the object at the image capturing time is displayed on the display unit in association with the image capturing time. Anomaly detection system.

2. The control unit When the number of change regions in the captured image data that continues along the time series exceeds a predetermined threshold, it is determined that an abnormality has occurred in the object. The anomaly detection system according to claim 1 .

3. The captured image data and the abnormality information are associated with each other based on the image capturing time, The display unit When one of the captured image data displayed on the display unit and the anomaly information displayed on the display unit is selected, the other anomaly information or captured image data associated with the image capturing time is displayed in accordance with the selected anomaly information or captured image data. The anomaly detection system according to claim 1 .

4. an imaging means configured to be able to capture images of an object at predetermined time intervals; a control unit configured to be able to detect an abnormality occurring in the object based on the captured image data generated by the imaging means, and to be able to perform predetermined image processing on the captured image data; an information processing means configured to acquire captured image data of the object captured by the imaging means, store the captured image data in the storage means, and output the captured image data to the control means; a display unit configured to be able to display information about the detected abnormality; and an abnormality detection method executed by the control unit in an abnormality detection system having the display unit, The control unit a plurality of pieces of image data of the object captured at predetermined intervals by the imaging means are compared in a time series of the image capturing times, and when a changed area is present in at least two of the compared image data, a marking is set to surround the changed area extracted in the image data associated with the image capturing times, and the image data and the marking are superimposed on the image data and displayed on the display unit; When it is determined that an abnormality has occurred in the object based on the changed region, abnormality information including details of the abnormality occurring in the object at the image capturing time is displayed on the display unit in association with the image capturing time. Anomaly detection methods.

5. an imaging means configured to be able to image an object at predetermined time intervals; a control unit configured to be able to detect an abnormality occurring in the object based on image data generated by the imaging means and to be able to perform predetermined image processing on the image data; an information processing means including a storage unit for storing information, configured to acquire image data of the object imaged by the imaging means and store it in the storage unit, and to be able to output the image data to the control unit; and a display unit configured to be able to display information related to the detected abnormality, The method sequentially compares a plurality of pieces of image data obtained by imaging the object at predetermined intervals by the imaging means in a time series of imaging times, and when a changed area is present in at least two of the compared image data, sets a marking surrounding the changed area for the changed area extracted in the image data associated with the imaging time, and displays the marking on the display unit in a superimposed manner on the image data and the image data, and when it is determined that an abnormality has occurred in the object based on the changed area, displays abnormality information including details of the abnormality occurring in the object at the imaging time on the display unit in association with the imaging time. A program that makes it happen.

Images