Oil outflow monitoring system and oil outflow monitoring method

Abstract

To quickly and reliably detect oil spills while reducing the burden on human monitoring oil spills.SOLUTION: An oil spill monitoring system includes: a video acquisition unit that acquires a video of a scene; a storage unit that stores an oil detection model which is a machine learning model for detecting oil in the video; an oil detection unit that identifies a region in which oil appears in the video using an oil detection model; and a video display unit that displays region information which is information indicating the region in which the oil appears as augmented reality together with the video. In addition, the oil spill monitoring system acquires attribute information which is information on the oil in the video by an oil attribute determination model which is a machine learning model, and the image display unit displays the acquired attribute information and the information based on the attribute information as augmented reality together with the video.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an oil leakage monitoring system and an oil leakage monitoring method.

Background Art

[0002] Patent Document 1 describes an outflow accident monitoring device configured for the purpose of detecting, before oil flows out through a drainage ditch or the like into a river or the coast and pollutes the environment, when the oil has flowed out due to damage or malfunction of equipment handling oil. The outflow accident monitoring device is supported so as to maintain a predetermined distance above the floor surface of a predetermined area around the equipment (such as a concrete floor surface forming an oil outflow surface), and includes a light source unit that projects monitoring light rays onto the predetermined area and a light receiving element that receives the reflected light of the monitoring light rays from the predetermined area and converts it into an electrical signal, a signal processing unit that receives the output electrical signal of the light receiving element, detects a change in the signal, and issues a signal when a predetermined change occurs, and an alarm that issues an oil spill alarm in response to the output signal of the signal processing unit.

[0003] Further, Patent Document 2 describes an oil leakage detection device that detects oil leakage from equipment, piping, etc. installed in various plants. The oil leakage detection device detects oil leakage by utilizing the characteristics that oil emits fluorescence when irradiated with ultraviolet rays and absorbs light when irradiated with visible light. Specifically, the oil leakage detection device alternately irradiates a sample surface with ultraviolet rays and visible light, and detects oil leakage by taking the difference between the obtained images.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In facilities that handle oil, such as factories and power equipment, the monitoring of oil spills is carried out visually by on-site workers and the like. However, it is difficult to detect spills quickly, the human burden is large, and it is also difficult to ensure detection accuracy because the modes of oil spills vary depending on the properties of the oil and the on-site environment.

[0006] Also, in such facilities, the detection of oil spills is also carried out based on changes in the liquid level of float-type liquid level gauges installed in the equipment and facilities. However, in the case of using a liquid level gauge, an abnormality cannot be detected unless a certain amount of oil has spilled. Also, the source of the oil spill is not necessarily specified. For example, even if the spill accident monitoring device described in Patent Document 1 or the oil leak detection device described in Patent Document 2 is installed at a specific location in the facility, it is difficult to reliably detect the oil spill.

[0007] The present invention has been made in view of such a background, and an object thereof is to provide an oil spill monitoring system and an oil spill monitoring method that can reduce the human burden associated with monitoring oil spills and can quickly and reliably detect oil spills.

Means for Solving the Problems

[0008] One aspect of the present invention for achieving the above object is an information processing system (oil spill monitoring system) that supports the monitoring of oil spills, including a video acquisition unit that acquires a video of the site, a storage unit that stores an oil detection model, which is a machine learning model for detecting oil shown in the video, an oil detection unit that identifies a region in the video where oil is shown using the oil detection model, and a video display unit that displays, as augmented reality, region information, which is information indicating the region where the oil is shown, together with the video.

[0009] Thus, the oil leakage monitoring system of the present invention detects oil shown in the video taken at the site by an oil detection model which is a machine learning model, and displays region information, which is information indicating the region where the oil is shown, as augmented reality together with the video. Therefore, a user such as an operator monitoring the oil leakage can easily and quickly discover the oil leakage and can take appropriate measures at an early stage.

[0010] Another invention of the present invention is the above-described oil leakage monitoring system, wherein the oil detection model includes an oil attribute determination model which is a machine learning model that acquires attribute information which is information regarding the oil shown in the video based on the video. When oil is shown in the video, the oil leakage monitoring system further includes an oil attribute acquisition unit that acquires the attribute information of the oil by the oil attribute determination model. The video display unit displays the acquired attribute information or information acquired based on the attribute information as augmented reality together with the video.

[0011] Thus, the oil leakage monitoring system of the present invention acquires attribute information which is information regarding the oil shown in the video by an oil attribute determination model which is a machine learning model, and displays the acquired attribute information as augmented reality together with the above-described video. Therefore, the user can easily acquire information regarding the oil.

[0012] Another invention of the present invention is the above-described oil leakage monitoring system, wherein the storage unit further stores oil usage facility information which is information associating the attribute information with information indicating the facility using the oil, and the oil leakage monitoring system further includes a related information acquisition unit that acquires information indicating the facility corresponding to the attribute information from the oil usage facility information. The video display unit displays the acquired information indicating the facility as augmented reality together with the video.

[0013] Thus, the oil leakage monitoring system of the present invention acquires information indicating the facility corresponding to the attribute information, and displays the acquired information as augmented reality together with the above-described video. Therefore, the user can easily acquire information regarding the oil leakage source and can efficiently proceed with the task of identifying the oil leakage source.

[0014] One of the other inventions of the present invention is the above oil leakage monitoring system, wherein the oil detection model is generated by learning the feature amounts of the respective videos when oil is shown in the video and when it is not shown.

[0015] In this way, the oil leakage monitoring system of the present invention uses a machine learning model generated by learning the feature amounts (color, shading, pattern, etc.) of the respective videos when oil is shown and when it is not shown to detect oil. Therefore, oil can be detected with high accuracy based on the difference in the reflected light of oil peculiar to each property of oil that is difficult to capture with the human eye.

[0016] One of the other inventions of the present invention is the above oil leakage monitoring system, wherein the oil detection model is generated by learning the time-series changes of the respective videos when oil is shown in the video and when it is not shown as feature amounts.

[0017] In this way, the oil leakage monitoring system of the present invention uses a machine learning model generated by learning the time-series changes of the respective videos when oil is shown and when it is not shown (such as the time fluctuation of the oil film reflected in the video of the color, shading, pattern, etc. of the oil film) as feature amounts to detect oil. Therefore, oil can be detected with high accuracy based on the time-series change of the reflected light of oil peculiar to each property of oil that is difficult to capture with the human eye.

[0018] In addition, the problems disclosed in the present application and the solutions thereto are clarified by the section of the mode for carrying out the invention and the drawings.

Effect of the Invention

[0019] According to the present invention, the human burden involved in monitoring oil leakage can be reduced, and oil leakage can be detected quickly and surely.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments for carrying out the invention will be described. In the following description, the same or similar components may be denoted by common reference numerals and the description thereof may be omitted.

[0022] FIG. 1 shows the main functions of an information processing system (hereinafter referred to as "oil leakage monitoring system 1") described as an embodiment. The oil leakage monitoring system 1 supports the work of an operator who monitors oil leakage in a facility that handles oil. The above-mentioned facility is not necessarily limited, but for example, it is a power-related facility where power facilities using oil such as power plants and substations are installed, a factory or various plants where devices and machines using oil are installed, and the like.

[0023] The oil leakage monitoring system 1 analyzes the video taken at the site (such as a river, a roadside ditch, etc.) where oil leakage is to be monitored using a machine learning model, generates information indicating the presence or absence of oil leakage and the oil leakage situation, and provides the generated information to users such as on-site workers. The above-mentioned video is, for example, a video taken by a photographing device (camera) equipped on a portable information terminal such as a smartphone or a tablet carried by an on-site worker. The oil leakage monitoring system 1 generates the above-mentioned information by analyzing the above-mentioned video, for example, in real time, and provides the generated information to the user as augmented reality (AR (Augmented Reality)) together with the above-mentioned video. By using the above-mentioned information, the user can easily and quickly detect oil leakage, and can quickly and surely grasp the oil leakage situation.

[0024] When analyzing the above-mentioned video, the oil leakage monitoring system 1 performs image analysis using a machine learning model to generate information indicating the presence or absence of oil leakage and the oil leakage situation. Further, the oil leakage monitoring system 1 generates information indicating the presence or absence of oil leakage and the oil leakage situation by performing time-series analysis on the above-mentioned video taken as a moving image (a group of time-series image data), for example.

[0025] In addition, the oil leakage monitoring system 1 identifies the attributes of the oil (such as oil type, properties, degree of contamination, etc.) by analyzing the above-mentioned video, and provides information indicating the identified attributes (hereinafter referred to as "attribute information") and information based on the attribute information to the user as augmented reality. The user can efficiently perform specific operations for the source of oil leakage, for example, by using the provided information.

[0026] In addition, the oil leakage monitoring system 1 generates the above-mentioned information using a machine learning model learned with a sufficient amount of learning data taken in various environments for various oil aspects. Therefore, the oil leakage monitoring system 1 can confirm the presence or absence of oil leakage and grasp the oil leakage situation with high accuracy without being affected by the oil aspect and the on-site environment.

[0027] In addition, the oil leakage monitoring system 1 improves the accuracy of confirming the presence or absence of oil leakage and grasping the oil leakage situation by appropriately removing noise (e.g., driftwood, garbage, etc. included in the above-mentioned video) included in the video.

[0028] FIG. 2A is a diagram showing a state where a user 3 such as an operator uses the oil leakage monitoring system 1 to confirm the presence or absence of oil leakage (or grasp the oil leakage situation) from a facility 4 handling oil to the environment 5 such as a surrounding river or a roadside ditch. As shown in the figure, while the user 3 directs the direction of the imaging device 21 of the portable information terminal 2 he / she has in the direction in which he / she wants to confirm the oil situation, the user 3 is checking the video displayed on the display of the portable information terminal 2 (hereinafter referred to as the "display device 22").

[0029] FIGS. 2B and 2C are examples of the video displayed on the display device 22. FIG. 2B is an example of the video displayed on the display device 22 when there is no oil leakage, and FIG. 2C is an example of the video displayed on the display device 22 when there is oil leakage.

[0030] As shown in FIG. 2C, on the display device 22, information indicating the area where oil exists (in this example, the area of the water surface where oil exists) (hereinafter referred to as the "area information 26a") is displayed as augmented reality overlaid on the video 25 being captured by the imaging device 21. In addition, when the user 3 changes the imaging direction (the direction of the camera) of the imaging device 21, for example, the area information 26a also follows according to the currently captured video 25. Therefore, the user 3 can easily and quickly confirm the presence or absence of oil leakage and grasp the oil leakage situation over a wide range of the site by, for example, moving (scanning) the imaging direction of the imaging device 21.

[0031] Thus, according to the oil leakage monitoring system 1, the user can easily and surely confirm the oil floating on the water surface and grasp the oil leakage situation, which is not necessarily easy to confirm with the naked eye. Note that the illustrated area information 26a is an image in which the area where the oil on the water surface exists is highlighted (colored, shaded, hatched, etc.), but the form of the area information 26a is not necessarily limited.

[0032] Also, the illustrated display device 22 also functions as a touch panel. For example, when the user 3 touches a part of the area information 26a of the display device 22, the attribute information of the oil shown at that position and information based on the attribute information (for example, a list of candidates for the source of the outflow retrieved using the attribute information as a search key, etc. Hereinafter, these information are referred to as "related information 26b") are displayed on the display device 22, as shown in FIG. 2D. By using the related information 26b, the user 3 can efficiently perform, for example, the task of identifying the source of the oil leakage.

[0033] Returning to FIG. 1, as shown in the figure, the oil leakage monitoring system 1 has the functions of a storage unit 110, a video acquisition unit 120, an oil detection unit 125, an oil attribute determination unit 130, a related information acquisition unit 140, a display information generation unit 145, a video display unit 150, and a model learning unit 160.

[0034] Among the above functions, the storage unit 110 stores image data 111, an oil detection model 112, learning data 113, oil usage facility information 114, and display information 115.

[0035] The image data 111 is data acquired by the video acquisition unit 120, and is data obtained by encoding the video taken at the site in a predetermined format. The image data 111 includes information indicating the date and time when the image data 111 was taken. The image data 111 may be one frame cut out from a video. The image data 111 may be taken by a person using the imaging device 21 (camera) provided in the portable information terminal 2 such as a smartphone as described above, or may be automatically taken by a fixed-point camera or the like.

[0036] The oil detection model 112 is a group of machine learning models that generate information about oil based on the captured video (image data 111) of the site, and includes each machine learning model of an image analysis model 1121, a time series analysis model 1122, and an oil attribute determination model 1123.

[0037] Among these, the image analysis model 1121 is a machine learning model that determines whether oil is shown in the captured video (image data 111), and identifies the area where oil is shown when oil is shown. The image analysis model 1121 learns, for example, the feature amounts (color, shading, pattern, etc.) of each video when oil is shown and when oil is not shown. Further, the image analysis model 1121 may be, for example, a machine learning model (anomaly detection model) learned by an autoencoder (such as VAE (Variational Autoencoder)) with the image data 111 where no oil is shown as a steady state.

[0038] The time series analysis model 1122 is a machine learning model that determines whether oil is shown in the image data 111 and identifies the area where oil is shown when oil is shown, based on the time series change of the image data 111 for the captured video (group of time series image data 111) of the site. For example, the time series analysis model 1122 learns the time series changes (such as the temporal fluctuation of the oil film reflected in the video of the color, shading, pattern, etc. of the oil film) of each video when oil is shown and when oil is not shown as feature amounts. Further, the time series analysis model 1122 may be realized, for example, by a machine learning model (anomaly detection model) learned by an autoencoder (such as VAE) with the time series change of the image data 111 where no oil is shown as a steady state.

[0039] In addition, the determination of whether oil is shown in the image data 111 and the identification of the area where oil is shown when oil is shown in the image data 111 may be performed using either one of the image analysis model 1121 and the time series analysis model 1122, or may be performed using the results of both, for example, by a comprehensive evaluation value obtained by weighting each result.

[0040] The oil attribute determination model 1123 is a machine learning model that acquires the attribute information of the oil shown in the captured video (image data 111) for the image data 111. The oil attribute determination model 1123 is, for example, a machine learning model learned by learning data including image data with oil shown or image data without oil shown and correct answer information, prepared for each type of oil or each on-site environment.

[0041] Note that the types of each machine learning model (image analysis model 1121, time series analysis model 1122, oil attribute determination model 1123) shown above are not necessarily limited. For example, they are realized by deep learning (DNN (Deep Neural Network), CNN (Convolutional Neural Network), etc.) and various time series analysis methods. Also, the identification of the oil and the area where the oil is shown (object detection) included in the image data 111 by each machine learning model is performed using, for example, the "sliding window method", "HOG (Histograms of Oriented Gradients) feature amount", "region proposal method", "Faster R-CNN", "YOLO (You Only Look Once)", "SSD (Single Shot Detector)", "End-to-END learning", etc.). Further, the feature amounts used by each machine learning model may be set by a human system, or for example, feature amounts automatically extracted by an information processing device using known feature amount extraction methods (SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), FAST (Features from Accelerated Segment Test), BRIEF (Binary Robust Independent Elementary Features), ORB (Oriented FAST and Rotated BRIEF), etc.) may be used.

[0042] As shown in FIG. 1, the learning data 113 includes learning data 1131 for image analysis used for learning the image analysis model 1121, learning data 1132 for time series analysis used for learning the time series analysis model 1122, and learning data 1133 for oil attribute determination used for learning the oil attribute determination model 1123.

[0043] The learning data 1131 for image analysis is data in which, for example, image data with oil or without oil prepared for each type of oil and the site environment, and correct answer information (information indicating whether oil is present or the area where oil is present) are associated with each other. In addition, in order to improve the accuracy of the image analysis model 1121, the learning of the image analysis model 1121 is performed using a sufficient amount of learning data 1131 for image analysis based on image data taken in various environments (environments with different locations, seasons, times, etc., environments including noise such as garbage floating on the water surface, etc.).

[0044] The learning data 1132 for time series analysis is data in which, for example, a time series image data group with oil and a time series image data group without oil prepared for each type of oil and the site environment, and correct answer information (information indicating whether oil is present or the area where oil is present) are associated with each other. In addition, in order to improve the accuracy of the time series analysis model 1122, the learning of the time series analysis model 1122 is performed using a sufficient amount of learning data 1132 for time series analysis based on a time series image data group taken in various environments (locations, seasons, times, noise such as garbage floating on the water surface).

[0045] The learning data 1133 for oil attribute determination is learning data that includes, for example, image data with oil or without oil prepared for each type of oil and the site environment, and correct answer information (attribute information). In addition, in order to improve the accuracy of the oil attribute determination model 1123, the learning of the oil attribute determination model 1123 is performed using a sufficient amount of learning data 1133 for oil attribute determination based on image data taken in various environments (locations, seasons, times, noise such as garbage floating on the water surface).

[0046] The oil-using facility information 114 shown in FIG. 1 includes information associating the identifier of each facility provided in Facility 4 (hereinafter referred to as "facility ID") with the attribute information. The oil-using facility information 114 is managed by a user such as the administrator of Facility 4 or an operator who monitors the oil spill.

[0047] FIG. 3 shows an example of the oil-using facility information 114. The exemplified oil-using facility information 114 is composed of one or more entries (records) having items such as facility ID 1141, facility type 1142, installation location 1143, and attribute information 1144. One entry of the oil-using facility information 114 corresponds to one facility provided in Facility 4.

[0048] Among the above items, the facility ID of the facility is set in the facility ID 1141. Information indicating the type of the facility (form, model number, etc.) is set in the facility type 1142. Information indicating the location where the facility is provided in Facility 4 is set in the installation location 1143. Attribute information (such as oil type) associated with the facility is set in the attribute information 1144.

[0049] The display information 115 shown in FIG. 1 is information generated by the display information generation unit 145 and is the above-mentioned area information 26a or related information 26b.

[0050] FIG. 4 shows an example of the display information 115. The exemplified display information 115 is composed of one or more entries (records) having items such as display information ID 1151, format 1152, display position (area) 1153, and display content 1154. One entry of the display information 115 corresponds to one of the above-mentioned area information 26a or related information 26b displayed on the display device 22.

[0051] Among the above items, the display information ID 1151 is set with a display information ID that is an identifier (object ID) of the area information 26a or the related information 26b. The format 1152 is set with information (such as an image, text, etc.) indicating the format of the area information 26a or the related information 26b. The display position (area) 1153 is set with information indicating the position or area on the screen of the display device 22 where the area information 26a or the related information 26b is displayed. The display content 1154 is set with the method and content of the display of the area information 26a or the related information 26b on the display device 22.

[0052] Returning to FIG. 1, the video acquisition unit 120 acquires the image data of the video (video or still image) acquired by the portable information terminal 2 or the like held by the user. The image data acquired by the video acquisition unit 120 is stored by the storage unit 110 as the image data 111.

[0053] The oil detection unit 125 includes the functions of the image analysis unit 1251 and the time-series image analysis unit 1252. Among these, the image analysis unit 1251 uses the image analysis model 1121 to determine whether oil is shown in the image data 111 acquired by the video acquisition unit 120, and to identify the area where oil is shown when oil is shown, etc. The time-series image analysis unit 1252 uses the time-series analysis model 1122 to determine whether oil is shown in the group of time-series image data 111 acquired by the video acquisition unit 120, and to identify the area where oil is shown when oil is shown, etc.

[0054] The oil attribute determination unit 130 uses the oil attribute determination model 1123 to acquire the attribute information of the oil shown in the image data 111 for the image data 111 acquired by the video acquisition unit 120.

[0055] The related information acquisition unit 140 acquires information related to the attribute information based on the attribute information acquired by the oil attribute determination unit 130.

[0056] The display information generation unit 145 generates the above-described area information 26a for the area in which the oil is captured by the image analysis unit 1251 or the time-series image analysis unit 1252. Further, the display information generation unit 145 generates the above-described related information 26b based on the information acquired by the related information acquisition unit 140.

[0057] The video display unit 150 displays the video captured by the imaging device 21 on the display device 22. Further, the area information 26a and the related information 26b generated by the display information generation unit 145 are displayed in a predetermined manner as augmented reality on the video 25 being captured by the imaging device 21. Note that the video display unit 150 may have a function of reproducing a video based on the image data 111 acquired in the past while displaying the area information 26a and the related information 26b generated for the image data 111 as augmented reality. Thereby, the user can perform post-analysis.

[0058] The model learning unit 160 performs learning of the oil detection model 112 using the learning data 113, that is, learning of the image analysis model 1121 using the learning data 1131 for image analysis, learning of the time-series analysis model 1122 using the time-series analysis learning data 1132, and learning of the oil attribute determination model 1123 using the oil attribute determination learning data 1133. Note that the learning data 113 may be divided into data used for learning and data used for verification, and verification may be performed on the oil detection model 112 after learning using the data used for learning with the data used for verification.

[0059] FIG. 5 is an example of the hardware of an information processing apparatus (computer) used to implement the oil leakage monitoring system 1. The illustrated information processing apparatus 10 includes a processor 11, a main storage device 12, an auxiliary storage device 13, an input device 14, an output device 15, a communication device 16, a timing device 17, and an imaging device 18. The oil leakage monitoring system 1 may be configured by a plurality of information processing apparatuses 10 connected communicably.

[0060] The information processing apparatus 10 may be realized using virtual information processing resources provided using virtualization technologies, process space separation technologies, etc., such as a virtual server provided by a cloud system, for example, in whole or in part. All or part of the functions provided by the information processing apparatus 10 may be realized by services provided by a cloud system via an API (Application Programming Interface), for example. All or part of the functions provided by the information processing apparatus 10 may be realized using, for example, SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service), etc.

[0061] Further, all or part of the information processing apparatus 10 may be a portable information terminal such as a smartphone, a tablet, a notebook personal computer, etc., used by a user at a site where oil leakage monitoring is performed, for example. For example, the above-described portable information terminal 2 is an example of the information processing apparatus 10 and constitutes all or part of the oil leakage monitoring system 1.

[0062] In the figure, the processor 11 is configured using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), an AI (Artificial Intelligence) chip, etc.

[0063] The main memory device 12 is a device that stores programs and data, and is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a non-volatile memory (NVRAM (Non Volatile RAM)), etc.

[0064] The auxiliary storage device 13 is, for example, an SSD (Solid State Drive), a hard disk drive, an optical storage device (such as a CD (Compact Disc), a DVD (Digital Versatile Disc)), a storage system, an IC card, a reading / writing device for recording media such as an SD card or an optical recording medium, a storage area of a cloud server, etc. Programs and data can be read into the auxiliary storage device 13 via a reading device for a recording medium or a communication device 16. Programs and data stored (memorized) in the auxiliary storage device 13 are read into the main storage device 12 at any time.

[0065] The input device 14 is an interface that receives external inputs, and is, for example, a touch panel, a keyboard, a mouse, a card reader, a pen-input type tablet, a voice input device, etc.

[0066] The output device 15 is an interface that outputs various kinds of information such as the progress of processing and the results of processing. The output device 15 is, for example, a display device (such as a liquid crystal monitor, an LCD (Liquid Crystal Display), a graphic card, etc.) that visualizes the above various kinds of information, a device (such as a voice output device (speaker, etc.)) that vocalizes the above various kinds of information, a device (such as a printing device, etc.) that converts the above various kinds of information into characters. Note that, for example, the information processing device 10 may be configured to input and output information to and from other devices via the communication device 16.

[0067] The input device 14 and the output device 15 constitute a user interface that receives information from and presents information to the user.

[0068] The communication device 16 is a device that realizes communication with other devices. The communication device 16 is a wired or wireless communication interface that realizes communication with other devices via a communication network such as the Internet, and is, for example, a NIC (Network Interface Card), a wireless communication module, a USB module, etc.

[0069] The timing device 17 is a device that generates a highly accurate current date and time, and is configured using an RTC (Real Time Clock) or the like. The timing device 17 may calibrate the date and time based on the date and time acquired by the communication device 16 using NTP (Network Time Protocol).

[0070] The imaging device 18 captures a still image or a moving image, and generates image data 111 based on the captured video. The imaging device 18 is, for example, a camera provided in a smartphone used by a user on site.

[0071] For the information processing device 10, for example, an operating system, a file system, a DBMS (DataBase Management System) (relational database, NoSQL, etc.), a KVS (Key-Value Store), etc. may be introduced.

[0072] Each of the above-described functions provided in the oil leakage monitoring system 1 is realized by the processor 11 of the information processing device 10 reading and executing a program stored in the main storage device 12, or by the hardware (FPGA, ASIC, AI chip, etc.) of the information processing device 10. The information processing device 10 stores the above-described various types of information (data) as, for example, a table in a database or a file managed by a file system.

[0073] Subsequently, the main processes performed by the oil leakage monitoring system 1 will be described.

[0074] FIG. 6 is a flowchart for explaining a process (hereinafter referred to as "learning process S600") performed when the oil leakage monitoring system 1 learns the oil detection model 112 (image analysis model 1121, time series analysis model 1122, oil attribute determination model 1123). As shown in the figure, the model learning unit 160 individually learns the image analysis model 1121, the time series analysis model 1122, and the oil attribute determination model 1123 using the learning data 1131 for image analysis, the learning data 1132 for time series analysis, and the learning data 1133 for oil attribute determination, respectively (S611). Note that since learning the oil detection model 112 usually requires a large amount of information processing resources, the learning of the oil detection model 112 may be performed using, for example, an API (Application Programming Interface) for machine learning provided by a cloud system.

[0075] FIG. 7 is a flowchart for explaining a process (hereinafter referred to as "oil leakage monitoring process S700") performed by the oil leakage monitoring system 1 when a user monitors oil leakage while using the oil leakage monitoring system 1. The user performs the monitoring operation of oil leakage at the site while operating a portable information terminal 2 such as a smartphone, which is a component of the oil leakage monitoring system 1. The oil leakage monitoring process S700 is performed, for example, when the application software of the portable information terminal 2 cooperates with the cloud. Hereinafter, the oil leakage monitoring process S700 will be described with reference to the figure.

[0076] First, the video acquisition unit 120 of the oil leakage monitoring system 1 acquires a video (video or still image) of a predetermined area at the site where oil leakage is to be monitored (S711). The above video may be image data cut out from video data or image data captured as a still image. The video (image data) acquired by the video acquisition unit 120 is stored by the storage unit 110 of the oil leakage monitoring system 1 as image data 111.

[0077] Subsequently, the oil detection unit 125 (image analysis unit 1251, time-series image analysis unit 1252) of the oil leakage monitoring system 1 detects oil (the area where oil appears) in the image data 111 (S712). If oil cannot be detected (S713: NO), the process returns to S711. On the other hand, if oil is detected (S713: YES), the oil attribute determination unit 130 of the oil leakage monitoring system 1 acquires the attribute information of the detected oil (S714).

[0078] Subsequently, the related information acquisition unit 140 of the oil leakage monitoring system 1 acquires information related to the acquired attribute information from the oil-using facility information 114 (S715). For example, the related information acquisition unit 140 acquires information from the oil-using facility information 114 using the attribute information as a search key.

[0079] Subsequently, the display information generation unit 145 of the oil leakage monitoring system 1 generates the above-mentioned area information 26a for the area where the oil identified by the image analysis unit 1251 or the time-series image analysis unit 1252 appears. Further, the display information generation unit 145 generates the above-mentioned related information 26b based on the information acquired by the related information acquisition unit 140 (S716).

[0080] Subsequently, the video display unit 150 of the oil leakage monitoring system 1 displays the video captured by the imaging device 21 on the display device 22, and expands and displays the area information 26a and the area information 26a generated by the display information generation unit 145 as augmented reality on the video 25 being captured by the imaging device 21 (S717).

[0081] Subsequently, the oil leakage monitoring system 1 determines whether an end operation has been performed (S718). If the user has not performed an end operation (S718: NO), the process returns to S711. If the user has performed an end operation (S718: YES), the oil leakage monitoring system 1 ends the oil leakage monitoring process S700.

[0082] As described above, the oil leakage monitoring system 1 detects oil shown in the video captured at the site by a machine learning model (image analysis model 1121, time series analysis model 1122), and displays area information, which is information indicating the area where the oil is shown, as augmented reality together with the video. Therefore, the user can quickly and surely discover the oil leakage and take appropriate measures early.

[0083] In addition, the oil leakage monitoring system 1 acquires attribute information, which is information about the oil shown in the video captured at the site, by an oil attribute determination model, and displays the acquired attribute information as augmented reality together with the above video. Therefore, the user can easily acquire information about the oil.

[0084] In addition, the oil leakage monitoring system 1 acquires the equipment ID corresponding to the attribute information, and displays the acquired equipment ID as augmented reality together with the above video. Therefore, the user can efficiently perform specific work on the oil leakage source, for example, using the acquired equipment ID.

[0085] In addition, the oil leakage monitoring system 1 detects oil using the image analysis model 1121, which is a machine learning model generated by learning the feature amounts (color, shading, pattern, etc.) of each video when the oil is shown and when it is not shown. Therefore, the oil can be detected with high accuracy based on the difference in the reflected light of the oil unique to each property of the oil that is difficult to capture with the human eye.

[0086] In addition, the oil leakage monitoring system 1 detects oil using the time series analysis model 1122, which is a machine learning model generated by learning the time series changes (such as the temporal fluctuation of the oil film reflected in the video of the color, shading, pattern, etc. of the oil film) of each video when the oil is shown and when it is not shown as feature amounts. Therefore, the oil can be detected with high accuracy based on the time series change of the reflected light of the oil unique to each property of the oil that is difficult to capture with the human eye.

[0087] As described above in detail for the embodiments of the present invention, the above description is for facilitating the understanding of the present invention and does not limit the present invention. The present invention can be changed and improved without departing from its gist, and it goes without saying that equivalents of the present invention are included therein. For example, the above embodiments have been described in detail for easy understanding of the present invention and are not necessarily limited to those having all the configurations described. Also, for a part of the configuration of the above embodiments, addition, deletion, and replacement with other configurations are possible.

[0088] For example, for the image data 111 of the video obtained by shooting a wide area with a shooting device mounted on a drone or the like, the oil detection unit 125 analyzes the image data 111, and the video display unit 150 provides the information obtained thereby, so that the user can easily and quickly grasp the situation of the oil diffused over a wide area.

Explanation of Reference Numerals

[0089] 1 Oil spill monitoring system, 2 Portable information terminal, 21 Shooting device, 22 Display device, 25 Video, 26a Region information, 26b Related information, 3 User, 4 Facility, 5 Environment, 10 Information processing device, 110 Storage unit, 111 Image data, 112 Oil detection model, 1121 Image analysis model, 1122 Time series analysis model, 1123 Oil attribute determination model, 113 Learning data, 1131 Learning data for image analysis, 1132 Learning data for time series analysis, 1133 Learning data for oil attribute determination, 114 Oil-using facility information, 115 Display information, 120 Video acquisition unit, 125 Oil detection unit, 1251 Image analysis unit, 1252 Time series image analysis unit, 130 Oil attribute determination unit, 140 Related information acquisition unit, 145 Display information generation unit, 150 Video display unit, 160 Model learning unit

Claims

1. An information processing system for assisting in monitoring oil leakage, comprising: a video acquisition unit that acquires a video of a site; a storage unit that stores an oil detection model, which is a machine learning model for detecting oil shown in the video; an oil detection unit that identifies, using the oil detection model, an area in the video where oil is shown; a video display unit that displays, as augmented reality together with the video, area information that is information indicating the area where the oil is shown; An oil leakage monitoring system.

2. The oil leakage monitoring system according to claim 1, wherein: the oil detection model includes an oil attribute determination model, which is a machine learning model for acquiring, based on the video, attribute information that is information regarding the oil shown in the video; when oil is shown in the video, the system further includes an oil attribute acquisition unit that acquires the attribute information of the oil using the oil attribute determination model; the video display unit displays, as augmented reality together with the video, the acquired attribute information or information acquired based on the attribute information. An oil leakage monitoring system.

3. The oil leakage monitoring system according to claim 2, wherein: the storage unit further stores oil usage equipment information, which is information associating the attribute information with information indicating equipment using the oil; the system further includes a related information acquisition unit that acquires, from the oil usage equipment information, information indicating the equipment corresponding to the attribute information; the video display unit displays, as augmented reality together with the video, the acquired information indicating the equipment. An oil leakage monitoring system.

4. A method for assisting in monitoring oil leakage, comprising the steps of: an information processing device acquiring a video of a site; storing an oil detection model, which is a machine learning model for detecting oil shown in the video; identifying, using the oil detection model, an area in the video where oil is shown; displaying, as augmented reality together with the video, area information that is information indicating the area where the oil is shown. An oil leakage monitoring method.

5. The oil leakage monitoring method according to claim 4, wherein: the oil detection model includes an oil attribute determination model, which is a machine learning model for acquiring, based on the video, attribute information that is information regarding the oil shown in the video; the information processing device when oil is shown in the video, acquiring the attribute information of the oil using the oil attribute determination model; displaying, as augmented reality, the obtained attribute information or information obtained based on the attribute information together with the video; An oil leakage monitoring method that further performs the following.

6. The oil leakage monitoring method according to claim 5, wherein the information processing device further stores oil usage facility information, which is information associating the attribute information with information indicating a facility using oil; obtains information indicating the facility corresponding to the attribute information from the oil usage facility information; and displays, as augmented reality, the obtained information indicating the facility together with the video. An oil leakage monitoring method that further performs the following.

Images