Device, method, and program for processing acoustic data detected in facility

Abstract

To provide a device that processes acoustic data detected in a facility to identify an apparatus in which an abnormality occurs.SOLUTION: The device includes: an acoustic data acquisition section that acquires acoustic data detected by an acoustic sensor of a robot capable of moving in a facility; a sound abnormality detection section that detects an abnormality degree of sound using the acoustic data; a LiDAR data acquisition section that acquires LiDAR data detected by a LiDAR sensor of the robot; an apparatus detection section that detects at least one apparatus using the LiDAR data; and an abnormal apparatus identification section that identifies an apparatus in which an abnormality occurs among the at least one apparatus on the basis of a change in the abnormality degree accompanying movement of the robot.SELECTED DRAWING: Figure 22

Description

Technical Field

[0001] The present invention relates to an apparatus, a method, and a program for processing acoustic data detected within a facility.

Background Art

[0002] Patent Document 1 describes that "the inspection support system 1 shown in FIG. 1 enables an inspector 4 at a monitoring location 3 remote from an inspection site 2 to inspect the equipment at the inspection site 2" (paragraph 0010), that "the sound information from the unmanned mobile body 10 includes the sound information acquired by each of a plurality of microphones 131 to 13m, and the sound determination unit 72 can determine the sound source direction, the sound source position, and the sound volume based on this sound information" (paragraph 0037), and that "the graphic addition unit 73 adds at least one of a graphic representing the position of the sound source and a graphic representing the sound volume to the captured image acquired from the unmanned mobile body 10 based on the determination result by the sound determination unit 72" (paragraph 0038).

[0003] Patent Document 2 describes that "a field patrol robot automatically patrols the field and acquires real-time field video and field audio, etc." (paragraph 0005), and that "the operating sound of field equipment is measured, and the sound band distribution of the operating sound of the equipment is compared with the sound band distribution of the operating sound of the corresponding equipment during normal operation corresponding to the plant operating state registered in advance in a database, and an abnormality of the equipment is automatically notified based on the volume difference of each sound band" (paragraph 0010). [Prior Art Documents] [Patent Documents] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2020-149349 [Patent Document 2] Japanese Unexamined Patent Application Publication No. 2000-039914

Summary of the Invention

Means for Solving the Problems

[0004] In a first aspect of the present invention, there is provided an apparatus comprising: an acoustic data acquisition unit that acquires acoustic data detected by an acoustic sensor of a robot capable of moving within a facility; a sound abnormality detection unit that detects the degree of sound abnormality using the acoustic data; a LiDAR data acquisition unit that acquires LiDAR data detected by a LiDAR sensor of the robot; a device detection unit that detects at least one device using the LiDAR data; and an abnormal device identification unit that identifies a device in which an abnormality has occurred among the at least one device based on a change in the degree of abnormality accompanying the movement of the robot.

[0005] In the above apparatus, the abnormal device identification unit may identify, as a device in which an abnormality has occurred, a device located on the advancing direction side among the at least one device in response to an increase in the degree of abnormality as the robot moves in the advancing direction.

[0006] Any of the above apparatuses may include a robot command unit that instructs the robot of an advancing direction based on a change in the degree of abnormality accompanying the movement of the robot.

[0007] In any of the above apparatuses, the robot command unit may instruct the robot to maintain the advancing direction of the robot in response to an increase in the degree of abnormality as the robot moves, and to change the advancing direction of the robot in response to the degree of abnormality ceasing to increase or decreasing as the robot moves.

[0008] In any of the above apparatuses, the sound abnormality detection unit may detect the degree of abnormality based on a result of comparing the acoustic data with normal acoustic data associated with the position where the acoustic data was detected.

[0009] In any of the above apparatuses, the device detection unit may acquire identification information of a device that should exist at the position of an object confirmed by the LiDAR data using a device database that records device data including position and identification information for each of a plurality of devices in the facility.

[0010] Any of the above devices may include a device control unit that, in response to identifying an abnormality in one device, performs control for dealing with the abnormality in the one device on at least one of the one device or other devices in the facility.

[0011] In a second aspect of the present invention, there is provided a method including: acquiring acoustic data detected by an acoustic sensor of a robot capable of moving within a facility; detecting an abnormality level of a sound using the acoustic data; acquiring LiDAR data detected by a LiDAR sensor of the robot; detecting at least one device using the LiDAR data; and identifying a device in which an abnormality has occurred among the at least one device based on a change in the abnormality level accompanying the movement of the robot.

[0012] In a third aspect of the present invention, there is provided a program that is executed by a computer and causes the computer to function as an acoustic data acquisition unit that acquires acoustic data detected by an acoustic sensor of a robot capable of moving within a facility, a sound abnormality detection unit that detects an abnormality level of a sound using the acoustic data, a LiDAR data acquisition unit that acquires LiDAR data detected by a LiDAR sensor of the robot, a device detection unit that detects at least one device using the LiDAR data, and an abnormal device identification unit that identifies a device in which an abnormality has occurred among the at least one device based on a change in the abnormality level accompanying the movement of the robot.

[0013] Note that the above summary of the invention does not list all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Embodiments for Carrying Out the Invention

[0015] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0016] FIG. 1 shows a schematic configuration of the facility 1 according to the present embodiment. The facility 1 may be an entire or partial segment such as a factory or a plant, and a plurality of devices 10 are arranged therein. Such a factory or plant may be, for example, a factory for producing various industrial products, an industrial plant such as a chemical or metal plant, a plant for managing and controlling a wellhead and its surroundings such as a gas field or an oil field, a plant for managing and controlling power generation such as hydraulic, thermal, or nuclear power, a plant for managing and controlling environmental power generation such as solar or wind power, a plant for managing and controlling water supply and sewerage or a dam, etc. Further, the facility 1 may be an entire or partial part of a building or a transportation vehicle that includes a plurality of devices 10 to be controlled and monitored. The facility 1 includes a plurality of devices 10, a control system 20, one or more robots 30, a monitoring device 40, a collected information database 50, a device database 60, and a robot database 70.

[0017] A plurality of devices 10 are provided at various locations within the facility 1. Each of the plurality of devices 10 may be installed either indoors or outdoors within the area of the facility 1. At least some of the plurality of devices 10 may be a process device, a power generation device, or any other device (or equipment) that is under the control of the control system 20, or may be a part of such a device. At least some of the devices 10 may be provided with field devices that operate under the control of the control system 20, other devices, or an operator. Also, at least some of the devices 10 may be the field devices themselves.

[0018] Such field devices may be, for example, sensor devices such as pressure gauges, flow meters, temperature sensors, valve devices such as flow control valves and on-off valves, actuator devices such as fans and motors, imaging devices such as cameras or videos that capture objects such as the status of plants or equipment, acoustic devices such as microphones or speakers that collect abnormal sounds or emit alarm sounds in plants or equipment, position detection devices that output the position information of the devices possessed by the facility 1, or other devices. Also, the other devices 10 among the plurality of devices 10 may be structures such as pipes, storage tanks, supports, partitions, or others that are not under the control of the control system 20.

[0019] The control system 20 is connected to at least one of the plurality of devices 10 that is a control target. The control system 20 may be, for example, a distributed control system (DCS). The control system 20 controls each device 10 according to the state of each device 10 measured by sensors or the like provided in each device 10 that is a control target.

[0020] Each of the one or more robots 30 is used to monitor at least some of the plurality of devices 10 within the facility 1. Each robot 30 can move within the facility 1 to collect sounds and capture images and so on.

[0021] The monitoring device 40 is communicably connected to each of one or more robots 30. The monitoring device 40 may be connected to each robot 30 via a wireless network such as a mobile phone network, a wireless WAN, a wireless LAN, or Bluetooth (registered trademark), or may be connected to each robot 30 via a wired network such as a wired Ethernet (registered trademark). The monitoring device 40 according to the present embodiment is installed inside the facility 1. Alternatively, the monitoring device 40 may be provided outside the facility 1 and remotely monitor each device 10 inside the facility 1. For example, the monitoring device 40 may be installed inside another facility, or may be realized by a cloud server on the Internet, for example.

[0022] The monitoring device 40 receives the collected data such as acoustic data and image data collected by each robot 30 inside the facility 1 from each robot 30, and monitors the state of each device 10 using the collected data. When the monitoring device 40 identifies an abnormality in any of the devices 10, it may instruct the control system 20 to perform control to address the abnormality.

[0023] The monitoring device 40 uses a collected information database 50, a device database 60, and a robot database 70 to monitor each device 10 inside the facility 1 using one or more robots 30. The collected information database 50 records the collected data collected from each robot 30. The device database 60 records device data about each device 10 inside the facility 1. The robot database 70 stores data related to each robot 30. These databases may be storage devices such as a hard disk connected to the monitoring device 40 by wire or wirelessly, or cloud storage on the Internet, or may be temporarily stored in the memory of the monitoring device 40 or the like.

[0024] FIG. 2 shows the configuration of the control system 20 according to the present embodiment together with the display device 240 and the input device 250. The display device 240 displays a display screen output by the control system 20. The input device 250 inputs instructions for the control system 20 from a user such as an operator, a worker, or a maintenance staff of the facility 1 and supplies them to the control system 20. The display device 240 and the input device 250 may be provided in a monitoring console or a user terminal connected to the control system 20.

[0025] The control system 20 may be a computer such as a workstation, a server computer, a general-purpose computer, or other computers, or may be a computer system to which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Further, the control system 20 may be implemented by one or more executable virtual computer environments in the computer. Instead of this, the control system 20 may be a dedicated computer designed for controlling each device 10, or may be dedicated hardware realized by a dedicated circuit. In the present embodiment, the control system 20 is installed in the facility 1, but the control system 20 may be provided outside the facility 1 by a cloud computing system or the like on the Internet.

[0026] The control system 20 includes a state acquisition unit 210, a state determination unit 220, a display processing unit 230, an instruction input unit 260, and a control unit 270. The state acquisition unit 210 acquires device state data indicating the state of each device 10 from each device 10, such as internal state values of each device 10 and measurement values measured by sensors provided in each device 10. The state acquisition unit 210 may receive the device state data of each device 10 by using communication according to a communication protocol such as HART (registered trademark), BRAIN, Foundation Fieldbus (registered trademark), ISA100.11a, etc. The state acquisition unit 210 may receive the device state data of the device 10 detected by the monitoring device 40 by using the collection data collected by the robot 30 from the monitoring device 40.

[0027] The state determination unit 220 is connected to the state acquisition unit 210. The state determination unit 220 determines whether each device 10 is normal or abnormal using the device state data acquired by the state acquisition unit 210. The state determination unit 220 may calculate a health index indicating the soundness of the operation in the entire facility 1 or a specific range within the facility 1 from at least one of the internal state values or measured values of two or more devices 10, and determine whether the operation is normal. The state determination unit 220 may determine the normality or abnormality of at least some of the devices 10 by receiving from the monitoring device 40 the determination result of the normality or abnormality of the device 10 determined by the monitoring device 40 using the collection data collected by the robot 30.

[0028] The display processing unit 230 is connected to the state acquisition unit 210 and the state determination unit 220. The display processing unit 230 performs display control to cause the display device 240 to display a display screen including device state data such as the internal state value and measured value of each device 10, a soundness index for the entire facility 1 or a specific range within the facility 1, and the occurrence status of abnormalities of each device 10.

[0029] The instruction input unit 260 receives an instruction from a user to the control system 20 from the input device 250. Such an instruction may be, for example, an instruction from a user who has seen the display screen displayed on the display device 240 to change the operation of at least one device 10. The instruction input unit 260 may be connected to the monitoring device 40 and may be able to communicate with the monitoring device 40 by wire or wirelessly. The instruction input unit 260 may receive from the monitoring device 40 an instruction to perform control for dealing with an abnormality in response to the detection of an abnormality in at least one device 10 by the monitoring device 40.

[0030] The control unit 270 is connected to the state acquisition unit 210 and the instruction input unit 260. The control unit 270 controls each device 10 according to the device state data acquired by the state acquisition unit 210. The control unit 270 may control each device 10 according to an instruction in response to receiving an instruction from a user or an instruction from the monitoring device 40.

[0031] Figure 3 shows the processing flow of the control system 20 according to the present embodiment. In step 300 (S300), the state acquisition unit 210 acquires device state data indicating the state of each device 10 from each device 10. The state acquisition unit 210 may receive device data indicating the state of at least one device 10 from the monitoring device 40.

[0032] In S310, the state determination unit 220 determines whether each device 10 is normal or abnormal. The state determination unit 220 may determine whether the device 10 is normal (not abnormal) according to whether at least one of the internal state value or the measured value of the device 10 is within a predetermined normal range corresponding to the value. Further, the state determination unit 220 calculates a health index indicating the soundness of the operation in the entire facility 1 or a specific range within the facility 1 from at least one of the internal state values or the measured values of two or more devices 10 using a pre-defined calculation formula or the like, and determines whether the operation of the entire facility 1 or a specific range within the facility 1 is normal according to whether the value of the health index is within the normal range.

[0033] In S320, the display processing unit 230 performs display control to cause the display device 240 to display a display screen including the internal state value and the measured value of each device 10, the health index for the entire facility 1 or a specific range within the facility 1, and the abnormal occurrence status of each device 10. The display processing unit 230 may generate a video output of the display screen and supply it to the display device 240, or may generate html or a script for generating the display screen and transmit it to the display device 240.

[0034] In S330, the instruction input unit 260 receives an instruction from the user for the control system 20 from the input device 250. The instruction input unit 260 may receive an instruction for performing control for dealing with an abnormality transmitted by the monitoring device 40 that has detected an abnormality in at least one device 10.

[0035] In S340, the control unit 270 controls each device 10 according to a predetermined control algorithm or control model or the like using the device state data acquired by the state acquisition unit 210. The control unit 270 may calculate a control value for each device 10 by PI control, PID control, or the like. The control unit 270 may calculate a control value for each device 10 according to the device state data acquired by the state acquisition unit 210 using various machine learning models or the like. The control unit 270 may change the control content for each device 10 in response to an instruction from the instruction input unit 260.

[0036] The control system 20 repeats the processes from S300 to S340. Thereby, the control system 20 can adaptively control each device 10 according to the state of each device 10.

[0037] FIG. 4 shows the configuration of the robot 30 according to the present embodiment. The robot 30 includes one or more sensors 400, one or more sensors 410, one or more actuators 420, a state acquisition unit 430, a communication unit 440, and a control unit 450.

[0038] Each of the one or more sensors 400 detects or measures the state within the facility 1. In the example of this figure, the sensor 400a is an acoustic sensor that detects the sound within the facility 1 and outputs it as acoustic data. The sensor 400b is an image sensor that captures an image within the facility 1 and outputs it as image data. The sensor 400c is a LiDAR sensor. The sensor 400c measures at least one of the shape of each object within the irradiation range of light or laser or the distance to each irradiation point by irradiating light or laser to the outside and detecting the reflected light, and outputs the measurement result as LiDAR data. The robot 30 may not include at least one of the sensors 400a to 400c. The robot 30 may include at least one of a temperature sensor, a humidity sensor, a gas sensor, or other various sensors.

[0039] Each of the one or more sensors 410 detects or measures the state of the robot 30. In the example of this figure, the sensor 410a is a position sensor that detects the position of the robot 30 and outputs it as position data. The sensor 410a may be a GPS, specifically, a GPS receiver that receives signals from GPS (Global Positioning System) satellites to identify the position of the robot 30. Alternatively, the sensor 410a may be a sensor for identifying the position of the robot 30 using any position detection system capable of detecting the position of the robot 30 within the facility 1.

[0040] The sensor 410b is an azimuth sensor such as a geomagnetic sensor, for example. The sensor 410b measures the azimuth of the robot 30 or the azimuth of the detection direction of each sensor 400. The sensor 410c is an angle sensor or an elevation angle sensor for measuring the detection direction of each sensor 400. The sensors 410b and 410c output direction data indicating at least one of the azimuth of the robot 30 or the detection direction of each sensor 400 to the state acquisition unit 430. Note that the robot 30 may not include at least one of the sensors 410a to 410c. The robot 30 may include at least one of a speed sensor, an acceleration sensor, or other various sensors.

[0041] Each of the one or more actuators 420 is a motor or the like for driving each part of the robot 30. The robot 30 may include various actuators 420 such as an actuator 420 for movement by traveling or flying, an actuator 420 for changing the orientation of the entire robot 30 or a part thereof, and an actuator 420 for operating equipment such as an arm.

[0042] The state acquisition unit 430 is connected to one or more sensors 400 and one or more sensors 410. The state acquisition unit 430 acquires the state inside the facility 1 measured by the one or more sensors 400 and the state of the robot 30 measured by the one or more sensors 410. The state acquisition unit 430 may acquire the state inside the facility 1 by receiving various measurement data such as acoustic data, image data, and LiDAR data from the one or more sensors 400. The state acquisition unit 430 may acquire the state of the robot 30 by receiving various measurement data such as position data and direction data from the one or more sensors 410.

[0043] The communication unit 440 is connected to the state acquisition unit 430. The communication unit 440 communicates wirelessly or by wire with the monitoring device 40. The communication unit 440 transmits the various measurement data acquired by the state acquisition unit 430 to the monitoring device 40. Also, the communication unit 440 receives various instructions for the robot 30 from the monitoring device 40.

[0044] The control unit 450 controls each actuator 420 according to the various measurement data acquired by the state acquisition unit 430 and the instructions from the monitoring device 40. Thereby, the robot 30 can perform operations corresponding to the state inside the facility 1, the state of the robot 30, and the instructions from the monitoring device 40.

[0045] Note that some robots 30 may be fixedly installed inside the facility 1, and such robots 30 do not need to be provided with actuators for movement. Also, inside the facility 1, monitoring devices (monitoring cameras, monitoring microphones, etc.) that are equipped with one or more sensors 400 but are not classified as robots may be provided. Such monitoring devices will have some functions and configurations of the robot 30. Therefore, in this specification, for convenience of explanation, at least one of the robots 30 may be such a monitoring device, and such a monitoring device shall perform the following processes, etc. within the scope of the implemented functions.

[0046] FIG. 5 shows the processing flow of the robot 30 according to the present embodiment. In S500, one or more sensors 400 observe the state of the facility 1 in the vicinity of the robot 30. One or more sensors 410 observe the state of the robot 30. The state acquisition unit 430 acquires various measurement data indicating the state of the facility 1 from one or more sensors 400. The state acquisition unit 430 acquires various measurement data indicating the state of the robot 30 from one or more sensors 410.

[0047] In S510, the communication unit 440 transmits various measurement data indicating the state in the facility 1 and the state of the robot 30 observed by one or more sensors 400 and one or more sensors 410 to the monitoring device 40. Here, when the central direction of measurement of each sensor 400 coincides with the direction of the robot 30, that is, for example, when the sensor unit or the like mounting each sensor 400 faces the front of the robot 30, the measurement direction (central direction of measurement) of each sensor 400 coincides with the direction of the robot 30. In such a case, if the robot 30 transmits its own direction to the monitoring device 40, the monitoring device 40 can obtain the measurement direction of each sensor 400.

[0048] When the direction of the sensor unit mounting each sensor 400 is variable with respect to the robot 30 main body, the central direction of measurement of each sensor 400 does not necessarily coincide with the direction of the robot 30. In this case, the robot 30 may transmit direction data including its own direction and the measurement direction of each sensor 400 to the monitoring device 40. Alternatively, the robot 30 may transmit direction data including its own direction and the difference or offset of the measurement direction of each sensor 400 with respect to its own direction to the monitoring device 40. In this case, the monitoring device 40 can obtain the measurement direction of each sensor 400 by adding the difference or offset of the measurement direction of each sensor 400 to its own direction.

[0049] In S520, the communication unit 440 receives an instruction from the monitoring device 40. In S530, the control unit 450 performs an operation according to the instruction from the monitoring device 40. The control unit 450 drives at least one actuator 420 according to an instruction for movement, an instruction for direction change, an instruction for direction change of an acoustic sensor, an image sensor, etc., received from the monitoring device 40, thereby operating the robot 30 as instructed. Further, the control unit 450 may set various parameters in the robot 30 according to an instruction from the monitoring device 40.

[0050] The robot 30 repeats the processes from S500 to S530. Thereby, the robot 30 can move inside the facility 1 according to an instruction from the monitoring device 40, and perform sound collection by the sensor 400a, image capturing by the sensor 400b, acquisition of LiDAR data by the sensor 400c, etc.

[0051] FIG. 6 shows the configuration of the monitoring device 40 according to the present embodiment together with the collected information database 50, the device database 60, the robot database 70, the input device 680, and the display device 690. The input device 680 inputs an instruction for the monitoring device 40 from a user such as an operator, a worker, or a maintenance staff of the facility 1 and supplies it to the monitoring device 40. The display device 690 displays a display screen output by the monitoring device 40. The input device 680 and the display device 690 may be provided in a monitoring console or a user terminal etc. connected to the monitoring device 40. The display device 690 and the input device 680 may be shared with the display device 240 and the input device 250 shown in FIG. 2.

[0052] The monitoring device 40 may be a facility management device that manages each device 10 in the facility 1, or may be a device that realizes some functions included in the facility management device. The monitoring device 40 may be a computer such as a workstation, a server computer, a general-purpose computer, or other computers, or may be a computer system to which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Further, the monitoring device 40 may be implemented by one or more executable virtual computer environments in the computer. Such a computer functions as the monitoring device 40 by executing a program for monitoring each device 10 using the robot 30. Alternatively, the monitoring device 40 may be a dedicated computer designed for monitoring each device 10, or may be dedicated hardware realized by a dedicated circuit.

[0053] In the present embodiment, the monitoring device 40 is installed in the facility 1. Alternatively, the monitoring device 40 may be provided outside the facility 1 by a cloud computing system or the like on the Internet. Further, in the present embodiment, the monitoring device 40 remotely operates each robot 30. Alternatively, the monitoring device 40 may be mounted on at least one robot 30, and directly operate on-site a robot group including each robot 30 or two or more robots 30.

[0054] The monitoring device 40 includes a communication unit 600, a collected information database connection unit 620, a device database connection unit 630, a robot database connection unit 640, a monitoring processing unit 650, an input processing unit 660, a display processing unit 665, and a communication unit 670. The communication unit 600 performs wireless or wired communication with the robot 30. The communication unit 600 includes an acoustic data acquisition unit 602, an image data acquisition unit 604, a LiDAR data acquisition unit 606, a position data acquisition unit 608, a direction data acquisition unit 610, and an instruction transmission unit 612.

[0055] The acoustic data acquisition unit 602 acquires acoustic data detected within Facility 1. The acoustic data acquisition unit 602 may acquire acoustic data by receiving the acoustic data transmitted by the robot 30. The image data acquisition unit 604 acquires image data captured within Facility 1. The image data acquisition unit 604 may acquire image data by receiving the image data transmitted by the robot 30. The LiDAR data acquisition unit 606 acquires LiDAR data detected by the sensor 400c of the robot 30 within Facility 1. The LiDAR data acquisition unit 606 may acquire LiDAR data by receiving the LiDAR data transmitted by the robot 30.

[0056] The position data acquisition unit 608 acquires position data indicating the position of the robot 30. The position data acquisition unit 608 may acquire position data by receiving the position data transmitted by the robot 30. The direction data acquisition unit 610 acquires direction data indicating the directions of the robot 30 and each sensor 400. The direction data acquisition unit 610 may acquire direction data by receiving the direction data transmitted by the robot 30. The instruction transmission unit 612 transmits an instruction for the robot 30 determined within the monitoring device 40 to the robot 30.

[0057] The collected information database connection unit 620 is connected to the collected information database 50, the communication unit 600, and the monitoring processing unit 650. The collected information database connection unit 620 records various measurement data obtained by the acoustic data acquisition unit 602, the image data acquisition unit 604, the LiDAR data acquisition unit 606, the position data acquisition unit 608, and the direction data acquisition unit 610 in the communication unit 600 as collected data in the collected information database 50. Further, the collected information database connection unit 620 accesses the collected information database 50 in response to a request from the monitoring processing unit 650. Note that the monitoring device 40 may not include the collected information database connection unit 620 if it is not necessary to record the history of the collected data, and the measurement data obtained by the communication unit 600 may be supplied to the monitoring processing unit 650 without passing through the collected information database connection unit 620. In this case, the collected information database 50 is unnecessary.

[0058] The device database connection unit 630 is connected to the device database 60 and the monitoring processing unit 650. The device database connection unit 630 accesses the device database 60 in response to a request from the monitoring processing unit 650. The robot database connection unit 640 is connected to the robot database 70 and the monitoring processing unit 650. The robot database connection unit 640 accesses the robot database 70 in response to a request from the monitoring processing unit 650.

[0059] The monitoring processing unit 650 is connected to the collected information database connection unit 620, the device database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670. The monitoring processing unit 650 reads out the robot data of each robot 30 via the robot database connection unit 640, and determines the monitoring actions in the facility 1 to be executed by each robot 30 using the robot data. The monitoring processing unit 650 instructs each robot 30 to perform each operation included in the determined monitoring action via the instruction transmission unit 612 in the communication unit 600.

[0060] Further, the monitoring processing unit 650 reads out the collection data collected by one or more robots 30 and recorded in the collection information database 50 via the collection information database connection unit 620. Then, the monitoring processing unit 650 uses the collection data to detect the state of each device 10 monitored by each robot 30. The monitoring processing unit 650 determines whether each device 10 is normal or not using the collection data. The monitoring processing unit 650 may read out the device data of each device 10 via the device database connection unit 630 and determine whether each device 10 is normal or abnormal based on the information registered in the device data.

[0061] The monitoring processing unit 650 may cause the display processing unit 665 to generate a display screen for displaying the state of each device 10 and the determination result of normal or abnormal, and display it on the display device 690. The monitoring processing unit 650 may transmit the state of each device 10 and the determination result of normal or abnormal to the control system 20 via the communication unit 670.

[0062] The monitoring processing unit 650 may generate an instruction to perform control for dealing with an abnormality in response to the detection of an abnormality in at least one device 10 as a result of monitoring each device 10 using one or more robots 30. The monitoring processing unit 650 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0063] The input processing unit 660 is connected to the input device 680. The input processing unit 660 receives an input of information to the input device 680 from the user and supplies it to the monitoring processing unit 650. The display processing unit 665 is connected to the monitoring processing unit 650. The display processing unit 665 performs display processing to generate a display screen according to an instruction from the monitoring processing unit 650 and display it on the display device 690.

[0064] The communication unit 670 is connected to the monitoring processing unit 650. The communication unit 670 may be connected to the control system 20 and may be capable of communicating with the control system 20 wirelessly or by wire. The communication unit 670 may transmit the device status data of each device 10 detected by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit the determination result of normal or abnormal for each device 10 determined by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit an instruction to perform control for dealing with an abnormality to the control system 20 in response to an abnormality being detected in at least one device 10. Note that the communication unit 670 may not have a function of transmitting at least one of the device status data of the device 10, the determination result of normal or abnormal for the device 10, or the instruction to perform control for dealing with an abnormality to the control system 20. When the monitoring device 40 does not have any of these functions, the communication unit 670 may not be provided.

[0065] According to the monitoring device 40 shown above, by monitoring the inside of the facility 1 using one or a plurality of robots 30 that can move inside the facility 1, a large number of devices 10 can be monitored using a relatively small number of robots 30. In addition, since the monitoring device 40 can control the robot 30 according to the situation at each observation point and collect necessary information, it is possible to detect the state of each device 10 that cannot be collected by sensors or the like pre-installed in each device 10.

[0066] FIG. 7 shows an example of the data structure of the collected information database 50 according to the present embodiment. The collected information database 50 records the measurement data collected from each robot 30 as collected data. The collected information database 50 may record one or more records, each of which is a unit of collected data. In the data structure of the collected information database 50 shown in this figure, each record of the collected data is arranged in the row direction, and the collected data of each record includes fields of "date and time", "position and direction", "robot identification information", and "measurement data".

[0067] "Date and Time" is a field that records the date and time when the measurement data of the corresponding record was acquired by the robot 30. "Position and Orientation" is a field that records at least one position and orientation of the robot 30 or each sensor 400 at the timing when the measurement data of the corresponding record was acquired. The collection information database 50 may record, as "Position and Orientation", at least one of the position detected by the sensor 410a of the robot 30, the azimuth of the robot 30 detected by the sensor 410b, or the direction of the robot 30 or each sensor 400 detected by the sensor 410c at the timing indicated by "Date and Time".

[0068] "Robot Identification Information" is a field that records data values such as an identification number, a serial number, or other unique number or character string that can identify the robot 30 that acquired the measurement data of the corresponding record, or can specify the robot 30 that acquired the measurement data within Facility 1. "Measurement Data" is a field that records the measurement data acquired by the robot 30 at the timing indicated by "Date and Time". The data recorded as "Measurement Data" may be various types of measurement data acquired by the robot 30, such as at least one of acoustic data, image data, or LiDAR data.

[0069] The communication unit 600 in the monitoring device 40 may receive, as separate packets or packet groups, etc., data including the acquisition date and time of measurement data by the robot 30, the robot identification information of the robot 30, and measurement data regarding the position and orientation of the robot 30, and data including the acquisition date and time of measurement data by the robot 30, the robot identification information of the robot 30, and measurement data regarding the state within the facility 1. In this case, each time the communication unit 600 receives measurement data regarding the position and orientation of the robot 30, it updates the position and orientation of the robot 30 managed by the monitoring device 40. When the communication unit 600 receives measurement data regarding the state within the facility 1, it generates a record of the collected data by associating the measurement data with the position and orientation of the robot 30 at the acquisition date and time of the measurement data. Alternatively, when the communication unit 440 in the robot 30 transmits various measurement data indicating the state within the facility 1 and the state of the robot 30 observed by each sensor 400 and each sensor 410 to the monitoring device 40 (see S510 in FIG. 5), it may assemble the collected data for one record from these measurement data, encode it into one or two or more packets, and transmit it to the monitoring device 40.

[0070] FIG. 8 shows an example of the data structure of the device database 60 according to the present embodiment. The device database 60 records device data for each of a plurality of devices 10 within the facility 1. The device database 60 may record records, each of which is device data for one device, for the number of devices 10. In the data structure of the device database 60 shown in this figure, each record of the device data is arranged in the row direction, and the device data of each record includes fields of "device identification information", "device name", "device information", "position", "template", and "acoustic data".

[0071] "Device identification information" is a field for recording a data value such as an identification number, a serial number, or other unique number or character string that can identify the corresponding device 10 within the facility 1. "Device name" is a field for recording the device name assigned to the corresponding device 10 by a user or the like.

[0072] "Device Information" is a field that records various information about the corresponding device 10. "Location" is a field that records the location of the device 10 within the facility 1. "Template" is a field that records image data (template image data) of the appearance of the corresponding device 10 for use in identifying the device 10 by image matching. "Acoustic Data" is a field that records at least one of the acoustic data collected from the corresponding device 10 or the acoustic data emitted by the device 10 when the corresponding device 10 is normal.

[0073] FIG. 9 shows an example of the data structure of the robot database 70 according to the present embodiment. The robot database 70 records robot data for each of one or more robots 30 in the facility 1. The robot database 70 may record records, each of which is robot data for one robot, for the number of robots 30. In the data structure of the robot database 70 shown in this figure, each record of the robot data is arranged in the row direction, and the robot data of each record includes fields of "Robot Identification Information", "Robot Name", "Robot Information", "Location and Direction", and "Schedule Information".

[0074] "Robot Identification Information" is a field that records a data value such as an identification number, a serial number, or other unique number or character string that can identify the corresponding robot 30 within the facility 1. "Robot Name" is a field that records the robot name assigned to the corresponding robot 30 by a user or the like.

[0075] "Robot Information" is a field that records various information about the corresponding robot 30. "Position and Direction" is a field that records the position and direction of the corresponding robot 30 within Facility 1, and, if necessary, the direction of each sensor 400 (absolute direction or offset with respect to the direction of robot 30). "Schedule Information" is a field that records the schedule for the corresponding robot 30 to patrol within Facility 1 and collect sounds within Facility 1 or the sounds of each device 10, etc.

[0076] Figure 10 shows the configuration of the monitoring processing unit 650 of the monitoring device 40 according to the present embodiment. The monitoring processing unit 650 includes a sound source separation unit 1000, a device detection unit 1010, a sound source identification unit 1020, a sound anomaly detection unit 1030, an abnormal device identification unit 1040, a device control unit 1050, and a robot command unit 1060.

[0077] The sound source separation unit 1000 is connected to the collection information database connection unit 620. The sound source separation unit 1000 reads out, via the collection information database connection unit 620, the collection data in which acoustic data is recorded as measurement data among the collection data registered in the collection information database 50. As a result, the sound source separation unit 1000 obtains the acoustic data, the date and time when the acoustic data was acquired, and the position and direction where the acoustic data was observed within Facility 1. The sound source separation unit 1000 extracts at least one sound component by performing sound source separation on the acquired acoustic data.

[0078] The device detection unit 1010 is connected to the collected information database connection unit 620. The device detection unit 1010 reads out, via the collected information database connection unit 620, the collected data in which image data is recorded as measurement data among the collected data registered in the collected information database 50. Thereby, the device detection unit 1010 acquires the image data, the date and time when the image data was acquired, and the position and direction where the image data was observed within the facility 1. The device detection unit 1010 detects at least one device 10 imaged in the image data by performing image recognition or the like on the image data. The device detection unit 1010 may search for the device 10 within the facility 1 corresponding to the device imaged in the image data by accessing the device database 60 via the device database connection unit 630 and referring to each record of the device data.

[0079] The sound source identification unit 1020 is connected to the sound source separation unit 1000 and the device detection unit 1010. The sound source identification unit 1020 identifies which of the at least one device 10 imaged in the image data is the sound source of each sound component among the at least one sound component included in the acoustic data. The sound source identification unit 1020 according to the present embodiment identifies which of the devices 10 detected from the image data by the device detection unit 1010 is the sound source of each sound component extracted from the acoustic data by the sound source separation unit 1000. The sound source identification unit 1020 instructs the device database connection unit 630 to record the sound component data regarding each sound component in the device database 60 in association with the device 10 that is the sound source among the at least one device 10 imaged in the image data. When the device 10 corresponding to any sound component cannot be identified, the sound source identification unit 1020 may receive the designation of the corresponding device 10 or the like by receiving the input of information from the user via the input processing unit 660.

[0080] The abnormal sound detection unit 1030 is connected to the sound source identification unit 1020. The abnormal sound detection unit 1030 detects the degree of abnormality of the sound component using the sound component data associated with each device 10.

[0081] The abnormal device identification unit 1040 is connected to the sound abnormality detection unit 1030. The abnormal device identification unit 1040 identifies an abnormality of the device 10 that is the sound source of the sound component based on the degree of abnormality of the sound component detected by the sound abnormality detection unit 1030. The abnormal device identification unit 1040 may cause the display processing unit 665 to generate a display screen for displaying the state of each device 10 detected by the robot 30 on the display device 690 and display it on the display device 690. Further, the abnormal device identification unit 1040 may cause the display processing unit 665 to generate a display screen for displaying the determination result of normal or abnormal of each device 10 on the display device 690 and display it on the display device 690.

[0082] The device control unit 1050 is connected to the abnormal device identification unit 1040. When an abnormality of any device 10 that is the sound source of the sound component is identified, the device control unit 1050 performs control for dealing with the abnormality of the device 10 on at least one of the device 10 in which the abnormality is identified or other devices 10 in the facility 1. The device control unit 1050 transmits an instruction for performing control for dealing with the abnormality of the device 10 to the instruction input unit 260 in the control system 20 via the communication unit 670.

[0083] The robot command unit 1060 is connected to the collection information database connection unit 620, the robot database connection unit 640, and the sound source identification unit 1020. The robot command unit 1060 determines the operation that each robot 30 should perform and instructs each robot 30 to perform the determined operation.

[0084] The monitoring processing unit 650 shown above has each component shown in FIG. 10 in order to realize each function such as identification of the device 10 that is the sound source of each sound component, identification of the abnormality of the device 10 of the sound source based on the degree of abnormality of the sound component, instruction of control for dealing with the abnormality of the device 10, and instruction of the operation for each robot 30. Instead of this, the monitoring processing unit 650 may adopt a configuration that does not have a part of the components shown in FIG. 10.

[0085] For example, the monitoring processing unit 650 may not have the device control unit 1050 and may not issue an instruction for control to handle an abnormality of the device 10. Further, the monitoring processing unit 650 may not have the robot command unit 1060 and may not issue an instruction for an operation to each robot 30. Further, the monitoring processing unit 650 may not have the sound abnormality detection unit 1030 and the abnormal device identification unit 1040 and may not identify an abnormality of the device 10 that is a sound source based on the degree of abnormality of the sound component. Even when the monitoring device 40 does not have some of the functions exemplified herein, the monitoring device 40 can cause the user to receive information output or displayed by the monitoring device 40, discover an abnormality of the device 10, and take technical measures to be taken in response to the abnormality of the device 10 for at least one device 10 in the facility 1.

[0086] According to the monitoring device 40 described above, it is possible to collect the acoustic data detected in the facility 1 and the image data captured in the facility 1, and identify the device 10 that is the sound source of each sound component included in the acoustic data. Thereby, the monitoring device 40 can analyze the potential state of each device 10 that may not be detected from the state of each device 10 collected by the control system 20 from the sound component having the device 10 as the sound source. Further, the monitoring device 40 can promptly collect abnormal sounds emitted by the devices 10 installed at various locations in the facility 1, and can promptly take measures against the abnormality of the devices 10.

[0087] FIGS. 11 and 12 show the processing flow of the monitoring device 40 according to the present embodiment. The processing flows of FIGS. 11 and 12 show the case where the monitoring device 40 monitors the facility 1 using one robot 30 for convenience of explanation. The monitoring device 40 may execute the processing flows of FIGS. 11 and 12 for each of the plurality of robots 30.

[0088] In S1100, the robot command unit 1060 instructs the robot 30 to be processed to move within the facility 1 via the instruction transmission unit 612. Here, the robot command unit 1060 may refer to the schedule information recorded in the robot database 70 in association with the robot 30 to be processed, and acquire the schedule for the robot 30 to patrol within the facility 1. This schedule may include information necessary for determining the operation of the robot 30, such as the positions of the observation points where the robot 30 should observe the state within the facility 1, the times when the robot 30 should arrive at each observation point, the movement routes between the observation points, or the observation directions at each observation point. The robot command unit 1060 moves the robot 30 to the next observation point according to the specification in this schedule, and supplies an instruction to the robot 30 to perform observation at the next observation point. In response to this, the robot 30 moves to the next observation point (see S520 and S530 in FIG. 5). If an area where an abnormality may occur within the facility 1 is specified, the robot command unit 1060 may instruct at least one robot 30 to head towards that area and move to various positions within that area to acquire acoustic data and image data, etc.

[0089] In S1110, the robot 30 observes the state within the facility 1 and the state of the robot 30 using each sensor 400 at the observation point (see S500 in FIG. 5). The robot 30 transmits various measurement data indicating the observed state within the facility 1 and the state of the robot 30 to the monitoring device 40 (see S510 in FIG. 5). The acoustic data acquisition unit 602 in the monitoring device 40 acquires the acoustic data from among the measurement data transmitted from the robot 30. The collection information database connection unit 620 may add the position data acquired by the position data acquisition unit 608, the direction data acquired by the direction data acquisition unit 610, etc. to the acoustic data acquired by the acoustic data acquisition unit 602, and record it in the collection information database 50 as collection data.

[0090] In S1120, the sound source separation unit 1000 extracts at least one sound component by performing sound source separation or the like on the acoustic data from the robot 30. The sound source separation unit 1000 may perform sound source separation by extracting at least one sound component having different directions of sound sources as viewed from the robot 30 from the acoustic data. The sound source separation unit 1000 may identify the direction of each sound component as viewed from the robot 30 (sound source localization).

[0091] Here, since each sensor 400a collects sound from a range of predetermined directions other than the central direction of measurement of the sensor 400a according to the directivity, the sensor 400a collects sound that is a synthesis of sounds from various sound sources. The sound source separation unit 1000 performs a process of separating or decomposing the sound collected by each sensor 400a into the sounds generated by each sound source. In this specification, for convenience of explanation, the sound obtained by extracting at least a part of the sound collected by the sensor 400a is referred to as a "sound component", but the "sound component" itself is also a sound. Therefore, the acoustic data that is the data of the sound collected by the sensor 400a and the sound component data that is the data of the sound component may be in the same data format.

[0092] The sound source separation unit 1000 may utilize various sound source separation and sound source localization techniques. For example, the robot 30 may have two or more sensors 400a, and the acoustic data received by the monitoring device 40 may include acoustic data for each channel that records the sound collected by each sensor 400a.

[0093] The sound source separation unit 1000 may calculate the direction of the sound source of each sound component with respect to the robot 30 from the time difference at which each sound component included in the acoustic data reaches each sensor 400a. Here, the sound source separation unit 1000 may calculate the relative direction of the sound source of each sound component with respect to the measurement direction of the sensor unit including two or more sensors 400a, and add it to the absolute direction (the central direction of measurement) in which the sensor unit itself is facing, thereby calculating the absolute direction of the sound source of each sound component at the observation point. Here, the absolute direction means an angle represented starting from at least a common reference direction (such as north) within the facility 1, such as the horizontal and vertical angles based on north. The relative direction means an angle represented starting from an arbitrarily selected reference direction (such as the traveling direction of the robot 30).

[0094] Further, the robot 30 may have a sensor 400a such as a directional microphone, and may acquire acoustic data while changing the orientation of the sensor 400a within a predetermined range of directions with respect to the robot 30. For example, the robot 30 may acquire acoustic data while changing the orientation of the sensor 400a within the angular range of the image data captured by the sensor 400b. The sound source separation unit 1000 may use such acoustic data to specify the orientation of the sensor 400a when the magnitude (sound pressure) of each sound component is maximized as the direction of the sound source of that sound component.

[0095] By separating the acoustic data into sound sources in this way, the monitoring device 40 can identify the sound components originating from each device 10 even when it has acquired acoustic data in which sounds from two or more devices 10 are superimposed. Thereby, the monitoring device 40 can increase the detection rate of abnormalities in the device 10 and can execute appropriate countermeasures against the abnormalities in the device 10.

[0096] In S1130, the image data acquisition unit 604 in the monitoring device 40 acquires image data from among the measurement data transmitted from the robot 30. The collection information database connection unit 620 may add the position data acquired by the position data acquisition unit 608, the direction data acquired by the direction data acquisition unit 610, etc. to the image data acquired by the image data acquisition unit 604, and record it in the collection information database 50 as collection data.

[0097] In S1140, the device detection unit 1010 detects one or more devices 10 imaged in the image data by performing image recognition or the like on the image data transmitted from the robot 30. The device detection unit 1010 may extract one or more objects imaged in the image data, search for the device 10 corresponding to each object from the device database 60, and detect the device 10 corresponding to each object. Here, the device detection unit 1010 may search for the device 10 located in the direction of the robot 30 or the sensor 400b at this timing from the position of the robot 30 at the timing when the image data was acquired, from the device database 60.

[0098] The device detection unit 1010 may recognize each device 10 included in the image data using vision-based AR (Augmented Reality) technology. For example, the device detection unit 1010 may recognize each device 10 in the image data using markerless AR technology. The device detection unit 1010 may detect each of at least one device 10 included in the image data by matching it with a predetermined template. For example, the device detection unit 1010 extracts one or more objects imaged in the image data, and searches the device database 60 for the device 10 associated with the template image data that matches or is most similar to the image of each object. The device detection unit 1010 may detect the device 10 retrieved for each object as the device 10 located in the image range including that object in the image data. When the template image data associated with the device 10 retrieved from the device database 60 using the position of the robot 30 and the direction of the robot 30 or the sensor 400b at the timing of acquiring the image data matches or is similar to the image of the object in the image range corresponding to the direction of the sensor 400b with respect to the position of the robot 30 in the image data with a similarity equal to or higher than a predetermined threshold, the device detection unit 1010 may determine that the device 10 corresponds to that object.

[0099] The device detection unit 1010 may also recognize each device 10 included in the image data using AR technology that uses markers. In this case, the device data stored in the device database 60 may store the identification information of the marker attached to the corresponding device 10 in, for example, the device information field. The device detection unit 1010 may search the device database 60 for the device 10 associated with the device data in which the identification information matching the identification information of the marker attached to the device 10 included in the image data is stored.

[0100] The device detection unit 1010 may detect a device 10 that is not registered in the device database 60. For example, the device detection unit 1010 may detect, as an unregistered device 10, a device 10 newly installed in the facility 1 or a part of a device 10 that is already registered in the device database 60. The device detection unit 1010 may estimate the type (e.g., reactor, measuring instrument, pipe, or flange, etc.) or model of the device 10 corresponding to the object from the external shape, etc. of the object extracted from the image data. The device detection unit 1010 may detect an unknown device 10 included in the image data by matching it with a template or the like predetermined for each type or model of the device, and specify the type or model of the device. The monitoring device 40 can extract, from the image data, a device 10 that is installed in the facility 1 but not registered and a part of a device 10 that has not been recognized as a failure unit capable of emitting abnormal sounds and has not been registered as an individual device 10, by performing image recognition on the image data using template matching or the like, and make it possible to detect the sound source.

[0101] The device detection unit 1010 may calculate the relative direction of the device 10 with respect to the center direction of the image from the position of the device 10 in the image corresponding to the image data. The device detection unit 1010 may calculate the absolute direction of each device 10 viewed from the observation point by adding the relative direction of the device 10 with respect to the center direction of the image to the absolute direction of the sensor 400b at the time of imaging the image data.

[0102] In S1150, the sound source identification unit 1020 identifies which device 10 imaged in the image data is the sound source of each sound component included in the acoustic data. The sound source identification unit 1020 may identify, for each sound component extracted from the acoustic data, among the devices 10 detected from the image data, the device 10 located in the region corresponding to the direction of the sound source within the image data as the sound source device 10. As an example, for a certain sound component and a certain device 10 detected from the acoustic data and the image data observed by the robot 30 at the same position at the same timing or different timings, the sound source identification unit 1020 may determine that the sound source of this sound component is this device 10 when the absolute direction of the sound component and the absolute direction of the device 10 as seen from the observation point are the same, or when the difference between these absolute directions is within a predetermined error range. For the device 10 already registered in the device database 60, the sound source identification unit 1020 may determine that the sound source of this sound component is this device 10 on the condition that the device 10 is located in the absolute direction of the sound component as seen from the observation point of the acoustic data.

[0103] Also, for a certain sound component and a certain device 10 obtained from acoustic data and image data observed by the robot 30 at different positions at the same timing or different timings, the sound source identification unit 1020 determines that the sound source of this sound component is this device 10 on the condition that the absolute direction of the sound component as seen from the observation point of the acoustic data and the absolute direction of the device 10 as seen from the observation point of the image data intersect or approximately intersect within a predetermined error range. Here, the device detection unit 1010 may calculate an approximate distance range from the observation point of the image data to each device 10 based on the size etc. of each device 10 in the image corresponding to the image data. In this case, in addition to the absolute direction of the sound component as seen from the observation point of the acoustic data and the absolute direction of the device 10 as seen from the observation point of the image data intersecting or approximately intersecting within a predetermined error range, when the distance from the observation point of the image data to this intersection position is within the calculated distance range, the sound source identification unit 1020 may determine that the sound source of this sound component is this device 10. The monitoring device 40 can identify the device 10 that is the sound source of each sound component by using the local acoustic data and image data observed by the robot 30 at the observation point, so that the sound information of each device 10 that could not be detected by the measurement by the control system 20 and was buried in the facility 1 can be made apparent as information usable for the management of each device 10.

[0104] In S1160, the sound source identification unit 1020 instructs the device database connection unit 630 to record the sound component data regarding each sound component in the device database 60 in association with the device 10 that is the sound source. The device database connection unit 630, upon receiving the instruction from the sound source identification unit 1020, records the sound component data regarding each sound component in the acoustic data field in the record of the device data corresponding to the device 10 that is the sound source in the device database 60. The device database connection unit 630 may record a plurality of sound component data obtained at different dates and times in the acoustic data field by adding the acquisition date and time of the sound component data and the sound component data to the acoustic data field.

[0105] In addition, when the sound source of a certain sound component is device 10 that is included in the image data but not registered in the device database 60, the sound source identification unit 1020 may register this device 10 in the device database 60 through the input of information from the user by the input processing unit 660, and record the sound component data in association with this device 10 in the device database 60. This process will be described later in relation to S1250.

[0106] In S1200, the sound abnormality detection unit 1030 detects the degree of abnormality of the sound component using the sound component data associated with each device 10. Here, the degree of abnormality may be represented by a real number or an integer within a predetermined range such as from 0 to 1 or from 0 to 100%, or may be a binary value such as normal or abnormal.

[0107] If the acoustic data emitted by device 10 when device 10 is normal is recorded in the device database 60, the sound abnormality detection unit 1030 may calculate, as the degree of abnormality, a value indicating how different the sound component data of the target with device 10 as the sound source is from the acoustic data emitted by device 10 during normal times. For example, the sound abnormality detection unit 1030 may determine the degree of abnormality according to the comparison result between the acoustic data during normal times and the sound component data of the target.

[0108] The sound abnormality detection unit 1030 may perform the comparison between the acoustic data during normal times and the sound component data of the target in either the time domain or the frequency domain. When performing the comparison in the time domain, the sound abnormality detection unit 1030 adjusts the phase difference so that the time integral of the difference (such as the absolute value of the difference) between the acoustic data during normal times and the sound component data of the target is minimized, and may use the time integral of the difference between the acoustic data and the sound component data of the target at the adjusted phase difference as the degree of abnormality. In this case, the sound abnormality detection unit 1030 may adjust the amplitude so that, for example, the average amplitudes match in order to match the loudness of the sound of the acoustic data and the sound component data of the target.

[0109] When comparing the normal acoustic data with the target sound component data in the frequency domain, the sound abnormality detection unit 1030 may calculate the abnormality degree by integrating the difference (such as the absolute value of the difference) for each frequency between the frequency spectrum of the normal acoustic data and the frequency spectrum of the target sound component data in the frequency direction. In this case, the sound abnormality detection unit 1030 may perform an adjustment to match the sound levels so as to make the acoustic data and the target sound component data comparable.

[0110] When the acoustic data collected from the device 10 in the past is recorded as one or more histories in the acoustic data field in the device database 60, the sound abnormality detection unit 1030 may calculate, as the abnormality degree, a value indicating how different the target sound component data with the device 10 as the sound source is from the one or more acoustic data collected from the device 10 in the past. The sound abnormality detection unit 1030 may calculate the abnormality degree of the target sound component data with respect to the acoustic data collected in the past in the same manner as the calculation method when calculating the abnormality degree of the target sound component data with respect to the acoustic data emitted by the device 10 during normal operation described above.

[0111] The sound abnormality detection unit 1030 may determine the abnormality degree according to the degree of deviation of the target sound component data from the distribution of the acoustic data collected from the device 10 in the past. Also, the sound abnormality detection unit 1030 may calculate, as the abnormality degree, the sum of the degrees of difference for each of a predetermined number of acoustic data with the smallest difference from the target sound component data among the acoustic data collected from the device 10 in the past (k-nearest neighbor method). In addition to the above, the sound abnormality detection unit 1030 may calculate the abnormality degree of the acoustic data and the target sound component data using various methods for calculating the difference, dissimilarity, similarity, or degree of deviation between two sounds.

[0112] The abnormal sound detection unit 1030 may detect the degree of abnormality of the sound component using AI technology. The abnormal sound detection unit 1030 may receive the sound components emitted by each device 10 from the sound source identification unit 1020 and cause them to be learned by a machine learning model for each device 10. The abnormal sound detection unit 1030 may store the machine learning model of each device 10 or the learned parameters representing the machine learning model in the device information field of the device database 60 or the like, and update the machine learning model each time it newly receives the sound components emitted by the device 10. Further, the abnormal sound detection unit 1030 may receive the determination result of normal or abnormal for each sound component. In this case, the abnormal sound detection unit 1030 may generate or update the machine learning model by learning so that the correct determination result is output when the sound component is input to the machine learning model.

[0113] The abnormal sound detection unit 1030 may use a neural network, statistical learning, or other machine learning algorithms. The abnormal sound detection unit 1030 may use the time-series data of the sound component as an input to the machine learning model, and may use the frequency spectrum of the sound component as an input to the machine learning model. The abnormal sound detection unit 1030 updates the parameters of the machine learning model so as to reduce the error between the output of the model when the sample of each sound component serving as learning data is input to the machine learning model and the learning label indicating whether the sound component is normal or abnormal. For example, when using a neural network, the abnormal sound detection unit 1030 inputs the data value at each time or the data value at each frequency in the sound component to each input node of the input layer of the neural network. The abnormal sound detection unit 1030 adjusts the weights between the neurons of the neural network and the biases of each neuron, etc. by a method such as backpropagation using the error between the output value output by the neural network in response to the input of each sample and the label.

[0114] Note that the machine learning model may output the abnormality degree of the sound component in response to inputting the acoustic data collected from the device 10 in the past or the normal acoustic data and the sound component data from the device 10. Also, instead of being prepared for each individual device 10, the machine learning model may be prepared for each type or model of the device 10. Further, the machine learning model may be common to all the devices 10.

[0115] For example, by using the machine learning model learned in this way, in response to receiving the sound component generated from the device 10, the sound abnormality detection unit 1030 reads out the machine learning model corresponding to the device 10 from the device database 60, inputs the sound component, and can use the determination result of normal or abnormal output by the machine learning model as the abnormality degree. The sound abnormality detection unit 1030 may calculate the abnormality degree based on the difference between the normal acoustic data and the target sound component data by using any other method.

[0116] Note that when the device 10 is not registered in the device database 60 but the type or model of the device 10 has been estimated, the sound abnormality detection unit 1030 may determine the abnormality degree of the target sound component data based on the acoustic data normally generated by the device 10 of the same type or model as the estimated type or model or the acoustic data that has occurred in the past.

[0117] In S1210, the abnormal device identification unit 1040 identifies the abnormality of the device 10 that is the sound source of the sound component based on the abnormality degree of the sound component associated with the device 10. For example, when the abnormality degree of the sound component exceeds a predetermined threshold, the abnormal device identification unit 1040 may identify that the device 10 that is the sound source of the sound component is abnormal.

[0118] In S1220, the abnormal device identification unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays the status of each device 10 detected by the robot 30. The abnormal device identification unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays information (such as the observation location, direction, image, etc.) regarding one or more devices 10 that have been identified as the source of a certain sound component data but are not registered in the device database 60. Further, the abnormal device identification unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays the observation location, the direction of the sound component data, etc. for the sound component data that is not associated with any of the devices 10.

[0119] In S1230, the monitoring device 40 determines whether there is any sound component data that is not associated with any of the devices 10. If all the sound component data are each associated with one of the devices 10 (''N'' in S1230), the monitoring device 40 advances the process to S1240. The monitoring device 40 may advance the process to S1240 when all the sound component data are associated with one of the devices 10 registered in the device database 60.

[0120] In S1240, when an abnormality of one of the devices 10 that is the source of the sound component is identified, the device control unit 1050 performs control for dealing with the abnormality of the device 10 on at least one of the device 10 in which the abnormality has been identified or other devices 10. The device control unit 1050 transmits an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0121] As an example, the device control unit 1050 may instruct the control system 20 to perform control to stop the operation of the device 10 in which an abnormality has been identified. The device control unit 1050 may instruct the control system 20 to perform control to stop other devices 10 related to the device 10 in which an abnormality has been identified (for example, devices 10 in the upstream or downstream processes of the device 10 in which an abnormality has been identified, etc.). The device control unit 1050 may also instruct the control system 20 to perform control to stop the operations of both the device 10 in which an abnormality has been identified and other devices 10 related to the device 10 in which an abnormality has been identified. The monitoring device 40 proceeds with the process to S1100 and continues the monitoring of the facility 1 by the robot 30.

[0122] If there is sound component data that is not associated with any device 10 in S1230 (''Y'' in S1230), the monitoring device 40 proceeds with the process to S1250. Here, the monitoring device 40 may also proceed with the process to S1250 even when the sound component data is associated with a device 10 that is not registered in the device database 60.

[0123] In S1250, in response to the fact that the input processing unit 660 cannot identify the device 10 located in the image area corresponding to the direction of the sound source in the acoustic data for any of the sound components, the input processing unit 660 accepts input of information from the user for identifying the device 10. Here, if the device 10 associated with the sound component is unregistered and its presence in the facility 1 has not been confirmed, the input processing unit 660 accepts input of information about at least a part of the device data regarding the unregistered device 10, and the sound source identification unit 1020 may register the device data of the device 10 corresponding to the received information in the device database 60. In response to such input of information (''Y'' in S1260), the monitoring device 40 proceeds with the process to S1150. Thereby, the sound source identification unit 1020 can identify the sound source of each sound component including the newly registered device 10.

[0124] When the device 10 associated with the sound component is unregistered, the input processing unit 660 may receive an input of information indicating whether to register the device 10 in the device database 60. The sound source identification unit 1020 may register the device data of the device 10 corresponding to the type or model of the device 10 detected from the image data, the position or approximate position, or other information that has already been determined, in the device database 60.

[0125] In addition, the input processing unit 660 may receive an input of information specifying the device 10 of the sound source for the sound component data that is not associated with any device 10. In response to such an input of information (''Y'' in S1260), the monitoring device 40 advances the process to S1150. Thereby, the sound source identification unit 1020 can identify that the sound source of the sound component data is the specified device 10.

[0126] In S1270, in response to the fact that the sound source device 10 of each device 10 cannot be identified for any of the sound components, the robot command unit 1060 instructs the robot 30 to perform at least one of changing the imaging direction of the image, changing the imaging magnification of the image, or changing the position of the robot 30. In the processing flow of this figure, the robot command unit 1060 performs the processing of S1270 when there is no input of information regarding the device 10 corresponding to the sound component (''N'' in S1260). Alternatively, the robot command unit 1060 may automatically instruct the robot 30 to change the imaging direction of the image, change the imaging magnification of the image, change the position of the robot 30, etc., when the sound source device 10 of any of the sound components cannot be identified.

[0127] For example, when it is impossible to identify whether the sound source of a certain sound component is one of the two devices 10, the robot command unit 1060 may instruct the robot 30 to move closer to the two devices 10. Also, when there is a sound component from outside the viewing angle of the image data, the robot command unit 1060 may instruct the robot 30 to move the imaging direction of the image closer to the direction of the sound component, reduce the imaging magnification of the image to image the range including the direction of the sound component, or move toward the sound component outside the viewing angle. After instructing the robot 30, the monitoring device 40 proceeds with the process to S1110 and performs the processes from S1110 to S1220 on the acoustic data and image data newly acquired by the robot 30. Thereby, even in a situation where the monitoring device 40 cannot determine which device 10 is the sound source of the sound component in the observation from one observation point or under one observation condition, by changing the observation point or observation condition and repeating the observation, the accuracy of associating the sound component with the device 10 of the sound source can be further improved.

[0128] FIG. 13 shows an example of image data 1300 captured by the robot 30 according to the present embodiment. The image data 1300 captures devices 10 such as a plurality of pipes 1305a to 1305h, a valve 1310, a reactor 1320, and a flow meter 1330. In the example of this figure, there is an abnormality in the valve 1310 and it is emitting abnormal noise.

[0129] The monitoring device 40 receives collection data including the image data 1300 and acoustic data obtained by collecting sounds from within the viewing angle range of the image data 1300 (corresponding to S1110 and S1130 in FIG. 11). The sound source separation unit 1000 separates the acoustic data by sound source and extracts sound components from each direction within the viewing angle range of the image data 1300 (corresponding to S1120). The device detection unit 1010 detects each device 10 imaged in the image data 1300 by performing image recognition or the like on the image data 1300 (corresponding to S1140).

[0130] When the direction of a certain sound component coincides with (or coincides within a predetermined error range) the direction corresponding to the area where a certain device 10 is located in the image data 1300, the sound source generation part 1020 may specify that the sound source of the sound component is the device 10 (corresponding to S1150). The sound abnormality detection part 1030 detects the degree of abnormality of the sound component by using the sound component data associated with each device 10 (corresponding to S1200). In the example of this figure, the sound abnormality detection part 1030 detects a high degree of abnormality for the sound component from the valve 1310 that is emitting abnormal noise. As a result, the abnormal device specification part 1040 can specify that the valve 1310 is abnormal based on the degree of abnormality of the sound component.

[0131] Here, when the pipes 1305a to 1305c and the valve 1310 are newly installed in the facility 1 and are not registered in the device database 60, the device detection part 1010 may estimate that the pipes 1305a to 1305c are pipes and the valve 1310 is a valve from the appearance shape and the like of the object extracted from the image data 1300. The sound abnormality detection part 1030 may detect the degree of abnormality of the sound component based on the result of comparing the sound component data from the valve 1310 with the acoustic data emitted when the valve is normal or the acoustic data emitted by one or a plurality of other valves in the facility 1 in the past.

[0132] Note that depending on the type of the sensor 400a used by the robot 30, the sound source separation part 1000 may be able to separate the sound source in the horizontal direction, for example, but may not be able to separate the sound source in the vertical direction. In such a case, the sound source generation part 1020 may specify the device 10 that is the sound source of each sound component while appropriately changing the imaging direction and position of the robot 30 (S1270 in FIG. 12) by using the horizontal position of each device 10 in the image data 1300 and the direction of each sound component in the horizontal direction.

[0133] FIG. 14 shows the configuration of the monitoring processing unit 1400 of the monitoring device 40 according to the first modification of the present embodiment. The monitoring device 40 shown with reference to FIGS. 14 to 21 is a modification of the monitoring device 40 shown with reference to FIGS. 1 to 13. The monitoring device 40 according to this modification includes a monitoring processing unit 1400 instead of the monitoring processing unit 650. Since the functions and configurations other than the monitoring processing unit 1400 in the monitoring device 40 according to this modification are basically the same as those of the monitoring device 40 according to FIGS. 1 to 13, the description will be omitted except for the following differences.

[0134] Similar to the monitoring processing unit 650, the monitoring processing unit 1400 is connected to the collection information database connection unit 620, the device database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670 of the monitoring device 40. The monitoring processing unit 1400 reads the robot data of each robot 30 via the robot database connection unit 640, and determines the monitoring actions in Facility 1 that each robot 30 should execute using the robot data. The monitoring processing unit 1400 instructs each robot 30 to perform each operation included in the determined monitoring actions via the instruction transmission unit 612 in the communication unit 600.

[0135] The monitoring processing unit 1400 may cause the display processing unit 665 to generate a display screen for displaying the status of each device 10 and the determination result of normal or abnormal, and display it on the display device 690. The monitoring processing unit 1400 may transmit the status of each device 10 and the determination result of normal or abnormal to the control system 20 via the communication unit 670.

[0136] When at least one device 10 is detected as abnormal as a result of monitoring each device 10 using one or more robots 30, the monitoring processing unit 1400 may generate an instruction to perform control for dealing with the abnormality. The monitoring processing unit 1400 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0137] The monitoring processing unit 1400 according to this modification example has a function of identifying an abnormal area where an abnormality may exist in Facility 1 by using the acoustic data detected at each of a plurality of positions in Facility 1. The monitoring processing unit 1400 includes an acoustic anomaly detection unit 1430, an abnormal device identification unit 1440, an abnormal area identification unit 1435, a device control unit 1450, and a robot command unit 1460.

[0138] The acoustic anomaly detection unit 1430 is connected to the collection information database connection unit 620. The acoustic anomaly detection unit 1430 acquires the acoustic data (also referred to as "first acoustic data") detected by one or more robots 30 at each of a plurality of positions (also referred to as "first positions") in Facility 1 from the collection information database 50 via the collection information database connection unit 620. The acoustic anomaly detection unit 1430 detects the degree of acoustic abnormality at each first position by using the first acoustic data detected at each of the plurality of first positions. Further, the acoustic anomaly detection unit 1430 may acquire the acoustic data (also referred to as "second acoustic data") detected at each of a plurality of positions (also referred to as "second positions") within the abnormal area specified by the following abnormal area identification unit 1435 from the collection information database 50 via the collection information database connection unit 620. In this case, the acoustic anomaly detection unit 1430 may detect the degree of acoustic abnormality at each second position by using the second acoustic data detected at each of the plurality of second positions.

[0139] The abnormal area identification unit 1435 is connected to the acoustic anomaly detection unit 1430. The abnormal area identification unit 1435 identifies an abnormal area where an abnormality may exist in Facility 1 based on the degree of acoustic abnormality detected at each of the plurality of first positions. The abnormal area identification unit 1435 may identify the device in which the abnormality has occurred if it is possible to narrow down the device in which the abnormality has occurred within the abnormal area.

[0140] The abnormal device identification unit 1440 is connected to the acoustic anomaly detection unit 1430. The abnormal device identification unit 1440 identifies the device 10 in which an abnormality has occurred in Facility 1 based on the degree of acoustic abnormality detected at each of the plurality of second positions.

[0141] The machine control unit 1450 is connected to the abnormal area identification unit 1435 and the abnormal device identification unit 1440. In response to the identification of an abnormality in one device 10 by at least one of the abnormal area identification unit 1435 or the abnormal device identification unit 1440, the machine control unit 1450 performs control for dealing with the abnormality of the one device 10 on at least one of the one device 10 or other devices 10 in the facility 1.

[0142] The robot command unit 1460 is connected to the collected information database connection unit 620, the robot database connection unit 640, the abnormal area identification unit 1435, and the abnormal device identification unit 1440. The robot command unit 1460 determines the operations to be performed by each robot 30 and instructs each robot 30 to perform the determined operations.

[0143] The monitoring processing unit 1400 shown above has each component shown in FIG. 14 in order to realize each function such as identification of an abnormal area, identification of the device 10 in which an abnormality has occurred, instruction of control for dealing with the abnormality of the device 10, and instruction of operations for each robot 30. Alternatively, the monitoring processing unit 1400 may adopt a configuration that does not have some of the components shown in FIG. 14.

[0144] For example, the monitoring processing unit 1400 may not have the device control unit 1450 and may not give an instruction for control to handle an abnormality of the device 10. Further, the monitoring processing unit 1400 may not have the robot command unit 1460 and may not give an instruction for an operation to each robot 30. Further, the monitoring processing unit 1400 may not have the abnormality area specifying unit 1435 and may not specify an abnormality area in the facility 1 by the abnormality area specifying unit 1435. Further, the monitoring processing unit 1400 may not have the abnormal device specifying unit 1440 and may not specify the device 10 in which an abnormality has occurred by the abnormal device specifying unit 1440. Even when the monitoring device 40 does not have some functions as exemplified herein, the monitoring device 40 can cause at least one device 10 in the facility 1 to take a technical measure to be taken in response to an abnormality in the facility 1 by the user receiving the information output or displayed by the monitoring device 40 and discovering the abnormality in the abnormality area or the device 10.

[0145] According to the monitoring device 40 according to the modification example shown above, it is possible to specify an abnormality area in which an abnormality may exist in the facility 1 by using the acoustic data detected at each of a plurality of first positions in the facility 1. Thereby, the monitoring device 40 can narrow down an abnormality area in which an abnormality may exist from the entire facility 1 by using the robot 30 or the like that can move in the facility 1 that may be provided on a vast site in some cases and collect sound, and can limit the range to be investigated and shorten the time required to discover the abnormality. As a result, the monitoring device 40 can be made to promptly handle the abnormality.

[0146] Figures 15, 16, and 17 show the processing flow of the monitoring device 40 according to the first modification. For the sake of convenience of explanation, the processing flows in FIGS. 15 to 17 show a case where the monitoring device 40 performs patrol monitoring in the facility 1 using one robot 30 (also referred to as the "first robot 30"), and when the first robot 30 detects an abnormal sound, it identifies the abnormal area and the equipment 10 where the abnormality occurred. The monitoring device 40 may cause each of the plurality of robots 30 to perform patrol monitoring in the facility 1 in parallel as the first robot 30 in the processing flows of FIGS. 15 to 17, and perform the processing shown in the processing flows of FIGS. 15 to 17 in response to any of the first robots 30 detecting an abnormal sound.

[0147] In S1500, the monitoring processing unit 1400 performs monitoring processing S1500 to S1530 to cause the first robot 30 to be processed to patrol in the facility 1. The robot command unit 1460 instructs the first robot 30 to be processed to move in the facility 1 via the instruction transmission unit 612. The robot command unit 1460 may perform the same processing as S1100 in FIG. 11. As a result, the first robot 30 to be processed moves to the observation point.

[0148] In S1510, the first robot 30 observes the state of the observation point in the facility 1 in the same manner as S1110 in FIG. 11. As a result, the acoustic data acquisition unit 602 acquires the first acoustic data detected by the robot 30 at the observation point (first position) in the facility 1. The acoustic data acquisition unit 602 may acquire the first acoustic data associated with the position data indicating the position of the robot 30 based on a sensor 410a such as GPS from the robot 30. The collection information database connection unit 620 records the first acoustic data as collection data in the collection information database 50.

[0149] In S1520, the abnormal sound detection unit 1430 detects the degree of abnormality of the sound at the first position (also referred to as the "first degree of abnormality") using the first acoustic data detected at the first position. The abnormal sound detection unit 1430 may detect the degree of abnormality of the sound at the first position in the same manner as S1200 in FIG. 12. In this case, the abnormal sound detection unit 1430 may use, as the normal acoustic data at the first position, one or more records of acoustic data collected at the first position among the past acoustic data recorded in the collection information database 50, or one or more records of acoustic data collected within a predetermined range near the first position. The abnormal sound detection unit 1430 may use, as the normal acoustic data, only the acoustic data collected by a robot 30 of the same model as the first robot 30, or only the acoustic data collected by the same individual as the first robot 30. Further, the abnormal sound detection unit 1430 may use, as the normal acoustic data, only the acoustic data collected in the same direction as the direction of sound collection by the first robot 30, reflecting the directivity of the acoustic sensor 400a.

[0150] Since the frequency of occurrence of abnormalities in individual areas within the facility 1 is low, the acoustic data collected from past history can be regarded as almost normal acoustic data as a whole. In addition, depending on the detection result of the abnormality by the abnormal area specifying unit 1435 or the abnormal device specifying unit 1440, or by the user of the monitoring device 40, if annotations such as normal or abnormal are added to each acoustic data in each record of the collection data in the collection information database 50, the abnormal sound detection unit 1430 may use, as the normal acoustic data, only the acoustic data with an annotation indicating that it is normal. When detecting the degree of abnormality of the sound at each of a plurality of first positions using AI technology, the abnormal sound detection unit 1430 may train the acoustic data for each first position using a machine learning model for each first position.

[0151] The monitoring device 40 may record acoustic data or a machine learning model, etc. for each position in a position database, and may include a position database connection unit connected to the position database and the monitoring processing unit 1400. In this case, the monitoring device 40 may record the collected data collected at each position in the facility 1 in the position database as the collected data for each position, and may refer to it as the collected data for each position.

[0152] In S1530, the abnormal area specifying unit 1435 determines whether there is an abnormality in the sound at the first position which is the observation point of the first robot 30. For example, the abnormal area specifying unit 1435 may determine that the sound at the first position where the first robot 30 is located is abnormal in response to the abnormality degree detected using the first acoustic data collected by the first robot 30 exceeding the threshold value.

[0153] In response to determining that the sound at the first position to be processed is not abnormal (NO in S1530), the monitoring processing unit 1400 advances the process to S1500 to move the first robot 30 to the next observation point, and may perform the processes of S1510 and S1520. Here, the first robot 30 may observe the state inside the facility 1 at each of a plurality of observation points separated from each other by repeatedly executing the processes from S1500 to S1530. Alternatively, the first robot 30 may observe the state inside the facility 1 with each continuous point during movement as an observation point while gradually advancing along the route specified by the monitoring device 40 by repeatedly executing the processes from S1500 to S1530. In response to determining that the sound at the first position to be processed is abnormal, the monitoring processing unit 1400 advances the process to S1540.

[0154] In S1540, in response to determining that the sound at the first position of the first robot 30 is abnormal, the monitoring processing unit 1400 performs specific processing S1540 to S1670 of an abnormal area where an abnormality may exist around the first position of the first robot 30. To specify the abnormal area, the robot command unit 1460 determines and instructs the actions of one or more second robots 30 other than the first robot 30 among the plurality of robots 30. The robot command unit 1460 may instruct one or more second robots 30 (one or more second robots 30 located within a predetermined linear distance or moving distance from the first robot 30, or a predetermined number of second robots 30 closest to the first robot 30, etc.) that are relatively close to the first robot 30 among the plurality of robots 30 to perform actions for specifying the abnormal area.

[0155] For example, the robot command unit 1460 may instruct one or more second robots 30 to approach the first robot 30 to collect sound. Further, in response to the abnormality degree detected using the first acoustic data collected by the first robot 30 exceeding a threshold value, the robot command unit 1460 may instruct one or more second robots 30 to move within a predetermined range from the first robot 30 to collect sound. As a result, one or more second robots 30 move to the designated observation points respectively.

[0156] As described above, the monitoring device 40 may cause each of the plurality of robots 30 to perform patrol monitoring in the facility 1 in parallel as the first robot 30 in the processing flows of FIGS. 15 to 17. In this case, in response to detection of an abnormal sound at the position of any one of the first robots 30, the monitoring device 40 may temporarily assign the role of the second robot 30 to one or more other first robots 30 located in the vicinity of the first robot 30. The other first robots 30 assigned the role of the second robot 30 may interrupt the assigned patrol monitoring and perform the task of identifying the abnormal area and the device 10 where the abnormality occurred until the device 10 where the abnormality occurred is identified by this processing flow.

[0157] In S1610, the second robot 30 and the monitoring device 40 observe the state of the observation points in the facility 1 in the same manner as in S1510 of FIG. 15 and record them in the collection information database 50 as collection data. Here, one or more second robots 30 may observe the state in the facility 1 at each of one or more observation points separated from each other, or may observe the state in the facility 1 using each consecutive point during movement as an observation point. The collection information database connection unit 620 can record these first acoustic data in the collection information database 50 as collection data. By the processes of S1510 and S1610, the acoustic data acquisition unit 602 can acquire the first acoustic data detected by the first and second robots 30 at each of the plurality of first positions in the facility 1.

[0158] In S1620, the abnormal sound detection unit 1430 detects the abnormality degree of the sound at each first position (also referred to as the "second abnormality degree") using the first acoustic data detected at the first position of each second robot 30. The abnormal sound detection unit 1430 may detect the abnormality degree of the sound at the first position of each second robot 30 in the same manner as in S1520. Through the processes of S1520 and S1620, the abnormal sound detection unit 1430 can detect the abnormality degree of the sound at each first position using the first acoustic data detected at each of a plurality of first positions including the first position of the first robot 30 and the first positions of each second robot 30.

[0159] In S1630, the abnormal area identification unit 1435 identifies an abnormal area in the facility 1 where an abnormality may exist based on the abnormality degree of the sound detected at each of the plurality of first positions. If the abnormal area identification unit 1435 can narrow down the device 10 (also referred to as the "abnormal device 10") in which the abnormality has occurred within the abnormal area, it may identify the device 10 in which the abnormality has occurred. The method for identifying the abnormal area and the abnormal device 10 by the abnormal area identification unit 1435 will be described later with reference to FIGS. 18, 19, and 21.

[0160] If the abnormal area can be identified (YES in S1640), the monitoring processing unit 1400 advances the process to S1700 and searches for the abnormal device 10 within the abnormal area. If the abnormal device 10 can be identified (YES in S1650), the monitoring processing unit 1400 advances the process to S1740. If neither the abnormal area nor the abnormal device 10 can be identified (NO in S1640 and S1650), the monitoring processing unit 1400 advances the process to S1660.

[0161] In S1660, the robot command unit 1460 compares the first abnormality degree, which is the degree of abnormality of the sound at the position of the first robot 30, with each second abnormality degree, which is the degree of abnormality of the sound at the position of each second robot 30. When the first abnormality degree is equal to or greater than each second abnormality degree, that is, when the sound at the position of the first robot 30 is more likely to be abnormal or equivalent to the sound at the position of each second robot 30, the robot command unit 1460 proceeds with the process to S1540, and moves each second robot 30 without moving the first robot 30 to narrow down the abnormal area.

[0162] When any second abnormality degree is greater than the first abnormality degree, that is, when the sound at the position of any second robot 30 is more likely to be abnormal than the sound at the position of the first robot 30, the robot command unit 1460 swaps the first robot 30 and the second robot 30 and proceeds with the process to S1540. Thereby, the robot command unit 1460 sets the robot 30 at the first position where the degree of abnormality of the sound is the maximum as the first robot 30, and moves one or two or more other second robots 30 around the first robot 30 to narrow down the abnormal area. Instead of this, the robot command unit 1460 may move the first and each second robot 30 regardless of the degree of abnormality of the sound at each first position.

[0163] In this way, in response to the detection of an abnormal sound at the position of the first robot 30, the monitoring processing unit 1400 brings one or two or more second robots 30 closer to the first robot 30, or moves them within a predetermined range from the first robot 30, and detects the presence or absence of an abnormal sound at the position of each second robot 30. Thereby, the monitoring processing unit 1400 can narrow down and specify the abnormal area where an abnormality may exist in the facility 1. Also, when it is known that there is only one device 10 in the specified abnormal area, the monitoring processing unit 1400 can specify the only device 10 in the abnormal area as the abnormal device 10.

[0164] When an abnormal area is identified in S1640 (YES in S1640), the monitoring processing unit 1400 performs processing S1700 to S1730 for searching for an abnormal device 10 in the abnormal area. In S1700, the robot command unit 1460 instructs each of the first robot 30 and each of the second robots 30 to move to each of a plurality of positions (also referred to as "second positions") in the identified abnormal area. Here, the robot command unit 1460 may instruct at least one robot 30 to move to each second position in the abnormal area after all of the robots 30 involved in the monitoring processing S1500 to S1530 and the abnormal area identification processing S1540 to S1670. The robot command unit 1460 may cause at least one robot 30 to patrol two or more second positions.

[0165] In S1710, each robot 30 at each second position observes the state of each second position in the identified abnormal area in the same manner as in S1510 of Fig. 15. As a result, the acoustic data acquisition unit 602 acquires second acoustic data detected at each of the multiple second positions in the identified abnormal area. The acoustic data acquisition unit 602 may acquire, from each robot 30, second acoustic data associated with position data indicating the position of the robot 30 based on the sensor 410a such as a GPS. The collected information database connection unit 620 records the second acoustic data in the collected information database 50 as collected data.

[0166] In S1720, the sound abnormality detection unit 1430 detects the abnormality degree of the sound at each of the second positions by using the second acoustic data detected at each of the second positions. The sound abnormality detection unit 1430 may detect the abnormality degree of the sound at the second position of each robot 30 in the same manner as in S1520.

[0167] In S1730, the abnormal device identifying unit 1440 identifies the device 10 in which an abnormality has occurred within the facility 1, based on the abnormality levels of the sounds detected at each of the multiple second positions. A method for the abnormal device identifying unit 1440 to identify a device in which an abnormality has occurred will be described later with reference to FIG.

[0168] In S1740, the abnormal area specifying unit 1435 and the abnormal device specifying unit 1440 may cause the display device 690 to display information regarding the abnormal area and the device 10 in which the abnormality has occurred. The abnormal area specifying unit 1435 causes the display processing unit 665 to display on the display device 690 a display screen that displays the position of the abnormal area within the facility 1, the first positions of the first robot 30 and each second robot 30 at the timing when the abnormal area was specified, the state detected at each of the first positions at the timing when the abnormal area was specified, or other information. The abnormal device specifying unit 1440 causes the display processing unit 665 to display on the display device 690 a display screen that displays the second positions of the respective robots 30 at the timing when the device 10 in which the abnormality has occurred within the abnormal area was specified, the state detected at each of the second positions at the timing when the device 10 in which the abnormality has occurred was specified, the state of the device 10 in which the abnormality has occurred, or other information.

[0169] In S1750, in response to the identification of an abnormality in one device 10, the device control unit 1450 performs control for dealing with the abnormality of the one device 10 on at least one of the one device 10 or other devices 10 within the facility 1. The device control unit 1450 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670 in the same manner as the processing of the abnormal device specifying unit 1040 in S1240 of FIG. 12. The monitoring device 40 proceeds with the process to S1500 and continues the monitoring of the facility 1 by the robot 30.

[0170] According to the monitoring device 40 shown above, in response to the first robot 30 detecting an abnormal sound in the facility 1, one or two or more other second robots 30 are directed to the vicinity of the position where the abnormal sound is detected, and the degree of abnormal sound can be detected at a plurality of first positions. Then, the monitoring device 40 can identify an abnormal area where an abnormality may exist in the facility 1 based on the degree of abnormal sound at each of the plurality of first positions. Furthermore, the monitoring device 40 can identify the device 10 in which an abnormality exists in the abnormal area by searching the abnormal area using a plurality of robots 30.

[0171] In this way, the monitoring device 40 can identify the abnormal area and the abnormal device 10 that are difficult to detect only by detecting an abnormal sound from only one observation point by using the sound observation results from a plurality of observation points. In addition, the monitoring device 40 can discover the abnormal area and the abnormal device 10 more quickly by causing a plurality of robots 30 to operate in cooperation. When there is a shortage of robots 30, or when no other robot 30 is located near the first robot 30, etc., the monitoring device 40 may perform the processes from FIGS. 15 to 17 using one robot 30. In this case, the monitoring device 40 may cause one robot 30 to acquire acoustic data at each of the plurality of first positions and the plurality of second positions.

[0172] Note that the monitoring device 40 according to this modification example may have the functions and configurations of the monitoring processing unit 650 shown in FIGS. 10 to 13. For example, the monitoring device 40 executes the monitoring process in the facility 1 according to the operation flow shown in S1500 to S1530 of FIG. 15, executes the identification process of the abnormal area according to the operation flow shown in S1540 to S1670 of FIG. 15, and in the abnormal area, the device 10 in which an abnormality has occurred may be identified using acoustic data and image data in the same manner as the operation flow shown in FIGS. 11 to 12.

[0173] FIG. 18 shows an example of a method for specifying an abnormal area by the monitoring device 40 according to the first modification. In the example of this figure, during the monitoring process by the robot A operating as the first robot 30 (S1500 to S1530 in FIG. 15), the sound abnormality detection unit 1430 in the monitoring device 40 detects that the sound abnormality degree is 0.8 at the position of the robot A. In the example of this figure, the abnormal area specifying unit 1435 determines that the sound at the position of the robot A is abnormal in response to the sound abnormality degree at the position of the robot A exceeding the threshold value (0.5 in the example of this figure).

[0174] In response to the determination that the sound at the position of the robot A as the first robot 30 is abnormal (YES in S1530 of FIG. 15), the monitoring device 40 performs the abnormal area specifying process according to the operation flow shown from S1540 in FIG. 15 to S1670 in FIG. 16. In the example of this figure, the monitoring device 40 performs the abnormal area specifying process using the robot B and the robot C operating as the second robot 30 in addition to the robot A.

[0175] In the example of this figure, the sound abnormality detection unit 1430 in the monitoring device 40 detects that the sound abnormality degree is 0.1 at the position of the robot B and that the sound abnormality degree is 0.2 at the position of the robot C (S1620 in FIG. 16). The abnormal area specifying unit 1435 determines that the sounds at the positions of the robot B and the robot C are normal because the sound abnormality degrees at the positions of the robot B and the robot C are equal to or less than the threshold value (0.5).

[0176] In the example of this figure, the monitoring device 40 may specify the abnormal area based on these two or more positions in response to the detection of sound abnormalities at each of the two or more positions (S1630 in FIG. 16). In the state shown in this figure, since the sound abnormality is detected only at the position of the robot A, the monitoring device 40 proceeds with the process to S1540 without specifying the abnormal area (NO in S1640 and S1650, YES in S1660), and determines the actions of the robot B and the robot C (S1540).

[0177] In the example of this figure, the robot command unit 1460 of the monitoring device 40 may instruct robots B and C to approach robot A for sound collection. Here, the robot command unit 1460 may move robots B and C straight toward robot A, may move them straight in the direction of approaching robot A, or may gradually approach robot A while making them change their directions left and right according to a random number or a pre-programmed algorithm.

[0178] Also, the robot command unit 1460 of the monitoring device 40 may instruct robots B and C to move within a predetermined range from robot A for sound collection. In the example of this figure, the robot command unit 1460 may move robots B and C to range A which is a range within a circle with a predetermined radius centered on the position of robot A. The robot command unit 1460 may direct robots B and C straight toward robot A or gradually approach it as described above.

[0179] The robot command unit 1460 may adjust the size of range A as a predetermined range according to the magnitude of the sound component indicating an abnormality in the first acoustic data collected by the first robot 30, i.e., robot A. For example, the robot command unit 1460 may, in the same manner as S1200 in FIG. 12, adjust the amplitude of the normal acoustic data to calculate the difference from the target acoustic data as the sound component indicating an abnormality, and adjust the size of range A according to the magnitude of the calculated difference. Since the larger the sound component indicating an abnormality, the closer robot A is to the abnormal location, the robot command unit 1460 may make range A smaller as the sound component indicating an abnormality is larger, and make range A larger as the sound component indicating an abnormality is smaller.

[0180] Further, the robot command unit 1460 may adjust a range A as a predetermined range according to the sensitivity or directivity of the acoustic sensor 400a used by the robot A, which is the first robot 30, for sound collection. When the sensitivity of the acoustic sensor 400a is higher, the robot command unit 1460 may make the range A larger, and when the sensitivity of the acoustic sensor 400a is lower, the robot command unit 1460 may make the range A smaller. Also, in a direction where the directivity of the acoustic sensor 400a is higher, the robot command unit 1460 may widen the range A, and in a direction where the directivity of the acoustic sensor 400a is lower, the robot command unit 1460 may narrow the range A.

[0181] Further, the robot command unit 1460 may adjust the range A according to the abnormality degree of the sound detected at the position of the robot A. Since the higher the abnormality degree of the sound, the more likely it is to accurately detect the abnormality of the sound, the higher the possibility that the abnormal location is closer when the abnormality degree of the sound is higher. Therefore, the robot command unit 1460 may make the range A smaller as the abnormality degree of the sound is higher, and make the range A larger as the abnormality degree of the sound is lower.

[0182] FIG. 19 shows an example of a method for specifying an abnormal area by the monitoring device 40 according to the first modification. In the example of this figure, the robots B and C are in a state closer to the robot A than in the state of FIG. 18. The sound abnormality detection unit 1430 in the monitoring device 40 has detected that the abnormality degree of the sound is 0.2 at the position after the movement of the robot B and the abnormality degree of the sound is 0.7 at the position after the movement of the robot C (S1620 in FIG. 16). Since the abnormality degree of the sound at the position of the robot B is equal to or less than the threshold value (0.5), the abnormality area specifying unit 1435 determines that the sound at the position of the robot B is normal. Also, since the abnormality degree of the sound at the position of the robot C exceeds the threshold value (0.5), the abnormality area specifying unit 1435 determines that the sound at the position of the robot C is abnormal.

[0183] The abnormal area specifying unit 1435 specifies an abnormal area where an abnormal location may exist within Facility 1, using the distances for each of two or more first positions among the plurality of first positions, where the degree of abnormality detected using the first acoustic data exceeds a threshold value. In the example of this figure, since abnormal sounds are detected at each of two or more first positions (the positions of Robot A and Robot C), the abnormal area specifying unit 1435 specifies the abnormal area based on these two or more positions (S1630 in FIG. 16). The abnormal area specifying unit 1435 specifies, as the abnormal area, an area where the distance to the position of Robot A is within a predetermined range and the distance to the position of Robot C is within a predetermined range. That is, the abnormal area specifying unit 1435 specifies, as the abnormal area, an area where range A where the distance to the position of Robot A is within a predetermined range and range B where the distance to the position of Robot C is within a predetermined range overlap (the area including Tank A and Tank B in the figure). Note that when abnormal sounds are detected at each of three or more first positions, the abnormal area specifying unit 1435 may specify the abnormal area using the distances for each of the three or more first positions.

[0184] In the example of this figure, the predetermined range in which the robot command unit 1460 moves Robots B and C in response to Robot A detecting an abnormal sound and the predetermined range used as the threshold value for the distance to the position of Robot A in specifying the abnormal area are the same range A, but these ranges may be different.

[0185] Also, the predetermined range used as the threshold value of the distance to the first position of each robot 30 may be the same, or may be different for each robot 30. The abnormal area specifying unit 1435 may adjust the distance ranges for each of the two or more first positions according to the sensitivity of the acoustic sensor 400a that performs sound collection, the directivity of the acoustic sensor 400a, or the degree of abnormality of the sound. For example, when the sensitivity of the acoustic sensor 400a is higher, the abnormal area specifying unit 1435 may increase the range (range A for robot A and range B for robot C), and when the sensitivity of the acoustic sensor 400a is lower, the abnormal area specifying unit 1435 may decrease the range. Also, the abnormal area specifying unit 1435 may widen the range in the direction where the directivity of the acoustic sensor 400a is higher and narrow the range in the direction where the directivity of the acoustic sensor 400a is lower. Further, the abnormal area specifying unit 1435 may decrease the range as the degree of abnormality of the sound increases and increase the range as the degree of abnormality of the sound decreases.

[0186] According to the monitoring device 40 described above, since the abnormal area is specified using the distances to two or more positions where an abnormal sound has been detected, the area where an abnormality may exist can be narrowed down to a smaller range. Also, when adjusting the range for moving another robot 30 or the range specified as the abnormal area according to at least one of the sensitivity of the acoustic sensor 400a that performs sound collection, the directivity of the sensor 400a, or the degree of abnormality of the sound, the monitoring device 40 can further appropriately narrow down the range for investigating the abnormality according to the sound collection characteristics of the robot 30 or the like. As a result, the monitoring device 40 can limit the range for investigating the abnormality and shorten the time required to discover the abnormality. As a result, the monitoring device 40 can promptly take action against the abnormality.

[0187] FIG. 20 shows an example of a method for identifying the abnormal device 10 by the monitoring device 40 according to the first modification. In response to the area where the ranges A and B in FIG. 19 overlap being identified as the abnormal area, the robot command unit 1460 instructs each of robots A to C to move to each of a plurality of second positions within the abnormal area (S1700 in FIG. 17). This figure shows the state where robots A to C have moved into the abnormal area. The abnormal device identification unit 1440 identifies the device 10 in which an abnormality has occurred in the facility 1 based on the abnormality degree of the sound detected at each of the plurality of second positions (S1730 in FIG. 17).

[0188] Here, the monitoring device 40 may cause robots A to C to perform patrol monitoring within the abnormal area to identify the device 10 in which an abnormality has occurred within the abnormal area. For example, the monitoring device 40 causes the other robots 30 to approach the position with the maximum sound abnormality degree among the positions of each robot 30 within the abnormal area in the same manner as from S1540 to S1670 in FIG. 15, so as to gather at the position with the maximum sound abnormality degree within the abnormal area. Thereby, the monitoring device 40 can identify the device 10 at that position as the device 10 in which an abnormality has occurred by identifying the position with the maximum sound abnormality degree within the abnormal area.

[0189] Further, the monitoring device 40 may use robots A to C to individually observe each device 10 within the abnormal area, thereby identifying the device 10 that is generating abnormal sound among the devices 10. Further, the monitoring device 40 may have the functions and configurations of the monitoring processing unit 650 shown in FIGS. 10 to 13, and execute the monitoring process within the abnormal area according to the operation flow shown from S1500 to S1530 in FIG. 15 to identify the device 10 in which an abnormality has occurred.

[0190] FIG. 21 shows an example of a method for identifying the abnormal device 10 by the monitoring device 40 according to the first modification. The monitoring device 40 may use the method for identifying the abnormal device 10 shown in this figure in S1630 of FIG. 16. In the method for identifying the abnormal device 10 according to this modification, the abnormal area specifying unit 1435 specifies the device 10 in which an abnormality has occurred in the facility 1 based on the direction of the sound component indicating an abnormality at each of a plurality of first positions.

[0191] In this modification, the sound abnormality detection unit 1430 extracts at least one sound component whose sound source directions viewed from each of the plurality of first positions are different from each other by performing sound source separation or the like on the first acoustic data detected at each of the plurality of first positions. The sound abnormality detection unit 1430 may perform sound source separation, sound source localization, etc. in the same manner as the method shown in relation to S1120 of FIG. 11. Note that the robot 30 may have two or more acoustic sensors 400a for use in sound source separation or the like, and may have an acoustic sensor array (such as an "array microphone" or a "microphone array") in which a plurality of acoustic sensors 400a are arranged.

[0192] The sound abnormality detection unit 1430 detects the degree of sound abnormality for each sound component in each direction detected at each of the first positions. The abnormal area specifying unit 1435 determines the presence or absence of an abnormality in each sound component detected at each of the first positions. Thereby, the abnormal area specifying unit 1435 can specify the direction of the sound component indicating an abnormality at each of the plurality of first positions where an abnormality in the sound component has been detected.

[0193] The abnormal area specifying unit 1435 specifies the device 10 that is emitting abnormal sounds by using the direction of the sound component indicating an abnormality viewed from each of the plurality of first positions where an abnormality in the sound component has been detected. In the example of this figure, the abnormal area specifying unit 1435 identifies the tank B, which is the device 10 provided at the position where the lines extending from each of the plurality of first positions (the positions of the robots A and B) toward the direction of the sound component indicating an abnormality intersect, as the device 10 in which an abnormality has occurred in the facility 1.

[0194] Thus, when the robot 30 capable of specifying the direction of the sound component can be used for the search process of the device 10 in which an abnormality has occurred, the abnormality area specifying unit 1435 can more quickly specify the device 10 in which the abnormality has occurred. Note that the abnormal device specifying unit 1440 may specify the device 10 in which an abnormality has occurred in the same manner as described above in S1730 of FIG. 17.

[0195] When two or more devices 10 overlap and exist in the direction of the sound, or when the resolution in the direction of the sound is low, etc., the abnormality area specifying unit 1435 may not be able to specify a single device 10 in which an abnormality has occurred. Therefore, the abnormality area specifying unit 1435 may specify an abnormality area based on the direction of the sound component indicating an abnormality at each of the plurality of first positions in the same manner as described above. In this case, the abnormality area specifying unit 1435 may specify, as the abnormality area, an area of a predetermined range including a position where lines extending from each of the plurality of first positions toward the direction of the sound component indicating an abnormality intersect.

[0196] FIG. 22 shows the configuration of the monitoring processing unit 2300 of the monitoring device 40 according to the second modification of the present embodiment. The monitoring device 40 shown with reference to FIGS. 22 to 24 is a modification of the monitoring device 40 shown with reference to FIGS. 1 to 13. The monitoring device 40 according to this modification includes a monitoring processing unit 2300 instead of the monitoring processing unit 650. Since the functions and configurations other than the monitoring processing unit 2300 in the monitoring device 40 according to this modification are basically the same as those of the monitoring device 40 according to FIGS. 1 to 13, the description will be omitted except for the following differences.

[0197] Similar to the monitoring processing unit 650, the monitoring processing unit 2300 is connected to the collection information database connection unit 620, the device database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670 of the monitoring device 40. The monitoring processing unit 2300 reads the robot data of each robot 30 via the robot database connection unit 640, and determines the monitoring actions in Facility 1 that each robot 30 should execute using the robot data. The monitoring processing unit 2300 instructs each robot 30 to perform each operation included in the determined monitoring actions via the instruction transmission unit 612 in the communication unit 600.

[0198] The monitoring processing unit 2300 may cause the display processing unit 665 to generate a display screen for displaying the status of each device 10 and the determination result of normal or abnormal, and display it on the display device 690. The monitoring processing unit 2300 may transmit the status of each device 10 and the determination result of normal or abnormal to the control system 20 via the communication unit 670.

[0199] When at least one device 10 is detected to be abnormal as a result of monitoring each device 10 using one or more robots 30, the monitoring processing unit 2300 may generate an instruction to perform control for dealing with the abnormality. The monitoring processing unit 2300 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0200] The monitoring processing unit 2300 according to this modification example has a function of identifying abnormal devices that may have abnormalities in Facility 1 using the acoustic data and LiDAR data collected in Facility 1. The monitoring processing unit 2300 includes an acoustic anomaly detection unit 2330, an abnormal device identification unit 2340, a device control unit 2350, and a robot command unit 2360.

[0201] The device detection unit 2310 is connected to the collected information database connection unit 620. The device detection unit 2310 reads out the collected data in which LiDAR data is recorded as measurement data among the collected data registered in the collected information database 50 via the collected information database connection unit 620. Thereby, the device detection unit 2310 acquires the LiDAR data, the date and time when the LiDAR data was acquired, and the position and direction where the LiDAR data was observed within the facility 1. The device detection unit 2310 detects at least one device 10 using the LiDAR data. When the device 10 measured in the LiDAR data cannot be specified, the device detection unit 2310 may receive the designation of the corresponding device 10 or the like by receiving the input of information from the user via the input processing unit 660.

[0202] The abnormal sound detection unit 2330 is connected to the collected information database connection unit 620. The abnormal sound detection unit 2330 reads out the collected data in which acoustic data is recorded as measurement data among the collected data registered in the collected information database 50 via the collected information database connection unit 620. Thereby, the abnormal sound detection unit 2330 acquires the acoustic data, the date and time when the acoustic data was acquired, and the position where the acoustic data was observed within the facility 1. The abnormal sound detection unit 2330 detects the degree of abnormality of the sound using the acquired acoustic data.

[0203] The abnormal device identification unit 2340 is connected to the device detection unit 2310 and the abnormal sound detection unit 2330. The abnormal device identification unit 2340 identifies the device 10 in which an abnormality has occurred among at least one device 10 based on the change in the degree of abnormality of the sound accompanying the movement of the robot 30.

[0204] The device control unit 2350 is connected to the abnormal device identification unit 2340. When the abnormality of one device 10 is identified by at least one of the abnormal device identification unit 2340, the device control unit 2350 performs control for dealing with the abnormality of the one device 10 on at least one of the one device 10 or other devices 10 within the facility 1.

[0205] The robot command unit 2360 is connected to the collected information database connection unit 620, the robot database connection unit 640, and the abnormal device identification unit 2340. The robot command unit 2360 determines the operations to be performed by each robot 30 and instructs each robot 30 to perform the determined operations.

[0206] As described above, the monitoring processing unit 2300 has each component shown in FIG. 22 in order to realize functions such as identification of the device 10 in which an abnormality has occurred, instruction of control for dealing with the abnormality of the device 10, and instruction of operations for each robot 30. Instead of this, the monitoring processing unit 2300 may adopt a configuration that does not have some of the components shown in FIG. 22.

[0207] For example, the monitoring processing unit 2300 may not have the device control unit 2350 and may not give an instruction for control to deal with the abnormality of the device 10. Also, the monitoring processing unit 2300 may not have the robot command unit 2360 and may not give an instruction for operations to each robot 30. Further, the monitoring processing unit 2300 may not have the abnormal device identification unit 2340 and may not identify the device 10 in which an abnormality has occurred by the abnormal device identification unit 2340. Even when the monitoring device 40 does not have some of the functions exemplified here, the user can receive the information output or displayed by the monitoring device 40, discover the abnormality of the device 10, and cause at least one device 10 in the facility 1 to take technical measures to be taken in response to the abnormality of the device 10.

[0208] According to the monitoring device 40 according to the modification example described above, by using the acoustic data and LiDAR data collected in the facility 1, the position of each object or device 10 discovered by the LiDAR data, and the change in the abnormal degree of the sound accompanying the movement of the robot 30, etc., it is possible to quickly identify the device 10 that emits abnormal sounds. Thereby, according to the monitoring device 40 according to this modification example, it is possible to quickly take measures against the abnormality of the device 10.

[0209] FIG. 23 shows the processing flow of the monitoring device 40 according to the second modification. For the sake of convenience of explanation, the processing flow in FIG. 23 shows the case where the monitoring device 40 performs monitoring within the facility 1 using one robot 30. The monitoring device 40 may execute the processing flow in FIG. 23 for each of the plurality of robots 30.

[0210] In S2400, the robot command unit 2360 instructs the robot 30 to be processed to move within the facility 1 via the instruction transmission unit 612. The robot command unit 2360 may perform the same processing as S1100 in FIG. 11. As a result, the robot 30 to be processed moves to the observation point.

[0211] In S2405, the robot 30 observes the state of the facility 1 and the state of the robot 30 using each sensor 400 at the observation point (see S500 in FIG. 5). The robot 30 transmits various measurement data indicating the observed state of the facility 1 and the state of the robot 30 to the monitoring device 40 (see S510 in FIG. 5). The acoustic data acquisition unit 602 in the monitoring device 40 acquires acoustic data from among the measurement data transmitted from the robot 30. The collection information database connection unit 620 may perform the same processing as S1110 in FIG. 11.

[0212] In S2410, the sound abnormality detection unit 2330 detects the degree of sound abnormality using the acoustic data at the observation point. Here, the degree of abnormality may be represented by a real number or an integer within a predetermined range such as from 0 to 1 or from 0 to 100%, or may be a binary value such as normal or abnormal. The sound abnormality detection unit 2330 may detect the degree of abnormality based on the result of comparing the acoustic data with the normal acoustic data associated with the position where the acoustic data was detected. The sound abnormality detection unit 2330 may access the collection information database 50 via the collection information database connection unit 620 and refer to the normal acoustic data. The sound abnormality detection unit 2330 may detect the degree of abnormality by the same processing as the sound abnormality detection unit 1030 in S1200 in FIG. 12 and the sound abnormality detection unit 1430 in S1520 in FIG. 15.

[0213] In S2415, the LiDAR data acquisition unit 606 in the monitoring device 40 acquires LiDAR data from among the measurement data transmitted from the robot 30. The collection information database connection unit 620 may add the position data acquired by the position data acquisition unit 608, the direction data acquired by the direction data acquisition unit 610, etc. to the LiDAR data acquired by the LiDAR data acquisition unit 606, and record it in the collection information database 50 as collection data. Here, the LiDAR data is a data set indicating the distances to objects (irradiation points) in each direction within a range such as one-dimensional (horizontal direction, etc.) or two-dimensional (horizontal and vertical directions, etc.) when viewed from the observation point, by irradiating light or a laser in each direction and detecting the reflected light. Note that the LiDAR data may be data obtained by the robot 30 while changing the direction of the LiDAR sensor 400c at the observation point, targeting a wide area around the observation point.

[0214] In S2420, the device detection unit 2310 detects one or more devices 10 based on the LiDAR data from the robot 30. The device detection unit 2310 obtains the shape of each object within the irradiation range of light or laser or the distance to each irradiation point on the surface of each object based on the LiDAR data. The device detection unit 2310 may confirm the presence of one or more objects within the observation range of the LiDAR sensor 400c and calculate the position of each object based on the observation location where the LiDAR data was observed, the direction of each irradiation point, and the distance to each irradiation point. The device detection unit 2310 may use the device database 60 to obtain the identification information of the device 10 that should be present at the position of the object confirmed by the LiDAR data. For example, the device detection unit 2310 may search the device database 60 for a record of the device 10 recorded as being present at the position of the object confirmed by the LiDAR data (or within a predetermined error range). Thereafter, the device detection unit may obtain the identification information of the searched device 10 from the record of the device 10. In addition, when the shape of the object can be determined to some extent from the LiDAR data, the device detection unit 2310 may search the device database 60 for a record of the device 10 having the same shape as the shape of the object confirmed by the LiDAR data or a shape similar with a similarity equal to or higher than a threshold value, and obtain the identification information of the device 10. Note that the device detection unit 2310 may specify the corresponding device 10 using both the position and the shape of the object.

[0215] For any object confirmed by LiDAR data, if the device 10 that should be present at its position cannot be identified, the input processing unit 660 may accept input of information from the user to identify the device 10. Here, if the device 10 is unregistered and its presence in Facility 1 has not been confirmed, the input processing unit 660 accepts input of information about at least a part of the device data regarding the unregistered device 10, and the device detection unit 2310 may register the device data of the device 10 corresponding to the received information in the device database 60. Also, when the device 10 that should be present at the position of the object confirmed by LiDAR data is unregistered, the input processing unit 660 may accept input of information indicating whether to register the device 10 in the device database 60. The device detection unit 2310 may register the device data of the device 10 corresponding to the shape, position, or other already known information of the device 10 confirmed from the LiDAR data in the device database 60.

[0216] In S2425, the monitoring device 40 determines whether the abnormality degree of the sound at the current observation point has increased compared to the abnormality degree of the sound at the previous observation point. If the abnormality degree of the sound does not increase (''N'' in S2425), the monitoring device 40 advances the process to S2430. The monitoring device 40 may advance the process to S2430 when the abnormality degree of the sound has decreased. If the abnormality degree of the sound has increased (''Y'' in S2425), the monitoring device 40 advances the process to S2435.

[0217] In S2430, the robot command unit 2360 instructs the robot 30 on the traveling direction based on the change in the degree of abnormality of the sound accompanying the movement of the robot 30. Here, when the degree of abnormality of the sound does not increase as the robot 30 moves (\"N\" in S2425), it means that the robot 30 is not approaching the device 10 that emits abnormal sounds. Therefore, in S2430, the robot command unit 2360 instructs the robot 30 to change its traveling direction. The robot command unit 2360 may instruct the robot 30 to travel toward the device 10 detected from the LiDAR data. In this case, the robot command unit 2360 may select any one of the one or more devices 10 detected from the LiDAR data and instruct the robot 30 to travel toward that device 10. If the degree of abnormality of the sound does not increase as a result of selecting an existing device 10 and advancing the robot 30 toward that device 10, the robot command unit 2360 may remove that device 10 from the selection target. Instead of instructing the robot 30 to travel straight toward the selected device 10, the robot command unit 2360 may determine the traveling direction of the robot 30 to travel somewhat randomly toward the selected device 10.

[0218] Further, the robot command unit 2360 may determine the traveling direction in which the robot 30 should travel according to the positional relationship of the one or more devices 10 detected from the LiDAR data. For example, the robot command unit 2360 may instruct the robot 30 to travel in the direction in which more devices 10 exist according to the distribution of the devices 10 seen from the observation point.

[0219] The robot command unit 2360 may determine a new traveling direction based on the position of the observation points and the degree of abnormality of the sound up to now. For example, if there is an observation point where the degree of abnormality of the sound detected is greater than the degree of abnormality of the sound at the current observation point, the robot command unit 2360 may instruct the robot 30 to return to that observation point.

[0220] In S2435, the robot command unit 2360 instructs the robot 30 to maintain its traveling direction. Here, when the degree of sound abnormality increases as the robot 30 moves ( "Y" in S2425), it means that the robot 30 is approaching the device 10 that emits abnormal sounds. Therefore, in S2435, the robot command unit 2360 causes the robot 30 to maintain its traveling direction in order to approach the device 10 that emits abnormal sounds. Note that the robot command unit 2360 may change the direction of the robot 30 to some extent randomly while generally maintaining the traveling direction of the robot 30.

[0221] In S2440, in response to the increase in the degree of sound abnormality as the robot 30 moves in the traveling direction, the abnormal device identification unit 2340 identifies, as the device 10 where an abnormality has occurred, the device 10 located on the traveling direction side among at least one device 10 confirmed by the LiDAR data in S2420. For the device 10 registered in the device database 60, the abnormal device identification unit 2340 may use the data stored in the device database 60 instead of the LiDAR data to identify the device 10 existing in the traveling direction. For example, the abnormal device identification unit 2340 may search the device database 60 for the record of the device 10 recorded as existing on the current traveling direction of the robot 30 from the current observation point.

[0222] In S2445, the monitoring device 40 determines whether it can identify the abnormal device. For example, when the degree of sound abnormality increases as the robot 30 moves, if there is only one device 10 located on the advancing direction side (that is, the device 10 approaching the robot 30 as the robot 30 moves), the monitoring device 40 can identify the abnormal device 10. On the contrary, for example, when the degree of sound abnormality increases as the robot 30 moves and there is no device 10 located on the advancing direction side, the monitoring device 40 cannot identify the abnormal device. Also, when there are multiple devices 10 located on the advancing direction side, the monitoring device 40 cannot identify which device 10 is the abnormal device 10. If the monitoring device 40 can identify the abnormal device 10 as a result of such determination (\"Y\" in S2445), the process proceeds to S2450. If the monitoring device 40 cannot identify the abnormal device 10 (\"N\" in S2445), the process proceeds to S2405.

[0223] In S2450, in response to the identification of the abnormality of one device 10, the device control unit 2350 performs control for dealing with the abnormality of the one device 10 on at least one of the one device 10 or other devices 10 in the facility 1. The device control unit 2350 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670 in the same manner as the processing of the abnormal device identification unit 1040 in S1240 of FIG. 12. The monitoring device 40 proceeds with the process to S2400 and continues the monitoring of the facility 1 by the robot 30.

[0224] According to the monitoring device 40 shown above, by having the robot command unit 2360 that indicates the advancing direction to the robot 30 based on the change in the degree of sound abnormality accompanying the movement of the robot 30, even when the robot 30 does not have a directional microphone, the direction in which the device 10 that emits abnormal sound exists can be identified. Furthermore, the monitoring device 40 can identify the device 10 that emits abnormal sound from among the devices 10 confirmed in the LiDAR data based on the identified direction, and can promptly take measures against the device 10.

[0225] Note that the monitoring device 40 according to this modification example may have the functions and configurations of the monitoring processing unit 650 shown in FIGS. 10 to 13. For example, the monitoring device 40 executes the movement process of the robot 30 in the direction in which the sound abnormality degree increases according to the operation flow shown in S2400 to S2435 of FIG. 23, and may identify the abnormal device 10 using the acoustic data and the image data in the same manner as the operation flow shown in FIGS. 11 to 12 at the movement destination of the robot 30. After identifying the abnormal device 10 using the acoustic data and the image data in the same manner as the operation flow shown in FIGS. 11 to 12 at the movement destination of the robot 30, the monitoring device 40 may identify the abnormal device 10 using the acoustic data and the LiDAR data according to the operation flow shown in S2340 of FIG. 23.

[0226] Also, the monitoring device 40 according to this modification example may have the functions and configurations of the monitoring processing unit 1400 shown in FIGS. 14 to 21. For example, the monitoring device 40 executes the monitoring process in the facility 1 according to the operation flow shown in S1500 to S1530 of FIG. 15, executes the identification process of the abnormal area according to the operation flow shown in S1540 to S1670 of FIG. 15, and may identify the abnormal device 10 using the acoustic data and the LiDAR data in the same manner as the operation flow shown in FIG. 23 within the abnormal area.

[0227] FIG. 24 shows an example of a method for identifying an abnormal device 10 by the monitoring device 40 according to the second modification. In the example of this figure, the tank B is making abnormal noises. In the example of this figure, the robot 30 is shown in a state where it has moved to the position of "robot" in this figure (S2400 in FIG. 23). The robot 30 acquires acoustic data and LiDAR data at the position of "robot" in FIG. 24 (S2405 and 2415 in FIG. 23). The abnormal sound detection unit 2330 detects the degree of abnormality of the sound at the position of "robot" in FIG. 24 (S2410 in FIG. 23). The device detection unit 2310 detects a plurality of devices 10 such as tanks A to C and fans A to C from the LiDAR data from the robot 30 (S2420 in FIG. 23). The monitoring device 40 determines whether the degree of abnormality of the sound at the current observation point has increased compared to the degree of abnormality of the sound at the previous observation point (S2425 in FIG. 23). In the example of FIG. 24, it is assumed that the position of "robot" is the first observation point to which the flow of this figure is applied. In this case, since the degree of abnormality of the sound at the previous observation point has not been detected, the monitoring device 40 cannot determine the increase or decrease in the degree of abnormality of the sound. In this case, the robot command unit 2360 may randomly determine the device 10 that the robot 30 should head towards from among two or more devices 10 detected using the LiDAR data, or may maintain the current traveling direction of the robot 30 (S2430 in FIG. 23).

[0228] When the robot 30 is advanced in the traveling direction A (S2430 in FIG. 23), the robot 30 will approach the tank B which is the source of the abnormal sound. Therefore, as the robot 30 moves, the monitoring device 40 observes an increase in the degree of abnormality of the sound (S2425 in FIG. 23). In this case, the robot command unit 2360 may instruct the robot 30 to continue advancing in the traveling direction A (S2435 in FIG. 23). When the robot 30 gets close enough to the tank B, the monitoring device 40 can determine that only the tank B exists in the traveling direction in which the degree of abnormality of the sound increases. Thereby, the abnormal device identification unit 2340 can identify that the device 10 in which the abnormality has occurred is the tank B (S2440 in FIG. 23).

[0229] In the case of the positional relationship illustrated in this figure, tank A also exists on the extension line of the traveling direction A of the robot 30. In such a case, since the abnormality degree of the sound starts to decrease in response to the robot 30 advancing in the traveling direction A and passing tank B, the abnormal device identification unit 2340 may exclude tank A located on the side of the traveling direction A from the candidates of the device 10 where an abnormality has occurred, as viewed from that point. The robot command unit 2360 continues to search for the source of the abnormal sound by changing the traveling direction of the robot 30 to the direction toward any one of the remaining devices 10 as candidates, or to the direction opposite to the traveling direction A, etc. Alternatively, the abnormal device identification unit 2340 may determine that the robot 30 has come closest to the device 10 that emits the abnormal sound in response to the sound pressure or volume of the abnormal sound being equal to or greater than the threshold value, and identify the closest device 10 as the device 10 where an abnormality has occurred.

[0230] When the robot 30 advances from the illustrated position in a direction such as the traveling direction B (S2430 in FIG. 23), the robot 30 will eventually move away from tank B, which is the source of the abnormal sound. Therefore, as the robot 30 moves, the monitoring device 40 observes a decrease in the abnormality degree of the sound (S2425 in FIG. 23). Therefore, the robot command unit 2360 instructs the robot 30 to advance in a direction different from the traveling direction B (S2430 in FIG. 23). In this way, by repeating the operation of changing the traveling direction of the robot 30 when the abnormality degree of the sound does not increase, the monitoring device 40 can finally make the robot 30 advance in the direction in which the abnormality degree of the sound increases, that is, toward tank B, which is the source of the abnormal sound. As a result, the abnormal device identification unit 2340 can identify that the device 10 where an abnormality has occurred is tank B (S2440 in FIG. 23).

[0231] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of a device having a role of performing operations. Specific stages and sections may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable medium. The dedicated circuit may include digital and / or analog hardware circuits, including integrated circuits (ICs) and / or discrete circuits. The programmable circuit may include reconfigurable hardware circuits including memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0232] A computer-readable medium may include any tangible device capable of storing instructions executable by an appropriate device, such that a computer-readable medium having instructions stored therein will comprise a product including instructions executable to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (registered trademark) disc, memory stick, integrated circuit card, etc.

[0233] Computer-readable instructions may include source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming languages such as Smalltalk®, JAVA®, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages.

[0234] Computer-readable instructions may be provided to a processor or programmable circuit of a programmable data processing apparatus such as a general-purpose computer, a special-purpose computer, or other computer, locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc., and the computer-readable instructions may be executed to create means for performing the operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0235] FIG. 25 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 can cause the computer 2200 to function as an operation associated with the apparatus according to an embodiment of the present invention or as one or more sections of the apparatus, or to execute the operation or the one or more sections, and / or can cause the computer 2200 to execute a process according to an embodiment of the present invention or a stage of the process. Such a program may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0236] The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are mutually connected by a host controller 2210. The computer 2200 also includes an input / output unit such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0237] The CPU 2212 operates according to programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphic controller 2216 acquires image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or in itself, and causes the image data to be displayed on the display device 2218.

[0238] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads a program or data from the DVD-ROM 2201 and provides the program or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0239] The ROM 2230 stores therein a boot program or the like executed by the computer 2200 upon activation, and / or a program that depends on the hardware of the computer 2200. The input / output chip 2240 may also be connected to the input / output controller 2220 via various input / output units such as a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0240] The program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium, installed in the hard disk drive 2224, the RAM 2214, or the ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. The information processing described in these programs is read by the computer 2200, resulting in the cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be configured by realizing the operation or processing of information according to the use of the computer 2200.

[0241] For example, when communication is executed between the computer 2200 and an external device, the CPU 2212 may execute a communication program loaded in the RAM 2214 and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. The communication interface 2222 reads the transmission data stored in the transmission buffer processing area provided in a recording medium such as the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or the IC card under the control of the CPU 2212, transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer processing area provided on the recording medium, etc.

[0242] Further, the CPU 2212 may cause all or a necessary part of a file or database stored in an external recording medium such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and execute various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.

[0243] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording medium and may undergo information processing. The CPU 2212 may execute various types of processing on the data read from the RAM 2214, including various types of operations, information processing, condition judgment, conditional branch, unconditional branch, information search / replacement, etc. described throughout this disclosure and specified by the instruction sequence of the program, and write back the result to the RAM 2214. Further, the CPU 2212 may search for information in files, databases, etc. within the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2212 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored within the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0244] The programs or software modules described above may be stored in a computer-readable medium on or near the computer 2200. Also, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network.

[0245] As described above, the present invention has been described using embodiments. However, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0246] In the claims, the specification, and the drawings, for the operations, procedures, steps, stages, and other processes in the apparatus, system, program, and method shown, the execution order of each process, such as the operations, procedures, steps, and stages, is not explicitly stated as "earlier" or "preceding" etc., and it should be noted that, unless the output of the previous process is used in the subsequent process, it can be realized in any order. Regarding the operation flow in the claims, the specification, and the drawings, even if it is described for convenience using "first," "next," etc., it does not mean that it is essential to be implemented in this order.

Explanation of Reference Numerals

[0247] 1 Facility 10 Equipment 20 Control System 30 Robot 40 Monitoring Device 50 Collected Information Database 60 Equipment Database 70 Robot Database 210 State Acquisition Unit 220 State Determination Unit 230 Display Processing Unit 240 Display Device 250 Input Device 260 Instruction Input Unit 270 Control Unit 400a~c Sensor 410a~b Sensor 420 Actuator 430 State Acquisition Unit 440 Communication Unit 450 Control Unit 600 Communication Unit 602 Acoustic Data Acquisition Unit 604 Image data acquisition unit 606 LiDAR data acquisition unit 608 Position data acquisition unit 610 Direction data acquisition unit 612 Instruction transmission unit 620 Collection information database connection unit 630 Equipment database connection unit 640 Robot database connection unit 650 Monitoring processing unit 660 Input processing unit 665 Display processing unit 670 Communication unit 680 Input device 690 Display device 1000 Sound source separation unit 1010 Equipment detection unit 1020 Sound source location unit 1030 Abnormal sound detection unit 1040 Abnormal equipment identification unit 1050 Equipment control unit 1060 Robot command unit 1300 Image data 1305a~h Pipes 1310 Valve 1320 Reactor 1330 Flow meter 1400 Monitoring processing unit 1430 Abnormal sound detection unit 1435 Abnormal area identification unit 1440 Abnormal equipment identification unit 1450 Equipment control unit 1460 Robot command unit 2200 Computer 2201 DVD-ROM 2210 Host controller 2212 CPU 2214 RAM 2216 Graphics controller 2218 Display device 2220 Input / output controller 2222 Communication interface 2224 Hard Disk Drive 2226 DVD-ROM Drive 2230 ROM 2240 Input / Output Chip 2242 Keyboard 2300 Monitoring Processing Unit 2310 Device Detection Unit 2330 Abnormal Sound Detection Unit 2340 Abnormal Device Identification Unit 2350 Device Control Unit 2360 Robot Command Unit

Claims

1. An acoustic data acquisition unit that acquires acoustic data detected by an acoustic sensor of a robot capable of moving within a facility; An abnormal sound detection unit that detects the degree of abnormality of a sound using the acoustic data; A LiDAR data acquisition unit that acquires LiDAR data detected by a LiDAR sensor of the robot; A device detection unit that detects at least one device using the LiDAR data; An abnormal device identification unit that identifies a device in which an abnormality has occurred among the at least one device based on a change in the degree of abnormality accompanying the movement of the robot A device comprising.

2. The abnormal device identification unit according to claim 1, wherein, in response to an increase in the degree of abnormality as the robot moves in the traveling direction, the device located on the traveling direction side among the at least one device is identified as the device in which an abnormality has occurred.

3. The device according to claim 2, further comprising a robot command unit that instructs the robot of a traveling direction based on a change in the degree of abnormality accompanying the movement of the robot.

4. The robot command unit according to claim 3 instructs the robot to maintain the traveling direction of the robot in response to an increase in the degree of abnormality accompanying the movement of the robot, and to change the traveling direction of the robot in response to the degree of abnormality not increasing or decreasing as the robot moves.

5. The abnormal sound detection unit according to any one of claims 1 to 4 detects the degree of abnormality based on a result of comparing the acoustic data with normal acoustic data associated with the position where the acoustic data was detected.

6. The device detection unit according to claim 1 uses a device database that records device data including position and identification information for each of a plurality of devices in the facility to acquire identification information of a device that should exist at the position of an object confirmed by the LiDAR data.

7. The device according to claim 1, further comprising a device control unit that performs control for dealing with an abnormality of the one device on at least one of the one device or another device in the facility in response to the abnormality of the one device being identified.

8. Acquiring acoustic data detected by an acoustic sensor of a robot capable of moving within a facility; Detecting the degree of abnormality of a sound using the acoustic data; Obtaining LiDAR data detected by the LiDAR sensor of the robot; Detecting at least one device using the LiDAR data; Identifying a device in which an abnormality has occurred among the at least one device based on a change in the degree of abnormality accompanying the movement of the robot A method comprising the steps of.

9. Executed by a computer, the computer is caused to function as: An acoustic data acquisition unit that acquires acoustic data detected by an acoustic sensor of a robot capable of moving within a facility; An acoustic abnormality detection unit that detects the degree of acoustic abnormality using the acoustic data; A LiDAR data acquisition unit that acquires LiDAR data detected by the LiDAR sensor of the robot; A device detection unit that detects at least one device using the LiDAR data; An abnormal device identification unit that identifies a device in which an abnormality has occurred among the at least one device based on a change in the degree of abnormality accompanying the movement of the robot A program for causing the computer to function as such.

Images