Apparatus, method, and program for processing acoustic data detected in facility

Abstract

To provide an apparatus for identifying an abnormal place in an abnormal area by processing acoustic data detected in a facility.SOLUTION: An apparatus includes: an acoustic data acquisition section for acquiring acoustic data detected in a facility; an abnormal area identification section for identifying an abnormal area having the possibility of existence of a device in which abnormality occurs in the facility by using first acoustic data detected with a first acoustic sensor; and an abnormal place identification section for identifying an abnormal place in the abnormal area by using second acoustic data from the abnormal area, which is detected by a second acoustic sensor with higher directivity than that of the first acoustic sensor.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an apparatus, a method, and a program for processing acoustic data detected within a facility.

Background Art

[0002] Patent Document 1 states that "three sound collection devices 31 to 33 are arranged around a monitoring target portion of a power generation plant 1 such that planes including sound collection surfaces 31 are orthogonal to each other" (paragraph 0015), that "for each area 32, the inherent frequency spectrum information of sound generated from the sound collection target space (a part of the monitoring target portion of the power generation plant 1) of the sound collection element 33 installed in the area 32 during normal operation of the power generation plant 1 is stored" (paragraph 0023), and that "the malfunction symptom occurrence device identification unit 52 searches the plant design information DB 56 for a record in which the occurrence position of the malfunction symptom included in the malfunction symptom detection information received by the malfunction symptom detection information receiving unit 51 is registered in the field 561. Then, the component device specified by the device information of the retrieved record is identified as the component device in which the malfunction symptom has occurred, and the device information of the record is output together with the malfunction symptom detection information" (paragraph 0039). [Prior Art Documents] [Patent Documents] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-215833

Summary of the Invention

[0003] In a first aspect of the present invention, there is provided an apparatus including: an acoustic data acquisition unit that acquires acoustic data detected within a facility; an abnormal area identification unit that uses first acoustic data detected by a first acoustic sensor to identify an abnormal area where there may be a device in which an abnormality has occurred within the facility; and an abnormal location identification unit that uses second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor to identify an abnormal location within the abnormal area.

[0004] In the above-described apparatus, the acoustic data acquisition unit may acquire first acoustic data detected by a first acoustic sensor mounted on a first robot capable of moving within the facility and second acoustic data detected by a second acoustic sensor mounted on a second robot capable of moving within the facility.

[0005] Any of the above-described apparatuses may include a robot command unit that, in response to an abnormal area being specified, instructs the second robot to head towards the abnormal area.

[0006] In the above-described apparatus, the robot command unit, in response to the second robot arriving at a position where it can detect second acoustic data from within the abnormal area, instructs the second robot to detect the second acoustic data while changing at least one of the position or direction of the second acoustic sensor, and the abnormal location specifying unit may specify the abnormal location using the second acoustic data detected by the second acoustic sensor at at least one of each position or each direction from within the abnormal area.

[0007] In any of the above-described apparatuses, the acoustic data acquisition unit may acquire first acoustic data and second acoustic data detected by a first acoustic sensor and a second acoustic sensor mounted on a third robot capable of moving within the facility.

[0008] Any of the above-described apparatuses may include an abnormal device specifying unit that specifies a device corresponding to the position of the abnormal location as the device in which the abnormality has occurred.

[0009] Any of the above-described apparatuses may include a device control unit that, in response to an abnormality of one device being detected, performs control for dealing with the abnormality of the one device on at least one of the one device or other devices within the facility.

[0010] In a second aspect of the present invention, there is provided a method comprising: obtaining acoustic data detected within a facility; an abnormal area specifying unit that specifies an abnormal area where there may be a device in which an abnormality has occurred within the facility, using first acoustic data detected by a first acoustic sensor; and specifying an abnormal location within the abnormal area, using second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor.

[0011] In a third aspect of the present invention, there is provided a program that is executed by a computer and causes the computer to function as an acoustic data acquisition unit that acquires acoustic data detected within a facility, an abnormal area specifying unit that specifies an abnormal area where there may be a device in which an abnormality has occurred within the facility, using first acoustic data detected by a first acoustic sensor, and an abnormal location specifying unit that specifies an abnormal location within the abnormal area, using second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor.

[0012] Note that the above summary of the invention does not list all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Embodiments for Carrying Out the Invention

[0014] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0015] FIG. 1 shows a schematic configuration of a facility 1 according to the present embodiment. The facility 1 may be an entire or partial segment such as a factory or a plant, and a plurality of devices 10 are arranged therein. Such a factory or a plant may be, for example, a factory for producing various industrial products, an industrial plant such as a chemical or metal plant, a plant for managing and controlling wells and their surroundings in a gas field or an oil field, a plant for managing and controlling power generation such as hydraulic, thermal, or nuclear power, a plant for managing and controlling environmental power generation such as solar or wind power, a plant for managing and controlling water supply and sewerage or a dam, etc. Further, the facility 1 may be an entire or partial part of a building or a transportation facility including a plurality of devices 10 to be controlled and monitored. The facility 1 includes a plurality of devices 10, a control system 20, one or more robots 30, a monitoring device 40, a collected information database 50, a device database 60, and a robot database 70.

[0016] The plurality of devices 10 are provided at various locations within the facility 1. Each of the plurality of devices 10 may be installed either indoors or outdoors within the area of the facility 1. At least some of the plurality of devices 10 may be a process device, a power generation device, or any other device (or equipment) that is controlled by the control system 20, or may be a part of such a device. At least some of the devices 10 may be provided with field devices that operate under the control of the control system 20 or other devices, or an operator. Further, at least some of the devices 10 may be the field devices themselves.

[0017] Such field devices may be, for example, sensor devices such as pressure gauges, flow meters, and temperature sensors, valve devices such as flow control valves and on-off valves, actuator devices such as fans and motors, imaging devices such as cameras or videos that capture objects such as the situation of a plant or equipment, acoustic devices such as microphones or speakers that collect abnormal sounds or emit alarm sounds in a plant or equipment, position detection devices that output the position information of the devices possessed by Facility 1, or other devices. Further, other devices 10 among the plurality of devices 10 may be structures such as pipes, storage tanks, supports, partition walls, or others that do not receive control by the control system 20.

[0018] The control system 20 is connected to at least one device 10 to be controlled among the plurality of devices 10. The control system 20 may be, for example, a distributed control system (DCS). The control system 20 controls each device 10 according to the state of each device 10 measured by sensors or the like provided in each device 10 to be controlled.

[0019] Each of the one or more robots 30 is used to monitor at least a part of the plurality of devices 10 in Facility 1. Each robot 30 can move within Facility 1 to collect sound and capture images and the like.

[0020] The monitoring device 40 is communicably connected to each of the one or more robots 30. The monitoring device 40 may be connected to each robot 30 via a wireless network such as a mobile phone network, wireless WAN, wireless LAN, or Bluetooth (registered trademark), or may be connected to each robot 30 via a wired network such as wired Ethernet (registered trademark). The monitoring device 40 according to the present embodiment is installed within Facility 1. Alternatively, the monitoring device 40 may be provided outside Facility 1 to remotely monitor each device 10 within Facility 1. For example, the monitoring device 40 may be installed in another facility or may be realized by, for example, a cloud server on the Internet.

[0021] The monitoring device 40 receives the collected data such as acoustic data and image data collected by each robot 30 in the facility 1 from each robot 30, and monitors the state of each device 10 using the collected data. When the monitoring device 40 identifies an abnormality in any one of the devices 10, it may instruct the control system 20 to perform control to address the abnormality.

[0022] The monitoring device 40 uses a collection information database 50, a device database 60, and a robot database 70 to monitor each device 10 in the facility 1 using one or more robots 30. The collection information database 50 records the collected data collected from each robot 30. The device database 60 records device data for each device 10 in the facility 1. The robot database 70 stores data related to each robot 30. These databases may be storage devices such as a hard disk connected to the monitoring device 40 by wire or wirelessly, or cloud storage on the Internet, and may be temporarily stored in the memory of the monitoring device 40 or the like.

[0023] FIG. 2 shows the configuration of the control system 20 according to the present embodiment together with a display device 240 and an input device 250. The display device 240 displays a display screen output by the control system 20. The input device 250 inputs an instruction to the control system 20 from a user such as an operator, worker, or maintenance staff of the facility 1 and supplies it to the control system 20. The display device 240 and the input device 250 may be provided in a monitoring console or a user terminal or the like connected to the control system 20.

[0024] The control system 20 may be a computer such as a workstation, a server computer, a general-purpose computer, or other computers, or may be a computer system to which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Further, the control system 20 may be implemented by one or more executable virtual computer environments within the computer. Alternatively, the control system 20 may be a dedicated computer designed for controlling each device 10, or may be dedicated hardware realized by a dedicated circuit. In the present embodiment, the control system 20 is installed in the facility 1, but the control system 20 may be provided outside the facility 1 by a cloud computing system or the like on the Internet.

[0025] The control system 20 includes a state acquisition unit 210, a state determination unit 220, a display processing unit 230, an instruction input unit 260, and a control unit 270. The state acquisition unit 210 acquires device state data indicating the state of each device 10 from each device 10, such as internal state values of each device 10 and measurement values measured by sensors provided in each device 10. The state acquisition unit 210 may receive the device state data of each device 10 by using communication according to a communication protocol such as HART (registered trademark), BRAIN, Foundation Fieldbus (registered trademark), ISA100.11a. The state acquisition unit 210 may receive from the monitoring device 40 the device state data of the device 10 detected by the monitoring device 40 by using the collection data collected by the robot 30.

[0026] The state determination unit 220 is connected to the state acquisition unit 210. The state determination unit 220 determines whether each device 10 is normal or abnormal by using the device state data acquired by the state acquisition unit 210. The state determination unit 220 may calculate a health index indicating the soundness of the operation in the entire facility 1 or a specific range within the facility 1 from at least one of the internal state values or measurement values of two or more devices 10, and determine whether the operation is normal or not. The state determination unit 220 may determine the normality or abnormality of at least some of the devices 10 by receiving from the monitoring device 40 the determination result of the normality or abnormality of the device 10 determined by the monitoring device 40 using the collection data collected by the robot 30.

[0027] The display processing unit 230 is connected to the state acquisition unit 210 and the state determination unit 220. The display processing unit 230 performs display control to cause the display device 240 to display a display screen including device state data such as the internal state value and measurement value of each device 10, a soundness index for the entire facility 1 or a specific range within the facility 1, and the occurrence status of abnormalities of each device 10.

[0028] The instruction input unit 260 receives an instruction from a user for the control system 20 from the input device 250. Such an instruction may be, for example, an instruction from a user who has seen the display screen displayed on the display device 240 to change the operation of at least one device 10. The instruction input unit 260 may be connected to the monitoring device 40 and may be communicable with the monitoring device 40 by wire or wirelessly. The instruction input unit 260 may receive from the monitoring device 40 an instruction to perform control for dealing with an abnormality in response to the detection of an abnormality in at least one device 10 by the monitoring device 40.

[0029] The control unit 270 is connected to the state acquisition unit 210 and the instruction input unit 260. The control unit 270 controls each device 10 according to the device state data acquired by the state acquisition unit 210. The control unit 270 may control each device 10 according to an instruction in response to receiving an instruction from a user or an instruction from the monitoring device 40.

[0030] Figure 3 shows the processing flow of the control system 20 according to the present embodiment. In step 300 (S300), the state acquisition unit 210 acquires device state data indicating the state of each device 10 from each device 10. The state acquisition unit 210 may receive device data indicating the state of at least one device 10 from the monitoring device 40.

[0031] In S310, the state determination unit 220 determines whether each device 10 is normal or abnormal. The state determination unit 220 may determine whether the device 10 is normal (not abnormal) according to whether at least one of the internal state value or the measured value of the device 10 is within a predetermined normal range corresponding to the value. Further, the state determination unit 220 calculates a health index indicating the soundness of the operation in the entire facility 1 or a specific range within the facility 1 from at least one of the internal state values or the measured values of two or more devices 10 using a predefined calculation formula or the like, and determines whether the operation of the entire facility 1 or a specific range within the facility 1 is normal according to whether the value of the health index is within the normal range.

[0032] In S320, the display processing unit 230 performs display control to cause the display device 240 to display a display screen including the internal state value and the measured value of each device 10, the health index for the entire facility 1 or a specific range within the facility 1, and the abnormal occurrence status of each device 10. The display processing unit 230 may generate a video output of the display screen and supply it to the display device 240, or may generate html or a script for generating the display screen and transmit it to the display device 240.

[0033] In S330, the instruction input unit 260 receives an instruction from the user for the control system 20 from the input device 250. The instruction input unit 260 may receive an instruction for performing control for dealing with an abnormality transmitted by the monitoring device 40 that has detected an abnormality in at least one device 10.

[0034] In S340, the control unit 270 controls each device 10 according to a predetermined control algorithm, control model, etc. using the device state data acquired by the state acquisition unit 210. The control unit 270 may calculate a control value for each device 10 by PI control, PID control, or the like. The control unit 270 may calculate a control value for each device 10 according to the device state data acquired by the state acquisition unit 210 using various machine learning models or the like. The control unit 270 may change the control content for each device 10 in response to an instruction from the instruction input unit 260.

[0035] The control system 20 repeats the processes from S300 to S340. Thereby, the control system 20 can adaptively control each device 10 according to the state of each device 10.

[0036] FIG. 4 shows the configuration of the robot 30 according to the present embodiment. The robot 30 includes one or more sensors 400, one or more sensors 410, one or more actuators 420, a state acquisition unit 430, a communication unit 440, and a control unit 450.

[0037] Each of the one or more sensors 400 detects or measures the state within the facility 1. In the example of this figure, the sensor 400a is an acoustic sensor that detects the sound within the facility 1 and outputs it as acoustic data. The sensor 400b is an image sensor that captures an image within the facility 1 and outputs it as image data. The sensor 400c is a LiDAR sensor. The sensor 400c measures at least one of the shape of each object within the irradiation range of light or laser or the distance to each irradiation point by irradiating light or laser to the outside and detecting the reflected light, and outputs the measurement result as LiDAR data. The robot 30 may not include at least one of the sensors 400a to 400c. The robot 30 may include at least one of a temperature sensor, a humidity sensor, a gas sensor, or other various sensors.

[0038] Each of the one or more sensors 410 detects or measures the state of the robot 30. In the example of this figure, the sensor 410a is a position sensor that detects the position of the robot 30 and outputs it as position data. The sensor 410a may be a GPS, specifically, a GPS (Global Positioning System) receiver that receives signals from GPS satellites to identify the position of the robot 30. Alternatively, the sensor 410a may be a sensor for identifying the position of the robot 30 using any position detection system capable of detecting the position of the robot 30 within the facility 1.

[0039] The sensor 410b is an azimuth sensor such as a geomagnetic sensor, for example. The sensor 410b measures the azimuth of the robot 30 or the azimuth of the detection direction of each sensor 400. The sensor 410c is an angle sensor or an elevation angle sensor or the like for measuring the detection direction of each sensor 400. The sensors 410b and 410c output direction data indicating at least one of the azimuth of the robot 30 or the detection direction of each sensor 400 to the state acquisition unit 430. Note that the robot 30 may not include at least one of the sensors 410a to 410c. The robot 30 may include at least one of a speed sensor, an acceleration sensor, or other various sensors.

[0040] Each of the one or more actuators 420 is a motor or the like for driving each part of the robot 30. The robot 30 may include various actuators 420 such as an actuator 420 for movement by traveling or flying, an actuator 420 for changing the orientation of the entire robot 30 or a part thereof, and an actuator 420 for operating equipment such as an arm.

[0041] The state acquisition unit 430 is connected to one or more sensors 400 and one or more sensors 410. The state acquisition unit 430 acquires the state inside the facility 1 measured by the one or more sensors 400 and the state of the robot 30 measured by the one or more sensors 410. The state acquisition unit 430 may acquire the state inside the facility 1 by receiving various measurement data such as acoustic data, image data, and LiDAR data from the one or more sensors 400. The state acquisition unit 430 may acquire the state of the robot 30 by receiving various measurement data such as position data and direction data from the one or more sensors 410.

[0042] The communication unit 440 is connected to the state acquisition unit 430. The communication unit 440 performs wireless or wired communication with the monitoring device 40. The communication unit 440 transmits various measurement data acquired by the state acquisition unit 430 to the monitoring device 40. Also, the communication unit 440 receives various instructions for the robot 30 from the monitoring device 40.

[0043] The control unit 450 controls each actuator 420 according to various measurement data acquired by the state acquisition unit 430 and instructions from the monitoring device 40. Thereby, the robot 30 can perform operations corresponding to the state inside the facility 1, the state of the robot 30, and instructions from the monitoring device 40.

[0044] Note that some robots 30 may be fixedly installed inside the facility 1, and such robots 30 do not need to be equipped with actuators for movement. Also, inside the facility 1, there may be provided monitoring devices (such as surveillance cameras and surveillance microphones) that are equipped with one or more sensors 400 but are not classified as robots. Such monitoring devices will have some functions and configurations of the robot 30. Therefore, in this specification, for the sake of convenience of explanation, at least one of the robots 30 may be such a monitoring device, and such a monitoring device may perform the following processes and the like within the scope of the implemented functions.

[0045] FIG. 5 shows the processing flow of the robot 30 according to the present embodiment. In S500, one or more sensors 400 observe the state of the facility 1 in the vicinity of the robot 30. One or more sensors 410 observe the state of the robot 30. The state acquisition unit 430 acquires various measurement data indicating the state of the facility 1 from one or more sensors 400. The state acquisition unit 430 acquires various measurement data indicating the state of the robot 30 from one or more sensors 410.

[0046] In S510, the communication unit 440 transmits various measurement data indicating the state within the facility 1 and the state of the robot 30 observed by one or more sensors 400 and one or more sensors 410 to the monitoring device 40. Here, when the central direction of measurement of each sensor 400 coincides with the direction of the robot 30, that is, for example, when a sensor unit or the like on which each sensor 400 is mounted faces the front of the robot 30, the measurement direction (central direction of measurement) of each sensor 400 coincides with the direction of the robot 30. In such a case, if the robot 30 transmits its own direction to the monitoring device 40, the monitoring device 40 can obtain the measurement direction of each sensor 400.

[0047] When the direction of the sensor unit on which each sensor 400 is mounted is variable with respect to the robot 30 main body, the central direction of measurement of each sensor 400 does not necessarily coincide with the direction of the robot 30. In this case, the robot 30 may transmit direction data including its own direction and the measurement direction of each sensor 400 to the monitoring device 40. Alternatively, the robot 30 may transmit direction data including its own direction and the difference or offset of the measurement direction of each sensor 400 with respect to its own direction to the monitoring device 40. In this case, the monitoring device 40 can obtain the measurement direction of each sensor 400 by adding the difference or offset of the measurement direction of each sensor 400 to its own direction.

[0048] In S520, the communication unit 440 receives an instruction from the monitoring device 40. In S530, the control unit 450 performs an operation according to the instruction from the monitoring device 40. The control unit 450 drives at least one actuator 420 according to an instruction to move, an instruction to change direction, an instruction to change the direction of an acoustic sensor, an image sensor, etc. received from the monitoring device 40, thereby operating the robot 30 as instructed. Further, the control unit 450 may set various parameters within the robot 30 according to an instruction from the monitoring device 40.

[0049] The robot 30 repeats the processes from S500 to S530. Thereby, the robot 30 can move within the facility 1 according to an instruction from the monitoring device 40, and perform sound collection by the sensor 400a, image capturing by the sensor 400b, acquisition of LiDAR data by the sensor 400c, and the like.

[0050] FIG. 6 shows the configuration of the monitoring device 40 according to the present embodiment together with the collected information database 50, the device database 60, the robot database 70, the input device 680, and the display device 690. The input device 680 inputs an instruction for the monitoring device 40 from a user such as an operator, a worker, or a maintenance staff of the facility 1 and supplies it to the monitoring device 40. The display device 690 displays a display screen output by the monitoring device 40. The input device 680 and the display device 690 may be provided in a monitoring console or a user terminal connected to the monitoring device 40. The display device 690 and the input device 680 may be shared with the display device 240 and the input device 250 shown in FIG. 2.

[0051] The monitoring device 40 may be a facility management device that manages each device 10 in the facility 1, or may be a device that realizes some functions included in the facility management device. The monitoring device 40 may be a computer such as a workstation, a server computer, a general-purpose computer, or other computers, or may be a computer system to which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. Further, the monitoring device 40 may be implemented by one or more executable virtual computer environments in the computer. Such a computer functions as the monitoring device 40 by executing a program for monitoring each device 10 using the robot 30. Instead of this, the monitoring device 40 may be a dedicated computer designed for monitoring each device 10, or may be dedicated hardware realized by a dedicated circuit.

[0052] In the present embodiment, the monitoring device 40 is installed in the facility 1. Instead of this, the monitoring device 40 may be provided outside the facility 1 by a cloud computing system or the like on the Internet. Further, in the present embodiment, the monitoring device 40 remotely operates each robot 30. Instead of this, the monitoring device 40 may be mounted on at least one robot 30, and directly operate on-site a robot group including each robot 30 or two or more robots 30.

[0053] The monitoring device 40 includes a communication unit 600, a collected information database connection unit 620, a device database connection unit 630, a robot database connection unit 640, a monitoring processing unit 650, an input processing unit 660, a display processing unit 665, and a communication unit 670. The communication unit 600 performs wireless or wired communication with the robot 30. The communication unit 600 includes an acoustic data acquisition unit 602, an image data acquisition unit 604, a LiDAR data acquisition unit 606, a position data acquisition unit 608, a direction data acquisition unit 610, and an instruction transmission unit 612.

[0054] The acoustic data acquisition unit 602 acquires the acoustic data detected within Facility 1. The acoustic data acquisition unit 602 may acquire the acoustic data by receiving the acoustic data transmitted by the robot 30. The image data acquisition unit 604 acquires the image data captured within Facility 1. The image data acquisition unit 604 may acquire the image data by receiving the image data transmitted by the robot 30. The LiDAR data acquisition unit 606 acquires the LiDAR data detected by the sensor 400c of the robot 30 within Facility 1. The LiDAR data acquisition unit 606 may acquire the LiDAR data by receiving the LiDAR data transmitted by the robot 30.

[0055] The position data acquisition unit 608 acquires the position data indicating the position of the robot 30. The position data acquisition unit 608 may acquire the position data by receiving the position data transmitted by the robot 30. The direction data acquisition unit 610 acquires the direction data indicating the directions of the robot 30 and each sensor 400. The direction data acquisition unit 610 may acquire the direction data by receiving the direction data transmitted by the robot 30. The instruction transmission unit 612 transmits the instruction for the robot 30 determined within the monitoring device 40 to the robot 30.

[0056] The collected information database connection unit 620 is connected to the collected information database 50, the communication unit 600, and the monitoring processing unit 650. The collected information database connection unit 620 records various measurement data obtained by the acoustic data acquisition unit 602, the image data acquisition unit 604, the LiDAR data acquisition unit 606, the position data acquisition unit 608, and the direction data acquisition unit 610 in the communication unit 600 as collected data in the collected information database 50. Further, the collected information database connection unit 620 accesses the collected information database 50 in response to a request from the monitoring processing unit 650. Note that the monitoring device 40 may not include the collected information database connection unit 620 if it is not necessary to record the history of the collected data, and the measurement data obtained by the communication unit 600 may be supplied to the monitoring processing unit 650 without passing through the collected information database connection unit 620. In this case, the collected information database 50 is unnecessary.

[0057] The device database connection unit 630 is connected to the device database 60 and the monitoring processing unit 650. The device database connection unit 630 accesses the device database 60 in response to a request from the monitoring processing unit 650. The robot database connection unit 640 is connected to the robot database 70 and the monitoring processing unit 650. The robot database connection unit 640 accesses the robot database 70 in response to a request from the monitoring processing unit 650.

[0058] The monitoring processing unit 650 is connected to the collected information database connection unit 620, the device database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670. The monitoring processing unit 650 reads out the robot data of each robot 30 via the robot database connection unit 640, and determines the monitoring actions in the facility 1 that each robot 30 should execute using the robot data. The monitoring processing unit 650 instructs each robot 30 to perform each operation included in the determined monitoring action via the instruction transmission unit 612 in the communication unit 600.

[0059] Further, the monitoring processing unit 650 reads out the collection data collected by one or more robots 30 and recorded in the collection information database 50 via the collection information database connection unit 620. Then, the monitoring processing unit 650 uses the collection data to detect the state of each device 10 monitored by each robot 30. The monitoring processing unit 650 determines whether each device 10 is normal or not using the collection data. The monitoring processing unit 650 may read out the device data of each device 10 via the device database connection unit 630 and determine whether each device 10 is normal or abnormal based on the information registered in the device data.

[0060] The monitoring processing unit 650 may cause the display processing unit 665 to generate a display screen for displaying the state of each device 10 and the determination result of normal or abnormal, and display it on the display device 690. The monitoring processing unit 650 may transmit the state of each device 10 and the determination result of normal or abnormal to the control system 20 via the communication unit 670.

[0061] When at least one device 10 is detected to be abnormal as a result of monitoring each device 10 using one or more robots 30, the monitoring processing unit 650 may generate an instruction to perform control for dealing with the abnormality. The monitoring processing unit 650 may transmit an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0062] The input processing unit 660 is connected to the input device 680. The input processing unit 660 receives an input of information from the user to the input device 680 and supplies it to the monitoring processing unit 650. The display processing unit 665 is connected to the monitoring processing unit 650. The display processing unit 665 performs display processing to generate a display screen according to an instruction from the monitoring processing unit 650 and display it on the display device 690.

[0063] The communication unit 670 is connected to the monitoring processing unit 650. The communication unit 670 may be connected to the control system 20 and may be capable of communicating with the control system 20 wirelessly or by wire. The communication unit 670 may transmit the device status data of each device 10 detected by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit the determination result of normal or abnormal for each device 10 determined by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit an instruction to perform control for dealing with an abnormality to the control system 20 in response to an abnormality being detected in at least one device 10. Note that the communication unit 670 may not have a function of transmitting at least one of the device status data of the device 10, the determination result of normal or abnormal for the device 10, or the instruction to perform control for dealing with an abnormality to the control system 20. When the monitoring device 40 does not have any of these functions, the communication unit 670 may not be provided.

[0064] According to the monitoring device 40 described above, by monitoring the inside of the facility 1 using one or a plurality of robots 30 that can move inside the facility 1, it is possible to monitor a large number of devices 10 using a relatively small number of robots 30. In addition, since the monitoring device 40 can control the robot 30 according to the situation at each observation point and collect necessary information, it is possible to detect the state of each device 10 that cannot be collected by sensors or the like pre-installed in each device 10.

[0065] FIG. 7 shows an example of the data structure of the collected information database 50 according to the present embodiment. The collected information database 50 records the measurement data collected from each robot 30 as collected data. The collected information database 50 may record one or more records, each of which is a unit of collected data. In the data structure of the collected information database 50 shown in this figure, each record of the collected data is arranged in the row direction, and the collected data of each record includes fields of "date and time", "position and direction", "robot identification information", and "measurement data".

[0066] "Date and Time" is a field that records the date and time when the measurement data of the corresponding record was acquired by the robot 30. "Position and Orientation" is a field that records at least one position and orientation of the robot 30 or each sensor 400 at the timing when the measurement data of the corresponding record was acquired. The collection information database 50 may record, as "Position and Orientation", at least one of the position detected by the sensor 410a of the robot 30, the orientation of the robot 30 detected by the sensor 410b, or the direction of the robot 30 or each sensor 400 detected by the sensor 410c at the timing indicated by "Date and Time". The collection information database 50 may record, as the detected position, at least one of the coordinates within the facility 1 or the area identification information that identifies each area when the facility 1 is divided into a plurality of areas.

[0067] "Robot Identification Information" is a field that records data values such as an identification number, a serial number, or other unique number or character string that can identify the robot 30 that acquired the measurement data of the corresponding record, which can specify the robot 30 that acquired the measurement data within the facility 1. "Measurement Data" is a field that records the measurement data acquired by the robot 30 at the timing indicated by "Date and Time". The data recorded as "Measurement Data" may be various types of measurement data acquired by the robot 30, such as at least one of acoustic data, image data, or LiDAR data.

[0068] The communication unit 600 within the monitoring device 40 may receive, as separate packets or packet groups, etc., data including the acquisition date and time of measurement data by the robot 30, the robot identification information of the robot 30, and measurement data regarding the position and orientation of the robot 30, and data including the acquisition date and time of measurement data by the robot 30, the robot identification information of the robot 30, and measurement data regarding the state within the facility 1. In this case, each time the communication unit 600 receives measurement data regarding the position and orientation of the robot 30, it updates the position and orientation of the robot 30 managed by the monitoring device 40. When the communication unit 600 receives measurement data regarding the state within the facility 1, it generates a record of the collected data by associating the measurement data with the position and orientation of the robot 30 at the acquisition date and time of the measurement data. Alternatively, when the communication unit 440 within the robot 30 transmits various measurement data indicating the state within the facility 1 and the state of the robot 30 observed by each sensor 400 and each sensor 410 to the monitoring device 40 (see S510 in FIG. 5), it may assemble the collected data for one record from these measurement data, encode it into one or two or more packets, and transmit it to the monitoring device 40.

[0069] FIG. 8 shows an example of the data structure of the device database 60 according to the present embodiment. The device database 60 records device data for each of a plurality of devices 10 within the facility 1. The device database 60 may record records, each of which is device data for one device, for the number of devices 10. In the data structure of the device database 60 shown in this figure, each record of the device data is arranged in the row direction, and the device data of each record includes fields of "device identification information", "device name", "device information", "position", "template", and "acoustic data".

[0070] "Device identification information" is a field for recording data values such as an identification number, a serial number, or other unique numbers or character strings that can identify the corresponding device 10 within the facility 1. "Device name" is a field for recording the device name assigned to the corresponding device 10 by a user or the like.

[0071] "Device Information" is a field that records various information about the corresponding device 10. "Location" is a field that records the location of the device 10 within the facility 1. "Template" is a field that records image data (template image data) of the appearance of the corresponding device 10 for use in identifying the device 10 by image matching. "Acoustic Data" is a field that records at least one of the acoustic data collected from the corresponding device 10 or the acoustic data emitted by the device 10 when the corresponding device 10 is normal.

[0072] Figure 9 shows an example of the data structure of the robot database 70 according to this embodiment. The robot database 70 records robot data for each of one or more robots 30 in the facility 1. The robot database 70 may record records, each of which is robot data for one robot, for the number of robots 30. In the data structure of the robot database 70 shown in this figure, each record of the robot data is arranged in the row direction, and the robot data of each record includes fields of "Robot Identification Information", "Robot Name", "Robot Information", "Location and Direction", and "Schedule Information".

[0073] "Robot Identification Information" is a field that records a data value such as an identification number, serial number, or other unique number or character string that can identify the corresponding robot 30 within the facility 1. "Robot Name" is a field that records the robot name assigned to the corresponding robot 30 by a user or the like.

[0074] "Robot Information" is a field that records various information about the corresponding robot 30. "Position and Direction" is a field that records the position and direction of the corresponding robot 30 within Facility 1, and, if necessary, the direction of each sensor 400 (absolute direction or offset relative to the direction of robot 30). "Schedule Information" is a field that records the schedule for the corresponding robot 30 to patrol within Facility 1 to collect sounds within Facility 1 or the sounds of each device 10.

[0075] Figure 10 shows the configuration of the monitoring processing unit 650 of the monitoring device 40 according to the present embodiment. The monitoring processing unit 650 includes an abnormal area identification unit 1000, a first sound source separation unit 1010, a first sound abnormality detection unit 1020, a second sound source separation unit 1030, a second sound abnormality detection unit 1035, an abnormal location identification unit 1037, an abnormal device identification unit 1040, a device control unit 1050, and a robot command unit 1060.

[0076] The abnormal area identification unit 1000 is connected to the collection information database connection unit 620. The abnormal area identification unit 1000 reads out, via the collection information database connection unit 620, the collection data in which acoustic data is recorded as measurement data among the collection data registered in the collection information database 50. The abnormal area identification unit 1000 acquires the acoustic data, the date and time when the acoustic data was acquired, and the position and direction where the acoustic data was observed within Facility 1. The abnormal area identification unit 1000 uses the acoustic data detected by the first acoustic sensor 400a read out via the collection information database connection unit 620 to identify an abnormal area where there may be a device in which an abnormality has occurred within Facility 1.

[0077] The first sound source separation unit 1010 is connected to the collection information database connection unit 620 and the abnormal area identification unit 1000. The first sound source separation unit 1010 reads out the collection data in which acoustic data is recorded as measurement data among the collection data registered in the collection information database 50 via the collection information database connection unit 620. The first sound source separation unit 1010 reads out the acoustic data in the abnormal area detected by the second acoustic sensor 400a which has higher directivity than the first acoustic sensor 400a. The first sound source separation unit 1010 acquires the acoustic data, the date and time when the acoustic data was acquired, and the position and direction where the acoustic data was observed within facility 1. The first sound source separation unit 1010 separates the read acoustic data into sound sources in the first direction.

[0078] The first sound anomaly detection unit 1020 is connected to the first sound source separation unit 1010. The first sound anomaly detection unit 1020 uses the acoustic data separated into sound sources in the first direction by the first sound source separation unit 1010 to detect abnormal sounds in the first direction with respect to the robot 30. The first sound anomaly detection unit 1020 may use the acoustic data detected by the second acoustic sensor 400a in the abnormal area to detect abnormal sounds in the first direction with respect to the robot 30.

[0079] The second sound source separation unit 1030 is connected to the collection information database connection unit 620. The second sound source separation unit 1030 reads out the collection data in which acoustic data is recorded as measurement data among the collection data registered in the collection information database 50 via the collection information database connection unit 620. The second sound source separation unit 1030 reads out the acoustic data in the abnormal area detected by the acoustic sensor 400a which has higher directivity than the first acoustic sensor 400a. The second sound source separation unit 1030 acquires the acoustic data, the date and time when the acoustic data was acquired, and the position and direction where the acoustic data was observed within facility 1. The second sound source separation unit 1030 separates the read acoustic data into sound sources in the second direction.

[0080] The second abnormal sound detection unit 1035 is connected to the second sound source separation unit 1030. The second abnormal sound detection unit 1035 may detect an abnormal sound in the second direction with respect to the robot 30 using the acoustic data separated in the second direction by the second sound source separation unit 1030.

[0081] The abnormal location identification unit 1037 is connected to the first abnormal sound detection unit 1020, the second abnormal sound detection unit 1035, and the second sound source separation unit 1030. The abnormal location identification unit 1037 uses the acoustic data from within the abnormal area detected by the second acoustic sensor 400a, which has a higher directivity than the first acoustic sensor 400a, to identify the abnormal location within the abnormal area. The abnormal location identification unit 1037 may identify the location where the abnormal sound occurred by identifying the direction of occurrence of the abnormal sound using the detection result in the first direction in the first abnormal sound detection unit 1020 and the detection result in the second direction in the second abnormal sound detection unit 1035. The abnormal location identification unit 1037 may cause the second sound source separation unit 1030 to acquire acoustic data and cause the second abnormal sound detection unit 1035 to detect an abnormal sound in the second direction in response to identifying the abnormal occurrence location in the first direction from the detection result of the first abnormal sound detection unit 1020.

[0082] The abnormal device identification unit 1040 is connected to the abnormal location identification unit 1037, the input processing unit 660, the device database connection unit 630, and the display processing unit 665. The abnormal device identification unit 1040 identifies the device corresponding to the location of the abnormal location identified by the abnormal location identification unit 1037 as the device in which the abnormality occurred. The abnormal device identification unit 1040 may cause the display processing unit 665 to generate a display screen for displaying the state of each device 10 detected by the robot 30 on the display device 690 and display it on the display device 690. Further, the abnormal device identification unit 1040 may cause the display processing unit 665 to generate a display screen for displaying the determination result of normal or abnormal for each device 10 on the display device 690 and display it on the display device 690. Further, the abnormal device identification unit 1040 may register the determination result of normal or abnormal for each device 10 in the device data of each device 10 in the device database 60 via the device database connection unit 630.

[0083] The machine control unit 1050 is connected to the abnormal device identification unit 1040, the device database connection unit 630, and the communication unit 670. In response to the detection of an abnormality in one device 10 by the abnormal device identification unit 1040, the machine control unit 1050 performs control for dealing with the abnormality of the one device 10 on at least one of the one device 10 or other devices 10 within the facility 1. The machine control unit 1050 transmits an instruction for performing control for dealing with the abnormality of the device 10 to the instruction input unit 260 within the control system 20 via the communication unit 670.

[0084] The robot command unit 1060 is connected to the collected information database connection unit 620, the device database connection unit 630, the robot database connection unit 640, the abnormal area identification unit 1000, and the abnormal location identification unit 1037. The robot command unit 1060 determines the operations to be performed by each robot 30 and instructs each robot 30 to perform the determined operations. In response to the identification of an abnormality by the abnormal area identification unit 1000 or the abnormal location identification unit 1037, the robot command unit 1060 may instruct the robot 30 to perform at least one of movement or sound collection.

[0085] The monitoring processing unit 650 shown above has each component shown in FIG. 10 in order to realize each function such as the identification of the device 10 that is the sound source of each sound component (hereinafter also simply referred to as acoustic data) indicated by the acoustic data, the identification of the abnormality of the device 10 that is the sound source based on the abnormality degree of the acoustic data, the instruction of control for dealing with the abnormality of the device 10, and the instruction of operations for each robot 30. Alternatively, the monitoring processing unit 650 may adopt a configuration that does not have some of the components shown in FIG. 10.

[0086] For example, the monitoring processing unit 650 may not have the device control unit 1050 and may not issue instructions for control to address abnormalities in the device 10. Also, the monitoring processing unit 650 may not have the robot command unit 1060 and may not issue instructions for operations to each robot 30. Even when the monitoring device 40 does not have some of the functions exemplified herein, the monitoring device 40 can enable the user to receive the information output or displayed by the monitoring device 40, discover abnormalities in the device 10, and cause at least one device 10 in the facility 1 to take technical measures to address the abnormalities in the device 10.

[0087] According to the monitoring device 40 described above, it is possible to collect the acoustic data detected in the facility 1 and identify the device 10 that is the source of each sound component included in the acoustic data. Thereby, the monitoring device 40 can analyze the potential state of each device 10, which may not be detectable from the states of the respective devices 10 collected by the control system 20, from the sound components having the device 10 as the source. Also, the monitoring device 40 can promptly collect abnormal sounds emitted by the devices 10 installed at various locations in the facility 1 and can promptly address abnormalities in the devices 10.

[0088] FIGS. 11 and 12 show the processing flow of the monitoring device 40 according to the present embodiment. In the processing flows of FIGS. 11 and 12, an example is shown in which the first direction in which the first sound abnormality detection unit 1020 detects an abnormal sound is the horizontal direction, and the second direction in which the second sound abnormality detection unit 1035 detects an abnormal sound is the vertical direction. However, it is not limited thereto, and the first direction and the second direction may be different from each other.

[0089] In S1100, the robot command unit 1060 instructs the robot 30 to be processed to move within the facility 1 via the instruction transmission unit 612. The robot command unit 1060 may refer to the robot database 70 and determine the robot 30 (hereinafter also referred to as the first robot 30) that can move within the facility 1 and is equipped with the omnidirectional sensor 400a (hereinafter also referred to as the first acoustic sensor 400a) as the robot 30 to be processed. Here, the directivity may be the relationship between the direction of the sensor 400a with respect to the sound source and the sound collection performance, and the omnidirectional first acoustic sensor 400a may collect sound with the same sensitivity of 360 degrees centered on the first acoustic sensor 400a.

[0090] The robot command unit 1060 may refer to the schedule information recorded in the robot database 70 in association with the first robot 30 to be processed and obtain the schedule for the first robot 30 to be processed to patrol within the facility 1. This schedule may include information necessary to determine the operation of the first robot 30, such as the position of each observation point where the first robot 30 should observe the state within the facility 1, the time to arrive at each observation point, the movement route between the observation points, or the observation direction at each observation point. The robot command unit 1060 moves the first robot 30 to the next observation point according to the specification in this schedule and supplies an instruction to the first robot 30 to perform observation at the next observation point. In response to this, the first robot 30 moves to the next observation point (see S520 and S530 in FIG. 5).

[0091] In S1110, the first robot 30 observes the state inside the facility 1 and the state of the first robot 30 using each sensor 400 at the observation point (see S500 in FIG. 5). The first robot 30 transmits various measurement data indicating the observed state inside the facility 1 and the state of the first robot 30 to the monitoring device 40 (see S510 in FIG. 5). The acoustic data acquisition unit 602 in the monitoring device 40 acquires the acoustic data detected by the first acoustic sensor 400a (hereinafter also referred to as the acoustic data to be detected) among the measurement data transmitted from the first robot 30. The collection information database connection unit 620 may add the position data acquired by the position data acquisition unit 608, the direction data acquired by the direction data acquisition unit 610, etc. to the acoustic data acquired by the acoustic data acquisition unit 602 and record it in the collection information database 50 as collection data.

[0092] In S1120, the abnormal area identification unit 1000 determines the presence or absence of abnormal sounds in each of a plurality of areas inside the facility 1, and identifies the area in which the abnormal sound is determined to have occurred as an abnormal area. The abnormal area identification unit 1000 acquires the acoustic data to be detected detected by the first acoustic sensor 400a via the collection information database connection unit 620. The abnormal area identification unit 1000 may identify, as an abnormal area, an area where the sound pressure of the acoustic data to be detected exceeds a predetermined threshold value. When the sound pressure of the acoustic data to be detected exceeds a predetermined threshold value in a plurality of adjacent areas, the abnormal area identification unit 1000 may identify the area with the highest sound pressure as the abnormal area.

[0093] Further, the abnormal area identification unit 1000 may calculate the abnormality degree of the acoustic data to be detected detected by the first acoustic sensor 400a and identify the abnormal area. For example, when all the devices 10 inside the facility 1 or all the devices 10 in the target area inside the facility 1 are normal, and the acoustic data detected by the first acoustic sensor 400a in the target area inside the facility 1 is recorded in the collection information database 50 as a history, the abnormality degree may be calculated using the acoustic data (hereinafter also referred to as reference acoustic data) in the target area during normal times.

[0094] The abnormal area identification unit 1000 may calculate, as the degree of abnormality, a value indicating how different the acoustic data to be detected is from the reference acoustic data. The abnormal area identification unit 1000 may determine the degree of abnormality according to the comparison result between the reference acoustic data and the acoustic data to be detected. For example, the abnormal area identification unit 1000 may determine a larger degree of abnormality as the difference in sound pressure between the reference acoustic data and the acoustic data to be detected is larger. The abnormal area identification unit 1000 may identify, as an abnormal area, an area where the degree of abnormality exceeds a predetermined threshold value. Further, when the degrees of abnormality of a plurality of adjacent areas exceed a predetermined threshold value, the abnormal area identification unit 1000 may identify, as an abnormal area, the area with the largest degree of abnormality. Here, the degree of abnormality may be represented by a real number or an integer within a predetermined range such as from 0 to 1 or from 0 to 100%, or may be a binary value such as normal or abnormal.

[0095] The abnormal area identification unit 1000 may supply data indicating the abnormal area (such as area identification information) to the first sound source separation unit 1010 and the robot command unit 1060. Thereby, the first sound source separation unit 1010 and the robot command unit 1060 may start the abnormal location identification operation.

[0096] When the abnormal area identification unit 1000 identifies an abnormal area (''Yes'' in S1120), the monitoring device 40 proceeds to the process in S1130. When the abnormal area identification unit 1000 does not identify an abnormal area in any of the one or more areas (''No'' in S1120), the monitoring device 40 proceeds to the process in S1100.

[0097] In S1130, in response to the abnormal area being specified by the abnormal area specifying unit 1000, the robot command unit 1060 instructs the robot (hereinafter also referred to as the second robot 30) to head towards the abnormal area. The robot command unit 1060 may refer to the robot database 70 and determine the robot 30 that can move within the facility 1 and is equipped with a sensor 400a (hereinafter also referred to as the second acoustic sensor 400a) with higher directivity than the first acoustic sensor 400a as the second robot 30 to be processed. The robot command unit 1060 may determine the robot 30 equipped with an acoustic sensor array in which two or more second acoustic sensors 400a are arranged in the first direction as the second robot 30 to be processed. Here, the plurality of second acoustic sensors 400a mounted on the second robot 30 may be arranged in a one-dimensional or two-dimensional array.

[0098] In response to the second robot 30 arriving at a position where it can detect acoustic data from within the abnormal area, the robot command unit 1060 may instruct the second robot 30 to detect acoustic data while changing at least one of the position or direction of the second acoustic sensor 400a. For example, the robot command unit 1060 may instruct the second robot 30 to detect acoustic data with the second acoustic sensor 400a while traveling all the travel routes within the abnormal area. The robot command unit 1060 may instruct the second robot 30 to detect acoustic data with the second acoustic sensor 400a while changing the orientation of the second acoustic sensor 400a at one or a plurality of predetermined positions (for example, the central position) within the abnormal area.

[0099] The second robot 30 observes the state inside the facility 1 and the state of the first robot 30 using each sensor 400 at the observation point according to an instruction from the robot command unit 1060 (see S500 in FIG. 5). The second robot 30 transmits various measurement data indicating the observed state inside the facility 1 and the state of the second robot 30 to the monitoring device 40 (see S510 in FIG. 5). The acoustic data acquisition unit 602 in the monitoring device 40 acquires the acoustic data detected by the second acoustic sensor 400a (hereinafter also referred to as the acoustic data to be detected) among the measurement data transmitted from the second robot 30. The collection information database connection unit 620 may add the position data acquired by the position data acquisition unit 608, the direction data acquired by the direction data acquisition unit 610, etc. to the acoustic data acquired by the acoustic data acquisition unit 602, and record it in the collection information database 50 as collection data.

[0100] In S1140, the first sound source separation unit 1010 acquires the acoustic data detected by the second acoustic sensor 400a at at least one of each position or each direction from within the abnormal area specified by the abnormal area specifying unit 1000, and extracts at least one sound component by separating the acoustic data in the first direction. The first sound source separation unit 1010 acquires the acoustic data detected by the second acoustic sensor 400a acquired by the acoustic data acquisition unit 602 via the collection information database connection unit 620.

[0101] The first sound source separation unit 1010 can separate the acquired acoustic data into sound components in a plurality of detection directions in the first direction (e.g., on a horizontal plane) centered on the second robot 30 (or the sensor unit including a plurality of second acoustic sensors 400a) in the abnormal area by separating the sound source in the first direction. Here, each second acoustic sensor 400a collects sound from a predetermined direction range other than the measurement direction of the second acoustic sensor 400a according to the directivity, so that the sound synthesized from sounds from various sound sources is collected. The first sound source separation unit 1010 performs a process of separating or decomposing the sound collected by each second acoustic sensor 400a into the sounds generated by each sound source in the first direction. The first sound source separation unit 1010 may utilize various sound source separation and sound source localization techniques.

[0102] The first sound source separation unit 1010 may calculate the sound components from each direction with respect to the second robot 30 from the time difference at which each sound component included in the acoustic data from the plurality of second acoustic sensors 400a arranged in the first direction reaches each second acoustic sensor 400a. Here, the first sound source separation unit 1010 adds the absolute direction (measurement direction) in which the sensor unit itself is facing to the relative direction (the direction in which the sound source is separated) of the sound source of each sound component with respect to the measurement direction of the sensor unit including the second acoustic sensor 400a, thereby calculating the absolute direction of the sound source of each sound component at the observation point. Here, the absolute direction means an angle represented starting from at least a common reference direction (such as north) within the facility 1, such as an angle in the horizontal and vertical directions with north as a reference. The relative direction means an angle represented starting from an arbitrarily selected reference direction (such as the traveling direction of the second robot 30).

[0103] Also, while changing the orientation of the second acoustic sensor 400a within the range of the first direction with respect to the second robot 30, the first sound source separation unit 1010 may calculate the magnitude (sound pressure) of at least one sound component within the range where the orientation of the second acoustic sensor 400a is changed, using the acoustic data detected by the second acoustic sensor 400a.

[0104] In S1150, the first sound abnormality detection unit 1020 calculates the respective abnormality degrees using the acoustic data separated by sound sources in a plurality of detection directions in the first direction (hereinafter also referred to as the acoustic data to be detected). Here, the abnormality degree may be represented by a real number or an integer within a predetermined range such as from 0 to 1 or from 0 to 100%, or may be a binary value such as normal or abnormal.

[0105] The first sound abnormality detection unit 1020 may calculate, as the abnormality degree, a value indicating how different the acoustic data to be detected is from the acoustic data (hereinafter also referred to as the reference acoustic data) when all the devices 10 in the facility 1 or all the devices 10 in the specified abnormal area are normal.

[0106] Here, when the past acoustic data in the same detection direction in the first direction at the same detection position of the abnormal area when all the devices 10 in the facility 1 or all the devices 10 in the abnormal area are normal is recorded in the collection information database 50 as one or more histories, the first sound abnormality detection unit 1020 may use the recorded past acoustic data as the reference acoustic data for calculating the abnormality degree.

[0107] For example, the first sound abnormality detection unit 1020 may determine the abnormality degree according to the comparison result between the reference acoustic data and the acoustic data to be detected. The first sound abnormality detection unit 1020 may perform the comparison between the reference acoustic data and the acoustic data to be detected in either the time domain or the frequency domain. When the comparison is performed in the time domain, the first sound abnormality detection unit 1020 adjusts the phase difference so that the time integral of the difference (such as the absolute value of the difference) between the reference acoustic data and the acoustic data to be detected is minimized, and may use the time integral of the difference between the reference acoustic data and the acoustic data to be detected at the adjusted phase difference as the abnormality degree. In this case, the first sound abnormality detection unit 1020 may adjust the amplitude so that, for example, the average amplitudes match in order to match the loudness of the sound of the reference acoustic data and the acoustic data to be detected.

[0108] When comparing the reference acoustic data with the acoustic data to be detected in the frequency domain, the first sound anomaly detection unit 1020 may calculate the degree of anomaly by integrating the difference (such as the absolute value of the difference) for each frequency between the frequency spectrum of the reference acoustic data and the frequency spectrum of the acoustic data to be detected in the frequency direction. In this case, the first sound anomaly detection unit 1020 may perform an adjustment to equalize the sound volume so as to enable comparison between the reference acoustic data and the acoustic data to be detected.

[0109] The first sound anomaly detection unit 1020 may determine the degree of anomaly according to the degree of deviation of the acoustic data to be detected from the distribution of the reference acoustic data. Also, the first sound anomaly detection unit 1020 may calculate, as the degree of anomaly, the sum of the degrees of difference for each of a predetermined number of acoustic data with the smallest difference from the acoustic data to be detected among the reference acoustic data (k-nearest neighbor method). In addition to the above, the first sound anomaly detection unit 1020 may calculate the degree of anomaly of the acoustic data to be detected using various methods for calculating the difference, dissimilarity, similarity, or degree of deviation between two sounds.

[0110] The first sound anomaly detection unit 1020 may detect the degree of anomaly of the acoustic data to be detected using AI technology. The first sound anomaly detection unit 1020 may cause the acoustic data in each detection direction in the first direction in each area within the facility 1 to be learned by a machine learning model. The first sound anomaly detection unit 1020 may store the machine learning model or the learned parameters representing the machine learning model in the collection information database 50 or the like, and may update the machine learning model each time new acoustic data in the first direction in each area is received. Also, the first sound anomaly detection unit 1020 may receive the determination result of normal or abnormal for the acoustic data in each detection direction in the first direction in each area. In this case, the first sound anomaly detection unit 1020 may generate or update the machine learning model by learning so that when the acoustic data is input to the machine learning model, the correct determination result is output.

[0111] The first sound abnormality detection unit 1020 may use a neural network, statistical learning, or other machine learning algorithms. The first sound abnormality detection unit 1020 may use time-series acoustic data as an input to the machine learning model, and may use the frequency spectrum of the sound components indicated by the acoustic data as an input to the machine learning model. The first sound abnormality detection unit 1020 updates the parameters of the machine learning model so as to reduce the error between the output of the model when each sample of the acoustic data serving as learning data is input to the machine learning model and the learning label indicating whether the acoustic data is normal or abnormal. For example, when using a neural network, the first sound abnormality detection unit 1020 inputs the data value at each time or the data value at each frequency in the acoustic data to each input node of the input layer of the neural network. The first sound abnormality detection unit 1020 uses the error between the output value output by the neural network in response to inputting each sample and the label, and adjusts the weights between the neurons of the neural network and the biases of each neuron by a method such as backpropagation.

[0112] Note that the machine learning model may output the degree of abnormality of the acoustic data to be detected in response to inputting the reference acoustic data and the acoustic data to be detected. Also, the machine learning model may be common for all detection directions in each area within Facility 1 or for all areas within Facility 1.

[0113] For example, by using the machine learning model learned in this way, in response to receiving the acoustic data to be detected in the first direction, the first sound abnormality detection unit 1020 reads out from the collection information database 50 the machine learning model corresponding to the position and detection direction where the acoustic data was detected, inputs the acoustic data to be detected, and can use the determination result of normal or abnormal output by the machine learning model as the degree of abnormality. The first sound abnormality detection unit 1020 may calculate the degree of abnormality based on the difference between the reference acoustic data and the acoustic data to be detected using any other method.

[0114] In S1160, the abnormal location specifying unit 1037 specifies an abnormal location in the abnormal area by using the acoustic data detected by the second acoustic sensor 400a at at least one of each position or each direction within the abnormal area. The abnormal location specifying unit 1037 may specify, for each detection direction in the first direction, a location where abnormal sound occurs as an abnormal location.

[0115] The abnormal location specifying unit 1037 may determine that abnormal sound is occurring in the detection direction when the abnormality degree of the acoustic data to be detected in the detection direction in the first direction exceeds a predetermined threshold value. When the abnormal location specifying unit 1037 determines that abnormal sound is occurring in a plurality of detection directions in the first direction in the abnormal area, it may determine that abnormal sound is occurring in the detection direction with the highest abnormality degree. The abnormal location specifying unit 1037 may specify a predetermined range located in the detection direction where abnormal sound is determined to be occurring as an abnormal location. For example, the abnormal location specifying unit 1037 may specify, as an abnormal location, the range of a straight line extending from the second robot 30 or the sensor unit at the position where the acoustic data to be detected is detected, in the detection direction where abnormal sound is determined to be occurring, or a predetermined error range centered on the straight line.

[0116] The abnormal location specifying unit 1037 may supply data indicating that an abnormal location has been specified to the second sound source separation unit 1030 and the robot command unit 1060. The abnormal location specifying unit 1037 may supply data indicating the abnormal location (such as the position and detection direction where the acoustic data to be detected in the abnormal area is detected) to the abnormal device specifying unit 1040. When the abnormal location specifying unit 1037 specifies an abnormal location in at least one detection direction in the first direction of the abnormal area (i.e., "Yes" in S1160), the monitoring device 40 proceeds to S1200. When no abnormal location is specified in any detection direction in the first direction of the abnormal area (i.e., "No" in S1160), the monitoring device 40 proceeds to S1100.

[0117] In S1200, the second sound source separation unit 1030 acquires acoustic data detected by the acoustic sensor 400a mounted on the second robot 30 via the collection information database connection unit 620, and extracts at least one sound component by separating the sound source in the second direction from the acoustic data.

[0118] The second sound source separation unit 1030 may acquire acoustic data detected by an acoustic sensor 400a (for example, the second acoustic sensor 400a) having higher directivity than the first acoustic sensor 400a in the abnormal area where the abnormal location in the first direction is specified. When a plurality of second acoustic sensors 400a mounted on the second robot 30 are arranged in a one-dimensional array, the robot command unit 1060 changes the orientation so as to arrange the plurality of second acoustic sensors 400a arranged in the first direction in the second direction in response to the specification of the abnormal location in the first direction by the abnormal location specifying unit 1037, and may instruct the second robot 30 to collect acoustic data for separating the sound source in the second direction. The second sound source separation unit 1030 acquires the acoustic data detected in response to the instruction via the collection information database connection unit 620. Further, when a plurality of second acoustic sensors 400a mounted on the second robot 30 are two-dimensionally arranged in the first direction and the second direction, the second sound source separation unit 1030 may acquire the acoustic data detected by the plurality of second acoustic sensors 400a simultaneously with the acoustic data used for specifying the abnormal location in the first direction via the collection information database connection unit 620.

[0119] The second sound source separation unit 1030 may separate the acquired acoustic data into sound sources in a plurality of detection directions in the second direction in the detection direction in which the abnormal portion in the first direction is specified by the abnormal portion specifying unit 1037. By separating the acoustic data into sound sources in the second direction, the second sound source separation unit 1030 can separate the acoustic data into sound components in a plurality of detection directions in the second direction (for example, on a vertical plane along the detection direction in which abnormal sounds on the horizontal plane are detected) centered on the second robot 30 (or the sensor unit including the plurality of acoustic sensors 400a) in the abnormal area. The second sound source separation unit 1030 performs a process of separating or decomposing the sound collected by each acoustic sensor 400a into the sound generated by the sound source in each detection direction in the second direction. The second sound source separation unit 1030 may utilize various sound source separation and sound source localization techniques.

[0120] The second sound source separation unit 1030 may calculate the sound components from each detection direction in the second direction with respect to the second robot 30 from the time difference at which each sound component included in the acoustic data from the plurality of acoustic sensors 400a arranged in the second direction reaches each acoustic sensor 400a. Here, the second sound source separation unit 1030 may calculate the absolute direction of the sound source of each sound component at the observation point by adding the absolute direction (measurement direction) in which the sensor unit itself is facing to the relative direction (the direction in which the sound source is separated) of the sound source of each sound component with respect to the measurement direction of the sensor unit including the plurality of acoustic sensors 400a.

[0121] Further, the second robot 30 may acquire acoustic data while changing the orientation of the acoustic sensor 400a within the range in the second direction with respect to the second robot 30. The second sound source separation unit 1030 may calculate the magnitude (sound pressure) of at least one sound component within the range in which the orientation of the acoustic sensor 400a is changed using such acoustic data.

[0122] In S1210, the second sound abnormality detection unit 1035 calculates the respective abnormality degrees using the acoustic data separated into sound sources in a plurality of detection directions in the second direction (hereinafter also referred to as the acoustic data to be detected). Here, the abnormality degree may be represented by a real number or an integer within a predetermined range such as from 0 to 1 or from 0 to 100%, or may be a binary value such as normal or abnormal.

[0123] The second sound abnormality detection unit 1035 may calculate, as an abnormality degree, a value indicating how different the acoustic data to be detected is from the acoustic data (hereinafter also referred to as reference acoustic data) when all the devices 10 in the facility 1 or all the devices 10 in a specified abnormal area are normal.

[0124] Here, when the past acoustic data in the same detection direction in the first direction and the second direction at the same detection position in the abnormal area when all the devices 10 in the facility 1 or all the devices 10 in the abnormal area are normal is recorded as one or more histories in the collection information database 50, the second sound abnormality detection unit 1035 may use the recorded past acoustic data as the reference acoustic data for calculating the abnormality degree.

[0125] For example, the second sound abnormality detection unit 1035 may determine the abnormality degree according to the comparison result between the reference acoustic data and the acoustic data to be detected. The second sound abnormality detection unit 1035 may perform the comparison between the reference acoustic data and the acoustic data to be detected in either the time domain or the frequency domain. When the comparison is performed in the time domain, the second sound abnormality detection unit 1035 adjusts the phase difference so that the time integral of the difference (such as the absolute value of the difference) between the reference acoustic data and the acoustic data to be detected is minimized, and may use the time integral of the difference between the reference acoustic data and the acoustic data to be detected at the adjusted phase difference as the abnormality degree. In this case, the second sound abnormality detection unit 1035 may adjust the amplitude so that, for example, the average amplitudes match in order to match the loudness of the reference acoustic data and the acoustic data to be detected.

[0126] When the comparison between the reference acoustic data and the acoustic data to be detected is performed in the frequency domain, the second sound abnormality detection unit 1035 may calculate the abnormality degree by integrating, in the frequency direction, the difference (such as the absolute value of the difference) for each frequency between the frequency spectrum of the reference acoustic data and the frequency spectrum of the acoustic data to be detected. In this case, the second sound abnormality detection unit 1035 may perform an adjustment to match the loudness so that the reference acoustic data and the acoustic data to be detected can be compared.

[0127] The second sound abnormality detection unit 1035 may determine the degree of abnormality according to the degree of deviation of the acoustic data to be detected with respect to the distribution of the reference acoustic data. Further, the second sound abnormality detection unit 1035 may calculate, as the degree of abnormality, the sum of the degrees of difference for each of a predetermined number of acoustic data having the smallest difference from the acoustic data to be detected among the reference acoustic data (k-nearest neighbor method). In addition to the above, the second sound abnormality detection unit 1035 may calculate the degree of abnormality of the acoustic data to be detected using various methods for calculating the difference, similarity, or degree of deviation between two sounds.

[0128] The second sound abnormality detection unit 1035 may detect the degree of abnormality of the acoustic data to be detected using AI technology. The second sound abnormality detection unit 1035 may cause the acoustic data in each detection direction in the second direction in each area in the facility 1 to be learned by a machine learning model. The second sound abnormality detection unit 1035 may store the machine learning model or the learned parameters representing the machine learning model in the collection information database 50 or the like, and may update the machine learning model each time newly received acoustic data in the second direction in each area. Further, the second sound abnormality detection unit 1035 may receive a determination result of normal or abnormal for the acoustic data in each detection direction in the second direction in each area. In this case, the second sound abnormality detection unit 1035 may generate or update the machine learning model by learning so that the correct determination result is output when the acoustic data is input to the machine learning model.

[0129] The second sound abnormality detection unit 1035 may use a neural network, statistical learning, or other machine learning algorithms. The second sound abnormality detection unit 1035 may use time-series acoustic data as an input to the machine learning model, and may use the frequency spectrum of the sound components indicated by the acoustic data as an input to the machine learning model. The second sound abnormality detection unit 1035 updates the parameters of the machine learning model so as to reduce the error between the output of the model when each sample of the acoustic data serving as learning data is input to the machine learning model and the learning label indicating whether the acoustic data is normal or abnormal. For example, when using a neural network, the second sound abnormality detection unit 1035 inputs the data value at each time or the data value at each frequency in the acoustic data to each input node of the input layer of the neural network. The second sound abnormality detection unit 1035 uses the error between the output value output by the neural network in response to inputting each sample and the label, and adjusts the weights between the neurons of the neural network and the biases of each neuron by a method such as backpropagation.

[0130] Note that the machine learning model may output the degree of abnormality of the acoustic data to be detected in response to inputting the reference acoustic data and the acoustic data to be detected. Also, the machine learning model may be common for all detection directions in each area within Facility 1 or for all areas within Facility 1.

[0131] For example, by using the machine learning model learned in this way, in response to receiving the acoustic data to be detected in the second direction, the second sound abnormality detection unit 1035 reads out from the collection information database 50 the machine learning model corresponding to the position and detection direction where the acoustic data was detected, inputs the acoustic data to be detected, and can use the determination result of normal or abnormal output by the machine learning model as the degree of abnormality. The second sound abnormality detection unit 1035 may calculate the degree of abnormality based on the difference between the reference acoustic data and the acoustic data to be detected using any other method.

[0132] In S1220, the abnormal location specifying unit 1037 may specify, for each detection direction in the second direction, a location where abnormal noise occurs as an abnormal location. The abnormal location specifying unit 1037 may determine that abnormal noise is occurring in a detection direction when the degree of abnormality of the acoustic data exceeds a predetermined threshold for each detection direction in the second direction. When the abnormal location specifying unit 1037 detects abnormal noise in a plurality of detection directions in the second direction in the abnormal area, it may determine that abnormal noise is occurring in the detection direction with the highest degree of abnormality. The abnormal location specifying unit 1037 may specify a predetermined range located in the detection direction in which it is determined that abnormal noise is occurring as an abnormal location. For example, the abnormal location specifying unit 1037 may specify, as an abnormal location, the range of a straight line extending from the second robot 30 or the sensor unit at the position where the acoustic data to be detected is detected, in the detection direction in which it is determined that abnormal noise is occurring, or a predetermined error range centered on the straight line.

[0133] The abnormal location specifying unit 1037 may supply data indicating an abnormal location (such as the position and detection direction where the acoustic data to be detected in the abnormal area is detected) to the abnormal device specifying unit 1040. When the abnormal location specifying unit 1037 specifies an abnormal location in at least one detection direction in the second direction of the abnormal area (i.e., "Yes" in S1220), the monitoring device 40 proceeds to S1230. When the abnormal location specifying unit 1037 does not specify an abnormal location in any detection direction in the second direction of the abnormal area (i.e., "No" in S1220), the monitoring device 40 proceeds to S1100.

[0134] In S1230, the abnormal device identification unit 1040 identifies the device 10 in which an abnormality has occurred, based on the abnormal area, the detection direction in the first direction, and the detection direction in the second direction. For example, the abnormal device identification unit 1040 may identify, by searching the device database 60, the device 10 provided in the detection direction in the second direction in the detection direction in the first direction, which is identified as an abnormal location by the abnormal location identification unit 1037, with reference to the abnormal area identified by the abnormal area identification unit 1000. For example, the abnormal device identification unit 1040 may identify, as the device 10 in which an abnormality has occurred, the device 10 provided at a position where a straight line extending from the second robot 30 in the abnormal area or the sensor unit mounted on the second robot 30 in the detection direction in the second direction (vertical direction) in the detection direction in the first direction (horizontal direction) intersects or substantially intersects within a predetermined error range.

[0135] The abnormal device identification unit 1040 may further identify a more detailed position where an abnormality has occurred in the device 10, based on the position of the identified device 10 and the detection direction in the second direction identified as an abnormal location by the abnormal location identification unit 1037 with reference to the abnormal area identified by the abnormal area identification unit 1000. For example, the abnormal device identification unit 1040 may identify, as the abnormal occurrence position, a position where a straight line extending from the second robot 30 in the abnormal area or the sensor unit mounted on the second robot 30 in the detection direction in the second direction (vertical direction) in the detection direction in the first direction (horizontal direction) intersects or substantially intersects within a predetermined error range with the identified device 10.

[0136] The abnormal device identification unit 1040 may cause the display processing unit 665 to display, on the display device 690, a display screen that displays the device 10 in which an abnormality has occurred and the position where the abnormality has occurred in the device 10. In addition, when the abnormal device identification unit 1040 cannot identify the device 10 in which an abnormality has occurred from the abnormal area, the detection direction in the first direction, and the detection direction in the second direction in the device database 60, the abnormal device identification unit 1040 may receive, via the input processing unit 660, information input from the user for identifying the device 10 in which an abnormality has occurred. In this case, the abnormal device identification unit 1040 may cause the display processing unit 665 to display, on the display device 690, a display screen that displays the abnormal area, the detection direction in the first direction identified as the abnormal location, and the detection direction in the second direction on the map of Facility 1. Thereby, the abnormal device identification unit 1040 may register the device data of the device 10 corresponding to the information received by the input processing unit 660 in the device database 60.

[0137] In S1240, the device control unit 1050 performs control for dealing with the abnormality of the device 10 on at least one of the device 10 in which an abnormality has been identified and another device 10. The device control unit 1050 transmits an instruction to perform control for dealing with the abnormality of the device 10 to the control system 20 via the communication unit 670.

[0138] As an example, the device control unit 1050 may instruct the control system 20 to perform control to stop the operation of the device 10 in which an abnormality has occurred. The device control unit 1050 may instruct the control system 20 to perform control to stop another device 10 that is related to the device 10 in which an abnormality has occurred or the position where the abnormality has occurred in the device 10 (for example, a device 10 in the upstream or downstream process of the device 10 in which an abnormality has occurred, etc.). The device control unit 1050 may also instruct the control system 20 to perform control to stop the operations of both the device 10 in which an abnormality has occurred and another device 10 that is related to the device 10 in which an abnormality has occurred. The monitoring device 40 proceeds to S1100 and continues the monitoring inside Facility 1 by the robot 30.

[0139] FIG. 13 shows an explanatory diagram for explaining the identification of an abnormal area using the acoustic data detected by the first acoustic sensor 400a. In the example of FIG. 13, four devices 10 and the first robot 30 in the facility 1 are shown together with the XYZ axes. In FIG. 13, the detection range of the acoustic data in the first acoustic sensor 400a is indicated by a dashed line. The first robot 30 moves within the facility 1 and detects acoustic data with the same sensitivity by the first acoustic sensor 400a within a 360-degree range centered on the first robot 30. The abnormal area identification unit 1000 identifies, as an abnormal area, an area among a plurality of areas in the facility 1 where the sound pressure of the acoustic data exceeds a predetermined threshold value. In the example of FIG. 13, the abnormal area identification unit 1000 identifies the area as an abnormal area in response to the sound pressure of the acoustic data detected by the first acoustic sensor 400a exceeding the predetermined threshold value in the area including the device 10d where an abnormality has occurred.

[0140] FIG. 14 shows an explanatory diagram for explaining the sound source separation by the first sound source separation unit 1010. In the example of FIG. 14, four devices 10 and the second robot 30 in the facility 1 are shown together with the XYZ axes indicating the same direction as in FIG. 13. The second robot 30 is different from the first robot 30. In FIG. 14, each detection direction in the horizontal direction is indicated by a dashed arrow. In the present embodiment, the first sound source separation unit 1010 performs sound source separation in the detection directions of 0°, α°, 2α°, 3α°, -α°, -2α°, and -3α° (α>0) in the horizontal direction (XY plane) with reference to the second robot 30 in the same manner as S1140. Here, the detection direction of 0° is, as an example, the measurement direction in the horizontal direction of the sensor unit including the second acoustic sensor 400a, and each detection direction in FIG. 14 is shown as a relative direction.

[0141] The first sound source separation unit 1010 may acquire acoustic data from two second acoustic sensors 400a arranged horizontally, for example, in the same manner as S1140, and calculate acoustic data from the sound source in each detection direction based on the time difference of arrival at each second acoustic sensor 400a. The first sound source separation unit 1010 may calculate acoustic data from each detection direction by changing the time difference according to the detection direction. The first sound anomaly detection unit 1020 calculates the degree of anomaly for each detection direction in the same manner as S1150, and the anomaly location identification unit 1037 compares the degree of anomaly with a predetermined threshold in the same manner as S1160. As an example, the anomaly location identification unit 1037 determines that abnormal sound is generated in the detection direction 3α° where the degree of anomaly exceeds the predetermined threshold, thereby identifying that there is a device 10d with an anomaly in the range of the direction of the detection direction 3α°.

[0142] FIG. 15 shows an explanatory diagram for explaining sound source separation by the second sound source separation unit 1030. In the example of FIG. 15, together with the XYZ axes indicating the same direction as in FIG. 13, the device 10d and the second robot 30 in the facility 1 are shown. In FIG. 15, each detection direction in the vertical direction is indicated by a dashed arrow. In the example of FIG. 15, the second sound source separation unit 1030 performs sound source separation in the vertical direction toward the horizontal detection direction 3α° specified as the abnormal location by the anomaly location identification unit 1037 with reference to the second robot 30. In the present embodiment, the second sound source separation unit 1030 performs sound source separation in the detection directions of 0°, β°, and 2β° (β>0) in the vertical direction. Here, the detection direction of 0° is, for example, the measurement direction in the vertical direction of the sensor unit including the second acoustic sensor 400a, and each detection direction is shown as a relative direction in FIG. 15.

[0143] The second sound source separation unit 1030 may acquire acoustic data detected by two second acoustic sensors 400a arranged, for example, in the vertical direction, in the same manner as S1200, and calculate acoustic data from the sound source in each detection direction from the time difference at which the acoustic data reaches each second acoustic sensor 400a. The second sound source separation unit 1030 may calculate the acoustic data from the detection direction by changing the time difference according to each detection direction. The second sound anomaly detection unit 1035 calculates the degree of anomaly for each detection direction in the vertical direction in the same manner as S1210, and the anomaly location identification unit 1037 compares the degree of anomaly with a predetermined threshold value in the same manner as S1220. The second sound anomaly detection unit 1035 determines, as an example, that the degree of anomaly in the detection direction 2β° exceeds a predetermined threshold value. The abnormal device identification unit 1040 identifies the device 10d provided at the position where the straight line from the second robot 30 toward the detection direction 2β° intersects, from the device database 60. The abnormal device identification unit 1040 may further identify the vertical position in the device 10d where the straight line from the second robot 30 toward the detection direction 2β° intersects as the abnormal occurrence position.

[0144] The monitoring device 40 monitors the entire facility 1 using the acoustic data from the acoustic sensor 400a with low directivity, and can identify the device 10 in which an abnormality has occurred using the acoustic data from the acoustic sensor 400a with high directivity detected within the abnormal area when the abnormal area is identified. Thereby, the monitoring device 40 can efficiently acquire and process the acoustic data in identifying the device 10 in which an abnormality has occurred, and can further execute appropriate countermeasures against the abnormality of the device 10.

[0145] In addition, when the first robot 30 is equipped with the first acoustic sensor 400a and the second acoustic sensor 400a, in S1110 and S1130, the acoustic data acquisition unit 602 may acquire the first acoustic data and the second acoustic data detected by the first acoustic sensor 400a and the second acoustic sensor 400a mounted on the first robot 30 that can move within the facility 1. Thereby, the abnormal area specifying unit 1000 specifies the abnormal area using the first acoustic data detected by the first acoustic sensor 400a in the same manner as in S1120, and the abnormal location specifying unit 1037 may specify the abnormal location using the second acoustic data detected by the second acoustic sensor 400a in the same manner as from S1130 to S1220. Also in this embodiment, since the device 10 in which an abnormality has occurred step by step according to the directivity can be specified, it is efficient. In addition, the time required for the movement of the robot 30 for the detection of the acoustic data by the second acoustic sensor 400a can be shortened, and the countermeasure against the abnormality can be executed promptly.

[0146] Further, the monitoring device 40 may not include the second sound source separation unit 1030 and the second sound abnormality detection unit 1035. Even in this case, by the abnormal location specifying unit 1037 specifying the abnormal location in the first direction, the abnormal device specifying unit 1040 can specify the device 10 in which the abnormality has occurred at the abnormal location.

[0147] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (1) stages of a process in which an operation is performed or (2) sections of a device having a role of performing an operation. Specific stages and sections may be implemented by a dedicated circuit, a programmable circuit supplied with computer-readable instructions stored on a computer-readable medium, and / or a processor supplied with computer-readable instructions stored on a computer-readable medium. The dedicated circuit may include digital and / or analog hardware circuits, and may include an integrated circuit (IC) and / or discrete circuits. The programmable circuit may include a reconfigurable hardware circuit including memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, a field programmable gate array (FPGA), a programmable logic array (PLA), etc.

[0148] The computer-readable medium may include any tangible device capable of storing instructions executable by an appropriate device, and as a result, a computer-readable medium having instructions stored therein will comprise a product including instructions executable to create means for performing the operations specified in the flowchart or block diagram. Examples of the computer-readable medium may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of the computer-readable medium may include a floppy (registered trademark) disk, a diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a Blu-ray (registered trademark) disc, a memory stick, an integrated circuit card, etc.

[0149] Computer-readable instructions may include any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in an object-oriented programming language such as Smalltalk®, JAVA®, C++, and a conventional procedural programming language such as the "C" programming language or a similar programming language.

[0150] Computer-readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus such as a general-purpose computer, a special-purpose computer, or other computer, either locally or via a wide area network (WAN) such as a local area network (LAN), the Internet, etc., and may be executed to create means for performing the operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0151] FIG. 16 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as an operation associated with the apparatus according to an embodiment of the present invention or as one or more sections of the apparatus, or to execute the operation or the one or more sections, and / or may cause the computer 2200 to execute a process according to an embodiment of the present invention or a stage of the process. Such a program may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0152] The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are mutually connected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0153] The CPU 2212 operates according to programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphic controller 2216 acquires image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or in itself, and causes the image data to be displayed on the display device 2218.

[0154] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads a program or data from the DVD-ROM 2201 and provides the program or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0155] ROM 2230 stores therein a boot program or the like executed by computer 2200 at activation and / or a program dependent on the hardware of computer 2200. Input / output chip 2240 may also be connected to input / output controller 2220 via various input / output units through a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0156] The program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium, installed in hard disk drive 2224, RAM 2214, or ROM 2230, which is also an example of a computer-readable medium, and executed by CPU 2212. The information processing described in these programs is read by computer 2200, resulting in cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be configured by realizing the operation or processing of information according to the use of computer 2200.

[0157] For example, when communication is executed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded in RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 2214, hard disk drive 2224, DVD-ROM 2201, or an IC card under the control of CPU 2212, transmits the read transmission data to the network, or writes the received data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0158] Further, the CPU 2212 may cause all or necessary parts of files or databases stored in external recording media such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and execute various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording media.

[0159] Various types of information such as various types of programs, data, tables, and databases may be stored in the recording media and may undergo information processing. The CPU 2212 may execute various types of processing on the data read from the RAM 2214, including various types of operations, information processing, condition judgment, conditional branch, unconditional branch, information search / replacement, etc. described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 2214. Further, the CPU 2212 may search for information in files, databases, etc. within the recording media. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording media, the CPU 2212 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0160] The programs or software modules described above may be stored in a computer-readable medium on or near the computer 2200. Also, a recording medium such as a hard disk or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network.

[0161] As described above, the present invention has been explained using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0162] It should be noted that the execution order of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the specification, and the drawings is not explicitly indicated as "earlier" or "preceding" etc., and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the specification, and the drawings, even if it is described for convenience using "first," "next," etc., it does not mean that it is essential to implement in this order.

Description of Reference Numerals

[0163] 1 Facility 10 Equipment 20 Control System 30 Robot 40 Monitoring Device 50 Collected Information Database 60 Equipment Database 70 Robot Database 210 State Acquisition Unit 220 State Judgment Unit 230 Display Processing Unit 240 Display Device 250 Input Device 260 Instruction Input Unit 270 Control Unit 400a~c Sensor 410a~b Sensor 420 Actuator 430 State Acquisition Unit 440 Communication Unit 450 Control Unit 600 Communication Unit 602 Acoustic Data Acquisition Unit 604 Image Data Acquisition Unit 606 LiDAR Data Acquisition Unit 608 Position Data Acquisition Unit 610 Direction Data Acquisition Unit 612 Instruction Sending Unit 620 Collection Information Database Connection Unit 630 Equipment Database Connection Unit 640 Robot Database Connection Unit 650 Monitoring Processing Unit 660 Input Processing Unit 665 Display Processing Unit 670 Communication Unit 680 Input Device 690 Display Device 1000 Abnormal Area Identification Unit 1010 First Sound Source Separation Unit 1020 First Sound Abnormality Detection Unit 1030 Second Sound Source Separation Unit 1035 Second Sound Abnormality Detection Unit 1037 Abnormal Location Identification Unit 1040 Abnormal Equipment Identification Unit 1050 Equipment Control Unit 1060 Robot Command Unit 2200 Computer 2201 DVD-ROM 2210 Host Controller 2212 CPU 2214 RAM 2216 Graphics Controller 2218 Display Device 2220 Input / Output Controller 2222 Communication Interface 2224 Hard Disk Drive 2226 DVD-ROM Drive 2230 ROM 2240 Input / Output Chip 2242 Keyboard

Claims

1. An acoustic data acquisition unit that acquires acoustic data detected within a facility; An abnormal area specifying unit that specifies an abnormal area where there may be a device in which an abnormality has occurred within the facility, using the first acoustic data detected by a first acoustic sensor; An abnormal location specifying unit that specifies an abnormal location within the abnormal area, using the second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor A device comprising:

2. The device according to claim 1, wherein the acoustic data acquisition unit acquires the first acoustic data detected by the first acoustic sensor mounted on a first robot capable of moving within the facility and the second acoustic data detected by the second acoustic sensor mounted on a second robot capable of moving within the facility.

3. The device according to claim 2, further comprising a robot command unit that instructs the second robot to head towards the abnormal area in response to the abnormal area being specified.

4. The robot command unit instructs the second robot to detect the second acoustic data while changing at least one of the position or direction of the second acoustic sensor in response to the second robot arriving at a position where it can detect the second acoustic data from within the abnormal area, The abnormal location specifying unit specifies the abnormal location using the second acoustic data detected by the second acoustic sensor at at least one of each position or each direction from within the abnormal area. The device according to claim 3.

5. The device according to claim 1, wherein the acoustic data acquisition unit acquires the first acoustic data and the second acoustic data detected by the first acoustic sensor and the second acoustic sensor mounted on a third robot capable of moving within the facility.

6. The device according to claim 1, further comprising an abnormal device specifying unit that specifies a device corresponding to the position of the abnormal location as a device in which an abnormality has occurred.

7. The device according to claim 6, further comprising a device control unit that performs control for dealing with an abnormality of one device on at least one of the one device or another device within the facility in response to the abnormality of the one device being detected.

8. A step of acquiring acoustic data detected within a facility; Identifying an abnormal area where there may be a device in which an abnormality has occurred within the facility, using the first acoustic data detected by the first acoustic sensor; Identifying an abnormal location within the abnormal area, using the second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor; A method comprising the steps of. **Claim 9** A program executed by a computer, causing the computer to function as an acoustic data acquisition unit that acquires acoustic data detected within a facility; function as an abnormal area identification unit that identifies an abnormal area where there may be a device in which an abnormality has occurred within the facility, using the first acoustic data detected by the first acoustic sensor; function as an abnormal location identification unit that identifies an abnormal location within the abnormal area, using the second acoustic data from within the abnormal area detected by a second acoustic sensor having higher directivity than the first acoustic sensor; and function as a program.

Images