Device, method, and program for processing acoustic data detected in facility

Abstract

Provided is a device comprising: an acoustic data acquisition unit that acquires acoustic data detected in a facility; an image data acquisition unit that acquires data of an image captured in the facility; a generation source identification unit that identifies which apparatus among at least one apparatus imaged in the image data serves as a generation source for each sound component among at least one sound component included in acoustic data; and an apparatus database connection unit that records, in an apparatus database, sound component data pertaining to each sound component among the at least one sound component in association with an apparatus that is the generation source among the at least one apparatus.

Description

Apparatus, method, and program for processing acoustic data detected within a facility

[0001] The present invention relates to an apparatus, method, and program for processing acoustic data detected within a facility.

[0002] Patent Document 1 describes that "the inspection support system 1 shown in FIG. 1 enables an inspector 4 to inspect equipment at an inspection site 2 at a monitoring location 3 remote from the inspection site 2" (paragraph 0010), that "sound information from the unmanned mobile body 10 includes sound information acquired by each of a plurality of microphones 131 to 13m, and the sound determination unit 72 can determine the direction, position, and volume of the sound source based on this sound information" (paragraph 0037), and that "the graphic addition unit 73 adds a graphic representing at least one of a graphic representing the position of the sound source and a graphic representing the volume of the sound to the captured image acquired from the unmanned mobile body 10 based on the determination result by the sound determination unit 72" (paragraph 0038). [Prior Art Literature] [Patent Document] [Patent Document 1] JP 2020-149349 A General disclosure

[0003] A first aspect of the present invention provides an apparatus including: an acoustic data acquisition unit that acquires acoustic data detected within a facility; an image data acquisition unit that acquires image data captured within the facility; a source identification unit that identifies which of at least one device captured in the image data is the source of each of at least one sound component included in the acoustic data; and an equipment database connection unit that records sound component data for each of the at least one sound component in an equipment database in association with the device that is the source of the at least one device.

[0004] The above device may include a sound source separation unit that performs sound source separation on the acoustic data to extract the at least one sound component.

[0005] In any of the above devices, the sound source separation section may extract the at least one sound component having sound source directions different from each other.

[0006] Any of the above devices may include a device detection unit that detects the at least one device captured in the image data by performing image recognition on the image data.

[0007] In any of the above devices, the device detection section may detect each of the at least one device included in the image data by matching it with a predetermined template.

[0008] In any of the above devices, the source identification unit may identify, for each of the at least one sound component, a device among the at least one device that is located in an area within the image data that corresponds to the direction of the sound source as the source device.

[0009] Any of the above devices may include an input processing unit that, when a device located in an area within the image data corresponding to a direction of a sound source for any of the at least one sound component cannot be identified, accepts input of information from a user for identifying the device. The source identification unit may register the device in the device database via the information input by the user via the input processing unit.

[0010] In any of the above devices, the acoustic data acquisition unit and the image data acquisition unit may acquire the acoustic data and the image data from a robot that can move within the facility and collect sound and take images.

[0011] Any of the above devices may include a robot command unit that instructs the robot to change at least one of the imaging direction of the image, the imaging magnification of the image, or the position of the robot, in response to the inability to identify the source device of any of the at least one sound component among the at least one device.

[0012] Any of the above devices may include a sound abnormality detection unit that detects the degree of abnormality of a sound component using the sound component data, and an abnormal device identification unit that identifies an abnormality in a device that is a source of the sound component based on the degree of abnormality of the sound component. In any of the above devices, the sound abnormality detection unit may determine the degree of abnormality based on a comparison result between normal sound data emitted by the device that is a source of the sound component and the sound component data. In any of the above devices, the sound abnormality detection unit may compare the normal sound data with the target sound component data in the time domain or the frequency domain. In any of the above devices, the sound abnormality detection unit may determine the degree of abnormality based on a comparison result between sound data previously collected from the device that is a source of the sound component and the target sound component data. In any of the above devices, the sound abnormality detection unit may determine the degree of abnormality of the target sound component data based on normal sound data or previously generated sound data emitted by a device of the same type or model as the device that is a source of the sound component. In any of the above devices, the abnormal device identifying unit may cause a display device to display a display screen that displays the status of each device.

[0013] Any of the above devices may be provided with an equipment control unit that, in response to identification of an abnormality in a piece of equipment that is the source of the sound component, controls the piece of equipment or at least one other piece of equipment within the facility to deal with the abnormality in the piece of equipment.

[0014] In a second aspect of the present invention, there is provided a method comprising: acquiring acoustic data detected within a facility; acquiring image data captured within the facility; identifying which of at least one device captured in the image data is the source of each of at least one sound component included in the acoustic data; and recording sound component data relating to each of the at least one sound component in an equipment database in association with the device that is the source of the at least one device.

[0015] In a third aspect of the present invention, there is provided a program that is executed by a computer and causes the computer to function as: an acoustic data acquisition unit that acquires acoustic data detected within a facility; an image data acquisition unit that acquires image data captured within the facility; a source identification unit that identifies which of at least one device captured in the image data is the source of each sound component of at least one sound component included in the acoustic data; and an equipment database connection unit that records sound component data for each sound component of the at least one sound component in an equipment database in association with the device that is the source of the sound component of the at least one device.

[0016] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions.

[0017] 1 shows a schematic configuration of a facility 1 according to the present embodiment. 2 shows a configuration of a control system 20 according to the present embodiment. 3 shows a processing flow of the control system 20 according to the present embodiment. 4 shows a configuration of a robot 30 according to the present embodiment. 5 shows a processing flow of the robot 30 according to the present embodiment. 6 shows a configuration of a monitoring device 40 according to the present embodiment. 7 shows an example of the data structure of a collected information database 50 according to the present embodiment. 8 shows an example of the data structure of an equipment database 60 according to the present embodiment. 9 shows an example of the data structure of a robot database 70 according to the present embodiment. 10 shows a configuration of a monitoring processing unit 650 of a monitoring device 40 according to the present embodiment. 11 shows a processing flow of the monitoring device 40 according to the present embodiment. 12 shows a processing flow of the monitoring device 40 according to the present embodiment. 13 shows an example of image data 1300 captured by a robot 30 according to the present embodiment. 14 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied in whole or in part.

[0018] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0019] FIG. 1 shows a schematic configuration of a facility 1 according to this embodiment. The facility 1 may be the entirety or a partial segment of a factory or plant, in which multiple devices 10 are located. Examples of such factories or plants include factories for producing various industrial products, chemical or metal industrial plants, plants for managing and controlling wellheads and their surrounding areas, plants for managing and controlling hydroelectric, thermal, or nuclear power generation, plants for managing and controlling solar or wind energy generation, and plants for managing and controlling water supply, sewage, dams, and the like. The facility 1 may also be the entirety or a part of a building or transportation facility equipped with multiple devices 10 to be controlled and monitored. The facility 1 includes multiple devices 10, a control system 20, one or more robots 30, a monitoring device 40, a collected information database 50, an equipment database 60, and a robot database 70.

[0020] The plurality of devices 10 are provided at various locations within the facility 1. Each of the plurality of devices 10 may be installed either indoors or outdoors within the area of ​​the facility 1. At least some of the devices 10 may be process devices, power generation devices, or any other devices (or facility equipment) that are controlled by the control system 20, or may be parts of such devices. At least some of the devices 10 may be field devices that operate under control from the control system 20, other devices, or operators. Furthermore, at least some of the devices 10 may be field devices themselves.

[0021] Such field devices may be, for example, sensor devices such as pressure gauges, flow meters, and temperature sensors, valve devices such as flow control valves and on-off valves, actuator devices such as fans and motors, imaging devices such as cameras or videos that capture images of the situation of a plant or an object such as equipment, audio devices such as microphones or speakers that collect abnormal sounds from a plant or equipment or emit alarm sounds, position detection devices that output position information of devices in the facility 1, or other devices. Furthermore, other devices 10 among the plurality of devices 10 may be pipes, storage tanks, supports, bulkheads, or other structures that are not controlled by the control system 20.

[0022] The control system 20 is connected to at least one device 10 to be controlled among the plurality of devices 10. The control system 20 may be, for example, a distributed control system (DCS). The control system 20 controls each device 10 in accordance with the state of each device 10 measured by a sensor or the like provided in each device 10 to be controlled.

[0023] Each of the one or more robots 30 is used to monitor at least some of the devices 10 in the facility 1. Each robot 30 can move within the facility 1 and collect sound, take images, and the like.

[0024] The monitoring device 40 is communicatively connected to each of one or more robots 30. The monitoring device 40 may be connected to each robot 30 via a wireless network such as a mobile phone network, a wireless WAN, a wireless LAN, or Bluetooth (registered trademark), or may be connected to each robot 30 via a wired network such as wired Ethernet (registered trademark). The monitoring device 40 according to this embodiment is installed within the facility 1. Alternatively, the monitoring device 40 may be installed outside the facility 1 and remotely monitor each device 10 within the facility 1. For example, the monitoring device 40 may be installed within another facility and may be realized, for example, by a cloud server on the Internet.

[0025] The monitoring device 40 receives collected data such as acoustic data and image data collected by each robot 30 within the facility 1 from each robot 30, and uses the collected data to monitor the status of each device 10. When the monitoring device 40 identifies an abnormality in any of the devices 10, it may instruct the control system 20 to perform control to deal with the abnormality.

[0026] The monitoring device 40 uses a collected information database 50, an equipment database 60, and a robot database 70 to monitor each piece of equipment 10 in the facility 1 using one or more robots 30. The collected information database 50 records collected data collected from each robot 30. The equipment database 60 records equipment data for each piece of equipment 10 in the facility 1. The robot database 70 stores data related to each robot 30. These databases may be storage devices such as a hard disk connected to the monitoring device 40 by wire or wirelessly or cloud storage on the Internet, or may be temporarily stored in the memory of the monitoring device 40.

[0027] 2 shows the configuration of the control system 20 according to this embodiment, together with a display device 240 and an input device 250. The display device 240 displays a display screen output by the control system 20. The input device 250 inputs instructions for the control system 20 from users such as operators, workers, or maintenance personnel of the facility 1 and supplies the instructions to the control system 20. The display device 240 and the input device 250 may be provided in a monitoring console, a user terminal, or the like connected to the control system 20.

[0028] The control system 20 may be a computer such as a workstation, a server computer, a general-purpose computer, or another type of computer, or may be a computer system in which multiple computers are connected. Such a computer system is also a computer in a broad sense. The control system 20 may also be implemented as one or more virtual computer environments executable within a computer. Alternatively, the control system 20 may be a dedicated computer designed for controlling each device 10, or may be dedicated hardware realized by a dedicated circuit. In this embodiment, the control system 20 is installed within the facility 1, but the control system 20 may also be installed outside the facility 1 using a cloud computing system on the Internet, for example.

[0029] The control system 20 includes a status acquisition unit 210, a status determination unit 220, a display processing unit 230, an instruction input unit 260, and a control unit 270. The status acquisition unit 210 acquires device status data indicating the status of each device 10 from each device 10, such as the internal state value of each device 10 and measurement values ​​measured by sensors provided in each device 10. The status acquisition unit 210 may receive the device status data of each device 10 using communication based on a communication protocol such as HART (registered trademark), BRAIN, Foundation Fieldbus (registered trademark), or ISA100.11a. The status acquisition unit 210 may receive, from the monitoring device 40, device status data of the device 10 detected by the monitoring device 40 using collected data collected by the robot 30.

[0030] The status determination unit 220 is connected to the status acquisition unit 210. The status determination unit 220 determines whether each device 10 is normal or abnormal using the device status data acquired by the status acquisition unit 210. The status determination unit 220 may calculate a health index indicating the health of operation in the entire facility 1 or in a specific area within the facility 1 from at least one of the internal status values ​​or measurement values ​​of two or more devices 10, and determine whether the operation is normal or abnormal. The status determination unit 220 may determine whether at least some of the devices 10 are normal or abnormal by receiving from the monitoring device 40 a determination result of whether the monitoring device 40 determines whether the devices 10 are normal or abnormal, made by using data collected by the robot 30.

[0031] The display processing unit 230 is connected to the status acquisition unit 210 and the status determination unit 220. The display processing unit 230 performs display control to cause the display device 240 to display a display screen including device status data such as internal status values ​​and measurement values ​​of each device 10, health indicators for the entire facility 1 or a specific area within the facility 1, and abnormality occurrence statuses of each device 10.

[0032] The instruction input unit 260 receives instructions from the user for the control system 20 from the input device 250. Such instructions may be instructions to change the operation of at least one device 10 from the user viewing the display screen displayed on the display device 240. The instruction input unit 260 may be connected to the monitoring device 40 and may be able to communicate with the monitoring device 40 via a wired or wireless connection. In response to the monitoring device 40 detecting an abnormality in at least one device 10, the instruction input unit 260 may receive, from the monitoring device 40, an instruction to perform control to deal with the abnormality.

[0033] The control unit 270 is connected to the status acquisition unit 210 and the instruction input unit 260. The control unit 270 controls each device 10 in accordance with the device status data acquired by the status acquisition unit 210. Upon receiving an instruction from a user or an instruction from the monitoring device 40, the control unit 270 may control each device 10 in accordance with the instruction.

[0034] 3 shows a processing flow of the control system 20 according to this embodiment. In step 300 (S300), the status acquisition unit 210 acquires device status data indicating the status of each device 10 from each device 10. The status acquisition unit 210 may receive device data indicating the status of at least one device 10 from the monitoring device 40.

[0035] In S310, the status determination unit 220 determines whether each device 10 is normal or abnormal. The status determination unit 220 may determine whether the device 10 is normal (whether it is not abnormal) based on whether at least one of the internal state values ​​or measurement values ​​of the device 10 is within a predetermined normal range corresponding to that value. The status determination unit 220 may also calculate a health index indicating the health of operations in the entire facility 1 or a specific area within the facility 1 from at least one of the internal state values ​​or measurement values ​​of two or more devices 10 using a predefined calculation formula or the like, and determine whether operations in the entire facility 1 or a specific area within the facility 1 are normal based on whether the value of the health index is within the normal range.

[0036] In S320, the display processing unit 230 performs display control to cause the display device 240 to display a display screen including the internal state values ​​and measurement values ​​of each device 10, the health index for the entire facility 1 or a specific area within the facility 1, and the abnormality occurrence status of each device 10. The display processing unit 230 may generate a video output of the display screen and supply it to the display device 240, or may generate HTML, a script, or the like for generating the display screen and send it to the display device 240.

[0037] In S330, the instruction input unit 260 receives an instruction from the user for the control system 20 from the input device 250. The instruction input unit 260 may receive an instruction to perform control to deal with the abnormality, which is transmitted by the monitoring device 40 that has detected an abnormality in at least one device 10.

[0038] In S340, the control unit 270 controls each device 10 in accordance with a predetermined control algorithm or control model, etc., using the device status data acquired by the status acquisition unit 210. The control unit 270 may calculate a control value for each device 10 using PI control, PID control, etc. The control unit 270 may calculate a control value for each device 10 according to the device status data acquired by the status acquisition unit 210, using various machine learning models, etc. The control unit 270 may change the control content for each device 10 in response to an instruction from the instruction input unit 260.

[0039] The control system 20 repeats the processes from S300 to S340, thereby enabling the control system 20 to adaptively control each device 10 according to the state of each device 10.

[0040] 4 shows the configuration of the robot 30 according to this embodiment. The robot 30 includes one or more sensors 400, one or more sensors 410, one or more actuators 420, a status acquisition unit 430, a communication unit 440, and a control unit 450.

[0041] Each of the one or more sensors 400 detects or measures conditions within the facility 1. In the example shown in this figure, sensor 400a is an acoustic sensor that detects sounds within the facility 1 and outputs the results as acoustic data. Sensor 400b is an image sensor that captures images within the facility 1 and outputs the images as image data. Sensor 400c is a LiDAR sensor. Sensor 400c measures at least one of the shape of each object within the light or laser irradiation range or the distance to each irradiation point by irradiating light or a laser beam to the outside and detecting the reflected light, and outputs the measurement results as LiDAR data. The robot 30 does not necessarily have to include at least one of sensors 400a to 400c. The robot 30 may also include at least one of a temperature sensor, a humidity sensor, a gas sensor, or various other sensors.

[0042] Each of the one or more sensors 410 detects or measures the state of the robot 30. In the example shown in the figure, the sensor 410a is a position sensor that detects the position of the robot 30 and outputs the position data. The sensor 410a may be a GPS (Global Positioning System), and more specifically, a GPS receiver that receives signals from GPS satellites to identify the position of the robot 30. Alternatively, the sensor 410a may be a sensor for identifying the position of the robot 30 using any position detection system that can detect the position of the robot 30 within the facility 1.

[0043] The sensor 410b is a direction sensor such as a geomagnetic sensor. The sensor 410b measures the orientation of the robot 30 or the orientation of the detection direction of each sensor 400. The sensor 410c is an angle sensor, elevation sensor, or the like for measuring the detection direction of each sensor 400. The sensors 410b and 410c output direction data indicating at least one of the orientation of the robot 30 or the detection direction of each sensor 400 to the status acquisition unit 430. Note that the robot 30 does not necessarily have to include at least one of the sensors 410a to 410c. The robot 30 may instead include at least one of a speed sensor, an acceleration sensor, or various other sensors.

[0044] Each of the one or more actuators 420 is a motor or the like for driving each part of the robot 30. The robot 30 may be equipped with various types of actuators 420, such as an actuator 420 for movement by running or flying, an actuator 420 for changing the orientation of the entire robot 30 or a part of it, and an actuator 420 for operating accessories such as an arm.

[0045] The status acquisition unit 430 is connected to one or more sensors 400 and one or more sensors 410. The status acquisition unit 430 acquires the status within the facility 1 measured by the one or more sensors 400 and the status of the robot 30 measured by the one or more sensors 410. The status acquisition unit 430 may acquire the status within the facility 1 by receiving various types of measurement data, such as acoustic data, image data, and LiDAR data, from the one or more sensors 400. The status acquisition unit 430 may acquire the status of the robot 30 by receiving various types of measurement data, such as position data and direction data, from the one or more sensors 410.

[0046] The communication unit 440 is connected to the status acquisition unit 430. The communication unit 440 communicates with the monitoring device 40 wirelessly or via a wire. The communication unit 440 transmits various measurement data acquired by the status acquisition unit 430 to the monitoring device 40. The communication unit 440 also receives various instructions for the robot 30 from the monitoring device 40.

[0047] The control unit 450 controls each actuator 420 in accordance with the various measurement data acquired by the status acquisition unit 430 and instructions from the monitoring device 40. This allows the robot 30 to perform operations corresponding to the status within the facility 1, the status of the robot 30, and instructions from the monitoring device 40.

[0048] Note that some of the robots 30 may be fixedly installed within the facility 1, and such robots 30 may not be equipped with actuators for movement. Also, within the facility 1, monitoring equipment (such as a monitoring camera or a monitoring microphone) that includes one or more sensors 400 but is not classified as a robot may be provided. Such monitoring equipment will have some of the functions and configuration of the robots 30. Therefore, for the sake of convenience of explanation, this specification assumes that at least one of the robots 30 may be such a monitoring equipment, and that such monitoring equipment will perform the processes described below within the scope of its implemented functions.

[0049] 5 shows a processing flow of the robot 30 according to this embodiment. In S500, one or more sensors 400 observe the state of the facility 1 in the vicinity of the robot 30. One or more sensors 410 observe the state of the robot 30. The state acquisition unit 430 acquires various measurement data indicating the state of the facility 1 from the one or more sensors 400. The state acquisition unit 430 acquires various measurement data indicating the state of the robot 30 from the one or more sensors 410.

[0050] In S510, the communication unit 440 transmits various measurement data indicating the state of the facility 1 and the state of the robot 30 observed by one or more sensors 400 and one or more sensors 410 to the monitoring device 40. Here, if the central direction of measurement of each sensor 400 coincides with the direction of the robot 30, that is, for example, if a sensor unit or the like that mounts each sensor 400 is facing the front of the robot 30, the measurement direction (central direction of measurement) of each sensor 400 coincides with the direction of the robot 30. In such a case, if the robot 30 transmits its own direction to the monitoring device 40, the monitoring device 40 can obtain the measurement direction of each sensor 400.

[0051] If the orientation of the sensor unit mounting each sensor 400 is variable with respect to the robot 30 body, the central direction of measurement of each sensor 400 does not necessarily coincide with the orientation of the robot 30. In this case, the robot 30 may transmit to the monitoring device 40 direction data including the orientation of the robot 30 itself and the measurement direction of each sensor 400. Alternatively, the robot 30 may transmit to the monitoring device 40 direction data including the orientation of the robot 30 itself and the difference or offset of the measurement direction of each sensor 400 relative to the orientation of the robot 30 itself. In this case, the monitoring device 40 can obtain the measurement direction of each sensor 400 by adding the difference or offset of the measurement direction of each sensor 400 to the orientation of the robot 30 itself.

[0052] In S520, the communication unit 440 receives an instruction from the monitoring device 40. In S530, the control unit 450 performs an operation in accordance with the instruction from the monitoring device 40. The control unit 450 drives at least one actuator 420 in accordance with an instruction to move, an instruction to change direction, an instruction to change the direction of an acoustic sensor, an image sensor, or the like, received from the monitoring device 40, thereby causing the robot 30 to operate as instructed. The control unit 450 may also set various parameters within the robot 30 in accordance with the instruction from the monitoring device 40.

[0053] The robot 30 repeats the processes from S500 to S530. As a result, the robot 30 can move within the facility 1 in response to instructions from the monitoring device 40, and perform operations such as collecting sound with the sensor 400a, capturing images with the sensor 400b, and acquiring LiDAR data with the sensor 400c.

[0054] 6 shows the configuration of the monitoring device 40 according to this embodiment, along with the collected information database 50, the equipment database 60, the robot database 70, the input device 680, and the display device 690. The input device 680 receives instructions from users, such as operators, workers, or maintenance personnel, of the facility 1, and supplies the instructions to the monitoring device 40. The display device 690 displays a display screen output by the monitoring device 40. The input device 680 and the display device 690 may be provided in a monitoring console or a user terminal connected to the monitoring device 40. The display device 690 and the input device 680 may be shared with the display device 240 and the input device 250 shown in FIG. 2.

[0055] The monitoring device 40 may be an equipment management device that manages each device 10 in the facility 1, or may be a device that realizes some of the functions included in the equipment management device. The monitoring device 40 may be a computer such as a workstation, a server computer, a general-purpose computer, or other computer, or may be a computer system in which multiple computers are connected. Such a computer system is also a computer in a broad sense. The monitoring device 40 may also be implemented using one or more virtual computer environments that can be executed within the computer. Such a computer functions as the monitoring device 40 by executing a program for monitoring each device 10 using the robot 30. Alternatively, the monitoring device 40 may be a dedicated computer designed for monitoring each device 10, or may be dedicated hardware realized by dedicated circuits.

[0056] In this embodiment, the monitoring device 40 is installed inside the facility 1, but instead, the monitoring device 40 may be installed outside the facility 1 using a cloud computing system on the Internet or the like. Also, in this embodiment, the monitoring device 40 remotely controls each robot 30, but instead, the monitoring device 40 may be mounted on at least one robot 30 and directly operate each robot 30 or a group of robots including two or more robots 30 on site.

[0057] The monitoring device 40 includes a communication unit 600, a collected information database connection unit 620, an equipment database connection unit 630, a robot database connection unit 640, a monitoring processing unit 650, an input processing unit 660, a display processing unit 665, and a communication unit 670. The communication unit 600 communicates with the robot 30 wirelessly or via a wired connection. The communication unit 600 includes an acoustic data acquisition unit 602, an image data acquisition unit 604, a LiDAR data acquisition unit 606, a position data acquisition unit 608, a direction data acquisition unit 610, and an instruction transmission unit 612.

[0058] The acoustic data acquisition unit 602 acquires acoustic data detected within the facility 1. The acoustic data acquisition unit 602 may acquire acoustic data by receiving acoustic data transmitted by the robot 30. The image data acquisition unit 604 acquires image data captured within the facility 1. The image data acquisition unit 604 may acquire image data by receiving image data transmitted by the robot 30. The LiDAR data acquisition unit 606 acquires LiDAR data detected by the sensor 400c of the robot 30 within the facility 1. The LiDAR data acquisition unit 606 may acquire LiDAR data by receiving LiDAR data transmitted by the robot 30.

[0059] The position data acquisition unit 608 acquires position data indicating the position of the robot 30. The position data acquisition unit 608 may acquire the position data by receiving position data transmitted by the robot 30. The direction data acquisition unit 610 acquires direction data indicating the directions of the robot 30 and each sensor 400. The direction data acquisition unit 610 may acquire the direction data by receiving direction data transmitted by the robot 30. The instruction transmission unit 612 transmits instructions for the robot 30, which have been determined within the monitoring device 40, to the robot 30.

[0060] The collected information database connection unit 620 is connected to the collected information database 50, the communication unit 600, and the monitoring processing unit 650. The collected information database connection unit 620 records various measurement data acquired by the acoustic data acquisition unit 602, the image data acquisition unit 604, the LiDAR data acquisition unit 606, the position data acquisition unit 608, and the direction data acquisition unit 610 in the communication unit 600 as collected data in the collected information database 50. The collected information database connection unit 620 also accesses the collected information database 50 in response to a request from the monitoring processing unit 650. Note that if there is no need to record the history of collected data, the monitoring device 40 does not need to include the collected information database connection unit 620, and the measurement data acquired by the communication unit 600 may be supplied to the monitoring processing unit 650 without going through the collected information database connection unit 620. In this case, the collected information database 50 is not necessary.

[0061] The equipment database connection unit 630 is connected to the equipment database 60 and the monitoring processing unit 650. The equipment database connection unit 630 accesses the equipment database 60 in response to a request from the monitoring processing unit 650. The robot database connection unit 640 is connected to the robot database 70 and the monitoring processing unit 650. The robot database connection unit 640 accesses the robot database 70 in response to a request from the monitoring processing unit 650.

[0062] The monitoring processing unit 650 is connected to the collected information database connection unit 620, the equipment database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670. The monitoring processing unit 650 reads out the robot data of each robot 30 via the robot database connection unit 640, and uses the robot data to determine the monitoring behavior within the facility 1 that each robot 30 should perform. The monitoring processing unit 650 instructs each robot 30 via the instruction transmission unit 612 in the communication unit 600 to perform each operation included in the determined monitoring behavior.

[0063] The monitoring processing unit 650 also reads out collected data collected by one or more robots 30, which is recorded in the collected information database 50, via the collected information database connection unit 620. Then, the monitoring processing unit 650 uses the collected data to detect the state of each device 10 monitored by each robot 30. The monitoring processing unit 650 uses the collected data to determine whether each device 10 is normal or abnormal. The monitoring processing unit 650 may read out the device data of each device 10 via the device database connection unit 630 and determine whether each device 10 is normal or abnormal based on the information registered in the device data.

[0064] The monitoring processing unit 650 may use the display processing unit 665 to generate a display screen that displays the status of each device 10 and the determination result of normality or abnormality, and display the generated screen on the display device 690. The monitoring processing unit 650 may transmit the status of each device 10 and the determination result of normality or abnormality to the control system 20 via the communication unit 670.

[0065] The monitoring processing unit 650 may generate an instruction to perform control to deal with the abnormality in response to detecting an abnormality in at least one of the devices 10 as a result of monitoring each device 10 using one or more robots 30. The monitoring processing unit 650 may transmit the instruction to perform control to deal with the abnormality in the device 10 to the control system 20 via the communication unit 670.

[0066] The input processing unit 660 is connected to the input device 680. The input processing unit 660 accepts information input to the input device 680 from the user and supplies it to the monitoring processing unit 650. The display processing unit 665 is connected to the monitoring processing unit 650. The display processing unit 665 performs display processing to generate a display screen in response to an instruction from the monitoring processing unit 650 and display it on the display device 690.

[0067] The communication unit 670 is connected to the monitoring processing unit 650. The communication unit 670 may be connected to the control system 20 and may be able to communicate with the control system 20 wirelessly or via a wire. The communication unit 670 may transmit device status data of each device 10 detected by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit the determination result of whether each device 10 is normal or abnormal determined by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit an instruction to perform control to deal with the abnormality to the control system 20 in response to detection of an abnormality in at least one device 10. Note that the communication unit 670 does not need to have the function of transmitting at least one of the device status data of the device 10, the determination result of whether the device 10 is normal or abnormal, or the instruction to perform control to deal with the abnormality to the control system 20. If the monitoring device 40 does not have any of these functions, the communication unit 670 may not be provided with the communication unit 670.

[0068] According to the monitoring device 40 described above, by monitoring the facility 1 using one or more robots 30 that can move within the facility 1, it is possible to monitor a large number of devices 10 using a relatively small number of robots 30. Furthermore, since the monitoring device 40 can control the robots 30 in accordance with the situation at each observation point to collect necessary information, it is possible to detect the state of each device 10 that cannot be collected by sensors or the like that are installed in each device 10 in advance.

[0069] 7 shows an example of the data structure of the collected information database 50 according to this embodiment. The collected information database 50 records the measurement data collected from each robot 30 as collected data. The collected information database 50 may record one or more records, each of which is a unit of collected data. In the data structure of the collected information database 50 shown in this figure, each record of collected data is arranged in the row direction, and the collected data of each record includes fields for "date and time," "position / direction," "robot identification information," and "measurement data."

[0070] "Date and time" is a field that records the date and time when the measurement data of the corresponding record was acquired by the robot 30. "Position / direction" is a field that records at least one of the position and direction of the robot 30 or each sensor 400 at the time when the measurement data of the corresponding record was acquired. The collected information database 50 may record, as "position and direction," at least one of the position detected by the sensor 410a of the robot 30, the orientation of the robot 30 detected by the sensor 410b, or the direction of the robot 30 or each sensor 400 detected by the sensor 410c at the time indicated by "date and time."

[0071] "Robot identification information" is a field for recording data values ​​such as an identification number or serial number that identifies the robot 30 that acquired the measurement data of the corresponding record, or a unique number or character string that enables identification of the robot 30 that acquired the measurement data within the facility 1. "Measurement data" is a field for recording measurement data acquired by the robot 30 at the timing indicated by "date and time." The data recorded as "measurement data" may be various types of measurement data acquired by the robot 30, such as at least one of acoustic data, image data, and LiDAR data.

[0072] The communication unit 600 in the monitoring device 40 may receive, as separate packets or packet groups, data including the date and time when the measurement data was acquired by the robot 30, the robot identification information of the robot 30, and measurement data related to the position and orientation of the robot 30, and data including the date and time when the measurement data was acquired by the robot 30, the robot identification information of the robot 30, and measurement data related to the state within the facility 1. In this case, the communication unit 600 updates the position and orientation of the robot 30 managed by the monitoring device 40 every time it receives measurement data related to the position and orientation of the robot 30. When the communication unit 600 receives measurement data related to the state within the facility 1, it generates a record of collected data by associating the measurement data with the position and orientation of the robot 30 at the date and time when the measurement data was acquired. Alternatively, when the communication unit 440 in the robot 30 transmits various measurement data indicating the state of the facility 1 and the state of the robot 30 observed by each sensor 400 and each sensor 410 to the monitoring device 40 (see S510 in Figure 5), it may assemble one record's worth of collected data from this measurement data, encode it into one or more packets, and transmit it to the monitoring device 40.

[0073] 8 shows an example of the data structure of the equipment database 60 according to this embodiment. The equipment database 60 records equipment data for each of the multiple equipment 10 in the facility 1. The equipment database 60 may record as many records as there are equipment 10, each record being equipment data for one equipment. In the data structure of the equipment database 60 shown in this figure, each record of equipment data is arranged row-wise, and the equipment data of each record includes the fields of "equipment identification information," "equipment name," "equipment information," "position," "template," and "acoustic data."

[0074] The "device identification information" field is a field for recording a data value such as an identification number or serial number that identifies the corresponding device 10, or a unique number or character string that enables the device 10 to be identified within the facility 1. The "device name" field is a field for recording a device name assigned to the corresponding device 10 by a user or the like.

[0075] "Device Information" is a field for recording various information related to the corresponding device 10. "Location" is a field for recording the location of the device 10 within the facility 1. "Template" is a field for recording image data (template image data) of the exterior of the corresponding device 10 to be used for identifying the device 10 by image matching. "Acoustic Data" is a field for recording at least one of acoustic data collected from the corresponding device 10 or acoustic data emitted by the device 10 when the corresponding device 10 is operating normally.

[0076] 9 shows an example of the data structure of the robot database 70 according to this embodiment. The robot database 70 records robot data for one or more robots 30 in the facility 1. The robot database 70 may record as many records as there are robots 30, each record containing robot data for one robot. In the data structure of the robot database 70 shown in this figure, each record of robot data is arranged row-wise, and the robot data in each record includes the following fields: "robot identification information," "robot name," "robot information," "position / direction," and "schedule information."

[0077] "Robot identification information" is a field for recording data values ​​such as an identification number or serial number that identifies the corresponding robot 30, or other unique numbers or character strings that enable the robot 30 to be identified within the facility 1. "Robot name" is a field for recording a robot name assigned to the corresponding robot 30 by a user or the like.

[0078] "Robot information" is a field for recording various information related to the corresponding robot 30. "Position / direction" is a field for recording the position and direction of the corresponding robot 30 within the facility 1, and, if necessary, the direction of each sensor 400 (absolute direction or offset relative to the direction of the robot 30). "Schedule information" is a field for recording a schedule for the corresponding robot 30 to patrol the facility 1 and collect sounds within the facility 1 or sounds from each device 10.

[0079] 10 shows the configuration of the monitoring processing unit 650 of the monitoring device 40 according to this embodiment. The monitoring processing unit 650 includes a sound source separation unit 1000, a device detection unit 1010, a sound source identification unit 1020, a sound abnormality detection unit 1030, an abnormal device identification unit 1040, a device control unit 1050, and a robot command unit 1060.

[0080] The sound source separation unit 1000 is connected to the collected information database connection unit 620. The sound source separation unit 1000 reads collected data in which acoustic data is recorded as measurement data from the collected data registered in the collected information database 50 via the collected information database connection unit 620. In this way, the sound source separation unit 1000 acquires the acoustic data, the date and time when the acoustic data was acquired, and the position and direction in the facility 1 where the acoustic data was observed. The sound source separation unit 1000 extracts at least one sound component by performing sound source separation or the like on the acquired acoustic data.

[0081] The device detection unit 1010 is connected to the collected information database connection unit 620. The device detection unit 1010 reads, via the collected information database connection unit 620, collected data in which image data is recorded as measurement data from among the collected data registered in the collected information database 50. As a result, the device detection unit 1010 acquires the image data, the date and time the image data was acquired, and the position and direction in the facility 1 at which the image data was observed. The device detection unit 1010 detects at least one device 10 captured in the image data by performing image recognition or the like on the image data. The device detection unit 1010 may access the device database 60 via the device database connection unit 630 and refer to each record of the device data, thereby searching for a device 10 in the facility 1 corresponding to the device captured in the image data.

[0082] The source identification unit 1020 is connected to the sound source separation unit 1000 and the device detection unit 1010. The source identification unit 1020 identifies which device 10, among the at least one device 10 captured in the image data, is the source of each of at least one sound component included in the sound data. The source identification unit 1020 according to this embodiment identifies which device 10, detected from the image data by the device detection unit 1010, is the source of each sound component extracted from the sound data by the sound source separation unit 1000. The source identification unit 1020 instructs the device database connection unit 630 to record sound component data relating to each sound component in the device database 60 in association with the device 10 that is the source of the sound component among the at least one device 10 captured in the image data. If the source identification unit 10 cannot identify the device 10 corresponding to any sound component, the source identification unit 1020 may receive information input from the user via the input processing unit 660 to specify the corresponding device 10.

[0083] The sound abnormality detection unit 1030 is connected to the sound source identification unit 1020. The sound abnormality detection unit 1030 uses the sound component data associated with each device 10 to detect the degree of abnormality of the sound component.

[0084] The abnormal device identifying unit 1040 is connected to the sound abnormality detection unit 1030. The abnormal device identifying unit 1040 identifies an abnormality in the device 10 that is the source of the sound component based on the abnormality level of the sound component detected by the sound abnormality detection unit 1030. The abnormal device identifying unit 1040 may use the display processing unit 665 to generate a display screen for displaying the status of each device 10 detected by the robot 30 on the display device 690, and display the display screen on the display device 690. The abnormal device identifying unit 1040 may also use the display processing unit 665 to generate a display screen for displaying the determination result of whether each device 10 is normal or abnormal on the display device 690, and display the display screen on the display device 690.

[0085] The equipment control unit 1050 is connected to the abnormal equipment identification unit 1040. In response to identification of an abnormality in any of the equipment 10 that is the source of the sound component, the equipment control unit 1050 performs control to address the abnormality in the equipment 10, on at least one of the equipment 10 in which the abnormality has been identified or the other equipment 10 within the facility 1. The equipment control unit 1050 transmits an instruction to the instruction input unit 260 in the control system 20 via the communication unit 670 to perform control to address the abnormality in the equipment 10.

[0086] The robot command unit 1060 is connected to the collected information database connection unit 620, the robot database connection unit 640, and the source identification unit 1020. The robot command unit 1060 determines the action that each robot 30 should take, and instructs each robot 30 to take the determined action.

[0087] The monitoring processing unit 650 described above has the components shown in Fig. 10 in order to realize functions such as identifying the device 10 that is the source of each sound component, identifying an abnormality in the source device 10 based on the abnormality level of the sound component, issuing control instructions to deal with the abnormality in the device 10, and issuing operation instructions to each robot 30. Alternatively, the monitoring processing unit 650 may have a configuration that does not include some of the components shown in Fig. 10.

[0088] For example, the monitoring processing unit 650 may not have the equipment control unit 1050 and may not issue control instructions to deal with an abnormality in the equipment 10. Furthermore, the monitoring processing unit 650 may not have the robot command unit 1060 and may not issue instructions to each robot 30 regarding its operation. Furthermore, the monitoring processing unit 650 may not have the sound abnormality detection unit 1030 and the abnormal equipment identification unit 1040 and may not identify an abnormality in the equipment 10 that is the source of the sound based on the degree of abnormality of the sound component. Even if the monitoring device 40 does not have some of the functions exemplified here, the user of the monitoring device 40 can receive information output or displayed by the monitoring device 40, discover an abnormality in the equipment 10, and cause at least one of the equipment 10 in the facility 1 to take technical measures to deal with the abnormality in the equipment 10.

[0089] The monitoring device 40 described above can collect acoustic data detected within the facility 1 and image data captured within the facility 1, and identify the equipment 10 that is the source of each sound component contained in the acoustic data. This allows the monitoring device 40 to analyze, from the sound components generated by the equipment 10, the latent state of each equipment 10 that may not be detectable from the state of each equipment 10 collected by the control system 20. Furthermore, the monitoring device 40 can quickly collect abnormal sounds emitted by equipment 10 installed in various locations within the facility 1, and can quickly respond to any abnormalities in the equipment 10.

[0090] 11 and 12 show a processing flow of the monitoring device 40 according to this embodiment. For convenience of explanation, the processing flows in Fig. 11 and 12 show a case where the monitoring device 40 monitors the inside of the facility 1 using one robot 30. The monitoring device 40 may execute the processing flows in Fig. 11 and 12 for each of the multiple robots 30.

[0091] In S1100, the robot command unit 1060 instructs the target robot 30 to move within the facility 1 via the instruction transmission unit 612. Here, the robot command unit 1060 may obtain a schedule for the target robot 30 to patrol the facility 1 by referring to schedule information recorded in the robot database 70 in association with the target robot 30. This schedule may include information necessary for determining the operation of the robot 30, such as the location of each observation point where the robot 30 should observe the state of the facility 1, the time when the robot 30 should arrive at each observation point, the movement route between the observation points, or the observation direction at each observation point. In accordance with the schedule, the robot command unit 1060 moves the robot 30 to the next observation point and provides the robot 30 with an instruction to perform observation at the next observation point. In response to this, the robot 30 moves to the next observation point (see S520 and S530 in FIG. 5 ). If an area within facility 1 where an abnormality may have occurred is identified, the robot command unit 1060 may instruct at least one robot 30 to head toward that area and to move to various positions within that area to acquire acoustic data and image data, etc.

[0092] In S1110, the robot 30 observes the state of the facility 1 and the state of the robot 30 using each sensor 400 at the observation point (see S500 in FIG. 5 ). The robot 30 transmits various measurement data indicating the observed state of the facility 1 and the state of the robot 30 to the monitoring device 40 (see S510 in FIG. 5 ). The acoustic data acquisition unit 602 in the monitoring device 40 acquires acoustic data from the measurement data transmitted from the robot 30. The collected information database connection unit 620 may add the position data acquired by the position data acquisition unit 608 and the direction data acquired by the direction data acquisition unit 610 to the acoustic data acquired by the acoustic data acquisition unit 602, and record the added data as collected data in the collected information database 50.

[0093] In S1120, the sound source separation unit 1000 extracts at least one sound component by performing sound source separation or the like on the sound data from the robot 30. The sound source separation unit 1000 may perform sound source separation by extracting from the sound data at least one sound component whose sound source directions as seen from the robot 30 are different from each other. The sound source separation unit 1000 may identify the direction of each sound component as seen from the robot 30 (sound source localization).

[0094] Here, each sensor 400a collects sound from a predetermined range of directions other than the center direction of measurement of the sensor 400a according to its directivity, and thus collects a sound that is a composite of sounds from various sound sources. The sound source separation unit 1000 performs processing to separate or decompose the sound collected by each sensor 400a into sounds generated by each source. For convenience of explanation, in this specification, sounds extracted from at least a portion of the sound collected by the sensor 400a are referred to as "sound components," but the "sound components" themselves are also sounds. Therefore, the acoustic data, which is data on the sound collected by the sensor 400a, and the sound component data, which is data on the sound components, may have the same data format.

[0095] The sound source separation unit 1000 may use various sound source separation and sound source localization techniques. For example, the robot 30 may have two or more sensors 400a, and the acoustic data received by the monitoring device 40 may include acoustic data for each channel that records the sounds collected by each sensor 400a.

[0096] The sound source separation unit 1000 may calculate the direction of the sound source of each sound component relative to the robot 30 from the time difference between the arrival times of each sound component included in the acoustic data at each sensor 400a. Here, the sound source separation unit 1000 may calculate the relative direction of the sound source of each sound component with respect to the measurement direction of a sensor unit including two or more sensors 400a, and add this to the absolute direction (measurement center direction) in which the sensor unit itself is facing, thereby calculating the absolute direction of the sound source of each sound component at the observation point. Here, the absolute direction refers to an angle expressed with a common reference direction (such as north) at least within the facility 1 as the origin, such as the horizontal and vertical angles with north as the reference. The relative direction refers to an angle expressed with an arbitrarily selected reference direction (such as the traveling direction of the robot 30) as the origin.

[0097] The robot 30 may also have a sensor 400a such as a directional microphone, and may acquire acoustic data while changing the orientation of the sensor 400a within a predetermined range of directions relative to the robot 30. For example, the robot 30 may acquire acoustic data while changing the orientation of the sensor 400a within the range of the angle of view of image data captured by the sensor 400b. Using such acoustic data, the sound source separation unit 1000 may identify the orientation of the sensor 400a when the magnitude (sound pressure) of each sound component is maximum as the direction of the sound source of that sound component.

[0098] By separating the sound sources from the acoustic data in this manner, the monitoring device 40 can identify the sound components originating from each device 10, even when the monitoring device 40 acquires acoustic data in which sounds from two or more devices 10 are superimposed. This enables the monitoring device 40 to increase the detection rate of abnormalities in the devices 10 and to take appropriate measures against the abnormalities in the devices 10.

[0099] In S1130, the image data acquisition unit 604 in the monitoring device 40 acquires image data from the measurement data transmitted from the robot 30. The collected information database connection unit 620 may add the position data acquired by the position data acquisition unit 608 and the direction data acquired by the direction data acquisition unit 610 to the image data acquired by the image data acquisition unit 604, and record the added data in the collected information database 50 as collected data.

[0100] In S1140, the device detection unit 1010 performs image recognition or the like on the image data from the robot 30 to detect one or more devices 10 captured in the image data. The device detection unit 1010 may extract one or more objects captured in the image data, and search the device database 60 for devices 10 corresponding to each object, thereby detecting the devices 10 corresponding to each object. Here, the device detection unit 1010 may search the device database 60 for devices 10 located in the direction of the robot 30 or the sensor 400b at the time the image data is acquired, based on the position of the robot 30 at this time.

[0101] The device detection unit 1010 may recognize each device 10 included in the image data using vision-based augmented reality (AR) technology. For example, the device detection unit 1010 may recognize each device 10 in the image data using markerless AR technology. The device detection unit 1010 may detect each of at least one device 10 included in the image data by matching it with a predetermined template. For example, the device detection unit 1010 may extract one or more objects captured in the image data and search the device database 60 for devices 10 associated with template image data that matches or is most similar to the image of each object. The device detection unit 1010 may detect the device 10 found for each object as a device 10 located in an image range in the image data that includes the object. The device detection unit 1010 may determine that the device 10 corresponds to the object when the template image data associated with the device 10 searched from the device database 60 using the position of the robot 30 and the direction of the robot 30 or the sensor 400b at the time the image data was acquired matches or is similar to the image of the object in the image data in the image range corresponding to the direction of the sensor 400b based on the position of the robot 30 with a similarity level equal to or greater than a predetermined threshold.

[0102] The device detection unit 1010 may recognize each device 10 included in the image data using AR technology that uses markers. In this case, the device data stored in the device database 60 may store identification information of the marker attached to the corresponding device 10, for example, in a device information field. The device detection unit 1010 may search the device database 60 for a device 10 associated with device data that stores identification information that matches the identification information of the marker attached to the device 10 included in the image data.

[0103] The equipment detection unit 1010 may detect equipment 10 that is not registered in the equipment database 60. For example, the equipment detection unit 1010 may detect equipment 10 newly installed in the facility 1 or a portion of equipment 10 already registered in the equipment database 60 as unregistered equipment 10. The equipment detection unit 1010 may estimate the type (e.g., reactor, measuring instrument, pipe, flange, etc.) or model of the equipment 10 corresponding to the object based on the external shape of the object extracted from the image data. The equipment detection unit 1010 may detect unknown equipment 10 included in the image data by matching the object with a template, etc., predetermined for each type or model of equipment, to identify the type or model of the equipment. The monitoring device 40 can perform image recognition on the image data using template matching, etc., to extract equipment 10 that is installed in the facility 1 but is not registered, and portions of equipment 10 that are not recognized as units of equipment that may emit abnormal noise and are not registered as individual equipment 10, from the image data, thereby making them detectable as sound sources.

[0104] The device detection unit 1010 may calculate the relative direction of the device 10 with respect to the center direction of the image from the position of the device 10 in the image corresponding to the image data. The device detection unit 1010 may calculate the absolute direction of each device 10 as seen from the observation point by adding the relative direction of the device 10 with respect to the center direction of the image to the absolute direction of the sensor 400b when the image data was captured.

[0105] In S1150, the source identification unit 1020 identifies which device 10 captured in the image data is the source of each sound component included in the sound data. For each sound component extracted from the sound data, the source identification unit 1020 may identify, among the devices 10 detected from the image data, the device 10 located in an area within the image data that corresponds to the direction of the sound source as the source device 10. As an example, for a certain sound component and a certain device 10 detected from the sound data and the image data observed by the robot 30 at the same position at the same time or at different times, the source identification unit 1020 may determine that the source of the sound component is the device 10 if the absolute direction of the sound component and the absolute direction of the device 10 as viewed from the observation point are the same, or if the difference between the absolute directions is within a predetermined error range. For a device 10 that has already been registered in the device database 60, the source identification unit 1020 may determine that the source of the sound component is this device 10, provided that the device 10 is located in the absolute direction of the sound component as seen from the observation point of the acoustic data.

[0106] Furthermore, with respect to a certain sound component and a certain device 10 obtained from acoustic data and image data observed by the robot 30 at different positions at the same time or different times, the source identification unit 1020 may determine that the source of the sound component is the device 10 on the condition that the absolute direction of the sound component as seen from the observation point of the acoustic data intersects or nearly intersects within a predetermined error range with the absolute direction of the device 10 as seen from the observation point of the image data. Here, the device detection unit 1010 may calculate an approximate distance range from the observation point of the image data to each device 10 based on, for example, the size of each device 10 in the image corresponding to the image data. In this case, the source identification unit 1020 may determine that the source of the sound component is the device 10 if the absolute direction of the sound component as seen from the observation point of the acoustic data intersects or nearly intersects within a predetermined error range with the absolute direction of the device 10 as seen from the observation point of the image data, and if the distance from the observation point of the image data to this intersection point is within the calculated distance range. The monitoring device 40 can identify the equipment 10 that is the source of each sound component using local acoustic data and image data observed by the robot 30 at the observation point, and can therefore make the sound information of each piece of equipment 10 that cannot be detected by measurements by the control system 20 and was buried within the facility 1 apparent as information that can be used to manage each piece of equipment 10.

[0107] In S1160, the generation source identification unit 1020 instructs the equipment database connection unit 630 to record sound component data for each sound component in the equipment database 60 in association with the equipment 10 that is the generation source. In response to the instruction from the generation source identification unit 1020, the equipment database connection unit 630 records the sound component data for each sound component in the acoustic data field of the record of the equipment data corresponding to the equipment 10 that is the generation source in the equipment database 60. The equipment database connection unit 630 may record a plurality of sound component data acquired at a plurality of different dates and times in the acoustic data field by adding the acquisition date and time of the sound component data and the sound component data to the acoustic data field.

[0108] If the source of a certain sound component is a device 10 that is included in the image data but is not registered in the device database 60, the sound source identification unit 1020 may register this device 10 in the device database 60 after the user inputs information via the input processing unit 660, and may record the sound component data in association with this device 10 in the device database 60. This processing will be described later in relation to S1250.

[0109] In S1200, the sound abnormality detection unit 1030 detects the degree of abnormality of a sound component by using the sound component data associated with each device 10. Here, the degree of abnormality may be expressed as a real number or an integer within a predetermined range, such as 0 to 1 or 0 to 100%, or may be a binary value, such as normal or abnormal.

[0110] When sound data emitted by the device 10 when the device 10 is normal is recorded in the device database 60, the sound abnormality detection unit 1030 may calculate, as the degree of abnormality, a value indicating how much the sound component data of the target generated from the device 10 differs from the sound data emitted by the device 10 when normal. For example, the sound abnormality detection unit 1030 may determine the degree of abnormality according to a comparison result between the sound data emitted by the device 10 when normal and the sound component data of the target.

[0111] The sound abnormality detection unit 1030 may compare the normal acoustic data with the target sound component data in either the time domain or the frequency domain. When performing the comparison in the time domain, the sound abnormality detection unit 1030 may adjust the phase difference so that the time integral of the difference (such as the absolute value of the difference) between the normal acoustic data and the target sound component data is minimized, and may use the time integral of the difference between the acoustic data and the target sound component data at the adjusted phase difference as the degree of abnormality. In this case, the sound abnormality detection unit 1030 may adjust the amplitude so that the average amplitude matches, for example, to match the loudness of the acoustic data and the target sound component data.

[0112] When comparing normal acoustic data with the target sound component data in the frequency domain, the sound abnormality detection unit 1030 may calculate the degree of abnormality by integrating the difference (such as the absolute value of the difference) for each frequency between the frequency spectrum of normal acoustic data and the frequency spectrum of the target sound component data in the frequency direction. In this case, the sound abnormality detection unit 1030 may adjust the volume of the sound to match so that the acoustic data and the target sound component data can be compared.

[0113] When one or more pieces of sound data collected from the device 10 in the past are recorded as history data in the sound data field in the device database 60, the sound abnormality detection unit 1030 may calculate, as the degree of abnormality, a value indicating how much the target sound component data originating from the device 10 differs from one or more pieces of sound data collected in the past from the device 10. The sound abnormality detection unit 1030 may calculate the degree of abnormality of the target sound component data relative to the sound data collected in the past, in the same manner as the calculation method for calculating the degree of abnormality of the target sound component data relative to the sound data emitted by the device 10 under normal conditions described above.

[0114] The sound abnormality detection unit 1030 may determine the degree of abnormality according to the degree of deviation of the target sound component data from the distribution of sound data previously collected from the device 10. Furthermore, the sound abnormality detection unit 1030 may calculate, as the degree of abnormality, the sum of the degrees of difference for each of a predetermined number of sound data that have the smallest differences from the target sound component data, among the sound data previously collected from the device 10 (k-nearest neighbor method). In addition to the above, the sound abnormality detection unit 1030 may calculate the degree of abnormality of the sound data and the target sound component data using various methods for calculating the difference, difference, similarity, or deviation between two sounds.

[0115] The sound abnormality detection unit 1030 may detect the degree of abnormality of a sound component using AI technology. The sound abnormality detection unit 1030 may receive sound components emitted by each device 10 from the sound source identification unit 1020 and train a machine learning model for each device 10. The sound abnormality detection unit 1030 may store the machine learning model of each device 10 or trained parameters representing the machine learning model in a device information field or the like of the device database 60, and update the machine learning model each time it receives a new sound component emitted by the device 10. The sound abnormality detection unit 1030 may also receive a determination result of whether each sound component is normal or abnormal. In this case, the sound abnormality detection unit 1030 may generate or update the machine learning model by learning so that a correct determination result is output when a sound component is input to the machine learning model.

[0116] The sound anomaly detection unit 1030 may use a neural network, statistical learning, or other machine learning algorithm. The sound anomaly detection unit 1030 may use time-series data of sound components as input to a machine learning model, or may use the frequency spectrum of the sound components as input to the machine learning model. The sound anomaly detection unit 1030 updates the parameters of the machine learning model so as to reduce the error between the model output when samples of each sound component, which serve as training data, are input to the machine learning model and the learning label indicating whether the sound component is normal or abnormal. For example, when a neural network is used, the sound anomaly detection unit 1030 inputs the data values ​​of the sound components at each time or each frequency to each input node in the input layer of the neural network. The sound anomaly detection unit 1030 adjusts the weights between each neuron in the neural network, the biases of each neuron, and the like, using a technique such as backpropagation, using the error between the output value and the label output by the neural network in response to the input of each sample.

[0117] The machine learning model may output the degree of abnormality of a sound component in response to input of sound data previously collected from the device 10 or sound data in normal conditions and sound component data from the device 10. Furthermore, instead of being prepared for each individual device 10, the machine learning model may be prepared for each type or model of device 10. Furthermore, the machine learning model may be common to all devices 10.

[0118] For example, by using the machine learning model trained in this manner, the sound abnormality detection unit 1030, upon receiving a sound component generated by the device 10, can read out the machine learning model corresponding to the device 10 from the device database 60, input the sound component, and use the normality / abnormality determination result output by the machine learning model as the degree of abnormality. The sound abnormality detection unit 1030 may use any other method to calculate the degree of abnormality based on the difference between normal acoustic data and the target sound component data.

[0119] In addition, when the device 10 is not registered in the device database 60 but the type or model of the device 10 has been estimated, the sound abnormality detection unit 1030 may determine the degree of abnormality of the target sound component data based on acoustic data emitted normally by a device 10 of the same type or model as the estimated type or model, or acoustic data generated in the past.

[0120] In S1210, the abnormal device identifying unit 1040 identifies an abnormality in the device 10 that is the source of the sound component, based on the abnormality degree of the sound component associated with the device 10. For example, the abnormal device identifying unit 1040 may identify the device 10 that is the source of the sound component as abnormal if the abnormality degree of the sound component exceeds a predetermined threshold.

[0121] In S1220, the abnormal device identifying unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays the status of each device 10 detected by the robot 30. The abnormal device identifying unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays information (observation point, direction, image, etc.) about one or more devices 10 that have been identified as the source of certain sound component data but are not registered in the device database 60. Furthermore, the abnormal device identifying unit 1040 may cause the display processing unit 665 to display on the display device 690 a display screen that displays the observation point, direction of the sound component data, etc. for sound component data that is not associated with any device 10.

[0122] In S1230, the monitoring device 40 determines whether there is any sound component data that is not associated with any device 10. If all of the sound component data is associated with any device 10 ("N" in S1230), the monitoring device 40 proceeds to S1240. If all of the sound component data is associated with any device 10 registered in the device database 60, the monitoring device 40 may proceed to S1240.

[0123] In S1240, if an abnormality is identified in any of the devices 10 that are the source of the sound component, the device control unit 1050 performs control to deal with the abnormality in the device 10, on at least one of the device 10 in which the abnormality is identified or the other devices 10. The device control unit 1050 transmits an instruction to the control system 20 via the communication unit 670 to perform control to deal with the abnormality in the device 10.

[0124] As an example, the equipment control unit 1050 may instruct the control system 20 to perform control to stop the operation of the equipment 10 in which an abnormality has been identified. The equipment control unit 1050 may instruct the control system 20 to perform control to stop other equipment 10 related to the equipment 10 in which an abnormality has been identified (for example, equipment 10 in an upstream or downstream process of the equipment 10 in which an abnormality has been identified). The equipment control unit 1050 may also instruct the control system 20 to perform control to stop the operation of both the equipment 10 in which an abnormality has been identified and the other equipment 10 related to the equipment 10 in which an abnormality has been identified. The monitoring device 40 proceeds to the process at S1100 and continues monitoring the facility 1 with the robot 30.

[0125] If there is sound component data that is not associated with any device 10 in S1230 ("Y" in S1230), the monitoring device 40 proceeds to S1250. Here, the monitoring device 40 may also proceed to S1250 if the sound component data is associated with a device 10 that is not registered in the device database 60.

[0126] In S1250, the input processing unit 660 accepts input of information from the user for identifying a device 10 when the device 10 located in the image region corresponding to the direction of the sound source in the acoustic data for any sound component cannot be identified. Here, if the device 10 associated with the sound component is unregistered and its presence in the facility 1 has not been confirmed, the input processing unit 660 may accept input of information regarding at least a portion of the device data for the unregistered device 10, and the source identification unit 1020 may register the device data of the device 10 corresponding to the accepted information in the device database 60. In response to the input of such information ("Y" in S1260), the monitoring device 40 proceeds to S1150. This allows the source identification unit 1020 to identify the source of each sound component, including the newly registered device 10.

[0127] If the device 10 associated with the sound component is not registered, the input processing unit 660 may accept input of information indicating whether or not the device 10 may be registered in the device database 60. The source identification unit 1020 may register in the device database 60 device data of the device 10 according to the type or model, location or approximate location of the device 10 detected from the image data, or other information that has already been determined.

[0128] Furthermore, the input processing unit 660 may accept input of information specifying the device 10 that is the source of sound component data that is not associated with any device 10. In response to the input of such information ("Y" in S1260), the monitoring device 40 proceeds to S1150. This allows the sound source identification unit 1020 to identify that the sound component data is generated by the specified device 10.

[0129] In S1270, the robot command unit 1060 instructs the robot 30 to change at least one of the image capturing direction, the image capturing magnification, or the position of the robot 30, in response to the fact that the device 10 that is the source of any of the sound components cannot be identified among the devices 10. In the processing flow of this figure, the robot command unit 1060 performs the processing of S1270 when information regarding the device 10 corresponding to the sound component has not been input ("N" in S1260). Alternatively, the robot command unit 1060 may automatically instruct the robot 30 to change the image capturing direction, the image capturing magnification, or the position of the robot 30, in response to the fact that the device 10 that is the source of any of the sound components cannot be identified.

[0130] For example, if it is not possible to identify which of the two devices 10 is the source of a certain sound component, the robot command unit 1060 may instruct the robot 30 to move closer to one of the two devices 10. Furthermore, if there is a sound component outside the angle of view of the image data, the robot command unit 1060 may instruct the robot 30 to move the imaging direction of the image closer to the direction of the sound component, to reduce the imaging magnification of the image so as to capture an area including the direction of the sound component, or to move toward the sound component outside the angle of view. After instructing the robot 30, the monitoring device 40 proceeds to S1110 and performs the processes from S1110 to S1220 on the acoustic data and image data newly acquired by the robot 30. Thus, even in a situation where it is not possible to determine which device 10 is the source of a sound component based on observation from a single observation point or under a single observation condition, the monitoring device 40 can further improve the accuracy of associating the sound component with the device 10 that is the source by changing the observation point or the observation conditions and repeating the observation.

[0131] 13 shows an example of image data 1300 captured by the robot 30 according to this embodiment. The image data 1300 includes images of devices 10 such as a plurality of pipes 1305a-h, a valve 1310, a reactor 1320, and a flow meter 1330. In the example shown in this figure, there is an abnormality in the valve 1310, and it is making an abnormal noise.

[0132] The monitoring device 40 receives collected data including image data 1300 and acoustic data obtained by collecting sounds from within the angle of view of the image data 1300 (corresponding to S1110 and S1130 in FIG. 11 ). The sound source separation unit 1000 separates the acoustic data into sound sources and extracts sound components from each direction within the angle of view of the image data 1300 (corresponding to S1120). The device detection unit 1010 detects each device 10 captured in the image data 1300 by performing image recognition or the like on the image data 1300 (corresponding to S1140).

[0133] The sound source identification unit 1020 may identify a sound component as a device 10 when the direction of the sound component matches (or matches within a predetermined error range) the direction corresponding to the area in the image data 1300 where the device 10 is located (corresponding to S1150). The sound abnormality detection unit 1030 detects the degree of abnormality of the sound component using the sound component data associated with each device 10 (corresponding to S1200). In the example shown in the figure, the sound abnormality detection unit 1030 detects a high degree of abnormality for the sound component from the valve 1310 that is emitting the abnormal sound. As a result, the abnormal device identification unit 1040 can identify the valve 1310 as abnormal based on the degree of abnormality of the sound component.

[0134] Here, if the pipes 1305a-c and the valve 1310 are newly installed in the facility 1 and have not yet been registered in the equipment database 60, the equipment detection unit 1010 may estimate that the pipes 1305a-c are pipes and the valve 1310 is a valve, based on the external shapes and the like of the objects extracted from the image data 1300. The sound abnormality detection unit 1030 may detect the degree of abnormality of the sound component based on the result of comparing the sound component data from the valve 1310 with sound data emitted by the valve when it is normal or sound data emitted in the past by one or more other valves in the facility 1.

[0135] Note that, depending on the type of sensor 400a used by the robot 30, the sound source separation unit 1000 may be able to separate sound sources in the horizontal direction, but may not be able to separate sound sources in the vertical direction. In such cases, the source identification unit 1020 may identify the device 10 that is the source of each sound component by using the horizontal position of each device 10 in the image data 1300 and the horizontal direction of each sound component while appropriately changing the imaging direction and position of the robot 30 (S1270 in FIG. 12 ).

[0136] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. Dedicated circuitry may include digital and / or analog hardware circuitry, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

[0137] A computer-readable medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture containing instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0138] The computer readable instructions may include either assembler instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0139] The computer-readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus, such as a general-purpose computer, special-purpose computer, or other computer, either locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, etc., which executes the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0140] 14 illustrates an example of a computer 2200 in which aspects of the present invention may be embodied, in whole or in part. Programs installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to embodiments of the present invention, and / or to perform a process or steps of a process according to embodiments of the present invention. Such programs may be executed by the CPU 2212 to cause the computer 2200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0141] A computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0142] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 acquires image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into the graphics controller 2216 itself, and causes the image data to be displayed on the display device 2218.

[0143] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0144] ROM 2230 stores therein a boot program or the like that is executed by computer 2200 upon activation, and / or programs that depend on the hardware of computer 2200. I / O chip 2240 may also connect various I / O units to I / O controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0145] The programs are provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. Information processing described in these programs is read by the computer 2200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be configured by implementing information manipulation or processing in accordance with the use of the computer 2200.

[0146] For example, when communication is performed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded in RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 2212, communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in RAM 2214, hard disk drive 2224, DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area or the like provided on the recording medium.

[0147] Furthermore, the CPU 2212 may cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and may perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.

[0148] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 2212 may perform various types of processing on data read from the RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 2214. The CPU 2212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored on the recording medium, the CPU 2212 may search for an entry that matches a condition specified by the attribute value of the first attribute from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0149] The above-described programs or software modules may be stored in a computer-readable medium on or near the computer 2200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable medium, thereby providing the programs to the computer 2200 via the network.

[0150] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0151] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0152] 1 Facility, 10 Equipment, 20 Control system, 30 Robot, 40 Monitoring device, 50 Collected information database, 60 Equipment database, 70 Robot database, 210 Status acquisition unit, 220 Status determination unit, 230 Display processing unit, 240 Display device, 250 Input device, 260 Instruction input unit, 270 Control unit, 400a-c Sensors, 410a-b Sensors, 420 Actuators, 430 Status acquisition unit, 440 Communication unit, 450 Control unit, 600 Communication unit, 602 Acoustic data acquisition unit, 604 Image data acquisition unit, 606 LiDAR data acquisition unit, 608 Position data acquisition unit, 610 Direction data acquisition unit, 612 Instruction transmission unit, 620 Collected information database connection unit, 630 Equipment database connection unit, 640 Robot database connection unit, 650 Monitoring processing unit, 660 Input processing unit, 665 display processing unit, 670 communication unit, 680 input device, 690 display device, 1000 sound source separation unit, 1010 equipment detection unit, 1020 source identification unit, 1030 sound abnormality detection unit, 1040 abnormal equipment identification unit, 1050 equipment control unit, 1060 robot command unit, 1300 image data, 1305a-h piping, 1310 valve, 1320 reactor, 1330 flow meter, 2200 computer, 2201 DVD-ROM, 2210 host controller, 2212 CPU, 2214 RAM, 2216 graphics controller, 2218 display device, 2220 input / output controller, 2222 communication interface, 2224 hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 Input / output chip, 2242 keyboard

Claims

1. An apparatus comprising: an acoustic data acquisition unit that acquires acoustic data detected within a facility; an image data acquisition unit that acquires image data captured within the facility; a source identification unit that identifies which of at least one device captured in the image data is the source of each of at least one sound component included in the acoustic data; and an equipment database connection unit that records sound component data regarding each of the at least one sound component in an equipment database in association with the device that is the source of the at least one device.

2. The device according to claim 1, further comprising a sound source separation unit that performs sound source separation on the acoustic data to extract the at least one sound component.

3. The device according to claim 2, wherein the sound source separation unit extracts the at least one sound component having sound source directions different from each other.

4. The device according to claim 3, further comprising a device detection unit that detects the at least one device captured in the image data by performing image recognition on the image data.

5. The apparatus according to claim 4, wherein said device detection unit detects each of said at least one device included in said image data by matching it with a predetermined template.

6. The device according to claim 4, wherein the source identification unit identifies, for each of the at least one sound component, one of the at least one devices that is located in an area within the image data that corresponds to the direction of the sound source as the source device.

7. The device according to claim 6, further comprising an input processing unit which, when it is not possible to identify a device located in an area within the image data corresponding to the direction of a sound source for any of the at least one sound component, accepts input of information from a user for identifying the device.

8. The device according to claim 6, wherein the acoustic data acquisition section and the image data acquisition section acquire the acoustic data and the image data from a robot capable of moving within the facility to collect sounds and take images.

9. The device of claim 8, further comprising a robot command unit that instructs the robot to at least one of change the imaging direction of the image, change the imaging magnification of the image, or change the position of the robot in response to failure to identify the source device of the at least one device for any of the at least one sound component.

10. An apparatus as described in any one of claims 1 to 9, comprising: a sound abnormality detection unit that detects the degree of abnormality of a sound component using the sound component data; and an abnormal device identification unit that identifies an abnormality in a device that is a source of the sound component based on the degree of abnormality of the sound component.

11. The device of claim 10, further comprising an equipment control unit which, in response to identification of an abnormality in a piece of equipment that is the source of the sound component, controls the piece of equipment or at least one other piece of equipment within the facility to address the abnormality in the piece of equipment.

12. A method comprising: acquiring acoustic data detected within a facility; acquiring image data captured within the facility; identifying which of at least one device captured in the image data is the source of each of at least one sound component included in the acoustic data; and recording sound component data relating to each of the at least one sound component in an equipment database in association with the device that is the source of the at least one device.

13. A program executed by a computer to cause the computer to function as: an acoustic data acquisition unit that acquires acoustic data detected within a facility; an image data acquisition unit that acquires image data captured within the facility; a source identification unit that identifies which of at least one device captured in the image data is the source of each of at least one sound component included in the acoustic data; and an equipment database connection unit that records sound component data regarding each of the at least one sound component in an equipment database in association with the device that is the source of the at least one device.

Images