Apparatus, method, and program

Abstract

The present invention provides an apparatus comprising: an acoustic data acquisition unit that acquires acoustic data detected by a sound collection device that can move around within a facility to collect sound; an image data acquisition unit that acquires image data captured within the facility; a range detection unit that detects the three-dimensional range of at least one device captured in the image data from the acquired image data; a source identification unit that uses the three-dimensional range detected by the range detection unit to identify which of the at least one device captured in the image data is the source of at least one sound component contained in the acoustic data; and a recording unit that records sound component data relating to the sound component in association with the device identified by the source identification unit.

Description

Technical Field

[0001] The present invention relates to an apparatus, a method, and a program.

Background Art

[0002] Patent Document 1 describes a maintenance plan support apparatus including "a device state diagnosis unit 21 that identifies a first device in which an abnormality has occurred". Patent Document 2 describes "acquiring sound source information indicating the position of a sound source and the direction in which the sound source emits sound". Patent Document 3 describes "a specifying device that facilitates specifying the position of an object by acquiring sound". [Prior Art Documents] [Patent Documents] Patent Document 1 Japanese Patent Application Laid-Open No. 2019-152941 Patent Document 2 Japanese Patent Application Laid-Open No. 2023-53670 Patent Document 3 Japanese Patent Application Laid-Open No. 2019-124513

Summary of the Invention

[0003] In a first aspect of the present invention, there is provided an apparatus including: an acoustic data acquisition unit that acquires acoustic data detected by a sound collection device capable of moving within a facility to collect sound; an image data acquisition unit that acquires image data captured within the facility; a range detection unit that detects, from the acquired image data, a three-dimensional range of each of at least one device imaged in the image data; a sound source identification unit that uses the three-dimensional range detected by the range detection unit to identify which device among at least one device imaged in the image data is the sound source of at least one sound component included in the acoustic data; and a recording unit that records sound component data regarding the sound component in association with the device identified by the sound source identification unit.

[0004] In the above apparatus, the range detection unit may acquire device type information corresponding to the three-dimensional range.

[0005] ​In the above-described apparatus, the source identification unit may use a three-dimensional map showing the three-dimensional range of each device detected by the range detection unit, and device type information corresponding to each three-dimensional range, to identify which of the at least one devices captured in the image data is the source of at least one sound component contained in the acoustic data.

[0006] In any of the above-described devices, the acoustic data acquisition unit acquires acoustic data including reference sound components from at least three reference sound sources located within the facility, the image data acquisition unit acquires image data in which reference marks of the reference sound sources are captured, and the source identification unit may identify the equipment that is the source of at least one sound component included in the acoustic data by identifying the positional relationship between a three-dimensional range and the sound source location of at least one sound component included in the acoustic data, based on the reference sound components and reference marks.

[0007] In any of the above-described devices, the source identification unit may use operation data indicating the operation of at least one piece of equipment within the facility to identify which of the at least one piece of equipment captured in the image data is the source of at least one sound component contained in the acoustic data.

[0008] In any of the above-described devices, the source identification unit may use previously acquired non-target sound component data to identify which of the at least one devices captured in the image data is the source of at least one sound component included in the acoustic data.

[0009] Any of the above devices may include a sound anomaly detection unit that uses sound component data to detect the degree of anomaly in sound components, and an anomaly device identification unit that identifies an anomaly in the device that is the source of the sound components based on the degree of anomaly in sound components.

[0010] In any of the above-described devices, the source identification unit may use the three-dimensional range detected by the range detection unit to identify a component of the equipment captured in the image data as the source of at least one sound component contained in the acoustic data.

[0011] A second aspect of the present invention provides a method comprising: acquiring acoustic data detected by a sound-collecting device that can move around within a facility to collect sound; acquiring image data captured within the facility; detecting the three-dimensional range of at least one device captured in the image data from the acquired image data; using the three-dimensional range detected by the range detection unit to identify which of the at least one device captured in the image data is the source of at least one sound component contained in the acoustic data; and recording sound component data relating to the sound component in association with the device identified by the source identification unit.

[0012] In a third aspect of the present invention, a program is provided that is executed by a computer and causes the computer to function as an acoustic data acquisition unit that acquires acoustic data detected by a sound collection device that can move around within a facility to collect sound; an image data acquisition unit that acquires image data captured within the facility; a range detection unit that detects the three-dimensional range of each of at least one device captured in the image data from the acquired image data; a source identification unit that uses the three-dimensional range detected by the range detection unit to identify which of the at least one device captured in the image data is the source of at least one sound component contained in the acoustic data; and a recording unit that records sound component data relating to the sound component in association with the device identified by the source identification unit.

[0013] It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. Furthermore, subcombinations of these features may also constitute an invention. [Brief explanation of the drawing]

[0014] [Figure 1] The schematic configuration of Facility 1 according to this embodiment is shown. [Figure 2] The configuration of the control system 20 according to this embodiment is shown. [Figure 3]Shows the processing flow of the control system 20 according to this embodiment. [Figure 4] Shows the configuration of the robot 30 according to this embodiment. [Figure 5] Shows the processing flow of the robot 30 according to this embodiment. [Figure 6] Shows the configuration of the monitoring device 40 according to this embodiment. [Figure 7] Shows an example of the data structure of the collected information database 50 according to this embodiment. [Figure 8] Shows an example of the data structure of the device database 60 according to this embodiment. [Figure 9] Shows an example of the data structure of the robot database 70 according to this embodiment. [Figure 10] Shows the configuration of the monitoring processing unit 650 of the monitoring device 40 according to this embodiment. [Figure 11] Shows the processing flow of the monitoring device 40 according to this embodiment. [Figure 12] Shows the processing flow of the monitoring device 40 according to this embodiment. [Figure 13] Shows an explanatory diagram for explaining the sound source position identification on the 3D map by the monitoring device 40 according to this embodiment. [Figure 14] Shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied in whole or in part.

Embodiments for Carrying Out the Invention

[0015] The present invention will be described through embodiments of the invention, but the embodiments do not limit the invention claimed in the claims. Also, not all combinations of features described in the embodiments are essential for the solution of the invention.

[0016] Figure 1 shows a schematic configuration of facility 1 according to this embodiment. Facility 1 may be a whole or a segment of a factory or plant, and multiple pieces of equipment 10 are arranged therein. Such a factory or plant may be, for example, a factory for producing various industrial products, an industrial plant for chemicals or metals, a plant for managing and controlling wellheads and surrounding areas of gas or oil fields, a plant for managing and controlling power generation such as hydroelectric, thermal, or nuclear power, a plant for managing and controlling environmental power generation such as solar or wind power, a plant for managing and controlling water and sewage systems or dams, etc. Facility 1 may also be a whole or a part of a building or transportation facility equipped with multiple pieces of equipment 10 to be controlled and monitored. Facility 1 comprises multiple pieces of equipment 10, a control system 20, one or more robots 30, a monitoring device 40, a collected information database 50, an equipment database 60, and a robot database 70.

[0017] Multiple devices 10 are installed at various locations within Facility 1. Each of the multiple devices 10 may be installed either indoors or outdoors within the area of ​​Facility 1. At least some of the multiple devices 10 may be process equipment, power generation equipment, or any other equipment (or facilities) controlled by the control system 20, or may be part of such equipment. At least some of the devices 10 may be field devices that operate under the control of the control system 20 or other devices, or from workers. At least some of the devices 10 may also be field devices themselves.

[0018] Such field devices may be, for example, sensor devices such as pressure gauges, flow meters, temperature sensors, valve devices such as flow control valves and on-off valves, actuator devices such as fans and motors, imaging devices such as cameras or videos that capture objects such as the situation of a plant or equipment, acoustic devices such as microphones or speakers that collect abnormal sounds or the like in a plant or equipment or emit alarm sounds, position detection devices that output the position information of the devices possessed by Facility 1, or other devices. Further, other devices 10 among the plurality of devices 10 may be structures such as pipes, storage tanks, support columns, partition walls, or others that do not receive control by the control system 20.

[0019] The control system 20 is connected to at least one device 10 that is a control target among the plurality of devices 10. The control system 20 may be, for example, a distributed control system (DCS: Distributed Control System). The control system 20 controls each device 10 according to the state of each device 10 measured by sensors or the like provided in each device 10 that is a control target.

[0020] Each of the one or more robots 30 is used to monitor at least a part of the plurality of devices 10 in Facility 1. Each robot 30 can move within Facility 1 to collect sound and capture images, etc. Each robot 30 is an example of a sound collection device.

[0021] The monitoring device 40 is communicably connected to each of the one or more robots 30. The monitoring device 40 may be connected to each robot 30 via a wireless network such as a mobile phone network, wireless WAN, wireless LAN, or Bluetooth (registered trademark), or may be connected to each robot 30 via a wired network such as wired Ethernet (registered trademark). The monitoring device 40 according to the present embodiment is installed inside Facility 1. Instead of this, the monitoring device 40 may be provided outside Facility 1 to remotely monitor each device 10 inside Facility 1. For example, the monitoring device 40 may be installed in another facility or may be realized by, for example, a cloud server on the Internet.

[0022] The monitoring device 40 receives collected data, such as acoustic data and image data, collected by each robot 30 within the facility 1, and uses the collected data to monitor the status of each piece of equipment 10. If the monitoring device 40 identifies an abnormality in any of the pieces of equipment 10, it may instruct the control system 20 to take action to address that abnormality.

[0023] The monitoring device 40 uses an information collection database 50, an equipment database 60, and a robot database 70 to monitor each piece of equipment 10 within the facility 1 using one or more robots 30. The information collection database 50 records collected data from each robot 30. The equipment database 60 records equipment data for each piece of equipment 10 within the facility 1. The robot database 70 stores data related to each robot 30. These databases may be storage devices such as hard disks connected to the monitoring device 40 by wired or wireless connections, or cloud storage on the internet, or they may be temporarily stored in the memory of the monitoring device 40.

[0024] Figure 2 shows the configuration of the control system 20 according to this embodiment, along with the display device 240 and the input device 250. The display device 240 displays the display screen output by the control system 20. The input device 250 receives and supplies instructions to the control system 20 from users such as operators, workers, or maintenance personnel of facility 1. The display device 240 and the input device 250 may be provided on a monitoring console or user terminal connected to the control system 20.

[0025] The control system 20 may be a computer such as a workstation, server computer, general-purpose computer, or other computer, and may also be a computer system in which multiple computers are connected. Such a computer system is also a computer in a broad sense. The control system 20 may also be implemented by a virtual computer environment that can run one or more times within the computer. Alternatively, the control system 20 may be a dedicated computer designed for controlling each of the devices 10, or dedicated hardware realized by dedicated circuits. In this embodiment, the control system 20 is installed within the facility 1, but the control system 20 may be provided outside the facility 1 via a cloud computing system on the internet or the like.

[0026] The control system 20 includes a state acquisition unit 210, a state determination unit 220, a display processing unit 230, an instruction input unit 260, and an equipment control unit 270. The state acquisition unit 210 acquires equipment state data from each equipment 10, such as the internal state value of each equipment 10 and measured values ​​measured by sensors provided on each equipment 10, indicating the state of each equipment 10. The state acquisition unit 210 may receive the equipment state data of each equipment 10 using communication protocols such as HART®, BRAIN, Foundation Fieldbus®, or ISA100.11a. The state acquisition unit 210 may also receive equipment state data of the equipment 10 detected by the monitoring device 40 using the collected data collected by the robot 30 from the monitoring device 40.

[0027] The status determination unit 220 is connected to the status acquisition unit 210. The status determination unit 220 uses the equipment status data acquired by the status acquisition unit 210 to determine whether each piece of equipment 10 is normal or abnormal. The status determination unit 220 may calculate a health index indicating the soundness of operations in the entire facility 1 or a specific area within the facility 1 from at least one of the internal status values ​​or measured values ​​of two or more pieces of equipment 10, and determine whether the operation is normal or not. The status determination unit 220 may also determine whether at least some of the pieces of equipment 10 are normal or abnormal by receiving the normal or abnormal determination result of the equipment 10 determined by the monitoring device 40 using the collected data collected by the robot 30 from the monitoring device 40.

[0028] The display processing unit 230 is connected to the status acquisition unit 210 and the status determination unit 220. The display processing unit 230 controls the display to show on the display device 240 a display screen that includes equipment status data such as internal status values ​​and measured values ​​for each piece of equipment 10, health indicators for the entire facility 1 or a specific area within facility 1, and the status of abnormalities occurring in each piece of equipment 10.

[0029] The instruction input unit 260 receives user instructions for the control system 20 from the input device 250. Such instructions may include instructions from a user who has viewed a display screen on the display device 240 to change the operation of at least one device 10. The instruction input unit 260 may be connected to a monitoring device 40 and may be able to communicate with the monitoring device 40 by wire or wireless. The instruction input unit 260 may also receive instructions from the monitoring device 40 to take control to address an abnormality detected by the monitoring device 40 in response to an abnormality detected in at least one device 10.

[0030] The device control unit 270 is connected to the status acquisition unit 210 and the instruction input unit 260. The device control unit 270 controls each device 10 according to the device status data acquired by the status acquisition unit 210. The device control unit 270 may control each device 10 according to instructions received from the user or from the monitoring device 40.

[0031] Figure 3 shows the processing flow of the control system 20 according to this embodiment. In step 300 (S300), the status acquisition unit 210 acquires device status data indicating the status of each device 10 from each device 10. The status acquisition unit 210 may receive device data indicating the status of at least one device 10 from the monitoring device 40.

[0032] In S310, the status determination unit 220 determines whether each piece of equipment 10 is normal or abnormal. The status determination unit 220 may determine whether the piece of equipment 10 is normal (not abnormal) based on whether at least one of the internal status values ​​or measured values ​​of the piece of equipment 10 is within a predetermined normal range corresponding to that value. Alternatively, the status determination unit 220 may calculate a health index indicating the soundness of operations in the entire facility 1 or a specific range within the facility 1 from at least one of the internal status values ​​or measured values ​​of two or more pieces of equipment 10 using a predetermined calculation formula, and determine whether the operations in the entire facility 1 or a specific range within the facility 1 are normal based on whether the value of the health index is within a normal range.

[0033] In S320, the display processing unit 230 performs display control to display a display screen on the display device 240 that includes the internal status values ​​and measured values ​​of each device 10, health indicators for the entire facility 1 or a specific area within facility 1, and the abnormality status of each device 10. The display processing unit 230 may generate a video output of the display screen and supply it to the display device 240, or it may generate HTML or a script for generating the display screen and send it to the display device 240.

[0034] In S330, the instruction input unit 260 receives instructions from the user to the control system 20 from the input device 250. The instruction input unit 260 may also receive instructions from the monitoring device 40, which has detected an abnormality in at least one piece of equipment 10, to perform control to address the abnormality.

[0035] In S340, the device control unit 270 controls each device 10 according to a predetermined control algorithm or control model using the device state data acquired by the state acquisition unit 210. The device control unit 270 may calculate control values ​​for each device 10 using PI control or PID control, etc. The device control unit 270 may also calculate control values ​​for each device 10 according to the device state data acquired by the state acquisition unit 210 using various machine learning models, etc. The device control unit 270 may change the control content for each device 10 in response to instructions from the instruction input unit 260.

[0036] The control system 20 repeats the process from S300 to S340. This allows the control system 20 to adaptively control each device 10 according to the state of each device 10.

[0037] Figure 4 shows the configuration of the robot 30 according to this embodiment. The robot 30 includes one or more sensors 400, one or more sensors 410, one or more actuators 420, a state acquisition unit 430, a communication unit 440, and an actuator control unit 450.

[0038] Each of the one or more sensors 400 detects or measures the conditions within facility 1. In the example shown in the figure, sensor 400a is an acoustic sensor that detects sound within facility 1 and outputs it as acoustic data. Sensor 400a may be an array microphone in which multiple microphones are arranged in a two-dimensional array. Sensor 400b is an image sensor that captures a three-dimensional or two-dimensional image within facility 1 and detects the image data. Sensor 400c is a LiDAR sensor. Sensor 400c measures at least one of the shape of each object within the irradiation range of the light or laser or the distance to each irradiation point by irradiating light or a laser to the outside and detecting the reflected light, and outputs the measurement result as LiDAR data. The robot 30 may be equipped with at least one of a temperature sensor, a humidity sensor, a gas sensor, or various other sensors.

[0039] Each of the one or more sensors 410 detects or measures the state of the robot 30. In the example shown in the figure, sensor 410a is a position sensor that detects the position of the robot 30 and outputs it as position data. Sensor 410a may be a GPS, specifically a GPS receiver that receives signals from GPS (Global Positioning System) satellites to determine the position of the robot 30. Alternatively, sensor 410a may be a sensor for determining the position of the robot 30 using any position detection system capable of detecting the position of the robot 30 within the facility 1.

[0040] Sensor 410b is an orientation sensor, such as a geomagnetic sensor. Sensor 410b measures the orientation of the robot 30 or the orientation of the detection direction of each sensor 400. Sensor 410c is an angle sensor or elevation sensor, etc., for measuring the detection direction of each sensor 400. Sensors 410b and 410c output direction data to the state acquisition unit 430 indicating at least one of the orientation of the robot 30 or the detection direction of each sensor 400. The robot 30 may be equipped with at least one of a speed sensor, an acceleration sensor, or various other sensors.

[0041] Each of the one or more actuators 420 is a motor or the like for driving parts of the robot 30. The robot 30 may be equipped with various actuators 420, such as actuators 420 for movement by driving or flying, actuators 420 for changing the orientation of the robot 30 or a part of it, and actuators 420 for operating equipment such as arms.

[0042] The state acquisition unit 430 is connected to one or more sensors 400 and one or more sensors 410. The state acquisition unit 430 acquires the state inside facility 1 measured by one or more sensors 400 and the state of robot 30 measured by one or more sensors 410. The state acquisition unit 430 may acquire the state inside facility 1 by receiving various measurement data such as acoustic data, image data, and LiDAR data from one or more sensors 400. The state acquisition unit 430 may acquire the state of robot 30 by receiving various measurement data such as position data and direction data from one or more sensors 410.

[0043] The communication unit 440 is connected to the status acquisition unit 430. The communication unit 440 communicates with the monitoring device 40 wirelessly or via wired connection. The communication unit 440 transmits various measurement data acquired by the status acquisition unit 430 to the monitoring device 40. The communication unit 440 also receives various instructions for the robot 30 from the monitoring device 40.

[0044] The actuator control unit 450 controls each actuator 420 according to various measurement data acquired by the state acquisition unit 430 and instructions from the monitoring device 40. As a result, the robot 30 can perform actions corresponding to the state within the facility 1, the state of the robot 30, and instructions from the monitoring device 40.

[0045] Some of the robots 30 may be permanently installed within the facility 1, and such robots 30 do not need to be equipped with actuators for movement. In addition, the facility 1 may be equipped with one or more sensors 400, but may also be equipped with monitoring equipment (surveillance cameras, monitoring microphones, etc.) that are not classified as robots. Such monitoring equipment will have some of the functions and configurations of the robots 30. Therefore, for the sake of explanation, in this specification, at least one of the robots 30 may be such monitoring equipment, and such monitoring equipment will perform processing etc. within the scope of its implemented functions.

[0046] Figure 5 shows the processing flow of the robot 30 according to this embodiment. In S500, one or more sensors 400 observe the state of facility 1 in the vicinity of robot 30. One or more sensors 410 observe the state of robot 30. The state acquisition unit 430 acquires various measurement data indicating the state of facility 1 from one or more sensors 400. The state acquisition unit 430 acquires various measurement data indicating the state of robot 30 from one or more sensors 410.

[0047] In S510, the communication unit 440 transmits various measurement data indicating the state of the facility 1 and the state of the robot 30, observed by one or more sensors 400 and one or more sensors 410, to the monitoring device 40. Here, if the measurement center direction of each sensor 400 coincides with the direction of the robot 30, that is, if, for example, the sensor unit on which each sensor 400 is mounted is facing the front of the robot 30, then the measurement direction (measurement center direction) of each sensor 400 coincides with the direction of the robot 30. In such a case, if the robot 30 transmits its own direction to the monitoring device 40, the monitoring device 40 can obtain the measurement direction of each sensor 400.

[0048] If the orientation of the sensor unit on which each sensor 400 is mounted is variable relative to the robot 30 body, the measurement center direction of each sensor 400 and the direction of the robot 30 do not necessarily coincide. In this case, the robot 30 may transmit direction data to the monitoring device 40 that includes the direction of the robot 30 itself and the measurement direction of each sensor 400. Alternatively, the robot 30 may transmit direction data to the monitoring device 40 that includes the direction of the robot 30 itself and the difference or offset between the measurement direction of each sensor 400 and the direction of the robot 30 itself. In this case, the monitoring device 40 can obtain the measurement direction of each sensor 400 by adding the difference or offset between the measurement direction of each sensor 400 and the direction of the robot 30 itself.

[0049] In S520, the communication unit 440 receives instructions from the monitoring device 40. In S530, the actuator control unit 450 performs actions in accordance with the instructions from the monitoring device 40. The actuator control unit 450 operates the robot 30 as instructed by driving at least one actuator 420 in response to movement instructions, direction change instructions, direction change instructions for acoustic sensors and image sensors, etc., received from the monitoring device 40. The actuator control unit 450 may also set various parameters within the robot 30 in response to instructions from the monitoring device 40.

[0050] The robot 30 repeats the processing from S500 to S530. This allows the robot 30 to move around the facility 1 in response to instructions from the monitoring device 40, and to perform tasks such as sound collection with sensor 400a, image capture with sensor 400b, and acquisition of LiDAR data with sensor 400c.

[0051] Figure 6 shows the configuration of the monitoring device 40 according to this embodiment, along with the collected information database 50, the equipment database 60, the robot database 70, the input device 680, and the display device 690. The input device 680 receives instructions from users such as operators, workers, or maintenance personnel of facility 1 and supplies them to the monitoring device 40. The display device 690 displays the display screen output by the monitoring device 40. The input device 680 and the display device 690 may be provided on a monitoring console or user terminal connected to the monitoring device 40. The display device 690 and the input device 680 may be shared with the display device 240 and input device 250 shown in Figure 2.

[0052] The monitoring device 40 may be an equipment management device that manages each piece of equipment 10 within the facility 1, or it may be a device that implements some of the functions included in the equipment management device. The monitoring device 40 may be a computer such as a workstation, server computer, general-purpose computer, or other computer, or it may be a computer system in which multiple computers are connected. Such a computer system is also a computer in a broad sense. The monitoring device 40 may also be implemented by a virtual computer environment that can run one or more programs within the computer. Such a computer functions as the monitoring device 40 by executing a program for monitoring each piece of equipment 10 using the robot 30. Alternatively, the monitoring device 40 may be a dedicated computer designed for monitoring each piece of equipment 10, or it may be dedicated hardware implemented by a dedicated circuit.

[0053] In this embodiment, the monitoring device 40 is installed within the facility 1, but alternatively, the monitoring device 40 may be provided outside the facility 1 via a cloud computing system on the internet or the like. Also, in this embodiment, the monitoring device 40 remotely operates each robot 30, but alternatively, the monitoring device 40 may be mounted on at least one robot 30 and directly operate each robot 30 or a group of robots including two or more robots 30 on site.

[0054] The monitoring device 40 includes a communication unit 600, an information collection database connection unit 620, an equipment database connection unit 630, a robot database connection unit 640, a monitoring processing unit 650, an input processing unit 660, a display processing unit 665, and a communication unit 670. The communication unit 600 communicates with the robot 30 wirelessly or via wired connection. The communication unit 600 includes an acoustic data acquisition unit 602, an image data acquisition unit 604, a LiDAR data acquisition unit 606, a position data acquisition unit 608, a direction data acquisition unit 610, and an instruction transmission unit 612.

[0055] The acoustic data acquisition unit 602 acquires acoustic data detected by the sensor 400a of the robot 30, which can move around the facility 1 and collect sound. The image data acquisition unit 604 acquires image data captured within the facility 1. The image data acquisition unit 604 may acquire image data by receiving image data detected by the sensor 400b of the robot 30. The LiDAR data acquisition unit 606 acquires LiDAR data detected by the sensor 400c of the robot 30 within the facility 1.

[0056] The position data acquisition unit 608 acquires position data indicating the position of the robot 30. The position data acquisition unit 608 may acquire position data by receiving position data transmitted by the robot 30. The direction data acquisition unit 610 acquires direction data indicating the direction of the robot 30 and each sensor 400. The direction data acquisition unit 610 may acquire direction data by receiving direction data transmitted by the robot 30. The instruction transmission unit 612 transmits instructions for the robot 30, which have been determined in the monitoring device 40, to the robot 30.

[0057] The data collection database connection unit 620 is connected to the data collection database 50, the communication unit 600, and the monitoring processing unit 650. The data collection database connection unit 620 records various measurement data acquired by the acoustic data acquisition unit 602, image data acquisition unit 604, LiDAR data acquisition unit 606, position data acquisition unit 608, and direction data acquisition unit 610 within the communication unit 600 as collected data in the data collection database 50. The data collection database connection unit 620 may add position data acquired by the position data acquisition unit 608 and direction data acquired by the direction data acquisition unit 610 to the acoustic data acquired by the acoustic data acquisition unit 602 and record them as collected data in the data collection database 50. The data collection database connection unit 620 also accesses the data collection database 50 in response to requests from the monitoring processing unit 650. Furthermore, if there is no need to record the history of collected data, the monitoring device 40 does not need to be equipped with a data collection database connection unit 620, and the measurement data acquired by the communication unit 600 may be supplied to the monitoring processing unit 650 without going through the data collection database connection unit 620. In this case, the data collection database 50 is not required.

[0058] The device database connection unit 630 is connected to the device database 60 and the monitoring processing unit 650. The device database connection unit 630 accesses the device database 60 in response to requests from the monitoring processing unit 650. The robot database connection unit 640 is connected to the robot database 70 and the monitoring processing unit 650. The robot database connection unit 640 accesses the robot database 70 in response to requests from the monitoring processing unit 650.

[0059] The monitoring processing unit 650 is connected to the collected information database connection unit 620, the equipment database connection unit 630, the robot database connection unit 640, the input processing unit 660, and the communication unit 670. The monitoring processing unit 650 reads robot data from each robot 30 via the robot database connection unit 640 and uses the robot data to determine the monitoring actions within facility 1 that each robot 30 should perform. The monitoring processing unit 650 instructs each robot 30 to perform each action included in the determined monitoring action via the instruction transmission unit 612 in the communication unit 600.

[0060] Furthermore, the monitoring processing unit 650 reads the collected data collected by one or more robots 30, which is recorded in the collected information database 50, via the collected information database connection unit 620. The monitoring processing unit 650 then uses the collected data to detect the status of each device 10 monitored by each robot 30. The monitoring processing unit 650 uses the collected data to determine whether each device 10 is normal or abnormal. The monitoring processing unit 650 records the processing results for each device 10 in the collected information database 50 via the collected information database connection unit 620. The monitoring processing unit 650 may also read the device data of each device 10 via the device database connection unit 630 and determine whether each device 10 is normal or abnormal based on the information registered in the device data.

[0061] The monitoring processing unit 650 may have the display processing unit 665 generate a display screen showing the status of each device 10 and the result of determining whether it is normal or abnormal, and display it on the display device 690. The monitoring processing unit 650 may transmit the status of each device 10 and the result of determining whether it is normal or abnormal to the control system 20 via the communication unit 670.

[0062] The monitoring processing unit 650 may, as a result of monitoring each device 10 using one or more robots 30, generate an instruction to perform control to address the abnormality if an abnormality is detected in at least one device 10. The monitoring processing unit 650 may transmit the instruction to perform control to address the abnormality in the device 10 to the control system 20 via the communication unit 670. The monitoring processing unit 650 may receive operation data from the control system 20 via the communication unit 670 indicating the operation of each device 10 (for example, whether it is running or stopped).

[0063] The input processing unit 660 is connected to the input device 680. The input processing unit 660 receives information input from the user to the input device 680 and supplies it to the monitoring processing unit 650. The display processing unit 665 is connected to the monitoring processing unit 650. The display processing unit 665 performs display processing to generate a display screen and display it on the display device 690 in response to instructions from the monitoring processing unit 650.

[0064] The communication unit 670 is connected to the monitoring processing unit 650. The communication unit 670 may be connected to the control system 20 and may be able to communicate with the control system 20 wirelessly or via wired connection. The communication unit 670 may transmit the device status data of each device 10 detected by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit the normal or abnormal determination result of each device 10 determined by the monitoring processing unit 650 to the control system 20. The communication unit 670 may transmit an instruction to the control system 20 to take control to address the abnormality in response to the detection of an abnormality in at least one of the devices 10. The communication unit 670 does not have to have the function of transmitting at least one of the following to the control system 20: the device status data of the device 10, the normal or abnormal determination result of the device 10, or an instruction to take control to address the abnormality. The monitoring device 40 does not have to include the communication unit 670 if it does not have any of these functions.

[0065] As described above, the monitoring device 40 allows for monitoring of the facility 1 using one or more robots 30 that can move within the facility 1, thereby enabling the monitoring of a large number of devices 10 using a relatively small number of robots 30. Furthermore, since the monitoring device 40 can control the robots 30 in accordance with the conditions at each observation point to collect necessary information, it becomes possible to detect the state of each device 10 that cannot be collected by sensors or other means pre-installed on each device 10.

[0066] Figure 7 shows an example of the data structure of the collected information database 50 according to this embodiment. The collected information database 50 records the measurement data collected from each robot 30 as collected data. The collected information database 50 may record one or more records, each of which is a single unit of collected data. In the data structure of the collected information database 50 shown in this figure, each record of collected data is arranged in rows, and the collected data in each record includes the fields "robot identification information", "date and time", "detection location", and "measurement data".

[0067] "Robot Identification Information" is a field that records data values ​​such as an identification number, serial number, or other unique number or string that can identify the robot 30 that acquired the measurement data for the corresponding record. "Date and Time" is a field that records the date and time when the measurement data for the corresponding record was detected by the robot 30. "Detection Position" is a field that records at least one position and orientation of the robot 30 or each sensor 400 at the time the measurement data for the corresponding record was detected. The collected information database 50 may record at least one of the following as "Detection Position" at the time indicated by "Date and Time": the position detected by the robot 30's sensor 410a, the orientation of the robot 30 detected by sensor 410b, or the orientation of the robot 30 or each sensor 400 detected by sensor 410c.

[0068] The "Measurement Data" field is a field that records measurement data detected by the robot 30 at the time indicated by "Date and Time". The data recorded as "Measurement Data" may be various types of measurement data acquired by the robot 30, such as at least one of acoustic data, image data, or LiDAR data.

[0069] The communication unit 600 within the monitoring device 40 may receive data including the date and time of acquisition of measurement data by the robot 30, the robot identification information of the robot 30, and measurement data relating to the position and direction of the robot 30, as separate packets or groups of packets, etc. In this case, the communication unit 600 updates the position and direction of the robot 30 managed by the monitoring device 40 each time it receives measurement data relating to the position and direction of the robot 30. When the communication unit 600 receives measurement data relating to the state of the facility 1, it generates a record of collected data by associating that measurement data with the position and direction (detection position) of the robot 30 at the date and time of acquisition of that measurement data. Alternatively, when the communication unit 440 in the robot 30 transmits various measurement data indicating the state of the facility 1 and the state of the robot 30 observed by each sensor 400 and each sensor 410 to the monitoring device 40 (see S510 in Figure 5), it may assemble one record of collected data from these measurement data, encode it into one or more packets, and transmit it to the monitoring device 40.

[0070] Figure 8 shows an example of the data structure of the equipment database 60 according to this embodiment. The equipment database 60 records equipment data for each of the multiple pieces of equipment 10 within the facility 1. The equipment database 60 may record equipment data for each piece of equipment 10 or for each type of equipment 10. Alternatively, the equipment database 60 may record records, each representing equipment data for one piece of equipment, for a number of pieces of equipment 10. In the data structure of the equipment database 60 shown in this figure, each record of equipment data is arranged in rows, and the equipment data in each record includes the fields "equipment information," "location," "template," and "acoustic data."

[0071] "Equipment Information" is a field for recording equipment type information (type of equipment 10, product name, or general name, etc.) of the corresponding equipment 10, such as a reactor, measuring instrument, piping, or flange. "Equipment Information" may further record data values ​​such as an identification number, serial number, or other unique number or string that allows the equipment 10 to be identified within Facility 1, and may also record the equipment name assigned to the corresponding equipment 10 by a user, etc. "Location" is a field for recording the location of the equipment 10 within Facility 1. "Template" is a field for recording three-dimensional image data (template image data) of the corresponding type of equipment 10 for use in identifying the type of equipment 10 using images. The template image data may be different for each piece of equipment 10, and may also be different for each type of equipment 10, that is, it may be common to multiple pieces of equipment 10 of the same type. "Acoustic data" is a field that records at least one of the following: acoustic data collected from the corresponding device 10 or a device of the same type as device 10, or acoustic data emitted by the corresponding device 10 or a device of the same type as device 10 when the corresponding device 10 or a device of the same type as device 10 is functioning correctly.

[0072] Figure 9 shows an example of the data structure of the robot database 70 according to this embodiment. The robot database 70 records robot data for each of the one or more robots 30 in the facility 1. The robot database 70 may record as many records as there are robots 30, each record representing the robot data for one robot. In the data structure of the robot database 70 shown in this figure, each record of robot data is arranged in rows, and the robot data in each record includes the fields "robot identification information", "robot name", "robot information", "position / direction", and "schedule information".

[0073] The "Robot Identification Information" field records data values ​​such as an identification number, serial number, or other unique number or string that allows the robot 30 to be identified within Facility 1. The "Robot Name" field records the robot name assigned to the corresponding robot 30 by the user or other party.

[0074] The "Robot Information" field records various information about the corresponding robot 30. The "Position / Direction" field records the position and direction of the corresponding robot 30 within Facility 1, and, if necessary, the direction of each sensor 400 (absolute direction or offset relative to the direction of the robot 30). The "Schedule Information" field records the schedule for the corresponding robot 30 to patrol Facility 1 and collect sounds within Facility 1.

[0075] Figure 10 shows the configuration of the monitoring processing unit 650 of the monitoring device 40 according to this embodiment. The monitoring processing unit 650 includes a sound source separation unit 1000, a range detection unit 1010, a source identification unit 1020, a sound abnormality detection unit 1030, an abnormal equipment identification unit 1040, an equipment command unit 1050, and a robot command unit 1060. The monitoring processing unit 650 uses image data to identify the equipment 10 that is the source of the sound components contained in the acoustic data and detects an abnormality in the equipment 10.

[0076] The sound source separation unit 1000 is connected to the collected information database connection unit 620. The sound source separation unit 1000 reads out the collected data registered in the collected information database 50, in which acoustic data is recorded as measurement data, via the collected information database connection unit 620. As a result, the sound source separation unit 1000 obtains the acoustic data, the date and time the acoustic data was acquired, and the location and direction in which the acoustic data was detected within the facility 1. The sound source separation unit 1000 extracts at least one sound component from the acquired acoustic data by separating the sound source, and detects the location of the source of at least one sound component (also called the sound source location).

[0077] The range detection unit 1010 is connected to the data collection database connection unit 620 and the equipment database connection unit 630. The range detection unit 1010 reads out the collected data registered in the data collection database 50, in which image data is recorded as measurement data, via the data collection database connection unit 620. As a result, the range detection unit 1010 obtains the image data, the date and time the image data was acquired, and the imaging position and imaging direction in which the image data was detected within the facility 1. From the acquired image data, the range detection unit 1010 detects the three-dimensional range of at least one equipment 10 captured in the image data.

[0078] The source identification unit 1020 is connected to the sound source separation unit 1000 and the range detection unit 1010. The source identification unit 1020 uses the three-dimensional range detected by the range detection unit 1010 to identify which of the at least one device 10 captured in the image data is the source of at least one sound component contained in the acoustic data. The source identification unit 1020 instructs the device database connection unit 630 to record the sound component data for each sound component in the device database 60, associating it with the device 10 that is the source among the at least one device 10 captured in the image data. If the source identification unit 1020 cannot identify the device 10 corresponding to any of the sound components, it may receive information input from the user via the input processing unit 660 to specify the corresponding device 10.

[0079] The sound anomaly detection unit 1030 is connected to the source identification unit 1020. The sound anomaly detection unit 1030 uses sound component data associated with each device 10 to detect the degree of anomaly in the sound components.

[0080] The abnormal device identification unit 1040 is connected to the sound abnormality detection unit 1030. Based on the degree of abnormality of the sound component detected by the sound abnormality detection unit 1030, the abnormal device identification unit 1040 identifies the abnormality of the device 10 that is the source of the sound component. The abnormal device identification unit 1040 may have the display processing unit 665 generate a display screen to display the status of each device 10 detected by the robot 30 on the display device 690 and display it on the display device 690. Alternatively, the abnormal device identification unit 1040 may have the display processing unit 665 generate a display screen to display the result of determining whether each device 10 is normal or abnormal on the display device 690 and display it on the display device 690.

[0081] The equipment command unit 1050 is connected to the abnormal equipment identification unit 1040. When an abnormality is identified in any of the equipment 10 that is the source of the sound component, the equipment command unit 1050 controls at least one of the equipment 10 that has been identified as abnormal in the facility 1 or other equipment 10 related to the operation of said equipment 10 to address the abnormality of equipment 10. The equipment command unit 1050 transmits an instruction to the instruction input unit 260 in the control system 20 via the communication unit 670 to control the operation of equipment 10 to address the abnormality of equipment 10.

[0082] The robot command unit 1060 is connected to the collected information database connection unit 620, the robot database connection unit 640, and the source identification unit 1020. The robot command unit 1060 determines the actions that each robot 30 should perform and instructs each robot 30 to perform the determined actions.

[0083] The monitoring processing unit 650 described above has the components shown in Figure 10 in order to realize various functions such as identifying the equipment 10 that is the source of each sound component, identifying abnormalities in the source equipment 10 based on the degree of abnormality of the sound component, issuing control instructions to deal with the abnormalities in the equipment 10, and issuing operation instructions to each robot 30. Alternatively, the monitoring processing unit 650 may adopt a configuration that does not have some of the components shown in Figure 10.

[0084] For example, the monitoring processing unit 650 does not have an equipment command unit 1050 and does not need to issue control instructions to deal with abnormalities in the equipment 10. Also, the monitoring processing unit 650 does not have a robot command unit 1060 and does not need to issue operation instructions to each robot 30. Even if the monitoring device 40 does not have some of the functions exemplified here, the monitoring device 40 can still be used by the user to receive information output or displayed by the monitoring device 40 to detect abnormalities in the equipment 10 and to have technical measures taken to deal with the abnormality in at least one piece of equipment 10 within the facility 1.

[0085] Figures 11 and 12 show the processing flow of the monitoring device 40 according to this embodiment. For the sake of explanation, the processing flows in Figures 11 and 12 show the case where the monitoring device 40 performs monitoring within the facility 1 using one robot 30. The monitoring device 40 may perform the processing flows in Figures 11 and 12 for each of multiple robots 30.

[0086] In S1100, the robot command unit 1060 instructs the robot 30 to be processed to move within the facility 1 via the instruction transmission unit 612. Here, the robot command unit 1060 may obtain a schedule for the robot 30 to patrol the facility 1 by referring to schedule information recorded in the robot database 70 associated with the robot 30 to be processed. This schedule may include information necessary to determine the operation of the robot 30, such as the location of each observation point where the robot 30 should observe the state of the facility 1, the time to arrive at each observation point, the movement route between observation points, or the observation direction at each observation point. In accordance with this schedule, the robot command unit 1060 supplies the robot 30 with an instruction to move to the next observation point and to perform observation at the next observation point. In response, the robot 30 moves to the next observation point (see S520 and S530 in Figure 5). If an area within Facility 1 where an abnormality may have occurred has been identified, the robot command unit 1060 may instruct at least one robot 30 to go to that area and to move to various locations within that area to acquire acoustic and image data.

[0087] In S1110, the robot 30 observes the state of the facility 1 and the state of the robot 30 using each sensor 400 at the observation point (see S500 in Figure 5). The robot 30 transmits various measurement data indicating the observed state of the facility 1 and the state of the robot 30 to the monitoring device 40 (see S510 in Figure 5). The image data acquisition unit 604 in the monitoring device 40 acquires, for example, three-dimensional image data from the measurement data transmitted from the robot 30. The collected information database connection unit 620 may add the acquisition time, position data acquired by the position data acquisition unit 608, and direction data acquired by the direction data acquisition unit 610 to the image data acquired by the image data acquisition unit 604 and record it in the collected information database 50 as collected data.

[0088] In S1120, the range detection unit 1010 detects the three-dimensional range of one or more objects captured in the image data by performing image recognition on the image data from the robot 30. The range detection unit 1010 may extract one or more three-dimensional ranges captured in the three-dimensional image data and obtain equipment type information by identifying the corresponding equipment type 10 for each three-dimensional range.

[0089] The range detection unit 1010 may use segmentation of the 3D image to recognize the 3D range of each object contained in the 3D image data. For example, the range detection unit 1010 may group the points in the 3D image data into regions based on similar features (at least one such feature, such as the color of each point or the density of points), and divide the image into multiple regions, thereby dividing it into the 3D ranges of multiple objects.

[0090] The range detection unit 1010 may acquire equipment type information indicating the type of equipment 10 corresponding to one or more three-dimensional ranges (also called the target three-dimensional ranges) contained in the image data. The range detection unit 1010 may identify the type of equipment 10 for each of at least one three-dimensional ranges contained in the image data by matching it with predetermined two-dimensional or three-dimensional template image data. For example, the range detection unit 1010 may extract one or more three-dimensional ranges captured in the image data and compare the template image data corresponding to each equipment 10 or type acquired from the equipment database 60 with each three-dimensional range. The range detection unit 1010 may acquire the type of equipment 10 associated with the target three-dimensional range as equipment type information corresponding to that three-dimensional range, which is either a match or similar to a predetermined threshold or higher (for example, the most similar) template image data.

[0091] The range detection unit 1010 may identify the type of device 10 associated with the template image data as corresponding to the 3D range, depending on whether the difference between the 3D range of the target and the occupied space of the template image data is less than or equal to a predetermined threshold. Here, the predetermined threshold may be fixed, or it may be a predetermined ratio to the total volume of the 3D range or template image data. The range detection unit 1010 may compare the 3D range of the target and the template image data by adjusting at least one direction and position of the 3D range of the target or the template image data so as to minimize the difference in occupied space.

[0092] Furthermore, the range detection unit 1010 may determine that the type of device 10 associated with the template image data corresponds to the three-dimensional range of the target and the template image data, depending on whether the difference between the shapes of the ranges visible from one direction (for example, from the image acquisition position in the imaging direction) is below a predetermined threshold.

[0093] The range detection unit 1010 may search the device database 60 to obtain template image data associated with one or more devices 10 located within a predetermined distance range from the acquisition location of the target image data, and use these for comparison.

[0094] Furthermore, the range detection unit 1010 may acquire equipment type information using a machine learning model that outputs equipment type information in response to input data indicating a three-dimensional range (for example, coordinates of a point cloud included in the three-dimensional range). The range detection unit 1010 may acquire the corresponding equipment type 10 output by inputting coordinates indicating a three-dimensional range into the machine learning model as equipment type information. The range detection unit 1010 may store a machine learning model trained using data indicating a corresponding three-dimensional range labeled for each type of equipment 10, or trained parameters representing the machine learning model, in the equipment information field of the equipment database 60, etc. The range detection unit 1010 may generate or update the machine learning model through training so that it outputs the correct type of equipment 10 when data indicating a three-dimensional range is input to the machine learning model. The range detection unit 1010 may use a neural network, statistical learning, or other machine learning algorithm. A machine learning model may be prepared for each individual equipment 10, or for each type of equipment 10. Also, the machine learning model may be common to all equipment 10.

[0095] The range detection unit 1010 may recognize a reference mark of a reference sound source placed within the facility 1 in the image data. The image data acquisition unit 604 acquires image data in which the reference marks of at least three reference sound sources are captured, and the range detection unit 1010 may recognize the reference marks in the image data. For example, the range detection unit 1010 may detect the position (3D coordinates, etc.) of the reference mark in image data at the same imaging position and imaging direction as the image data in which the 3D range of the device 10 was detected (for example, the same image data as the image data in which the 3D range was detected). Here, the reference sound source may be a device such as a speaker that outputs a reference sound component fixed at a predetermined position within the facility 1. The reference sound component may be a sound component that has different characteristics from the sound component generated by the device 10 within the facility 1, and for example, may have a different frequency range from the frequency range of the sound component generated by the device 10. The reference mark may be a feature that allows the range detection unit 1010 to identify the reference sound source through image recognition, and may be an external feature of the reference sound source (shape, pattern) or an identification mark (e.g., a one-dimensional or two-dimensional code) attached to the outer surface of the reference sound source. The range detection unit 1010 may store information for identifying the reference mark in advance.

[0096] The range detection unit 1010 can generate a 3D map that includes multiple 3D ranges located at different positions by detecting the 3D range of each device 10 in the image data.

[0097] In S1130, the acoustic data acquisition unit 602 in the monitoring device 40 acquires acoustic data from the measurement data transmitted from the robot 30. The collected information database connection unit 620 may add position data acquired by the position data acquisition unit 608 and direction data acquired by the direction data acquisition unit 610 to the acoustic data acquired by the acoustic data acquisition unit 602 and record it in the collected information database 50 as collected data.

[0098] In S1140, the sound source separation unit 1000 extracts at least one sound component from the acoustic data of the robot 30 by sound source separation or the like. The sound source separation unit 1000 may perform sound source separation by extracting at least one sound component from the acoustic data whose sound source directions are different from those of the robot 30. The sound source separation unit 1000 may identify the sound source position of each sound component as seen from the robot 30 (sound source localization).

[0099] Here, each sensor 400a collects sound from a predetermined range of directions other than the measurement center direction of the sensor 400a according to its directivity, thus collecting sound that is a synthesis of sounds from various sound sources. The sound source separation unit 1000 processes the sound collected by each sensor 400a to separate or decompose it into the sounds generated by each source. In this specification, for the sake of explanation, the sound extracted from at least a portion of the sound collected by the sensor 400a is referred to as a "sound component," but the "sound component" itself is also sound. Therefore, the acoustic data, which is the data of the sound collected by the sensor 400a, and the sound component data, which is the data of the sound components, may be in the same data format.

[0100] The sound source separation unit 1000 may utilize various sound source separation and sound source localization techniques. For example, the robot 30 may have two or more sensors 400a, and the acoustic data received by the monitoring device 40 may include acoustic data for each channel, such as recordings of sounds collected by each sensor 400a.

[0101] The sound source separation unit 1000 may calculate the sound source position of each sound component relative to the robot 30 from the time difference in which each sound component included in the acoustic data reaches each sensor 400a. Here, the sound source separation unit 1000 may calculate the relative sound source position of each sound component with respect to a sensor unit including two or more sensors 400a. The sound source separation unit 1000 may calculate the sound source position in three axes with the sensor unit as the origin as the relative sound source position. Alternatively, the sound source separation unit 1000 may calculate the sound source position in three axes with respect to a single point within the facility 1 (for example, the center of the facility) as the absolute sound source position. For example, the sound source separation unit 1000 may calculate the absolute sound source position by adding the coordinates of the relative sound source position with respect to the direction of movement of the robot to the position (coordinates) of the robot 30 within the facility 1 in three axes with respect to a single point within the facility 1 as the origin.

[0102] The sound source separation unit 1000 may separate the reference sound component output by a reference sound source located within the facility 1 in the acoustic data. The acoustic data acquisition unit 602 acquires acoustic data containing reference sound components from at least three reference sound sources located within the facility 1, and the sound source separation unit 1000 may detect the sound source position of the reference sound component from the acquired acoustic data. Here, the sound component used to identify the equipment 10 separated by the sound source separation unit 1000, which is different from the reference sound component, is also called the target sound component. The sound source separation unit 1000 may separate the reference sound component and the target sound component from the same acoustic data, or from different acoustic data. The sound source separation unit 1000 may detect the sound source position (3D coordinates, etc.) of the reference sound component in the acoustic data at the same detection position and direction as the position and direction in which the image data of the reference mark was captured. The sound source separation unit 1000 may store information for identifying the reference sound component (a predetermined frequency range, etc.) in advance.

[0103] By separating the sound sources of the acoustic data in this manner, the monitoring device 40 can identify the sound source location of each sound component originating from each device 10, even when acquiring acoustic data in which sounds from two or more devices 10 are superimposed. As a result, the monitoring device 40 can increase the detection rate of abnormalities in the devices 10 and take appropriate action in response to abnormalities in the devices 10.

[0104] In S1150, the source identification unit 1020 may use a three-dimensional map showing the three-dimensional range of each device 10 detected by the range detection unit 1010 and device type information corresponding to each three-dimensional range to identify which of the at least one device 10 captured in the image data is the source of at least one sound component included in the acoustic data. The source identification unit 1020 may identify which of the three-dimensional ranges in the three-dimensional map the sound source position of each target sound component included in the acoustic data is included in. The source identification unit 1020 receives device type information corresponding to the three-dimensional range located at the sound source position of the target sound component from the range detection unit 1010 and may identify one type of device 10 indicated by the received device type information as the source.

[0105] For example, with respect to a certain target sound component and a certain three-dimensional range detected from acoustic and image data observed by the robot 30 at the same location and at the same or different timings, the source identification unit 1020 may determine that the source of this target sound component is the device 10 corresponding to this three-dimensional range if the sound source position of the target sound component as seen from the observation point (e.g., relative sound source position) is included in the three-dimensional range, or if the difference between the sound source position and the three-dimensional range is within a predetermined error range.

[0106] The source identification unit 1020 may identify the device 10 that is the source of at least one sound component included in the acoustic data by identifying the positional relationship between the three-dimensional range and the sound source position of at least one sound component included in the acoustic data, based on the reference sound component and reference marks. The source identification unit 1020 may use the reference sound component and reference marks to convert the coordinates of the sound source position of the target sound component into coordinates on the three-dimensional map. For example, the source identification unit 1020 may calculate a coordinate transformation matrix to transform the coordinates of the sound source positions of at least three reference sound components to the coordinates of the positions of at least three reference marks on the three-dimensional map. By applying the coordinate transformation matrix to the sound source position of the target sound component, the source identification unit 1020 can identify the positional relationship between the three-dimensional range and the sound source position of the target sound component. The source identification unit 1020 may identify which three-dimensional range on the three-dimensional map the coordinate-transformed sound source position of the target sound component falls into. Because the coordinate transformation causes the three axes of the sound source location's coordinates to coincide with the three axes of the 3D map's coordinates, the source identification unit 1020 can identify the source device 10 with greater accuracy.

[0107] The source identification unit 1020 may use operation data indicating the operation of at least one piece of equipment 10 within facility 1 to identify which of the at least one piece of equipment 10 captured in the image data is the source of at least one sound component contained in the acoustic data. The source identification unit 1020 may receive operation data indicating the operation of each piece of equipment 10 from the control system 20 via the communication unit 670. The source identification unit 1020 may identify which of the multiple pieces of equipment 10 is the source of the target sound component, excluding the piece of equipment 10 that the operation data indicates is stopped at the time of acquisition of the acoustic data of the target sound component. The source identification unit 1020 may identify only the piece of equipment 10 that the operation data indicates is in operation at the time of acquisition of the acoustic data of the target sound component as the source of the target sound component. This reduces the processing required for equipment identification.

[0108] The source identification unit 1020 may use non-target sound component data corresponding to the environment in which facility 1 is located to identify which of the at least one device 10 captured in the image data is the source of at least one sound component included in the acoustic data. The source identification unit 1020 may exclude from the target sound components any sound components separated by the sound source separation unit 1000 that match the non-target sound component data or are similar to a predetermined threshold or higher, thereby omitting the source identification operation. The source identification unit 1020 may pre-acquire and store sound components of ambient sound in facility 1 that are not sourced by device 10 (sound components such as noise or static) as non-target sound component data. Examples of non-target sound component data include sound components caused by rain inside facility 1 when the environment in which facility 1 is located is rainy (sound of rain hitting the roof or ground of facility 1), or sound components caused by ocean waves inside facility 1 when the environment in which facility 1 is located is near the sea. The sensor 400a may pre-detect sound components within facility 1 as non-target sound component data when device 10 is not operating.

[0109] The source identification unit 1020 may use the equipment type information corresponding to the three-dimensional range to identify more detailed information about the equipment 10 located at the sound source location of the target sound component. The source identification unit 1020 may access the equipment database 60 via the equipment database connection unit 630 and refer to each record of the equipment data to retrieve equipment information for the equipment 10 in facility 1 that corresponds to the type of equipment 10 and location information related to the sound source (at least one of the sound source location, the acoustic data detection location, or the image data detection location) indicated by the received equipment type information. For example, if the equipment data records that the piping indicated by the equipment type information is located at or within a predetermined range from the sound source location, the source identification unit 1020 may obtain equipment information (identification number, etc.) for that piping.

[0110] The source identification unit 1020 may use the three-dimensional range detected by the range detection unit 1010 to identify which of the at least one equipment 10 captured in the image data originates from a component of that equipment 10. If a single three-dimensional range contains the sound source locations of multiple target sound components, the source identification unit 1020 may access the equipment database 60 and refer to each record of the equipment data to identify the component of the equipment 10 corresponding to each sound source location. For example, in a three-dimensional range of type pump, the source identification unit 1020 may use the positions of the impeller and motor in the pump indicated by the equipment data (for example, relative positions such as the top or bottom of the pump) to identify that the sources of two target sound components are the impeller and the motor, respectively. If the source identification unit 1020 identifies that multiple sound source locations are included in a single three-dimensional range, the source identification unit 1020 may supply information on the multiple sound source locations for the equipment 10 corresponding to that three-dimensional range to the range detection unit 1010, so that the range detection unit 1010 can detect a three-dimensional range for each component of the equipment 10 that acts as a sound source through segmentation. Alternatively, if the source identification unit 1020 identifies that multiple sound source locations are included in a single three-dimensional range, the monitoring device 40 may acquire template image data corresponding to each sound source location. In this case, the equipment database 60 records multiple template image data for one type of equipment 10.

[0111] In S1160, the source identification unit 1020 instructs the equipment database connection unit 630 to record sound component data for each sound component in the equipment database 60, associating it with the equipment 10 identified by the source identification unit 1020. The source identification unit 1020 may also instruct the equipment database 60 to record sound component data in association with the equipment 10 in facility 1 that corresponds to the type of equipment 10 indicated by the equipment type information and the location information related to the sound source (at least one of the sound source location, the acoustic data detection location, or the image data detection location). Upon receiving instructions from the source identification unit 1020, the equipment database connection unit 630 records sound component data for each sound component in the acoustic data field of the record of equipment data corresponding to the equipment 10 that is the source in the equipment database 60. The equipment database connection unit 630 may record multiple sound component data acquired at different dates and times in the acoustic data field by appending the date and time of acquisition of the sound component data and the sound component data itself to the acoustic data field.

[0112] Furthermore, if the source of a certain sound component is a device 10 that is included in the image data but is not registered in the device database 60, the source identification unit 1020 may register this device 10 in the device database 60 after receiving information from the user via the input processing unit 660, and record the sound component data in the device database 60 in association with this device 10.

[0113] In S1200, the sound abnormality detection unit 1030 detects the degree of abnormality of the sound components using sound component data associated with each device 10. Here, the degree of abnormality may be expressed as a real number or integer within a predetermined range such as 0 to 1 or 0 to 100%, or it may be a binary value such as normal or abnormal.

[0114] If the acoustic data emitted by the device 10 when the device 10 is functioning normally is recorded in the device database 60, the sound anomaly detection unit 1030 may calculate an anomaly degree that indicates how much the target sound component data originating from the device 10 differs from the acoustic data emitted by the device 10 when it is functioning normally. For example, the sound anomaly detection unit 1030 may determine the anomaly degree based on the comparison result between the acoustic data when it is functioning normally and the target sound component data.

[0115] The sound anomaly detection unit 1030 may compare the normal acoustic data with the target sound component data in either the time domain or the frequency domain. When the comparison is performed in the time domain, the sound anomaly detection unit 1030 may adjust the phase difference so that the time integral of the difference (absolute value of the difference, etc.) between the normal acoustic data and the target sound component data is minimized, and use the time integral of the difference between the acoustic data and the target sound component data at the adjusted phase difference as the anomaly degree. In this case, the sound anomaly detection unit 1030 may adjust the amplitude so that the loudness of the acoustic data and the target sound component data are matched, for example, so that the average amplitude matches.

[0116] When comparing normal acoustic data with target sound component data in the frequency domain, the sound anomaly detection unit 1030 may calculate the degree of anomaly by integrating the frequency differences (absolute values ​​of differences, etc.) between the frequency spectrum of the normal acoustic data and the frequency spectrum of the target sound component data in the frequency direction. In this case, the sound anomaly detection unit 1030 may adjust the volume of the sound to match the acoustic data and the target sound component data so that they can be compared.

[0117] If acoustic data previously collected from the device 10 is recorded as one or more history entries in the acoustic data field of the device database 60, the sound anomaly detection unit 1030 may calculate a value as the degree of anomaly indicating how much the target sound component data originating from the device 10 differs from the one or more acoustic data previously collected from the device 10. The sound anomaly detection unit 1030 may calculate the degree of anomaly of the target sound component data for previously collected acoustic data in the same manner as the calculation method for calculating the degree of anomaly of the target sound component data for acoustic data emitted by the device 10 under normal conditions, as described above.

[0118] The sound anomaly detection unit 1030 may determine the degree of anomaly based on the degree of deviation of the target sound component data from the distribution of acoustic data previously collected from the device 10. Alternatively, the sound anomaly detection unit 1030 may calculate the degree of anomaly as the sum of the differences for each of a predetermined number of acoustic data from the acoustic data previously collected from the device 10 that have the smallest difference from the target sound component data (k-nearest neighbor method). In addition to the above, the sound anomaly detection unit 1030 may calculate the degree of anomaly of the acoustic data and the target sound component data using various methods for calculating the difference, similarity, or deviation between two sounds.

[0119] The sound anomaly detection unit 1030 may use AI technology to detect the degree of abnormality of sound components. The sound anomaly detection unit 1030 may receive sound components emitted by each device 10 from the source identification unit 1020 and train a machine learning model for each device 10. The sound anomaly detection unit 1030 may store the machine learning model for each device 10 or trained parameters representing the machine learning model in the device information field of the device database 60, and update the machine learning model each time it receives new sound components emitted by the device 10. The sound anomaly detection unit 1030 may also receive a judgment result of whether each sound component is normal or abnormal. In this case, the sound anomaly detection unit 1030 may generate or update the machine learning model through training so that when sound components are input to the machine learning model, it outputs the correct judgment result.

[0120] The sound anomaly detection unit 1030 may use a neural network, statistical learning, or other machine learning algorithms. The sound anomaly detection unit 1030 may use time-series data of sound components as input to the machine learning model, or it may use the frequency spectrum of sound components as input to the machine learning model. The sound anomaly detection unit 1030 updates the parameters of the machine learning model to reduce the error between the output of the machine learning model when samples of each sound component that serve as training data are input to the machine learning model, and the training label indicating whether the sound component is normal or abnormal. For example, when using a neural network, the sound anomaly detection unit 1030 inputs the data value of each time point or the data value of each frequency in the sound component to each input node of the input layer of the neural network. The sound anomaly detection unit 1030 uses the error between the output value output by the neural network in response to each sample input and the label to adjust the weights between each neuron of the neural network and the bias of each neuron using a method such as backpropagation.

[0121] The machine learning model may output the degree of abnormality of sound components in response to inputs of acoustic data previously collected from device 10 or normal acoustic data, and sound component data from device 10. Furthermore, instead of having a separate machine learning model for each individual device 10, one may be prepared for each type of device 10. Also, the machine learning model may be common to all devices 10.

[0122] For example, by using a machine learning model trained in this manner, the sound anomaly detection unit 1030 can, upon receiving sound components originating from the device 10, read the corresponding machine learning model from the device database 60, input the sound components, and use the normal or abnormal judgment result output by the machine learning model as the anomaly score.

[0123] If a specific device 10 is not registered in the device database 60, the sound abnormality detection unit 1030 may determine the degree of abnormality of the target sound component data based on sound data emitted by a device 10 of the same type as the device type information when it is functioning normally, or sound data that has occurred in the past.

[0124] In S1210, the abnormal device identification unit 1040 identifies an abnormality in the device 10, which is the source of the sound component, based on the degree of abnormality of the sound component associated with the device 10. For example, the abnormal device identification unit 1040 may identify the device 10, which is the source of the sound component, as abnormal if the degree of abnormality of the sound component exceeds a predetermined threshold.

[0125] In S1220, if the equipment command unit 1050 identifies an abnormality in any of the equipment 10 that is the source of the sound component, it controls at least one of the equipment 10 that has been identified as abnormal or other related equipment 10 to address the abnormality in equipment 10. The equipment command unit 1050 transmits an instruction to the control system 20 via the communication unit 670 to control the equipment 10 to address the abnormality.

[0126] For example, the equipment command unit 1050 may instruct the control system 20 to stop the operation of the equipment 10 in which an abnormality has been identified. The equipment command unit 1050 may also instruct the control system 20 to stop other equipment 10 related to the equipment 10 in which an abnormality has been identified (for example, equipment 10 in an upstream or downstream process of the equipment 10 in which an abnormality has been identified). The equipment command unit 1050 may also instruct the control system 20 to stop the operation of both the equipment 10 in which an abnormality has been identified and the other equipment 10 related to the equipment 10 in which an abnormality has been identified. The monitoring device 40 proceeds to process S1100 and continues monitoring of the facility 1 by the robot 30.

[0127] In addition, in S1100, the robot command unit 1060 may instruct the robot 30 to change the image imaging direction, change the image imaging magnification, or change the position of the robot 30, depending on whether it was not possible to identify the source device 10 among the devices 10 for any sound component.

[0128] Furthermore, the processes S1110-S1120 for detecting the 3D range and S1130-S1220 for processing acoustic data may be performed at different times. For example, the monitoring device 40 may perform S1110-S1120 in all or part of the area within facility 1, and pre-store a 3D map of all or part of the area within facility 1 in the source identification unit 1020, associating it with the imaging position and direction of the image data.

[0129] Figure 13 is an explanatory diagram illustrating how the source identification unit 1020 identifies the sound source location on the 3D map 1300. Figure 13 shows the 3D map 1300 along with dashed arrows indicating the x, y, and z axes, and a robot 30 that has captured image data at the origin position.

[0130] In S1140, the sound source separation unit 1000 separates the target sound component originating from sound source position a, the target sound component originating from sound source position b, and the target sound component originating from sound source position c. The sound source separation unit 1000 further separates the reference sound component originating from reference sound source A, the reference sound component originating from reference sound source B, and the reference sound component originating from reference sound source C. The sound components and reference sound components may be included in the same acoustic data, or they may be included in acoustic data detected at the same position at different timings. The range detection unit 1010 segments the image data detected at the same position as the target sound component to detect 3D range a, 3D range b, and 3D range c, respectively. The range detection unit 1010 further detects the positions of the reference marks of reference sound source A, reference sound source B, and reference sound source C included in the image data, respectively.

[0131] The source identification unit 1020 maps the sound source location to a 3D range (3D map 1300) where the coordinate origin is common but the three axes are different, by coordinate transformation. The source identification unit 1020 can calculate a 3x3 coordinate transformation matrix by multiplying a 3x3 matrix consisting of the coordinates of the reference marks of reference sound source A (xA', yA', zA), reference sound source B (xB', yB', zB'), and reference sound source C (xC', yC', zC') detected by the range detection unit 1010 by the inverse matrix of that 3x3 matrix consisting of the coordinates of the reference marks of reference sound source A (xA', yA', zA'), reference sound source B (xB', yB', zB'), and reference sound source C (xC', yC', zC'). The source identification unit 1020 can transform the coordinates of sound source position ac to the same coordinate axes as the 3D map 1300 by multiplying the coordinate transformation matrix by a 3x3 matrix consisting of the coordinates of sound source position a (xa, ya, za), sound source position b (xb, yb, zb), and sound source position c (xc, yc, zc) in the three axes detected by the sound source separation unit 1000.

[0132] The source identification unit 1020 may identify the device 10 corresponding to the 3D range a as the source of the sound component corresponding to sound source position a, since the coordinates (xa', ya', za') of the sound source position a after coordinate transformation overlap with the 3D range a. The source identification unit 1020 may identify the device 10 corresponding to the 3D range b as the source of the sound component corresponding to sound source position b, since the coordinates (xb', yb', zb') of the sound source position b after coordinate transformation overlap with the 3D range b. The source identification unit 1020 may identify the device 10 corresponding to the 3D range c as the source of the sound component corresponding to sound source position c, since the coordinates (xc', yc', zc') of the sound source position c after coordinate transformation overlap with the 3D range c.

[0133] In this embodiment, the monitoring device 40 can acquire information by linking acoustic data with equipment 10 by having the robot 30, which is a sound collection device, walk through the facility 1, and perform abnormality diagnosis of equipment 10. Furthermore, by detecting a three-dimensional range, the monitoring device 40 can quickly and accurately identify the type of equipment 10.

[0134] In this embodiment, an example of performing a coordinate transformation between coordinates with a common origin was shown, but the invention is not limited to this. If the image data and acoustic data are observed at different locations, the source identification unit 1020 may perform a coordinate transformation that translates the coordinates so that the origins of the coordinate axes of the image data and the coordinate axes of the acoustic data coincide.

[0135] Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where a block may represent (1) a stage in a process in which an operation is performed or (2) a section of a device having the role of performing the operation. Specific stages and sections may be implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable medium, and / or processors supplied with computer-readable instructions stored on a computer-readable medium. Dedicated circuits may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits may include reconfigurable hardware circuits, including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logic operations, flip-flops, registers, memory elements such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0136] Computer-readable media may include any tangible device capable of storing instructions to be executed by a suitable device, and as a result, computer-readable media having instructions stored therein will comprise a product containing instructions that can be executed to create means for performing operations specified in a flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital multipurpose disc (DVD), Blu-ray® disc, memory stick, integrated circuit card, etc.

[0137] Computer-readable instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, Java®, C++, and traditional procedural programming languages ​​such as the C programming language or similar programming languages.

[0138] Computer-readable instructions may be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the internet to the processor or programmable circuit of a programmable data processing device such as a general-purpose computer, a special-purpose computer, or another computer, and the computer-readable instructions may be executed to create means for performing operations specified in a flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0139] Figure 14 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied in whole or in part. A program installed on the computer 2200 can cause the computer 2200 to function as an operation or one or more sections of an apparatus according to an embodiment of the present invention, or to execute such operation or one or more sections, and / or to cause the computer 2200 to execute a process or a stage of such process according to an embodiment of the present invention. Such a program may be executed by the CPU 2212 to cause the computer 2200 to perform a particular operation associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0140] The computer 2200 according to this embodiment includes a CPU 2212, RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0141] The CPU 2212 operates according to programs stored in the ROM 2230 and RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 from a frame buffer provided in RAM 2214 or from itself, and displays the image data on the display device 2218.

[0142] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides them to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

[0143] The ROM 2230 stores boot programs and / or programs that depend on the computer 2200's hardware, which are executed by the computer 2200 when activated. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via parallel ports, serial ports, keyboard ports, mouse ports, etc.

[0144] The program is provided on a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium and installed on a hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable medium, and executed by the CPU 2212. The information processing described within these programs is read by the computer 2200, resulting in coordination between the program and the various types of hardware resources described above. The apparatus or method may be configured to realize the manipulation or processing of information in accordance with the use of the computer 2200.

[0145] For example, when communication is performed between a computer 2200 and an external device, the CPU 2212 may execute a communication program loaded into RAM 2214 and, based on the processing described in the communication program, instruct the communication interface 2222 to perform communication processing. Under the control of the CPU 2212, the communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in a recording medium such as RAM 2214, a hard disk drive 2224, a DVD-ROM 2201, or an IC card, transmits the read transmission data to the network, or writes received data received from the network to a reception buffer processing area provided on the recording medium.

[0146] Furthermore, the CPU 2212 may read all or necessary parts of files or databases stored on external storage media such as the hard disk drive 2224, DVD-ROM drive 2226 (DVD-ROM 2201), or IC card into the RAM 2214, and perform various types of processing on the data in the RAM 2214. The CPU 2212 then writes the processed data back to the external storage media.

[0147] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and subjected to information processing. The CPU 2212 may perform various types of processing on the data read from RAM 2214, including various types of operations, information processing, conditional judgments, conditional branching, unconditional branching, information retrieval / replacement, etc., as described throughout this disclosure and specified by the program instruction sequence, and write the results back to RAM 2214. The CPU 2212 may also retrieve information in files, databases, etc., within the recording medium. For example, if multiple entries are stored in the recording medium, each having an attribute value of a first attribute associated with an attribute value of a second attribute, the CPU 2212 may search among the multiple entries for an entry that matches the condition for which the attribute value of the first attribute is specified, read the attribute value of the second attribute stored in that entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0148] The programs or software modules described above may be stored on or near computer 2200 on a computer-readable medium. Alternatively, recording media such as hard disks or RAM provided within a server system connected to a dedicated communication network or the Internet can be used as computer-readable media, thereby providing programs to computer 2200 via the network.

[0149] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above embodiments. It will be clear from the claims that such modified or improved forms may also be included in the technical scope of the present invention.

[0150] It should be noted that the execution order of operations, procedures, steps, and stages in the apparatus, systems, programs, and methods shown in the claims, specifications, and drawings is not explicitly stated as "before," "prior to," etc., and that these can be implemented in any order unless the output of a previous process is used in a later process. Even if the operation flow in the claims, specifications, and drawings is described using phrases such as "first," "next," etc. for convenience, it does not mean that it is essential to perform the operations in that order. [Explanation of symbols]

[0151] 1 facility 10 equipment 20 Control Systems 30 robots 40 Monitoring equipment 50. Database of collected information 60 Device Database 70 Robot Database 210 Status acquisition unit 220 State determination unit 230 Display Processing Unit 240 Display device 250 Input Devices 260 Instruction Input Section 270 Control Unit 400a~c sensor 410a~b Sensor 420 Actuators 430 Status acquisition unit 440 Communications Department 450 Control Unit 600 Communications Department 602 Acoustic Data Acquisition Unit 604 Image Data Acquisition Unit 606 LiDAR data acquisition unit 608 Location data acquisition unit 610 Directional data acquisition unit 612 Instruction transmission unit 620 Information Collection Database Connection Unit 630 Device Database Connection Unit 640 Robot Database Connection Unit 650 Monitoring and Processing Unit 660 Input Processing Unit 665 Display Processing Unit 670 Communications Department 680 Input Device 690 Display device 1000 Sound source separation section 1010 Range detection unit 1020 Source Identification Department 1030 Sound Anomaly Detection Unit 1040 Abnormal device identification section 1050 Equipment Control Department 1060 Robot Command Center 2200 Computers 2201 DVD-ROM 2210 Host Controller 2212 CPU 2214 RAM 2216 Graphics Controller 2218 Display Devices 2220 Input / Output Controller 2222 Communication Interface 2224 Hard Disk Drive 2226 DVD-ROM drive 2230 ROM 2240 Input / Output Chip 2242 keyboard

Claims

1. An acoustic data acquisition unit that acquires acoustic data detected by a sound collection device that can move around the facility to collect sound, An image data acquisition unit that acquires image data captured within the aforementioned facility, A range detection unit detects the three-dimensional range of at least one device captured in the image data from the acquired image data, A source identification unit that uses the three-dimensional range detected by the range detection unit to identify which of the at least one device captured in the image data is the source of at least one sound component included in the acoustic data, The system includes a recording unit that records sound component data relating to the aforementioned sound components in association with the equipment identified by the source identification unit. Device.

2. The range detection unit acquires equipment type information corresponding to the three-dimensional range. The apparatus according to claim 1.

3. The source identification unit uses a three-dimensional map showing the three-dimensional range of each device detected by the range detection unit, and the device type information corresponding to each of the three-dimensional ranges, to identify which of the at least one devices captured in the image data is the source of at least one sound component contained in the acoustic data. The apparatus according to claim 2.

4. The acoustic data acquisition unit acquires the acoustic data, which includes reference sound components from at least three reference sound sources located within the facility. The image data acquisition unit acquires the image data in which the reference mark of the reference sound source is captured, The source identification unit identifies the device that is the source of at least one sound component included in the acoustic data by determining the positional relationship between the three-dimensional range and the sound source location of at least one sound component included in the acoustic data, based on the reference sound component and the reference mark. The apparatus according to claim 1.

5. The source identification unit uses operation data indicating the operation of at least one device within the facility to identify which of the at least one device captured in the image data is the source of at least one sound component included in the acoustic data. The apparatus according to claim 1.

6. The source identification unit uses non-asymmetric sound component data corresponding to the environment in which the facility is located to identify which of the at least one devices captured in the image data is the source of at least one sound component included in the acoustic data. The apparatus according to claim 1.

7. A sound anomaly detection unit that uses the sound component data to detect the degree of abnormality of the sound component, The system includes an abnormal equipment identification unit that identifies an abnormality in the equipment that is the source of the sound component based on the degree of abnormality of the sound component. The apparatus according to claim 1.

8. The source identification unit uses the three-dimensional range detected by the range detection unit to identify a component of the equipment captured in the image data as the source of at least one sound component included in the acoustic data. The apparatus according to claim 1.

9. Acoustic data is acquired by a sound collection device that can move around the facility to collect sound, To acquire image data captured within the aforementioned facility, From the acquired image data, the three-dimensional range of at least one device captured in the image data is detected. Using the three-dimensional range, identify which of the at least one devices captured in the image data is the source of at least one sound component contained in the acoustic data, The system includes recording sound component data relating to the aforementioned sound components in association with the identified device. method.

10. It is executed by a computer, and the computer, An acoustic data acquisition unit that acquires acoustic data detected by a sound collection device that can move around the facility to collect sound, An image data acquisition unit that acquires image data captured within the aforementioned facility, A range detection unit detects the three-dimensional range of at least one device captured in the image data from the acquired image data, A source identification unit that uses the three-dimensional range detected by the range detection unit to identify which of the at least one device captured in the image data is the source of at least one sound component included in the acoustic data, The recording unit is configured to record the sound component data relating to the aforementioned sound components in association with the equipment identified by the source identification unit. program.

Images