Remote monitoring system, abnormality detection system, remote monitoring method and program

Abstract

This remote monitoring system (101) detects an abnormality in the state of an object (102) to be monitored that autonomously operates, and comprises: a state acquisition unit (201) which acquires, from the object (102) to be monitored, state information that indicates a state of the object (102) to be monitored; an information acquisition unit (202a) which acquires first sensing information, that indicates a result of sensing the object (102) to be monitored, from an external information source (103a) that is installed outside the object (102) to be monitored and senses the object (102) to be monitored; a state estimation unit (203a) which estimates a first state of the object (102) to be monitored on the basis of the first sensing information; a state comparison unit (207) which compares the state information and the estimated state information based on the first state; and an alert notification unit (206) which notifies a monitoring person of the remote monitoring system (101) of the occurrence of the abnormality on the basis of the comparison result of the state comparison unit (207).

Description

[Technical Field]

[0001] The present disclosure relates to a remote monitoring system, an anomaly detection system, a remote monitoring method, and a program. [Background technology]

[0002] In recent years, advances in AI (Artificial Intelligence) and communication technology have enabled robots to perform increasingly sophisticated activities autonomously. As a result, it is expected that robots will be able to make up for the labor shortage caused by the future decline in the working population.

[0003] To operate these robots efficiently, centralized management through remote control or monitoring is essential, but if a vulnerability exists in this system, an attacker could gain unauthorized control of the robot.It is predicted that an attacker who has infiltrated a robot will tamper with the status reported to the remote monitoring system, concealing any abnormalities so that the monitor does not notice them.

[0004] To detect this, an anomaly detection method is being considered for in-vehicle networks that detects tampering by comparing the driving conditions on the network with the driving conditions estimated from surrounding information obtained from in-vehicle cameras, etc. (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 216306 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the technology disclosed in Patent Document 1, if an attacker tampers with peripheral information as well, it is not possible to detect that the attacker has tampered with the status of a monitored object such as a robot.

[0007] Therefore, the present disclosure provides a remote monitoring system, an anomaly detection system, a remote monitoring method, and a program that can detect that an attacker has tampered with the state of a monitored object even if the attacker has tampered with surrounding information. [Means for solving the problem]

[0008] A remote monitoring system according to one embodiment of the present disclosure is a remote monitoring system that detects an abnormality in the state of an autonomously operating monitored object, and includes: a first acquisition unit that acquires state information indicating the state of the monitored object from the monitored object; a second acquisition unit that acquires first sensing information indicating a sensing result of the monitored object from a first sensing device that is provided outside the monitored object and senses the monitored object; a state estimation unit that estimates a first state, which is the state of the monitored object, based on the first sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the first state of the monitored object estimated by the state estimation unit; and a notification unit that notifies a monitor of the remote monitoring system that an abnormality has occurred based on the comparison result of the state comparison unit.

[0009] An anomaly detection system according to one aspect of the present disclosure is an anomaly detection system that detects an anomaly in the state of an autonomously operating monitored object, and includes: a first acquisition unit that acquires state information indicating the state of the monitored object from the monitored object; a second acquisition unit that acquires sensing information indicating a sensing result of the monitored object from a sensing device that is provided outside the monitored object and senses the monitored object; a state estimation unit that estimates the state of the monitored object based on the sensing information acquired by the second acquisition unit; and a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the state of the monitored object estimated by the state estimation unit.

[0010] A remote monitoring method according to one aspect of the present disclosure is a remote monitoring method for detecting an abnormality in the state of an autonomously operating monitored object, which includes acquiring status information indicating the state of the monitored object from the monitored object, acquiring sensing information indicating the sensing results of the monitored object from a sensing device that senses the monitored object and is installed outside the monitored object, estimating the state of the monitored object based on the acquired sensing information, comparing the acquired status information with estimated status information based on the estimated state of the monitored object, and notifying a supervisor of the remote monitoring system that an abnormality has occurred based on the comparison result between the status information and the estimated status information.

[0011] A program according to one aspect of the present disclosure is a program for causing a computer to execute the above remote monitoring method. [Effects of the Invention]

[0012] According to a remote monitoring system according to one aspect of the present disclosure, even if an attacker tampers with peripheral information, it is possible to detect that the attacker has tampered with the state of the monitored object. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram showing the configuration of a monitoring system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing a functional configuration of the anomaly detection system according to the first embodiment. [Figure 3] FIG. 3 is a block diagram showing a functional configuration of the state estimation unit learning system according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing the learning process in the state estimation unit learning system according to the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of an environment information table managed by the environment management unit of the state estimation unit learning system according to the first embodiment. [Figure 6A] FIG. 6A is a flowchart showing the processing of the remote monitoring system according to the first embodiment. [Figure 6B] FIG. 6B is a flowchart showing the processing of the state selection unit according to the first embodiment. [Figure 7] FIG. 7 is a block diagram showing a functional configuration of the anomaly detection system according to the second embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of an environment information table managed by the environment management unit of the anomaly detection system according to the second embodiment. [Figure 9] FIG. 9 is a block diagram showing a functional configuration of the anomaly detection system according to the third embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of an environment information table managed by the environment management unit of the anomaly detection system according to the third embodiment. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of an anomaly detection system according to the fourth embodiment. [Figure 12] FIG. 12 is a diagram illustrating an example of an environment information table managed by the environment management unit of the anomaly detection system according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] (Findings that formed the basis of this disclosure) Conventional autonomous robots have mainly been limited to limited operations in confined spaces, such as cleaning robots. However, in recent years, robots that can perform more advanced operations (e.g., more advanced tasks) in wider spaces, such as security robots or delivery robots, have appeared.

[0015] As the range of robot activities expands, it becomes necessary to manage a large number of robots scattered across various locations, making the ability to remotely and centrally control and monitor robots increasingly important.

[0016] On the other hand, if robots are equipped with a function to connect to a communication network for remote control and monitoring, it becomes possible for attackers to access the robot remotely. If the robot is controlled illegally, there is a concern that damage may occur due to the leakage of information within the facility or the unauthorized operation of equipment within the facility.

[0017] To prevent such damage, a remote monitoring system is being considered that communicates with the robot and monitors whether there are any abnormalities in the robot's status. However, if an attacker tampers with the surrounding information in addition to tampering with the robot's status notified to the remote monitoring system, the remote monitoring system will not be able to detect that the robot's status has been tampered with. The robot's status may be, for example, but is not limited to, the robot's location.

[0018] Therefore, the inventors of the present application have conducted extensive research into remote monitoring systems and the like that can detect tampering by an attacker with the state of a monitored object, such as a robot, even if the attacker has tampered with surrounding information, and have devised the remote monitoring system and the like described below. For example, in this disclosure, information obtained by sensing the monitored object is acquired from an external information source (external information source) independent of the monitored object, and the state of the monitored object estimated from that information is compared with the state acquired from the monitored object, such as a robot, to detect tampering with the state of the monitored object by an attacker and notify an alert to the monitor. As a result, the monitor who receives the alert can take the necessary action early, thereby minimizing damage.

[0019] A remote monitoring system according to one embodiment of the present disclosure is a remote monitoring system that detects an abnormality in the state of an autonomously operating monitored object, and includes: a first acquisition unit that acquires state information indicating the state of the monitored object from the monitored object; a second acquisition unit that acquires first sensing information indicating a sensing result of the monitored object from a first sensing device that is provided outside the monitored object and senses the monitored object; a state estimation unit that estimates a first state, which is the state of the monitored object, based on the first sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the first state of the monitored object estimated by the state estimation unit; and a notification unit that notifies a monitor of the remote monitoring system that an abnormality has occurred based on the comparison result of the state comparison unit.

[0020] This allows the remote monitoring system to detect an abnormality in the status information from the monitored object (tampering with the status information) using the first sensing information from the first sensing device installed outside the monitored object. In other words, the remote monitoring system can detect an abnormality without using peripheral information obtained from an in-vehicle camera or the like installed in the monitored object. Furthermore, because the first sensing device is installed outside the monitored object, the first sensing information is information that has not been tampered with by an attacker. Therefore, according to the remote monitoring system, even if the peripheral information is tampered with by an attacker, it is possible to detect that the attacker has tampered with the status of the monitored object.

[0021] Also, for example, the second acquisition unit may further acquire second sensing information different from the first sensing information, which indicates a sensing result of the monitored object, from a second sensing device that senses the monitored object and is provided outside the monitored object, and the state estimation unit may estimate a second state, which is the state of the monitored object, based on the second sensing information acquired by the second acquisition unit, and the estimated state information may be information further based on the second state.

[0022] This allows the remote monitoring system to detect an abnormality in the monitored object using two pieces of sensing information, thereby enabling more accurate detection than when one piece of sensing information is used.

[0023] Furthermore, for example, the monitoring device may further include an environment management unit that manages environmental information that associates the environment when the monitored object is sensed with the estimation accuracy of the state estimated by the state estimation unit, and a state determination unit that uses the environmental information managed by the environment management unit to determine one state from the first state and the second state, and outputs the determined one state as the estimated state information.

[0024] This allows the remote monitoring system to detect an abnormality in the monitored object using the first state and the second state, thereby enabling more accurate detection than when using a single estimated state.

[0025] Furthermore, for example, the state determination unit may select one of the first state and the second state using the environmental information, and output the selected one as the estimated state information.

[0026] This allows the remote monitoring system to select the state with the higher estimation accuracy, thereby enabling more accurate detection.

[0027] Furthermore, for example, the state determination unit may use the environmental information to perform a weighting calculation on the first state and the second state, and output the weighted states as the estimated state information.

[0028] As a result, the remote monitoring system uses the weighted calculated state, and therefore can take both the first state and the second state into consideration, thereby enabling more accurate detection.

[0029] Also, for example, the first acquisition unit may acquire the current location of the monitored object as the state, the state estimation unit may estimate the current location of the monitored object as the first state and the second state, and the environmental information may include estimation accuracy of the first state and the second state for each time.

[0030] This allows the remote monitoring system to detect abnormalities in the location of the monitored object (location tampering) even if the location information has been tampered with by an attacker.

[0031] Furthermore, for example, the monitoring target may be a robot, the first sensing device may include a fixed camera, and the second sensing device may include a fixed microphone.

[0032] This allows the remote monitoring system to effectively detect abnormalities in the robot's position by using fixed cameras and fixed microphones as sensing devices.

[0033] Furthermore, for example, the monitored object may be a robot that performs security work, the first sensing device may include a fixed camera, and the second sensing device may include an illuminance sensor.

[0034] As a result, the remote monitoring system can effectively detect abnormalities in the position of a robot performing security duties by using a fixed camera and an illuminance sensor as sensing devices.

[0035] Furthermore, for example, the monitored object may be a robot that performs cleaning work, the first sensing device may include a fixed camera, and the second sensing device may include a dust sensor.

[0036] As a result, the remote monitoring system can effectively detect abnormalities in the position of the robot performing cleaning work by using the fixed camera and dust sensor as sensing devices.

[0037] Furthermore, for example, the monitoring device may further include a state learning unit that calculates information indicating an estimation accuracy of at least one of the first state and the second state based on the state of the monitored object acquired by the first acquisition unit and at least one of the first state and the second state of the monitored object estimated by the state estimation unit, and outputs the calculated information indicating the estimation accuracy to the state estimation unit and the environment management unit.

[0038] This allows the state learning unit to update the machine learning model and environmental information used by the state estimation unit.

[0039] Furthermore, for example, the environment management unit may determine the estimation accuracy of the state estimated by the state estimation unit based on information indicating the estimation accuracy acquired from the state learning unit.

[0040] This allows the environmental information managed by the environment management unit to be information that corresponds to updates made to the state estimation unit. Therefore, in the remote monitoring system, even if the state estimation unit is updated, the state selection unit can select a state that takes the update into consideration. This contributes to accurately detecting if an attacker has tampered with the state of the monitored object.

[0041] Furthermore, for example, the state learning unit may output information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit before the start of operation of the monitored object.

[0042] As a result, the status learning unit performs the learning process before the monitoring target is put into operation, and the remote monitoring system can detect if an attacker has tampered with the status of the monitoring target during operation.

[0043] Furthermore, for example, the state learning unit may output information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit during operation of the monitored object.

[0044] This allows the state learning unit to perform learning processing even while the monitored object is in operation, thereby improving the estimation accuracy of the state estimation unit.

[0045] Furthermore, for example, the monitoring device may further include an environment management unit that manages environmental information that corresponds the environment when the monitored object is sensed with the estimation accuracy of the state estimated by the state estimation unit, and the state comparison unit may compare the state information with the estimated state information based on the environmental information.

[0046] This allows the remote monitoring system to use the environmental information for comparison in the status comparison unit. In other words, the remote monitoring system can make comparisons according to the environment indicated by the environmental information. This allows the remote monitoring system to more accurately determine abnormalities.

[0047] Furthermore, an anomaly detection system according to one aspect of the present disclosure is an anomaly detection system that detects an anomaly in the state of an autonomously operating monitored object, and includes a first acquisition unit that acquires state information indicating the state of the monitored object from the monitored object, a second acquisition unit that acquires sensing information indicating a sensing result of the monitored object from a sensing device that is provided outside the monitored object and senses the monitored object, a state estimation unit that estimates the state of the monitored object based on the sensing information acquired by the second acquisition unit, and a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the state of the monitored object estimated by the state estimation unit.

[0048] This allows the anomaly detection system to detect an anomaly in the status information from the monitored object (tampering with the status information) using sensing information from a sensing device installed outside the monitored object. In other words, the anomaly detection system can detect anomalies without using peripheral information obtained from an in-vehicle camera or the like installed in the monitored object. Furthermore, because the sensing device is installed outside the monitored object, the sensing information is information that has not been tampered with by an attacker. Therefore, the anomaly detection system can detect that an attacker has tampered with the status of the monitored object, even if the peripheral information has been tampered with by an attacker.

[0049] A remote monitoring method according to one aspect of the present disclosure is a remote monitoring method for detecting an abnormality in the state of an autonomously operating monitoring target, comprising the steps of: acquiring status information indicating the state of the monitoring target from the monitoring target; acquiring sensing information indicating a sensing result of the monitoring target from a sensing device that senses the monitoring target and is provided outside the monitoring target; estimating the state of the monitoring target based on the acquired sensing information; comparing the acquired status information with estimated status information based on the estimated state of the monitoring target; and notifying a monitor of the remote monitoring system that an abnormality has occurred based on a comparison result between the status information and the estimated status information. A program according to one aspect of the present disclosure is a program for causing a computer to execute the above remote monitoring method.

[0050] This provides the same effects as the above-mentioned remote monitoring system.

[0051] These general or specific aspects may be realized as a system, a method, an integrated circuit, a computer program, or a non-transitory recording medium such as a computer-readable CD-ROM, or as any combination of the system, method, integrated circuit, computer program, or recording medium. The program may be pre-stored in the recording medium, or may be supplied to the recording medium via a wide area communication network including the Internet.

[0052] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0053] The embodiments described below each illustrate a specific example of the present disclosure. The numerical values, shapes, components, steps, and step orders shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components that are not described in the independent claims that represent the highest concepts are described as optional components. Furthermore, in all embodiments, the respective contents can be combined.

[0054] Furthermore, each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, for example, the scales and the like do not necessarily match in each figure. Furthermore, in each figure, substantially the same configurations are assigned the same reference numerals, and duplicate explanations are omitted or simplified.

[0055] Furthermore, in this specification, terms indicating relationships between elements such as the same, as well as numerical values ​​and numerical ranges, are not expressions that only express a strict meaning, but also expressions that include a substantially equivalent range, for example, a difference of about a few percent (e.g., about 10%).

[0056] Furthermore, in this specification, ordinal numbers such as "first" and "second" do not refer to the number or order of components unless otherwise specified, but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0057] (Embodiment 1) [1. Overall system configuration] FIG. 1 is a diagram showing the configuration of a monitoring system 100 according to this embodiment.

[0058] As shown in FIG. 1, the monitoring system 100 includes a remote monitoring system 101, a monitored object 102, external information sources 103a and 103b, a network 104, and a state estimation unit learning system 300.

[0059] The remote monitoring system 101 is an information processing system for detecting abnormalities in the state of a remotely located autonomously operating monitoring target 102. The remote monitoring system 101 is connected to the monitoring target 102 and external information sources 103a and 103b via a network 104. The remote monitoring system 101 includes an anomaly detection system 200 that detects abnormalities in the state of the monitoring target 102 notified by the monitoring target 102, using information obtained by sensing the monitoring target 102 from each of the external information sources 103a and 103b. Hereinafter, the information obtained from each of the external information sources 103a and 103b will also be referred to as sensing information or estimation information.

[0060] The monitored object 102 is an autonomously operating robot, such as, but not limited to, a security robot, a delivery robot, a cleaning robot, etc. The monitored object 102 is equipped with sensors such as a camera, a microphone, an infrared sensor, an ultrasonic sensor, etc. that detect the state (e.g., location information) of the monitored object 102, and transmits the state of the monitored object 102 (state information indicating the state of the monitored object 102), such as video information captured by the camera, audio information picked up by the microphone, and location information estimated using the sensor, to the remote monitoring system 101 via the network 104.

[0061] The robot may be, for example, a wheeled, crawler, or legged (including walking) robot. The monitored object 102 may be any mobile object that operates autonomously, and is not limited to a robot. The monitored object 102 may be, for example, a mobile object that moves autonomously (for example, a vehicle that runs autonomously), an air vehicle that flies autonomously (for example, a drone), or the like. The monitored object 102 may also be a device in which some of the components of the monitored object 102 (for example, an arm) move (the device does not move, and only some of the components move). Autonomous movement, autonomous running, and autonomous flying are examples of autonomous behavior.

[0062] The state of the monitored object 102 is, for example, the current position (current position coordinates) of the monitored object 102, but may also be, for example, whether the monitored object 102 is operating or not, or whether the function of the monitored object 102 itself is operating or not (for example, whether it is currently cleaning or not), etc. The operation of the monitored object 102 includes the monitored object 102 moving (the current position is changing), a part of each component of the monitored object 102 (for example, an arm) moving (the current position is not changing, and only a part of each component is moving), etc.

[0063] The external information sources 103a and 103b are devices for detecting the state of the monitoring target 102. The external information sources 103a and 103b are installed at positions where they can detect the state of the monitoring target 102, and are cameras, microphones, various sensors such as illuminance sensors, clocks, etc. The external information sources 103a and 103b transmit information such as video information captured by cameras, audio information collected by microphones, sensor values ​​acquired by various sensors, or time information to the remote monitoring system 101 via the network 104. The external information sources 103a and 103b transmit the information to the remote monitoring system 101 without going through the monitoring target 102.

[0064] The external information sources 103a and 103b are provided outside the monitored object 102 and sense the monitored object 102. The external information sources 103a and 103b are, for example, devices that cannot communicate with the monitored object 102. The external information sources 103a and 103b may be, for example, fixed or mobile. The external information sources 103a and 103b are examples of sensing devices that sense the monitored object 102, and video information, audio information, sensor values, time information, etc. are examples of sensing information obtained by sensing. The sensor values ​​are, for example, but not limited to, measurements from a dust sensor.

[0065] The external information sources 103a and 103b may be devices that are installed in advance in the space where the monitored object 102 is used, or may be devices dedicated to the remote monitoring system 101. In this embodiment, the monitoring system 100 is only required to include at least two external information sources.

[0066] The network 104 is a communications network, and may be a closed network or the Internet, depending on the manner in which the monitored object 102 and the external information sources 103a and 103b operate.

[0067] The anomaly detection system 200 uses information acquired from external information sources 103a and 103b to perform anomaly detection of the state of the monitored object 102 notified by the monitored object 102.

[0068] The state estimation unit learning system 300 performs learning to improve the accuracy of state estimation by the state estimation unit 303, using the state notified from the monitored object 102 and information acquired from external information sources 103a and 103b.

[0069] The state estimation unit learning system 300 may exist within the remote monitoring system 101. In other words, the remote monitoring system 101 may have some or all of the functions of the state estimation unit learning system 300.

[0070] [1-1. Configuration of anomaly detection system] FIG. 2 is a block diagram showing the functional configuration of an anomaly detection system 200 according to this embodiment.

[0071] As shown in FIG. 2, the anomaly detection system 200 includes a state acquisition unit 201, information acquisition units 202a and 202b, state estimation units 203a and 203b, an environment management unit 204, a state selection unit 205, an alert notification unit 206, and a state comparison unit 207.

[0072] The status acquisition unit 201 receives status information indicating the status of the monitoring target 102 transmitted by the monitoring target 102. In this embodiment, the status acquisition unit 201 acquires the current location of the monitoring target 102 as the status of the monitoring target 102. The status acquisition unit 201 is configured to include, for example, a communication module (communication circuit). The status acquisition unit 201 is an example of a first acquisition unit.

[0073] The information acquiring unit 202a receives information transmitted by the external information source 103a, and the information acquiring unit 202b receives information transmitted by the external information source 103b. Specifically, the information acquiring unit 202a acquires, from the external information source 103a, sensing information (e.g., first sensing information) indicating the sensing result of the external information source 103a sensing the monitoring target 102, and the information acquiring unit 202b acquires, from the external information source 103b, sensing information (e.g., second sensing information) indicating the sensing result of the external information source 103b sensing the monitoring target 102. The first sensing information and the second sensing information are different types of sensing information.

[0074] The information acquisition units 202a and 202b are configured to include, for example, a communication module (communication circuit). The information acquisition units 202a and 202b are an example of a second acquisition unit.

[0075] The state estimation unit 203a estimates the state (an example of a first state) of the monitoring target 102 based on the information acquired by the information acquisition unit 202a, and the state estimation unit 203b estimates the state (an example of a second state) of the monitoring target 102 based on the information acquired by the information acquisition unit 202b. In this embodiment, the state estimation unit 203a estimates the current location of the monitoring target 102 as the first state and the second state. Since the sensing information used to estimate the first state and the second state are different types of information, the first state and the second state can be different states (for example, different locations).

[0076] For example, the information acquired by the information acquiring unit 202a is any one of video information, audio information, sensor values, time information, etc., and the information acquired by the information acquiring unit 202b is information other than any one of video information, audio information, sensor values, time information, etc. In other words, the information acquired by the information acquiring unit 202a (first sensing information) and the information acquired by the information acquiring unit 202b (second sensing information) are different types of information.

[0077] The state estimation unit 203a estimates a first state using, for example, a machine learning model, and the state estimation unit 203b estimates a second state using, for example, another machine learning model. The machine learning model is trained in advance so as to output the state of the monitored object 102 when sensing information is input. The machine learning model used by the state estimation unit 203a and the machine learning model used by the state estimation unit 203b are machine learning models trained using different input information.

[0078] The environment management unit 204 manages the accuracy of the state estimated by the state estimation units 203a and 203b as environment information (see FIG. 5, described later) for each environment when the state estimation units 203a and 203b estimate the state. The environment information is, for example, information that associates the environment when the monitoring target 102 is sensed with the estimation accuracy of the state of the monitoring target 102 estimated by the state estimation units 203a and 203b. Here, the environment is, for example, the time, but may also be the weather, the brightness around the monitoring target 102, etc. For example, the environment information may include the estimation accuracy of the first state and the second state for each time.

[0079] The state selection unit 205 selects which estimated state to adopt from the estimated states of the monitoring target 102 estimated by the state estimation units 203a and 203b, using environmental information managed by the environment management unit 204. The state selection unit 205 selects one of the first state and the second state using the environmental information, and outputs the selected one as an estimated state (estimated state information).

[0080] The state selection unit 205 is an example of a state determination unit. The state determination unit is not limited to estimating the state of the monitoring target 102 by selection. The state determination unit may calculate one state from the first state and the second state using environmental information, and output the calculated one state as an estimated state. For example, the state determination unit may perform a weighting calculation on the first state and the second state using environmental information, and output the weighted state as an estimated state.

[0081] The state comparison unit 207 compares the state (state information) acquired by the state acquisition unit 201 with the estimated state (estimated state information) acquired from the state selection unit 205. The estimated state information is information based on at least the first state, and in this embodiment, it is information based on the first state and the second state. The state comparison unit 207 compares, for example, the state of the monitoring target 102 acquired by the state acquisition unit 201 with the estimated state of the monitoring target 102 selected by the state selection unit 205, and detects an abnormality if there is a difference of a predetermined amount or more, and notifies the alert notification unit 206 of the detection of the abnormality.

[0082] The alert notification unit 206 alerts (notifies) the monitor of the remote monitoring system 101 that an abnormality has occurred, based on the comparison result of the state comparison unit 207.

[0083] The estimation accuracy of the states estimated by the state estimation units 203a and 203b, which is managed by the environment management unit 204, may be specified in advance by a supervisor or the like, or may be set using the results of learning performed by the state estimation unit learning system 300. The environment management unit 204 may determine the estimation accuracy of the estimated states estimated by the state estimation units 203a and 203b based on, for example, information indicating the estimation accuracy acquired from the state learning unit 301 (see FIG. 3). The determination of the estimation accuracy may be performed using, for example, a table indicating the correspondence between the information indicating the estimation accuracy and the estimation accuracy.

[0084] The remote monitoring system 101 according to this embodiment detects tampering by an attacker by estimating the state of the robot, which is the monitored object 102, using information (sensing information) from external information sources 103a, 103b, etc. Furthermore, the remote monitoring system 101 further selects from a plurality of states an estimated state using information from the optimal external information sources 103a, 103b, etc., depending on the environment in which the monitored object 102 operates, thereby making it possible to maintain high estimation accuracy in a variety of environments.

[0085] [1-2. Configuration of the state estimation unit learning system] Next, a description will be given of the configuration of the state estimation unit learning system 300. Fig. 3 is a block diagram showing the functional configuration of the state estimation unit learning system 300 according to this embodiment. Fig. 3 shows a configuration for performing learning processing on one state estimation unit 303. Note that, for convenience, Fig. 3 illustrates external information source 103, but the state estimation unit learning system 300 acquires sensing information from each of external information sources 103a and 103b.

[0086] As shown in FIG. 3, the state estimation unit learning system 300 includes a state acquisition unit 201 a, an information acquisition unit 202 c, a state learning unit 301, an environment management unit 302, and a state estimation unit 303.

[0087] The status acquisition unit 201a has the same configuration as the status acquisition unit 201 of the anomaly detection system 200. The status acquisition unit 201a acquires status information from the monitored object 102. The information acquisition unit 202c has the same configuration as the information acquisition units 202a and 202b of the anomaly detection system 200. The information acquisition unit 202c acquires sensing information from the external information source 103.

[0088] The state estimation unit 303 estimates the state of the monitored object 102 from the information of the external information source 103 acquired from the information acquisition unit 202c, and outputs the time when the state of the monitored object 102 was estimated and the estimated state to the state learning unit 301.

[0089] The state learning unit 301 compares the state of the monitoring target 102 acquired by the state acquisition unit 201a with the estimated state estimated by the state estimation unit 303 to calculate the estimation accuracy, and outputs the environment and the calculated estimation accuracy to the environment management unit 302 and the state estimation unit 303. The estimation accuracy here is, for example, information based on the difference (estimation error) between the state of the monitoring target 102 acquired from the state acquisition unit 201a and the estimated state acquired from the state estimation unit 303.

[0090] The environment management unit 302 manages the combination of the environment (here, time) and estimation accuracy acquired from the state learning unit 301 as environment information.

[0091] The state estimation unit 303 trains a machine learning model based on the estimation accuracy acquired from the state learning unit 301 so as to improve the estimation accuracy.

[0092] As an example of learning by the state estimation unit 303, a regression model is used in which the information acquired from the information acquisition unit 202c is used as an explanatory variable and the state of the monitored object 102 is used as a dependent variable, and the objective function is a square sum error, which is the sum of two values ​​of the difference between the state acquired from the state acquisition unit 201a and the state of the monitored object 102 estimated by the state estimation unit 303, and learning is performed so as to minimize the square sum error. Note that the square sum error is an example of estimation accuracy.

[0093] The state estimation unit 303 estimates the state of the monitored object 102 from internal parameters such as the weight or bias of the regression model and information from the external information source 103 acquired from the information acquisition unit 202, and outputs the time at which the state of the monitored object 102 was estimated and the estimated state to the state learning unit 301.

[0094] The state learning unit 301 calculates the sum of squares error, which is an objective function, from the state of the monitored object 102 acquired from the state acquisition unit 201a and the estimated state acquired from the state estimation unit 303 to calculate the estimation accuracy, and outputs the environment (here, the time) and the calculated estimation accuracy to the environment management unit 302 and the state estimation unit 303.

[0095] The environment management unit 302 manages the combination of the environment and the estimation accuracy acquired from the state learning unit 301 as environment information.

[0096] The state estimation unit 303 uses a differential equation to calculate the slope of the change in estimation accuracy when the internal parameters are changed, based on a combination of the internal parameters from a previous estimation and the estimation accuracy obtained from the state learning unit 301, and a combination of the internal parameters from the current estimation and the estimation accuracy obtained from the state learning unit 301, and updates the internal parameters so that the slope becomes a negative value, i.e., so that the square sum error becomes small.

[0097] While updating the internal parameters of state estimation unit 303, state learning unit 301 repeats learning, adjusts the internal parameters so that the state of monitored object 102 can be estimated with high accuracy, and manages the estimation accuracy for each environment in environment management unit 302. Then, state estimation unit learning system 300 can hand over the learning results to anomaly detection system 200 by replacing the learned parameters of state estimation unit 303 and the learned environmental information managed by environment management unit 204 with the parameters (internal parameters) of state estimation units 203a and 203b of anomaly detection system 200 and the environmental information of environment management unit 204.

[0098] For example, the adjustment (e.g., update) of the internal parameters of the machine learning model and the update of the environmental information are performed together. The environmental information includes the estimation accuracy for the internal parameters after the adjustment. In this way, the environmental information may be generated based on the learning process of the machine learning model.

[0099] In addition, when the remote monitoring system 101 has some of the functions of the state estimation unit learning system 300, the state learning unit 301 may calculate information (e.g., a square sum error) indicating the estimation accuracy of at least one of the first state and the second state of the monitored object 102 based on the state of the monitored object 102 acquired by the state acquisition unit 201 and at least one of the first state and the second state of the monitored object 102 estimated by the state estimation unit 303, and output the calculated information indicating the estimation accuracy to the state estimation units 203a, 203b and the environment management unit 204.

[0100] Note that the state learning unit 301 may output information indicating the calculated estimation accuracy to the state estimation units 203a and 203b and the environment management unit 204 before the start of operation of the monitoring target 102. Furthermore, the state learning unit 301 may output information indicating the calculated estimation accuracy to the state estimation units 203a and 203b and the environment management unit 204 while the monitoring target 102 is in operation. During operation of the monitoring target 102, learning is performed using the state of the monitoring target 102 acquired by the state acquisition unit 201 when no abnormality is detected by the state comparison unit 207 and the estimated states estimated by the state estimation units 203a and 203b.

[0101] [1-3. Learning process in the state estimation unit learning system] Next, a description will be given of a process for learning the estimation accuracy of the state of the monitored object 102 in the state estimation unit learning system 300. Fig. 4 is a flowchart showing the learning process (remote monitoring method) in the state estimation unit learning system 300 according to this embodiment.

[0102] As shown in FIG. 4, first, learning (learning processing) is performed in the state estimation unit learning system 300 (S401). In step S401, the state estimation unit 303, as learning, estimates the state of the monitored object 102 using the state extracted by the information acquisition unit 202c and outputs it to the state learning unit 301. The state learning unit 301 compares the state estimated by the state estimation unit 303 with the state of the monitored object 102 extracted by the state acquisition unit 201, calculates the estimation accuracy, and outputs it to the state estimation unit 303. The state estimation unit 303 performs processing to adjust internal parameters so as to improve the estimation accuracy based on the estimation accuracy calculated by the state learning unit 301. In step S401, these processing steps are repeated a certain number of times.

[0103] The state estimation unit learning system 300 determines whether the estimation accuracy is sufficient (S402). The state estimation unit learning system 300 checks whether the estimation accuracy held by the environment management unit 302 exceeds a preset accuracy.

[0104] If it is determined that the estimation accuracy is sufficient (YES in S402), the state estimation unit learning system 300 replaces the internal parameters learned by the state estimation unit 303 and the learned environmental information managed by the environmental management unit 302 with the internal parameters of the state estimation units 203a and 203b of the anomaly detection system 200 and the environmental information of the environmental management unit 204, and operates the system (S403).

[0105] If it is determined that the estimation accuracy is insufficient (NO in S402), the state estimation unit learning system 300 determines that the estimation accuracy of the state estimation unit 303 cannot be expected to improve, and after reviewing the learning method (S404), executes step S401 again.

[0106] As an example of a learning method, the state estimation unit 303 uses the information acquired from the information acquisition unit 202c as an explanatory variable and the state of the monitored object 102 as a dependent variable as a regression model, and the state learning unit 301 uses the least squares method to train the regression model so as to minimize the difference between the estimated state of the monitored object 102 estimated by the state estimation unit 303 and the state acquired from the state acquisition unit 201a.

[0107] If the state learning unit 301 determines that the estimation accuracy after learning is insufficient and that improvement in estimation accuracy is not expected, it reviews the learning method by, for example, reviewing the mathematical model, such as changing the regression model from linear regression to a regression tree, and improves accuracy until the error is within an acceptable range.

[0108] As described above, the monitoring system 100 (or the remote monitoring system 101) has a learning mode for learning a machine learning model. The learning mode is executed, for example, in the space where the monitored object 102 is used.

[0109] [1-4. Contents managed by the Environmental Management Department] Next, a description will be given of the contents managed by the environment management unit 204 of the anomaly detection system 200. Fig. 5 is a diagram showing an example of an environment information table managed by the environment management unit 204 according to this embodiment.

[0110] 5, the environment information table includes an environment 501 and an estimation accuracy 502. The estimation accuracy 502 includes the estimation accuracy of the state estimation unit 203a and the estimation accuracy of the state estimation unit 203b.

[0111] An environment 501 represents the environment when the state estimation units 203a and 203b estimated the state. In Fig. 5, the environment 501 represents time, and the estimation accuracy 502 for each time is shown.

[0112] Estimation accuracy 502 represents the estimation accuracy in environment 501. Estimation accuracy 502 indicates the ratio between the estimation accuracy of state estimation unit 203a and the estimation accuracy of state estimation unit 203b in percentage. In FIG. 5, the estimation accuracy is indicated by a ratio to make it easier to understand the difference in accuracy between state estimation units 203a and 203b, but any unit may be used as long as it allows comparison of the accuracy between state estimation units 203a and 203b. Furthermore, estimation accuracy is not limited to being a numerical value, and may be expressed as a scale such as "high," "medium," or "low," for example.

[0113] 5, the estimation accuracy of the state of the monitoring target 102 estimated by the state estimation unit 203a based on the sensing information of the external information source 103a at 12:00 is 80%, and the estimation accuracy of the state of the monitoring target 102 estimated by the state estimation unit 203b based on the sensing information of the external information source 103b at 12:00 is 20%. From this, it can be seen that at 12:00, the state of the monitoring target 102 estimated by the state estimation unit 203a has a higher estimation accuracy than the state of the monitoring target 102 estimated by the state estimation unit 203b.

[0114] The environment information table may be prepared in advance, may be created by learning in the state estimation unit learning system 300, or may be created from the environment information of the environment management unit 302 in the state estimation unit learning system 300.

[0115] When the state estimation unit is created by learning in the state estimation unit learning system 300, the environment when the estimation is performed is recorded in the environment 501, and the estimation accuracy at that time is recorded in the estimation accuracy 502.

[0116] [1-5. Processing of the remote monitoring system and status selection unit] Next, a description will be given of the processing of the remote monitoring system 101 and the state selection unit 205. First, a description will be given of the processing of the remote monitoring system 101. Fig. 6A is a flowchart showing the processing (remote monitoring method) of the remote monitoring system 101 according to this embodiment.

[0117] 6A, first, the status acquisition unit 201 acquires status information indicating the status of the monitoring target 102 from the monitoring target 102 (S601). The status information includes time information indicating the time at which sensing was performed by a sensor or the like provided in the monitoring target 102.

[0118] Next, the information acquiring units 202a and 202b acquire estimation information from the external information sources 103a and 103b (S602). Specifically, the information acquiring unit 202a acquires the first sensing information from the external information source 103a as estimation information, and the information acquiring unit 202b acquires the second sensing information from the external information source 103b as estimation information.

[0119] The timing of acquiring the first sensing information and the second sensing information is not particularly limited, and may be acquired at predetermined time intervals, for example. The first sensing information and the second sensing information may be acquired synchronously. For example, the first sensing information and the second sensing information are information sensed at a time within a predetermined time difference (for example, several seconds to several minutes) from the time indicated by the time information included in the status information.

[0120] Next, the state estimation units 203a and 203b estimate the state of the monitoring target 102 in each of the two or more pieces of estimation information (S603). Specifically, the state estimation unit 203a acquires, as an estimation result, the state (first state) of the monitoring target 102, which is an output obtained by inputting the first sensing information into a machine learning model. Furthermore, the state estimation unit 203b acquires, as an estimation result, the state (second state) of the monitoring target 102, which is an output obtained by inputting the second sensing information into a machine learning model. Here, the first state and the second state are information indicating the current position of the monitoring target 102 (for example, current position coordinates).

[0121] The state estimation units 203 a and 203 b output the estimation results to the state selection unit 205 .

[0122] Next, the state selection unit 205 determines the state of one of the monitoring targets 102 from the two or more estimated states of the monitoring targets 102 (S604). In this embodiment, the state selection unit 205 selects the first state or the second state, whichever has the higher estimation accuracy, based on the environmental information, and determines the selected state as the estimated state of the monitoring target 102. The state selection unit 205 acquires the estimation accuracy of each of the state estimation units 203a and 203b at the time indicated by the time information included in the state information from the environmental information table, and selects the first state or the second state, which has the higher estimation accuracy, based on the acquired estimation accuracy.

[0123] The state selection unit 205 outputs the selected estimated state of the monitoring target 102 to the state comparison unit 207 .

[0124] Next, the state comparison unit 207 determines whether the difference between the state acquired from the monitoring target 102 by the state acquisition unit 201 and the estimated state is equal to or less than a threshold value (S605). The threshold value is set in advance.

[0125] Next, if the state comparison unit 207 determines that the difference is equal to or less than the threshold value (YES in S605), the state acquired from the monitoring target 102 is within the normal range, and therefore ends the process. A YES determination in step S605 means that the state of the monitoring target 102 has not been tampered with in the monitoring target 102. Furthermore, if the state comparison unit 207 determines that the difference is not equal to or less than the threshold value (NO in S605), it outputs to the alert notification unit 206 that an abnormality has been detected. A NO determination in step S605 means that the state of the monitoring target 102 has been tampered with in the monitoring target 102, or there is a high possibility that tampering has occurred.

[0126] Next, the alert notification unit 206 notifies the monitor of the remote monitoring system 101 of an alert indicating that an abnormality has been detected (S606). The alert may be issued, for example, by displaying an alert on a display, by emitting a sound from a sound output device, by emitting light from a light emitting device, or by any other method or a combination thereof.

[0127] Next, a description will be given of the processing of status selection section 205. Fig. 6B is a flowchart showing the processing (remote monitoring method) of status selection section 205 according to this embodiment. Fig. 6B is a flowchart showing in detail step S604 of Fig. 6A.

[0128] As shown in FIG. 6B, the state selection unit 205 acquires the states estimated by the state estimation units 203a and 203b at, for example, 12:00 (S611).

[0129] The state selection unit 205 acquires, from the environment management unit 204, environment information that links the environment with the estimation accuracy of the state estimation units 203a and 203b in that environment (S612).

[0130] The state selection unit 205 selects a final estimated state from the estimated state, the environment when the state was estimated, and the estimation accuracy in that environment (S613). In the environmental information of Fig. 5, the estimation accuracy of the state estimation unit 203a at 12:00 is 80%, and the estimation accuracy of the state estimation unit 203b is 20%, so the state selection unit 205 selects the state estimated by the state estimation unit 203a, which has a high estimation accuracy of 80%. This is an example of determining one state of the monitoring target 102.

[0131] In addition to selecting the state with the highest estimation accuracy, the state selection unit 205 may weight the estimated states based on their respective estimation accuracies and calculate a new estimated state according to the weighting. For example, the new estimated state may be calculated so that the weight increases as the estimation accuracy increases.

[0132] (Embodiment 2) The following describes an anomaly detection system for monitoring an autonomously moving robot. In this embodiment, the system detects whether or not the location information notified by the robot has been tampered with.

[0133] FIG. 7 is a block diagram showing the functional configuration of an anomaly detection system 700 according to this embodiment.

[0134] As shown in FIG. 7, the anomaly detection system 700 includes a state acquisition unit 702, information acquisition units 711 and 721, state estimation units 712 and 722, an environment management unit 704, a state selection unit 703, an alert notification unit 206, and a state comparison unit 207.

[0135] The object of monitoring of the anomaly detection system 700 is a robot 701, and the external information sources are a fixed camera 710 and a fixed microphone 720.

[0136] The robot 701 transmits its position information to the anomaly detection system 700. As the position information, positioning information from a Global Positioning System (GPS) device mounted on the robot 701 or position information estimated by Simultaneous Localization and Mapping (SLAM) is used.

[0137] Fixed camera 710 is installed in a position where it is easy to capture an image of robot 701, for example, near the ceiling of the room, and transmits captured image 713 to anomaly detection system 700, and fixed microphone 720 is installed in a position where it is easy to pick up sounds associated with the activities of robot 701, for example, near the entrance to the room, and transmits picked up audio 723 to anomaly detection system 700. Fixed camera 710 is an example of a first sensing device, and fixed microphone 720 is an example of a second sensing device.

[0138] The state acquisition unit 702 receives position information from the robot 701. The state acquisition unit 702 acquires the position information of the robot 701 as the state of the robot 701.

[0139] The information acquisition unit 711 receives the video 713 transmitted by the fixed camera 710 and extracts video data to be used for state estimation. An example of the video data is a bitmap image converted to grayscale. The video 713 is an example of first sensing information.

[0140] The state estimation unit 712 uses the video data extracted by the information acquisition unit 711 to estimate the position of the robot 701 and outputs it as an estimated state 714 .

[0141] The information acquisition unit 721 receives the audio 723 transmitted by the fixed microphone 720 and extracts audio data to be used for state estimation. An example of the audio data is, but is not limited to, PCM (Pulse Code Modulation) data that is divided at regular intervals and band-pass filtered so as to include only frequencies that make it easy to identify the active sounds of the robot 701. The audio 723 is an example of second sensing information.

[0142] The state estimation unit 722 uses the voice data extracted by the information acquisition unit 721 to estimate the position of the robot 701 and outputs it as an estimated state 724 .

[0143] The environment management unit 704 manages environment information 705 (see FIG. 8) that holds the estimated accuracy of the fixed camera 710 and the fixed microphone 720 at each time.

[0144] The state selection unit 703 selects the estimated state with the highest estimation accuracy from the estimated state 714 and the estimated state 724 according to the environment information 705 managed by the environment management unit 704 .

[0145] The alert notification unit 206 has the same configuration as that of the anomaly detection system 200 .

[0146] The state comparison unit 207 compares the location information extracted by the state acquisition unit 702 with the estimated state (estimated location information) selected by the state selection unit 703, and if there is a difference of a certain amount or more, detects an abnormality and notifies the alert notification unit 206 of the detection of the abnormality.

[0147] More specifically, the state comparison unit 207 acquires the position information of the robot 701 in the form of latitude and longitude from the state acquisition unit 702, acquires the estimated position information of the robot 701 in the form of latitude and longitude from the state selection unit 703, calculates the distance between the two pieces of position information, and detects an abnormality if the calculated distance exceeds a preset threshold.

[0148] When the robot 701 is active in a facility such as an office building or a commercial facility, it is assumed that the illumination level within the facility will be high and the surrounding noise will be louder during the robot 701's activity hours, and that the illumination level within the facility will be low and the surrounding noise will be quieter outside the robot 701's activity hours.

[0149] In this case, it is highly likely that higher estimation accuracy can be achieved by estimating the state from the image 713 of the fixed camera 710 during the robot 701's activity period, and estimating the state from the audio 723 of the fixed microphone 720 outside the robot 701's activity period.

[0150] FIG. 8 is a diagram showing an example of an environment information table managed by the environment management unit 704 of the anomaly detection system according to this embodiment.

[0151] As shown in FIG. 8, in the environmental information table, the estimation accuracy of the state estimation unit 712 is set high between 12:00 and 22:00, which is within the activity time period of the robot 701, and the estimation accuracy of the state estimation unit 722 is set high between 22:00 and 12:00, which is outside the activity time period of the robot 701.

[0152] The state selection unit 703 can select the position with the higher estimation accuracy 1002 at a given time in the environment 1001 of the environment information table as the position of the robot 701, thereby selecting the optimal estimated state in accordance with changes in time.

[0153] The environment information table shown in FIG. 8 may be created from the environment information of the environment management unit 302 in the state estimation unit learning system 300 after the state estimation units 712 and 722 have been trained in the state estimation unit learning system 300.

[0154] In this embodiment, for simplicity of explanation, the number of fixed cameras 710 and fixed microphones 720 is limited to one, but a plurality of each may be used.

[0155] (Embodiment 3) Next, an anomaly detection system for a case where a monitored object is a security robot engaged in security work will be described. In this embodiment, a case where it is detected whether or not location information notified from the security robot has been tampered with will be described.

[0156] FIG. 9 is a block diagram showing the functional configuration of an anomaly detection system 800 according to this embodiment.

[0157] As shown in FIG. 9, the anomaly detection system 800 includes a state acquisition unit 802, information acquisition units 811 and 821, state estimation units 812 and 822, an environment management unit 804, a state selection unit 803, an alert notification unit 206, and a state comparison unit 207.

[0158] The object of monitoring of the anomaly detection system 800 is a security robot 801, and the external information sources are a fixed camera 810 and an illuminance sensor 820.

[0159] The security robot 801 transmits its position information to the anomaly detection system 800. As the position information, positioning information from a GPS device mounted on the security robot 801 or position information estimated by SLAM is used.

[0160] Fixed camera 810 is installed in a position that is easy for security robot 801 to take pictures of, for example, near the ceiling of a room, and transmits captured video 813 to anomaly detection system 800, while illuminance sensor 820 is installed on the movement path of security robot 801 and transmits acquired illuminance value 823 to anomaly detection system 800. Fixed camera 810 is an example of a first sensing device, and illuminance sensor 820 is an example of a second sensing device.

[0161] The status acquisition unit 802 receives position information from the security robot 801 .

[0162] The information acquisition unit 811 receives the video 813 transmitted by the fixed camera 810 and extracts video data to be used for state estimation. An example of the video data is a bitmap image converted to grayscale. The video 813 is an example of first sensing information.

[0163] The state estimation unit 812 uses the video data extracted by the information acquisition unit 811 to estimate the position of the security robot 801 and outputs it as an estimated state 814 .

[0164] The information acquisition unit 821 receives the illuminance value 823 transmitted by the illuminance sensor 820 and extracts illuminance data to be used for state estimation. The illuminance data is a value obtained by converting the illuminance value into lux, which is a unit of brightness. The illuminance value 823 is an example of second sensing information.

[0165] The state estimation unit 822 uses the illuminance data extracted by the information acquisition unit 821 to estimate the position of the security robot 801 and outputs it as an estimated state 824 .

[0166] Specifically, when the surrounding lights are not on, such as at night, the security robot 801 turns on its own lights when patrolling, so the information acquisition unit 821 acquires the illuminance value of the illuminance sensor 820, and the state estimation unit 822 estimates how far away the security robot 801 is from the acquired illuminance value.

[0167] The environment management unit 804 manages environment information 805 (see FIG. 10) that holds the estimated accuracy of the fixed camera 810 and the illuminance sensor 820 at each time.

[0168] The state selection unit 803 selects the estimated state with the highest estimation accuracy from the estimated state 814 and the estimated state 824 according to the environment information 805 managed by the environment management unit 804 .

[0169] The alert notification unit 206 has the same configuration as that of the anomaly detection system 200 .

[0170] The state comparison unit 207 compares the location information extracted by the state acquisition unit 802 with the estimated state (estimated location information) selected by the state selection unit 803, and if there is a difference of a certain amount or more, detects an abnormality and notifies the alert notification unit 206 of the detection of the abnormality.

[0171] When security robot 801 patrols as part of its security work, it is assumed that the built-in camera of security robot 801 records the patrol while it is on patrol, and the recording serves as evidence of the patrol. Therefore, when security robot 801 patrols at night, security robot 801 turns on the lights to ensure the illumination required for recording or for security purposes, and the illumination value around security robot 801 becomes high.

[0172] FIG. 10 is a diagram showing an example of an environment information table managed by the environment management unit 804 of the anomaly detection system 800 according to this embodiment.

[0173] As shown in Figure 10, in the environmental information table, the estimation accuracy of the state estimation unit 812 is set high from 08:00 to 20:00, when the surroundings are bright, such as during the day, and the security robot 801 is clearly visible in the image captured by the fixed camera 810, and the estimation accuracy of the state estimation unit 822 is set high from 22:00 to 08:00, when the surroundings are dark, such as at night, and the security robot 801 is not clearly visible in the image captured by the fixed camera 810, but fluctuations in illuminance values ​​due to the lighting turned on by the cleaning robot are easily detected.

[0174] The state selection unit 803 can select the optimal estimated state according to changes in time by selecting the position with the higher estimation accuracy 1102 as the position of the security robot 801 at a given time in the environment 1101 of the environmental information table.

[0175] The environment information table may be created from the environment information of the environment management unit 302 in the state estimation unit learning system 300 after the state estimation units 812 and 822 have been trained in the state estimation unit learning system 300 .

[0176] In this embodiment, for simplicity of explanation, the number of fixed cameras 810 and illuminance sensors 820 is limited to one, but a plurality of each may be used.

[0177] (Fourth embodiment) Next, an anomaly detection system for monitoring a cleaning robot that performs cleaning work will be described. In this embodiment, a case will be described in which it is detected whether or not location information notified by the cleaning robot has been tampered with.

[0178] FIG. 11 is a block diagram showing the functional configuration of an anomaly detection system 900 according to this embodiment.

[0179] As shown in FIG. 11, the anomaly detection system 900 includes a state acquisition unit 902, information acquisition units 911 and 921, state estimation units 912 and 922, an environment management unit 904, a state selection unit 903, an alert notification unit 206, and a state comparison unit 207.

[0180] The object to be monitored by the anomaly detection system 900 is a cleaning robot 901, and the external information sources are a fixed camera 910 and a dust sensor 920.

[0181] The cleaning robot 901 transmits its position information to the anomaly detection system 900. As the position information, positioning information from a GPS device mounted on the cleaning robot 901 or position information estimated by SLAM is used.

[0182] The fixed camera 910 is installed in a position where the cleaning robot 901 can easily take pictures, for example, near the ceiling of a room, and transmits the captured image 913 to the anomaly detection system 900, while the dust sensor 920 is installed in the vicinity of the place where the cleaning robot 901 performs cleaning work, and transmits the acquired sensor value 923 to the anomaly detection system 900. The fixed camera 910 is an example of a first sensing device, and the dust sensor 920 is an example of a second sensing device.

[0183] The status acquisition unit 902 receives position information from the cleaning robot 901 .

[0184] The information acquisition unit 911 receives the video 913 transmitted by the fixed camera 910 and extracts video data to be used for state estimation. An example of the video data is a bitmap image converted to grayscale. The video 913 is an example of first sensing information.

[0185] The state estimation unit 912 uses the video data extracted by the information acquisition unit 911 to estimate the position of the cleaning robot 901 and outputs it as an estimated state 914 .

[0186] The information acquisition unit 921 receives the sensor value 923 transmitted by the dust sensor 920 and extracts the sensor value to be used for state estimation. An example of the sensor value is the amount of particles present in the atmosphere. The sensor value 923 is an example of second sensing information.

[0187] The state estimation unit 922 estimates the position of the cleaning robot 901 using the sensor data extracted by the information acquisition unit 921 and outputs the estimated state 924 .

[0188] Specifically, the cleaning robot 901 performs cleaning work in a situation where there are no humans around, such as at night. As the cleaning work by the cleaning robot 901 stirs up dust in the surrounding area, the information acquisition unit 921 acquires the sensor value of the dust sensor 920, and the state estimation unit 922 estimates how far away the cleaning robot 901 is from the acquired sensor value.

[0189] The environment management unit 904 manages environment information 905 (see FIG. 12) that holds the estimated accuracy of the fixed camera 910 and the dust sensor 920 at each time.

[0190] The state selection unit 903 selects the estimated state with the highest estimation accuracy from the estimated state 914 and the estimated state 924 according to the environment information 905 managed by the environment management unit 904 .

[0191] The alert notification unit 206 has the same configuration as that of the anomaly detection system 200 .

[0192] The state comparison unit 207 compares the location information extracted by the state acquisition unit 902 with the estimated state (estimated location information) selected by the state selection unit 903, and if there is a difference of a certain amount or more, detects an abnormality and notifies the alert notification unit 206 of the detection of the abnormality.

[0193] When the cleaning robot 901 performs cleaning work, it is expected that dust will be stirred up into the air, and therefore the sensor value of the dust sensor will fluctuate greatly around the cleaning robot 901 during cleaning work.

[0194] FIG. 12 is a diagram showing an example of an environment information table managed by the environment management unit 904 of the anomaly detection system 900 according to this embodiment.

[0195] As shown in FIG. 12, in the environmental information table, the estimation accuracy of the state estimation unit 912 is set high during the period from 07:00 to 21:00 when the surroundings are bright, such as during the day, and the cleaning robot 901 is clearly visible in the image captured by the fixed camera 910. The estimation accuracy of the state estimation unit 922 is set high during the period from 21:00 to 07:00 when the surroundings are dark, such as at night, and the cleaning robot 901 is not clearly visible in the image captured by the fixed camera 910.

[0196] The state selection unit 903 can select the position of the cleaning robot 901 that has the higher estimation accuracy 1202 at a given time in the environment 1201 of the environment information table, thereby selecting the optimal estimated state according to the change in time.

[0197] The environment information table may be created from the environment information of the environment management unit 302 in the state estimation unit learning system 300 after the state estimation units 912 and 922 have been trained in the state estimation unit learning system 300 .

[0198] In this embodiment, for the sake of simplicity, the number of fixed cameras 910 and dust sensors 920 is limited to one, but a plurality of each may be used.

[0199] (Other embodiments) While the remote monitoring system according to one or more aspects has been described above based on each embodiment, the present disclosure is not limited to these embodiments. As long as it does not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments may also be included in the present disclosure.

[0200] For example, the monitoring target according to each of the above embodiments may be a mobile object used indoors or an outdoor mobile object.

[0201] Furthermore, the monitoring system according to each of the above embodiments may include only one external information source. In this case, the anomaly detection system may not include a state selection unit. Furthermore, the environment management unit may output environmental information to the state comparison unit, and the state comparison unit may compare the state information with the estimated state information based on the environmental information. For example, the state comparison unit may change a threshold value used to compare the difference between the state information and the estimated state information in accordance with the environmental information. For example, the state comparison unit may change the threshold value to a larger value as the estimation accuracy indicated by the environmental information increases.

[0202] Furthermore, in each of the above embodiments, the square sum error is used as an example of the loss function, but the loss function is not limited to the square sum error, and any loss function may be used.

[0203] Furthermore, in the above-described second to fourth embodiments, an example in which a fixed camera is used as the first sensing device has been described, but the present invention is not limited to this. The first sensing device and the second sensing device may be, for example, a sensing device other than a fixed camera, as long as they are capable of acquiring different sensing information from each other.

[0204] In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0205] The order in which the steps in the flowchart are executed is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. Also, some of the steps may be executed simultaneously (in parallel) with other steps, or some of the steps may not be executed.

[0206] The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.

[0207] Furthermore, the remote monitoring system according to each of the above embodiments may be realized as a single device or may be realized by multiple devices. When the remote monitoring system is realized by multiple devices, the components of the remote monitoring system may be distributed among the multiple devices in any manner. When the remote monitoring system is realized by multiple devices, the communication method between the multiple devices is not particularly limited and may be wireless communication or wired communication. Furthermore, wireless communication and wired communication may be combined between the devices.

[0208] Furthermore, each component described in the above embodiments may be implemented as software or, typically, as an LSI, which is an integrated circuit. These components may be individually integrated into a single chip, or some or all of them may be integrated into a single chip. While LSI is used here, it may also be referred to as an IC, system LSI, super LSI, or ultra LSI depending on the level of integration. Furthermore, the integration method is not limited to LSI; it may be implemented using a dedicated circuit (e.g., a general-purpose circuit that executes a dedicated program) or a general-purpose processor. It is also possible to use a field programmable gate array (FPGA), which can be programmed after LSI fabrication, or a reconfigurable processor, which allows the connection or settings of circuit cells within an LSI to be reconfigured. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or a derivative technology, that technology may naturally be used to integrate the components.

[0209] A system LSI is an ultra-multifunctional LSI manufactured by integrating multiple processing units on a single chip, and is specifically a computer system comprising a microprocessor, ROM (Read Only Memory), RAM (Random Access Memory), etc. Computer programs are stored in the ROM. The system LSI achieves its functions when the microprocessor operates in accordance with the computer program.

[0210] Furthermore, one aspect of the present disclosure may be a computer program that causes a computer to execute each of the characteristic steps included in the remote monitoring method shown in any of FIG. 4, FIG. 6A, and FIG. 6B.

[0211] Furthermore, for example, the program may be a program to be executed by a computer. Another aspect of the present disclosure may be a computer-readable non-transitory recording medium on which such a program is recorded. For example, such a program may be recorded on a recording medium and distributed or circulated. For example, the distributed program may be installed in a device having another processor, and the program may be executed by the processor, thereby causing the device to perform each of the above processes.

[0212] (Addendum) The above description of the embodiments discloses the following techniques.

[0213] (Technology 1) A remote monitoring system that detects an abnormality in the state of an autonomously operating monitoring target, a first acquisition unit that acquires status information indicating a status of the monitoring target from the monitoring target; a second acquisition unit that acquires first sensing information obtained by sensing the monitoring target from a first sensing device that is provided outside the monitoring target and senses the monitoring target; a state estimation unit that estimates a first state, which is a state of the monitoring target, based on the first sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the first state of the monitoring target estimated by the state estimation unit; a notification unit that notifies a supervisor of the remote monitoring system that an abnormality has occurred based on the comparison result of the status comparison unit. Remote monitoring system.

[0214] (Technology 2) the second acquisition unit further acquires second sensing information obtained by sensing the monitoring target, the second sensing information being different from the first sensing information, from a second sensing device that is provided outside the monitoring target and senses the monitoring target; the state estimation unit estimates a second state, which is the state of the monitoring target, based on the second sensing information acquired by the second acquisition unit; The estimated state information is information further based on the second state. The remote monitoring system according to technique 1.

[0215] (Technology 3) moreover, an environment management unit that manages environment information in which an environment when the monitoring target is sensed is associated with an estimation accuracy of the state estimated by the state estimation unit; a state determination unit that determines one state from the first state and the second state using the environmental information managed by the environment management unit, and outputs the determined one state as the estimated state information. The remote monitoring system according to Art. 2.

[0216] (Technology 4) The state determination unit selects one of the first state and the second state using the environmental information, and outputs the selected one as the estimated state information. The remote monitoring system described in technique 3.

[0217] (Technology 5) The state determination unit performs a weighting calculation on the first state and the second state using the environmental information, and outputs the weighted state as the estimated state information. The remote monitoring system described in technique 3.

[0218] (Technology 6) the first acquisition unit acquires a current location of the monitoring target as the state; the state estimation unit estimates a current position of the monitoring target as the first state and the second state; The environmental information includes estimation accuracy of the first state and the second state at each time. The remote monitoring system according to any one of techniques 3 to 5.

[0219] (Technology 7) the monitoring target is a robot, the first sensing device includes a fixed camera; The second sensing device includes a fixed microphone. The remote monitoring system described in technique 6.

[0220] (Technology 8) the monitoring target is a robot that performs security work, the first sensing device includes a fixed camera; The second sensing device includes an illuminance sensor. The remote monitoring system described in technique 6.

[0221] (Technology 9) the monitoring target is a robot that performs cleaning work, the first sensing device includes a fixed camera; The second sensing device includes a dust sensor. The remote monitoring system described in technique 6.

[0222] (Technology 10) The monitoring system further includes a state learning unit that calculates information indicating an estimation accuracy of at least one of the first state and the second state based on the state of the monitoring object acquired by the first acquisition unit and at least one of the first state and the second state of the monitoring object estimated by the state estimation unit, and outputs the calculated information indicating the estimation accuracy to the state estimation unit and the environment management unit. The remote monitoring system according to any one of techniques 3 to 9.

[0223] (Technology 11) The environment management unit determines the estimation accuracy of the state estimated by the state estimation unit based on the information indicating the estimation accuracy acquired from the state learning unit. The remote monitoring system described in Technology 10.

[0224] (Technology 12) The state learning unit outputs information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit before the start of operation of the monitored object. The remote monitoring system according to any one of claims 10 to 11.

[0225] (Technology 13) The state learning unit outputs information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit during operation of the monitored object. The remote monitoring system according to any one of techniques 10 to 12.

[0226] (Technology 14) further comprising an environment management unit that manages environment information in which an environment when the monitoring target is sensed is associated with an estimation accuracy of the state estimated by the state estimation unit, The state comparison unit compares the state information with the estimated state information based on the environmental information. The remote monitoring system according to technique 1.

[0227] (Technology 15) An anomaly detection system that detects an anomaly in a state of an autonomously operating monitoring target, a first acquisition unit that acquires status information indicating a status of the monitoring target from the monitoring target; a second acquisition unit that acquires sensing information obtained by sensing the monitoring target from a sensing device that is provided outside the monitoring target and senses the monitoring target; a state estimation unit that estimates a state of the monitoring target based on the sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the state of the monitoring target estimated by the state estimation unit. Anomaly detection system.

[0228] (Technology 16) A remote monitoring method for detecting an abnormality in the state of an autonomously operating monitoring target, comprising: acquiring status information indicating a status of the monitoring target from the monitoring target; acquiring sensing information obtained by sensing the monitoring target from a sensing device that is provided outside the monitoring target and senses the monitoring target; Estimating a state of the monitored object based on the acquired sensing information; comparing the acquired state information with estimated state information based on the estimated state of the monitored object; Based on the result of comparing the state information with the estimated state information, a supervisor of the remote monitoring system is notified that an abnormality has occurred. Remote monitoring methods.

[0229] (Technology 17) A program for causing a computer to execute the remote monitoring method described in Technology 16. [Industrial Applicability]

[0230] The present disclosure is useful for an anomaly detection system that, when remotely monitoring an autonomously operating robot, determines whether the state that the monitored robot notifies a remote monitoring system of differs from its actual state. [Explanation of symbols]

[0231] 100 Surveillance System 101 Remote Monitoring System 102 Monitoring Targets 103, 103a, 103b External information source (sensing device) 104 Network 200, 700, 800, 900 Anomaly Detection System 201, 201a, 702, 802, 902 Status acquisition unit (first acquisition unit) 202, 202a, 202b, 202c, 711, 721, 811, 821, 911, 921 Information acquisition section (second acquisition section) 203a, 203b, 303, 712, 722, 812, 822, 912, 922 State estimation unit 204, 302, 704, 804, 904 Environmental Management Department 205, 703, 803, 903 State selection unit (state determination unit) 206 Alert notification section 207 State comparison section 300 State Estimation Unit Learning System 301 State Learning Unit 501, 1001, 1101, 1201 environment 502, 1002, 1102, 1202 Estimated accuracy 701 Robot (Monitored) 705, 805, 905 Environmental information 710, 810, 910 Fixed cameras (first sensing device) 713, 813, 913 Video (first sensing information) 714, 724, 814, 824, 914, 924 Estimated state 720 Fixed microphone (second sensing device) 723 Voice (second sensing information) 801 Security Robot (Monitoring Target) 820 Illuminance sensor (second sensing device) 823 Illuminance value (second sensing information) 901 Cleaning robot (monitored) 920 Dust sensor (second sensing device) 923 Sensor value (second sensing information)

Claims

1. A remote monitoring system that detects an abnormality in the state of an autonomously operating monitoring target, a first acquisition unit that acquires status information indicating a status of the monitoring target from the monitoring target; a second acquisition unit that acquires first sensing information indicating a sensing result of the monitoring target from a first sensing device that is provided outside the monitoring target and senses the monitoring target; a state estimation unit that estimates a first state, which is a state of the monitoring target, based on the first sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the first state of the monitoring target estimated by the state estimation unit; a notification unit that notifies a supervisor of the remote monitoring system that an abnormality has occurred based on the comparison result of the status comparison unit. Remote monitoring system.

2. the second acquisition unit further acquires second sensing information, which is different from the first sensing information, from a second sensing device that senses the monitoring target and is provided outside the monitoring target; the state estimation unit estimates a second state, which is the state of the monitoring target, based on the second sensing information acquired by the second acquisition unit; The estimated state information is information further based on the second state. The remote monitoring system of claim 1 .

3. moreover, an environment management unit that manages environment information in which an environment when the monitoring target is sensed is associated with an estimation accuracy of the state estimated by the state estimation unit; a state determination unit that determines one state from the first state and the second state using the environmental information managed by the environment management unit, and outputs the determined one state as the estimated state information. The remote monitoring system according to claim 2 .

4. The state determination unit selects one of the first state and the second state using the environmental information, and outputs the selected one as the estimated state information. The remote monitoring system according to claim 3 .

5. The state determination unit performs a weighting calculation on the first state and the second state using the environmental information, and outputs the weighted states as the estimated state information. The remote monitoring system according to claim 3 .

6. the first acquisition unit acquires a current location of the monitoring target as the state; the state estimation unit estimates a current position of the monitoring target as the first state and the second state; The environmental information includes estimation accuracy of the first state and the second state at each time. The remote monitoring system according to any one of claims 3 to 5.

7. the monitoring target is a robot, the first sensing device includes a fixed camera; The second sensing device includes a fixed microphone. The remote monitoring system of claim 6.

8. the monitoring target is a robot that performs security work, the first sensing device includes a fixed camera; The second sensing device includes an illuminance sensor. The remote monitoring system of claim 6.

9. the monitoring target is a robot that performs cleaning work, the first sensing device includes a fixed camera; The second sensing device includes a dust sensor. The remote monitoring system of claim 6.

10. The monitoring system further includes a state learning unit that calculates information indicating an estimation accuracy of at least one of the first state and the second state based on the state of the monitoring object acquired by the first acquisition unit and at least one of the first state and the second state of the monitoring object estimated by the state estimation unit, and outputs the calculated information indicating the estimation accuracy to the state estimation unit and the environment management unit. The remote monitoring system according to any one of claims 3 to 5.

11. The environment management unit determines the estimation accuracy of the state estimated by the state estimation unit based on the information indicating the estimation accuracy acquired from the state learning unit. The remote monitoring system of claim 10.

12. The state learning unit outputs information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit before the start of operation of the monitored object. The remote monitoring system of claim 10.

13. The state learning unit outputs information indicating the calculated estimation accuracy to the state estimation unit and the environment management unit during operation of the monitored object. The remote monitoring system of claim 10.

14. further comprising an environment management unit that manages environment information in which an environment when the monitoring target is sensed is associated with an estimation accuracy of the state estimated by the state estimation unit, The state comparison unit compares the state information with the estimated state information based on the environmental information. The remote monitoring system of claim 1 .

15. An anomaly detection system that detects an anomaly in a state of an autonomously operating monitoring target, a first acquisition unit that acquires status information indicating a status of the monitoring target from the monitoring target; a second acquisition unit that acquires sensing information indicating a sensing result of the monitoring target from a sensing device that is provided outside the monitoring target and senses the monitoring target; a state estimation unit that estimates a state of the monitoring target based on the sensing information acquired by the second acquisition unit; a state comparison unit that compares the state information acquired by the first acquisition unit with estimated state information based on the state of the monitoring target estimated by the state estimation unit. Anomaly detection system.

16. A remote monitoring method for detecting an abnormality in the state of an autonomously operating monitoring target, comprising: acquiring status information indicating a status of the monitoring target from the monitoring target; acquiring sensing information indicating a sensing result of the monitoring target from a sensing device that senses the monitoring target and is provided outside the monitoring target; Estimating a state of the monitored object based on the acquired sensing information; comparing the acquired state information with estimated state information based on the estimated state of the monitored object; Based on the result of comparing the state information with the estimated state information, a supervisor of the remote monitoring system is notified that an abnormality has occurred. Remote monitoring methods.

17. A program for causing a computer to execute the remote monitoring method according to claim 16.

Images