Risk estimation device, risk estimation method, and storage medium
The risk estimation device accurately assesses risks in monitored spaces by estimating relationships between objects and elements, addressing the inaccuracy of existing systems that rely on similar scene graphs, thereby enhancing risk estimation accuracy.
Patent Information
- Application Number
- PCT/JP2023/020278
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
Existing risk estimation systems struggle to accurately assess risks in monitored spaces with multiple people and objects, as similar scene graphs can represent different relationships, leading to inaccurate risk estimation.
A risk estimation device and method that estimate relationships between objects and elements using inter-object and inter-element relationship estimation, incorporating these relationships into the risk estimation model to provide accurate risk assessments in monitored spaces.
Enables accurate risk estimation by considering the specific relationships between people and objects, improving the accuracy of risk assessments compared to systems that rely solely on scene graphs.
Smart Images

Figure JP2023020278_05122024_PF_FP_ABST
Abstract
Description
Risk estimation device, risk estimation method, and storage medium
[0001] The present disclosure relates to the technical fields of a risk estimation device, a risk estimation method, and a storage medium for estimating a risk in a monitored space.
[0002] There is known a system that estimates risks within a monitored area based on images from a surveillance camera, etc. For example, Patent Literature 1 discloses a system that generates a scene graph representing the relationships between people and objects detected based on images from a camera, etc., and estimates risks within a monitored area using a co-occurrence model regarding the co-occurrence of elements of the scene graph and assumed risks.
[0003] Japanese Patent Application Laid-Open No. 2022-170790
[0004] In situations where there are multiple people and multiple objects in a space observed by a camera, etc., the correspondence between people and objects, and the relative positions of people may differ, but they may appear identical in the scene graph. In such cases, it is difficult to accurately estimate risk from the scene graph alone.
[0005] In view of the above-mentioned problems, one of the objects of the present disclosure is to provide a risk estimation device, a risk estimation method, and a storage medium that are capable of accurately estimating risks within an observed space.
[0006] One aspect of the risk estimation device is a risk estimation device having: an inter-object relationship estimation means for estimating relationships between objects detected from observation data; an inter-element relationship estimation means for estimating relationships between the objects and relationships between the elements when the objects having the relationships are elements; and a risk estimation means for estimating risks in a space observed from the observation data based on the elements and the relationships between the elements.
[0007] One aspect of the risk estimation method is a risk estimation method in which a computer estimates relationships between objects detected from observation data, estimates relationships between the objects and relationships between elements when the objects having the relationships are elements, and estimates risks in a space observed from the observation data based on the elements and the relationships between the elements. Note that the term "computer" includes any electronic device (which may be a processor included in an electronic device) and may be composed of multiple electronic devices.
[0008] One aspect of the storage medium is a storage medium that stores a program that causes a computer to execute a process of estimating relationships between objects detected by observation data, estimating relationships between the objects and relationships between elements when the objects having the relationships are used as elements, and estimating risks in the space observed by the observation data based on the elements and the relationships between the elements.
[0009] One example of the effect of the present disclosure is that it becomes possible to accurately estimate risks within the observed space.
[0010] 1 shows a schematic configuration of a risk estimation system according to a first embodiment. FIG. 2 shows an example of the hardware configuration of a risk estimation device common to all embodiments. FIG. 3 shows an example of functional blocks possessed by a processor of a risk estimation device according to a first embodiment. FIG. 4 shows an example of a scene graph. FIG. 5 shows a scene graph specifying the relationships between elements. (A) shows a situation in which person p1 is holding drug s1 and drug s2. (B) shows a situation in which person p1 is holding drug s1 and person p2 is holding drug s2. (C) is a graph showing the inter-element relationships between element ea, element eb, and element ea and element eb. FIG. 5 shows an example of a co-occurrence model. FIG. 6 shows an example of a flowchart related to processing executed by the risk estimation device according to the first embodiment. FIG. 7 shows an example of functional blocks possessed by a processor of a risk estimation device according to a modified example. FIG. 7 shows a schematic configuration of a risk estimation system according to a second embodiment. FIG. 8 shows a schematic configuration of a video monitoring system according to a third embodiment. FIG. 9 is a block diagram of a risk estimation device according to a fourth embodiment. FIG. 10 shows an example of a flowchart executed by the risk estimation device according to the fourth embodiment.
[0011] Hereinafter, embodiments of a risk estimation device, a risk estimation method, and a storage medium will be described with reference to the drawings.
[0012] <First Embodiment> (1) System Configuration Fig. 1 shows a schematic configuration of a risk estimation system 100 according to the first embodiment. The risk estimation system 100 is a system that estimates (evaluates) risks in a space to be monitored (also referred to as a "monitored space") based on information obtained by observing the monitored space, and mainly includes a risk estimation device 1, an input device 2, an output device 3, a storage device 4, and an observation device 5. The monitored space may be a factory, a construction site, a warehouse, a hospital, a vehicle, a train station, or any other space (area) to be monitored.
[0013] The risk estimation device 1 detects objects present in the monitored space based on the observation data generated by the observation device 5, and estimates the risk in the monitored space based on the relationships between the detected objects, etc. The risk estimation device 1 then outputs information related to the risk estimation results. Here, "objects" include people and non-human objects, and "objects" may be control panels, arms, pillars, fire extinguishers, walkways, workbenches, and any other non-human objects that affect risk.
[0014] The risk estimation device 1 performs data communication with the input device 2, output device 3, storage device 4, and observation device 5 via a communication network or by direct wireless or wired communication.
[0015] The input device 2 is an interface that accepts input (user input) from a user who manages risks in a monitored space. The input device 2 may be, for example, any of various user input interfaces such as a touch panel, a button, a keyboard, a mouse, or a voice input device. The input device 2 supplies an input signal generated based on the user input to the risk estimation device 1.
[0016] The output device 3 outputs predetermined information based on the output signal supplied from the risk estimation device 1. In this case, the output signal includes at least one of a display signal and an audio signal. The output device 3 displays information based on the display signal supplied from the risk estimation device 1, and outputs information as audio based on the audio signal supplied from the risk estimation device 1. Examples of the output device 3 include display devices such as a display or a projector, and audio output devices such as a speaker.
[0017] The storage device 4 is a memory that stores various information related to risk estimation. The storage device 4 may be an external storage device such as a hard disk connected to or built into the risk estimation device 1, or may be a storage medium such as a flash memory. The storage device 4 may also be a server device that performs data communication with the risk estimation device 1. The storage device 4 may also be composed of multiple devices.
[0018] The storage device 4 stores object detection model information D1 and risk estimation model information D2. The object detection model information D1 is information such as parameters required to configure the object detection model. The object detection model is a model (engine) that outputs detection results for objects included in observation data when observation data is input. The risk estimation model information D2 is information such as parameters required to configure the risk estimation model. The risk estimation model is a model (engine) that outputs estimation results for risks in the monitored space when a scene graph (described below) and data indicating the relationships between elements of the scene graph are input. The object detection model and risk estimation model are, for example, machine learning models. Details of the object detection model information D1 and the risk estimation model information D2 will be described later.
[0019] The observation device 5 is a sensor that observes the monitored space, and may be, for example, one or more cameras (imaging means) that capture images of the monitored space as an observation range. In addition to a camera, the observation device 5 may also have an external sensor other than a camera (for example, a lidar, etc.), a microphone (audio input device) that collects sounds within the monitored space, and various sensors that detect the date, time, air pressure, illuminance, temperature, etc.
[0020] 1 is an example, and various modifications may be made to the configuration. For example, the risk estimation device 1 may be configured integrally with at least one of an input device 2, an output device 3, a storage device 4, and an observation device 5. In this case, the risk estimation system 100 may be realized by a single terminal device such as a smartphone or a tablet terminal.
[0021] (2) Hardware Configuration Fig. 2 shows the hardware configuration of the risk estimation device 1. The risk estimation device 1 includes, as hardware, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 90.
[0022] The processor 11 executes programs stored in the memory 12 to function as a controller (arithmetic unit) that performs overall control of the risk estimation device 1. The processor 11 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.
[0023] The memory 12 is composed of various types of volatile and non-volatile memory, such as RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The memory 12 also stores programs for executing the processes performed by the risk estimation device 1. Some of the information stored in the memory 12 may be stored in one or more external storage devices capable of communicating with the risk estimation device 1, or may be stored in a storage medium that is detachable from the risk estimation device 1. The memory 12 may also function as at least a part of the storage device 4.
[0024] The interface 13 is an interface for electrically connecting the risk estimation device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.
[0025] The hardware configuration of the risk estimation device 1 is not limited to the configuration shown in Fig. 2. For example, the risk estimation device 1 may include at least one of an input device 2, an output device 3, a storage device 4, and an observation device 5.
[0026] (3) Overview of Risk Estimation Next, we will explain the risk estimation performed by the risk estimation device 1. In summary, the risk estimation device 1 estimates risks in the monitored space based on a scene graph representing relationships between objects detected based on observation data output by the observation device 5 and the relationships between elements in the scene graph. In this way, the risk estimation device 1 accurately estimates risks in the monitored space and outputs information related to the estimation results.
[0027] 3 shows an example of functional blocks included in the processor 11 of the risk estimation device 1. Functionally, the processor 11 of the risk estimation device 1 includes an object detection unit 14, a scene graph generation unit 15, an inter-element relationship estimation unit 16, a risk estimation unit 17, and an output control unit 18. Note that in FIG. 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to that shown. The same applies to other functional block diagrams described below.
[0028] When the time for risk estimation arrives, the object detection unit 14 detects objects present in the monitored space of the observation device 5 based on the observation data output by the observation device 5. In this case, the object detection unit 14 configures an object detection model based on the object detection model information D1, and detects objects based on information output by the object detection model when observation data is input to the configured object detection model. The object detection unit 14 assigns, for example, to each detected object, an object ID that identifies the object and a label that indicates the type of the object (i.e., the class into which the object is classified).
[0029] The timing of risk estimation may be at predetermined time intervals (for example, intervals based on the intervals at which observation data is acquired), may be the timing when an instruction signal instructing execution of risk estimation is received from the input device 2 or the like, or may be the timing when a risk estimation start condition stored in advance in the memory 12 or the like is satisfied. Furthermore, the object detection unit 14 may detect an object based on observation data obtained from the observation device 5 at the timing of risk estimation, or may detect an object based on time-series observation data obtained during the period from a predetermined time before the timing of risk estimation to the timing of risk estimation.
[0030] Here, a supplementary explanation of the object detection model is provided. The object detection model is, for example, a machine learning model (including a statistical model; the same applies hereinafter) having any architecture, such as a neural network or a support vector machine. The object detection model may also be a model used in segmentation, such as instance segmentation. The object detection model is trained using learning data (training data), and parameters of the object detection model obtained by the training are pre-stored in the memory 12 as object detection model information D1. Here, the training data includes, for example, multiple records each consisting of a pair of observation data that matches the input format (a predetermined tensor format) of the object detection model and correct answer data that indicates the correct answer to be output by the object detection model when the observation data is input. When the object detection model is configured using a neural network, the object detection model information D1 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and filter size of filters in each layer, and the weight of each element of each filter.
[0031] The scene graph generation unit 15 estimates a scene graph representing the relationship between the people and objects detected by the object detection unit 14. The scene graph generation unit 15 estimates whether or not there is a relationship between the detected people and objects, and estimates the relationship between the people and objects that are estimated to be related. A scene graph is a graph in which people as objects and objects as objects are nodes, and the relationships between the nodes are represented by directed edges, and the relationships between the nodes (including the nodes that make up the relationships) are elements, and the scene graph represents a set of elements. Specific examples of scene graphs will be described later. Various methods have been proposed for generating scene graphs, and any of these methods can be adopted. In general, scene graphs are not limited to relationships between people and objects, but also include relationships between objects.
[0032] The inter-element relationship estimation unit 16 estimates the relationship between elements in a scene graph (also referred to as an "inter-element relationship"). For example, for any two elements (referred to as a "first element" and a "second element" for convenience of explanation), the inter-element relationship estimation unit 16 estimates the inter-element relationship between the first element and the second element and determines a class into which the inter-element relationship between the first element and the second element is classified. Then, the inter-element relationship estimation unit 16 regards each pair of elements in the scene graph as a first element and a second element, and estimates the inter-element relationship between each pair of elements in the scene graph. Then, the inter-element relationship estimation unit 16 supplies the estimation result of the inter-element relationship (e.g., information indicating the classified class) to the risk estimation unit 17.
[0033] Here, a specific example of an inter-element relationship will be described. In a first example of an inter-element relationship, the inter-element relationship is a relationship related to the distance between the elements, and the inter-element relationship estimation unit 16 estimates a relationship (i.e., a positional relationship) based on the distance between a person and an object of a first element and a person and an object of a second element. In this case, the inter-element relationship estimation unit 16 calculates the distance between the person and an object of the first element and the person and an object of the second element, and if the calculated distance is equal to or greater than a threshold, determines that the relationship between the first element and the second element belongs to a class corresponding to "close distance." If the calculated distance is less than the threshold, determines that the relationship between the first element and the second element belongs to a class corresponding to "far distance." The above-mentioned threshold is, for example, a predetermined compatibility value and is stored in the memory 12, etc. Note that the classification of relationships based on distance is not limited to two classes, "close distance" and "far distance," but may be three or more classes. In this case, for example, the inter-element relationship estimation unit 16 uses a plurality of threshold values with different values to classify the relationships based on the distance into one of three or more classes.
[0034] Here, the "distance" may be a two-dimensional distance on the image if the observation data is an image without distance information. Alternatively, it may be a three-dimensional distance in real space if the observation data is an image or three-dimensional point cloud data with distance information. Furthermore, the distance calculated by the inter-element relationship estimation unit 16 may be a distance based on an object, a distance based on a person, or a distance based on a person and an object. In this case, the inter-element relationship estimation unit 16 calculates, for example, the distance between any one point on the circumscribing rectangle of the person and / or object of the first element and any one point on the circumscribing rectangle of the person and / or object of the second element. As a specific example of such a distance calculation method, in an example where a distance based on an object is calculated, the inter-element relationship estimation unit 16 calculates, for example, the center-to-center distance between the circumscribing rectangle (bounding box) of the object of the first element and the circumscribing rectangle of the object of the second element in the image. Furthermore, in an example where a distance based on a person is calculated, the inter-element relationship estimation unit 16 calculates, for example, the center-to-center distance between the circumscribing rectangle of the person of the first element and the circumscribing rectangle of the person of the second element in the image. In an example of calculating the distance based on a person and an object, the inter-element relationship estimation unit 16 calculates, for example, the distance between the centers of the circumscribing rectangle of the person and object as a first element and the circumscribing rectangle of the person and object as a second element in the image. Note that when using three-dimensional distance in real space, the inter-element relationship estimation unit 16 calculates, for example, the distance between the centers of the circumscribing rectangular parallelepipeds of the person and / or object instead of calculating the distance between the centers of the circumscribing rectangles of the person and / or object.
[0035] In a second example of the inter-element relationship, the inter-element relationship indicates whether or not the elements share a common person, and the inter-element relationship estimation unit 16 estimates whether or not the person of the first element and the person of the second element are the same person. In this case, if the person of the first element and the person of the second element are the same (the same), the inter-element relationship estimation unit 16 determines that the inter-element relationship between the first element and the second element belongs to a class corresponding to "same person," and if the person of the first element and the person of the second element are different people, the inter-element relationship between the first element and the second element belongs to a class corresponding to "different people."
[0036] Note that the first and second examples of the inter-element relationship described above may be applied in combination. For example, if the person of the first element and the person of the second element are the same, the inter-element relationship estimation unit 16 determines, based on the second example, that the inter-element relationship between the first element and the second element belongs to a class corresponding to "the same person." On the other hand, if the person of the first element and the person of the second element are different people, the inter-element relationship estimation unit 16 further calculates a distance based on the first example and determines a class according to the distance.
[0037] The risk estimation unit 17 estimates the risk in the monitored space based on the scene graph generated by the scene graph generation unit 15 and the estimation results of the inter-element relationships generated by the inter-element relationship estimation unit 16. In this case, the risk estimation unit 17 configures a risk estimation model based on the risk estimation model information D2, and acquires the estimation results of the risk output by the risk estimation model when data indicating the scene graph and the estimation results of the inter-element relationships is input to the risk estimation model.
[0038] Here, the risk estimation model is, for example, a co-occurrence model, which is a machine learning model regarding the co-occurrence of elements and inter-element relationships in a scene graph with assumed risks. A specific example of the co-occurrence model will be described later.
[0039] The risk estimation model is not limited to a co-occurrence model. For example, the risk estimation model may be a model that outputs a risk estimation result when a scene graph and data in a predetermined tensor format indicating inter-element relationships are input to the risk estimation model. The risk estimation result output by the risk estimation model includes, for example, the most likely risk class and the confidence level that the risk will occur. The risk estimation model is, for example, a machine learning model (including a statistical model; the same applies hereinafter) having any architecture, such as a neural network or a support vector machine. Learning is performed using training data, and the parameters of the risk estimation model obtained by learning are pre-stored in the memory 12 as risk estimation model information D2. Here, the learning data includes, for example, multiple records each consisting of a pair of input data and correct answer data indicating the correct answer that the risk estimation model should output when the input data is input. When the risk estimation model is configured using a neural network, the risk estimation model information D2 includes various parameters (including hyperparameters), such as the layer structure, the neuron structure of each layer, the number and size of filters in each layer, and the weight of each element of each filter.
[0040] The output control unit 18 outputs information related to the risk estimation result output by the risk estimation unit 17. In this case, the output control unit 18 generates an output signal related to the risk estimation result and supplies the generated output signal to the output device 3 via the interface 13, thereby causing the output device 3 to execute display or audio output. For example, when the risk estimation unit 17 estimates the occurrence of a risk with a certainty level equal to or higher than a predetermined level, the output control unit 18 outputs, via the output device 3, information indicating a warning related to the risk and / or measures (countermeasures) for the risk. The above-mentioned predetermined level may be a different value for each risk.
[0041] Furthermore, the output control unit 18 may identify elements of a scene graph that contribute to the occurrence of a risk and relationships between elements, and output information about the identified elements and relationships between elements via the output device 3. When a co-occurrence model is used as the risk estimation model, the output control unit 18 may output information about relationships added by filtering using label propagation via the output device 3. When a co-occurrence model is used as the risk estimation model, the output control unit 18 is not limited to the above output examples, and may perform various outputs disclosed in Patent Document 1.
[0042] In addition, instead of displaying or outputting as audio using the output device 3, the output control unit 18 may transmit information regarding the risk estimation results (e.g., a report indicating the risk) to another device that manages the risk, or may store it in the memory device 4, etc.
[0043] Each of the components, including the object detection unit 14, the scene graph generation unit 15, the inter-element relationship estimation unit 16, the risk estimation unit 17, and the output control unit 18 described in FIG. 3, can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program on an arbitrary non-volatile storage medium and installing it as needed. Note that at least a portion of each of these components is not limited to being realized by software programs, but may also be realized by any combination of hardware, firmware, and software. Furthermore, at least a portion of each of these components may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, a program consisting of each of the above components may be realized using this integrated circuit. Furthermore, at least a portion of each component may be configured by an ASSP (Application Specific Standard Product), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.
[0044] (4) Scene Graph and Inter-Element Relationships Figure 4 is a diagram showing an example of a scene graph generated by the scene graph generation unit 15. In the example of Figure 4, eight objects are detected, and relationships r1 to r7 are defined between people with object IDs "o1" and "o2" and things with object IDs "o3" to "o8". The graph shown in Figure 4 corresponds to a bipartite graph divided into a set of person nodes and a set of thing nodes, and each relationship is represented by a directed edge. The scene graph shown in Figure 4 has seven elements, the same number as the number of directed edges.
[0045] 5 shows a scene graph specifying the inter-element relationships determined by the inter-element relationship estimation unit 16. In the example of FIG. 5, the inter-element relationship estimation unit 16 estimates a relationship r12 between an element e1 corresponding to a person with object ID "o1" and an object with object ID "o4" having a relationship r2, and an element e2 corresponding to a person with object ID "o2" and an object with object ID "o8" having a relationship r7. Note that the inter-element relationship estimation unit 16 also estimates inter-element relationships between pairs of elements other than the pair of element e1 and element e2.
[0046] The risk estimation unit 17 then performs risk estimation by taking into consideration not only the scene graph but also the relationships between elements, thereby enabling the risk estimation unit 17 to perform risk estimation that accurately takes into consideration the correspondence between people and objects, the positional relationships between people, and the like.
[0047] Here, the effect of risk estimation by the risk estimation unit 17 will be supplementally explained using a comparative example in which risk estimation is performed using only a scene graph as input to a co-occurrence model. In this comparative example, the positional relationship between elements is not taken into consideration, and risk may be erroneously estimated.
[0048] FIG. 6A shows a situation in which person p1 holds a drug s1 and a drug s2. FIG. 6B shows a situation in which person p1 holds a drug s1 and person p2 holds a drug s2. Here, since drugs s1 and s2 generate toxic gas when they come into contact with each other, as shown in FIG. 6A, when person p1 holds both drugs s1 and s2 at the same time, this corresponds to a situation in which the risk of toxic gas generation is high. On the other hand, as shown in FIG. 6B, when people p1 and p2, who are far apart, separately hold drugs s1 and s2, the risk of toxic gas generation is lower than in the situation shown in FIG. 6A. However, since all elements of the scene graph are input into the co-occurrence model, the input to the co-occurrence model is the same for the situation shown in FIG. 6A and the situation shown in FIG. 6B. Therefore, in the comparative example, the risk estimation results are the same for the situation shown in FIG. 6(A) and the situation shown in FIG. 6(B), and as a result, the risk estimation accuracy is low in the comparative example.
[0049] Taking the above into consideration, the risk estimation device 1 in this embodiment performs risk estimation by further considering the inter-element relationships. For example, in the situation shown in Figure 6(B), the risk estimation device 1 classifies the inter-element relationship between element ea, which corresponds to the relationship in which person p1 has drug s1, and element eb, which corresponds to the relationship in which person p2 has drug s2, as "distant."
[0050] FIG. 6(C) is a graph showing the element ea, element eb, and the inter-element relationship between element ea and element eb. The element ea, element eb, and the inter-element relationship between element ea and element eb correspond to the information input by the risk estimation unit 17 to the risk estimation model in the situation shown in FIG. 6(B). In the situation shown in FIG. 6(B), the risk estimation unit 17 adds the inter-element relationship "distant" between element ea and element eb to the input of the risk estimation model in addition to element ea and element eb. This enables the risk estimation unit 17 to perform risk estimation in accordance with the situation shown in FIG. 6(B). Note that in the situation shown in FIG. 6(A), for example, the inter-element relationship estimation unit 16 classifies the inter-element relationship between element ea and element eb as "same person," and the risk estimation unit 17 adds the inter-element relationship "same person" to the input of the risk estimation model in addition to element ea and element eb. This enables the risk estimation unit 17 to perform risk estimation in accordance with the situation shown in FIG. 6(A).
[0051] (5) Co-occurrence Model Next, a supplementary explanation will be given of the case where a co-occurrence model is used as a risk estimation model. Details of the co-occurrence model are disclosed in, for example, Patent Document 1.
[0052] FIG. 7 illustrates an example of a co-occurrence model. The nodes on the left side of the co-occurrence model list combinations of elements and inter-element relationships in the scene graph, hereafter referred to as "element-inter-element relationship t_ijk." Here, "i" (1≦i≦M) is an index indicating the class of an object (also referred to as an "entity class"), "j" (1≦j≦N) is an index indicating the class of the inter-object relationship and inter-element relationship (also referred to as a "relationship class"), and "k" (1≦k≦R) is an index indicating the class of the person (also referred to as an "entity or attribute class"). The inter-object relationship and inter-element relationship classes included in the co-occurrence model are relationships observed in the observation data (e.g., images) used as input for the co-occurrence model in the training data used to learn the co-occurrence model. The nodes on the right side of the co-occurrence model list the classes of expected risks. "l" (1≦l≦L) indicates the index of the expected risk class, and "rl" is data related to the risk of index l included in the training data for the co-occurrence model. The risk rl also includes a normal state.
[0053] The co-occurrence model defines the weighting factor "W_l[ijk]" representing the co-occurrence relationship between the element / inter-element relationship t_ijk and the risk rl using normalized self-mutual information. By using normalized self-mutual information, the possible values of the weighting factor W_l[ijk] can be set to a range from -1 to 1. The co-occurrence model is trained in advance using training data, and the weighting factor W_l[ijk] between the element / inter-element relationship t_ijk and the risk rl is determined. The training data can be prepared by generating a scene graph from observation data and manually labeling the risks occurring in the space observed by the observation data.
[0054] The risk estimation unit 17 performs label propagation and logistic regression. The risk estimation unit 17 performs label propagation using a scene graph as a seed. Label propagation is a message passing method. Label propagation is a semi-supervised learning method that trains a model by propagating data labels to related labels. In label propagation, label propagation of the relationship score of the element-element relationship t_ijk is performed. The relationship score is a score regarding the possibility that the element-element relationship t_ijk exists, and its initial value is "1" for the element-element relationship t_ijk identified by the scene graph generation unit 15 and the inter-element relationship estimation unit 16, and "0" for the element-element relationship t_ijk that has not been identified. When label propagation of the relationship score is performed, the relationship score for the element-element relationship t_ijk that has not been estimated becomes a value greater than "0". By performing label propagation, even an element-to-element relationship t_ijk that was not identified by the scene graph generation unit 15 and the inter-element relationship estimation unit 16 can be inferred to exist due to propagation from other element-to-element relationships t_ijk, and conversely, even an element-to-element relationship t_ijk that was identified by the scene graph generation unit 15 and the inter-element relationship estimation unit 16 can be inferred to be a false detection due to propagation from other element-to-element relationships t_ijk. By performing label propagation in this way, feature quantities such as a scene graph (graph structure) are updated.
[0055] The risk estimation unit 17 determines whether the value of the relationship score after label propagation is equal to or greater than a predetermined value, and filters out element-to-element relationships t_ijk having a relationship score equal to or greater than a predetermined threshold. The risk estimation unit 17 then calculates a risk score based on the filtered element-to-element relationships t_ijk using a formula for calculating a risk score. For example, formula (6) disclosed in Patent Document 1 is used as the above formula.
[0056] (6) Processing Flow Fig. 8 is an example of a flowchart relating to the processing executed by the risk estimation device 1. The risk estimation device 1 executes the flowchart shown in Fig. 8 when it is time to estimate a risk.
[0057] First, the risk estimation device 1 acquires observation data output by the observation device 5 (step S11). In this case, the risk estimation device 1 may store the observation data acquired from the observation device 5 in the storage device 4, memory 12, etc., and when the time to estimate the risk arrives, acquire the observation data to be used for risk estimation from the storage device 4, memory 12, etc. The observation data to be used for risk estimation may be the latest observation data among the observation data generated by the observation device 5 at a predetermined cycle, or may be the latest observation data for a predetermined length of time.
[0058] Next, the risk estimation device 1 detects objects in the monitored space based on the observation data acquired in step S11 and the object detection model information D1 (step S12).The risk estimation device 1 then generates a scene graph showing the relationships between people and objects included in the objects detected in step S12 (step S13).Furthermore, the risk estimation device 1 estimates the relationships between elements in the scene graph (step S14).
[0059] The risk estimation device 1 then estimates the risk in the monitored space based on the scene graph generated in step S13, the inter-element relationships in the scene graph estimated in step S14, and the risk estimation model information D2 (step S15). The risk estimation device 1 then outputs information relating to the estimation result of the risk estimated in step S15 (step S16). In this case, the risk estimation device 1 may display or output the information relating to the estimation result of the risk by sound using the output device 3, transmit it to another device, or store it in the storage device 4 or memory 12.
[0060] (6) Modifications Modifications of the above-described embodiment will now be described. The following modifications may be applied to the above-described embodiment in any combination.
[0061] (Variation 1) The risk estimation device 1 may predict the state of an object after a predetermined time has elapsed based on observation data, and perform risk estimation based on the prediction results of the scene graph and inter-element relationships after the predetermined time has elapsed.
[0062] 9 shows an example of functional blocks included in the processor 11 of the risk estimation device 1 in this modified example. Functionally, the processor 11 of the risk estimation device 1 in this modified example includes an object detection unit 14, a scene graph generation unit 15, an inter-element relationship estimation unit 16, a risk estimation unit 17, an output control unit 18, and a prediction unit 19. The object detection unit 14 and the scene graph generation unit 15 detect objects and generate scene graphs, as described in FIG. 3.
[0063] The prediction unit 19 predicts the state of each object after a predetermined time has elapsed. In this case, for example, the prediction unit 19 predicts the state of each object (including position, posture, relationships between objects, etc.) after a predetermined time has elapsed using any prediction technology (including human behavior prediction, etc.) based on time-series observation data, etc., acquired by the risk estimation device 1 from the observation device 5. The prediction unit 19 then updates the scene graph generated by the scene graph generation unit 15 based on the predicted states of each object after a predetermined time has elapsed, and supplies the updated scene graph to the risk estimation unit 17. The inter-element relationship estimation unit 16 estimates inter-element relationships after a predetermined time has elapsed from the states of each object after a predetermined time has elapsed predicted by the prediction unit 19. The inter-element relationship estimation unit 16 then supplies the estimated results of the inter-element relationships after a predetermined time has elapsed to the risk estimation unit 17.
[0064] The risk estimation unit 17 estimates the risk in the monitored space after a predetermined time has elapsed, based on the scene graph supplied from the prediction unit 19 (i.e., a scene graph reflecting the predicted state of each object after a predetermined time has elapsed) and the estimation result of the inter-element relationships after a predetermined time has elapsed supplied from the inter-element relationship estimation unit 16. In this case, the risk estimation unit 17 inputs data indicating the scene graph after a predetermined time has elapsed and the estimation result of the inter-element relationships into a risk estimation model, and obtains the risk estimation result output by the risk estimation model. The risk estimation unit 17 then supplies the risk estimation result to the output control unit 18.
[0065] In this way, the risk estimation device 1 in this modified example can estimate a risk that will occur within the monitored space after a predetermined time has elapsed.
[0066] (Modification 2) When the observation device 5 includes a sound input device such as a microphone, the object detection unit 14 may detect an object based on sound data generated by the sound input device.
[0067] In this case, for example, the object detection unit 14 detects a sound object having a predetermined pattern from the sound data acquired from the observation device 5. In this way, the sound detection result is also considered as a type of object. This makes it possible to take into account the power spectrum of environmental sounds in the monitored space and the presence or absence of collision sounds in risk estimation. Furthermore, the object detection unit 14 may identify the direction from which the sound is coming based on sound data generated by multiple audio input devices. This makes it possible to take into account the direction from which the collision sound or the like is coming, the location where the sound is being uttered, etc. in risk estimation.
[0068] Similarly, if the observation device 5 includes a sensor that detects data such as date, time, temperature, pressure (barometric pressure), and illuminance, the object detection unit 14 detects an object in the sensor data having a predetermined pattern from the collected sensor data acquired by the sensor using a sensor pattern detection model or the like. Parameters of the sensor pattern detection model and the like are stored in advance in the storage device 4 or the memory 12, etc. The object detection unit 14 may thus also regard the detection result of the sensor as a type of object.
[0069] <Second Embodiment> Figure 10 shows a schematic configuration of a risk estimation system 100A according to the second embodiment. The risk estimation system 100A according to the second embodiment has a risk estimation device 1A functioning as a server and a terminal device 8 functioning as a client. The risk estimation device 1A and the terminal device 8 communicate data via a network 99. Hereinafter, the same components as those in the first embodiment will be appropriately designated by the same reference numerals, and their description will be omitted.
[0070] The terminal device 8 is a terminal used by a user who performs operations within the monitored space, and has input functions, display functions, communication functions, and observation functions (such as an imaging function), and functions as the input device 2, output device 3, and observation device 5 shown in Fig. 1. The terminal device 8 may be, for example, a personal computer, a tablet terminal such as a smartphone, a PDA (Personal Digital Assistant), etc. Note that an observation device separate from the terminal device 8 may be provided in the risk estimation system 100A and supply observation data to the risk estimation device 1A.
[0071] The risk estimation device 1A has the same hardware configuration as the hardware configuration of the risk estimation device 1 shown in Fig. 2, and the processor 11 of the risk estimation device 1A has the functional blocks shown in Fig. 3 or 9 described in the first embodiment. The risk estimation device 1A receives observation data from the terminal device 8 or an observation device separate from the terminal device 8 via a network 99, and executes risk estimation processing. The risk estimation device 1A also transmits an output signal for outputting information related to the risk estimation result to the terminal device 8 via the network 99.
[0072] In this way, the risk estimation device 1A in the second embodiment can suitably notify the user of the terminal device 8 of information relating to the estimation result of the risk in the monitored space.
[0073] Third Embodiment The third embodiment relates to a video monitoring system, as an example of application in the medical and healthcare fields, that aims to ensure the safety of people moving around in a hospital, which is an example of a medical site.
[0074] 11 shows a schematic configuration of a video monitoring system 100B according to the third embodiment. The video monitoring system 100B includes a risk estimation device 1B and one or more cameras 51 installed in the hospital. Hereinafter, the same components as those in the first or second embodiment will be appropriately designated by the same reference numerals, and their description will be omitted.
[0075] The camera 51 generates time-series images (video data) of the inside of the hospital and supplies the generated video data to the risk estimation device 1 B. For example, the camera 51 is installed in each room, corridor, elevator, etc. in the hospital and acquires video data of the inside of the hospital.
[0076] The risk estimation device 1B has, for example, the hardware configuration shown in FIG. 2 and the functional block configuration shown in FIG. 3 or FIG. 9. The risk estimation device 1B regards the video data acquired from the camera 51 as the observation data in the first or second embodiment and estimates risks (e.g., dangerous behaviors) related to patients, hospital staff (including medical professionals), visitors, etc. from the video data in the same manner as in the first or second embodiment. In this case, the risk estimation device 1B may perform risk estimation based on the video data acquired from the camera 51 in real time, or may store the video data acquired from the camera 51 in a memory or the like accessible by the risk estimation device 1B and perform risk estimation using all of the stored video data at a predetermined timing. Note that the details of risk estimation in the third embodiment are the same as the processing of the object detection unit 14, scene graph generation unit 15, inter-element relationship estimation unit 16, and risk estimation unit 17 described in the first embodiment.
[0077] When the risk estimation device 1B detects a risk with a certainty level equal to or higher than a predetermined threshold, it outputs a warning (alert) to notify hospital staff of the occurrence of the risk. The risk estimation device 1B may function as the risk estimation device 1 in the first embodiment, or as the risk estimation device 1A in the second embodiment. For example, when the risk estimation device 1B functions as the risk estimation device 1 in the first embodiment, the risk estimation device 1B has an input device 2, an output device 3, and a storage device 4 built in or connected to these, and outputs a warning to the output device 3 when a risk is detected. Furthermore, when the risk estimation device 1B functions as the risk estimation device 1A in the second embodiment, for example, the risk estimation device 1B may be located outside the hospital, and when a risk is detected, it may output a warning to a terminal device 8 (not shown) used by hospital staff within the hospital.
[0078] The risk estimation device 1B may output a warning only to hospital staff or specific targets, or may control broadcasting equipment within the hospital to output a warning that can be heard by an unspecified number of people via in-hospital broadcasting or the like.
[0079] Instead of outputting a warning when a risk is detected, the risk estimation device 1B may generate a report indicating the detected risk and output the generated report at a predetermined timing. In this case, the risk estimation device 1B may transmit the report indicating the detected risk to another device or may store it in a memory or the like. For example, the risk estimation device 1B may generate a message to the subject indicating the detected risk and notify the subject of the message on a device owned by the subject. The processing related to the output of such warnings, reports, etc. corresponds to the processing of the output control unit 18 described in the first embodiment.
[0080] Here, examples of the "person" class in this embodiment include patients, medical staff (including medical professionals), visitors, etc. Furthermore, examples of the "object" class in this embodiment include medical equipment, intravenous drips, wagons, wheelchairs, etc. Furthermore, examples of the "risk" class in this embodiment include a situation in which a patient is about to touch a specific medical equipment, a situation in which a patient with an intravenous drip and a visitor pushing a wheelchair are about to collide, a situation in which a person with a white cane and a patient with an intravenous drip are about to collide, etc.
[0081] The risk estimation device 1B may perform risk estimation in real time based on video data acquired from the camera 51 and output a warning regarding the detected risk. The video data acquired from the camera 51 may be accumulated in a memory or the like, and risk estimation may be performed collectively based on the accumulated video data and output as a report.
[0082] Here, we will provide a supplementary explanation of the process that follows after issuing a warning about a detected risk. When hospital staff receive a warning about the occurrence of a risk, they confirm the detected risk and the abnormality that caused the risk based on the information output as a warning by the risk estimation device 1, and take appropriate countermeasures. Examples of countermeasures include rushing to the location where the abnormality occurred, providing appropriate care to the patient, and rearranging medical equipment.
[0083] Here, the risk estimation device 1B may determine a countermeasure according to the detected risk and notify the hospital staff of the determined countermeasure. In this case, for example, the risk estimation device 1B determines the countermeasure to be taken based on the content and level of the estimated risk using rule information that indicates the correspondence between the combination of the content and level of the expected risk and the countermeasure to be determined. The "content and level of the risk" corresponds to, for example, the class of risk indicated by the risk estimation result and its certainty. The above-mentioned rule information is stored in advance in a memory or the like that can be referenced by the risk estimation device 1B.
[0084] Furthermore, instead of rule information, the risk estimation device 1B may determine the countermeasure to be taken using a machine learning model that has learned the relationship between the content and level of risk and the countermeasure to be taken. In this case, the machine learning model is a model that has been trained to output the countermeasure to be taken when data indicating the content and level of risk is input. This machine learning model is, for example, a machine learning model (including a statistical model; the same applies hereinafter) having any architecture such as a neural network or a support vector machine. Furthermore, the trained parameters of the machine learning model are stored in advance in a memory or the like that can be referenced by the risk estimation device 1B.
[0085] The video data from the camera 51 acquired by the video monitoring system 100B may be transmitted from the camera 51 or the risk estimation device 1B to a server device (not shown), stored therein, and analyzed as necessary. Such analysis can be useful for developing algorithms (including machine learning models) for more effective risk estimation, predicting risk occurrence, and the like.
[0086] Here, a supplementary explanation will be given of the effects of the third embodiment. According to the third embodiment, by detecting risks such as abnormal behavior using video recognition technology, it is possible to detect abnormalities that pose risks more quickly than with conventional manual monitoring, and accidents and problems can be prevented before they occur. This ensures the safety of patients in the hospital and reduces the burden on hospital staff.
[0087] Furthermore, the video monitoring system 100B in the third embodiment can optimize the behavior of hospital staff by suggesting ways to deal with risks when they occur. Furthermore, by suggesting such ways of dealing with risks, the video monitoring system 100B can support the decision-making of hospital staff.
[0088] Furthermore, the video surveillance system 100B in the third embodiment can accumulate and analyze video data generated by the camera 51, which can be used to develop algorithms for more effective risk estimation and predict risk occurrence.
[0089] 12 is a block diagram of a risk estimation device 1X according to the fourth embodiment. The risk estimation device 1X mainly includes an inter-object relation estimation means 15X, an inter-element relation estimation means 16X, and a risk estimation means 17X. Note that the risk estimation device 1X may be configured by multiple devices.
[0090] The inter-object relationship estimation means 15X estimates the relationship between objects detected from the observation data. The inter-object relationship estimation means 15X can be, for example, the scene graph generation unit 15 in the first to third embodiments.
[0091] The inter-element relationship estimation means 16X estimates the relationships between objects and the inter-element relationships when the objects having the relationships are elements. The inter-element relationship estimation means 16X can be, for example, the inter-element relationship estimation unit 16 in the first to third embodiments.
[0092] The risk estimation means 17X estimates risks in the space observed by the observation data based on the elements and the relationships between the elements. The risk estimation means 17X can be, for example, the risk estimation unit 17 in the first to third embodiments.
[0093] 13 is an example of a flowchart executed by the risk estimation device 1X in the fourth embodiment. First, the inter-object relationship estimation means 15X estimates the relationships between objects detected from observation data (step S21). The inter-element relationship estimation means 16X estimates the relationships between objects and the relationships between elements when the objects having the relationships are considered as elements (step S22). The risk estimation means 17X estimates the risk in the space observed from the observation data based on the elements and the relationships between the elements (step S23).
[0094] According to the fourth embodiment, the risk estimation device 1X can accurately estimate the risk in the space observed from the observation data.
[0095] In the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can be supplied to a computer via wired communication paths such as electric wires and optical fibers, or via wireless communication paths.
[0096] In addition, some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.
[0097] [Supplementary Note 1] A risk estimation device comprising: an inter-object relation estimation means for estimating a relation between objects detected from observation data; an inter-element relation estimation means for estimating a relation between the objects and a relation between the elements when the objects having the relation are elements; and a risk estimation means for estimating a risk in a space observed from the observation data based on the elements and the relation between the elements. [Supplementary Note 2] The risk estimation device according to Supplementary Note 1, wherein the inter-object relation estimation means generates a scene graph representing the relation between the objects, and the risk estimation means estimates the risk based on the scene graph and the relation between the elements. [Supplementary Note 3] The risk estimation device according to Supplementary Note 1, wherein the inter-element relation estimation means estimates a relation related to a distance between the elements as the relation between the elements. [Supplementary Note 4] The risk estimation device according to Supplementary Note 1, wherein the inter-object relation estimation means estimates a relationship between a person included in the object and an object other than the person included in the object as the relation between the objects. [Supplementary Note 5] The risk estimation device according to Supplementary Note 4, wherein the inter-element relationship estimation means estimates whether the person is common between the elements as the relationship between the elements. [Supplementary Note 6] The risk estimation device according to Supplementary Note 1, further comprising prediction means for predicting a state of the object, wherein the risk estimation means estimates the risk based on the elements determined based on the predicted state and the predicted relationship between the elements. [Supplementary Note 7] The risk estimation device according to Supplementary Note 1, wherein the risk estimation means estimates the risk based on the elements and the relationships between the elements and a machine learning model, wherein the machine learning model is a model trained to output an estimation result of the risk when data indicating the elements and the relationships between the elements is input. [Supplementary Note 8] The risk estimation device according to Supplementary Note 1, further comprising output control means for outputting at least one of information regarding the occurrence of the risk or a countermeasure for dealing with the risk based on the estimated risk.[Supplementary Note 9] The risk estimation device according to Supplementary Note 7, wherein the space is a medical site, the countermeasures are countermeasures that optimize the behavior of medical professionals working at the medical site, and the output control means outputs to notify the medical professional of the countermeasures. [Supplementary Note 10] The risk estimation device according to Supplementary Note 7, wherein the output control means decides the countermeasures based on the estimated risk and a machine learning model, and the machine learning model is a model that has learned the relationship between the estimated risk and the countermeasures. [Supplementary Note 11] A risk estimation method, wherein a computer estimates relationships between objects detected from observation data, estimates the relationships between the objects and the relationships between the elements when the relationships between the objects and the objects having the relationships are elements, and estimates a risk in a space observed from the observation data based on the elements and the relationships between the elements. [Supplementary Note 12] A storage medium storing a program that causes a computer to execute a process of estimating relationships between objects detected by observation data, estimating relationships between the objects and relationships between elements when the objects having the relationships are used as elements, and estimating risks in a space observed by the observation data based on the elements and the relationships between the elements.
[0098] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference.
[0099] 1, 1A, 1B, 1X Risk estimation device 2 Input device 3 Output device 4 Storage device 5 Observation device 8 Terminal device 11 Processor 12 Memory 13 Interface 51 Camera 90 Data bus 99 Network 100, 100A Risk estimation system 100B Video monitoring system
Claims
1. A risk estimation device having an inter-object relationship estimation means for estimating relationships between objects detected from observation data, an inter-element relationship estimation means for estimating the relationships between the objects and the relationships between the elements when the objects having the relationships are treated as elements, and a risk estimation means for estimating risks in a space observed from the observation data based on the elements and the relationships between the elements.
2. The risk estimation device according to claim 1, wherein the inter-object relationship estimation means generates a scene graph representing the relationships between the objects, and the risk estimation means estimates the risk based on the scene graph and the relationships between the elements.
3. A risk estimation device according to claim 1, wherein said inter-element relationship estimation means estimates a relationship regarding a distance between said elements as the relationship between said elements.
4. A risk estimation device according to claim 1, wherein the inter-object relationship estimation means estimates a relationship between a person included in the object and an object other than the person included in the object as the relationship between the objects.
5. A risk estimation device according to claim 4, wherein said inter-element relationship estimation means estimates, as the relationship between said elements, whether or not said person is common between said elements.
6. A risk estimation device as described in claim 1, further comprising a prediction means for predicting a state of the object, wherein the risk estimation means estimates the risk based on the elements determined based on the predicted state and the predicted relationships between the elements.
7. The risk estimation device of claim 1, wherein the risk estimation means estimates the risk based on the elements and the relationships between the elements and a machine learning model, and the machine learning model is a model trained to output an estimation result of the risk when data indicating the elements and the relationships between the elements is input.
8. A risk estimation device as claimed in claim 1, further comprising an output control means for outputting at least one of information relating to the occurrence of said risk or a countermeasure for dealing with said risk based on said estimated risk.
9. The risk estimation device as described in claim 8, wherein the space is a medical site, the countermeasure is a countermeasure that optimizes the actions of medical personnel working in the medical site, and the output control means outputs to inform the medical personnel of the countermeasure.
10. A risk estimation device as described in claim 8, wherein the output control means determines the countermeasure based on the estimated risk and a machine learning model, and the machine learning model is a model that has learned the relationship between the estimated risk and the countermeasure.
11. A risk estimation method in which a computer estimates relationships between objects detected by observation data, estimates the relationships between the objects and the relationships between the elements when the objects having the relationships are considered as elements, and estimates risks in a space observed by the observation data based on the elements and the relationships between the elements.
12. A storage medium storing a program that causes a computer to execute a process of estimating relationships between objects detected by observation data, estimating the relationships between said objects and the relationships between said elements when the objects having said relationships are considered as elements, and estimating risks in the space observed by the observation data based on said elements and the relationships between said elements.
Citation Information
Patent Citations
Image monitoring system
JP2017028561A
Disease information management system, disease information management server, disease information management method and disease information management program
JP2021135623A
Monitoring device, monitoring method, and program
JP2022170790A
