Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to appropriately detect a motion section of a person. An information processing device according to one aspect of the present technology generates, on the basis of a measurement result from a sensor, a time series of a scene graph, which is graph structure information in which a plurality of first nodes respectively indicating a plurality of objects are connected by a first edge indicating a relationship between the objects, and, when the relationship between the objects is changed by a motion of a person, generates an event scene graph, which is graph structure information in which a time when the change in the relationship between the objects has been measured is included and second nodes respectively indicating the objects between which the relationship has changed are connected by a second edge indicating content of the change in the relationship. The present technology can be applied to a device that recognizes a behavior of a person.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and particularly to an information processing device, an information processing method, and a program that are capable of appropriately detecting a movement section of a person.

[0002] Human behavior recognition technology is attracting attention because of its many applications, such as analyzing surveillance camera footage, sports footage, and robot interaction. Behavior recognition is a technology that recognizes what a person is doing. For example, behavior recognition is performed using an inference model generated by machine learning.

[0003] In conventional behavior recognition, for example, motion data such as skeleton point information acquired by analyzing a short video of about a few seconds is used as input for an inference model. Therefore, when performing behavior recognition, it is necessary to detect and pre-divide sections of a long video continuously captured by a camera where a person performs a motion as sections to be analyzed for the motion data.

[0004] Various techniques have been proposed called motion segment detection, which estimates the start and end frames of a person's motion and extracts those segments before performing action recognition based on motion data. For example, there is a technique (Patent Document 1) that detects the feature values ​​of the joint movements of a person from motion capture data and detects the frames where the feature values ​​are minimal as the start and end frames of the motion, and a technique (Non-Patent Document 1) that inputs the optical flow of a video of a person into a neural network and detects motion segments.

[0005] JP 2011-227792 A

[0006] "Bsn: Boundary sensitive network for temporal action proposal generation.", Shanghai Jiao Tong University, Lin et al. "Slow motion matters: A slow motion enhanced network for weakly supervised temporal action localization.", Beihang University, Sun et al.

[0007] There are some movements that are difficult to detect using conventional methods that use joint movement features or optical flow. For example, in slow movements, it is not possible to obtain sufficient features to properly detect movement segments.

[0008] The present technology has been made in view of such circumstances, and makes it possible to appropriately detect a person's movement intervals.

[0009] An information processing device according to one aspect of the present technology includes an information processing unit that generates, based on measurement results from a sensor, a time series of a scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of objects are connected by a first edge indicating a relationship between the objects, and when a relationship between the objects changes due to a person's actions, generates an event scene graph, which is graph-structured information in which second nodes representing each of the objects whose relationship has changed are connected by a second edge indicating the content of the change in the relationship, and which includes the time at which the change in the relationship between the objects was measured.

[0010] An information processing device according to another aspect of the present technology acquires a plurality of subgraphs generated for each action of a person as graph structure information in which inter-object relationship nodes indicating the relationships between a plurality of first objects before and after a change, respectively, are connected by relationship change edges indicating the time at which the change in the relationship between the first objects was measured and the direction of the change, detects changes in the relationships between a plurality of second objects used by the person being measured based on measurement results by a sensor, searches for the subgraph of a predetermined action that indicates a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects, and detects a section in which the person being measured performed the same action as the predetermined action.

[0011] In one aspect of the present technology, a time series of a scene graph is generated based on the measurement results from a sensor, which is graph-structured information in which a plurality of first nodes each representing a plurality of objects are connected by a first edge indicating the relationship between the objects, and when the relationship between the objects changes due to the actions of a person, an event scene graph is generated which is graph-structured information in which second nodes representing each object whose relationship has changed are connected by a second edge indicating the content of the change in the relationship and which includes the time when the change in the relationship between the objects was measured.

[0012] In another aspect of the present technology, a plurality of subgraphs are acquired for each action of a person as graph structure information in which inter-object relationship nodes indicating the relationships between a plurality of first objects before and after the change, respectively, are connected by relationship change edges indicating the time at which the change in the relationship between the first objects was measured and the direction of the change, and changes in the relationships between a plurality of second objects used by the person being measured are detected based on the measurement results by a sensor, a subgraph of a predetermined action indicating changes in the relationships between the first objects that are the same as or similar to the changes in the relationships between the second objects is searched for, and a section in which the person being measured performed the same action as the predetermined action is detected.

[0013] 19 is a diagram showing an example of a configuration of an information processing system according to an embodiment of the present technology. FIG. 19 is a block diagram showing an example of a functional configuration of a motion interval structuring information recording unit. FIG. 19 is a diagram showing an example of a scene graph. FIG. 19 is a flowchart illustrating motion interval recording processing. FIG. 19 is a flowchart illustrating an event scene graph generation processing performed in step S2 of FIG. 4. FIG. 19 is a diagram showing an example of a scene graph. FIG. 19 is a diagram showing an example of generation of a scene graph. FIG. 19 is a diagram showing an example of generation of an event scene graph. FIG. 19 is a diagram showing an example of a user's motion represented by the event scene graph. FIG. 19 is a flowchart illustrating a state transition graph generation processing performed in step S3 of FIG. 4. FIG. 19 is a diagram showing an example of a state transition graph. FIG. 19 is a flowchart illustrating a connection processing of directed edges. FIG. 19 is a diagram showing an example of nodes in a state transition graph. FIG. 19 is a diagram showing an example of update of a state transition graph. FIG. 19 is a diagram showing an example of motion interval structuring information. FIG. 19 is a diagram showing an example of motion interval structuring information. FIG. 19 is a block diagram showing an example of a functional configuration of a motion interval detection unit. FIG. 19 is a flowchart illustrating motion interval detection processing. FIG. 19 is a flowchart illustrating graph search processing performed in step S104 of FIG. FIG. 1 is a diagram illustrating another example of the configuration of an information processing system. FIG. 2 is a diagram illustrating an example of differences in environments. FIG. 3 is a diagram illustrating an example of node weights. FIG. 4 is a diagram illustrating an example of node weight settings. FIG. 5 is a diagram illustrating an example of a scene graph. FIG. 6 is a diagram illustrating an example of recording an action section. FIG. 7 is a diagram illustrating a flow of generating a subgraph. FIG. 8 is a block diagram illustrating an example of the configuration of a computer.

[0014] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Activity section recording phase 3. Activity section detection phase 4. Configuration example of information processing device 5. Modification

[0015] <<Overview of this technology>> This technology is related to movement segment detection, which enables accurate detection of human movement segments performed at various speeds and scales with a small amount of computation. With this technology, difference information (event scene graph) that represents changes in the relationships between objects present around the person whose movement segment is being measured is generated, and movement segments are detected using graph-structured information that represents state transitions of the relationships between objects, which is generated based on the difference information.

[0016] This technology achieves the following, for example: 1. It makes it possible to detect sections of slow or small movements that are difficult to determine from a person's joint information. 2. Because processing is performed using lightweight data that represents the difference in changes in the relationships between objects, it makes it possible to reduce the amount of calculation and memory used for detecting movement sections. It also makes it possible to reduce the storage capacity of the data used for detecting movement sections.

[0017] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present technology.

[0018] 1, the information processing system 1 includes a measuring device 11 and an information processing device 12. The measuring device 11 and the information processing device 12 are connected via wired or wireless communication.

[0019] The measurement device 11 is a sensor device equipped with various sensors such as a color image sensor and a depth sensor. The measurement device 11 is installed in a room where a real person is present. In the example of FIG. 1 , the measurement device 11 is installed in a space where a person H is present. The person H to be measured lives his or her daily life by eating meals, taking medicine, brushing his or her teeth, etc.

[0020] The measuring device 11 outputs measurement data such as color images and distance images to the information processing device 12. The color images output by the measuring device 11 show not only the person H performing various actions, but also objects around the person H. The objects around the person H include objects used by the person H when performing actions. The distance images measured by the measuring device 11 show the distance to the person and the distance to the objects.

[0021] The information processing device 12 performs a movement segment recording process and a movement segment detection process based on the image data supplied as measurement data from the measuring device 11. For example, the movement segment recording process is repeatedly performed, and movement segment structuring information for various movements is recorded, and then the movement segment detection process is performed. The movement segment structuring information is information that represents, in a structured manner, the state transitions of objects that appear when person H performs a specific movement.

[0022] The motion section recording process mainly involves the following steps (1) to (4), each of which will be described in detail later.

[0023] (1) Obtaining a scene graph: A scene graph is information with a graph structure in which each object is represented by a node and the relationships between the objects are represented by edges. A method for obtaining a scene graph involves detecting human and object regions from the images of each frame that constitutes a video and obtaining the semantic positional relationships between them. (2) Obtaining scene graph difference information (time-series information): Scene graph difference information is obtained when the scene graph changes. The difference information is also graph-structured information composed of nodes and edges. A timestamp is set for the difference information obtained at each timing as time-series information. (3) Converting to a state transition structure (structural information): Based on the scene graph difference information, a state transition graph is generated that represents the transitions in the relationships between objects together with timestamps. The scene graph difference information, which is time-series information, is converted into information that structuredly represents the transitions in the relationships between objects. (4) Recording action interval structure information: From the state transition graph, a subgraph corresponding to the section in which a person performed a specific action is extracted as action interval structure information. Movement interval structuring information for various movements is generated, and a movement interval structuring information DB, which is a DB of the movement interval structuring information, is generated.

[0024] Meanwhile, as the operation interval detection process, the following process (5) is performed following the processes (1) to (3) above. (5) Operation interval detection process: Operation interval structured information (subgraph) having the same or similar graph structure as the state transition graph generated during operation interval detection is searched from the operation interval structured information recorded in the operation interval structured information DB. If operation interval structured information having the same or similar graph structure is found, the start time and end time added as timestamps to the state transition graph generated during operation interval detection are output.

[0025] The start time and end time output by the movement segment detection process represent the start time and end time of a certain movement of person H at the time of movement segment detection. The movement segment represented by the start time and end time output by the movement segment detection process is used for various processes such as behavior recognition.

[0026] 1, the movement section recording process and the movement section detection process are performed in the same space, but they may be performed in different spaces. When the movement section recording process and the movement section detection process are performed in different spaces, a movement section recording configuration and a movement section detection configuration are provided separately.

[0027] 1, the person to be measured when recording a movement segment and the person to be measured when detecting a movement segment are the same person H, but the measurement targets may be different people. The movement segment recording process may be performed using various people as measurement targets, and movement segment structuring information for various movements may be generated.

[0028] Each phase of the movement section recording process and the movement section detection process will be described below.

[0029] <<Movement Section Recording Phase>> <Configuration of the Movement Section Recording Side> FIG. 2 is a block diagram showing an example of the functional configuration of the movement section structured information recording unit 101. As shown in FIG.

[0030] 2, the motion section structuring information recording unit 101 is made up of a data acquisition unit 111, a scene graph difference generation unit 112, a state transition structure generation unit 113, and a recording unit 114. The motion section structuring information recording unit 101 functions as an information processing unit that performs motion section recording processing.

[0031] The data acquisition unit 111 acquires image data output from the measurement device 11 and outputs the image data to the scene graph difference generation unit 112. The scene graph difference generation unit 112 is sequentially supplied with image data measured at each time point for the person who will be the user.

[0032] The scene graph difference generation unit 112 generates a 3D spatial recognition map based on the image data of each frame supplied from the data acquisition unit 111. Specifically, the scene graph difference generation unit 112 performs segmentation (semantic segmentation and instance segmentation) such as PanopticFusion on the RGB image (2D image) supplied from the data acquisition unit 111, and recognizes the label for each pixel. The scene graph difference generation unit 112 integrates the label into a 3D spatial map generated by SLAM using the depth image, and generates a 3D spatial recognition map. PanopticFusion, one method of panoptic segmentation, is described in, for example, Literature 1. Literature 1: Gaku Narita, et al., “PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things”, Proc. IROS, 2019

[0033] The scene graph difference generator 112 generates a scene graph that represents the relationships between objects in space based on the 3D space recognition map.

[0034] FIG. 3 is a diagram showing an example of a scene graph.

[0035] If a user's environment contains a sofa, a table, a television, chair A, and chair B, the scene graph representing the user's environment will contain five nodes representing these objects, as shown in FIG.

[0036] In the example of Figure 3, the sofa node and the television node are connected by edge E1, which has the label "in_front_of". The label of edge E1 indicates that the sofa is in front of the television. The sofa node and the table node are connected by edge E2, which has the label "on_right". The label of edge E2 indicates that the table is on the right side of the sofa.

[0037] The television node and the table node are connected by edge E3 labeled "on_left." The label for edge E3 indicates that the table is on the left side of the television. The table node and chair A node, and the table node and chair B node are also connected by edges E4 and E5, which have labels set to indicate their respective positional relationships. Labels that indicate spatial positional relationships (front / behind / left / right / on / above / under / near, etc.) are used as labels set for edges.

[0038] In this way, a scene graph is information with a graph structure in which multiple objects, such as furniture, that exist in the environment are represented as nodes, and the relationship between two objects, such as their positional relationship, is represented by edges. The generation of a scene graph is described, for example, in Reference 2. Reference 2: Tomu Tahara, et al., "Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph," 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020

[0039] 2 generates a time series of a scene graph having such a graph structure based on the image data of each frame supplied from the data acquisition unit 111. The scene graph difference generation unit 112 generates an event scene graph, which is difference information of the scene graph, and outputs it to the state transition structure generation unit 113. For example, a change occurs in the scene graph when a user performs an action using an object in space. An event scene graph is generated each time a change occurs in the scene graph.

[0040] The state transition structure generating unit 113 generates a state transition graph based on the event scene graph supplied from the scene graph difference generating unit 112 and outputs it to the recording unit 114 .

[0041] The recording unit 114 extracts, from the state transition graph generated by the state transition structure generating unit 113, a subgraph corresponding to a section where a person performed a specific action as action section structuring information, and records the extracted information in a action section structuring information DB. Annotation information specifying an action section is input to the recording unit 114 as appropriate. The annotation information is input, for example, by an administrator who manages the recording of action section structuring information operating the information processing device 12.

[0042] <Movement Section Recording Process> FIG. 4 is a flowchart illustrating a series of processes when recording a movement section.

[0043] In step S1 , the data acquisition unit 111 acquires image data output from the measuring device 11 .

[0044] In step S2, an event scene graph generation process is performed, the details of which will be described later with reference to the flowchart of FIG.

[0045] In step S3, a state transition graph generation process is performed, which will be described in detail later with reference to the flowchart of FIG.

[0046] In step S4, the recording unit 114 records the subgraph of the movement interval as movement interval structuring information.

[0047] By repeating the above process, movement segment structure information for various movements is recorded in the DB.

[0048] Event Scene Graph Generation Processing Fig. 5 is a flowchart illustrating the event scene graph generation processing performed in step S2 of Fig. 4. The processing shown in Fig. 5 is performed each time image data for each frame is supplied from the data acquisition unit 111.

[0049] In step S11, the scene graph difference generation unit 112 performs panoptic fusion to generate a 3D space recognition map.

[0050] In step S12, the scene graph difference generation unit 112 generates a scene graph that represents the relationship between objects in space based on the 3D space recognition map.

[0051] FIG. 6 is a diagram showing an example of a scene graph.

[0052] When an RGB image showing a medicine and a cup placed on a table (desk) is captured and these objects are recognized, as shown on the left side of Figure 6, a scene graph consisting of nodes for the medicine, the cup, and the desk is generated, as indicated by the arrows. In the scene graph of Figure 6, the medicine node and the desk node are connected by an edge labeled "on," and the medicine node and the cup node are connected by an edge labeled "near." Furthermore, the cup node and the desk node are connected by an edge labeled "on." Such a scene graph is generated based on the image data of each frame.

[0053] Returning to the explanation of FIG. 5, in step S13, the scene graph difference generation unit 112 selects one relationship between objects represented by the generated scene graph.

[0054] In step S14, the scene graph difference generation unit 112 determines whether or not the relationship between the selected objects has changed.

[0055] If it is determined in step S14 that the relationship between the objects has changed, then in step S15 the scene graph difference generator 112 generates an event scene graph including information on the objects whose relationship has changed, the relationships before and after the change, and timestamps. If it is determined in step S14 that the relationship between the objects has not changed, the processing of step S15 is skipped.

[0056] In step S16, the scene graph difference generation unit 112 determines whether or not all relationships between objects have been selected. If it is determined in step S16 that all relationships between objects have not been selected, the process returns to step S13, where the next relationship is selected, and the same processing is repeated.

[0057] If it is determined in step S16 that all the relationships between objects have been selected, the process returns to step S2 in FIG. 4, and the subsequent processes are carried out.

[0058] 7 and 8 are diagrams showing an example of generating an event scene graph.

[0059] In the example of Figure 7, a time series of scene graphs consisting of nodes for medicine, cup, and desk is generated based on RGB images captured at times t1 to t4. The scene graph at time t1 is the same as the scene graph described with reference to Figure 6. In the scene graph at time t2, the label of the edge between the medicine node and the desk node, i.e., the relationship between the medicine and the desk, has changed from "on" to "near."

[0060] Similarly, in the scene graph at time t3, the relationship between the medicine and the desk changes from "near" to "on," and in the scene graph at time t4, the relationship between the desk and the cup changes from "on" to "near." For example, if a user picks up a medicine placed on the desk at time t2 and drinks it, puts the medicine back on the desk at time t3, and then picks up the cup at time t4, scene graphs for times t1 to t4 are generated.

[0061] When such a scene graph is generated, as shown in the lower part of Figure 8, whenever the label of the edge between nodes changes, that is, whenever the relationship between objects changes, the difference between the scene graph before and after the change is generated as an event scene graph.

[0062] In the example of FIG. 8 , in response to a change in the relationship between the medicine and the desk from "on" to "near," an event scene graph #1 is generated, which represents the difference between the scene graph at time t1 and the scene graph at time t2. The event scene graph #1 is composed of a medicine node and a desk node, which are objects whose relationship has changed, and an "on to near" label is set on the edge between these nodes, indicating the relationship before and after the change. In addition, a timestamp such as "19:36" is added to the event scene graph #1. The event scene graph #1 is expressed in a format such as "desk-medicine: on to near, 19:36."

[0063] Similarly, in response to the change in the relationship between the medicine and the desk from "near" to "on," an event scene graph #2 is generated that represents the difference between the scene graph at time t2 and the scene graph at time t3. A label "near to on" is set to the edge between the medicine node and the desk node that make up the event scene graph #2. In addition, a timestamp of "19:38" is added to the event scene graph #2. The event scene graph #2 is expressed in a format such as "desk-medicine: near to on, 19:38."

[0064] In response to the change in the relationship between the cup and the desk from "on" to "near," an event scene graph #3 is generated that represents the difference between the scene graph at time t3 and the scene graph at time t4. The edge between the desk node and the cup node that make up the event scene graph #3 is labeled "on to near." In addition, a timestamp of "19:39" is added to the event scene graph #3. The event scene graph #3 is expressed in a format such as "cup-desk: on to near, 19:39."

[0065] Event scene graphs #1 to #3 represent the user's actions at each time, such as "picking up the medicine on the desk and taking it," "putting the medicine on the desk," and "picking up the glass on the desk and drinking water," as shown in Figure 9.

[0066] In this way, when a relationship between objects changes due to a person's action or the like, the event scene graph is generated as graph-structured information in which nodes indicating each object whose relationship has changed are connected by edges indicating the nature of the change in relationship. The event scene graph includes a timestamp indicating the time at which the change in relationship between the objects was measured. If a node constituting the scene graph is referred to as a first node, then a node constituting the event scene graph generated based on the scene graph is referred to as a second node. Furthermore, a node constituting the state transition graph is referred to as a third node. Similarly, if an edge constituting the scene graph is referred to as a first edge, then an edge constituting the event scene graph is referred to as a second edge, and a node constituting the state transition graph is referred to as a third edge.

[0067] An event scene graph is generated not only when the relationship between objects changes, but also when a node representing a new object is added to the scene graph. An event scene graph is generated whenever the graph structure of the scene graph changes.

[0068] State Transition Graph Generation Processing Fig. 10 is a flowchart illustrating the state transition graph generation processing performed in step S3 of Fig. 4. The processing shown in Fig. 10 is performed every time the scene graph difference generation unit 112 generates an event scene graph.

[0069] In step S21, the state transition structure generation unit 113 acquires the event scene graph generated by the scene graph difference generation unit 112.

[0070] In step S22, the state transition structure generation unit 113 converts the acquired event scene graph into a state transition graph.

[0071] FIG. 11 is a diagram illustrating an example of a state transition graph.

[0072] As shown in Figure 11, when an event scene graph #1 with a timestamp of "19:36" is generated, a state transition graph #11 is generated that structures the contents of the event scene graph #1. The state transition graph #11 is composed of a "desk-medicine: on" node, which indicates that the medicine is on the desk, and a "desk-medicine: near" node, which indicates that the desk and the medicine are close to each other. These nodes are connected by a directed edge from the "desk-medicine: on" node to the "desk-medicine: near" node. The timestamp of the event scene graph #1, i.e., the timestamp of "19:36", when the relationship between the desk and the medicine changed from "on" to "near", is added to the directed edge from the "desk-medicine: on" node to the "desk-medicine: near" node.

[0073] In this way, the state transition graph is composed of multiple nodes, which are inter-object relationship nodes, that indicate the relationship between two objects. The edges connecting the nodes of the state transition graph are relationship change edges that indicate the time when the relationship between the objects changed and the direction of the state change.

[0074] FIG. 12 is a flowchart illustrating a process of connecting directed edges that form a state transition graph.

[0075] In step S31, the state transition structure generation unit 113 generates a state transition structure for the node R before and Node R after When the above-described event scene graph #1 is generated, the node "desk-medicine:on" is generated as node R as shown in FIG. before The node "Desk-Medicine:near" is generated as node R after is generated as:

[0076] In step S32, the state transition structure generation unit 113 generates a state transition structure for the node R prev and Node R before Let R be a directed edge (R prev →R before ) and connect to Node R. prev As shown in FIG. beforeThe node indicating the previous state and the "Desk-Medicine: on" node are connected by a directed edge E0 pointing from the node indicating the previous state to the "Desk-Medicine: on" node.

[0077] In step S33, the state transition structure generation unit 113 before and Node R after Let R be a directed edge (R before →R after ) As shown in FIG. 13, the "Desk-Medicine: On" node and the "Desk-Medicine: Near" node are connected by a directed edge E1 pointing from the "Desk-Medicine: On" node to the "Desk-Medicine: Near" node. The timestamp of event scene graph #1, "19:36", is added to the directed edge E1.

[0078] In step S34, the state transition structure generation unit 113 prev The pointer of node R after This pointer is used to point to the current node R when the state transition graph is updated. after is the next node R prev This shows that

[0079] 14 and 15 are diagrams showing an example of updating the state transition graph. In the following, the node R in FIG. prev will be omitted for explanation.

[0080] For example, when an event scene graph #2 is generated with a timestamp of "19:38" added, as shown in the upper right corner of Figure 14, state transition graph #11 is updated to state transition graph #12 based on event scene graph #2.

[0081] In the state transition graph #12, the node “desk-medicine:near” is node R before The node "Desk-Medicine:on" is node R after The node for "desk-medicine:near" is node R prevAlso, the "Desk-Medicine: Near" node and the "Desk-Medicine: On" node are connected by a directed edge E2 pointing from the "Desk-Medicine: Near" node to the "Desk-Medicine: On" node. The timestamp of event scene graph #2, "19:38", is added to the directed edge E2.

[0082] Similarly, when an event scene graph #3 is generated with a timestamp of "19:39" as shown in the upper left of Figure 15, state transition graph #12 is updated to state transition graph #13 based on event scene graph #3.

[0083] In the state transition graph #13, the nodes "desk-cup: on" and "desk-cup: near" are added. The "desk-cup: on" node is node R. before The node "desk-cup:near" is node R. after The node "Desk-Medicine:on" is node R prev The "Desk-Medicine: On" node and the "Desk-Cup: On" node are connected by a directed edge E3 pointing from the "Desk-Medicine: On" node to the "Desk-Cup: On" node. The timestamp of event scene graph #3, "19:39", is added to the directed edge E3.

[0084] Node R of "Desk-Medicine:on" prev is a node indicating the relationship between the desk and the medicine as the first kind of object at the previous timing, and is a node R of "desk-cup: on" before is a node indicating a relationship between the desk and the cup as the second type of object before the change. In this way, the directed edge E3 is an edge connecting a node indicating a relationship between the first type of object after the change and a node indicating a relationship between the second type of object before the change.

[0085] The "desk-cup: on" node and the "desk-cup: near" node are connected by a directed edge E4 pointing from the "desk-cup: on" node to the "desk-cup: near" node. The timestamp of event scene graph #3, "19:39", is added to the directed edge E4.

[0086] When event scene graph #4 is generated with the timestamp "19:40" added, as shown in the upper right corner of Figure 15, state transition graph #13 is updated to state transition graph #14 based on event scene graph #4.

[0087] In the state transition graph #14, the node "desk-cup: near" is node R before The node "desk-cup: on" is node R after The node for "desk-cup:near" is node R prev Also, the "desk-cup: near" node and the "desk-cup: on" node are connected by a directed edge E5 pointing from the "desk-cup: near" node to the "desk-cup: on" node. The timestamp of event scene graph #4, "19:40", is added to the directed edge E5.

[0088] This process of updating the state transition graph is repeated each time an event scene graph is generated. When annotation information specifying a movement interval is input, the recording unit 114 extracts a subgraph of the state transition graph generated as described above, within the range specified by the annotation information, and records it as movement interval structuring information (step S4 in FIG. 4). The subgraph recorded as movement interval structuring information is information generated using the measurement results while the person being measured is performing a specific movement.

[0089] FIG. 16 is a diagram showing an example of the movement section structuring information.

[0090] For example, if the entire state transition graph #14 in Fig. 15 is designated as the action section of "take medicine," a subgraph of the entire state transition graph #14 is recorded in the DB as action section structured information, as shown by the box in Fig. 16. The numbers shown at each node in Fig. 16 indicate the state transitions between objects in the action of "take medicine." In the action of "take medicine," the states between objects transition in the order of "desk-medicine: on," "desk-medicine: near," "desk-medicine: on," "desk-cup: on," "desk-cup: near," and "desk-cup: on."

[0091] By repeating the above process, subgraphs of various actions are recorded in the DB, as shown in Fig. 17. In the example of Fig. 17, a subgraph of the action "taking medicine" and a subgraph of the action "brushing teeth" are recorded. The DB in which the subgraphs of various actions are recorded is used in the action section detection process.

[0092] <<Movement Section Detection Phase>> <Configuration of the Movement Section Detection Side> FIG. 18 is a block diagram showing an example of the functional configuration of the movement section detection unit 201. As shown in FIG.

[0093] 18, the movement section detection unit 201 is made up of a data acquisition unit 211, a scene graph difference generation unit 212, a state transition structure generation unit 213, and a search unit 214. The movement section structuring information DB generated by the movement section recording process is acquired and prepared in the movement section detection unit 201. The movement section detection unit 201 acquires multiple subgraphs generated as movement section structuring information for each movement, and functions as an information processing unit that detects movement sections.

[0094] The data acquisition unit 211, scene graph difference generation unit 212, and state transition structure generation unit 213 have the same functions as the data acquisition unit 111, scene graph difference generation unit 112, and state transition structure generation unit 113 in Figure 2, respectively. When the movement segment structuring information recording unit 101 and the movement segment detection unit 201 are realized in the same device, the data acquisition unit 111, scene graph difference generation unit 112, and state transition structure generation unit 113 may be used when detecting movement segments instead of the data acquisition unit 211, scene graph difference generation unit 212, and state transition structure generation unit 213. In the description of Figure 18, descriptions that overlap with the above description will be omitted as appropriate.

[0095] The data acquisition unit 211 acquires image data output from the measurement device 11 and outputs it to the scene graph difference generation unit 212. The scene graph difference generation unit 212 is sequentially supplied with image data at each time that shows a user who is the target of behavior recognition.

[0096] The scene graph difference generation unit 212 generates a 3D space recognition map based on the image data of each frame supplied from the data acquisition unit 211, and generates a scene graph that represents the relationships between objects in space. The scene graph difference generation unit 212 generates an event scene graph based on the time series of the scene graph, and outputs it to the state transition structure generation unit 213.

[0097] The state transition structure generation unit 213 generates a state transition graph based on the event scene graph supplied from the scene graph difference generation unit 212 and outputs it to the search unit 214. An event scene graph is generated, and the search unit 214 is supplied with the updated state transition graph each time the state transition graph is updated. The updated state transition graph indicates state transitions between objects according to the current movement of the user who is the target of behavior recognition. The state transition structure generation unit 213 detects changes in the relationships between objects used by the person to be measured based on the measurement results from the measurement device 11. If the object on the movement segment recording side is the first object, the object on the movement segment detection side is the second object.

[0098] The search unit 214 searches for a subgraph that is identical to or similar to the state transition graph generated by the state transition structure generation unit 213 from among the subgraphs recorded in the operation interval structuring information DB as operation interval structuring information. As an algorithm for searching for a subgraph, for example, a graph search algorithm such as the gBoost method is used. The similarity (distance) between the state transition graph generated by the state transition structure generation unit 213 and the subgraph of each operation interval structuring information is calculated, and a subgraph that satisfies the conditions is acquired. For example, a subgraph with the highest similarity or a subgraph with a similarity higher than a threshold value is acquired as a subgraph that satisfies the conditions.

[0099] When a subgraph that satisfies the conditions is acquired, the search unit 214 references the timestamps of the edges that make up the state transition graph, and detects the section from the first timestamp to the last timestamp as a movement section. The subgraph searched from among the subgraphs recorded in the movement section structured information DB becomes information that indicates a change in the relationship between objects on the movement section recording side that is the same as or similar to the change in the relationship between objects on the movement section detection side. Based on the subgraph of a predetermined movement acquired by such a search, the search unit 214 detects a section in which the person being measured performed the same movement as the predetermined movement.

[0100] <Movement Section Detection Processing> FIG. 19 is a flowchart illustrating a series of processing steps when detecting a movement section.

[0101] In step S101 , the data acquisition unit 211 acquires image data output from the measuring device 11 .

[0102] In step S102, an event scene graph generation process is performed by the scene graph difference generation unit 212. In step S102, the process is basically the same as the process described with reference to the flowchart in FIG.

[0103] In step S103, a state transition graph generation process is performed by the state transition structure generation unit 213. In step S103, the process is basically the same as the process described with reference to the flowchart in FIG.

[0104] In step S104, the search unit 214 performs a graph search process.

[0105] Graph Search Processing FIG. 20 is a flowchart illustrating the graph search processing performed in step S104 of FIG.

[0106] In step S111, the search unit 214 acquires a state transition graph.

[0107] In step S112, the search unit 214 reads out, from among the subgraphs recorded in the motion interval structuring information DB, a subgraph whose final inter-object state is common to the state transition graph, as a candidate subgraph.

[0108] FIG. 21 is a diagram illustrating an example of a state transition graph at the time of detection.

[0109] For example, when the state transition graph #21 shown in FIG. 21 is generated by the state transition structure generation unit 213 as the state transition graph at the time of detection (current), the subgraph that shares the node "desk-cup: on", which is the state between the last objects, is read out as the candidate subgraph.

[0110] The state transition graph #21 shown in FIG. 21 has the same graph structure as the state transition graph in FIG. 15. That is, the state transition graph #21 is composed of a "Desk-Medicine: On" node, a "Desk-Medicine: Near" node, a "Desk-Cup: On" node, and a "Desk-Cup: Near" node. The "Desk-Medicine: On" node and the "Desk-Medicine: Near" node are connected by a directed edge E11 pointing from the "Desk-Medicine: On" node to the "Desk-Medicine: Near" node, and a directed edge E12 pointing from the "Desk-Medicine: Near" node to the "Desk-Medicine: On" node. A timestamp of "20:26" is added to the directed edge E11, and a timestamp of "20:28" is added to the directed edge E21.

[0111] The "Desk-Medicine: On" node and the "Desk-Cup: On" node are connected by a directed edge E13 pointing from the "Desk-Medicine: On" node to the "Desk-Cup: On" node. A timestamp of "20:29" is added to the directed edge E13.

[0112] The "Desk-Cup: on" node and the "Desk-Cup: near" node are connected by a directed edge E14 pointing from the "Desk-Cup: on" node to the "Desk-Cup: near" node, and a directed edge E15 pointing from the "Desk-Cup: near" node to the "Desk-Cup: on" node. A timestamp of "20:29" is added to directed edge E14, and a timestamp of "20:30" is added to directed edge E15.

[0113] Returning to the explanation of FIG. 20, in step S113, the search unit 214 uses a graph search algorithm to search for a subgraph from among the candidate subgraphs that has the closest graph structure to the state transition graph.

[0114] In step S114, when a subgraph that satisfies the conditions is obtained by the search, the search unit 214 refers to the timestamps of the edges that make up the state transition graph and detects the section from the time of the first timestamp to the time of the last timestamp as the operation section.

[0115] FIG. 22 is a diagram illustrating an example of a subgraph search.

[0116] Candidate subgraphs #31 and #32 shown on the right side of Fig. 22 are subgraphs of the action "take medicine" and the action "brush teeth," respectively. In these subgraphs, the node "desk-cup: on," which is the last inter-object state, is common to state transition graph #21.

[0117] A search is performed on candidate subgraphs #31 and #32, and candidate subgraph #31 for the action of "taking medicine" is found as a subgraph that satisfies the conditions. The timestamps of the edges that make up state transition graph #21 are referenced, and the section from the first timestamp, "20:26," to the last timestamp, "20:30," is detected as an action section. The first timestamp, "20:26," is the time when a change in the relationship between the desk and medicine on the action section detection side was detected, which was the same as the first change in the relationship between the desk and medicine on the action section recording side represented by candidate subgraph #31. Furthermore, the last timestamp, "20:30," is the time when a change in the relationship between the desk and cup on the action section detection side was detected, which was the same as the last change in the relationship between the desk and cup on the action section recording side represented by candidate subgraph #31.

[0118] The above processing makes it possible to properly detect sections of slow or small movements that are difficult to determine from a person's joint information. Furthermore, because processing is performed using lightweight data that represents the difference in changes in the relationships between objects, it is possible to reduce the amount of calculation and memory used for detecting movement sections. Furthermore, it is possible to reduce the storage capacity of the data used for detecting movement sections.

[0119] <<Configuration Example of Information Processing Apparatus>> <Configuration Related to Behavior Recognition> FIG. 23 is a block diagram showing a configuration example of the information processing apparatus 12. As shown in FIG.

[0120] 23 shows, as functional units that perform processing related to behavior recognition, an image data acquisition unit 301, a behavior data generation unit 302, and a behavior recognition unit 303 in addition to the above-mentioned behavior segment detection unit 201. When the behavior segment recording process is performed in the information processing device 12, the behavior segment structuring information recording unit 101 is also provided in the information processing device 12. The above description will be omitted as appropriate.

[0121] The image data acquisition unit 301 acquires image data such as color images and distance images supplied from the measurement device 11. The image data acquired by the image data acquisition unit 301 is supplied to the movement interval detection unit 201 and the movement data generation unit 302. The movement interval detection unit 201 performs movement interval detection processing based on the image data acquired by the image data acquisition unit 301, and outputs information about the movement intervals to the behavior recognition unit 303.

[0122] The motion data generation unit 302 performs skeleton estimation on the image supplied from the image data acquisition unit 301 and acquires skeleton point information. The skeleton point information acquired by the motion data generation unit 302 is output to the behavior recognition unit 303. A time series of skeleton point information acquired based on the image of each frame is output from the motion data generation unit 302 to the behavior recognition unit 303. Information on feature points other than skeleton points may also be acquired.

[0123] The behavior recognition unit 303 recognizes the user's behavior based on the skeleton point information of the frames of the movement intervals detected by the movement interval detection unit 201, out of the skeleton point information of each frame supplied from the movement data generation unit 302. The behavior recognition unit 303 has an inference model prepared in advance that receives, for example, time-series data of skeleton point information as input and outputs information on the behavior of a person. The behavior recognition unit 303 uses the skeleton point information of the movement intervals as input to the inference model to recognize behaviors such as "taking medicine" and "brushing teeth."

[0124] In this way, information on the movement sections detected by the movement section detection process is used to extract information to be used for recognizing a person's behavior, for example.

[0125] Information on movement sections detected by the movement section detection process may be used in the following applications: (1) Monitoring the elderly: Movement sections in which the person being monitored takes medicine are detected, and the time and number of times the medicine is taken are recorded. To accurately recognize that the person is taking medicine, behavior recognition is used. (2) Recording of behavior logs: From one day's worth of movement data, a life log is created and recorded for each hour, such as periods when the person was taking a break. Because the amount of data recorded is small, even an edge device can hold several months' worth of data. (3) Automatic annotation of movement sections: For example, this is used to specify movement sections when creating a dataset to be used for training an inference model for behavior recognition.

[0126] <Modification of the Configuration of the Information Processing Device> FIG. 24 is a block diagram showing another example of the configuration of the information processing device 12. In FIG.

[0127] The information processing device 12 shown in Fig. 24 is a device that has both the function of performing a motion segment recording process and the function of performing a motion segment detection process. The information processing device 12 is provided with a data acquisition unit 111, a scene graph difference generation unit 112, a state transition structure generation unit 113, a recording unit 114, and a search unit 214. When recording a motion segment, the data acquisition unit 111, the scene graph difference generation unit 112, the state transition structure generation unit 113, and the recording unit 114 perform a motion segment recording process. When detecting a motion segment, the data acquisition unit 111, the scene graph difference generation unit 112, the state transition structure generation unit 113, and the search unit 214 perform a motion segment detection process.

[0128] In this way, it is possible to realize the function of performing the movement section recording process and the function of performing the movement section detection process in the same device such as a PC.

[0129] FIG. 25 is a diagram illustrating an example of the configuration of an information processing system.

[0130] 25 is configured by connecting a robot 401 and a server 402 via a network such as a LAN or the Internet. Communication between the robot 401 and the server 402 is carried out via the network.

[0131] The robot 401 is a mobile object that acts autonomously in a space where a user is present. A predetermined program is executed on the computer of the robot 401, and a data acquisition unit 111, a scene graph difference generation unit 112, and a state transition structure generation unit 113 are realized. When recording a movement segment, the robot 401 performs the process up to generating a transition state graph of the movement segment recording process. Similarly, when detecting a movement segment, the robot 401 performs the process up to generating a transition state graph of the movement segment detection process. Information on the state transition graph generated by the robot 401 is sent to the server 402.

[0132] The server 402 is an information processing device on the cloud. The server 402 implements a recording unit 114 and a search unit 214. When recording a movement section, the recording unit 114 records movement section structuring information generated based on the transition state graph transmitted from the robot 401 in a DB. When detecting a movement section, the search unit 214 detects the movement section based on the transition state graph transmitted from the robot 401.

[0133] In this way, the movement section recording process and the movement section detection process up to the generation of the state transition graph may be performed on the device used by the user (robot 401), and the subsequent processes may be performed on an external information processing device.In place of the robot 401, other devices such as PCs, smartphones, tablet terminals, and HMDs (Head Mounted Displays) may be used as the user's device.

[0134] FIG. 26 is a diagram illustrating another example of the configuration of an information processing system.

[0135] The information processing system shown in Fig. 26 is composed of a smartphone 411, a robot 412, a server 413, and a server 414. The smartphone 411 and the robot 412 are devices used by a user. The smartphone 411 and the robot 412 are connected via a wireless LAN or the like. The robot 412 and the server 413, and the robot 412 and the server 414 are each connected via a network such as a LAN or the Internet.

[0136] A predetermined program is executed in the smartphone 411, and the data acquisition unit 111 is realized. When a movement section is recorded or detected, image data acquired by the data acquisition unit 111 is transmitted to the robot 412.

[0137] The robot 412 has the same hardware configuration as the robot 401 in Fig. 25. A predetermined program is executed on the computer of the robot 412, and a scene graph difference generation unit 112 and a state transition structure generation unit 113 are realized. When recording a movement section, the process up to the generation of a transition state graph in the movement section recording process is performed using image data transmitted from the smartphone 411, and information on the state transition graph is transmitted to the server 413. Similarly, when detecting a movement section, the process up to the generation of a transition state graph in the movement section detection process is performed using image data transmitted from the smartphone 411, and information on the state transition graph is transmitted to the server 414.

[0138] Both the server 413 and the server 414 are information processing devices on the cloud. The recording unit 114 is implemented in the server 413, and the search unit 214 is implemented in the server 414. When recording a movement section, movement section structuring information generated based on the transition state graph transmitted from the robot 412 is recorded in a DB by the recording unit 114 of the server 413. When detecting a movement section, the search unit 214 of the server 414 detects the movement section based on the transition state graph transmitted from the robot 412.

[0139] In this way, it is possible to arbitrarily change which components of the movement segment structuring information recording unit 101 and the movement segment detection unit 201 are realized in which devices. An outside-in type HMD or a robot that requires a separate camera to photograph the user may be used as the user's device. The function of the data acquisition unit 111 may be realized in an RGB camera or a smartphone. Furthermore, a computer that performs movement segment recording processing and a computer that performs movement segment detection processing may be separately provided.

[0140] <<Modifications>> <Example of Search Using Node Weights> FIG. 27 is a diagram showing an example of different environments.

[0141] With this technology, it is possible to perform the movement section detection process even when an object that was present in the room that was the target of the movement section when it was recorded is not present in the room that was the target of the detection, as shown in Fig. 27. In the example of Fig. 27, the cup that was present in the room that was the target of the movement section when it was recorded is not present in the room that was the target of the detection.

[0142] To enable the motion section detection process to be performed even when the environments of objects are different, a weight is assigned to each node that makes up the subgraph recorded as motion section structured information according to the importance of the state between the objects. For example, when focusing on the motion of "taking medicine," some medicines require water and some do not, so the node that indicates the state of the cup and desk is less important than the node that indicates the state of the medicine and desk.

[0143] FIG. 28 is a diagram illustrating an example of node weights.

[0144] Subgraph #51 shown on the left side of Figure 28 is a subgraph recorded in the DB as action interval structured information for the action "drink medicine." A weight of 0.9 is assigned to the "desk-medicine: on" node, and a weight of 0.85 is assigned to the "desk-medicine: near" node. Furthermore, a weight of 0.1 is assigned to the "desk-cup: on" node, and a weight of 0.2 is assigned to the "desk-cup: near" node.

[0145] FIG. 29 is a diagram illustrating an example of setting the weights of nodes.

[0146] As shown in Figure 29, weights are set by integrating multiple subgraphs generated as movement segment structuring information when recording a movement segment. For example, machine learning is performed using multiple subgraphs for the same movement as training data, and an inference model for setting weights is generated. Subgraphs #61 to #63 on the left side of Figure 29 are subgraphs for the movement of "taking medicine." For example, when recording movement segment structuring information, the recording unit 114 of the movement segment structuring information recording unit 101 uses the inference model to calculate and set the weights of each node that makes up the subgraphs generated as movement segment structuring information.

[0147] This makes it possible to search for subgraphs while taking into account the weights of the nodes when detecting an operation section, as shown in Fig. 28. In the example of Fig. 28, subgraph #51, which has a different node configuration from the state transition graph acquired when detecting an operation section, is searched for as a subgraph that satisfies the conditions.

[0148] The search for detecting the movement intervals may be performed by calculating the distance between graphs (a state transition graph and a subgraph prepared as movement interval structuring information), or may be performed using an inference model for search generated by machine learning. In the latter case, the search unit 214 of the movement interval detection unit 201 is provided with an inference model that receives graphs to be compared as input and outputs their similarity (likelihood).

[0149] <Modified Scene Graph> Although a scene graph composed of nodes representing objects is generated by the scene graph difference generation unit 112 (scene graph difference generation unit 212), feature points such as skeletal points (joint positions) of a person may also be included as nodes. When creating a scene graph using PanopticFusion or the like, not only are the relationships between the objects estimated, but the semantic positional relationships between the feature points of the person and the objects are also estimated.

[0150] FIG. 30 is a diagram showing an example of a scene graph.

[0151] Nodes N1 to N4 shown in FIG. 30 are nodes that represent objects. Labels indicating the relationship between the objects are set on the edges connecting the respective nodes. Meanwhile, nodes N11 to N13 are nodes that represent feature points of a person. Node N11 represents the head, and nodes N12 and N13 represent the right hand and left hand, respectively. The edge connecting node N11 and node N2 is set to the label "above," indicating the relationship between the head and the object represented by node N2, and the edge connecting node N13 and node N2 is set to the label "near," indicating the relationship between the left hand and the object represented by node N2. In this way, the coordinates of each joint of the person may be estimated by skeletal estimation or the like, and the above-described processing may be performed using a scene graph that includes information indicating the positional relationship between the person and objects.

[0152] <Automation of annotation using a generative model> When recording a behavior section, the range of a subgraph to be extracted from a state transition graph as behavior section structuring information is manually specified as shown in Fig. 31 , but the generation of the subgraph may be automated using a generative model. An inference model (generative model) for generating a subgraph generated by machine learning is prepared in the recording unit 114.

[0153] FIG. 32 is a diagram showing the flow of generating a subgraph.

[0154] The motion interval subgraph generation model M shown in FIG. 32 is an inference model for subgraph generation. The motion interval subgraph generation model M is input with the entire state transition graph generated when recording the motion interval and information specifying which motion a subgraph is to be generated for. The information specifying the subgraph is input, for example, by voice. In the example of FIG. 32, the generation of a subgraph for the motion of "taking medicine" is specified by voice using natural language. The information processing device 12 recognizes the voice and uses it as input to the motion interval subgraph generation model M. A voice recognition function may be provided in the motion interval subgraph generation model M.

[0155] When information specifying the entire state transition graph and a subgraph is input, the specified subgraph is output from the behavior interval subgraph generation model M. The behavior interval subgraph generation model M is generated by machine learning using, for example, sets of the state transition graph, information specifying the subgraph, and various behaviors of the subgraph as learning data.

[0156] In this way, the process of analyzing the input state transition graph and generating the specified subgraph may be performed automatically using an inference model.

[0157] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0158] 33 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program. For example, the information processing device 12 in FIG. 1 is configured by a computer having the configuration shown in FIG.

[0159] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0160] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0161] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0162] The program executed by the CPU 1001 is provided, for example, by being recorded on a removable medium 1011 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 1008.

[0163] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0164] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0165] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0166] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0167] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0168] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0169] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0170] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0171] (1) An information processing device comprising an information processing unit that generates, based on measurement results by a sensor, a time series of a scene graph that is graph-structured information in which a plurality of first nodes representing a plurality of objects are connected by first edges that indicate relationships between the objects, and when a relationship between the objects changes due to a person's movement, generates an event scene graph that is graph-structured information that includes the time at which the change in the relationship between the objects was measured and connects second nodes representing each of the objects whose relationship has changed by second edges that indicate the content of the change in the relationship. (2) The information processing device described in (1), wherein the information processing unit generates the event scene graph every time a relationship between the objects changes. (3) The information processing device described in (1) or (2), wherein the information processing unit generates, based on the event scene graph, a state transition graph that is graph-structured information in which two inter-object relationship nodes that respectively indicate the relationship between the objects before and after the change are connected by relationship change edges that indicate the time and the direction of the change. (4) The information processing device according to (3), wherein the information processing unit generates the state transition graph in which the inter-object relation node indicating the relationship after the change between the first type of objects and the inter-object relation node indicating the relationship before the change between the second type of objects are connected by the relationship change edge. (5) The information processing device according to (4), wherein the information processing unit records a subgraph of the state transition graph generated using the measurement results while a person is performing a specific action. (6) The information processing device according to (5), wherein the information processing unit generates the subgraph based on information indicating a section of the action input by an administrator. (7) The information processing device according to (5), wherein the information processing unit generates the subgraph for each action using a generative model that receives the state transition graph and information specifying an action as input and outputs the subgraph.(8) An information processing method in which an information processing device generates, based on measurement results by a sensor, a time series of a scene graph that is graph-structured information in which a plurality of first nodes that respectively represent a plurality of objects are connected by first edges that indicate relationships between the objects, and when a relationship between the objects changes due to a person's movement, generates an event scene graph that is graph-structured information that includes the time when the change in the relationship between the objects was measured and connects second nodes that indicate each of the objects whose relationship has changed by a second edge that indicates the content of the change in the relationship. (9) A program that causes a computer to execute processes of: generating, based on measurement results by a sensor, a time series of a scene graph that is graph-structured information in which a plurality of first nodes that respectively represent a plurality of objects are connected by first edges that indicate relationships between the objects, and when a relationship between the objects changes due to a person's movement, generates an event scene graph that is graph-structured information that includes the time when the change in the relationship between the objects was measured and connects second nodes that indicate each of the objects whose relationship has changed by a second edge that indicates the content of the change in the relationship. (10) An information processing device comprising: an information processing unit that acquires a plurality of subgraphs generated for each action of a person as graph structure information in which inter-object relationship nodes indicating the relationships between a plurality of first objects before and after a change, respectively, are connected by relationship change edges indicating the time when the change in the relationship between the first objects was measured and the direction of the change; detects changes in the relationships between a plurality of second objects used by the person being measured based on measurement results by a sensor; searches for the subgraph of a predetermined action that indicates a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects; and detects a section in which the person being measured performed the same action as the predetermined action.(11) The information processing device according to (10), wherein the information processing unit detects, as a change in the relationship between the second objects, a section from a time when a change identical to a first change in the relationship between the first objects represented by the subgraph of the predetermined behavior is detected to a time when a change identical to a last change is detected, as a section in which the person to be measured performed the predetermined behavior. (12) The information processing unit generates, based on the measurement results by the sensor, a time series of a scene graph which is graph-structured information in which a plurality of first nodes representing a plurality of the second objects are connected by first edges which indicate the relationships between the second objects, and when the relationships between the second objects change due to the behavior of the person to be measured, generates an event scene graph which is graph-structured information in which second nodes representing the second objects whose relationships have changed are connected by second edges which indicate the content of the change in the relationship, including the time when the change in the relationship between the second objects was measured. (13) The information processing device according to (12), wherein the information processing unit generates the event scene graph every time the relationship between the second objects changes. (14) The information processing device according to (13), wherein the information processing unit generates, based on the event scene graph, a state transition graph which is graph-structured information in which third nodes indicating pre-change relationships and post-change relationships between the second objects are connected by a third edge indicating the time and the direction of the change. (15) The information processing device according to (14), wherein the information processing unit generates the state transition graph in which the third node indicating the post-change relationship between the second objects of a first type and the third node indicating the pre-change relationship between the second objects of a second type are connected by the third edge. (16) The information processing device according to (14) or (15), wherein the information processing unit extracts, as a search target, the subgraph including the inter-object relation node indicating the same change as the last change among the changes between the second objects indicated by the third nodes constituting the state transition graph.(17) An information processing method, in which an information processing device acquires a plurality of subgraphs generated for each action of a person as graph structure information in which inter-object relationship nodes indicating relationships between a plurality of first objects before and after a change, respectively, are connected by relationship change edges indicating the time when the change in the relationship between the first objects was measured and the direction of the change, detects changes in the relationships between a plurality of second objects used by the person being measured based on measurement results by a sensor, searches for the subgraph of a predetermined action indicating a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects, and detects a section in which the person being measured performed the same action as the predetermined action. (18) A program that causes a computer to execute the following process: acquire a plurality of subgraphs generated for each of a person's actions as graph structure information in which inter-object relationship nodes indicating relationships between a plurality of first objects before and after a change, respectively, are connected by relationship change edges indicating the time when the change in the relationship between the first objects was measured and the direction of the change; detect changes in the relationships between a plurality of second objects used by the person being measured based on the measurement results from a sensor; search for the subgraph of a predetermined action that indicates a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects; and detect a section in which the person being measured performed the same action as the predetermined action.

[0172] REFERENCE SIGNS LIST 1 Information processing device, 11 Measurement device, 12 Information processing device, 101 Action section structuring information recording unit, 111 Data acquisition unit, 112 Scene graph difference generation unit, 113 State transition structure generation unit, 114 Recording unit, 201 Action section detection unit, 211 Data acquisition unit, 212 Scene graph difference generation unit, 213 State transition structure generation unit, 214 Search unit, 301 Image data acquisition unit, 302 Action data generation unit, 303 Action recognition unit

Claims

1. An information processing device comprising an information processing unit that generates, based on the results of measurements by a sensor, a time series of a scene graph, which is graph-structured information in which multiple first nodes each representing multiple objects are connected by first edges that indicate the relationships between the objects, and when the relationships between the objects change due to a person's movements, generates an event scene graph, which is graph-structured information in which second nodes representing each of the objects whose relationships have changed are connected by second edges that indicate the content of the change in the relationships and include the time when the change in the relationships between the objects was measured.

2. The information processing device according to claim 1, wherein the information processing section generates the event scene graph every time a relationship between the objects changes.

3. The information processing device according to claim 1, wherein the information processing unit generates a state transition graph, which is graph-structured information in which two inter-object relationship nodes indicating the relationship between the objects before and after the change, respectively, are connected by a relationship change edge indicating the time and the direction of the change, based on the event scene graph.

4. The information processing device according to claim 3, wherein the information processing unit generates the state transition graph in which the inter-object relation nodes indicating the post-change relations between the first type of objects and the inter-object relation nodes indicating the pre-change relations between the second type of objects are connected by the relation change edges.

5. The information processing device according to claim 4, wherein the information processing unit records a subgraph of the state transition graph that is generated using the measurement results while the person is performing a specific action.

6. The information processing device according to claim 5, wherein the information processing unit generates the subgraph based on information indicating an operation section input by an administrator.

7. The information processing device according to claim 5, wherein the information processing unit receives the state transition graph and information specifying an operation as input, and generates the subgraph for each operation using a generation model that outputs the subgraph.

8. An information processing method in which an information processing device generates, based on measurement results from a sensor, a time series of a scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of objects are connected by a first edge indicating the relationship between the objects, and when the relationship between the objects changes due to a person's movement, generates an event scene graph, which is graph-structured information in which second nodes representing each of the objects whose relationship has changed are connected by a second edge indicating the content of the change in the relationship, and which includes the time when the change in the relationship between the objects was measured.

9. A program that causes a computer to execute the following process: based on the results of measurements by a sensor, generate a time series of a scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of objects are connected by a first edge that indicates the relationship between the objects; and when the relationship between the objects changes due to a person's movements, generate an event scene graph, which is graph-structured information in which second nodes representing each of the objects whose relationship has changed are connected by a second edge that indicates the content of the change in the relationship and includes the time when the change in the relationship between the objects was measured.

10. An information processing device comprising: an information processing unit that acquires multiple subgraphs generated for each person's behavior as graph structure information in which inter-object relationship nodes indicating the relationships between multiple first objects before and after a change, respectively, are connected by relationship change edges indicating the time when the change in the relationship between the first objects was measured and the direction of the change; detects changes in the relationships between multiple second objects used by the person being measured based on measurement results from a sensor; searches for the subgraph of a specified behavior that indicates a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects; and detects a section in which the person being measured performed the same behavior as the specified behavior.

11. The information processing device according to claim 10, wherein the information processing unit detects, as a change in the relationship between the second objects, a section from the time when a change identical to the first change in the relationship between the first objects represented by the subgraph of the specified action is detected to the time when a change identical to the last change is detected as a section in which the person being measured performed the specified action.

12. The information processing device according to claim 10, wherein the information processing unit generates, based on the measurement results by the sensor, a time series of a scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of the second objects are connected by a first edge indicating the relationship between the second objects, and when the relationship between the second objects changes due to the action of the person being measured, generates an event scene graph, which is graph-structured information in which second nodes representing each of the second objects whose relationship has changed are connected by a second edge indicating the content of the change in the relationship, and which includes the time when the change in the relationship between the second objects was measured.

13. The information processing device according to claim 12, wherein the information processing section generates the event scene graph every time a relationship between the second objects changes.

14. The information processing device according to claim 13, wherein the information processing unit generates, based on the event scene graph, a state transition graph, which is graph-structured information in which third nodes indicating the relationships between the second objects before and after the change are connected by third edges indicating the time and direction of the change.

15. The information processing device according to claim 14, wherein the information processing unit generates the state transition graph in which the third node indicating the relationship between the second objects of the first type after the change and the third node indicating the relationship between the second objects of the second type before the change are connected by the third edge.

16. The information processing device according to claim 14, wherein the information processing unit extracts, as a search target, the subgraph including the inter-object relation node that indicates the same change as the last change among the changes between the second objects indicated by the third node constituting the state transition graph.

17. An information processing method comprising: an information processing device acquiring a plurality of subgraphs generated for each action of a person as graph structure information in which inter-object relationship nodes indicating the relationships between a plurality of first objects before and after a change, respectively, are connected by relationship change edges indicating the time at which the change in the relationship between the first objects was measured and the direction of the change; detecting changes in the relationships between a plurality of second objects used by the person being measured based on measurement results from a sensor; searching for the subgraph of a predetermined action indicating a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects; and detecting a section in which the person being measured performed the same action as the predetermined action.

18. A program causing a computer to execute the following processes: acquire multiple subgraphs generated for each person's behavior as graph structure information in which inter-object relationship nodes indicating the relationships between multiple first objects before and after a change, respectively, are connected by relationship change edges indicating the time when the change in the relationship between the first objects was measured and the direction of the change; detect changes in the relationships between multiple second objects used by the person being measured based on the measurement results from a sensor; search for the subgraph of a predetermined behavior that indicates a change in the relationship between the first objects that is the same as or similar to the change in the relationship between the second objects; and detect a section in which the person being measured performed the same behavior as the predetermined behavior.

Images