Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to easily reflect actions of a dynamic object in actions of another dynamic object. In the present technology, a first scene graph is generated, said first scene graph being information on a graph structure in which a plurality of first nodes respectively indicating a plurality of objects in a space are connected via first edges indicating relationships between the objects in the space, and each of the first nodes is connected to a second node indicating a body part of a dynamic object via a second edge indicating a relationship between the object in the space indicated by the first node and the body part when the dynamic object is in a specific posture. The present technology can be applied to a computer that recognizes actions of a user in a real space and reflects the actions as actions of an avatar in a virtual space.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that enable the behavior of a dynamic object to be easily reflected in the behavior of another dynamic object.

[0002] There is a technology that senses a user's actions in the real world and allows an avatar in a virtual space to express the same actions while maintaining semantic consistency. Other users viewing the virtual space using devices such as PCs can observe the actions of a certain user in the real world through the avatar in the virtual space.

[0003] However, because real space and virtual space differ in terms of the presence or absence of objects, the shapes of objects, etc., if an attempt is made to directly represent the actions of a user in real space as the actions of an avatar, the behavior of the avatar may become unnatural. Patent Literature 1 discloses a technique for manually adjusting the position of an avatar to match the situation in virtual space, and manually adjusting the positions of joints that determine the avatar's pose.

[0004] Japanese Patent Application Laid-Open No. 2015-176495

[0005] Virtual spaces known as metaverses are continually being created by providers and creators, and it is not realistic to manually adjust poses so that they can be applied to all virtual spaces.

[0006] The present technology has been made in view of such circumstances, and makes it possible to easily reflect the behavior of a dynamic object in the behavior of another dynamic object.

[0007] An information processing device according to a first aspect of the present technology includes an information processing unit that generates a first scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of in-space objects are connected by a first edge that indicates the relationship between the in-space objects, and the first nodes and second nodes that indicate physical parts of a dynamic object are connected by a second edge that indicates the relationship between the in-space objects represented by the first nodes and the physical parts when the dynamic object assumes a specific pose, based on measurement results from a sensor.

[0008] An information processing device according to a second aspect of the present technology includes an information processing unit that acquires recognition results of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, connects a plurality of first nodes each representing a plurality of second intra-space objects existing in a second space with a first edge indicating the relationship between the second intra-space objects, and reflects the behavior of the first dynamic object in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information that connects the first nodes and second nodes indicating physical parts of the second dynamic object with second edges indicating the relationship between the second intra-space object indicated by the first nodes and the physical part when the second dynamic object takes a specific pose that appears in the recognized behavior of the first dynamic object.

[0009] In a first aspect of the present technology, a first scene graph is generated based on measurement results from a sensor, the first scene graph being information of a graph structure in which a plurality of first nodes each representing a plurality of in-space objects are connected by a first edge indicating the relationship between the in-space objects, and the first nodes and second nodes indicating physical parts of a dynamic object are connected by a second edge indicating the relationship between the in-space object represented by the first node and the physical part when the dynamic object assumes a specific pose.

[0010] In a second aspect of the present technology, a recognition result of a behavior of a first dynamic object performed using a first intra-space object existing in a first space is acquired, and the behavior of the first dynamic object is reflected in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information in which a plurality of first nodes representing a plurality of second intra-space objects existing in a second space are connected by first edges indicating relationships between the second intra-space objects, and the first nodes and second nodes indicating physical parts of the second dynamic object are connected by second edges indicating relationships between the second intra-space objects represented by the first nodes and the physical parts when the second dynamic object assumes a specific pose that appears in the recognized behavior of the first dynamic object.

[0011] 1 is a diagram illustrating an example of a use case of the present technology. FIG. 1 is a diagram illustrating an example of processing using a Scene Graph. FIG. 2 is a diagram illustrating an example of a Scene Graph with joint. FIG. 3 is a diagram illustrating an example of a skeleton. FIG. 4 is a diagram illustrating an example of a general Scene Graph. FIG. 5 is a diagram illustrating a series of processing flows in an information processing device. FIG. 6 is a block diagram illustrating an example of the functional configuration of an important relationship extraction unit. FIG. 7 is a diagram illustrating an example of a Scene Graph with joint. FIG. 8 is a diagram illustrating an example of integrating Scene Graphs with joints. FIG. 9 is a diagram illustrating an example of generating an Integrated Scene Graph. FIG. 10 is a block diagram illustrating an example of generating an Essential Scene Graph. FIG. 11 is a block diagram illustrating an example of the functional configuration of an Optimal Position calculation unit. FIG. 12 is a diagram illustrating an example of affordance. FIG. 13 is a diagram illustrating an example of an Essential Scene Graph and a Scene Graph of a virtual space. FIG. 14 is a diagram illustrating an example of calculation of a distance between graphs. FIG. 15 is a diagram illustrating another example of calculation of a distance between graphs. FIG. 16 is a block diagram illustrating an example of a configuration of an information processing device. FIG. 17 is a block diagram illustrating another example of a configuration of an information processing device. FIG. 18 is a block diagram illustrating an example of a configuration of a computer.

[0012] Hereinafter, embodiments of the present technology will be described. The description will be made in the following order: 1. Overview of the present technology 2. Processing flow of the present technology 3. Important relationship extraction flow 4. Avatar reflection flow 5. Configuration of information processing device 1 6. Modified examples

[0013] <<Outline of the Present Technology>> <Use Case of the Present Technology> FIG. 1 is a diagram illustrating an example of a use case of the present technology.

[0014] This technology is used to sense a user's behavior in real space and have an avatar in a virtual space express the same behavior as the user's behavior while maintaining semantic consistency. FIG. 1 shows how the behavior of user A, a person existing in real space, is sensed and the behavior with the same meaning is reflected as the behavior of avatar A. The humanoid avatar A shown on the right side of FIG. 1 is user A's avatar existing in the virtual space. Such processing is realized, for example, by an information processing device 1 and a sensor device 2.

[0015] User A's actions are reflected not by having avatar A directly express small movements such as the movements of user A's hands and feet, but by extracting the meaning of user A's actions and having avatar A express actions with the same meaning. For example, as shown in Figure 1, when the action of "sitting on a chair" is detected, the direction of the chair in the virtual space is calculated based on the position of avatar A, and avatar A expresses the action of moving to the position of the chair and sitting down.

[0016] This allows user A to express his or her experience in the real world in the virtual space. In addition, other users viewing the virtual space using devices such as PCs or smartphones can observe user A's actions in the real world through avatar A in the virtual space. For example, the virtual space shown in Figure 1 contains avatars for multiple users. Each user can operate their own avatar to move it to any position in the virtual space and see what it looks like at that location.

[0017] In the use case of Figure 1, the main issues are as follows:

[0018] 1. Differences in spatial layout between real space and virtual space Differences in spatial layout between real space and virtual space arise from differences in coordinate systems, the presence or absence of objects, and differences in the shapes of objects. It is difficult to directly represent the user's actions recognized in real space as the actions of an avatar due to differences in spatial layout. The real space and virtual space shown in Figure 1 also have different spatial layouts. Spatial layout is determined by the size and shape of the space, as well as the number, shape, and size of the objects placed in the space.

[0019] 2. Real-time performance when reflecting user actions as avatar actions: Because the computational load involved in avatar placement is large, it is difficult to adjust the position and posture of an avatar in real time to match the spatial layout of the virtual space. In order to reflect user actions as avatar actions in real time, these processes must be performed automatically with minimal computational effort.

[0020] <Scene Graph Used in the Present Technology> FIG. 2 is a diagram showing an example of processing using a Scene Graph.

[0021] In the information processing device 1, the spatial layouts of the real space and the virtual space are abstracted and represented as Scene Graphs, as indicated by arrows A1 and A2 in Fig. 2. Furthermore, as indicated by the bidirectional arrows in Fig. 2, the position of an avatar that minimizes the graph distance between the Scene Graphs of the real space and the virtual space is searched for. The avatar's behavior in the virtual space is expressed by placing the avatar at the searched position.

[0022] If the real space is the first space, the virtual space is the second space. A spatial object existing in the real space as the first space corresponds to a first spatial object, and the user corresponds to a first dynamic object. On the other hand, a spatial object existing in the virtual space as the second space corresponds to a second spatial object, and the avatar corresponds to a second dynamic object.

[0023] As will be described in detail later, a Scene Graph for the real space is generated that is composed of nodes of the minimum number of related objects required for each user action, and is used for comparison with the Scene Graph for the virtual space. To generate the Scene Graph for the real space that is used for comparison with the Scene Graph for the virtual space, a Scene Graph with joint is used, which is a Scene Graph that includes nodes that indicate the physical parts (joints) of a person along with objects in the space.

[0024] FIG. 3 is a diagram showing an example of a Scene Graph with joint.

[0025] A typical Scene Graph is information with a graph structure that shows people as nodes along with objects in the space, as shown in A of Fig. 3. The Scene Graph shown in A of Fig. 3 is composed of three nodes that show objects in the space: Sofa #1, Chair #2, and Table #3, and one node that shows Human #1 as a user. The nodes are connected by edges that have labels that show the relationships between them.

[0026] While a typical Scene Graph is graph-structured information that represents one person with one node, a Scene Graph with joint is graph-structured information that represents each joint of one person, namely, Head #1, Neck #1, Left Hand #1, Right Hand #1, Spine Base #1, Left Leg #1, and Right Leg #1, with a node, as shown in FIG. 3B. Nodes representing objects in space and nodes representing joints are also connected by edges with labels indicating their respective relationships. In the example of FIG. 3B, the Spine Base #1 node and the Chair #2 node are connected by an edge labeled "on." For example, when a user is sitting in Chair #2, the spatial layout of the real space at that time is represented by the Scene Graph with joint in FIG. 3B.

[0027] <Terminology> Here, we will briefly explain the main terms.

[0028] Dynamic objects are objects that have skeletal information (skeleton). Dynamic objects are divided into real-world dynamic objects and virtual-world dynamic objects. For example, dynamic objects include people, avatars, and animals.

[0029] Static objects are the opposite concept of dynamic objects, and are objects that do not have skeletal information. There are static objects in real space and static objects in virtual space. For example, chairs, desks, TVs, lighting equipment, etc. are included in static objects. Hereinafter, static objects will be referred to as in-space objects where appropriate. Furthermore, when there is no need to distinguish between dynamic and static objects, they will simply be referred to as objects.

[0030] Pose is information that defines the body position and posture of a dynamic object at a specific time. Pose is mainly expressed based on skeletal information.

[0031] Motion is composed of a series of poses of a dynamic object that are continuous over time. The movement or behavior of a dynamic object corresponds to motion.

[0032] Position: A position is the location in space of a static or dynamic object. Position is expressed by three-dimensional coordinates.

[0033] Skeleton Information Skeleton information is structural information about the skeleton that constitutes a dynamic object. Skeleton information is data necessary for controlling movement and posture. As shown in FIG. 4, the skeleton is mainly represented by bones and joints that control the rotational movement of the bones. If the dynamic object is a humanoid object, the joints include specific physical parts such as the head, hands, and feet as well as the joints of a human body. In this technology, for example, the head, neck, right hand, left hand, right leg, and left leg, which are indicated by dots in FIG. 4, are treated as joints. For example, the position of an avatar is calculated using the skeleton.

[0034] Bone: A bone is information that connects physical parts in the structural information of the skeleton that makes up a dynamic object. Bones contain coordinate data.

[0035] Joint: A joint is a physical part that contains coordinate data in the structural information of the skeleton that makes up a dynamic object. There are joints that are connected to one bone and joints that are connected to multiple bones.

[0036] Node: A node represents a dynamic object or an object in space in graph structure information. In this technology, a node is used that represents a single bone, a single joint, or a combination of a bone and a joint of a dynamic object.

[0037] ・Scene Graph A Scene Graph is information with a graph structure that represents the relationships between objects. A Scene Graph is made up of multiple nodes and edges that connect the nodes.

[0038] Figure 5 shows an example of a typical Scene Graph. If a space contains a sofa, a table, a television, chair A, and chair B, the Scene Graph will contain five nodes representing these objects, as shown in Figure 5.

[0039] In the example of FIG. 5, the sofa node and the TV node share an edge E １ Edge E １ The label "on_right" indicates that the sofa is in front of the TV. The sofa node and the table node share an edge E with the label "on_right". ２ Edge E ２ The label indicates that the table is to the right of the sofa.

[0040] The TV node and the table node have an edge E with the label “on_left”. ３ Edge E ３ The label indicates that the table is on the left side of the TV. The edge E between the table node and the chair A node, and the edge E between the table node and the chair B node, are also labeled with labels indicating their respective positional relationships. ４ , E ５ The edges are connected by a label (front / behind / left / right / on / above / under / near, etc.) that indicates the spatial positional relationship. In the following explanation, the direction of the edge will be omitted as appropriate.

[0041] In this way, a Scene Graph is a graph-structured information in which multiple objects in a space are represented as nodes, and the relationship between two objects in a space, such as their positional relationship, is represented by edges. Scene Graphs are described, for example, in Reference 1. Reference 1: Tomu Tahara, et al., "Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph," 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020

[0042] ・Scene Graph with joint Scene Graph with joint is a graph structure that represents the joints of dynamic objects when they take a specific pose as nodes, and connects nodes with joint names (nodes that represent joints) and nodes that represent objects in space with edges that represent the relationships between them.

[0043] ・Integrated Scene Graph The Integrated Scene Graph is a graph structure information that integrates multiple Scene Graphs with joints into a single Scene Graph based on common nodes, and connects nodes with edges that indicate coefficients calculated by weighting the occurrence probability based on the graph distance from the joint node of the dynamic object. The Integrated Scene Graph is a Scene Graph that expresses the relationship between the joints of dynamic objects in a specific pose and objects in space, starting from the dynamic object.

[0044] ・Essential Scene Graph The Essential Scene Graph is generated by filtering edges in the Integrated Scene Graph based on a threshold. The Essential Scene Graph is a Scene Graph that expresses the dynamic object joints in a specific pose and the minimum necessary relationships with in-space objects for that specific pose, starting from the dynamic object.

[0045] <<Processing Flow of the Present Technology>> FIG. 6 is a diagram showing the flow of a series of processes in the information processing device 1.

[0046] The series of processes in the information processing device 1 can be divided into two processes: an important relationship extraction flow and an avatar reflection flow. The important relationship extraction flow and the avatar reflection flow each include the following processes. Details of each process will be described later.

[0047] 1. Important Relationship Extraction Flow (1-1) Graph Integration Processing The graph integration processing is a processing in which a Scene Graph with joint is generated multiple times based on real space information obtained by measuring the real space using the sensor device 2, and multiple Scene Graphs with joints are integrated into one Integrated Scene Graph. The graph integration processing includes the space recognition processing shown as the processing of step S1 in the upper part of Fig. 6 and the Scene Graph integration processing shown as the processing of step S2.

[0048] (1-2) Relationship Extraction Processing The relationship extraction processing is a processing for extracting relationships between objects originating from a dynamic object based on the appearance probability of the object in the space represented by the Integrated Scene Graph and the graph distance from the joint node of the dynamic object. A subgraph having an important relationship with respect to the user's pose is extracted from the Integrated Scene Graph as an Essential Scene Graph. The important relationship extraction processing shown as the processing of step S3 in the upper part of Fig. 6 corresponds to the relationship extraction processing.

[0049] The process of the important relationship extraction flow is repeated, and multiple Essential Scene Graphs are generated for each type of specific pose that appears in the user's actions. The action of "sitting on a chair" includes a pose where the knees are bent while sitting. The action of "lying down on a bed" includes a pose where the whole body is lying down.

[0050] 2. Avatar Reflection Flow (2-1) Related Object Search Process The related object search process searches for in-space objects included in the Scene Graph with joint of the virtual space that have the same affordance as the affordance of a related object represented by the Essential Scene Graph, and determines the area around the searched in-space object as the target area of ​​action. The target area of ​​action is a range that serves as a candidate location for placing an avatar. The related object search process includes a space / pose integration process shown as step S11 in the lower part of FIG. 6. As indicated by arrows A11 and A12, the Scene Graph indicating the spatial layout of the virtual space and the avatar pose information are used in the space / pose integration process. Furthermore, as indicated by arrow A13, the Essential Scene Graph generated by the important relationship extraction flow is used to determine the target area of ​​action in the space / pose integration process. Of the Essential Scene Graphs generated by the important relationship extraction flow, the Essential Scene Graph corresponding to the recognized user pose is used.

[0051] (2-2) Graph Distance Calculation Processing The graph distance calculation processing calculates the graph distance between the Essential Scene Graph and the Scene Graph with joint, which indicates the spatial layout of the virtual space when an avatar in a specific pose is placed in the action target range. The position where the graph distance is smallest is determined as the placement position of the avatar, and the specific pose at the determined placement position becomes the retarget pose. The graph distance calculation processing includes a Scene Graph comparison processing shown as the processing of step S12 in the lower part of Figure 6. As indicated by arrow A14, the Essential Scene Graph generated by the important relationship extraction flow is used in the Scene Graph comparison processing.

[0052] (2-3) Avatar Display Processing The avatar display processing is a processing for placing an avatar in the virtual space while maintaining consistency with the mesh of the object in the space. The avatar display processing is performed based on the retarget pose generated by the Scene Graph comparison processing.

[0053] <<Important Relationship Extraction Flow>> <Functional Configuration> Fig. 7 is a block diagram showing an example of the functional configuration of the important relationship extraction unit 11. The important relationship extraction unit 11 in Fig. 7 performs each process of the important relationship extraction flow.

[0054] The important relationship extraction unit 11 is composed of a space recognition unit 21, a Scene Graph with joint generation unit 22, an Integrated Scene Graph generation unit 23, and an Essential Scene Graph generation unit 24. At least some of the functional units shown in Fig. 7 are realized by executing a predetermined program on a computer constituting the information processing device 1. The important relationship extraction unit 11 receives as input real space information output from the sensor device 2.

[0055] The sensor device 2 is a device equipped with various sensors such as a color image sensor and a depth sensor. The sensor device 2 is installed in a room or the like in the real space where the user is present. The user to be measured goes about their daily life by sitting in a chair to eat meals or sitting on a sofa to watch TV. The color image output by the sensor device 2 shows not only the user performing various actions but also objects in the space in the room. The distance image measured by the sensor device 2 indicates the distance to the user and the distance to the objects in the space.

[0056] The spatial recognition unit 21 generates a 3D spatial recognition map based on image data of each frame supplied from the sensor device 2 as real space information. Specifically, the spatial recognition unit 21 performs segmentation (semantic segmentation and instance segmentation) such as PanopticFusion on the RGB image (2D image) that constitutes the real space information, and recognizes the label for each pixel. The spatial recognition unit 21 integrates the label into a 3D spatial map generated by SLAM using depth images, and generates a 3D spatial recognition map. PanopticFusion, one method of panoptic segmentation, is described in, for example, Reference 2. Reference 2: Gaku Narita, et al., “PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things”, Proc. IROS, 2019

[0057] Furthermore, the space recognition unit 21 generates a Scene Graph that indicates the spatial layout of the real space based on the 3D space recognition map. The space recognition unit 21 performs skeletal estimation based on the RGB image that constitutes the real space information, and generates skeletal information. The skeletal information includes the coordinates of each body part when the user assumes a specific pose. The space recognition unit 21 associates the skeletal information generated based on image data at the same time with the Scene Graph and outputs them.

[0058] The Scene Graph with joint generation unit 22 generates and outputs a Scene Graph with joint based on the skeleton information and the Scene Graph at each time supplied from the space recognition unit 21. The space recognition process (S1) in FIG. 6 is performed by the space recognition unit 21 and the Scene Graph with joint generation unit 22.

[0059] The Integrated Scene Graph generation unit 23 integrates multiple Scene Graphs with joints supplied from the Scene Graph with joint generation unit 22 into one Scene Graph with joint. The Integrated Scene Graph generation unit 23 integrates the Scene Graphs with joints by grouping together common nodes. The Integrated Scene Graph generation unit 23 also generates and outputs an Integrated Scene Graph by setting coefficients indicating the importance of relationships on edges connecting all common nodes. The Scene Graph integration process (S2) in FIG. 6 is performed by the Integrated Scene Graph generation unit 23.

[0060] The Essential Scene Graph generation unit 24 generates an Essential Scene Graph based on the Integrated Scene Graph supplied from the Integrated Scene Graph generation unit 23. The important relationship extraction process (S3) in Fig. 6 is performed by the Essential Scene Graph generation unit 24. Details of the processing of each unit will be described below.

[0061] <(1-1) Graph Integration Processing> FIG. 8 is a diagram showing an example of a Scene Graph with joint generated by the Scene Graph with joint generating unit 22. As shown in FIG.

[0062] When a user is taking a specific pose at any position in real space, a Scene Graph with joint as shown in Fig. 8 is generated based on the Scene Graph that shows the spatial layout of real-space objects at that time and skeletal information that shows the user's pose. Scene Graph with joint 1 to N shown in Fig. 8 are Scene Graph with joint generated based on measurement results at different times.

[0063] The Scene Graph with joint 1 consists of four nodes representing spatial objects: Sofa #1, Chair #1, Wall #1, and Floor #1, and seven joint nodes representing each of the user's physical parts. The Sofa #1 node and Chair #1 node are connected by an edge labeled "near," indicating that they are close to each other. Nodes of other objects in the space are also connected by edges labeled with labels indicating their relationships.

[0064] The Chair#1 node and the Spine Base#1 node are connected by an edge labeled "on." The Floor#1 node and the Left Leg#1 node, and the Floor#1 node and the Right Leg#1 node are also connected by edges labeled "on."

[0065] Similarly, Scene Graph with joint 2 consists of four nodes representing spatial objects: TV#1, Chair#1, Wall#1, and Floor#1, and seven joint nodes representing each of the user's physical parts. The TV#1 node and Chair#1 node are connected by an edge labeled "facing," indicating that they face each other. Nodes of other spatial objects are also connected by edges labeled with labels indicating their relationships.

[0066] Additionally, the TV#1 node and the Head#1 node are connected by an edge labeled "facing." The Chair#1 node and the Spine Base#1 node are connected by an edge labeled "on." The Floor#1 node and the Left Leg#1 node are connected by an edge labeled "on." The Floor#1 node and the Right Leg#1 node are connected by an edge labeled "above."

[0067] The Scene Graph with joint N consists of three nodes representing spatial objects: Chair #1, Wall #1, and Floor #1, and seven joint nodes representing each of the user's physical parts. The Chair #1 node and the Wall #1 node are connected by an edge labeled "in front of," indicating that Chair #1 is in front of Wall #1. Nodes of other objects in the space are also connected by edges labeled with labels indicating their relationships.

[0068] The Chair#1 node and the Spine Base#1 node are connected by an edge labeled "on." The Floor#1 node and the Left Leg#1 node, and the Floor#1 node and the Right Leg#1 node are also connected by edges labeled "on."

[0069] 8 are all Scene Graphs with joint 1 to N generated based on the measurement results when the user is sitting on Chair #1 and doing something. In addition to the Scene Graph with joint that shows the spatial layout when sitting on Chair #1, various Scene Graphs with joint are generated, such as a Scene Graph with joint that shows the spatial layout when sitting on a sofa, a Scene Graph with joint that shows the spatial layout when lying on a bed, and so on.

[0070] In this way, the important relationship extraction unit 11 having the Scene Graph with joint generation unit 22 has the function of an information processing unit that generates a Scene Graph with joint (first scene graph) in which multiple nodes (first nodes) representing in-space objects in real space are connected by edges (first edges) indicating the relationships between them, and nodes representing in-space objects and joint nodes (second nodes) are connected by edges (second edges) indicating the relationships between the in-space objects and physical parts when the user assumes a specific pose.

[0071] The Scene Graph with joints contains multiple joint nodes that represent the user's seven body parts as dynamic objects. Depending on the pose the user assumes, different joint nodes are connected to nodes that represent objects in space.

[0072] FIG. 9 is a diagram showing an example of Scene Graph with joint integration by the Integrated Scene Graph generating unit 23.

[0073] As shown on the right side of Figure 9, Scene Graph with joint 1 to N are integrated by grouping together common nodes. The integrated Scene Graph with joint is a Scene Graph consisting of six nodes representing spatial objects (Sofa #1, Chair #1, Wall #1, Floor #1, TV #1, and Chair #2) and seven joint nodes representing each of the user's physical parts. Each node is connected by an edge with a label indicating the relationship between each object and the probability of occurrence.

[0074] For example, the node for Chair #1 and the node for Spine Base #1 are connected by an edge labeled "on: 0.8" and an edge labeled "over: 0.2." The label "on: 0.8" indicates that the probability of occurrence of the relationship "on" between Chair #1 and Spine Base #1 is "0.8." The label "over: 0.2" indicates that the probability of occurrence of the relationship "over" between Chair #1 and Spine Base #1 is "0.2."

[0075] Similarly, the TV#1 node and the Head#1 node are connected by an edge labeled "facing: 0.2", and the Floor#1 node and the Left Leg#1 node, and the Floor#1 node and the Right Leg#1 node are connected by an edge labeled "on: 0.7" and an edge labeled "over: 0.3", respectively.

[0076] FIG. 10 is a diagram showing an example of the generation of an Integrated Scene Graph by the Integrated Scene Graph generation unit 23.

[0077] In the Integrated Scene Graph generation unit 23, the appearance probability of edges constituting the integrated Scene Graph with joint is weighted according to the graph distance from the joint node. In the example of Figure 10, among the nodes of the objects in the space, the nodes Chair #1, TV #1, and Floor #1 that are directly connected to the joint node are considered to be nodes in the first layer. In addition, the nodes Sofa #1, Wall #1, and Chair #2 that are connected to the nodes in the first layer are considered to be nodes in the second layer.

[0078] The appearance probability of an edge connecting a joint node and a node in the first layer is weighted by 1 / 1, and the appearance probability of an edge connecting a node in the first layer and a node in the second layer is weighted by 1 / 2. In this way, the Integrated Scene Graph generation unit 23 weights the appearance probability of each edge constituting the integrated Scene Graph with joint according to the distance from the joint node, and generates an Integrated Scene Graph (third scene graph) including edges whose coefficients are the weighted appearance probabilities.

[0079] The process of integrating multiple Scene Graphs with joint into an Integrated Scene Graph, which is information with a single graph structure, is performed according to the following rules: 1. Connect all nodes and edges included in the Scene Graph with joint. 2. Calculate the occurrence probability of edges connecting nodes, and integrate edges connecting common nodes. 3. Starting from the joint node, define a layer K (K = 1, 2, ..., N (N is the shortest travel distance from the joint node to the farthest node among the nodes of the objects in the space)) of nodes for each object in the space. Multiply the occurrence probability of edges connecting each node by 1 / K, and set this as the edge coefficient in the label.

[0080] The Integrated Scene Graph provides quantitative user-driven information on the relationship between the user's physical parts and objects in the space when the user is in a specific pose.

[0081] <(1-2) Relationship Extraction Process> FIG. 11 is a diagram showing an example of generation of an Essential Scene Graph by the Essential Scene Graph generation unit 24. As shown in FIG.

[0082] As shown in Fig. 11, the Essential Scene Graph is generated by retaining, from among the edges constituting the Integrated Scene Graph, edges that are labeled with coefficients exceeding a threshold, as well as nodes of objects in the space connected to joint nodes and edges between those nodes. In the example of Fig. 11, a value of 0.5 is used as the threshold, and a subgraph in which only the Chair #1 node and Floor #1 node remain, along with the joint nodes, is generated as the Essential Scene Graph.

[0083] The Essential Scene Graph (second Scene Graph) is a Scene Graph consisting of joint nodes and nodes representing objects in space that are commonly included in multiple Scene Graphs with joint. Nodes representing objects in space extracted from the Integrated Scene Graph as nodes constituting the Essential Scene Graph are nodes connected to joint nodes by edges labeled with coefficients exceeding a threshold. The coefficients set for the labels are values ​​calculated by weighting according to the graph distance from the joint nodes.

[0084] The process of extracting an Essential Scene Graph from an Integrated Scene Graph using threshold processing is performed according to the following rules: 1. The node in Spine Base #1 is the starting point. 2. The nodes to be left are searched for using threshold processing. (1) A threshold is set. (2) Of the edges connected to joint nodes, edges with coefficients below the threshold are deleted. (3) Nodes that are not connected to other nodes by edges are deleted. (4) When a joint node is considered a parent node, joint nodes that do not have a child node connected to them and the edges connected to them are deleted.

[0085] This makes it possible to extract the relationship between the specific pose and the in-space objects that are the minimum necessary for a specific pose at the user's starting point, which are dynamic objects.

[0086] For example, when considering the pose of "sitting on a chair," one chair on the floor is sufficient for the pose of "sitting on a chair." At the Integrated Scene Graph stage shown on the left side of Figure 11, two chairs exist in the graph, but by deleting unnecessary chairs, an Essential Scene Graph is extracted that includes information about the one chair that is the minimum required for the pose of "sitting on a chair." Similarly, an Essential Scene Graph is generated for each type of specific pose that appears when the user performs various actions, such as "sitting on a sofa" or "lying down on a bed."

[0087] The term "essential" here means that the Essential Scene Graph is a Scene Graph that extracts common elements among the in-space objects contained in multiple Scene Graphs with joints obtained when a user takes multiple poses with similar meanings, based on the appearance probability of the in-space objects and the weighting results according to the graph distance from the user's starting point. The Essential Scene Graph intuitively represents the in-space objects that are the minimum necessary for a specific pose and the relationship between that pose. Hereinafter, the in-space objects that are the minimum necessary for a specific pose, as represented by the Essential Scene Graph, will be referred to as related objects, where appropriate.

[0088] <<Avatar Reflection Flow>> <Functional Configuration> Fig. 12 is a block diagram showing an example of the functional configuration of the optimal position calculation unit 12A. The optimal position calculation unit 12A in Fig. 12 performs related object search processing and graph distance calculation processing in the avatar reflection flow.

[0089] 12, the optimal position calculation unit 12A is configured with a pose placement unit 31, a scene graph with joint generation unit 32, an inter-graph distance calculation unit 33, and a determination unit 34. At least some of the functional units shown in FIG. 12 are realized by a computer constituting the information processing device 1 executing a predetermined program.

[0090] The optimal position calculation unit 12A has a virtual space information DB and a pose information DB. The virtual space information DB is a DB of Scene Graphs that indicate the spatial layout of the virtual space. The pose information DB is a DB of information related to various poses of the avatar. The information related to the poses includes coordinate information of each body part of the avatar when it assumes each pose.

[0091] The pose placement unit 31 reads and acquires the Scene Graph of the virtual space from the virtual space information DB. The Scene Graph of the virtual space includes attribute information indicating the affordance of each in-space object indicated by the nodes that make up the Scene Graph.

[0092] FIG. 13 is a diagram showing examples of affordances. FIG. 13 shows the affordances of the in-space objects Floor, Stone, and Bed. In the virtual space, at least Floor, Stone, and Bed are placed as in-space objects. The affordances of Floor, Stone, and Bed are "stand able," "sittable," and "lie down able," respectively. Stone is an in-space object that is not a chair but can allow an avatar to sit on it. In this example, affordances correspond to the functions of the in-space objects.

[0093] Furthermore, the pose placement unit 31 reads and acquires information about the avatar's pose from the pose information DB. When the user is in the "sitting in a chair" pose, information about the sitting pose is acquired. The pose placement unit 31 places the avatar in the specific pose in the virtual space, and outputs information about the avatar placement and a Scene Graph of the virtual space to the Scene Graph with joint generation unit 32.

[0094] The Scene Graph with joint generation unit 32 generates and outputs a Scene Graph with joint of the virtual space based on the information supplied from the pose placement unit 31. The space / pose integration process (S11) in FIG. 6 is performed by the pose placement unit 31 and the Scene Graph with joint generation unit 32.

[0095] The graph distance calculation unit 33 calculates the graph distance between the Scene Graph with joint generated by the Scene Graph with joint generation unit 32 and the Essential Scene Graph, and outputs the calculation result.

[0096] If the distance between the Scene Graph with joint and the Essential Scene Graph is equal to or smaller than a threshold, the determination unit 34 determines the position of the avatar represented by the Scene Graph with joint as the optimal position and outputs information indicating the optimal position. The Scene Graph comparison process (S12) in FIG. 6 is performed by the graph distance calculation unit 33 and the determination unit 34.

[0097] The optimal position calculation unit 12A acquires the recognition result of the user's behavior at the start of the avatar reflection flow processing. For example, the spatial recognition unit 21 of the important relationship extraction unit 11 recognizes the user's behavior based on the measurement result from the sensor device 2, and the recognition result is supplied to the optimal position calculation unit 12A. The optimal position calculation unit 12A determines the placement position of the avatar based on an essential scene graph corresponding to a specific pose appearing in the recognized user's behavior and information about that pose. The optimal position calculation unit 12A functions as an information processing unit that acquires the recognition result of the user's behavior and reflects the user's behavior in the avatar's behavior.

[0098] FIG. 14 is a diagram showing an example of an Essential Scene Graph and a Scene Graph of a virtual space.

[0099] By comparing the Essential Scene Graph with the Scene Graph of the virtual space, in-space objects that have the same affordance as the related object are searched for among the in-space objects of the virtual space. The Essential Scene Graph shown on the left side of Figure 14 is the Essential Scene Graph generated when the user is in the "sitting in a chair" pose. Chair #1 is identified as the related object, and Stone #1, an in-space object that has the same affordance as Chair #1, is searched for. Both Chair #1 and Stone #1 are in-space objects that have the affordance of "sittable."

[0100] As shown in the lower right of Fig. 14, the pose placement unit 31 determines the area around Stone #1, which has the affordance of "sittable," as the placement position of the avatar, and a Scene Graph with joint indicating the spatial layout in that state is generated by the Scene Graph with joint generation unit 32. In the example in the lower right of Fig. 14, an avatar in a skeleton state is placed. The process of determining the area around Stone #1, which has the affordance of "sittable," as the placement position of the avatar is the related object linking process included in the space / pose integration process in Fig. 6.

[0101] By positioning an avatar around an in-space object that has the same affordance as the related object, it is possible to have the avatar perform an action with the same meaning as the user's action, even if an in-space object of the same type as the related object does not exist in the virtual space. In the example of Figure 14, the action of "sitting" is expressed by the avatar using Stone #1 instead of a Chair. The user's action is reflected in the avatar's action based on the Scene Graph with joint that shows the state in which the avatar is assuming the "sitting" pose using Stone #1 that has the same affordance as Chair #1 used by the user.

[0102] <(2-2) Graph Distance Calculation Processing> FIG. 15 is a diagram showing an example of the calculation of the graph distance by the graph distance calculation unit 33. In FIG.

[0103] The Scene Graph with joint shown on the left side of FIG. 15 is a Scene Graph with joint that shows the spatial layout of the virtual space generated by the Scene Graph with joint generation unit 32. Some nodes and edges are omitted. In the example of FIG. 15, a Scene Graph with joint is generated that includes two nodes representing Stone #1 and Floor #1 as spatial objects, and seven joint nodes representing each of the avatar's physical parts. The Stone #1 node and the Spine Base #1 node are connected by an edge labeled "on." The Floor #1 node and the Left Leg #1 node, and the Floor #1 node and the Right Leg #1 node are also connected by edges labeled "on."

[0104] The Scene Graph with joint generated by the Scene Graph with joint generation unit 32 is graph-structured information (first scene graph) in which multiple nodes (first nodes) representing in-space objects existing in the virtual space are connected by edges (first edges) indicating the relationships between them, and nodes representing in-space objects and joint nodes representing physical parts of an avatar are connected by edges (second edges) indicating the relationships between the in-space objects and physical parts when the avatar assumes the same pose as a specific pose that appears in the recognized user behavior.

[0105] The distance between the graphs of the Scene Graph with joint and the Essential Scene Graph, which show the spatial layout of such a virtual space, is calculated as shown by the bidirectional arrow. The calculation of the distance between the graphs is performed according to the following rules. 1. The spine base of both graphs is the starting point. 2. Cost c = 0 is set (cost is initialized). 3. The following processes are performed: (1) Edges that are not in the Essential Scene Graph are searched for in the Scene Graph with joint of the virtual space, and the edges found by the search and the nodes connected to those edges are deleted. Cost c = the number of deleted edges and nodes is set. (2) Edges in the Essential Scene Graph are added to the Scene Graph with joint of the virtual space. 1 is added as the cost when adding a node, and a coefficient defined in the Essential Scene Graph (occurrence probability x weight 1 / K according to the distance between the graphs) is added as the cost when adding an edge.

[0106] This rule is used when calculating the graph editing cost as the graph distance. In the example of Figure 15, the graph distance (cost c) = 3 can be calculated as follows: 1. Set cost c = 0. 2. Delete the node on Floor #1 and add cost c = 1. 3. Delete the two edges connected to the node on Floor #1 and add cost c = 2.

[0107] 16 is a diagram showing another example of the calculation of the distance between graphs. The distance between graphs (cost c) = 4.8 is calculated as follows.

[0108] 1. Set cost c = 0. 2. Delete the Floor #1 node and add cost c = 1. 3. Delete the three edges connected to the Floor #1 node and add cost c = 3. 4. The edge connecting the Chair #1 node and the Spine Base #1 node is an edge that exists in the Essential Scene Graph but does not exist in the Scene Graph with joint. An edge corresponding to this edge (the edge connecting the Stone #1 node and the Spine Base #1 node) is added to the Scene Graph with joint in the virtual space and add cost c = 0.8. In Figure 16, the edge connecting the Stone #1 node and the Spine Base #1 node is shown with a dashed line, indicating that this edge is an added edge.

[0109] As described above, the Essential Scene Graph used in calculating the graph distance with the Scene Graph with joint in the virtual space is graph-structured information (second scene graph) configured with nodes (other first nodes) indicating in-space objects that are commonly included in a plurality of Scene Graphs with joint (other first scene graphs) generated for the real space. The Scene Graph with joint generated for the real space is graph-structured information in which a plurality of nodes indicating in-space objects in the real space are connected by edges (other first edges) indicating their relationships, and nodes indicating in-space objects and joint nodes (other second nodes) indicating physical parts of the user are connected by edges (other second edges) indicating the relationships between the in-space objects and the physical parts when the user assumes a specific pose.

[0110] As described above, the closer the graph structures of the Essential Scene Graph and the Essential Scene Graph in the virtual space are, the smaller the required value for the graph distance.The positions of each body part of the avatar indicated by the Scene Graph with joint where the graph distance with the Essential Scene Graph is below the threshold are used as the placement position of the avatar.

[0111] The determination unit 34 determines whether the inter-graph distance calculated as described above is equal to or less than a threshold. If it is determined that the inter-graph distance is equal to or greater than the threshold, the avatar is rearranged, and the same process is repeated a predetermined number of times. The avatar is rearranged so that the avatar is placed in a position different from the previous position.

[0112] As described above, the position of an avatar in virtual space is determined based on the graph distance between Scene Graphs. This allows for a reduction in the amount of calculation compared to calculations based on the geometric shapes of the avatar and objects in the virtual space. Furthermore, by narrowing down the avatar's placement position to positions around related objects, it is possible to reduce the amount of calculation compared to placing an avatar from all possible positions in the virtual space and calculating the graph distance.

[0113] <(2-3) Avatar display processing> After the placement position of an avatar taking a specific pose is determined, the avatar's pose is corrected to match the geometric shape of the in-space object using a method such as PLACE. The pose is corrected based on the mesh information and pose information of the in-space object in the virtual space to prevent the avatar from being displayed as being embedded in the in-space object. PLACE is described in Reference 3. Reference 3: Proximity Learning of Articulation and Contact in 3D Environments, Siwei Zhang, Yan Zhang et al.

[0114] Through the above process, the user's action of sitting on a chair is reflected as the action of an avatar sitting on Stone #1 in the virtual space. Similarly, when the user performs other actions such as sitting on a sofa or lying down on a bed, these are reflected as the action of an avatar using an in-space object with the same affordance as a sofa or a bed.

[0115] Even if the spatial layouts of the real space and the virtual space are different, it is possible to make the avatar perform the same actions as the user in a natural pose while maintaining semantic consistency. Furthermore, it is possible to place avatars in virtual spaces with various spatial layouts in real time processing, without the need for work to adjust for differences in spatial layout. In other words, it is possible to easily reflect the user's actions in the real space as the actions of an avatar in the virtual space.

[0116] <<Configuration of Information Processing Device 1>> Fig. 17 is a block diagram showing an example configuration of the information processing device 1. Of the configuration shown in Fig. 17, the same components as those described above are denoted by the same reference numerals. Duplicate explanations will be omitted.

[0117] 17 , the information processing device 1 is provided with an important relationship extraction unit 11 and an avatar reflection processing unit 12. The avatar reflection processing unit 12 is composed of an optimal position calculation unit 12A and a display control unit 12B. The display control unit 12B performs the above-mentioned avatar display process and controls the display of the avatar. That is, the display control unit 12B places the avatar in the virtual space based on the optimal position generated by the optimal position calculation unit 12A, and displays the avatar by adjusting the pose, etc.

[0118] FIG. 18 is a block diagram showing another example of the configuration of the information processing device.

[0119] As shown in A of Fig. 18, the important relationship extraction unit 11 and the avatar reflection processing unit 12 may be realized in the same device, or as shown in B of Fig. 18, the important relationship extraction unit 11 and the avatar reflection processing unit 12 may be realized in different devices. In the example of B of Fig. 18, the important relationship extraction unit 11 is realized in an information processing device 1-1, and the avatar reflection processing unit 12 is realized in an information processing device 1-2. The information processing device 1-1 and the information processing device 1-2 constitute an information processing system. The information processing device 1-1 and the information processing device 1-2 are connected via a network such as the Internet or a LAN.

[0120] <<Modifications>> Although the case where the behavior of a user in real space is reflected in the behavior of an avatar in a virtual space has been described, this technology can also be applied to a case where the behavior of a user in real space is reflected in the behavior of an AR character expressed using the real space. The AR character is a CG (Computer Graphics) character that is displayed overlaid on the real space. The AR character is observed using an AR device such as an HMD or a smartphone.

[0121] This technology can also be applied to cases where the behavior of an avatar in a virtual space is reflected in the behavior of a robot in a real space. In other words, as long as the spatial layout can be represented by a Scene Graph, including the physical parts of dynamic objects, the real space can be the source space, or the virtual space can be the source space. Conversely, the real space can be the destination space, or the virtual space can be the destination space.

[0122] Similarly, the present technology can be applied to cases where the behavior of an avatar in a virtual space is reflected in the behavior of an avatar in another virtual space, and can also be applied to cases where the behavior of a user in the real space is reflected in the behavior of an AR character in the real space. In this way, the combination of the source space as the first space and the destination space as the second space can be changed as desired.

[0123] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0124] 19 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. For example, the information processing device 1 is configured by a computer having the configuration shown in FIG.

[0125] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0126] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0127] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0128] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0129] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0130] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0131] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0132] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0133] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0134] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0135] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0136] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0137] (1) An information processing device comprising an information processing unit that generates, based on measurement results from a sensor, a first scene graph, which is graph-structured information in which a plurality of first nodes representing a plurality of objects in space are connected by first edges that indicate relationships between the objects in space, and a second node representing a physical part of a dynamic object is connected by a second edge that indicates a relationship between the object in space represented by the first node and the physical part when the dynamic object assumes a specific pose. (2) The information processing device described in (1), wherein the information processing unit generates the first scene graph in which the first node is connected by the second edge to a second node corresponding to the specific pose among a plurality of second nodes that respectively represent a plurality of the physical parts. (3) The information processing device described in (1) or (2), wherein the information processing unit generates a second scene graph composed of the second node and the first node included in common in a plurality of the first scene graphs. (4) The information processing device according to (3), wherein the information processing unit generates the second scene graph including the first node extracted based on a graph distance between the second node and the first node among the first nodes included in common to a plurality of the first scene graphs. (5) The information processing device according to (4), wherein the information processing unit generates a third scene graph by integrating the first node included in common to a plurality of the first scene graphs and the first edge indicating a relationship between the same objects in the space, and generates the second scene graph based on the third scene graph. (6) The information processing device according to any of (3) to (5), wherein the information processing unit generates the second scene graph for each type of the specific pose.(7) An information processing method, in which an information processing device connects a plurality of first nodes representing a plurality of intraspace objects, respectively, with first edges indicating relationships between the intraspace objects, and generates a first scene graph, which is graph-structured information in which the first nodes and second nodes representing physical parts of a dynamic object are connected with second edges indicating relationships between the intraspace objects represented by the first nodes and the physical parts when the dynamic object assumes a specific pose, based on measurement results by a sensor. (8) A program, in which a computer executes a process of generating a first scene graph, which is graph-structured information in which the first nodes representing a plurality of intraspace objects, respectively, are connected with first edges indicating relationships between the intraspace objects, and the first nodes and second nodes representing physical parts of a dynamic object are connected with second edges indicating relationships between the intraspace objects represented by the first nodes and the physical parts when the dynamic object assumes a specific pose, based on measurement results by a sensor. (9) An information processing device comprising: an information processing unit that acquires a recognition result of an action of a first dynamic object performed using a first intra-space object existing in a first space, and reflects the action of the first dynamic object in the action of the second dynamic object based on a first scene graph that is graph-structured information in which a plurality of first nodes representing a plurality of second intra-space objects existing in a second space are connected by first edges that indicate relationships between the second intra-space objects, and a second node that indicates a physical part of the second dynamic object is connected by second edges that indicate relationships between the second intra-space objects represented by the first nodes and the physical part when the second dynamic object assumes a specific pose that appears in the recognized action of the first dynamic object. (10) The information processing device according to (9), wherein the information processing unit causes the second dynamic object to perform an action using the second intra-space object that has the same function as the first intra-space object used by the first dynamic object.(11) The information processing device according to (9) or (10), wherein the information processing unit acquires a recognition result of an action of a person existing as the first dynamic object in the first space, which is a real space, recognized based on a measurement result by a sensor, and reflects the action of the person in an action of an avatar existing as the second dynamic object in the second space, which is a virtual space. (12) The information processing device according to any of (9) to (11), wherein the information processing unit reflects the action of the first dynamic object in an action of the second dynamic object based on the first scene graph indicating a state of the second space when the second dynamic object takes the specific pose, using the second intra-space object having the same function as the first intra-space object used by the first dynamic object. (13) The information processing device according to any of (9) to (12), wherein the information processing unit acquires a second scene graph constituted by other first nodes indicating the first intra-space objects included in common in a plurality of other first scene graphs generated as the first scene graphs targeting the first space. (14) The information processing device according to (13), wherein the other first scene graph is information with a graph structure in which a plurality of the other first nodes are connected by other first edges indicating relationships between the objects in the first space, and the other first nodes and other second nodes indicating other physical parts of the first dynamic object are connected by other second edges indicating relationships between the objects in the first space indicated by the other first nodes and the other physical parts when the first dynamic object takes the specific pose, and the second scene graph is information with a graph structure in which the other first nodes and the other second nodes included in common to a plurality of the other first scene graphs are connected by the other second edges. (15) The information processing device according to (13), wherein the information processing unit determines the position in the second space of the second dynamic object taking the specific pose based on a graph distance between the first scene graph and the second scene graph.(16) An information processing method in which an information processing device acquires recognition results of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, connects a plurality of first nodes each representing a plurality of second intra-space objects existing in a second space with a first edge indicating the relationship between the second intra-space objects, and reflects the behavior of the first dynamic object in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information in which the first nodes and second nodes indicating physical parts of the second dynamic object are connected with second edges indicating the relationship between the second intra-space object indicated by the first nodes and the physical part when the second dynamic object takes a specific pose that appears in the recognized behavior of the first dynamic object. (17) A program that causes a computer to execute a process of acquiring a recognition result of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, connecting a plurality of first nodes that respectively represent a plurality of second intra-space objects existing in a second space with first edges that indicate the relationship between the second intra-space objects, and reflecting the behavior of the first dynamic object in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information that connects the first nodes and second nodes that indicate physical parts of the second dynamic object with second edges that indicate the relationship between the second intra-space object represented by the first nodes and the physical part when the second dynamic object takes a specific pose that appears in the recognized behavior of the first dynamic object.

[0138] REFERENCE SIGNS LIST 1 Information processing device, 2 Sensor device, 11 Important relationship extraction unit, 12 Avatar reflection processing unit, 12A Optimal position calculation unit, 12B Display control unit, 21 Space recognition unit, 22 Scene graph with joint generation unit, 23 Integrated scene graph generation unit, 24 Essential scene graph generation unit, 31 Pose placement unit, 32 Scene graph with joint generation unit, 33 Inter-graph distance calculation unit, 34 Determination unit

Claims

1. An information processing device having an information processing unit that generates a first scene graph, which is graph-structured information in which multiple first nodes each representing multiple in-space objects are connected by first edges that indicate the relationship between the in-space objects, and the first nodes and second nodes representing physical parts of a dynamic object are connected by second edges that indicate the relationship between the in-space objects represented by the first nodes and the physical parts when the dynamic object assumes a specific pose, based on measurement results from a sensor.

2. The information processing device according to claim 1, wherein the information processing unit generates the first scene graph in which the first node and the second node corresponding to the specific pose among the plurality of second nodes each representing a plurality of the body parts are connected by the second edge.

3. The information processing device according to claim 1, wherein the information processing unit generates a second scene graph configured from the second node and the first node that is included in common in a plurality of the first scene graphs.

4. The information processing device according to claim 3, wherein the information processing unit generates the second scene graph including the first node extracted based on the graph distance between the first node and the second node from among the first nodes commonly included in a plurality of the first scene graphs.

5. The information processing device according to claim 4, wherein the information processing unit generates a third scene graph by integrating the first nodes that are commonly included in multiple first scene graphs and the first edges that indicate relationships between objects in the same space, and generates the second scene graph based on the third scene graph.

6. The information processing device according to claim 3, wherein the information processing unit generates the second scene graph for each type of the specific pose.

7. An information processing method in which an information processing device generates a first scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of in-space objects are connected by a first edge indicating the relationship between the in-space objects, and the first nodes are connected to second nodes representing physical parts of a dynamic object by a second edge indicating the relationship between the in-space objects represented by the first nodes and the physical parts when the dynamic object assumes a specific pose, based on measurement results from a sensor.

8. A program that causes a computer to execute a process of generating a first scene graph based on the results of measurements by a sensor, the first scene graph being information of a graph structure in which a plurality of first nodes each representing a plurality of objects in space are connected by a first edge that indicates the relationship between the objects in space, and the first nodes are connected to second nodes representing physical parts of a dynamic object by a second edge that indicates the relationship between the objects in space represented by the first nodes and the physical parts when the dynamic object assumes a specific pose.

9. An information processing device comprising an information processing unit that acquires recognition results of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, connects a plurality of first nodes each representing a plurality of second intra-space objects existing in a second space with a first edge indicating the relationship between the second intra-space objects, and reflects the behavior of the first dynamic object in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information in which the first nodes and second nodes indicating physical parts of the second dynamic object are connected with second edges indicating the relationship between the second intra-space object indicated by the first nodes and the physical part when the second dynamic object assumes a specific pose that appears in the recognized behavior of the first dynamic object.

10. The information processing device according to claim 9, wherein the information processing unit causes the second dynamic object to perform an action using the second in-space object having the same function as the first in-space object used by the first dynamic object.

11. The information processing device according to claim 9, wherein the information processing unit acquires a recognition result of the behavior of a person existing as the first dynamic object in the first space, which is a real space, recognized based on measurement results from a sensor, and reflects the behavior of the person in the behavior of an avatar existing as the second dynamic object in the second space, which is a virtual space.

12. The information processing device according to claim 9, wherein the information processing unit reflects the behavior of the first dynamic object in the behavior of the second dynamic object based on the first scene graph indicating the state of the second space when the second dynamic object assumes the specific pose using the second intra-space object having the same function as the first intra-space object used by the first dynamic object.

13. The information processing device according to claim 9, wherein the information processing unit acquires a second scene graph composed of other first nodes indicating objects in the first space that are commonly included in other first scene graphs generated as the first scene graphs targeting the first space.

14. The information processing device described in claim 13, wherein the other first scene graph is graph-structured information in which a plurality of the other first nodes are connected by other first edges indicating the relationships between the objects in the first space, and the other first nodes and other second nodes indicating other physical parts of the first dynamic object are connected by other second edges indicating the relationships between the objects in the first space indicated by the other first nodes and the other physical parts when the first dynamic object assumes the specific pose, and the second scene graph is graph-structured information in which the other first nodes and the other second nodes included in common to a plurality of the other first scene graphs are connected by the other second edges.

15. The information processing device according to claim 13, wherein the information processing unit determines the position in the second space of the second dynamic object taking the specific pose based on the graph distance between the first scene graph and the second scene graph.

16. An information processing method in which an information processing device acquires recognition results of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, and reflects the behavior of the first dynamic object in the behavior of the second dynamic object based on a first scene graph, which is graph-structured information in which a plurality of first nodes each representing a plurality of second intra-space objects existing in a second space are connected by first edges indicating the relationship between the second intra-space objects, and the first nodes and second nodes indicating physical parts of the second dynamic object are connected by second edges indicating the relationship between the second intra-space objects indicated by the first nodes and the physical parts when the second dynamic object assumes a specific pose that appears in the recognized behavior of the first dynamic object.

17. A program causing a computer to execute a process of acquiring recognition results of the behavior of a first dynamic object performed using a first intra-space object existing in a first space, connecting a plurality of first nodes representing a plurality of second intra-space objects existing in a second space with first edges indicating the relationships between the second intra-space objects, and connecting the first nodes and second nodes indicating physical parts of the second dynamic object with second edges indicating the relationships between the second intra-space objects represented by the first nodes and the physical parts when the second dynamic object assumes a specific pose that appears in the recognized behavior of the first dynamic object, based on a first scene graph, which is graph-structured information.

Images