Information processing device, information processing method, and recording medium
The information processing device uses spatio-temporal scene graphs to synchronize and predict avatar actions, addressing delays and ensuring natural behavior, thus improving remote communication by aligning avatar actions with intended actions and enhancing the communication experience.
Patent Information
- Application Number
- PCT/JP2024/015671
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-14
AI Technical Summary
Existing remote communication systems using avatars face challenges in smooth communication due to delays in behavioral data transmission, leading to discrepancies between expected and actual actions of avatars, which disrupt the communication experience.
An information processing device and method that generates and utilizes spatio-temporal scene graphs to predict and synchronize the actions of remote avatars by analyzing sensor data, adjusting action speed, and correcting movement destinations to minimize latency and maintain natural avatar behavior.
Enables smooth and natural remote communication by compensating for transmission delays and ensuring that avatars' actions align closely with the intended actions, enhancing the communication experience.
Smart Images

Figure JP2024015671_14112024_PF_FP_ABST
Abstract
Description
Information processing device, information processing method, and recording medium
[0001] The present technology relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that enable people in remote locations to smoothly communicate with each other via their avatars.
[0002] One technology for replicating human behavior with an avatar is to use wearable or optical motion capture systems. By using a motion capture system to measure the position of a person's joints and obtain 3D pose data of the entire body with low latency and high frequency, it is possible to present an avatar that replicates the same actions as the person with high tracking accuracy.
[0003] International Publication No. 2022 / 209450 International Publication No. 2020 / 012727
[0004] There is a remote communication system that allows people in different locations to communicate with each other through their avatars. Each person wears a display device such as AR (Augmented Reality) glasses and communicates by talking to the other person's avatar, which appears to be present in the same space as them, and by reacting to the words and actions of the other person's avatar.
[0005] The actions of the avatar that one person is looking at are performed based on the actual actions of the remote person, and the action data is transmitted. If there is a discrepancy between the timing of the other person's expected action and the timing of the other person's actual action due to delays in the transmission of action data, appropriate communication becomes difficult.
[0006] This technology was developed in light of these circumstances, and enables people in distant locations to smoothly communicate with each other via their avatars.
[0007] An information processing device according to a first aspect of the present technology generates a first scene graph based on measurement results from a sensor indicating a plurality of objects including a first person looking at a second avatar that reproduces an action corresponding to an action of a second person and the second person looking at the first avatar that reproduces an action corresponding to an action of the first person, in which nodes indicating the plurality of objects at different times are connected by change edges that indicate temporal changes in state; and includes an information processing unit that selects an action to be reproduced by the second avatar based on similar scene graphs that include a graph structure similar to that of the first scene graph and are included in a plurality of second scene graphs that are generated in advance based on measurement results from a sensor indicating a plurality of arbitrary objects including a plurality of arbitrary people, in which nodes indicating the plurality of arbitrary objects at different times are connected by change edges that indicate temporal changes in state.
[0008] An information processing device according to a second aspect of the present technology includes a generation unit that generates, based on measurement results of a sensor indicating a plurality of objects including a plurality of people, a scene graph in which nodes indicating the plurality of objects at each time are connected by relationship edges indicating relationships between the plurality of objects, and nodes indicating the plurality of objects at different times are connected by change edges indicating changes in state over time.
[0009] In a first aspect of the present technology, a first scene graph is generated based on measurement results of a sensor indicating a plurality of objects including a first person looking at a second avatar that reproduces an action corresponding to an action of a second person, and the second person looking at the first avatar that reproduces an action corresponding to an action of the first person, in which nodes indicating the plurality of objects at different times are connected by change edges that indicate changes in state over time; and an action to be reproduced by the second avatar is selected based on similar scene graphs including a graph structure similar to that of the first scene graph, which are included in a plurality of second scene graphs that are generated in advance based on measurement results of a sensor indicating a plurality of arbitrary objects including any plurality of people, and in which nodes indicating the plurality of arbitrary objects at different times are connected by change edges that indicate changes in state over time.
[0010] In a second aspect of the present technology, based on the measurement results of a sensor indicating a plurality of objects including a plurality of people, a scene graph is generated in which nodes indicating the plurality of objects at each time are connected by relationship edges indicating the relationships between the plurality of objects, and nodes indicating the plurality of objects at different times are connected by change edges indicating temporal changes in state.
[0011] 1 is a diagram illustrating an example of communication realized using an AR application. A diagram illustrating an example of discrepancy in remote communication. A diagram illustrating a series of processing flows in an information processing system according to an embodiment of the present technology. A diagram illustrating an example of processing in a human behavior learning phase. A diagram illustrating an example of a graph showing spatial information. A diagram illustrating an example of a graph showing person attribute information. A diagram illustrating an example of a graph showing relationship information. A flowchart illustrating spatiotemporal scene graph construction and recording processing. A diagram illustrating an example of a graph showing a recognition result. A diagram illustrating an example of a spatial scene graph. A diagram illustrating an example of time-series state likelihood. A diagram illustrating an example of a spatiotemporal scene graph. A diagram illustrating an example of processing in a human behavior reproduction phase. A flowchart illustrating human behavior candidate prediction processing. A diagram illustrating an example of inter-graph distance. A diagram illustrating an example of a spatiotemporal scene graph including predicted behavior candidates. A flowchart illustrating predicted behavior space adaptation processing. A diagram illustrating an example of calculation of a magnification of an avatar's behavior speed. A diagram illustrating an example of change of destination. A block diagram illustrating an example of the functional configuration of an information processing device. A block diagram illustrating an example of the functional configuration of an information processing device. A block diagram illustrating an example of the configuration of an information processing system. A block diagram illustrating an example of the configuration of a computer.
[0012] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Processing flow in information processing system 3. Human behavior learning phase 4. Human behavior reproduction phase 5. Summary 6. Configuration of each device 7. Modified example
[0013] <<Outline of this technology>> An AR application that applies this technology is an application that allows a real person (hereinafter simply referred to as "person") wearing an AR display device to communicate with a person in a remote space. By wearing the AR display device, the other person's avatar is displayed as if they were in the person's vicinity. Each person communicates with the other person in a remote space by talking to the other person's avatar displayed by the AR display device and acting in response to the other person's avatar's words and actions.
[0014] FIG. 1 is a diagram showing an example of communication realized using an AR application.
[0015] Space A shown on the right side of FIG. 1 is a space where person A is present. An AR display device 1A worn by person A displays an avatar of person B, who will be communicating with person A. The AR display device 1A is, for example, an optical see-through HMD (Head Mounted Display). In the present disclosure, an optical see-through AR HMD having a shape similar to glasses may be referred to as AR glasses. The display of person B's avatar is controlled so that person B behaves in a similar manner to person B in response to the recognition of person B's behavior.
[0016] The AR display device 1A is provided with a sound output function using a speaker or the like and a sound detection function using a microphone. The sound emitted by person B is output from the AR display device 1A as the sound of person B's avatar and delivered to person A. By seeing the avatar behaving in accordance with the action that person B is actually taking in space B and by listening to the sound of person B emitted by the avatar, person A can get the feeling that person B, represented by the avatar, is close to him or her.
[0017] On the other hand, space B shown on the left side of Figure 1 is the space where person B is located. Space A and space B are remote spaces located at different locations. An avatar of person A, who is the communication partner, is displayed on the AR display device 1B worn by person B. Like the AR display device 1A, the AR display device 1B is also an optical see-through type AR glass. The display of person A's avatar is controlled so that it behaves in the same way as person A in response to the recognition of person A's behavior.
[0018] The AR display device 1B is provided with a voice output function using a speaker or the like and a voice detection function using a microphone. The voice emitted by person A is output from the AR display device 1B as the voice of person A's avatar and delivered to person B. By seeing the avatar behaving in accordance with the actions that person A is actually performing in space A and by listening to the voice of person A emitted by the avatar, person B can get the feeling that person A, represented by the avatar, is nearby.
[0019] In the example of Fig. 1, the avatar of person B, which person A is looking at, waves its hand in response to person B waving his hand. The fact that person B has waved his hand is recognized based on sensor data, and control is performed to reflect this in the behavior of person B's avatar, which person A is looking at.
[0020] 1, the dot patterns applied to the avatars of person A and person B indicate that each avatar is a virtual object displayed by an AR display device. Although the avatars are depicted as human-shaped objects, virtual objects with shapes different from humans, such as objects representing real animals or virtual creatures, may also be used as avatars.
[0021] Instead of optical see-through AR glasses, other display devices such as a video see-through HMD or a smartphone may be used as the AR display devices 1A and 1B. Hereinafter, when it is not necessary to distinguish between the AR display devices 1A and 1B, they will be collectively referred to as the AR display device 1.
[0022] In this way, an AR application to which this technology is applied enables remote communication via each person's avatar.
[0023] When reproducing a person's actions with an avatar, as in the remote communication shown in Figure 1, a wearable or optical motion capture system is used to detect the person's actions. By measuring the joint positions of a person using a motion capture system, 3D pose data of the entire body can be obtained with low latency and high frequency, but in order to reproduce the same actions as the person's actions as the actions of an avatar, the measurement space and the reproduction space must have similar environments.
[0024] This technology uses spatial retargeting to make virtual characters such as avatars behave in the same way in different spaces. Spatial retargeting is described, for example, in Reference 1: [Tahara+, ISMAR2020] T. Tahara, T. Seno, G. Narita and T. Ishikawa, “Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph,” ISMAR, 2020.
[0025] Spatial retargeting is a technology that abstracts and expresses the meaning of a person's actions in one space, and then reproduces the same actions as avatar actions in another space. Although spatial retargeting can avoid the spatial constraints mentioned above, discrepancies in remote communication can occur due to the time delay required to recognize a person's actions, transmit information about the actions, and reproduce them as avatar actions.
[0026] FIG. 2 is a diagram showing an example of a discrepancy in remote communication.
[0027] The upper part of Figure 2 shows the states of Space A and Space B at time t, and the lower part shows the states of Space A and Space B at time t+1. At each time, Person A in Space A and Person B in Space B communicate with each other via avatars as described with reference to Figure 1. Audio is transmitted with low latency.
[0028] At time t, person B, who is sitting on a sofa in space B shown on the left, proposes to person A that they sit on the sofa and watch TV together, as shown in speech bubble #1. The proposal to person A, who is standing on the floor, is made by person B speaking to the avatar of person A that person B is looking at. The voice of person B is output with low delay as the voice of person B's avatar by the AR display device 1A worn by person A in space A, as shown in speech bubble #2.
[0029] When person A responds to person B's suggestion with voice as shown in speech bubble #3, person A's voice is output with low latency as the voice of person A's avatar by AR display device 1B worn by person B in space B as shown in speech bubble #4. Upon hearing person A's reply, person B expects person A (person A's avatar) to take the action of sitting on the sofa and watching TV. At this point, person A, who responded to person B's suggestion, will begin the action of approaching the sofa in space A and sitting down.
[0030] Time t+1 is, for example, the timing immediately after person A sits down on the sofa in space A. At time t+1, person A sits down on the sofa in space A and starts watching TV, but as described above, it takes time for person A's behavior to be recognized, information about the behavior to be transmitted, and for the behavior to be reproduced as an avatar's behavior, so person A's avatar in space B remains standing, just as it was at time t. Because the behavior of person A's avatar at time t+1 differs from the behavior expected by person B, the situation shown in FIG. 2 is one in which a discrepancy in communication has occurred.
[0031] In remote communication enabled by AR applications, processing is performed to compensate for the time delay required between the initiation of a remote person's action and the reproduction of that action by an avatar, enabling smooth communication even when remote people communicate with each other via avatars.
[0032] <<Processing Flow in Information Processing System>> FIG. 3 is a diagram showing a series of processing flows in an information processing system according to an embodiment of the present technology.
[0033] As shown in Fig. 3, a series of processes in the information processing system consists of two processing phases: a "human behavior learning phase" and a "human behavior reproduction phase." The information processing system includes a configuration on the human behavior learning side and a configuration on the human behavior reproduction side.
[0034] 3, the human behavior learning system includes a measurement device 11 and an information processing device 12. The measurement device 11 and the information processing device 12 are connected via wired or wireless communication.
[0035] The measurement device 11 is a sensor device equipped with various sensors such as a color image sensor and a depth sensor. The measurement device 11 is installed in a room where real people are present. In the example of FIG. 3 , the measurement device 11 is installed in a space where person A and person B are present, but the measurement device 11 may also be installed in a space where multiple people other than person A and person B are present. The multiple people to be measured communicate with each other by talking and using gestures.
[0036] The measurement device 11 measures a person as a measurement target and outputs measurement data such as a color image and a distance image to the information processing device 12. The color image output by the measurement device 11 shows not only the person as a measurement target, but also surrounding objects such as furniture around the person. The distance image measured by the measurement device 11 indicates the distance to the person and the distance to surrounding objects such as furniture.
[0037] The information processing device 12 analyzes the color image supplied from the measurement device 11 as measurement data and performs image recognition, etc., to recognize human attributes such as age, sex, height, etc. The information processing device 12 also analyzes the color image and distance image to recognize objects around the person, and to recognize relationships between objects and between people, etc. The information processing device 12 generates a spatiotemporal scene graph based on the recognition results and records it in a time-series human behavior DB.
[0038] The time-series human behavior DB is a collection of spatiotemporal scene graphs. The processing of the human behavior learning phase is repeatedly performed with the space where various people exist as the measurement target. The time-series human behavior DB records information on multiple spatiotemporal scene graphs generated with the space where various people exist as the measurement target.
[0039] The time-series human behavior DB generated by the information processing device 12 is supplied to an information processing device 22A installed in space A and an information processing device 22B installed in space B as part of the human behavior reproduction configuration. The information processing device 12, the information processing device 22A, and the information processing device 22B are each connected via a network such as the Internet. Space A and space B are spaces where person A and person B are located, respectively. Person A and person B, who are located far away, are people who communicate with each other via their avatars as described above.
[0040] 3, an AR display device 1A, a measurement device 21A, and an information processing device 22A are provided in space A. The measurement device 21A and the information processing device 22A are connected via wired or wireless communication. The AR display device 1A worn by person A who communicates with person B as an AR application user is also connected to the information processing device 22A via wired or wireless communication.
[0041] The measurement device 21A is a sensor device equipped with various sensors such as a color image sensor and a depth sensor, similar to the measurement device 11. The measurement device 21A measures the movement of person A and outputs measurement data such as a color image and a distance image as the measurement result to the information processing device 22A.
[0042] The information processing device 22A transmits data of the AR content to the AR display device 1A and displays an avatar of person B. Hereinafter, the avatar of person B that person A is looking at will be referred to as avatar B, and the avatar of person A that person B is looking at will be referred to as avatar A, as appropriate. In FIG. 3 , avatar B that person A is looking at is shown as an illustration of person B with a dot pattern added. Also, avatar A that person B is looking at is shown as an illustration of person A with a dot pattern added. The same applies to other figures described later.
[0043] Furthermore, the information processing device 22A performs various recognition processes based on the measurement data supplied from the measurement device 21A, and generates a spatiotemporal scene graph representing the context of space A where person A is located. The information processing device 22A compares the spatiotemporal scene graph representing the context of space A with the spatiotemporal scene graph generated by the processing of the human behavior learning phase, and predicts the next behavior of person B who is located remotely. The information processing device 22A reproduces the behavior of person B as the behavior of avatar B according to the predicted behavior of person B and the spatial layout of space A. The behavior of person B is reproduced by displaying avatar B performing the predicted behavior.
[0044] 3, an AR display device 1B, a measurement device 21B, and an information processing device 22B are provided in space B. The measurement device 21B and the information processing device 22B are connected via wired or wireless communication. The AR display device 1B worn by person B, who communicates with person A as an AR application user, is also connected to the information processing device 22B via wired or wireless communication.
[0045] The measurement device 21B is a sensor device equipped with various sensors such as a color image sensor and a depth sensor, similar to the measurement device 11. The measurement device 21B measures the movement of person B and outputs measurement data such as a color image and a distance image, which are the measurement results, to the information processing device 22B.
[0046] The information processing device 22B transmits data of the AR content to the AR display device 1B, and displays an avatar A that is an avatar of the person A.
[0047] Furthermore, the information processing device 22B performs various recognition processes based on the measurement data supplied from the measurement device 21B, and generates a spatiotemporal scene graph representing the context of space B where person B is located. The information processing device 22B compares the spatiotemporal scene graph representing the context of space B with the spatiotemporal scene graph generated by the processing in the human behavior learning phase, and predicts the next behavior of person A who is located remotely. The information processing device 22B reproduces the behavior of person A as the behavior of avatar A according to the predicted behavior of person A and the spatial layout of space B. The behavior of person A is reproduced by displaying avatar A performing the predicted behavior.
[0048] Recognition data, which is data on the respective recognition results, is transmitted and received between the information processing device 22A and the information processing device 22B. Recognition data representing the recognition results of person A's behavior, the recognition results of the relationships between objects in space A, etc. is transmitted from the information processing device 22A to the information processing device 22B. Furthermore, recognition data representing the recognition results of person B's behavior, the recognition results of the relationships between objects in space B, etc. is transmitted from the information processing device 22B to the information processing device 22A. The respective contexts of space A and space B are shared between the information processing device 22A and the information processing device 22B, and are used in each process of the human behavior reproduction phase, including generation of a spatiotemporal scene graph.
[0049] The behavior of avatar B seen by person A is controlled based on the behavior predicted by the information processing device 22A, and the behavior of avatar A seen by person B is controlled based on the behavior predicted by the information processing device 22B, thereby realizing smooth remote communication with low latency.
[0050] 3, the measurement device 11 and the measurement device 21A, and the measurement device 11 and the measurement device 21B are configured as devices in different housings, but they may be configured as devices in the same housing. In this case, measurements in the human behavior learning phase and measurements in the human behavior reproduction phase are performed in the same space.
[0051] The functions of the information processing device 12 and the information processing devices 22A and 22B may be realized by a server on the Internet. The information processing device 12 and the information processing devices 22A and 22B may be configured as devices housed in the same housing.
[0052] The functions of the measurement device 11 and the information processing device 12 may be implemented in a single device. Similarly, on the human behavior reproduction side, the functions of the measurement device 21A and the information processing device 22A, and the functions of the measurement device 21B and the information processing device 22B may each be implemented in a single device. Furthermore, the functions of the information processing device 22A may be implemented in the AR display device 1A, and the functions of the information processing device 22B may be implemented in the AR display device 1B.
[0053] In this way, it is possible to change which device implements the functions of each device as needed. Hereinafter, when there is no need to distinguish between the measurement device 21A and the measurement device 21B, they will be collectively referred to as the measurement device 21. Furthermore, when there is no need to distinguish between the information processing device 22A and the information processing device 22B, they will be collectively referred to as the information processing device 22.
[0054] The details of the processes in each phase, the human behavior learning phase and the human behavior reproduction phase, will be described below.
[0055] <<Personal Behavior Learning Phase>> FIG. 4 is a diagram showing an example of processing in the personal behavior learning phase.
[0056] The processing in the human behavior learning phase includes a recognition process (Step 1) and a spatiotemporal scene graph construction and recording process (Step 2).
[0057] <Step 1: Recognition Processing> The recognition processing is processing for recognizing the behavior of people in real space, attribute information of each person, spatial information, and relationships between objects based on measurement data supplied from the measurement device 11. The recognition processing includes space recognition processing, person attribute recognition processing, and person behavior recognition processing as processing of Steps 1-1 to 1-3. The person attribute processing may not be performed.
[0058] Step 1-1: Space Recognition Processing In the space recognition processing, space information is generated based on the measurement data supplied from the measurement device 11. For example, the following information is generated as the space information.
[0059] ・Geometric information of each object: shape, size, etc. ・Semantic information of each object: category (wall, floor, chair, etc.), parts (seat back, doorknob) ・Information indicating the relationship between objects: distance between objects, such as "object A is near object B," positional relationship between objects, such as "object A is in front of object B," etc.
[0060] For example, the technologies described in References 2 and 3 can be used for spatial recognition. Reference 2: [Narita+, IROS2019] G. Narita, T. Seno, T. Ishikawa and Y. Kaji, “PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things”, IROS, 2019. Reference 3: [Tahara+, ISMAR2020] T. Tahara, T. Seno, G. Narita and T. Ishikawa, “Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph”, ISMAR, 2020.
[0061] Step 1-2: Person Attribute Recognition Processing In the person attribute recognition processing, person attribute information is generated based on the measurement data supplied from the measurement device 11. For example, the following information is generated as person attribute information.
[0062] ・Information obtained from appearance, such as age, gender, and race ・Information that can be used to identify an individual in combination with the above information or image information
[0063] Person attributes can be recognized using a library such as OpenCV.
[0064] Step 1-3: Human behavior recognition processing In the human behavior recognition processing, relationship information indicating object-person relationships and relationship information indicating person-person relationships are generated based on the measurement data supplied from the measurement device 11. For example, the following information is generated as the relationship information:
[0065] Object-person interaction: An action in which a person acts on an object, such as person A sitting on a sofa ("Person A Vs to Object B"). Person-person interaction: An action in which a person acts on another person, such as person A talking to person B ("Person A Vs to Person B").
[0066] For example, the technology described in Reference 4 can be used for human behavior recognition. Reference 4: [Gao+, BMVC2018] C. Gao, Y. Zou and J.-B. Huang, “iCAN: Instance-Centric Attention Network for Human-Object Interaction”, BMVC, 2018.
[0067] The information processing device 12 is provided with recognizers that perform, for example, spatial recognition processing, human attribute recognition processing, and human behavior recognition processing. Each recognizer is configured, for example, by an inference model that receives measurement data supplied from the measurement device 11 as input and outputs a recognition result. An inference model generated in advance by machine learning is prepared in the information processing device 12. Each recognizer calculates the recognition result as well as the likelihood of the recognition result.
[0068] The spatial information generated by the spatial recognition process, the person attribute information generated by the person attribute recognition process, and the relationship information generated by the person behavior recognition process are represented as graphs, which are data having a graph structure.
[0069] FIG. 5 is a diagram showing an example of a graph showing spatial information.
[0070] For example, suppose it is recognized that a sofa, a chair, and a table are located around a person to be measured, and that they are positioned in a predetermined relationship. In this case, the graph showing the spatial information will be data consisting of three nodes (sofa#1, chair#2, table#3) representing these objects, as shown in Figure 5.
[0071] In the example of Figure 5, the sofa node and the chair node are connected by edge E1 labeled "in front of" and edge E2 labeled "on left of." Edge E1, which is represented as an arrow pointing from the sofa node to the chair node, indicates that the chair is located in front of the sofa. Edge E2, which is represented as an arrow pointing from the chair node to the sofa node, indicates that the sofa is located on the left side of the chair in front of it.
[0072] The sofa node and the table node, and the chair node and the table node are also connected by edges E3 and E4, respectively, to which labels indicating their positional relationship are set.
[0073] FIG. 6 is a diagram showing an example of a graph showing person attribute information.
[0074] When the age of the person being measured is recognized as 60 years old, the graph showing the person attribute information is composed of a node representing the person being measured (Person#1) and a node representing the age of 60 (Age:60), as shown in A of Fig. 6. The node representing the person being measured and the node representing the age of 60 are connected by an edge E11 labeled "has".
[0075] Furthermore, if the height of the person being measured is recognized as 1.8 m, the graph showing the person attribute information will be composed of a node representing the person being measured (Person#1) and a node representing that the height is 1.8 m (Height: 1.8 m), as shown in B of Fig. 6. The node representing the person being measured and the node representing that the height is 1.8 m are connected by an edge E12 labeled "has".
[0076] FIG. 7 is a diagram illustrating an example of a graph showing relationship information.
[0077] As a relationship between an object and a person, it is assumed that the person being measured is using a table and sitting in a chair. In this case, the graph showing the relationship information is composed of a node representing the person being measured (Person#1), a node representing the table (table#1), and a node representing the chair (chair#2), as shown in Figure 7.
[0078] The node representing the person being measured and the node representing the table are connected by an edge E21 labeled "use." The node representing the person being measured and the node representing the chair are connected by an edge E22 labeled "sitting on."
[0079] The results of the recognition process in Step 1 are thus obtained as data having a graph structure in which objects, people, and person attributes are represented as nodes, and the relationships between them are represented as edges.
[0080] <Step 2: Spatiotemporal scene graph construction and recording process> The spatiotemporal scene graph construction and recording process is a process of constructing and recording a spatial scene graph that expresses the spatial meaning at each time and a spatiotemporal scene graph that includes change information between spatial scene graphs, based on the information obtained by the recognition process.
[0081] The spatiotemporal scene graph construction and recording process will be described with reference to the flowchart of FIG.
[0082] Step 2-1: Comparison of human behavior recognition results The human behavior recognition result obtained from each recognizer at time t, which is the current time, is compared with the human behavior recognition result obtained similarly at time t-1, which is the previous measurement time, to determine whether or not there has been a change in the human behavior. If it is determined that there has been no change in the human behavior, the comparison of the human behavior recognition results is repeated based on information newly acquired by the recognition process.
[0083] On the other hand, if it is determined that there has been a change in human behavior, the process proceeds to a later stage to construct a spatiotemporal scene graph.
[0084] Step 2-2: Construction of spatial scene graph at time t-1 / time t When a change in person behavior is detected, a spatial scene graph is constructed for each of time t-1, when the most recent change in person behavior was detected, and time t, which is the current time. The spatial scene graph is constructed by integrating the graph of spatial information, the graph of person attribute information, and the graph of object-person relationship information for each time into a single scene graph for each time.
[0085] FIG. 9 is a diagram showing an example of a graph representing the recognition result.
[0086] As shown on the left side of Fig. 9, it is assumed that at time t-1, person B, who was sitting on the sofa, talks to person A, and that at time t after a predetermined time has elapsed, person A sits on the sofa where person B is sitting. In this case, the graphs of space information, person attribute information, and object-person relationship information at time t-1 and time t are as shown on the right side of Fig. 9.
[0087] For example, the spatial information graph at time t-1 is composed of a floor node and a sofa node as objects. The sofa node and the floor node are connected by an edge labeled "on", which indicates that the sofa is on the floor. The spatial information graph at time t is also composed of a floor node and a sofa node as objects.
[0088] The graph of person attribute information at time t-1 is composed of a graph in which person A's node and a node representing that person's age is 20 (Age: 20) are connected by an edge labeled "has", and a graph in which person B's node and a node representing that person's age is 19 (Age: 19) are connected by an edge labeled "has". The graph of person attribute information at time t is also composed of a graph with the same structure.
[0089] The graph of object-person relationship information at time t-1 includes a graph in which the node for person A is connected to the node for the floor by an edge labeled "stand", and the node for person A is connected to the node for person B by an edge labeled "Talk to" that points from the node for person B to the node for person A. The edge labeled "Talk to" that points from the node for person B to the node for person A indicates that person B has spoken to person A.
[0090] When there are multiple people in the space to be measured, the information on the relationship between objects and people includes not only information indicating the relationship between objects and people, but also information indicating the relationship between people. When appropriate, information indicating the relationship between objects is also included in the information on the relationship between objects and people.
[0091] The relationship information graph at time t-1 also includes a graph in which the node for Person B and the node for the sofa are connected by an edge labeled "Sit to." The edge labeled "Sit to" from the node for Person B to the node for the sofa indicates that Person B is sitting on the sofa.
[0092] The graph of object-person relationship information at time t includes a graph in which the node for person A and the node for the sofa are connected by an edge labeled "Sit to." The edge labeled "Sit to" from the node for person A to the node for the sofa indicates that person A is sitting on the sofa.
[0093] 9 shows an example in which a change occurs in the graph of object-person relationship information as person A sits on the sofa, and it is determined that there has been a change in the person's behavior. In this way, when a change is detected between the graph representing the recognition result at time t-1 and the graph representing the recognition result at time t, the graphs are integrated and a spatial scene graph is generated.
[0094] FIG. 10 is a diagram illustrating an example of a spatial scene graph.
[0095] 10 is a spatial scene graph that integrates the respective graphs of spatial information, person attribute information, and relationship information at time t-1, while the graph in the lower part is a spatial scene graph that integrates the respective graphs of spatial information, person attribute information, and relationship information at time t.
[0096] Step 2-3: Evaluating the positional relationship between a person and an object The positional relationship between a person and an object from time t-1 to time t is evaluated. This evaluation uses a measure that can be calculated instantly from the position of each person and each object. For example, if a change in the positional relationship of "Close to" is recognized, it is evaluated that the distance between the person and the object has shortened from time t-1 to time t. The change in the positional relationship is represented by the edge connecting the person node and the object node.
[0097] A similar measure is also used in the human behavior reproduction phase. In order to enable behavior prediction to be performed in a short time in the human behavior reproduction phase, it is desirable that the evaluation of the positional relationship here be performed using a measure that can be calculated in a short time.
[0098] Step 2-4: Evaluation of time-series state likelihood How the likelihood of the recognition result of human behavior has changed from time t-1 to time t is evaluated using time-series data. In human behavior recognition, when a behavior with a significantly high probability (likelihood) is performed among the discretized behaviors, the behavior with the high probability is output as the recognition result of the human behavior. For example, when a behavior with a likelihood equal to or greater than a threshold is recognized, the behavior is output as the recognition result of the human behavior.
[0099] When a person changes their behavior, the time it takes for a behavior with a significantly higher likelihood to be recognized from the behavior before the change can cause the above-mentioned communication discrepancy. By recording the temporal change in likelihood of the recognition result of each behavior obtained in the human behavior recognition process, it becomes possible to easily predict the target person's next behavior when a similar change in likelihood is observed.
[0100] FIG. 11 is a diagram illustrating an example of time-series state likelihood.
[0101] The left side of Fig. 11 shows the states of person A and person B at time t. The state shown on the left side of Fig. 11 is a state in which person B, sitting on a sofa, suggests to person A, who is standing near a table, that they sit on the sofa and watch TV together. In person A's time-series state likelihood table, "1", "0", and "0" are recorded as the likelihoods of "stand", "sit", and "talk", respectively. "stand", which has a likelihood higher than the threshold, is output as the action recognition result for person A.
[0102] The center of Fig. 11 shows the states of person A and person B at time t+1. In the state shown in the center of Fig. 11, person A starts moving from a standing state and approaches the sofa. In the time-series state likelihood table for person A, the likelihoods of "stand," "sit," and "talk" are recorded as "0.8," "0.1," and "0.1," respectively. As person A approaches the positions of person B and the sofa, recognition is performed such that the likelihoods of "sit" and "talk" increase and the likelihood of "stand" decreases.
[0103] The right side of Fig. 11 shows the states of person A and person B at time t + Δt. The state shown on the right side of Fig. 11 is the state immediately after person A sits down on the sofa. In this case, "0", "1", and "0" are recorded as the likelihoods of "stand", "sit", and "talk", respectively, in the time-series state likelihood table for person A. "sit", which has a likelihood higher than the threshold, is output as the behavior recognition result for person A.
[0104] In this way, the time-series state likelihood table records the likelihood of the behavior recognition result at each time. The information recorded in the time-series state likelihood table is, for example, a time series of likelihoods until the next behavior with a likelihood higher than the threshold is recognized.
[0105] Step 2-5: Edge connection between spatial scene graphs at time t-1 / time t By connecting the spatial scene graphs constructed in Step 2-2 with edges indicating the change in positional relationship evaluated in Step 2-3 and edges indicating the time-series state likelihood evaluated in Step 2-4, the spatial scene graphs at each time are integrated into a single spatiotemporal scene graph.
[0106] FIG. 12 is a diagram illustrating an example of a spatiotemporal scene graph.
[0107] The spatiotemporal scene graph shown in Fig. 12 is a scene graph constructed by connecting the spatial scene graph at time t-1 and the spatial scene graph at time t described with reference to Fig. 10 with edges indicating changes in positional relationship and edges indicating time-series state likelihood. An edge indicating a change in positional relationship is called a positional relationship edge, and an edge indicating time-series state likelihood is called a time-series state likelihood edge.
[0108] The node of person A in the spatial scene graph at time t-1 and the node of person A in the spatial scene graph at time t are connected by a time-series state likelihood edge E31 that indicates the time-series state likelihood of person A from time t-1 to time t. As shown in the speech bubble, the labels set on the time-series state likelihood edge E31 indicate the time series of the likelihoods of "stand," "sit," and "talk," respectively.
[0109] Furthermore, the node of person A in the spatial scene graph at time t-1 and the node of the sofa in the spatial scene graph at time t are connected by a positional relationship edge E32, which indicates that the change in the positional relationship between person A and the sofa from time t-1 to time t is "Close to." The label set on the positional relationship edge E32 indicates that person A has approached the sofa.
[0110] The node of person A in the spatial scene graph at time t-1 and the node of person B in the spatial scene graph at time t are connected by a positional relationship edge E33, which indicates that the change in the positional relationship between person A and person B from time t-1 to time t is "Close to." The label set on the positional relationship edge E33 indicates that person A has moved closer to person B.
[0111] Similarly, the node of person B in the spatial scene graph at time t-1 and the node of person B in the spatial scene graph at time t are connected by a time-series state likelihood edge E34 indicating the time-series state likelihood of person B from time t-1 to time t.
[0112] In this way, the spatiotemporal scene graph is constructed by connecting nodes of the same person in the spatial scene graphs at different times with time-series state likelihood edges, which indicate the time-series state likelihood. Also, the spatiotemporal scene graph is constructed by connecting nodes of each person and object, and nodes of each person and other people, with positional relationship edges, which indicate the change in positional relationship.
[0113] The time-series state likelihood edge is an edge that indicates the time series of the likelihood of the action recognition result, and corresponds to an edge that indicates a temporal change in the state of the recognition result of a person's action. Furthermore, the positional relationship edge is an edge that indicates a change in the positional relationship between objects, and corresponds to an edge that indicates a temporal change in the placement state of the object in space. The spatial scene graph is a scene graph that is constructed by connecting nodes that constitute the spatial scene graph at different times using two types of change edges that indicate a temporal change in the state.
[0114] In the example of Figure 12, one spatiotemporal scene graph is generated by integrating spatial scene graphs for two time periods, time t-1 and time t, but one spatiotemporal scene graph may also be generated by integrating spatial scene graphs for three or more time periods.
[0115] A spatiotemporal scene graph generated in advance by integrating multiple spatial scene graphs in this manner is recorded in the time-series human behavior DB and is used to predict the next human behavior in the human behavior reproduction phase. In the present disclosure, the multiple scene graphs recorded in advance in the time-series human behavior DB may be referred to as a second scene graph.
[0116] As described above, in the spatiotemporal scene graph construction and recording process, the spatiotemporal scene graph is constructed and recorded when a semantic change in the context of the real space occurs. The timing of a semantic change in the context of the real space also includes the timing when the positional relationships of objects, including people and peripheral objects, change. The necessary data is recorded in the time-series human behavior DB regardless of the measurement time length or time resolution, making it possible to efficiently generate the time-series human behavior DB.
[0117] Furthermore, the information recorded in the time-series human behavior DB has a graph structure. Because data can be accessed by navigating the nodes of interest and the edges connected to those nodes, access frequency can be reduced compared to DBs that record information in table format, making it possible to improve processing efficiency.
[0118] <<Human Behavior Reproduction Phase>> FIG. 13 is a diagram showing an example of processing in the human behavior reproduction phase.
[0119] The processing in the human behavior reproduction phase is a processing for sequentially predicting the next behavior of a remote person by referring to the spatiotemporal scene graph recorded in the time-series human behavior DB by the processing in the human behavior learning phase, and displaying the avatar of the remote person based on the predicted behavior. The information processing device 22A performs a processing for displaying the avatar of person B, and the information processing device 22B performs a processing for displaying the avatar of person A.
[0120] The human behavior reproduction phase includes a recognition process (Step 11), a human behavior candidate prediction process (Step 12), a predicted behavior space adaptation process (Step 13), and an avatar behavior presentation process (Step 14).
[0121] <Step 11: Recognition Processing> The recognition processing in the human behavior reproduction phase is basically the same as the recognition processing in the human behavior learning phase (Step 1 in FIG. 4). Duplicate explanations will be omitted as appropriate. The recognition processing in the human behavior reproduction phase is performed with a space where a person communicating with a remote person is present as the measurement target.
[0122] Step 11-1: Spatial Recognition Processing In the spatial recognition processing, spatial information is generated based on the measurement data supplied from the measurement device 21. As the spatial information, geometric information of each object, semantic information of each object, and information indicating the relationship between the objects are generated.
[0123] Step 11-2: Person attribute recognition processing In the person attribute recognition processing, person attribute information of the person being measured is generated based on the measurement data supplied from the measurement device 21. As the person attribute information, information such as age, sex, and race obtained from the person's appearance, and information that allows personal identification, is generated.
[0124] Step 11-3: Human Behavior Recognition Processing In the human behavior recognition processing, relationship information between an object and a person and between people is generated based on the measurement data supplied from the measurement device 21.
[0125] The spatial information generated by the spatial recognition process, the person attribute information generated by the person attribute recognition process, and the relationship information generated by the person behavior recognition process are acquired as data having a graph structure.
[0126] <Step 12: Person behavior candidate prediction processing> The person behavior candidate prediction processing is a processing for searching for a similar spatiotemporal scene graph, which is a spatiotemporal scene graph similar to the spatiotemporal scene graph at the current time, from among the spatiotemporal scene graphs recorded in the time-series person behavior DB, and acquiring candidates for the next behavior of a remote person. In the present disclosure, the similar spatiotemporal scene graph may be simply referred to as a similar scene graph.
[0127] The human behavior candidate prediction process will be described with reference to the flowchart of FIG.
[0128] Step 12-1: Construction of spatial scene graphs at time t-1 / time t As in Step 2-2 of FIG. 8, spatial scene graphs at time t-1 and time t are constructed.
[0129] Step 12-2: Evaluation of the positional relationship between the person and the object As in Step 2-3 of FIG. 8, the positional relationship between the person and the object from time t-1 to time t is evaluated.
[0130] Step 12-3: Evaluation of time-series state likelihood As in Step 2-4 in FIG. 8, how the likelihood of recognizing human behavior has changed from time t-1 to time t is evaluated using time-series data.
[0131] Step 12-4: Edge connection between spatial scene graphs at time t-1 / time t As in Step 2-5 of Figure 8, the spatial scene graphs constructed in Step 12-1 are connected with edges indicating the change in positional relationship evaluated in Step 12-2 and edges indicating the time-series state likelihood evaluated in Step 12-3, thereby integrating the spatial scene graphs at each time into a single spatiotemporal scene graph.
[0132] 14 is the same as the processing of Steps 2-2 to 2-5 in Fig. 8 in the human behavior learning phase. A spatiotemporal scene graph is generated that represents the spatial context of a person who is communicating with a remote person via an avatar. In the present disclosure, a spatiotemporal scene graph in which nodes of the spatial scene graphs at different times, time t-1 / time t, are connected by edges indicating changes in positional relationships and edges indicating time-series state likelihoods may be referred to as a first scene graph.
[0133] Step 12-5: Search for similar spatiotemporal scene graphs The current spatiotemporal scene graph from time t-1 to time t constructed by the processing up to Step 12-4 is compared with the spatiotemporal scene graphs recorded in the time-series human behavior DB to search for similar spatiotemporal scene graphs. The search for similar spatiotemporal scene graphs is performed, for example, by comparing the inter-graph distances of the respective spatiotemporal scene graphs and listing spatiotemporal scene graphs whose inter-graph distance is equal to or less than a threshold.
[0134] If the person targeted for measurement when constructing the spatiotemporal scene graph in the time-series human behavior DB is different from the person targeted by the current spatiotemporal scene graph, each person targeted by the current spatiotemporal scene graph is assigned to the person targeted for measurement when constructing the spatiotemporal scene graph in the time-series human behavior DB, and then the distances between the graphs are compared.
[0135] Furthermore, when evaluating the identity of time-series state likelihood edges in a search for similar spatiotemporal scene graphs, the evaluation is performed, for example, by determining that the two edges are the same edge when the average cross-correlation of the time-series state likelihoods indicated by the two edges to be evaluated is equal to or greater than a threshold value.
[0136] Various measures such as edit distance can be used as the distance between graphs. The edit distance between spatiotemporal scene graphs is an index that evaluates the distance between graphs by the number of edits to nodes and edges required for two graphs to be compared to become the same graph. Edits to nodes and edges include adding, deleting, and replacing nodes and edges.
[0137] FIG. 15 is a diagram illustrating an example of the graph distance.
[0138] Graph A shown in Figure 15A is composed of nodes for Person A, Person B, the floor, and the sofa. Person A's node and Person B's node are connected by an edge labeled "talk", and Person A's node and the floor node are connected by an edge labeled "stand". Person B's node and the sofa node are connected by an edge labeled "sit".
[0139] Graph B, which is compared with Graph A, consists of nodes for Person A, Person B, the floor, and the chair. Person A's node and Person B's node are connected by an edge labeled "talk", Person A's node and the floor node are connected by an edge labeled "stand", and Person B's node and the chair node are connected by an edge labeled "sit".
[0140] For example, if graph A is used as a reference, replacing the chair node in graph B with a sofa node will make graph B the same graph as graph A. The graph distance between graph A and graph B shown in FIG. 15A is 1.
[0141] 15B, the distance between graphs C and D is similarly calculated as 5. Graphs A and B, which have a closer distance between them, are more similar than graphs C and D.
[0142] Based on the inter-graph distances thus obtained, multiple similar spatiotemporal scene graphs are acquired as spatiotemporal scene graphs containing candidates for predicted behavior.From the spatiotemporal scene graphs recorded in the time-series human behavior DB, spatiotemporal scene graphs whose graph structures are at least partially similar to the current spatiotemporal scene graph are acquired as similar spatiotemporal scene graphs.
[0143] Step 12-6: Evaluating the Likelihood of Predicted Actions Based on the multiple similar spatiotemporal scene graphs retrieved in Step 12-5, a predicted action to be reproduced by the avatar is selected so as to select the action that the remote person will most likely perform next.
[0144] FIG. 16 is a diagram illustrating an example of a spatiotemporal scene graph including candidates for predicted behavior.
[0145] 16, three spatiotemporal scene graphs, behavior candidate spatiotemporal scene graphs 1 to 3, are acquired from the time-series human behavior DB as similar spatiotemporal scene graphs including candidates for predicted behavior. The behavior candidate spatiotemporal scene graphs 1 to 3 are spatiotemporal scene graphs configured by the spatial scene graph at time t-1 and the spatial scene graph at time t for person X and person Y.
[0146] When person B is assigned to person X and person A is assigned to person Y, and attention is paid to the behavior candidate spatiotemporal scene graphs 1 to 3, the candidates for the predicted behavior that person Y will perform next (at time t) are "sit on the sofa" and "stand on the floor."
[0147] The behavior candidate spatiotemporal scene graph 1 and the behavior candidate spatiotemporal scene graph 3 in FIG. 16 show that person Y, who is standing on the floor at time t-1, approaches the sofa and sits on the sofa at time t. The behavior candidate spatiotemporal scene graph 2 shows that person Y, who is standing on the floor at time t-1, continues to stand on the floor at time t. The probability that person Y will sit on the sofa at time t (likelihood: 2 / 3) is higher than the probability that person Y will continue to stand on the floor at time t (likelihood: 1 / 3). The behavior that person Y, who is standing on the floor at time t-1, is most likely to perform next is identified as "sitting on the sofa."
[0148] In this way, from among the predicted behavior candidates, the candidate with the highest probability, "sit on the sofa," is selected as the final predicted behavior to be reproduced by the avatar. The behavior of "sit on the sofa" is a behavior that realizes the state change indicated by the time-series state likelihood edge connected to the node at time t-1 of person Y corresponding to person A in the behavior candidate space-time scene graphs 1 and 2, which are similar space-time scene graphs.
[0149] Furthermore, "sofa" is selected as the next destination. The destination "sofa" is indicated by the "Close to" positional relationship edge connected to the node of person Y corresponding to person A in the behavior candidate space-time scene graphs 1 and 2, which are similar space-time scene graphs.
[0150] Furthermore, the time Tgoal until the completion of the behavior of “sit on the sofa” is acquired. The time Tgoal is calculated, for example, as the difference between time t−1 and time t in the behavior candidate spatiotemporal scene graph 1 and the behavior candidate spatiotemporal scene graph 3 used to select the behavior of “sit on the sofa.”
[0151] <Step 13: Predicted Action Space Adaptation Processing> The predicted action space adaptation processing adjusts the action speed of the avatar of a remote person and corrects the destination so that the predicted action selected in the human action candidate prediction processing in Step 12 is completed by the time Tgoal.
[0152] The adjustment of the action speed includes adjustment of the movement speed and adjustment of the motion (action) speed. The adjustment of the avatar's action speed and correction of the destination are performed based on the destination of the predicted action and the time Tgoal until the predicted action is completed. If the predicted action involves movement, the destination is corrected appropriately.
[0153] The layout of the space targeted in the human behavior learning phase and the reproduction space targeted in the human behavior reproduction phase may differ. For example, even when an avatar is asked to reproduce the same action of "sitting on a sofa," the time it takes to sit on the sofa will differ depending on the layout of the space. Depending on the difference in layout between the reproduction space and the space measured when constructing the spatiotemporal scene graph (spatiotemporal scene graph in the time-series human behavior DB) used to select predicted actions, the avatar's behavior speed will be adjusted.
[0154] The predicted action space adaptation process will be described with reference to the flowchart of FIG.
[0155] Step 13-1: Calculation of required time Tstd When a predicted action is reproduced in the reproduction space at a standard speed, the required time Tstd for the predicted action to be completed is calculated. The standard speed at which each action is performed by an avatar is determined in advance. The required time Tstd varies depending on the type of predicted action, the layout of the space, the position of the avatar at the start of the predicted action, etc.
[0156] Step 13-2: Calculation of action speed multiplier Tstd / Tgoal Based on the time Tgoal until the predicted action is completed, the action speed multiplier Tstd / Tgoal of the avatar is calculated.
[0157] FIG. 18 is a diagram showing an example of calculation of the magnification of the action speed of an avatar.
[0158] The space shown on the left side of Fig. 18 is the space that was the measurement target when constructing the spatiotemporal scene graph used to select the predicted behavior (during the human behavior learning phase), and the space shown on the right side is the reproduction space. In the human behavior learning phase, the time required for person A to complete the behavior of "sit on the sofa" is recorded as 3 seconds. The time Tgoal until the predicted behavior is completed is 3 seconds.
[0159] On the other hand, the sofa in the reproduction space is located away from the position where avatar A, who is reproducing the action of "sitting on the sofa," is standing. As shown by arrow A1, if avatar A reproduces the action of "sitting on the sofa" at a standard speed, the time required to complete the predicted action, Tstd, is 6 seconds.
[0160] In this case, the speed multiplier Tstd / Tgoal for the action "sit on the sofa" is calculated as 2, and the action speed of avatar A is adjusted to be twice the standard speed, as shown by arrow A2. By executing the action "sit on the sofa", which is a predicted action, at twice the standard speed, it becomes possible to have avatar A sit on the sofa within three seconds.
[0161] In this way, by adjusting the action speed of the avatar, it is possible to complete the next action within the predicted time.
[0162] On the other hand, significantly increasing the speed of an avatar's movement or motion can make the avatar appear unnatural. In the predicted action space adaptation process, if the avatar appears unnatural and there is a synonymous destination, the predicted action is corrected to change the destination. For example, a speed threshold is set for each action reproduced by the avatar so that the avatar does not appear unnatural.
[0163] Step 13-3: Correction of predicted behavior If the behavior selected as the predicted behavior to be reproduced by the avatar involves movement, a destination is also selected, such as a sofa, a chair, etc. If there is a destination synonymous with the selected destination in the reproduction space and the avatar would look more natural if it were moved toward that destination, the destination is changed.
[0164] FIG. 19 illustrates an example of changing the destination.
[0165] The space shown on the left side of Figure 19 is the space that was the target of the human behavior learning phase, and the space shown on the right side is the reproduction space. In both spaces, chairs #1 to #3 are prepared in addition to the chair in which person B is sitting. Chair #1 is located next to the chair in which person B is sitting, and chairs #2 and #3 are located across the table from the chair in which person B is sitting.
[0166] In the human behavior learning phase targeting the space shown on the left side of Figure 19, a spatiotemporal scene graph is constructed that includes a graph structure showing the behavior of person A sitting in chair #2. Chair #2 is the chair opposite the chair in which person A is sitting. The time required for the behavior of sitting in chair #2 is 3 seconds. Based on the spatiotemporal scene graph representing this context, we will explain how to change the destination when the destination is chair #2 and the behavior of "sit in a chair" is selected as the predicted behavior.
[0167] When avatar A is made to reproduce the action of "sitting on a chair," chairs #1, #2, and #3 are synonymous destinations (nodes) in the spatiotemporal scene graph constructed during the human action reproduction phase for the reproduction space. There are multiple synonymous destinations in the spatiotemporal scene graph representing the context of the playback space. The times required to perform the actions of sitting on chairs #1, #2, and #3 in the reproduction space are 6 seconds, 5 seconds, and 3 seconds, respectively, as indicated by arrows A11 to A13.
[0168] In this way, when there are multiple synonymous destinations in the reproduction space, the difference (||Tgoal - Tstd||) between the time required to complete the next action, Tgoal, and the time required if the predicted action is executed at standard speed, Tstd, is calculated, and the destination is corrected so that the destination with the smallest difference between the required time, Tgoal, and the time, Tstd, is selected. In the example of Figure 19, as shown by arrow A13, the destination is corrected so that chair #3, with the smallest difference between the required time, Tgoal (3 seconds), and the time, Tstd (3 seconds), is selected as the destination.
[0169] In this way, by selecting the optimal destination based on the change in the appearance of the avatar due to movement, it is possible to prevent the avatar from looking unnatural. Such destination correction is performed only within the range where the shape of the spatiotemporal scene graph does not change (the range where the meaning expressed by the space does not change). For example, the range where the graph containing the destination node before correction and the graph containing the destination node after correction are isomorphic and the distance between the graphs is 0 is the range where the meaning expressed by the space does not change.
[0170] <Step 14: Presenting Avatar Behavior> The process of presenting avatar behavior is a process of controlling the display of the avatar of a remote person so that the predicted behavior can be completed within a predetermined time, and reproducing the avatar performing the predicted behavior. The behavior is presented by displaying an image of the avatar performing the predicted behavior on the AR display device 1.
[0171] The generation of motions according to specific spaces is described in, for example, Reference 4. Reference 4: [Starke+, SIGGRAPHasia2019] S. Starke, H. Zhang, T. Komura and J. Saito, “Neural State Machine for Character-Scene Interactions”, SIGGRAPH Asia, 2019.
[0172] After the avatar behavior presentation process is performed, the processes from Step 11 onward are repeatedly executed.
[0173] <<Summary>> As described above, in an information processing system to which the present technology is applied, communication between people in remote locations is carried out via each other's avatars.
[0174] In the human behavior learning phase, a spatial scene graph is generated in which nodes represent people, surrounding objects, and attribute information in the real world, and nodes are connected by edges that indicate their relationships. A spatiotemporal scene graph is also constructed and recorded by connecting nodes that make up the spatial scene graph at different times with edges that indicate temporal changes in state. Edges that indicate temporal changes in state include time-series state likelihood edges that indicate how a person's state has changed over time, and positional relationship edges that indicate how the positional relationship between a person or object at a certain time and another person or object at a different time has changed.
[0175] In the human behavior reproduction phase, the next action of a remote person is predicted by comparing the spatiotemporal scene graph, which is composed of spatial scene graphs from the past to the present, with the spatiotemporal scene graph recorded in the time-series human behavior DB. Avatars are presented to reproduce the predicted action, compensating for time delays and enabling smooth communication.
[0176] In the human behavior reproduction phase, the movement speed and motion speed of the avatar are adjusted based on the time required to complete the next action in each of the target spaces when constructing the spatiotemporal scene graph and the reproduction space. This allows the avatar to complete the reproduction of the action by the predicted completion time of the next action.
[0177] Furthermore, in the human behavior reproduction phase, the destination and target of the avatar's action are corrected within the range where the graphs are isomorphic (the range where the distance between graphs is 0) so that the avatar can complete the reproduction of the action by the time the predicted action is completed. This makes it possible to prevent the avatar from looking unnatural.
[0178] By expressing the context of the real world using a spatiotemporal scene graph, which connects nodes and edges spatiotemporally, it becomes possible to obtain changes in the state of a specific person or object simply by focusing on and accessing the edges connected to the node of that person or object. This makes it possible to access data more efficiently than when using a table-type DB.
[0179] The information in the spatiotemporal scene graph is a semantic abstraction of the real-world context. It allows the real-world context to be described with less information. Furthermore, it only needs to be updated when semantic changes occur, which helps prevent data from becoming bloated.
[0180] <<Configuration of Each Device>> <Configuration of Information Processing Device 12> Fig. 20 is a block diagram showing an example of the functional configuration of the information processing device 12. At least some of the functional units shown in Fig. 20 are realized by the CPU of a computer constituting the information processing device 12 executing a predetermined program.
[0181] 20, a human behavior learning processing unit 101 is realized in the information processing device 12. The human behavior learning processing unit 101 is made up of a recognition unit 111, a time-space scene graph generation unit 112, and a recording control unit 113.
[0182] The recognition unit 111 is composed of a space recognition unit 121 , a person attribute recognition unit 122 , and a person behavior recognition unit 123 .
[0183] The space recognition unit 121 performs space recognition processing (Step 1-1 in FIG. 4) based on the measurement data supplied from the measurement device 11, and generates space information.
[0184] The person attribute recognition unit 122 performs person attribute recognition processing (Step 1-2 in FIG. 4) based on the measurement data supplied from the measurement device 11, and generates person attribute information.
[0185] The human behavior recognition unit 123 performs human behavior recognition processing (Steps 1-3 in FIG. 4) based on the measurement data supplied from the measurement device 11, and generates related information. The human behavior recognition unit 123 functions as a recognition unit that recognizes the state of the person being measured.
[0186] The spatial information generated by the spatial recognition unit 121 , the person attribute information generated by the person attribute recognition unit 122 , and the relationship information generated by the person behavior recognition unit 123 are supplied to the spatiotemporal scene graph generation unit 112 .
[0187] The spatiotemporal scene graph generation unit 112 performs spatiotemporal scene graph construction and recording processing (Step 2 in FIG. 4, FIG. 8) based on information supplied from each unit of the recognition unit 111. Information on the spatiotemporal scene graph generated by the spatiotemporal scene graph generation unit 112 is supplied to the recording control unit 113. The spatiotemporal scene graph generation unit 112 functions as a generation unit that generates multiple spatiotemporal scene graphs by performing spatiotemporal scene graph construction and recording processing.
[0188] The recording control unit 113 records the spatiotemporal scene graph supplied from the spatiotemporal scene graph generation unit 112 in the time-series human behavior DB.
[0189] <Configuration of information processing device 22> Fig. 21 is a block diagram showing an example of the functional configuration of the information processing device 22. At least some of the functional units shown in Fig. 21 are realized by the CPU of a computer constituting the information processing device 22 executing a predetermined program.
[0190] 21, a human behavior reproduction processing unit 151 is realized in the information processing device 22. The human behavior reproduction processing unit 151 is made up of a recognition unit 161, a human behavior prediction unit 162, and a presentation unit 163.
[0191] The recognition unit 161 is composed of a space recognition unit 171 , a person attribute recognition unit 172 , and a person behavior recognition unit 173 .
[0192] The space recognition unit 171 performs space recognition processing (Step 11-1 in FIG. 13) based on the measurement data supplied from the measurement device 21, and generates space information.
[0193] The person attribute recognition unit 172 performs person attribute recognition processing (Step 11-2 in FIG. 13) based on the measurement data supplied from the measurement device 21, and generates person attribute information.
[0194] The human behavior recognition unit 173 performs human behavior recognition processing (Step 11-3 in FIG. 13 ) based on the measurement data supplied from the measurement device 21, and generates related information. The human behavior recognition unit 173 functions as a recognition unit that recognizes the behavior of person A (first person) who is looking at avatar B (second avatar) that reproduces behavior corresponding to the behavior of person B (second person), and the behavior of person B who is looking at avatar A (first avatar) that reproduces behavior corresponding to the behavior of person A.
[0195] The space information generated by the space recognition unit 171 , the person attribute information generated by the person attribute recognition unit 172 , and the relationship information generated by the person behavior recognition unit 173 are supplied to the person behavior prediction unit 162 .
[0196] The human behavior prediction unit 162 performs human behavior candidate prediction processing (Step 12 in FIG. 13, FIG. 14) based on the information supplied from each unit of the recognition unit 161.
[0197] The human behavior prediction unit 162 includes a spatiotemporal scene graph generation unit 162 A. Based on the information supplied from the recognition unit 161, the spatiotemporal scene graph generation unit 162 A generates a spatiotemporal scene graph that represents the context of the reproduction space.
[0198] The human behavior prediction unit 162 compares the spatiotemporal scene graph generated by the spatiotemporal scene graph generation unit 162A with the spatiotemporal scene graph recorded in the time-series human behavior DB to obtain a similar spatiotemporal scene graph. The human behavior prediction unit 162 selects candidates for predicted behavior based on the similar spatiotemporal scene graph, and selects a predetermined predicted behavior from the candidates for predicted behavior as a final predicted behavior. Information on the predicted behavior selected by the human behavior prediction unit 162 is supplied to the presentation unit 163. In this way, the human behavior prediction unit 162 functions as an information processing unit that selects a predicted behavior based on a similar spatiotemporal scene graph (similar scene graph) included in multiple scene graphs (second scene graphs) pre-recorded in the time-series human behavior DB.
[0199] The presentation unit 163 generates display data for an avatar of a remote person by playing back the AR content and performing rendering, etc. The presentation unit 163 transmits the display data to the AR display device 1 and displays the avatar on the AR display device 1. The presentation unit 163 performs an avatar behavior presentation process (Step 14 in FIG. 13 ). The presentation unit 163 also performs a predicted behavior space adaptation process (Step 13 in FIG. 13 , FIG. 17 ) as appropriate.
[0200] <<Modifications>> <Configuration Example of Information Processing System> FIG. 22 is a block diagram showing a configuration example of an information processing system.
[0201] In the above, the human behavior learning processing unit 101 and the human behavior reproduction processing unit 151 are realized in different devices, but they may also be realized in a single device, an information processing device 201, as shown in A of Figure 22.
[0202] The information processing device 201 performs processing of the human behavior learning phase in the human behavior learning processing unit 101, and performs processing of the human behavior reproduction phase in the human behavior reproduction processing unit 151. The human behavior reproduction processing unit 151 of the information processing device 201 performs processing of the human behavior reproduction phase and displays an avatar that reproduces the predicted behavior on the AR display device 1.
[0203] As shown in FIG. 22B, the human behavior learning processing unit 101 and the human behavior reproduction processing unit 151 may be realized in the AR display device 1.
[0204] The human behavior reproduction processing unit 151 of the AR display device 1 performs processing in the human behavior reproduction phase and displays an avatar that reproduces the predicted behavior on the display unit 211. The display unit 211 is configured with a display that displays the avatar, etc.
[0205] <Other> Although the above explanation has been given mainly on cases where an avatar is made to reproduce human actions such as talking and sitting on a sofa, it is also possible to make an avatar reproduce various actions other than these.
[0206] The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.
[0207] 23 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. The computers that function as the information processing devices 12 and 22 have the same configuration as that shown in FIG.
[0208] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .
[0209] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are connected to the input / output interface 1005. A time-series human behavior DB is constructed in the storage unit 1008.
[0210] In a computer configured as described above, the CPU 1001, for example, loads a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program, thereby realizing the human behavior learning processing unit 101 and the human behavior reproduction processing unit 151.
[0211] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.
[0212] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.
[0213] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
[0214] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.
[0215] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.
[0216] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.
[0217] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.
[0218] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.
[0219] <Examples of Combinations of Configurations> The present technology can also have the following configurations.
[0220] (1) An information processing device comprising: an information processing unit that generates, based on measurement results of a sensor that indicates a plurality of objects including a first person watching a second avatar that reproduces an action corresponding to an action of a second person and the second person watching the first avatar that reproduces an action corresponding to an action of the first person, a first scene graph in which nodes representing the plurality of objects at different times are connected by change edges that indicate temporal changes in state, and selects an action to be reproduced by the second avatar based on similar scene graphs that include a graph structure similar to that of the first scene graph and are included in a plurality of second scene graphs that are generated in advance based on measurement results of a sensor that indicates a plurality of arbitrary objects including a plurality of arbitrary people and in which nodes representing the plurality of arbitrary objects at different times are connected by change edges that indicate temporal changes in state. (2) The information processing device described in (1), wherein the graph structure of the first scene graph further includes relationship edges that connect nodes representing the plurality of objects at each time and indicate relationships between the plurality of objects. (3) The information processing device according to (1) or (2), wherein the change edges of the first scene graph include: a first edge indicating a temporal change in a state of the same person; and a second edge indicating a change in a positional relationship between the plurality of objects. (4) The information processing device according to (3), further comprising a recognition unit that recognizes an action of the first person and an action of the second person, wherein the information processing unit generates the first scene graph using the first edge indicating a temporal change in the state of the same person by a time series of likelihoods of recognition results by the recognition unit. (5) The information processing device according to (3) or (4), wherein the second edges include: an edge that connects a node indicating the first person or the second person with a node indicating a peripheral object and indicates a change in the positional relationship between the person and a peripheral object; and an edge that connects nodes indicating the first person and the second person and indicates a change in the positional relationship between the people.(6) The information processing device according to any one of (3) to (5), wherein the information processing unit selects, as the action to be reproduced by the second avatar, an action that realizes a change in state indicated by the first edge connected to a node representing the second person in the similar scene graph. (7) The information processing device according to (6), wherein the information processing unit adjusts a speed of the action to be reproduced by the second avatar according to a difference between a layout of a reproduction space in which the first person exists and a layout of the space measured when the similar scene graph is generated. (8) The information processing device according to any one of (3) to (7), wherein the information processing unit selects, as a destination of the second avatar, a position of a peripheral object indicated by a node connected to the node representing the second person by the second edge in the similar scene graph. (9) The information processing device according to (8), wherein, when the first scene graph includes a plurality of nodes synonymous with a node connected to a node representing the second person by the second edge, the information processing unit selects movement of the second avatar to a position of a peripheral object represented by one of the plurality of nodes based on a change in appearance of the second avatar resulting from movement to the plurality of nodes. (10) An information processing method in which an information processing device generates a first scene graph in which nodes representing a plurality of objects including a first person looking at a second avatar that reproduces an action corresponding to an action of a second person and the second person looking at the first avatar that reproduces an action corresponding to an action of the first person at different times are connected by change edges that indicate temporal changes in state, based on measurement results from a sensor that indicates the plurality of objects, and selects an action to be reproduced by the second avatar based on similar scene graphs that include a graph structure similar to that of the first scene graph and are included in a plurality of second scene graphs that are generated in advance based on measurement results from a sensor that indicates a plurality of arbitrary objects including any plurality of people and in which nodes representing the plurality of arbitrary objects at different times are connected by change edges that indicate temporal changes in state.(11) A recording medium having recorded thereon a program for causing a computer to execute the following processes: based on measurement results of a sensor indicating a plurality of objects including a first person looking at a second avatar that reproduces an action corresponding to the action of a second person and the second person looking at the first avatar that reproduces an action corresponding to the action of the first person, generate a first scene graph in which nodes indicating the plurality of objects at different times are connected by change edges that indicate temporal changes in state; and select an action to be reproduced by the second avatar based on similar scene graphs including a graph structure similar to that of the first scene graph, which are included in a plurality of second scene graphs that are generated in advance based on measurement results of a sensor indicating a plurality of arbitrary objects including a plurality of arbitrary people and in which nodes indicating the plurality of arbitrary objects at different times are connected by change edges that indicate temporal changes in state. (12) An information processing device comprising: a generation unit that generates a scene graph based on measurement results of a sensor that indicates a plurality of objects including a plurality of people, in which nodes that indicate the plurality of objects at each time are connected by relationship edges that indicate relationships between the plurality of objects, and nodes that indicate the plurality of objects at different times are connected by change edges that indicate changes in state over time. (13) The information processing device described in (12), in which the change edges of the scene graph include a first edge that indicates a change in state of the same person over time, and a second edge that indicates a change in positional relationship between the plurality of objects. (14) The information processing device described in (13), further comprising: a recognition unit that recognizes behavior of a person, in which the generation unit generates the scene graph using the first edge that indicates a change in the state of the same person over time by a time series of likelihood of a recognition result by the recognition unit. (15) The information processing device according to (13) or (14), wherein the second edges include: an edge that connects a node representing a person with a node representing a peripheral object and indicates a change in a positional relationship between the person and the peripheral object; and an edge that connects nodes representing people and indicates a change in a positional relationship between the people.(16) The information processing device according to any one of (12) to (15), wherein the generation unit generates the scene graph at a timing when a positional relationship between the objects changes. (17) An information processing method, wherein an information processing device generates a scene graph based on measurement results of a sensor indicating a plurality of objects including a plurality of people, in which nodes indicating the plurality of objects at each time are connected by relationship edges indicating relationships between the plurality of objects, and nodes indicating the plurality of objects at different times are connected by change edges indicating changes in state over time. (18) A recording medium having recorded thereon a program that causes a computer to execute a process of generating a scene graph based on measurement results of a sensor indicating a plurality of objects including a plurality of people, in which nodes indicating the plurality of objects at each time are connected by relationship edges indicating relationships between the plurality of objects, and nodes indicating the plurality of objects at different times are connected by change edges indicating changes in state over time.
[0221] REFERENCE SIGNS LIST 1 AR display device, 11 measurement device, 12 information processing device, 21A, 21B measurement device, 22A, 22B information processing device, 101 human behavior learning processing unit, 111 recognition unit, 112 spatiotemporal scene graph generation unit, 113 recording control unit, 151 human behavior reproduction processing unit, 161 recognition unit, 162 human behavior prediction unit, 162A spatiotemporal scene graph generation unit, 163 presentation unit
Claims
1. An information processing device comprising: an information processing unit that generates a first scene graph in which nodes representing a first person looking at a second avatar that reproduces behavior corresponding to the behavior of a second person and a second person looking at the first avatar that reproduces behavior corresponding to the behavior of the first person are connected with change edges indicating changes in state over time based on measurement results from a sensor that indicates a plurality of objects including the first person; and selects an action to be reproduced by the second avatar based on similar scene graphs including a graph structure similar to that of the first scene graph, the similar scene graphs being included in a plurality of second scene graphs that are generated in advance based on measurement results from a sensor that indicates a plurality of arbitrary objects including an arbitrary plurality of persons and in which nodes representing the plurality of arbitrary objects at different times are connected with change edges indicating changes in state over time.
2. The information processing device according to claim 1, wherein the graph structure of the first scene graph further includes relationship edges that connect nodes representing the plurality of objects at each time and indicate relationships between the plurality of objects.
3. The information processing device according to claim 1, wherein the change edges of the first scene graph include a first edge indicating a temporal change in the state of the same person, and a second edge indicating a change in the positional relationship of the multiple objects.
4. An information processing device as described in claim 3, further comprising a recognition unit that recognizes the behavior of the first person and the behavior of the second person, wherein the information processing unit generates the first scene graph using the first edge that indicates a change in the state of the same person over time by a time series of the likelihood of the recognition result by the recognition unit.
5. The information processing device according to claim 3, wherein the second edges include: edges that connect a node representing the first person or the second person with a node representing a peripheral object and indicate a change in the positional relationship between the person and a peripheral object; and edges that connect the nodes representing the first person and the second person and indicate a change in the positional relationship between the people.
6. The information processing device according to claim 3, wherein the information processing unit selects, as an action to be reproduced by the second avatar, an action that realizes a change in state indicated by the first edge connected to a node indicating the second person in the similar scene graph.
7. The information processing device according to claim 6, wherein the information processing unit adjusts a speed of the action reproduced by the second avatar in accordance with a difference between a layout of a reproduction space in which the first person exists and a layout of the space measured when the similar scene graph is generated.
8. The information processing device according to claim 3, wherein the information processing unit selects, in the similar scene graph, a position of a surrounding object indicated by a node connected to a node indicating the second person by the second edge, as a destination of the second avatar.
9. The information processing device according to claim 8, wherein, when the first scene graph includes a plurality of nodes synonymous with the node connected to the node representing the second person by the second edge, the information processing unit selects to move the second avatar to a position of a surrounding object indicated by one of the plurality of nodes based on a change in the appearance of the second avatar resulting from movement to the plurality of nodes.
10. An information processing method comprising: an information processing device generating a first scene graph in which nodes representing a first person looking at a second avatar reproducing an action corresponding to the action of a second person and a second person looking at the first avatar reproducing an action corresponding to the action of the first person, based on measurement results from a sensor indicating a plurality of objects including the first person, and in which nodes representing the plurality of objects at different times are connected by change edges indicating changes in state over time; and selecting an action to be reproduced by the second avatar based on similar scene graphs including a graph structure similar to that of the first scene graph, the similar scene graphs being included in a plurality of second scene graphs that have been generated in advance based on measurement results from a sensor indicating a plurality of arbitrary objects including an arbitrary plurality of persons, and in which nodes representing the plurality of arbitrary objects at different times are connected by change edges indicating changes in state over time.
11. A recording medium having a program recorded thereon to cause a computer to execute the following processes: based on measurement results of a sensor indicating a plurality of objects including a first person looking at a second avatar which reproduces an action corresponding to the action of a second person, and the second person looking at the first avatar which reproduces an action corresponding to the action of the first person, generate a first scene graph in which nodes indicating the plurality of objects at different times are connected by change edges indicating changes in state over time; and select an action to be reproduced by the second avatar based on similar scene graphs including a graph structure similar to that of the first scene graph, the similar scene graphs being included in a plurality of second scene graphs which are generated in advance based on measurement results of a sensor indicating a plurality of arbitrary objects including any plurality of persons, and in which nodes indicating the plurality of arbitrary objects at different times are connected by change edges indicating changes in state over time.
12. An information processing device comprising: a generation unit that generates a scene graph based on measurement results of a sensor indicating multiple objects including multiple people, in which nodes indicating the multiple objects at each time are connected by relationship edges indicating relationships between the multiple objects, and nodes indicating the multiple objects at different times are connected by change edges indicating changes in state over time.
13. The information processing device according to claim 12, wherein the change edges of the scene graph include a first edge indicating a temporal change in the state of the same person, and a second edge indicating a change in the positional relationship of the multiple objects.
14. An information processing device as described in claim 13, further comprising a recognition unit that recognizes the behavior of a person, wherein the generation unit generates the scene graph using the first edge that indicates a change in the state of the same person over time by a time series of the likelihood of the recognition result by the recognition unit.
15. The information processing device according to claim 13, wherein the second edges include: edges that connect nodes representing people and nodes representing surrounding objects and indicate a change in the positional relationship between the people and surrounding objects; and edges that connect nodes representing people and indicate a change in the positional relationship between the people.
16. The information processing device according to claim 12, wherein the generation unit generates the scene graph when a positional relationship between the objects changes.
17. An information processing method in which an information processing device generates a scene graph based on measurement results of a sensor indicating multiple objects including multiple people, in which nodes indicating the multiple objects at each time are connected by relationship edges indicating relationships between the multiple objects, and nodes indicating the multiple objects at different times are connected by change edges indicating changes in state over time.
18. A recording medium having recorded thereon a program that causes a computer to execute a process of generating a scene graph based on the measurement results of a sensor indicating multiple objects including multiple people, in which nodes indicating the multiple objects at each time are connected by relationship edges indicating the relationships between the multiple objects, and nodes indicating the multiple objects at different times are connected by change edges indicating changes in state over time.
Citation Information
Patent Citations
Information processing device, information processing method, and program
WO2020012727A1
Data analysis device and data analysis method
JP2022086650A
Effectively Stream Augmented Reality Data from Third-Party Systems
JP2022549986A
Scene-aware video dialogue
JP2023510430A
Information processing device and information processing method
WO2021192686A1
