Information processing device and information processing method

Abstract

A contents display device (20) that functions as an information processing device is provided with: an acquisition unit (251); a determination unit (252); and a display control unit (253). The acquisition unit (251) acquires template data that define a relative relationship among a first real object, a second real object, and a virtual object. The determination unit (252) determines the relative relationship between the first real object and the second real object from three-dimensional data of a real space which has been detected by a sensor. The display control unit (253) controls a display device, on the basis of the relative relationship and the template data, such that the virtual object is arranged above or near the first real object so as to face the second real object.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and an information processing method. [Background technology]

[0002] Conventionally, in AR (Augmented Reality) technology, a real object and one or more virtual digital contents are provided to a user in an augmented reality space perceived by the user in a state where they are blended together as if they coexist. Such AR technology is used in a wide variety of applications, including industrial design, medicine, and entertainment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2016-516241 Summary of the Invention [Problem to be solved by the invention]

[0004] The challenge with the above AR technology is how to naturally blend virtual content into real space.

[0005] Therefore, the present disclosure proposes an information processing device and an information processing method that can realize content arrangement corresponding to the spatial context of the real space. [Means for solving the problem]

[0006] In order to solve the above problem, an information processing device according to an embodiment of the present disclosure includes an acquisition unit, a determination unit, and a display control unit. The acquisition unit acquires template data that defines a relative relationship between a first real object, a second real object, and a virtual object. The determination unit determines the relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by a sensor. The display control unit controls the display device to place a virtual object on or near the first real object so as to face the second real object, based on the relative relationship and the template data. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of an information processing system according to an embodiment. [Diagram 2] FIG. 1 illustrates an example of the configuration of a template creation device. [Diagram 3] FIG. 13 is a diagram showing an example of an AR content placement image and a corresponding template scene graph. [Figure 4] FIG. 11 is a diagram illustrating an example of priority setting. [Diagram 5] FIG. 13 is a diagram illustrating an example of a relation condition. [Figure 6] FIG. 1 is a diagram showing an example of the configuration of a content display device. [Figure 7] FIG. 1 is a diagram showing an example of a three-dimensional space recognition map. [Figure 8] FIG. 13 is a diagram showing an example of information accessible from a scene graph generated from a three-dimensional spatial recognition map. [Figure 9] FIG. 2 is a diagram showing an example of a three-dimensional space recognition map at a certain time and a scene graph corresponding to the three-dimensional space recognition map. [Figure 10] FIG. 13 is a diagram illustrating an example of mapping of AR content to a scene graph at a certain time. [Figure 11] 10A and 10B are diagrams illustrating an example of a three-dimensional space recognition map and a scene graph after a certain time has elapsed from the situation illustrated in FIG. 9. [Figure 12]FIG. 13 is a diagram illustrating an example of a comparison of scene graphs at different times. [Figure 13] FIG. 11 is a diagram showing an example of a positional relationship candidate that can realize AR content. [Figure 14] FIG. 11 is a diagram showing an example of a positional relationship candidate that can realize AR content. [Figure 15] 13A and 13B are diagrams illustrating an example of a list after matching each positional relationship candidate with a relationship condition, and a corresponding scene graph. [Figure 16] 13A and 13B are diagrams illustrating examples of evaluation of the suitability of positional relationship candidates. [Figure 17] 13A and 13B are diagrams illustrating examples of evaluation of the suitability of positional relationship candidates. [Figure 18] 13A and 13B are diagrams illustrating an example of a final determination of positional relationship candidates. [Figure 19] FIG. 2 is a diagram showing the relationship between the functions of the template creation device and the content display device. [Figure 20] 13 is a flowchart illustrating an example of a processing procedure of the template creation device. [Figure 21] 13 is a flowchart showing an example of a processing procedure of a content display device. [Figure 22] FIG. 11 is a diagram showing a detailed procedure for listing up positional relationship candidates. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts may be denoted by the same reference numerals, and duplicated descriptions may be omitted. In addition, the present disclosure relates to various XR technologies such as AR, VR, and MR, but in the following embodiments, in particular, an information processing method that recognizes a real space (actual space) and naturally blends the real space with a virtual object (AR content) to provide the user with the information is described.

[0009] The present disclosure will be described in the following order. 1. System configuration example 2. Functional configuration example 2-1. Functional configuration of the template creation device 2-2. Functional configuration of content display device 2-3. Relationships between functions 3. Example of processing procedure 3-1.Template creation device processing procedure 3-2. Processing procedure of content display device 4.Other 5. Conclusion

[0010] <<1. System configuration example>> Fig. 1 is a diagram showing an example of the configuration of an information processing system according to an embodiment. As shown in Fig. 1, the information processing system 1 according to an embodiment includes an RGB-D sensor 2, a template creation device 10, and a content display device 20. The template creation device 10 and the content display device 20 can realize transmission and reception or exchange of various information via a network or an external storage device.

[0011] The RGB-D sensor 2 measures the real space and acquires three-dimensional data including RGB images and depth data. The RGB-D sensor 2 can be realized by, for example, an RGB-D camera, a combination of an RGB camera and a depth sensor, or a sensor such as LiDAR. The RGB-D sensor 2 functions as a sensor that detects three-dimensional data of the real space.

[0012] The template creation device 10 is an information processing device that executes, for example, various processes related to the template creation phase in an offline state, and can be realized by, for example, a PC (Personal Computer).

[0013] As shown in FIG. 1, the template creation device 10 includes an input / output interface 11, a storage device 12, and a control device 13.

[0014] The input / output interface 11 acquires the template space model and inputs it to the control device 13. The template space model is a three-dimensional space model that is used by a content creator to consider how to arrange AR content in the real space.

[0015] The storage device 12 stores data and the like used in various processes related to the template creation phase. The storage device 12 is realized by a data readable / writable storage device such as a DRAM, an SRAM, a flash memory, a hard disk, etc. The storage device 12 has, for example, a content DB, a related condition DB, a template SG (scene graph) DB, and the like, which will be described later.

[0016] The control device 13 is a controller that executes various processes of the template creating device 10. The controller is realized by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the processor executes various programs stored in a storage device inside the template creating device 10 using a RAM (Random Access Memory) or the like as a working area, thereby realizing various processes related to the template creation phase described below. The controller may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Any of the CPU, MPU, ASIC, and FPGA can be considered as a controller.

[0017] The content display device 20 is an information processing device that executes various processes related to the experience phase of AR content in an online state. The content display device 20 can be realized, for example, by a see-through or video-through type HMD (Head Mounted Display), a wearable terminal such as a smartphone, or a laptop PC (Personal Computer).

[0018] As shown in FIG. 1, the content display device 20 includes an RGB-D data input interface 21, a display device 22, a storage device 23, and a control device 24.

[0019] The RGB-D data input interface 21 acquires RGB-D data, which is three-dimensional data of the real space, from the RGB-D sensor 2, and inputs it to the control device 24.

[0020] The display device 22 displays various information such as AR content.

[0021] The storage device 23 stores data and the like used for various processes related to the experience phase of the AR content. The storage device 23 is realized by a data readable / writable storage device such as a DRAM, an SRAM, a flash memory, a hard disk, etc. The storage device 23 has, for example, a content DB, a relational condition DB, a template SG (scene graph) DB, and a past (template) SG DB, which will be described later.

[0022] The control device 24 is a controller that executes various processes of the content display device 20. The controller is realized by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). For example, the processor executes various programs stored in a storage device inside the content display device 20 using a RAM (Random Access Memory) or the like as a working area, thereby realizing various processes related to the experience phase of the AR content described below. The controller may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Any of the CPU, MPU, ASIC, and FPGA can be considered as a controller.

[0023] A content creator creates AR content by imagining a scene to be realized and operating the template creation device 10 so that the content arrangement is in accordance with the image. The video data of AR content is created so as to reflect the intention and scene image of the content creator as much as possible. In the creation of AR content, it is an issue to realize content mapping that maintains the intention and scene image of the content creator even if the spatial context (spatial shape, relationship, etc.) changes. Therefore, as described below, in the embodiment of the present disclosure, a template scene graph that defines the scene image to be realized is generated, and content mapping according to the spatial context and the change in the spatial context is performed based on this template scene graph. This realizes the creation of video data of AR content that is close to the scene image of the content creator and has the same meaning as a scene even if the spatial context (spatial shape, relationship, etc.) changes.

[0024] <<2. Example of functional configuration>> <2-1. Configuration example of template creation device> An example of the configuration of a template creating device will be described with reference to Figures 2 to 5. Figure 2 is a diagram illustrating an example of the configuration of a template creating device.

[0025] As shown in FIG. 2, the template creation device 10 includes an input unit 110, an output unit 120, a communication unit 130, a storage unit 140, and a control unit 150.

[0026] Each block (input unit 110 to control unit 150) included in the template creation device 10 is a functional block indicating a function of the template creation device 10. These functional blocks may be software blocks or hardware blocks. For example, each of the above-mentioned functional blocks may be one software module realized by software (including a microprogram), or one circuit block on a semiconductor chip (die). Of course, each functional block may be one processor or one integrated circuit. The functional blocks may be configured in any manner. Each block included in the template creation device 10 may be configured in a functional unit different from the example shown in FIG. 2.

[0027] The input unit 110 is realized by various input devices such as a keyboard, a mouse, a touch panel, etc., and inputs various information. The input unit 110 acquires a template space model, which is a typical space layout for executing an AR application, and inputs it to the template creation device 10.

[0028] The template space model is a three-dimensional space model that a content creator uses to consider how to arrange AR content, and includes the three-dimensional shape of the space (shape of each object), identification information of each object, object category, and relationships between objects. The template space model is generated by three-dimensional space recognition technology. Note that the template space model may be a CAD (Computer-Aided Design) model as long as it includes the three-dimensional shape of the space (shape of each object), identification information of each object, object category, and relationships between objects.

[0029] The output unit 120 is realized by various output devices such as a display and a speaker, and outputs various information.

[0030] The communication unit 130 is realized by a communication module for performing communication via a network. The communication unit 130 transmits and receives information to and from the content display device 20 via the network, for example.

[0031] The storage unit 140 is realized by the above-mentioned storage device 12 or the like, and stores data and the like used in various processes related to the template creation phase. The storage unit 140 has a content DB 141, a relational condition DB 142, and a template scene graph DB 143, as shown in FIG.

[0032] The content DB 141 stores data of AR content. The relationship condition DB 142 stores relationship conditions for AR content to perform appropriate placement and interaction in response to a spatial context and a change in the spatial context. The template scene graph DB 143 stores placement rules for AR content that abstractly express the relationship between the template space model and the AR content.

[0033] The control unit 150 is realized by the above-mentioned control device 13, and executes various processes related to the template creation phase. The control unit 150 has a creation unit 151 and a setting unit 152, as shown in FIG.

[0034] The creation unit 151 executes generation of a template scene graph by content arrangement for the template space model. Fig. 3 is a diagram showing an example of an arrangement image of AR content and a corresponding template scene graph.

[0035] For example, it is assumed that a person 3-1 and a sitable object 3-3 exist in the input template space model. Then, it is assumed that the content creator assumes the realization of a scene SC in which a virtual character 3-2 (hereinafter, appropriately described as "character A"), which is an AR content, sits next to the person 3-1 who is sitting on the sitable object 3-3 as shown in the left diagram of FIG. 3, and talks with the person 3-1. At this time, the content creator assumes a scene in the template space model in which the person 3-1 sits on the sitable object 3-3 ("sitting on"), the "character A" sits on the sitable object 3-3 ("sitting on"), and the "character A" talks with the person 3-1 ("talking to"), and sets the relationship (relationship label) between each node ("person", "character A", "sitable object"). Upon receiving the input of such settings, the creation unit 151 generates a template scene graph TSG shown in the right diagram of FIG. 3. That is, the template scene graph TSG is template data that defines the relative relationship between a first real object (e.g., a sitting object 3-3), a second real object (e.g., a person 3-1), and a virtual object (e.g., a character A) in the AR content. The relative relationship includes the relative positional relationship between the first real object, the second real object, and the virtual object, as well as an interaction action that the virtual object can take with at least one of the first real object or the second real object. In the template scene graph, a relationship label such as "sitting on" or "talking to" that defines the relationship between nodes is assigned. For example, the relationship label "sitting on" is a label for defining the interaction of "character A" to make it sit on the sitting object 3-3. Also, the relationship label "talking to" is a label for defining the interaction of "character A" to make it talk with the person 3-1.

[0036] In the content experience phase, the layout of the AR content is determined to realize this template scene graph TSG. This allows the content layout to reflect the intention of the content creator. In addition, multiple template scene graphs TSG can be included in an AR application, and by switching to another template scene graph after "Character A" finishes talking, it is possible to express the temporal behavioral changes and situational changes of the AR content. In this way, by using the template scene graph TSG that abstractly expresses the relationship between objects in the template space model, it is possible to perform processing that quickly tracks the temporal behavioral changes and situational changes of the AR content without the need to process low-level data such as meshes.

[0037] The template scene graph TSG may be generated in response to the operation of the content creator, or the creation unit 151 may automatically infer a scene that can be realized in the template space model, and automatically generate a template scene graph TSG according to the inferred scene.

[0038] Setting unit 152 sets priorities for relationships between nodes in template scene graph TSG generated by creation unit 151. These priorities indicate the degree of relative priority of each relationship among the relationships between nodes in template scene graph TSG. Fig. 4 is a diagram showing an example of priority setting.

[0039] When the relationships between the nodes in the template scene graph TSG are treated equally, the setting unit 152 sets, for example, the priority between all the nodes in the template scene graph TSG to the same "1", as in the template scene graph TSG1-1 shown in the left diagram of Fig. 4. Also, when priority is given to "character A" conversing with a person over sitting on an object on which a person can sit and sitting on an object on which "character A" can sit, the setting unit 152 sets, for example, the priority of the relationship in which "character A" converses with a person to "2", as in the template scene graph TSG1-2 shown in the right diagram of Fig. 4, and sets the priority of relationships other than the relationship in which "character A" converses with a person to "1", which is lower than the priority of the relationship in which "character A" converses with a person.

[0040] Next, the setting unit 152 sets a relation condition for the AR content to perform appropriate placement and interaction in response to the spatial context and the change in the spatial context. Fig. 5 is a diagram showing an example of the relation condition.

[0041] The relationship conditions are set by the content creator to allow the AR content to be placed and interacted with as intended, and are set based on physical norms, social norms, time norms, etc.

[0042] The conditions based on physical norms predetermine whether the AR content ("Character A"), which is a virtual object, can physically interact with an object. As an example of the conditions based on physical norms, as shown in FIG. 5, in order for the positional relationship of the relationship label "sitting on" to be established between nodes of the template scene graph TSG, there must be no obstacle on the seat of the target object to be sat on. Also, as shown in FIG. 5, in order for the positional relationship of the relationship label "talking to" to be established between nodes of the template scene graph TSG, there must be an example of the condition that the conversation partner is within 2 meters.

[0043] In addition, conditions based on social norms prescribe whether AR content ("Character A") can interact with an object, assuming a situation that may be physically possible but is socially unacceptable, such as sitting on a chair where someone else is sitting. An example of a condition based on social norms is that the target object to be sat on must not be a chair on which a person is already sitting (an object that can be sat on), in order to establish a positional relationship, for example, of the related label "sitting on" between nodes in the template scene graph TSG.

[0044] In addition, the conditions based on the time norm are set in advance so that the AR content can interact with the object as naturally as possible, assuming that the relationship between objects including the AR content ("Character A") changes according to the movement of the object in the real space with the passage of time, such as from the past to the present or from the present to the future. An example of the conditions based on the time norm is that, when the positional relationship of the relationship label "sitting on" was established in the past, 10 seconds or more must have passed since the positional relationship of the relationship label "sitting on" was established in the past in order for the positional relationship of the relationship label "sitting on" to be established in the present. In addition, regardless of the past relationship label ("any"), in order for the positional relationship of the relationship label "sitting on" to be established in the present, it is possible to sit down within 5 seconds. In addition, when performing control such as switching the template scene graph TSG according to the progress of the AR application, an example of the condition is that, when switching to the next template scene graph TSG, in order for the relationship of the relationship label "sitting on" to be established in the present, it is necessary for the positional relationship of the relationship label "talking to" or "holding" to be established in the future.

[0045] Among the above-mentioned relational conditions, the conditions based on physical norms and social norms have no time constraints and function as conditions for determining the time-dependent positional relationship of AR content according to the recognition result of the real space, whereas the conditions based on temporal norms have time constraints and function as conditions for determining the time-dependent positional relationship of AR content from past → present → future.

[0046] The above-mentioned relationship conditions are used in the content experience phase to narrow down a list of positional relationship candidates for placing AR content in a current scene graph that abstractly represents the current real space and evaluating the degree of conformance with the template scene graph TSG. By abstractly representing the placement of AR content in the template scene graph TSG, it becomes possible to narrow down positional relationship candidates based on abstract relationships. In addition, conditions based on temporal norms set as relationship conditions make it possible to place AR content according to temporal changes in the real space. The template scene graph TSG and relationship conditions created in the above-mentioned template creation phase are recorded and used for appropriate AR content placement processing (content mapping processing) in the content experience phase.

[0047] <2-2. Example of the configuration of a content display device> An example of the configuration of a content display device will be described with reference to Fig. 6 to Fig. 18. Fig. 6 is a diagram showing an example of the configuration of a content display device.

[0048] As shown in FIG. 6, content display device 20 includes an input section 210, an output section 220, a communication section 230, a storage section 240, and a control section 250.

[0049] Each block (input section 210 to control section 250) provided in content display device 20 is a functional block indicating a function possessed by content display device 20. These functional blocks may be software blocks or hardware blocks. For example, each of the above-mentioned functional blocks may be a single software module realized by software (including a microprogram), or may be a single circuit block on a semiconductor chip (die). Of course, each functional block may be a single processor or a single integrated circuit. The functional blocks may be configured in any manner. Each block provided in content display device 20 may be configured in functional units different from the example shown in FIG. 6.

[0050] The input unit 210 is realized by various input devices such as a keyboard, a mouse, a touch panel, etc., and inputs various information. The input unit 210 also includes an RGB-D data input interface 21, and acquires RGB-D data from, for example, the RGB-D sensor 2 (see FIG. 1) and inputs it to the content display device 20. The RGB-D data is composed of RGB image and depth data. The RGB-D data functions as three-dimensional data of the real space detected by the RGB-D sensor 2.

[0051] The output unit 220 is realized by various output devices such as a display and a speaker that function as the above-mentioned display device 22, and outputs various information. The output unit 220 can output, for example, video data onto which AR content is appropriately mapped.

[0052] The communication unit 230 is realized by a communication module for performing communication via a network. The communication unit 230 transmits and receives information to and from the template creation device 10 via the network, for example.

[0053] The storage unit 240 is realized by the above-mentioned storage device 23 or the like, and stores data and the like used in various processes related to the content experiencing phase. As shown in Fig. 2, the storage unit 240 has a content DB 241, a relational condition DB 242, a template scene graph DB 243, and a past template scene graph DB 244.

[0054] The content DB 241 stores data of AR content acquired from the template creation device 10. The relationship condition DB 242 stores data of relationship conditions acquired from the template creation device 10. As described above, the relationship conditions are conditions for AR content to perform appropriate placement and interaction in response to a spatial context and a change in the spatial context (see FIG. 5). The template scene graph DB 143 and the past template scene graph DB 144 store data of the template scene graph TSG acquired from the template creation device 10. As described above, the template scene graph TSG is data indicating a placement rule of AR content that abstractly expresses the relationship between the template space model and the AR content.

[0055] The control unit 250 is realized by the above-mentioned control device 24, and executes various processes related to the content experiencing phase. The control unit 250 has an acquisition unit 251, a determination unit 252, and a display control unit 253, as shown in FIG.

[0056] The acquisition unit 251 acquires the template scene graph TSG. The template scene graph TSG is template data (an example of scene graph data) that defines a relative relationship between a first real object, a second real object, and a virtual object in, for example, AR content. The relative relationship includes a relative positional relationship between the first real object, the second real object, and the virtual object, as well as an interaction action that the virtual object can take with at least one of the first real object or the second real object. The interaction action includes an AR content (for example, "character A") that is a virtual object sitting on the first real object (for example, an object that can be sat on), holding the first real object (for example, an object that can be held), and talking to the second real object (for example, a person).

[0057] The determination unit 252 determines the relative relationship between the first real object and the second real object from RGB-D data, which is three-dimensional data of the real space detected by the sensor (RGB-D sensor 2). That is, the determination unit 252 determines the relative relationship between the first real object and the second real object based on scene graph data that abstractly represents the real space from the RGBD-D data.

[0058] Specifically, first, the determination unit 252 generates a three-dimensional space recognition map from the RGB-D data. The three-dimensional space recognition map is a map constructed based on the real world including information on the shape (mesh, point cloud, etc.) and category (chair, sofa, table, etc.) of objects in the real space, and individual object identification (chair #1, chair #2, etc.). For the process of generating the three-dimensional space recognition map, for example, the technology disclosed in the non-patent literature "Narita et al., "PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things", IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS2019), 2019." and the non-patent literature "Grinvald et al., "Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery", IEEE Robotics and Automation Letters, 2019." can be used. FIG. 7 is a diagram showing an example of the three-dimensional space recognition map. Based on the real space RS shown in the left diagram of Fig. 7, it is displayed as a three-dimensional space recognition map MP shown in the right diagram of Fig. 7. In the three-dimensional space recognition map MP, not only are the shapes of individual objects in the real space expressed, but they can also be expressed separated in different colors, although this is not shown in detail in Fig. 7.

[0059] Next, the determination unit 252 recognizes the relationship between objects (a first real object and a second real object) based on the three-dimensional space recognition map (see FIG. 7, for example), and generates a current scene graph (hereinafter, referred to as a "scene graph") that abstractly represents the space. The relationship between objects may be a positional relationship based on distance or direction, or an interaction (sitting, talking) that represents an action between objects. Among the former positional relationships, the relationship based on distance norms such as close and far can be easily calculated from the three-dimensional space recognition map. As for the relationship based on the direction norms of objects such as front, back, left, right, up, and down, it is necessary to estimate the pose of each object recognized in the three-dimensional space recognition map. For pose estimation, a technology for estimating the pose of an object from RGB-D data disclosed in the literature "Wang et al., "Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019." may be used.

[0060] A method of recognizing interactions as relationships can be used that uses a method of recognizing interactions from RGB images disclosed in the literature "Xu et al., "Scene Graph Generation by Iterative Message Passing", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017., and integrates the image recognition results into a three-dimensional space recognition map. Alternatively, a method of recognizing interactions from three-dimensional positional relationships between objects disclosed in the literature "Savva et al., "PiGraphs: Learning Interaction Snapshots from Observations", ACM Transactions on Graphics (TOG), 2016., can be used. By integrating the relationships between the objects obtained in this way, a scene graph that abstractly represents the entire scene can be generated. The content display device 20 executes a process of content placement (content mapping) based on this scene graph. FIG. 8 is a diagram showing an example of information that can be accessed from a scene graph generated from a three-dimensional space recognition map.

[0061] As shown in FIG. 8, the scene graph 8-1 includes information such as a bounding box 8-2, a mesh 8-3, and a TSDF & label map 8-4 used to generate the scene graph 8-1. The objects included in each piece of information are linked by the same object ID. The scene graph 8-1 is an abstract representation of the relationship between objects, while the bounding box 8-2 is an abstract representation of the shape and position of each object, and includes information such as the outer dimensions, position, and direction of the object. The mesh 8-3 is a mesh representation of the 3D space recognition map, and includes information on the point cloud of the object, polygons (surfaces) (connections between points), and labels for each point and polygon. The TSDF & label map 8-4 is a volumetric representation of the 3D space using multiple voxels, and is configured by giving each voxel the distance to the surface of the object, weight, and reliability of the label to be assigned.

[0062] Fig. 9 is a diagram illustrating an example of a three-dimensional space recognition map at a certain time and a scene graph corresponding to the three-dimensional space recognition map, and Fig. 10 is a diagram illustrating an example of mapping of AR content to the scene graph at a certain time.

[0063] The above process generates a time t 0 3D spatial recognition map MP_t 0 (See the left diagram in Figure 9) is a scene graph CSG1_t 0 (See the right diagram in Figure 9.) The 3D spatial recognition map MP_t 0 As a result of spatial recognition, for example, a "person" 9-1, a "sofa" 9-2, a "chair A" 9-3, a "table" 9-4, a "chair B" 9-5, and a "TV" 9-6 are included in the image. The "sofa" 9-2, "chair A" 9-3, the "table" 9-4, "chair B" 9-5, and the "TV" 9-6 are examples of a first real object. The "person" 9-1 is an example of a second real object. For convenience of explanation, the time t 0 The content mapping process in the scene graph CSG1_t is explained below. 0 The display control unit 253 performs content mapping using a template scene graph TSG (see the right diagram in FIG. 3) on the scene graph CSG2_t shown in FIG. 0 is obtained. In this way, content arrangement corresponding to the spatial context is realized based on a scene graph that abstractly represents the relationships between objects existing in the real space and a template scene graph that abstractly represents the scene image of the content creator. In the following explanation, "person" 9-1 is simply referred to as "person", "sofa" 9-2 is simply referred to as "sofa", "chair A" 9-3 is simply referred to as "chair A", "table" 9-4 is simply referred to as "table", "chair B" 9-5 is simply referred to as "chair B", and "TV" 9-6 is simply referred to as "TV".

[0064] FIG. 11 is a diagram showing an example of a three-dimensional space recognition map and a scene graph after a certain time has elapsed from the situation shown in FIG. 9. FIG. 12 is a diagram showing a comparative example of a scene graph at a different time. For convenience of explanation, 0 From time t 1 It is assumed that 10 seconds or more have elapsed until

[0065] As shown in FIG. 11, in the real space, at time t 1 Suppose that at some time t 1 3D spatial recognition map MP_t 1 (See the bottom left diagram in Figure 11) is a scene graph CSG_t 1 (See the lower right diagram in FIG. 11). The determination unit 252 uses the scene graph CSG2_t 0 and the scene graph CSG_t 1 The determination unit 252 compares the scene graph CSG2_t with the scene graph CSG2_t to determine the movement of the first real object (such as "sofa", "chair A", or "chair B") and the second real object (such as "person"). 0 and the scene graph CSG1_t 1 By comparing with time t 0 From time t 1 Detect changes in the relationships between objects over time.

[0066] time t 0 From time t 1 In detecting a change in the relationship between objects over time, the determination unit 252 determines whether or not a change in the relationship has occurred for each object having a relationship with the AR content "Character A." The determination unit 252 then assigns a mark indicating that a change has occurred (hereinafter, referred to as a "change mark") to the AR content related to the object whose relationship has changed. For example, as shown in FIG. 12, the determination unit 252 applies a mark indicating that a change has occurred to the AR content related to the object whose relationship has changed. 0 and the scene graph CSG_t 1By comparing with the above, the "person" who had a relationship with "Character A" has changed from "sitting on" with the "sofa" to "sitting on" with "Chair B", so a change mark (e.g., figure 12-1) is added to "Character A" related to the "person". There may also be cases where you want to consider changes in objects that are not directly related to the AR content. In such cases, you can set the distance from the AR content to the object that will be candidates for change detection based on the distance on the scene graph (how many edges need to be passed from the AR content to the object on the scene graph) or the Euclidean distance between the AR content and the object, etc.

[0067] The display control unit 253 controls the display device 22 to place a virtual object (e.g., character A) on or near a first real object (e.g., chair A or chair B) so as to face a second real object (e.g., a person) based on a relative relationship (e.g., scene graph) between the first real object and the second real object and template data (e.g., template scene graph TSG). After placing the virtual object, the display control unit 253 executes an interaction operation of the virtual object with respect to the second real object. When the three-dimensional data (RGB-D data) includes a plurality of first real objects, the display control unit 253 places a virtual object on a first real object on which no other real object is placed, based on the template data. When the three-dimensional data includes a plurality of first real objects, the display control unit 253 places a virtual object on one of the plurality of first real objects that is closer to the second real object. Moreover, the display control unit 253 controls the display device 22 not to place a virtual object when the distance between the first real object and the second real object is greater than or equal to a predetermined distance in the relative relationship. Moreover, the display control unit 253 rearranges the virtual object on the first real object on the condition that a certain time or more has passed since the previous interaction operation of the virtual object. Moreover, the display control unit 253 rearranges the virtual object on the first real object on the condition that the virtual object can perform a predetermined interaction operation on the first real object within a certain time. Moreover, the display control unit 253 rearranges the virtual object on the first real object on the condition that the virtual object is in a positional relationship capable of performing a predetermined interaction operation on the second real object when a scene of a video content including the first real object, the second real object, and the virtual object is switched. The display control unit 253 can perform the arrangement and rearrangement of the virtual object based on the relationship condition DB 241.

[0068] Specifically, the display control unit 253 lists, in the scene graph, positional relationship candidates that can realize the relationships that each AR content has in the template scene graph for content that has a change mark added (hereinafter referred to as "changed content") and AR content that has not yet been set as a relationship in the scene graph in the template scene graph (hereinafter referred to as "unused content").

[0069] First, the display control unit 253 makes a list of positional relationship candidates for clarifying the positions of changed content and unused content. A positional relationship candidate refers to a relationship label that clarifies (determines) the positional relationship between objects (e.g., the placement position of AR content) by the relationship label, such as the relationship label "sitting on" described in the template scene graph TSG (see the right diagram in FIG. 3). The display control unit 253 holds information regarding which relationship label can clarify the position as a list in advance. FIGS. 13 and 14 are diagrams showing examples of positional relationship candidates that can realize AR content.

[0070] Scene Graph CSG_t 1In this case, the positional relationship candidates that can realize the positional relationship of the relationship label "sitting on" set for "Character A" in the template scene graph TSG are three (arrangements (1) to (3)) shown in the relationship list L1 of FIG. 13. The objects that can realize the positional relationship of the relationship label "sitting on" for "Character A" correspond to objects that can be sat on, such as "sofa", "chair A", and "chair B", and do not include "table" or "TV". The display control unit 253 determines objects that can be sat on from among the multiple objects included in the scene graph by creating a list of positional relationships and interaction labels that can be taken for object categories in advance. By creating a list of positional relationships and interaction labels that can be taken for object categories in advance, for example, "table" and the like can be excluded from the search target in advance, and a full search of candidates in the scene graph can be avoided. If the contents shown in the relationship list L1 are expressed in a scene graph, it will be, for example, a scene graph CSG_L1 shown in FIG. 13.

[0071] Next, the display control unit 253 further lists positional relationship candidates between the changed content and the object in each of the positional relationship candidates listed in the relationship list L1. In the template scene graph TSG (see, for example, the right diagram in FIG. 3), the positional relationship other than "sitting on" required for "character A" is "talking to" with respect to "person". Accordingly, as shown in the relationship list L2 in FIG. 14, positional relationship candidates between "character A" and "person", which are the changed content in each of the positional relationship candidates listed in the relationship list L1, are listed. That is, in the arrangements (1) to (3), "talking to" between "character A" and "person" is further listed. If the contents shown in the relationship list L2 are expressed in a scene graph, it will become like the scene graph CSG_L2 shown in FIG. 14.

[0072] When the list of positional relationship candidates between the changing content and the object is completed, the display control unit 253 checks each positional relationship candidate against a relationship condition (for example, see FIG. 5) and extracts a valid positional relationship candidate from each positional relationship candidate. That is, the display control unit 253 determines whether or not a relationship establishment condition defined in advance is satisfied for a relationship label corresponding to each positional relationship candidate.

[0073] Specifically, the display control unit 253 searches for a condition corresponding to the current relationship label "talking to" from the relationship conditions for the positional relationship of "character A" to "person" in arrangement (1) of the relationship list L2, and determines whether the relationship establishment condition "conversational partner is within 2 m (meters)" is satisfied. The position of each object in the scene graph can be acquired from lower-level information as shown in FIG. 7, and it can be determined whether the relationship establishment condition corresponding to the relationship label "talking to" is satisfied, for example, from the Euclidean distance between each object. The display control unit 253 also determines whether the relationship establishment condition is satisfied for the positional relationship of "character A" to "person" in arrangements (2) and (3) of the relationship list L2, in the same manner as in the case of arrangement (1) described above.

[0074] Furthermore, the display control unit 253 searches for a condition corresponding to the current relationship label "sitting on" from among the relationship conditions for "character A" to be in a positional relationship of "sitting on" with "chair B" in arrangement (3) of the relationship list L2, and judges whether or not the condition is satisfied. It is assumed that, among the conditions for "character A" to be in a positional relationship of "sitting on" with "chair B", the condition "no obstacle on the seat surface" based on the physical norm and the condition "not a chair on which a person is already sitting" based on the social norm are satisfied. In this case, the display control unit 253 judges whether or not the remaining condition for "character A" to be in a positional relationship of "sitting on" with "chair B" when the current relationship label is "sitting on" and the past relationship label is "any (any relationship is applicable)", that is, the condition "able to sit within 5 seconds" related to the temporal norm, is satisfied. The display control unit 253 determines whether or not the remaining condition for "character A" to be in a positional relationship of "sitting on" with "chair B" is satisfied when the current relationship label is "sitting on" and the past relationship label is "any (any relationship is applicable)". 0 From 1 Character A can sit on Chair B within 5 seconds, and the time required for this action is judged based on the physical distance and the moving speed of Character A. 0 At time t 0 From time t 1 Since it is assumed that 10 seconds or more have passed since sitting down once, the condition based on the time norm, "10 seconds or more have passed since sitting down once," is satisfied. Note that the display control unit 253 also determines whether the relationship establishment condition is satisfied when "Character A" is "sitting on" with respect to the "sofa" in arrangement (1) of the relationship list L2 and when "Character A" is "sitting on" with respect to "Chair A" in arrangement (2) of the relationship list L2, in the same manner as in the case of arrangement (3) described above.

[0075] In addition, by including past and future relationship labels in addition to present relationship labels as relationship conditions, it becomes possible to arrange AR content in accordance with changes in space over time.

[0076] When the display control unit 253 determines, as a result of the matching, that the positional relationship of the relationship label "talking to" corresponding to the arrangement (1) among the positional relationship candidates shown in the relationship list L2 shown in FIG. 14 does not satisfy the condition related to the physical norm ("the conversation partner is within 2 m"), it deletes the corresponding positional relationship candidate from the relationship list L2 as the positional relationship condition is not satisfied. In addition, when the display control unit 253 determines that the positional relationship of the relationship label "sitting on" corresponding to the arrangement (3) does not satisfy the condition related to the social norm ("this is not a chair where a person is already sitting"), it deletes the corresponding positional relationship candidate from the relationship list L2 as the positional relationship condition is not satisfied. Note that when deleting the positional relationship of the relationship label "sitting on" corresponding to the arrangement (3) in the relationship list L2, the positional relationship (arrangement position) of the AR content is not determined, so the display control unit 253 also deletes the positional relationship of the relationship label "talking to" corresponding to the arrangement (3) in the same manner. FIG. 15 is a diagram showing a list after matching each positional relationship candidate with the relationship condition and an example of a corresponding scene graph.

[0077] As shown in Fig. 15, the display control unit 253 extracts valid positional relationship candidates from among the positional relationship candidates by comparing the relationship list L2 with the relationship condition DB 241, and generates a relationship list L3. If the contents of the relationship list L3 are represented in a scene graph, it will become like the scene graph CSG_L3 shown in Fig. 15. In the scene graph CSG_L3, only three positional relationship candidates that satisfy the relationship establishment condition are represented.

[0078] The display control unit 253 finally selects the positional relationship candidate at time t 1The display control unit 253 determines an optimal relationship to be used in the scene graph of the scene graph TSG. The optimal relationship is evaluated by a degree of compatibility that indicates how well each of the relationships (between each node) in each positional relationship candidate matches the positional relationship set in the template scene graph TSG. This degree of compatibility is evaluated as the sum of the priorities of positional relationship candidates that can be realized in the scene graph, among the priorities set for the positional relationships that define the relationships between each node constituting the template scene graph TSG. The display control unit 253 selects a positional relationship candidate whose degree of compatibility, evaluated as the sum of priorities, is equal to or greater than a threshold value and is evaluated to be the highest degree of compatibility, at time t 1 The optimal positional relationship between objects to be used in the scene graph is determined. Figs. 16 and 17 are diagrams showing an example of evaluating the suitability of positional relationship candidates. Fig. 18 is a diagram showing an example of final determination of positional relationship candidates. Figs. 16 and 17 show an example of the use of template scene graph TSG1-2 which prioritizes a relationship in which "character A" converses with "person".

[0079] As shown in Fig. 16, in the scene graph CSG_L3-1 expressing the positional relationship between each node such that "Character A" is "sitting on" to the "sofa" in the arrangement (1) shown in the relationship list L3, two of the relationships in the template scene graph TSG1-2 are compatible. That is, the positional relationship of the relationship label "sitting on" set between "Character A" and "sofa" and the positional relationship of the relationship label "sitting on" set between "Person" and "Chair B" are compatible. Based on these compatible relationships, the degree of compatibility, which is the sum of the priorities calculated by the display control unit 253, is "1+1=2" as shown in Fig. 16.

[0080] On the other hand, as shown in Fig. 17, in the scene graph CSG_L3-2 expressing the positional relationship between each node such that the positional relationship of "character A" is "sitting on" "chair A" and "character A" is "talking to" "person" in the arrangement (2) shown in the relationship list L3, three relationships among the relationships in the template scene graph TSG1-2 are matched. That is, the positional relationship of the relationship label "sitting on" set between "character A" and "chair A", the positional relationship of the relationship label "talking to" set between "character A" and "person", and the positional relationship of the relationship label "sitting on" set between "person" and "chair B" are matched. Therefore, the degree of match, which is the sum of the priorities calculated by the display control unit 253 based on these matched relationships, is "1+2+1=4" as shown in Fig. 17.

[0081] Therefore, the display control unit 253 judges whether the degree of conformance corresponding to FIG. 16 and the degree of conformance corresponding to FIG. 17 are each equal to or greater than a threshold. If the degrees of conformance of FIG. 16 and FIG. 17 are equal to or greater than a threshold, the display control unit 253 finally displays the positional relationship corresponding to the layout (2) shown in the relationship list L4 of FIG. 18 at time t 1 The optimal relationship (positional relationship between objects) to be used in the scene graph of 1 The content "Character A" is "sitting on" with respect to "Chair A" and "talking to" with respect to the "Person", which enables the placement and interaction of AR content that conforms to the template scene graph TSG1-2. In other words, content placement that reflects the intention of the content creator is realized while responding to changes in the spatial context.

[0082] 16 and 17 are less than a threshold, the display control unit 253 selects a positional relationship candidate shown in FIG. 16 and FIG. 17 at time t 1 It is determined that there is no optimal relationship to utilize in the scene graph.

[0083] The display control unit 253 uses the scene graph CSG_t shown in FIG. 1 At time t 1 The scene graph is used as the 3D spatial recognition map MP_t 1 The content mapping is executed to place the AR content "Character A" in the scene graph, and the display of the AR content (the video data onto which the AR content is mapped) on the display constituting the output unit 220 is controlled. The positional relationship expressed in the scene graph is used to place the AR content, and if there is a "sitting on" relationship, the AR content is transformed into a sitting state and then placed on the horizontal surface of an object on which the user can sit (such as a chair). In addition, if there is a relationship expressing a positional relationship defined by other relationship labels such as "on," "near," or "in front of," the AR content is placed, for example, randomly, within the range indicated by the definition of each relationship label. If the AR content is placed in the three-dimensional space recognition map, the AR content is superimposed on the RGB image based on the three-dimensional position and orientation of the RGB image calculated based on the RGB-D data.

[0084] Furthermore, when the determination unit 252 determines that the first real object (such as "sofa", "chair A", or "chair B") has been moved, the display control unit 253 can also rearrange the AR content (such as "character A"), which is a virtual object, to the first real object. This allows the content to be rearranged so as to realize an interaction such as making a virtual character stand up and then sit down again.

[0085] The control unit 250 repeatedly executes the above-mentioned process until the display of the AR content is completed, thereby making it possible to display the AR content in accordance with the change over time in the real space.

[0086] <2-3. Relationships between functions> The relationship between the functions of the template creation device 10 and the content display device 20 that constitute the information processing system 1 will be described with reference to Fig. 19. Fig. 19 is a diagram showing the relationship between the functions of the template creation device and the content display device.

[0087] 19, when the template creation device 10 acquires a template space model, it generates a template SG (scene graph) based on the AR content stored in the content DB 141. Furthermore, the template creation device 10 sets a relationship priority for the template SG (scene graph) and stores it in the template SG (scene graph) DB 143. Furthermore, the template creation device 10 sets a relationship condition for the template SG (scene graph) and stores it in the relationship condition DB 142.

[0088] When the content display device 20 acquires the RGB-D data, it creates a three-dimensional space recognition map. After creating the three-dimensional space recognition map, the content display device 20 recognizes the relationships between objects based on the three-dimensional space recognition map, compares the past scene graph with the current scene graph, adds relationships for unused content whose relationships have not yet been set in the scene graph, and detects changes in the relationships between objects.

[0089] The content display device 20 assigns a change mark indicating that a change has occurred to the AR content related to the object whose relationship has changed, and for the changed content with the change mark and the unused content, lists positional relationship candidates that can be realized in the scene graph of the relationship of each AR content in the template scene graph.The content display device 20 then compares each positional relationship candidate with the relationship conditions, extracts an optimal relationship from each positional relationship candidate, and evaluates the suitability of the extracted optimal relationship.

[0090] The content display device 20 performs content mapping based on a scene graph that represents the positional relationship candidate with the highest degree of suitability, and displays the content.

[0091] <<3. Processing procedure example>> <3-1. Processing procedure of template creation device> An example of a processing procedure of the template creation device 10 will be described with reference to Fig. 20. Fig. 20 is a flowchart showing an example of a processing procedure of the template creation device.

[0092] 20, the input unit 110 receives a template space model (step S101). The creation unit 151 generates a template scene graph by arranging contents (step S102).

[0093] Next, the setting unit 152 sets a relationship priority for the template scene graph (step S103), sets a relationship condition (step S104), and ends the processing shown in FIG.

[0094] <3-2. Processing procedure of content display device> An example of the processing procedure of the content display device 20 will be described with reference to Figures 21 and 22. Figure 21 is a flowchart showing an example of the processing procedure of the content display device. Figure 22 is a diagram showing the detailed procedure for listing up positional relationship candidates.

[0095] 21, the input unit 210 inputs RGB-D data (step S201). The determination unit 252 generates a three-dimensional space recognition map from the RGB-D data (step S202).

[0096] The determination unit 252 recognizes the relationship between objects based on a three-dimensional space recognition map (see FIG. 7, for example) (step S203), and generates a scene graph (current scene graph) that abstractly expresses the space.

[0097] The determination unit 252 detects a change in the relationship between the objects over time (step S204).

[0098] The display control unit 253 lists positional relationship candidates that can realize the relationship between each AR content in the template scene graph for the changed content, which is the AR content in which a change (such as a change in position) has occurred in an object related to the AR content (step S205). Note that the display control unit 253 can also list unused content, which is the AR content in which a relationship has not yet been set in the scene graph in the template scene graph, in the same manner.

[0099] The procedure of step S205 will be described in detail with reference to FIG. 22. As shown in FIG. 22, the display control unit 253 lists positional relationship candidates for clarifying the positions of the change contents (step S205-1). Next, the display control unit 253 further lists positional relationship candidates between the change contents and the objects in each positional relationship candidate (step S205-2). Then, the display control unit 253 compares each positional relationship candidate with the relationship condition, and deletes those other than the matching condition (positional relationship candidates that do not satisfy the relationship condition) from each positional relationship candidate (step S205-3). That is, the display control unit 253 extracts a valid positional relationship candidate from each positional relationship candidate.

[0100] 21, the display control unit 253 executes an optimal relationship evaluation of the changed content (step S206). That is, the display control unit 253 compares the suitability of the positional relationship candidates of the changed content and determines the optimal relationship.

[0101] The display control unit 253 executes content mapping based on the optimal relationship determined in step S206 (step S207), and performs content display (step S208).

[0102] <<4.Other>> In the above embodiment, the content display device 20 may execute only various processes related to the experience phase of the AR content, and transmit video data of the AR content to a wearable device (such as an HMD or a smartphone) of a user who experiences the AR content.

[0103] In this case, the content display device 20 transmits, for example, video data of the AR content to the see-through HMD via the network. This allows the user wearing the HMD to see the superimposed image of the foreground of the user through the HMD. The projection method of the HMD is not particularly limited, and may be any projection method such as a virtual image projection method or a virtual image projection method.

[0104] The content display device 20 transmits video data of the AR content to a video-through type HMD via a network. This allows a user wearing the HMD to see an image captured by the HMD and a virtual object superimposed on a display provided on the HMD.

[0105] In addition, the content display device 20 transmits video data of the AR content to the smartphone via the network, allowing a user of the smartphone to view, on a display provided on the smartphone, an image captured by the smartphone and a virtual object superimposed thereon.

[0106] In the information processing system 1, a projector, a tablet terminal, a television receiver, or the like may be used as a display device for displaying video data of AR content.

[0107] Furthermore, the content display device 20 according to the above embodiment may be realized by a dedicated computer system, or may be realized by a general-purpose computer system.

[0108] Furthermore, a program for executing the operation of the content display device 20 according to the embodiment can be stored in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk and distributed. Then, for example, the program is installed in a computer, and a control device is configured to execute the program to execute the operation of the content display device 20 and execute various processes related to the experience phase of the AR content. In this case, the control device may be the content display device 20 according to the embodiment.

[0109] The above program may also be stored in a disk device provided in a server device on a network such as the Internet, and may be downloaded to a computer. The functions for executing various processes related to the experience phase of the AR content may be realized by cooperation between an OS (Operating System) and application software. In this case, the parts other than the OS may be stored in a medium and distributed, or the parts other than the OS may be stored in a server device and may be downloaded to a computer.

[0110] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0111] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0112] The above-described embodiments can be combined as appropriate in the areas where the processing contents are not contradictory. The order of each step shown in the sequence diagram or flow chart of the present embodiment can be changed as appropriate.

[0113] The technology for providing AR content realized by the content display device 20 according to the embodiment can be applied to any industrial field in which XR technology such as AR or VR is used, such as industrial design, medicine, entertainment, and the like.

[0114] <<5. Conclusion>> As described above, according to an embodiment of the present disclosure, the content display device 20 (an example of an information processing device) includes an acquisition unit 251, a determination unit 252, and a display control unit 253. The acquisition unit 251 acquires template data that defines a relative relationship between a first real object, a second real object, and a virtual object. The determination unit 252 determines the relative relationship between the first real object and the second real object from three-dimensional data of the real space detected by a sensor. The display control unit 253 controls the display device 22 to place a virtual object on or near the first real object so as to face the second real object, based on the relative relationship and the template data.

[0115] This allows the content display device 20 to realize content arrangement corresponding to the spatial context of the real space and changes in the spatial context. In particular, the content display device 20 can realize appropriate content mapping in response to changes over time in the real space, and can display video data of AR content that has been appropriately content mapped.

[0116] In addition, the relative relationship includes a relative positional relationship between a first real object, a second real object, and a virtual object, as well as an interaction action of a virtual object with at least one of the first real object and the second real object, thereby realizing more specific content mapping.

[0117] Furthermore, after arranging the virtual object, the display control unit 253 executes an interaction operation of the virtual object with respect to the second real object, thereby realizing an interaction operation based on an appropriate content arrangement.

[0118] The interaction actions include sitting on the first real object, holding the first real object, and talking to the second real object, which allows for more natural interaction actions.

[0119] Furthermore, when the three-dimensional data includes a plurality of first real objects, the display control unit 253 places a virtual object on a first real object on which no other real object is placed, based on the template data. This makes it possible to realize physically natural content mapping.

[0120] Furthermore, when the three-dimensional data includes a plurality of first real objects, the display control unit 253 places the virtual object in one of the plurality of first real objects that is closer to the second real object. This makes it possible to realize physically natural content mapping.

[0121] Furthermore, the display control unit 253 controls the display device not to place a virtual object when the distance between the first real object and the second real object in the relative relationship is equal to or greater than a predetermined distance, thereby achieving physically natural content mapping.

[0122] Furthermore, when the determination unit 252 determines that the first real object has moved, the display control unit 253 rearranges the virtual object on the first real object. This makes it possible to realize content mapping that follows changes in the real space.

[0123] Furthermore, the display control unit 253 rearranges the virtual object to the first real object on the condition that a certain amount of time has passed since the previous interaction operation of the virtual object. This makes it possible to realize natural content mapping that takes into account the passage of time from the past to the present.

[0124] Furthermore, the display control unit 253 rearranges the virtual object on the first real object under the condition that the virtual object can perform a predetermined interaction operation with the first real object within a certain period of time. This makes it possible to realize natural content mapping that takes into account the passage of time from the past to the present.

[0125] Furthermore, the display control unit 253 rearranges the first real object on the virtual object under the condition that the virtual object is in a positional relationship that allows a predetermined interaction operation to be performed with the second real object when a scene of a video content including the first real object, the second real object, and the virtual object is switched. This makes it possible to realize natural content mapping that takes into account the passage of time from the present to the future.

[0126] Furthermore, the template data is scene graph data, which allows content mapping to be performed using more abstract data while still reflecting the intent of the content creator, thereby reducing the processing load.

[0127] In addition, the determination unit determines the relative relationship between the first real object and the second real object based on scene graph data that abstractly represents real space from three-dimensional data. This reduces the processing load and enables faster processing than when content mapping is performed using low-level data such as a three-dimensional mesh.

[0128] In addition, the first real object is an object that can be sat on or held, the second real object is a person, and the virtual object is a human character that changes depending on the relative relationship. This makes it possible to realize content mapping that assumes everyday scenes.

[0129] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above-described embodiments as they are, and various modifications are possible without departing from the gist of the present disclosure. In addition, components of different embodiments and modifications may be appropriately combined.

[0130] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be provided. The present technology can be configured as follows. (1) an acquisition unit that acquires template data that defines a relative relationship between a first real object, a second real object, and a virtual object; a determination unit that determines a relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by a sensor; a display control unit that controls a display device to place the virtual object on or near the first real object so as to face the second real object based on the relative relationship and the template data; An information processing device comprising: (2) The relative relationship includes a relative positional relationship between the first real object, the second real object, and the virtual object, and an interaction action of the virtual object with at least one of the first real object or the second real object. The information processing device according to (1). (3) After placing the virtual object, the display control unit executes an interaction action of the virtual object with respect to the second real object. The information processing device according to (2). (4) The interaction actions include sitting on the first real object, holding the first real object, and talking to the second real object. The information processing device according to (3). (5) When the three-dimensional data includes a plurality of the first real objects, the display control unit places the virtual object on the first real object on which no other real object is placed, based on the template data. The information processing device according to (2). (6) When the three-dimensional data includes a plurality of the first real objects, the display control unit places the virtual object on one of the plurality of the first real objects that is closer to the second real object. The information processing device according to (2). (7) The display control unit controls the display device so as not to place the virtual object when a distance between the first real object and the second real object in the relative relationship is equal to or greater than a predetermined distance. The information processing device according to (2). (8) The display control unit rearranges the virtual object on the first real object when the determination unit determines that the first real object has moved. The information processing device according to (2). (9) The display control unit rearranges the virtual object on the first real object on condition that a certain time or more has elapsed since a previous interaction operation of the virtual object. The information processing device according to (8). (10) The display control unit rearranges the virtual object on the first real object on the condition that the virtual object can perform a predetermined interaction operation with the first real object within a certain time. The information processing device according to (8). (11) The display control unit rearranges the virtual object to the first real object on the condition that a positional relationship is established in which the virtual object can perform a predetermined interaction operation with the second real object when a scene of a video content including the first real object, the second real object, and the virtual object is changed. The information processing device according to (8). (12) The template data is scene graph data. The information processing device according to (1). (13) The determination unit determines a relative relationship between the first real object and the second real object based on scene graph data that abstractly represents the real space from the three-dimensional data. The information processing device according to (1). (14) The first real object is an object that can be sat on or held, the second real object is a person, and the virtual object is a human character that changes according to the relative relationship. The information processing device according to (1). (15) obtaining template data defining a relative relationship between a first real object, a second real object, and a virtual object; determining a relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by a sensor; controlling a display device to position the virtual object on or near the first real object so as to point towards the second real object based on the relative relationship and the template data; An information processing method comprising: [Explanation of symbols]

[0131] 1. Information Processing Systems 2 RGB-D sensor 10 Template Creation Device 20 Content display device 110 Input section 120 Output section 130 Communications Department 140 Storage section 150 Control section 210 Input section 220 Output section 230 Communications Department 240 Storage section 250 Control section

Claims

1. an acquisition unit that acquires template data that defines a relative relationship between a first real object, a second real object, and a virtual object; a determination unit that determines a relative relationship in real space between the first real object and the second real object from three-dimensional data of the real space detected by a sensor; a display control unit that controls a display device to place the virtual object on or near the first real object so as to face the second real object, based on the relative relationship in real space determined by the determination unit and the template data; Equipped with The display control unit is After placing the virtual object, an interaction action of the virtual object with respect to the second real object is executed. Information processing device.

2. The relative relationship defined in the template data includes a relative positional relationship between the first real object, the second real object, and the virtual object, and an interaction action of the virtual object with respect to at least one of the first real object or the second real object. The information processing device according to claim 1 .

3. The interaction actions include sitting on the first real object, holding the first real object, and talking to the second real object.

3. The information processing device according to claim 1 or 2.

4. When the three-dimensional data includes a plurality of the first real objects, the display control unit places the virtual object on the first real object on which no other real object is placed, based on the template data.

3. The information processing device according to claim 1 or 2.

5. When the three-dimensional data includes a plurality of the first real objects, the display control unit places the virtual object on one of the plurality of the first real objects that is closer to the second real object.

3. The information processing device according to claim 1 or 2.

6. The display control unit controls the display device so as not to place the virtual object when a distance between the first real object and the second real object is equal to or greater than a predetermined distance in the relative relationship in the real space determined by the determination unit.

3. The information processing device according to claim 1 or 2.

7. The display control unit rearranges the virtual object on the first real object when the determination unit determines that the first real object has moved.

3. The information processing device according to claim 1 or 2.

8. The display control unit rearranges the virtual object on the first real object on condition that a certain time or more has elapsed since a previous interaction action of the virtual object.

3. The information processing device according to claim 1 or 2.

9. The display control unit rearranges the virtual object on the first real object on the condition that the virtual object is capable of performing a predetermined interaction operation with the first real object within a certain time.

3. The information processing device according to claim 1 or 2.

10. The display control unit rearranges the virtual object to the first real object on the condition that a positional relationship is established in which the virtual object can perform a predetermined interaction operation with the second real object when a scene of a video content including the first real object, the second real object, and the virtual object is changed.

3. The information processing device according to claim 1 or 2.

11. The template data is scene graph data.

3. The information processing device according to claim 1 or 2.

12. The determination unit determines a relative relationship between the first real object and the second real object in the real space based on scene graph data that abstractly represents the real space from the three-dimensional data.

3. The information processing device according to claim 1 or 2.

13. The first real object is a sitable object or a holdable object, the second real object is a person, and the virtual object is a person character that changes according to a relative relationship in the real space determined by the determination unit.

3. The information processing device according to claim 1 or 2.

14. obtaining template data defining a relative relationship between a first real object, a second real object, and a virtual object; determining a relative relationship in real space between the first real object and the second real object from three-dimensional data of the real space detected by a sensor; controlling a display device to position the virtual object on or near the first real object so as to point towards the second real object based on the determined relative relationship in real space and the template data; Including, Controlling the display device includes: After placing the virtual object, an interaction action of the virtual object with respect to the second real object is executed. Information processing methods.

Images