Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program that make it possible to determine the existence probability of an object represented by a node of real space graph information. An information processing device according to one aspect of the present technology acquires, on the basis of a captured image representing a real space at a first time, real space graph information that is information of a graph structure including a first real space node and a second real space node that respectively represent a first real object and a second real object in the real space and a real space edge representing an arrangement relationship between the first real object and the second real object, and calculates a first existence probability of the first real object and a second existence probability of the second real object on the basis of an elapsed time from the first time to a second time. The present invention can be applied to information processing devices which control the action of autonomously moving robots.

Description

Information processing device, information processing method, and program

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable determination of the existence probability of an object represented by a node of real-space graph information.

[0002] In order to control the behavior of autonomous mobile objects such as robots, it is important to sense the surrounding environment and understand it in an abstract manner. One of the technologies for abstractly representing the environment is a technology using 3D Scene Graph (Non-Patent Document 1).

[0003] 3D Scene Graph is information with a graph structure in which real-world objects such as furniture in the environment are represented as nodes, and relationships such as the placement of objects are represented as edges. By using 3D Scene Graph, it becomes possible to plan the behavior of autonomous mobile vehicles without relying on the absolute position of each object.

[0004] JP 2022-46408 A

[0005] Tomu Tahara, et al., “Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph”, 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020

[0006] In the environment representation using 3D Scene Graph, it is not assumed that information about objects that existed in the past will be deleted. Even if an object that is easily moved, such as stationery, is carried, it will be treated as an object that exists in a place that was previously observed.

[0007] The present technology has been developed in light of these circumstances, and makes it possible to determine the probability of existence of an object represented by a node in real-space graph information.

[0008] An information processing device according to one aspect of the present technology includes: a real-space graph acquisition unit that acquires, based on a first captured image representing a real space at a first time, real-space graph information, which is information of a graph structure including first real-space nodes and second real-space nodes that respectively represent a first real object and a second real object in the real space, and real-space edges that represent a positional relationship between the first real object and the second real object; and an existence probability management unit that calculates a first existence probability of the first real object and a second existence probability of the second real object based on the elapsed time from the first time to a second time.

[0009] In one aspect of the present technology, real-space graph information is obtained based on a first captured image representing a real space at a first time, the real-space graph information being information of a graph structure including a first real-space node and a second real-space node representing a first real object and a second real object, respectively, in the real space, and real-space edges representing the positional relationship between the first real object and the second real object, and a first existence probability of the first real object and a second existence probability of the second real object are calculated based on the elapsed time from the first time to a second time.

[0010] 1 is a diagram illustrating an example of an environment in which an autonomous moving body operates according to an embodiment of the present technology. FIG. 1 is a diagram illustrating an example of a 3D Scene Graph. FIG. 2 is a diagram illustrating an example of a flow of generating a 3D Scene Graph with presence probabilities. FIG. 2 is a block diagram illustrating an example of a functional configuration of an autonomous moving body. FIG. 3 is a block diagram illustrating an example of a configuration of a 3D Scene Graph with presence probabilities generation unit. FIG. 4 is a flowchart illustrating an update process of intermediate presence probabilities. FIG. 5 is a diagram illustrating time changes of intermediate presence probabilities. FIG. 6 is a flowchart illustrating a generation process of final presence probabilities. FIG. 7 is a diagram illustrating an example of input / output of a final presence probability determination network. FIG. 8 is a diagram illustrating an example of a relation. FIG. 9 is a flowchart illustrating an update process of a weight of a final presence probability determination network. FIG. 10 is a flowchart illustrating a series of processes of an autonomous moving body. FIG. 11 is a block diagram illustrating an example of a hardware configuration of an autonomous moving body. FIG. 12 is a diagram illustrating an example of an information processing system. FIG. 13 is a diagram illustrating a first implementation example of a 3D Scene Graph with presence probabilities generation unit. FIG. 14 is a diagram illustrating a second implementation example of a 3D Scene Graph with presence probabilities generation unit. FIG. 15 is a diagram illustrating a third implementation example of a 3D Scene Graph with presence probabilities generation unit. FIG. 16 is a diagram illustrating a fourth implementation example of a 3D Scene Graph with presence probabilities generation unit. 10 is a diagram illustrating a sixth implementation example of a 3D Scene Graph with existence probabilities generation unit.

[0011] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Configuration and operation of an autonomous moving body 3. Hardware configuration 4. Modifications

[0012] <<Outline of the Present Technology>> <Regarding Autonomous Moving Body> FIG. 1 is a diagram illustrating an example of an environment in which an autonomous moving body according to an embodiment of the present technology operates.

[0013] The autonomous mobile body 1 is, for example, a pet-type robot that moves indoors. If the environment in which the autonomous mobile body 1 exists is a living room of a house, various objects exist around the autonomous mobile body 1, including furniture such as a sofa and a table, and home appliances such as a television and lighting equipment.

[0014] The autonomous mobile body 1 is equipped with various sensors such as an RGB camera and a depth sensor. The autonomous mobile body 1 has the function of autonomously taking various actions such as making an action plan based on information acquired by the sensors and communicating with a user.

[0015] The autonomous moving body 1 generates a 3D Scene Graph as shown in the balloon in Figure 1 based on RGB images captured by an RGB camera, depth images detected by a depth sensor, etc., and performs behavior planning based on the generated 3D Scene Graph.

[0016] This technology can also be applied to controlling the behavior of an AR (Augmented Reality) character as an autonomous moving object based on a 3D Scene Graph. The AR character is a virtual character that is displayed superimposed on real space using an AR display device. A technology that enables a user wearing an AR display device such as an HMD (Head Mounted Display) to communicate with an AR character is disclosed, for example, in International Publication No. 2022 / 224522.

[0017] <Example of 3D Scene Graph> FIG. 2 is a diagram showing an example of a 3D Scene Graph.

[0018] If a sofa, a table, a television, chair A, and chair B exist in the environment of the autonomous mobile body 1 and are arranged so as to have a predetermined positional relationship, as shown in Figure 2, the 3D Scene Graph representing the environment of the autonomous mobile body 1 includes five nodes representing these objects.

[0019] In the example of FIG. 2, the sofa node and the TV node share an edge E with the label “in_front_of”. １ Edge E １ The label "on_right" indicates that the sofa is in front of the TV. The sofa node and the table node share an edge E with the label "on_right". ２ Edge E ２The label indicates that the table is to the right of the sofa.

[0020] The TV node and the table node have an edge E with the label “on_left”. ３ Edge E ３ The label indicates that the table is on the left side of the TV. The edge E between the table node and the chair A node, and the edge E between the table node and the chair B node are also labeled with labels indicating their respective positional relationships. ４ , E ５ The labels assigned to the edges are labels that represent spatial relationships (front / behind / left / right / on / above / under / near, etc.).

[0021] In this way, the 3D Scene Graph is real-space graph information having a graph structure in which real objects such as furniture that exist in the environment are represented as nodes, and relationships such as the positional relationship of the objects are represented as edges. The nodes are real-space nodes that represent real objects in the real space, and the edges are real-space edges that represent the positional relationship of the real objects in the real space. If the sofa shown in FIG. 2 is the first real object, another piece of furniture, for example, a table, is the second real object. Also, the edge E connecting the sofa node and the table node is ２ becomes a real space edge that represents the positional relationship between the first real object and the second real object. The generation of a 3D Scene Graph is described, for example, in Literature 1 (the same as Non-Patent Document 1 above). Literature 1: Tomu Tahara, et al., "Retargetable AR: Context-aware Augmented Reality in Indoor Scenes based on 3D Scene Graph," 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 2020

[0022] In this technology, information on the probability of an object's existence is added to the 3D Scene Graph described above. The probability of an object's existence is used in the behavior planning of an autonomous mobile vehicle as an index to determine whether a previously observed object has moved. It is conceivable to use a value that increases when an object is actually observed and decreases over time as the probability of existence, but a probability calculated by such a simple calculation cannot distinguish between objects that are easy to move and objects that are difficult to move.

[0023] This technology uses a neural network that inputs a 3D scene graph to calculate the presence probability, taking into account the ease of movement of an object and the relative positions of surrounding objects. In addition, by using a framework that updates the weights of the neural network each time the presence probability is calculated, a neural network that can calculate presence probabilities specialized for each environment is generated without any training data.

[0024] This technology makes it possible to calculate the probability of an object's existence, taking into account the environment and ease of movement, enabling autonomous mobile objects to execute sophisticated behavior plans that adapt to the environment.

[0025] <Flow of generating 3D Scene Graph with existence probability> Fig. 3 is a diagram showing an example of the flow of generating a 3D Scene Graph with existence probability. The generation of a 3D Scene Graph with existence probability, which is a 3D Scene Graph used in the present technology, is mainly performed in the following procedure.

[0026] (1) As shown on the left side of Figure 3, for each object, the existence probability according to the elapsed time is calculated as an intermediate existence probability and added to the 3D Scene Graph. The elapsed time is the time from the last time the object was observed to the current time. The 3D Scene Graph with the intermediate existence probability added is used as a temporary Scene Graph.

[0027] (2) As shown in the center of Figure 3, a final presence probability that reflects the object's ease of movement is calculated as the final presence probability using a final presence probability determination network, which is a graph convolutional network (GCN). The final presence probability determination network is a neural network that inputs a temporary scene graph. The 3D scene graph with the final presence probability information added is used to plan the behavior of the autonomous mobile body 1.

[0028] (3) As shown on the right side of Figure 3, it is determined whether an object within the observation range has moved or not. The actual determination result as to whether the object has moved or not is taken as correct data, and the weights of the final presence probability determination network are updated based on the error between the correct data and the prediction result of the final presence probability determination network.

[0029] <<Configuration and Operation of Autonomous Mobile Body>> <Overall Configuration of Autonomous Mobile Body> FIG. 4 is a block diagram showing an example of the functional configuration of the autonomous mobile body 1. As shown in FIG.

[0030] 4, the autonomous moving body 1 implements a 3D Scene Graph with existence probability generation unit 11 and a behavior planning unit 12. The 3D Scene Graph with existence probability generation unit 11 and the behavior planning unit 12 are implemented, for example, by the CPU of a computer constituting the autonomous moving body 1 executing a predetermined program.

[0031] The 3D Scene Graph with existence probabilities generation unit 11 acquires various sensor data detected by sensors mounted on the autonomous mobile body 1 and generates a 3D Scene Graph with existence probabilities. Captured image data such as RGB images captured by an RGB camera and depth images detected by a depth sensor are input to the 3D Scene Graph with existence probabilities generation unit 11. The final 3D Scene Graph with existence probabilities generated by the 3D Scene Graph generation unit 11 is supplied to the behavior planning unit 12. Details of the 3D Scene Graph with existence probabilities generation unit 11 will be described later with reference to FIG. 5 .

[0032] The behavior planning unit 12 performs behavior planning based on the 3D Scene Graph with final existence probabilities generated by the 3D Scene Graph with existence probabilities generation unit 11. The behavior planning unit 12 controls each driving unit of the autonomous mobile body 1 to execute an operation for performing the planned behavior. For example, under the control of the behavior planning unit 12, a behavior according to the situation of the object, such as going to get a toy that is placed on a table in a room, is expressed.

[0033] FIG. 5 is a block diagram showing an example of the configuration of the 3D Scene Graph with existence probabilities generating unit 11.

[0034] 5, the 3D Scene Graph with existence probability generation unit 11 includes an image acquisition unit 51, a space reconstruction unit 52, a 3D Scene Graph generation unit 53, an intermediate existence probability generation unit 54, a final existence probability generation unit 55, a moving object detection unit 56, and a weight update unit 57. As shown enclosed by a dashed line, the space reconstruction unit 52 and the 3D Scene Graph generation unit 53 constitute a real space graph acquisition unit 61, and the intermediate existence probability generation unit 54 and the final existence probability generation unit 55 constitute a existence probability management unit 62. Furthermore, the moving object detection unit 56 and the weight update unit 57 constitute an information update unit 63. Each unit will be described in detail below.

[0035] <Image Acquisition Unit 51> The image acquisition unit 51 controls the sensing device to acquire an RGB image and a depth image. A sequence of image data including the RGB image and the depth image acquired by the image acquisition unit 51 is sequentially supplied to the spatial reconstruction unit 52.

[0036] In the above example, an RGB camera and a depth sensor are provided, but other configurations can be used as sensor devices used to acquire RGB images and depth images. For example, a device such as an RGB-D camera that combines an RGB camera and a ToF sensor may be used, or a device that acquires depth images using monocular depth estimation technology based on RGB images captured by an RGB camera may be used.

[0037] <Real Space Graph Acquisition Unit 61> Space Reconstruction Unit 52 The space reconstruction unit 52 performs segmentation using Panoptic Fusion and label integration into a 3D space map based on the captured image data supplied from the image acquisition unit 51, thereby reconstructing a 3D space. Panoptic Fusion is described, for example, in Reference 2. Reference 2: Gaku Narita, et al., "Panoptic Fusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things", Proc. IROS, 2019

[0038] Specifically, the spatial reconstruction unit 52 performs semantic segmentation and instance segmentation on the 2D RGB image to predict a panoptic label for each pixel. The panoptic label includes the ID of the object region that each pixel constitutes (an object ID and an ID indicating the type (category) of the object). The spatial reconstruction unit 52 also integrates the panoptic label into a 3D spatial map generated by SLAM using the depth image to generate a volumetric map.

[0039] The spatial reconstruction unit 52 also calculates the reliability c of each object. i Calculate the reliability c i is calculated as a moving average in the time direction of the reliability of the prediction result for each object, which is a general output in panoptic segmentation. Instead of a moving average, the average of the reliability for a certain period of time is the reliability c i Alternatively, the reliability of the prediction result at a predetermined time may be used as the reliability c i It may be used as:

[0040] The spatial reconstruction unit 52 generates mesh data by performing segmentation, and a reliability c iThe spatial reconstruction unit 52 outputs information on the classification, classification likelihood, and category likelihood. The mesh data is labeled mesh data extracted from the volumetric map. The mesh data represents the shape of an object, etc. The category likelihood is the likelihood of an object category, such as the probability that an object is a chair, the probability that an object is a table, etc. The mesh data output from the spatial reconstruction unit 52 is supplied to the 3D Scene Graph generation unit 53 and the intermediate existence probability generation unit 54, and the reliability information is supplied to the intermediate existence probability generation unit 54. The category likelihood information output from the spatial reconstruction unit 52 is supplied to the final existence probability generation unit 55.

[0041] A spatial reconstruction technique other than PanopticFusion may be used to reconstruct a 3D space including semantic information. For example, when the 3DSG described in Literature 3 is used to generate a 3D Scene Graph in the 3D Scene Graph generation unit 53, a spatial reconstruction technique other than PanopticFusion is used in the spatial reconstruction unit 52. Literature 3: Antoni Rosinol, et al., "3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans," Robotics: Science and Systems, 2020

[0042] 3D Scene Graph Generation Unit 53 The 3D Scene Graph generation unit 53 generates a 3D Scene Graph having a graph structure as described with reference to FIG. 2 based on the mesh data supplied from the spatial reconstruction unit 52. The generation of the 3D Scene Graph by the 3D Scene Graph generation unit 53 is executed each time the real space is reconstructed by the spatial reconstruction unit 52. For example, when the real space is reconstructed based on a first captured image acquired at a first time, a 3D Scene Graph representing the real space at the first time is generated, and when the real space is reconstructed based on a second captured image acquired at a second time after the first time, a 3D Scene Graph representing the real space at the second time is generated. Information about the 3D Scene Graph generated by the 3D Scene Graph generation unit 53 is output to a subsequent processing unit.

[0043] <Existence Probability Management Unit 62> Intermediate Existence Probability Generation Unit 54 The intermediate existence probability generation unit 54 calculates the existence probability of each observed object based on the mesh data and reliability information supplied from the spatial reconstruction unit 52. The existence probability calculated by the intermediate existence probability generation unit 54 is an intermediate existence probability, and is calculated taking into account a decrease over time. The intermediate existence probability generation unit 54 calculates the existence probability of each observed object based on the observation time t i and the intermediate existence probability p i are linked and managed.

[0044] Here, the update process of the intermediate existence probability will be described with reference to the flowchart of Fig. 6. The flowchart of Fig. 6 also shows the process of the space reconstruction unit 52.

[0045] In step S1, the space reconstruction unit 52 reconstructs the 3D space using Panoptic Fusion, and generates mesh data and reliability for each object.

[0046] In step S2, the intermediate existence probability generating unit 54 calculates the set of mesh data and reliability {c i} and select one object.

[0047] In step S3, the intermediate existence probability generating unit 54 determines whether one-third or more of the meshes constituting the selected object have been updated. An object in which one-third or more of the meshes have been updated is treated as an observed object. A value other than one-third may be used as the threshold value used to determine whether an object is observed.

[0048] If it is determined in step S3 that one-third or more of the mesh data constituting the object has been updated, in step S4, the intermediate existence probability generation unit 54 calculates the observation time t i For an object for which it is determined in step S3 that one-third or more of the mesh data has not been updated, the process of step S4 is skipped.

[0049] In step S5, the intermediate existence probability generating unit 54 calculates the intermediate existence probability p i is calculated and updated according to the following formula (1). i is the observation time t i It is managed in association with

[0050] In formula (1), α and β are coefficients that can take any value. By calculation using formula (1), the intermediate existence probability of each object is calculated by the reliability c i is the initial value, and the value decreases over time.

[0051] FIG. i 7 is a diagram showing the intermediate existence probability when α=1, α=0.5, β=15. In FIG. 7, the horizontal axis represents time, and the vertical axis represents the intermediate existence probability. Equation (1) is based on the hypothesis that the existence probability hardly changes while the elapsed time is short, and the existence probability suddenly decreases from a certain timing. In this way, the intermediate existence probability is calculated from the first observation time t i is calculated based on the elapsed time from the first time to the current time t, which is the second time.

[0052] 6, in step S6, the intermediate presence probability generator 54 determines whether or not all objects have been selected. If it is determined in step S6 that all objects have not yet been selected, the process returns to step S2, where the next object is selected, and the above processing is repeated.

[0053] If it is determined in step S6 that all objects have been selected, the process in FIG. 6 ends. i and observation time t i The information is integrated into the 3D Scene Graph generated by the 3D Scene Graph generating unit 53. For example, the intermediate existence probability generating unit 54 assigns an intermediate existence probability p i and observation time t i By adding this information, the information on the intermediate existence probability is integrated into the 3D Scene Graph, and a 3D Scene Graph with intermediate existence probability is generated.

[0054] The information of each node in the 3D Scene Graph with intermediate existence probability includes the intermediate existence probability p i and observation time t i As described above, when the sofa is the first real object and the table is the second real object, the information of the sofa node includes the intermediate existence probability p i The node information in the table includes the intermediate existence probability p i The 3D Scene Graph with intermediate existence probabilities, which is the first real space graph information with existence probabilities into which the intermediate existence probabilities are integrated, is supplied to the final existence probability generation unit 55 and the moving object detection unit 56 as a temporary Scene Graph.

[0055] Final Presence Probability Generation Unit 55 The final presence probability generation unit 55 generates a final presence probability, which is the final presence probability of each object, based on the 3D Scene Graph with intermediate presence probabilities and the category likelihoods supplied from the spatial reconstruction unit 52. Since the presence probability is generated based on the information of the 3D Scene Graph, the final presence probability of each object is a presence probability that takes into account the relationship with surrounding objects. Furthermore, since the presence probability is generated using category information, the final presence probability is a presence probability that takes into account the ease of movement according to the object category.

[0056] Here, the process of generating the final existence probabilities will be described with reference to the flowchart in Fig. 8. The process in Fig. 8 starts when information on the 3D Scene Graph with intermediate existence probabilities is supplied.

[0057] In step S11, the final existence probability generation unit 55 acquires the 3D Scene Graph with intermediate existence probabilities supplied from the preceding processing unit. The final existence probability generation unit 55 also acquires category likelihood information supplied from the spatial reconstruction unit 52.

[0058] In step S12, the final existence probability generation unit 55 generates a final existence probability using a final existence probability determination network. The final existence probability generation unit 55 is provided with a final existence probability determination network that is a graph convolutional network.

[0059] 9 is a diagram showing an example of input and output of the final existence probability determination network. As shown in Fig. 9, the final existence probability determination network is a network that receives, as input, information on node features, edge features, and adjacency matrices, for example.

[0060] The node features are expressed as an N × F matrix, where N is the number of nodes and F is the number of categories + 2. The node features are the likelihood of each object category and the intermediate existence probability p i and elapsed time t-t i The intermediate existence probability p i , time t, time t i is obtained from a 3D Scene Graph with intermediate presence probabilities.

[0061] Edge features are expressed as an E x D matrix, where E is the number of edges and D is the number of relations associated with the edge. Edge features are information indicating the IDs of the relations associated with each edge. Relations are obtained from a 3D Scene Graph with intermediate existence probabilities.

[0062] Fig. 10 is a diagram showing an example of a relation. The relation indicating the layout relationship between objects as described with reference to Fig. 2 is identified by the id shown in Fig. 10. The relation shown in Fig. 10 is the same as the relation used in the technology described in Literature 1.

[0063] The adjacency matrix is ​​expressed as an N x N matrix, and is a matrix in which connections between nodes are expressed as 0 or 1.

[0064] Using the node features, edge features, and adjacency matrix information as input, the final existence probability determination network outputs the final existence probability of each of the N objects. The final existence probability is expressed, for example, by a value of [0, 1] (a value ranging from 0 to 1).

[0065] In this way, the final presence probability generation unit 55 uses information included in the 3D Scene Graph with intermediate presence probabilities and information such as elapsed time as input, and calculates (infers) the final presence probability of each object using a prediction network that outputs information on the final presence probability.

[0066] In step S13 of Fig. 8 , the final presence probability generation unit 55 generates a 3D Scene Graph with final presence probabilities by adding information about the final presence probabilities to each node of the 3D Scene Graph. Then, the processing of Fig. 8 ends. The information about each node of the 3D Scene Graph with final presence probabilities includes information about the final presence probabilities of the respective objects. As described above, if the sofa is the first real object and the table is the second real object, the information about the sofa node includes information about the final presence probability of the sofa as the first final presence probability, and the information about the table node includes information about the final presence probability of the table as the second final presence probability.

[0067] The 3D Scene Graph with final existence probabilities generated as described above as second real space graph information with existence probabilities is supplied to the behavior planning unit 12 and used for behavior planning of the autonomous mobile body 1. The 3D Scene Graph with final existence probabilities generated by the final existence probability generation unit 55 is also supplied to the weight update unit 57 in FIG.

[0068] <Information Update Unit 63> Moving Object Detection Unit 56 The moving object detection unit 56 detects moving objects from among the observed objects. The moving object detection is performed by comparing the immediately preceding 3D Scene Graph with the current 3D Scene Graph to distinguish between objects that have moved and objects that have not moved.

[0069] For example, the moving object detection unit 56 calculates Intersection over Union (IoU) based on the bounding box information of the same object included in the node information of the previous 3D Scene Graph and the node information of the current 3D Scene Graph. The moving object detection unit 56 determines that an object whose IoU is equal to or less than a threshold is a moved object, and determines that an object whose IoU is greater than the threshold is a non-moving object.

[0070] IoU is an index used to evaluate the degree of overlap between bounding box regions of objects. The larger the IoU value, the greater the overlap, i.e., the smaller the amount of movement of the object is determined to be. An object with an IoU value of 1.0 is considered to be a stationary object. Conversely, the smaller the IoU value, the smaller the overlap, i.e., the larger the amount of movement of the object is determined to be. Information indicating the detection result by the moving object detection unit 56 is supplied to the weight update unit 57.

[0071] Weight Update Unit 57 The weight update unit 57 updates the weights, which are parameters of the final presence probability determination network, based on the detection results of the moving object by the moving object detection unit 56 and the final presence probability inferred by the final presence probability generation unit 55.

[0072] The process of updating the weights of the final existence probability determination network will be described with reference to the flowchart of FIG.

[0073] In step S21, the weight update unit 57 selects one object in the observation range.

[0074] In step S22, the weight update unit 57 determines whether the selected object is an object whose bounding box IoU is 0.3 or more. In this example, an object whose bounding box IoU is less than 0.3 is determined to be a moving object. A value other than 0.3 may be used as the threshold value that serves as a criterion for determining whether an object has moved.

[0075] If it is determined in step S22 that the IoU of the bounding box is 0.3 or more, that is, that the object is not moving, in step S23, the weight update unit 57 sets the value 1 as the existence probability of the selected object and adds it to the correct answer data.

[0076] On the other hand, if it is determined in step S22 that the object has an IoU of less than 0.3, i.e., is a moving object, then in step S24 the weight update unit 57 sets the value 0 as the existence probability of the selected object and adds it to the correct answer data.

[0077] After adding the correct answer data in step S23 or step S24, the weight update unit 57 determines whether or not all objects have been selected in step S25. If it is determined that all objects have not been selected, the process returns to step S21, and the above-described process is repeated.

[0078] If it is determined in step S25 that all objects have been selected, in step S26, the weight update unit 57 performs error back propagation for each object whose final existence probability has been predicted, using the difference between the final existence probability and the correct data (0 or 1) as the error, and updates the weights of the final existence probability determination network. After the weights of the final existence probability determination network have been updated, the processing of FIG. 11 ends.

[0079] In this way, in the process of updating the weights of the final presence probability determination network, a value according to the determination result of whether or not the observed object has moved is used as the correct answer data. If it is determined that the object has not moved, a value of 1 is used as the correct answer data, and if it is determined that the observed object has moved, a value of 0 is used as the correct answer data.

[0080] Furthermore, the weight update process for the final presence probability determination network is performed in parallel with the action plan using the final presence probability determination network, for example, every time the final presence probability determination network is generated. Because no pre-learning is performed, it is possible to perform learning specific to the environment without requiring learning data.

[0081] Before updating the weights to specialize for the environment, initial learning is performed to learn weights that output the intermediate existence probability as the final existence probability as is. Furthermore, the final existence probability determination network may be trained using prior knowledge according to the object category and attributes, as follows: ・For chairs, the final existence probability is set to 0.5 times the intermediate existence probability, and for tables, the final existence probability is set to 1.0 times the intermediate existence probability. ・For small TVs, the final existence probability is set to 0.8 times the intermediate existence probability, and for large TVs, the final existence probability is set to 1.0 times the intermediate existence probability.

[0082] <Overall Processing Flow> A series of processing steps performed by the autonomous moving body 1 to control its behavior will be described with reference to the flowchart of FIG.

[0083] In step S31, the image acquisition unit 51 acquires data of a captured image including an RGB image and a depth image.

[0084] In step S32, an intermediate existence probability update process is performed. The process described with reference to Fig. 6 is performed, and the intermediate existence probability of each observed object is updated.

[0085] In step S33, the final existence probability generation process is performed. The process described with reference to Fig. 8 is performed to generate a 3D Scene Graph with final existence probabilities.

[0086] In step S34, the moving object detection unit 56 detects a moving object from among the objects being observed in the manner described above.

[0087] In step S35, a network weight updating process is performed. The process described with reference to Fig. 11 is performed based on the detection result of the moving object, and the weight of the final existence probability determination network is updated.

[0088] In step S36, the behavior planning unit 12 performs a behavior plan based on the 3D Scene Graph with final existence probability, and executes an arbitrary behavior by controlling each driving unit of the autonomous mobile body 1, for example.

[0089] The series of processes shown in FIG. 12 realizes the following application.

[0090] - Object Search in a Household Pet Robot If the autonomous mobile body 1 is a household pet robot, a scenario is envisioned in which the robot will bring a pet toy to a user and ask the user to play with it. The robot will determine whether to retrieve the toy from a toy storage area prepared in advance or from a previously observed location using the final presence probability represented by a 3D Scene Graph.

[0091] For example, if a toy has been previously observed on the living room table and the final probability of the toy's presence is higher than a threshold, an action is planned to go to the table to get the toy. On the other hand, if the final probability of the toy's presence is lower than the threshold, an action is planned to go to the toy storage area to get the toy. This makes it possible to plan actions to go to places where there is a high probability that the toy will be found.

[0092] Object Search in a Helper Robot When the autonomous mobile body 1 is a helper robot, a scenario is assumed in which the robot is instructed by a user to bring a cup. The robot determines whether to go to the cupboard or to a previously observed location to get the cup, using the final presence probability represented by a 3D Scene Graph.

[0093] Even in this case, for example, if a glass has been previously observed on a table and the final probability of the glass being there is higher than a threshold, an action is planned to go to the table to get the glass. On the other hand, if the final probability of the glass being there is lower than the threshold, an action is planned to go to the cupboard to get the glass. This makes it possible to plan an action to go to a place where there is a high probability that the glass will be there.

[0094] Through the above processing, the existence probability of an object is determined in association with the object. For example, if there are two chairs, Chair A and Chair B, they are distinguished and the existence probability of each is determined. For example, when taking an action such as sitting on a chair, it is possible to plan an action by taking into account the existence probability of each chair to sit on.

[0095] Furthermore, the existence probability of an object is updated not only over time but also in accordance with its relative position relative to surrounding objects. If the rate of decrease in existence probability were fixed for each category, it would only be possible to decrease the existence probability at a rate that corresponds to the object's category over time. By decreasing the existence probability in accordance with its relative position relative to surrounding objects, it becomes possible to calculate a more sophisticated existence probability that takes into account the characteristics of the object, such as whether the chair is easy to move or difficult to move.

[0096] Furthermore, the existence probability is calculated specifically for the environment. If the rate at which the existence probability decreases is assumed to be constant regardless of the environment, it would be impossible to take into account, for example, the different characteristics of each house. Even if the furniture layout in house A and house B is the same, there will be differences, such as chairs being easy to move in house A and difficult to move in house B. By reflecting the ease of movement of objects in the environment, it becomes possible to calculate the existence probability more accurately, specific to the environment.

[0097] <<Hardware Configuration>> <Hardware Configuration of Autonomous Moving Body> FIG. 13 is a block diagram showing an example of the hardware configuration of the autonomous moving body 1. As shown in FIG.

[0098] As shown in FIG. 13, the autonomous moving body 1 is configured by connecting a camera 102, a sensor 103, a communication unit 104, a drive control unit 105, and a memory 106 to a control unit 101.

[0099] The control unit 101 is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The control unit 101 executes programs stored in the ROM and memory 106 to control the overall operation of the autonomous mobile body 1. The control unit 101 executes predetermined programs to realize a 3D Scene Graph with existence probability generation unit 11 and a behavior planning unit 12 shown in FIG. 4 .

[0100] The camera 102 captures an image in front of the autonomous moving body 1 and outputs an RGB image to the control unit 101 .

[0101] The sensor 103 is configured with various sensors such as a ToF sensor. The ToF sensor that configures the sensor 103 measures the distance to each position ahead of the autonomous moving body 1 and outputs a depth image to the control unit 101.

[0102] The sensor 103 includes various sensors such as an acceleration sensor, a gyro sensor, a positioning sensor, a microphone, and a temperature sensor as appropriate. In this case, the measurement results of the acceleration sensor, the gyro sensor, and the positioning sensor are output to the control unit 101. The measurement results of the acceleration sensor, the gyro sensor, and the positioning sensor are used to estimate the position and attitude of the autonomous moving body 1, etc.

[0103] The communication unit 104 is configured by a communication module such as a wireless LAN. The communication unit 104 communicates with external devices via a network and transmits data supplied from the control unit 101 to the external devices. The communication unit 104 also receives data transmitted from external devices and outputs the data to the control unit 101. External devices include not only devices present in the environment of the autonomous mobile body 1 but also servers connected via the Internet.

[0104] The drive control unit 105 drives the motors provided in each part of the autonomous mobile body 1 according to the control by the control unit 101, thereby realizing the behavior of the autonomous mobile body 1.

[0105] The memory 106 is a storage medium such as a flash memory, etc. The memory 106 stores various data such as programs executed by the CPU of the control unit 101.

[0106] <Modification of Configuration> At least some of the functional units of the existence-probability-attached 3D Scene Graph generation unit 11 described with reference to FIG. 5 can be realized by a device external to the autonomous moving body 1.

[0107] Fig. 14 is a diagram showing an example of an information processing system including an external device. The information processing system shown in Fig. 14 is composed of an autonomous mobile body 1, a charging stand 2, and a server 201. The autonomous mobile body 1 and the charging stand 2 are connected via a wireless LAN, for example. The autonomous mobile body 1 and the server 201, and the charging stand 2 and the server 201 are connected via a network, respectively.

[0108] In the information processing system shown in Fig. 14, the autonomous mobile body 1 functions as an edge device. The charging stand 2 is used to charge the battery of the autonomous mobile body 1 and also functions as an intermediary terminal that mediates communication between the autonomous mobile body 1 and the server 201. The charging stand 2 is equipped with a computer having a CPU, ROM, RAM, etc. Instead of the charging stand 2, an information processing device such as a smartphone, tablet terminal, or PC may be used as the intermediary terminal. The server 201 is an information processing device on the cloud. An implementation example of each functional unit of the 3D Scene Graph with existence probability generation unit 11 will be described below.

[0109] 15 is a diagram showing a first implementation example of the 3D Scene Graph with presence probability generator 11. In the example of Fig. 15 , all of the functional units constituting the 3D Scene Graph with presence probability generator 11, namely, the image acquisition unit 51, the spatial reconstruction unit 52, the 3D Scene Graph generator 53, the intermediate presence probability generator 54, the final presence probability generator 55, the moving object detection unit 56, and the weight update unit 57, are implemented in the autonomous moving body 1.

[0110] The implementation example shown in Fig. 15 is similar to the example described with reference to Fig. 5 etc. The autonomous moving body 1 including a real space graph acquisition unit 61 configured with a space reconstruction unit 52 and a 3D Scene Graph generation unit 53, and a presence probability management unit 62 configured with an intermediate presence probability generation unit 54 and a final presence probability generation unit 55 serves as an information processing device.

[0111] 16 is a diagram showing a second implementation example of the 3D Scene Graph with presence probability generation unit 11. In the example of Fig. 16 , of the functional units constituting the 3D Scene Graph with presence probability generation unit 11, the image acquisition unit 51, the spatial reconstruction unit 52, the 3D Scene Graph generation unit 53, the intermediate presence probability generation unit 54, the final presence probability generation unit 55, and the moving object detection unit 56 are implemented in the autonomous moving body 1, and the weight update unit 57 is implemented in the server 201.

[0112] In this case as well, the autonomous moving body 1 including the real space graph acquisition unit 61 configured by the space reconstruction unit 52 and the 3D Scene Graph generation unit 53, and the presence probability management unit 62 configured by the intermediate presence probability generation unit 54 and the final presence probability generation unit 55, serves as the information processing device.

[0113] 17 is a diagram showing a third implementation example of the 3D Scene Graph with presence probability generator 11. In the example of Fig. 17 , of the functional units constituting the 3D Scene Graph with presence probability generator 11, the image acquisition unit 51 is implemented in the autonomous moving body 1, and the space reconstruction unit 52, the 3D Scene Graph generator 53, the intermediate presence probability generator 54, the final presence probability generator 55, the moving object detection unit 56, and the weight update unit 57 are implemented in the server 201.

[0114] In this case, the server 201 including a real space graph acquisition unit 61 configured by a space reconstruction unit 52 and a 3D Scene Graph generation unit 53, and an existence probability management unit 62 configured by an intermediate existence probability generation unit 54 and a final existence probability generation unit 55 serves as the information processing device.

[0115] 18 is a diagram showing a fourth implementation example of the 3D Scene Graph with presence probability generator 11. In the example of Fig. 18 , of the functional units constituting the 3D Scene Graph with presence probability generator 11, the image acquisition unit 51, the spatial reconstruction unit 52, the 3D Scene Graph generator 53, the intermediate presence probability generator 54, the final presence probability generator 55, and the moving object detection unit 56 are implemented in the autonomous moving body 1, and the weight update unit 57 is implemented in the charging stand 2.

[0116] In this case, the autonomous moving body 1 including a real space graph acquisition unit 61 configured by a space reconstruction unit 52 and a 3D Scene Graph generation unit 53, and a presence probability management unit 62 configured by an intermediate presence probability generation unit 54 and a final presence probability generation unit 55 becomes the information processing device.

[0117] 19 is a diagram showing a fifth implementation example of the 3D Scene Graph with presence probability generator 11. In the example of Fig. 19, of the functional units constituting the 3D Scene Graph with presence probability generator 11, the image acquisition unit 51 is implemented in the autonomous moving body 1, and the space reconstruction unit 52, the 3D Scene Graph generator 53, the intermediate presence probability generator 54, the final presence probability generator 55, the moving object detection unit 56, and the weight update unit 57 are implemented in the charging stand 2.

[0118] In this case, the charging stand 2, which includes a real space graph acquisition unit 61 configured by a space reconstruction unit 52 and a 3D Scene Graph generation unit 53, and an existence probability management unit 62 configured by an intermediate existence probability generation unit 54 and a final existence probability generation unit 55, serves as the information processing device.

[0119] 20 is a diagram showing a sixth implementation example of the 3D Scene Graph with presence probability generator 11. In the example of Fig. 20 , of the functional units constituting the 3D Scene Graph with presence probability generator 11, the image acquisition unit 51 is implemented in the autonomous moving body 1, and the space reconstruction unit 52, the 3D Scene Graph generator 53, the intermediate presence probability generator 54, the final presence probability generator 55, and the moving object detection unit 56 are implemented in the charging stand 2. In addition, the weight update unit 57 is implemented in the server 201.

[0120] In this case, too, the charging stand 2 including the real space graph acquisition unit 61 configured by the space reconstruction unit 52 and the 3D Scene Graph generation unit 53, and the existence probability management unit 62 configured by the intermediate existence probability generation unit 54 and the final existence probability generation unit 55, serves as the information processing device.

[0121] The present invention is not limited to the examples shown in FIGS. 15 to 20, and it is possible to arbitrarily change which device realizes each functional unit of the existence probability-added 3D Scene Graph generation unit 11.

[0122] <<Modifications>> The present technology is applicable to controlling the behavior of various autonomous moving objects that perform behavior planning based on a 3D Scene Graph. Autonomous moving objects that perform behavior planning based on a 3D Scene Graph include robots of a non-robot type, drones, and AR characters. When controlling the behavior of an AR character, at least some of the functional units of the 3D Scene Graph with presence probability generation unit 11 are implemented in an HMD worn by a user or in an external device that controls the display of the HMD.

[0123] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0124] 21 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. For example, the control unit 101 in FIG. 13 is configured by a computer having the configuration shown in FIG.

[0125] A CPU (Central Processing Unit) 1001 , a ROM (Read Only Memory) 1002 , and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004 .

[0126] An input / output interface 1005 is also connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, etc., and an output unit 1007 including a display, a speaker, etc. are connected to the input / output interface 1005. In addition, a storage unit 1008 including a hard disk, a nonvolatile memory, etc., a communication unit 1009 including a network interface, etc., and a drive 1010 that drives removable media 1011 are also connected to the input / output interface 1005.

[0127] In a computer configured as described above, the CPU 1001 performs the above-described series of processes by, for example, loading a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0128] The program executed by the CPU 1001 is installed in the storage unit 1008 by being recorded on, for example, a removable medium 1011 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0129] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0130] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0131] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0132] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0133] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0134] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0135] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0136] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0137] (1) An information processing device comprising: a real space graph acquisition unit that acquires, based on a first captured image representing a real space at a first time, real space graph information that is information of a graph structure including first real space nodes and second real space nodes representing a first real object and a second real space node, respectively, in the real space, and real space edges that represent a positional relationship between the first real object and the second real object, and an existence probability management unit that calculates a first existence probability of the first real object and a second existence probability of the second real object based on the elapsed time from the first time to a second time. (2) The information processing device described in (1), wherein the real space graph acquisition unit acquires the real space graph information based on a second captured image representing the real space at the second time after the first time. (3) The information processing device according to (2), wherein the existence probability management unit generates first real space graph information with existence probabilities, in which information on the first real space nodes of the real space graph information acquired based on the second captured image includes information on the first existence probability, and information on the second real space nodes includes information on the second existence probability. (4) The information processing device according to (3), wherein the existence probability management unit calculates the first existence probability and the second existence probability, which decrease according to the elapsed time. (5) The information processing device according to (4), wherein the existence probability management unit calculates the first existence probability and the second existence probability using reliability of segmentation results of the first real object and the second real object as initial values. (6) The information processing device according to any of (3) to (5), wherein the existence probability management unit calculates a first final existence probability of the first real object and a second final existence probability of the second real object using information included in the first real space graph information with existence probabilities and the elapsed time as inputs to a prediction network. (7) The information processing device according to (6), wherein the prediction network is a network that receives information including information on the first real space node, information on the second real space node, and information on the real space edge as input, and outputs information on the first final existence probability and information on the second final existence probability.(8) The information processing device according to (6) or (7), wherein the existence probability management unit generates second real space graph information with existence probabilities, in which information about the first real space node includes information about the first final existence probability and information about the second real space node includes information about the second final existence probability. (9) The information processing device according to (7), wherein the information about the first real space node input to the prediction network includes a category likelihood of the first real object, and the information about the second real space node includes a category likelihood of the second real object. (10) The information processing device according to (8) or (9), further comprising: a behavior planning unit that performs a behavior plan for an autonomous moving body based on the second real space graph information with existence probabilities. (11) The information processing device according to any of (8) to (10), further comprising: an information updating unit that updates parameters of the prediction network using a value according to a determination result of whether the first real object and the second real object have moved as correct answer data. (12) The information processing device according to (11), wherein the information updating unit updates parameters of the prediction network each time the second real space graph information with existence probability is generated. (13) The information processing device according to any of (2) to (12), further comprising an image acquisition unit that acquires the first captured image and the second captured image from a sensor. (14) An information processing method, wherein an information processing device acquires real space graph information, which is information of a graph structure including first real space nodes and second real space nodes that represent a first real object and a second real object in the real space, respectively, and real space edges that represent a positional relationship between the first real object and the second real object, based on a captured image that represents a real space at a first time point, and calculates a first existence probability of the first real object and a second existence probability of the second real object based on the elapsed time from the first time point to a second time point.(15) A program causing a computer to execute a process of: acquiring, based on a captured image representing a real space at a first time, real space graph information that is information of a graph structure including first real space nodes and second real space nodes representing a first real object and a second real object in the real space, respectively, and real space edges representing a positional relationship between the first real object and the second real object; and calculating, based on the elapsed time from the first time to a second time, a first existence probability of the first real object and a second existence probability of the second real object.

[0138] REFERENCE SIGNS LIST 1 Autonomous moving body, 2 Charging stand, 11 3D Scene Graph with presence probability generation unit, 12 Behavior planning unit, 51 Image acquisition unit, 52 Space reconstruction unit, 53 3D Scene Graph generation unit, 54 Intermediate presence probability generation unit, 55 Final presence probability generation unit, 56 Moving object detection unit, 57 Weight update unit, 61 Real space graph acquisition unit, 62 Presence probability management unit, 63 Information update unit

Claims

1. An information processing device comprising: a real space graph acquisition unit that acquires, based on a first captured image representing a real space at a first time, real space graph information, which is information of a graph structure including first real space nodes and second real space nodes representing a first real object and a second real object in the real space, respectively, and real space edges representing the positional relationship between the first real object and the second real object; and an existence probability management unit that calculates a first existence probability of the first real object and a second existence probability of the second real object based on the elapsed time from the first time to a second time.

2. The information processing device according to claim 1, wherein the real space graph acquisition unit acquires the real space graph information based on a second captured image representing the real space at the second time point that is later than the first time point.

3. The information processing device according to claim 2, wherein the existence probability management unit generates first real-space graph information with existence probability, the first real-space node information of the real-space graph information acquired based on the second captured image including information on the first existence probability, and the second real-space node information including information on the second existence probability.

4. The information processing device according to claim 3, wherein the existence probability management unit calculates the first existence probability and the second existence probability, which decrease according to the elapsed time.

5. The information processing device according to claim 4, wherein the existence probability management unit calculates the first existence probability and the second existence probability using the reliability of the segmentation results of the first real object and the second real object as initial values.

6. The information processing device according to claim 3, wherein the existence probability management unit calculates a first final existence probability of the first real object and a second final existence probability of the second real object using information contained in the first real space graph information with existence probability and the elapsed time as inputs to a prediction network.

7. The information processing device according to claim 6, wherein the prediction network is a network that receives information including information on the first real space node, information on the second real space node, and information on the real space edge as input, and outputs information on the first final existence probability and information on the second final existence probability.

8. The information processing device according to claim 6, wherein the existence probability management unit generates second real space graph information with existence probabilities, the second real space graph information including information on the first real space node about the first final existence probability and information on the second real space node about the second final existence probability.

9. The information processing device according to claim 7, wherein the information of the first real space node input to the prediction network includes a category likelihood of the first real object, and the information of the second real space node includes a category likelihood of the second real object.

10. The information processing device according to claim 8, further comprising a behavior planning unit that performs behavior planning for the autonomous mobile body based on the second real space graph information with existence probability.

11. The information processing device according to claim 8, further comprising an information updating unit that updates the parameters of the prediction network using a value according to the determination result of whether the first real object and the second real object have moved as correct answer data.

12. The information processing device according to claim 11, wherein the information update unit updates the parameters of the prediction network every time the second real space graph information with existence probabilities is generated.

13. The information processing device according to claim 2, further comprising an image acquisition unit that acquires the first captured image and the second captured image from a sensor.

14. An information processing method, comprising: an information processing device, based on a captured image representing a real space at a first time, acquiring real space graph information, which is information of a graph structure including first real space nodes and second real space nodes representing a first real object and a second real object in the real space, respectively, and real space edges representing the positional relationship between the first real object and the second real object; and calculating a first existence probability of the first real object and a second existence probability of the second real object based on the elapsed time from the first time to a second time.

15. A program causing a computer to execute the following process: acquire, based on a captured image representing a real space at a first time, real space graph information, which is information of a graph structure including first real space nodes and second real space nodes representing a first real object and a second real object in the real space, respectively, and real space edges representing the positional relationship between the first real object and the second real object; and calculate, based on the elapsed time from the first time to a second time, a first existence probability of the first real object and a second existence probability of the second real object.

Images