Object relationship estimation from 3D semantic grid

Abstract

The present disclosure relates to object relationship estimation from 3D semantic meshes. The implementations disclosed herein provide systems and methods for determining relationships between objects based on a raw semantic mesh representing vertices and faces of a 3D geometry of a physical environment. Such raw semantic meshes can be generated and used to provide input to a machine learning model that estimates relationships between objects in the physical environment. For example, the machine learning model can output a graph of nodes and edges that indicates that a vase is on a table or that a particular instance V1 of a vase is on a particular instance T1 of a table.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 898,049, filed September 10, 2019, which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to determining objects in a physical environment and relationships between the objects, and specifically to systems, methods, and devices that use machine learning to estimate relationships and classify objects in a physical environment. BACKGROUND

[0004] Various computer vision techniques are used to identify physical objects in a physical environment. For example, images captured by a camera can be automatically evaluated to determine that a physical environment includes a table, a chair, and a vase. However, such techniques can provide little or no information about relationships between the objects. For example, these techniques can not determine whether the vase is on the table top or whether the chair is next to the table and facing the table. SUMMARY

[0005] The implementations disclosed herein provide systems and methods that determine relationships between objects based on a raw semantic mesh of vertices and faces representing 3D geometry of a physical environment. Such raw semantic meshes can be generated and used to provide input to a machine learning model that estimates relationships between objects in a physical environment. For example, the machine learning model can output a graph of nodes and edges that indicates that a vase is on a table top or that a particular instance of a vase, VI, is on a particular instance of a table, Tl.

[0006] Some implementations provide a method of estimating or otherwise determining relationships between objects in a physical environment. The example method can be implemented by a computing device that uses a processor to execute instructions. The method generates a semantic mesh of vertices and faces. The 3D shape of the semantic mesh represents 3D geometry of object surfaces of a physical environment. Further, at least a portion of the vertices have semantic labels that identify object types (e.g., table, chair, vase, etc.). For example, in such a semantic mesh, vertices on a floor surface can be labeled as “floor,” and vertices on a chair surface can be labeled as “chair.” Adjacent vertices in the semantic mesh form faces, e.g., three “floor” vertices can define a face that represents a portion of a floor surface. Each vertex can store or otherwise be associated with a location (e.g., x, y, z location information).

[0007] One or more machine learning algorithms can be used to generate the semantic mesh. In some implementations, the semantic mesh can be generated based on a set of one or more images of the physical environment, e.g., using a 3D reconstruction technique (e.g., an algorithm or a machine learning model) that provides a three-dimensional (3D) triangular mesh representing the surface of the physical environment and / or a semantic image segmentation technique (e.g., an algorithm or a machine learning model) for annotating or classifying objects of the physical environment.

[0008] The method can convert the original semantic mesh into a graph representation having nodes and edges determined based on the vertices and faces of the semantic mesh. The graph representation can be in a full form or in a simplified form. For example, the method can perform a simplification technique to reduce the complexity of the graph representation by reducing the number of nodes.

[0009] The method uses one or more machine learning models to estimate relationships between the objects. In some implementations, the method inputs a representation of the graph (e.g., a full graph representation or a simplified version of the graph) into one or more machine learning models (e.g., neural networks). The method can also use one or more machine learning models to classify the objects or update previously determined object classification labels based on the representation of the graph and / or the estimated relationships. In some implementations, the method classifies the objects or updates the classification labels. Using a simplified representation of the original semantic mesh (e.g., a simplified graph) as input to the machine learning model can improve the accuracy and efficiency of the machine learning model. Using a simplified version as input can allow for accurate results using a representation of the semantic mesh (e.g., a graph representation or a simplified graph representation) rather than using randomly or all initialized connected nodes. In some implementations, the machine learning model outputs a graph representation representing the objects and their relationships. In some implementations, the machine learning model outputs pairwise relationships between the objects.

[0010] In some implementations, the relationships between the objects produced by the machine learning model are estimates using probabilities, e.g., providing a 99% likelihood that table Tl is on floor Fl, a 90% likelihood that container Cl is on table Tl, a 75% likelihood that chair Cl is next to table Tl, a 75% likelihood that chair C2 is facing television TVl, a 75% likelihood that handle Hl is attached to door Dl, and so on.

[0011] Some implementations disclosed herein provide systems and methods of generating synthetic data such as semantic meshes that are similar to scanned meshes of real physical environments and contain objects with known / labelled relationships. Such synthetic data can be used to train machine learning models to classify objects and determine relationships from semantic meshes using the techniques disclosed herein and for various other purposes. An example method can be implemented by a computing device that uses a processor to execute instructions. The method models multiple meshes for independent objects of a synthetic environment (e.g., individual meshes for each of a chair, a table, a floor, etc.). Each of the independent objects is associated with an object type and has an independent mesh that includes semantic labels (e.g., a chair mesh includes all nodes labelled as “chair”). In some implementations, a user creates a synthetic room by separately creating or inserting a floor, a chair, a table, a wall, etc.

[0012] The method determines a volumetric representation based on the multiple meshes, e.g., based on all of the meshes created by the user for the synthetic room. For example, this can involve converting the individual meshes of the independent objects to form a single density cloud or point set that represents all of the meshes. The method then determines a combined mesh based on the volumetric representation. For example, the density cloud generated from the multiple individual meshes can be converted back to a single mesh that represents all of the objects.

[0013] The method also determines relationships between the independent objects of the single mesh. For example, the relationships can be determined based on manual input information from the creator of the synthetic scene or automatically determined based on evaluating the individual meshes or the single mesh. The single mesh, the identifications of the synthetic objects known to be represented by the mesh, and / or the relationships between such objects can then be provided to train a machine learning model to classify objects and estimate relationships from semantic meshes using the techniques disclosed herein and for various other purposes. In some implementations, a machine learning model is trained to receive a semantic mesh (or a simplified version of a semantic mesh) and produce an output that identifies relationships between objects represented in the semantic mesh (or the simplified version of the semantic mesh).

[0014] According to some implementations, a non-transitory computer-readable storage medium has stored therein instructions that are computer-executable to perform or cause performance of any of the methods described herein. According to some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to, working with the one or more processors, carry out or cause carrying out of any of the methods described herein. BRIEF DESCRIPTION OF DRAWINGS

[0015] Accordingly, the disclosure can be understood by those of ordinary skill in the art, and a more complete appreciation of the disclosure can be gained by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0016] Figure 1 is a block diagram of an example operating environment according to some implementations.

[0017] Figure 2 is a block diagram of an example controller according to some implementations.

[0018] Figure 3 is a block diagram of an example electronic device according to some implementations.

[0019] Figure 4 is a flow diagram illustrating an example method for determining relationships between objects in a physical environment using a semantic grid according to some implementations.

[0020] Figure 5 is a block diagram illustrating an example method for creating a simplified version of a semantic grid.

[0021] Figure 6 is a block diagram illustrating an example semantic grid.

[0022] Figure 7 is a block diagram illustrating a simplified version of a semantic grid of Figure 6 according to some implementations.

[0023] Figure 8 is a block diagram illustrating a simplified version of a semantic grid of Figure 7 according to some implementations.

[0024] Figure 9 is a block diagram illustrating a simplified version of a semantic grid of Figure 8 according to some implementations.

[0025] Figure 10 is a block diagram illustrating an example method for determining relationships between objects in a physical environment using a simplified version of a semantic grid according to some implementations.

[0026] Figure 11 is a block diagram representing the addition of a virtual object to a CGR environment represented by an object relationship graph of Figure 10 according to some implementations.

[0027] Figure 12 is a block diagram illustrating the generation of synthetic data and the training of a machine learning model using the synthetic data.

[0028] In accordance with common practice the various features illustrated in the drawings can not be drawn to scale. Accordingly, the dimensions of the various features can be arbitrarily expanded or reduced for the clarity of presentation. Also, some of the drawings can not depict all of the components of a given system, method or device. Finally, like reference numerals can be used to denote like features throughout the specification and figures. DETAILED DESCRIPTION

[0029] Many details are described to provide a thorough understanding of the example implementations shown in the drawings. The drawings, however, merely illustrate some example aspects of the present disclosure and should not be construed to be limiting. One of ordinary skill in the art would understand that other effective aspects or variants do not include all of the specific details described herein. Moreover, well-known systems, methods, components, devices and circuits have not been described in exhaustive detail so as not to obscure more pertinent aspects of the example implementations described herein. While the example implementations are amenable to various modifications and alternative forms, specific embodiments have been shown by way of example in Figures 1 to 3 Example implementations are shown that involve a handheld electronic device, but other implementations do not necessarily involve a handheld device and can involve other types of devices, including but not limited to televisions, set-top box devices, laptop computers, desktop computers, gaming devices, home automation devices, watches, head-mounted devices (HMDs) and other wearable electronic devices, and other devices that process or display content.

[0030] Figure 1 is a block diagram of an example operating environment 100 according to some implementations. While relevant features are shown, one of ordinary skill in the art will appreciate from the disclosure herein that various other features have not been shown for brevity and so as not to obscure more pertinent aspects of the example implementations disclosed herein. To that end, as a non-limiting example, the operating environment 100 includes a controller 110 and an electronic device 120, one or both of which can be in a physical environment.

[0031] The electronic device 120 is configured to process or display content. In some implementations, the electronic device 120 includes a suitable combination of software, firmware or hardware. The content can be provided for display on the electronic device 120 from a recorded source or a live source. For example, the content can be stored in a memory on the electronic device 120, the controller 110 or other location. In another example, the content can be a stream of frames captured or processed in real-time by a camera on the electronic device 120, the controller 110 or other location. Reference is made to the following description of the electronic device 120 for further details. Figure 3 The electronic device 120 is described in further detail. In some implementations, the functionality of the controller 110 is provided by or combined with the electronic device 120, for example, in the case of an electronic device used as a standalone unit.

[0032] In some implementations, the controller 110 is a computing device that is in a local or remote location relative to the physical environment 105. In one example, the controller 110 is a local server located within the physical environment 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside of the physical environment 105. In some implementations, the controller 110 is communicatively coupled with the electronic device 120 via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802. l lx, IEEE 802.16x, IEEE 802.3x, etc.).

[0033] Figure 2 is a block diagram of an example of a controller 110 according to some implementations. While certain specific features are illustrated, one of ordinary skill in the art will appreciate from the disclosure herein that various other features have not been illustrated for the sake of brevity and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the controller 110 includes one or more processing units 202 (e.g., microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802. l lx, IEEE 802.16x, global system for mobile (GSM) communication, code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), BLUETOOTH, ZIGBEE, or similar type interface), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0034] In some implementations, the one or more communication buses 204 include circuitry that interconnects system components and controls communication between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0035] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid state memory devices. In some implementations, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 220 optionally includes one or more storage devices remotely located from the one or more processing units 202. Memory 220 includes non-transitory computer readable media. In some implementations, memory 220 or the non-transitory computer readable media of memory 220 stores instructions, modules, and data structures used by processor(s) 202 in executing the processes of FIGS. 1-6. For example, memory 220 includes an optional operating system 230 and computer vision (CV) and computer generated reality (CGR) modules 240.

[0036] Operating system 230 includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0037] In some implementations, CV and CGR modules 240 include a semantic mesh unit 242, a relationship classification unit 244, a CGR unit 246, and a training unit 248. Semantic mesh unit 242 can be configured to generate a semantic mesh, e.g., using a 3D reconstruction technique (e.g., an algorithm or a machine learning model) that provides a 3D triangle mesh representing surfaces of a physical environment and / or a semantic image segmentation technique (e.g., an algorithm or a machine learning model) for annotating or classifying objects of the physical environment. Semantic mesh unit 242 can be configured to reduce the semantic mesh, e.g., by reducing the number of vertices / nodes in the semantic mesh via the techniques disclosed herein.

[0038] Relationship classification unit 244 can be configured to classify objects and estimate relationships between objects, e.g., using a machine learning model (e.g., a neural network) that uses a representation of the raw semantic mesh provided by semantic mesh unit 242.

[0039] CGR unit 246 can be configured to render a CGR environment that includes a depiction of a physical environment, virtual content, or both. CGR unit 246 can be configured to utilize objects and / or relationships classified or estimated by relationship classification unit 244. For example, CGR unit 246 can respond to a user’s voice command to place a virtual vase on the “table next to the window” based on the relationship defining that table T3 is next to window W2, e.g., it knows to place the virtual vase on table T3 and not another table that is not next to a window.

[0040] The training unit 248 can be configured to generate synthetic data, such as semantic grids, that are similar to scanned grids of real physical environments and have objects with known / labelled relationships. Such synthetic data can be used to train the machine learning model of the relationship classification unit 244 to classify objects and estimate relationships from semantic grids generated by the semantic grid unit 242 based on images of physical environments.

[0041] Although these modules and units are shown as residing on a single device (e.g., the controller 110), it will be appreciated that, in other implementations, any combination of these modules and units can be located in separate computing devices. Moreover, Figure 2 More functionality is used as a functional description of the various features present in particular implementations, as opposed to structural diagrams of the implementations described herein. As will be appreciated by those skilled in the art, items shown separately can be combined, and items shown separately can be divided. For example, Figure 2 Some of the functional modules shown separately in the can be implemented in a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of the particular functions between them, as well as how the features are allocated among the modules, will vary from one implementation to another and, in some implementations, depend in part on the particular combination of hardware, software, or firmware chosen to implement the particular implementation.

[0042] Figure 3 is a block diagram of an example of the electronic device 120 according to some implementations. Although some specific features are shown, one of skill in the art will appreciate from the disclosure herein that, for brevity and so as not to obscure more relevant aspects of the implementations disclosed herein, various other features are not shown. To that end, as a non-limiting example, in some implementations, the electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.1 lx, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more inward- or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0043] In some implementations, the one or more communication buses 304 include circuitry that interconnects and controls communications between system components. In some implementations, the one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptics engine, or one or more depth sensors (e.g., structured light, time-of-flight, etc.), among others.

[0044] In some implementations, the one or more displays 312 are configured to present a CGR experience to a user. In some implementations, the one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), micro-electromechanical system (MEMS), or similar display types. In some implementations, the one or more displays 312 correspond to waveguide, diffractive, reflective, polarized, holographic, etc. display types. For example, the electronic device 120 includes a single display. As another example, the electronic device 120 includes a display for each eye of a user.

[0045] The memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some implementations, the memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. The memory 320 optionally includes one or more storage devices remotely located from the one or more processing units 302. The memory 320 comprises a non-transitory computer readable storage medium. In some implementations, the memory 320, or the non-transitory computer readable storage medium of the memory 320, stores the following programs, modules, and data structures, among others, including an optional operating system 330 and a computer vision (CV) and computer generated reality (CGR) module 340.

[0046] The operating system 330 includes procedures for handling various basic system services and for performing hardware dependent tasks.

[0047] In some implementations, the CV and CGR module 340 includes a semantic mesh unit 342, a relationship classification unit 344, a CGR unit 346, and a training unit 248. The semantic mesh unit 342 can be configured to generate a semantic mesh, e.g., using a 3D reconstruction technique (e.g., an algorithm or a machine learning model) that provides a 3D triangle mesh representing the surface of a physical environment and / or a semantic image segmentation technique (e.g., an algorithm or a machine learning model) for annotating or classifying objects of the physical environment. The semantic mesh unit 342 can be configured to reduce the semantic mesh, e.g., by reducing the number of vertices / nodes in the semantic mesh via the techniques disclosed herein.

[0048] The relationship classification unit 344 can be configured to classify objects and estimate relationships between objects, e.g., using a machine learning model (e.g., a neural network) that uses a representation of the raw semantic mesh provided by the semantic mesh unit 342.

[0049] The CGR unit 346 can be configured to render a CGR environment that includes a depiction of the physical environment and / or virtual content. The CGR unit 346 can be configured to utilize objects and / or relationships determined by the relationship classification unit 344. For example, the CGR unit 346 can respond to a voice command by a user to place a virtual vase on the “table next to the window” based on a relationship defining that table T3 is next to window W2.

[0050] The training unit 348 can be configured to generate synthetic data, such as semantic meshes, that are similar to scanned meshes of real physical environments and have objects with known / labelled relationships. Such synthetic data can be used to train the machine learning model of the relationship classification unit 344 to classify objects and estimate relationships from semantic meshes generated by the semantic mesh unit 342 based on images of a physical environment.

[0051] Furthermore, Figure 3 More functionality is described in terms of various features that are present in particular implementations, as opposed to structural diagrams of the implementations described herein. As will be recognized by those skilled in the art, items individually listed can be combined, and some items can be separated. For example, Figure 3 Some of the functional modules shown separately in the implementations can be implemented in a single module, and various functions of a single functional block can be implemented by one or more functional blocks in various implementations. The actual number of modules and the division of particular functions between the modules can vary from implementation to implementation depending on the hardware, software, or firmware chosen for a particular implementation, and in some implementations, partially depends on the particular combination of hardware, software, or firmware selected for the particular implementation.

[0052] Figure 4is a flowchart showing an example method for determining relationships between objects in a physical environment using a semantic mesh, according to some implementations. In some implementations, the method 400 is performed by a controller 100 or electronic device 120 of the system 100. Figures 1 to 3 The method 400 can be performed at a television, set-top box, mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices in communication with one another. In some implementations, the method 400 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer readable medium, such as a memory.

[0053] At block 402, the method 400 generates a semantic mesh. The 3D shape of the semantic mesh represents surfaces of objects of the physical environment, and at least a portion of the vertices of the raw semantic mesh have semantic labels that identify object types (e.g., table, chair, vase, etc.). For example, in such a mesh, vertices on the floor surface can be labeled as “floor,” and vertices on a chair surface can be labeled as “chair.”

[0054] In some implementations, generating the semantic mesh involves estimating depths of different portions (e.g., pixels) of multiple images of the physical environment, correlating relationships (e.g., transforms) between poses of cameras used to capture the images, and generating a 3D map or other reconstruction of surfaces in the physical environment. Such a map / reconstruction can include locations for different surface points in a 3D coordinate system. In some implementations, a 3D triangle-based mesh is generated, and each vertex of the mesh stores or is otherwise associated with a location (e.g., x, y, z location information). The 3D reconstruction can be generated using one or more machine learning algorithms.

[0055] In some implementations, generating the semantic mesh involves techniques that do not utilize machine learning models. For example, one example approach projects vertices of a 3D reconstruction mesh from different viewpoints onto semantic segmentation images using an algorithm, and then uses a majority voting technique to determine final vertex labels.

[0056] In some implementations, semantic information is included in or added to the 3D reconstruction of the physical environment. One or more machine learning models can be used to classify objects, including or adding semantic information to the 3D reconstruction. In some implementations, a machine learning model is used to generate semantic labels for pixels of one or more images of the physical environment. Such semantic labels can be associated with corresponding portions of the 3D reconstruction based on determining an association between the images and the 3D reconstruction. In some implementations, a machine learning model uses the 3D reconstruction and semantically labeled images (e.g., automatically labeled images) as input to produce a semantically labeled mesh, e.g., a semantic mesh having at least some vertices associated with semantic labels.

[0057] At block 404, the method 400 converts the semantic mesh to a graph. This can involve converting vertices of the semantic mesh to nodes and determining edges that connect nearby nodes to each other. The edges of the graph can represent sides of faces of the semantic mesh. The nodes can store information about coordinates (e.g., x, y, z coordinates), semantic labels, and other information extracted from the semantic mesh.

[0058] At block 406, the method 400 performs graph simplification and instance detection. Graph simplification can reduce the complexity of the original semantic mesh or a graph representing the original semantic mesh, e.g., reducing the number of nodes in such a graph. Figure 5 Exemplary graph simplification techniques are provided. Instance detection involves identifying or estimating individual instances of objects in the semantic mesh or a graph representing the semantic mesh. In some implementations, instance detection is performed using a machine learning model that utilizes semantic locations and 3D locations of nodes of the graph to identify instances.

[0059] At block 408, the method 400 uses a machine learning model to estimate relationships between objects of the semantic mesh or a graph representing the semantic mesh. Examples of relationships include, but are not limited to, estimating that a first object is on top of a second object, a first object is below a second object, a first object is next to a second object, a first object is in contact with a second object, a first object is attached to a second object, a first object is leaning on a second object, a first object (e.g., a person) is holding a second object, a first object is within a second object, a first object is hanging from a second object, a first object is near a second object, a first object is facing a second object, a first object is facing away from a second object, a first object is in front of a second object, a first object is behind a second object, and a first object and a second object are aligned with each other.

[0060] In some implementations, the relationships between objects produced by the machine learning model are estimated as probabilities, e.g., providing a likelihood of 99% that table Tl is on floor Fl, a likelihood of 90% that container Cl is on table Tl, a likelihood of 75% that chair Cl is next to table Tl, a likelihood of 75% that chair C2 is facing television TVl, a likelihood of 75% that handle Hl is attached to door Dl, etc.

[0061] Additionally, images of the physical environment and / or pose information regarding the position and orientation of the camera that captured the image can be input to the machine learning model to further improve its accuracy or efficiency. Camera pose information can be used to produce viewpoint-specific relationships. For example, if a computer-generated reality (CGR) environment is produced using live images, the pose (e.g., position and orientation) of the user’s current device can be used to facilitate viewpoint-specific relationship determinations, e.g., identifying that a chair is in front of a table from the user’s current viewpoint.

[0062] In some implementations, the machine learning model outputs a graph representing the objects and their relationships. Figure 9 An object relationship graph 920 that can be output by the method 400 is shown. In some implementations, the machine learning model outputs pairwise relationships between objects. Relationships between two or more objects can be represented in any suitable format for output and use.

[0063] At block 408, the method 400 uses the machine learning model to update the object classification labels of the vertices of the semantic grid or nodes of a graph representing the semantic grid. In some implementations, the relationships between objects are used as input to update the labels of the vertices of the semantic grid or nodes of a graph representing the semantic grid. In some implementations, the machine learning model used to update the classification labels is a classifier implemented using a neural network.

[0064] In some implementations, the objects and / or relationships estimated in the method 400 and other techniques disclosed herein are used to facilitate the provision of a computer-generated reality (CGR) environment. A CGR environment refers to a fully or partially simulated environment with which a person senses and / or interacts via an electronic system. In CGR, a subset of a person’s physical motions, or representations thereof, are tracked, and, in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner that comports with at least one physical law. For example, a CGR system can detect a person’s head turning, and in response, adjust graphical content and an acoustic field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to characteristics of virtual objects in a CGR environment can be made in response to representations of physical motions (e.g., voice commands).

[0065] A person can sense and / or interact with a CGR object using any of their senses, including sight, hearing, touch, taste, and smell. For example, a person can sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides a perception of point audio sources in a 3D space. As another example, an audio object can enable audio transparency that selectively incorporates ambient sound from a physical environment with or without computer-generated audio. In some CGR environments, a person can sense and / or only interact with audio objects.

[0066] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment refers to a simulated environment designed to be completely based on computer-generated sensory inputs for one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of a person's presence within the computer-generated environment, and / or through a simulation of a subset of a person's physical movements within the computer-generated environment.

[0067] In contrast to a VR environment designed to be completely based on computer-generated sensory inputs, a mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory inputs from a physical environment or representations thereof in addition to computer-generated sensory inputs (e.g., virtual objects). On a virtual continuum, a mixed reality environment is anywhere between a completely physical environment as one end and a virtual reality environment as the other end, but does not include either end.

[0068] In some MR environments, computer-generated sensory inputs can respond to changes in sensory inputs from a physical environment. Additionally, some electronic systems for presenting MR environments can track position and / or orientation with respect to a physical environment to enable virtual objects to interact with real objects (i.e., physical articles from the physical environment or representations thereof). For example, a system can cause movement such that a virtual tree appears to be stationary with respect to a physical ground.

[0069] Examples of mixed reality include augmented reality and augmented virtuality. An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment or its representation. For example, an electronic system for presenting an AR environment can have a transparent or translucent display through which a person can view the physical environment directly. The system can be configured to present virtual objects on the transparent or translucent display so that the person, using the system, perceives the virtual objects superimposed over the physical environment. Alternatively, a system can have an opaque display and one or more imaging sensors that capture images or video of the physical environment, which are representations of the physical environment. The system combines the images or video with virtual objects and presents the combination on the opaque display. A person, using the system, views the physical environment indirectly via the images or video of the physical environment and perceives the virtual objects superimposed over the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Further alternatively, a system can have a projection system that projects virtual objects into the physical environment, for example, as a hologram or on a physical surface, so that a person, using the system, perceives the virtual objects superimposed over the physical environment.

[0070] An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system can transform the one or more sensor images to impose a selected perspective (e.g., viewpoint) that is different from the perspective captured by the imaging sensors. As another example, a representation of a physical environment can be transformed by graphically modifying (e.g., enlarging) portions thereof so that the modified portions can be representative but not true versions of the originally captured images. As another example, a representation of a physical environment can be transformed by graphically eliminating or blurring portions thereof.

[0071] An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. The sensory inputs can be representations of one or more features of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but a person’s face is realistically reproduced from images taken of a physical person. As another example, a virtual object can take on the shape or color of a physical article imaged by one or more imaging sensors. As another example, a virtual object can take on a shadow that is consistent with the position of the sun in the physical environment.

[0072] There are many different types of electronic systems that enable people to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields that integrate display capabilities, windows that integrate display capabilities, displays formed as lenses that are designed to be placed on a person’s eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors to capture images or video of a physical environment, and / or one or more microphones to capture audio of the physical environment. A head-mounted system can have a transparent or translucent display instead of an opaque display. A transparent or translucent display can have a medium through which light representative of an image is directed to a person’s eyes. The display can utilize digital light projection, OLEDs, LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one implementation, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects graphical images onto a person’s retina. Projection systems can also be configured to project virtual objects into a physical environment, for example, as a hologram or on a physical surface.

[0073] In some implementations, the object classifications and / or relationship estimates produced in the method 400 and other techniques disclosed herein are used to facilitate the provision of a computer-generated reality (CGR) environment. For example, during rendering of a CGR environment that includes objects classified by the method 400 and other techniques disclosed herein and estimated relationships, input can be received to position a virtual object in the CGR environment. For example, a user can provide a voice command to “place a virtual gyroscope on the table next to the wall.” A CGR unit (e.g., the CGR unit 246 of FIG. 2 or the CGR unit 346 of FIG. 3) providing the CGR environment can determine a location for the virtual object in the CGR environment based on the input and relationships between objects provided by the method 400 or other techniques disclosed herein. For example, it can identify the table Tl, rather than the table T2, as the table on which to automatically place the virtual gyroscope based on determining that the relationship “table Tl next to wall Wl” best matches the input “table next to the wall.” The virtual object (e.g., the virtual gyroscope) can then be automatically positioned at the appropriate location in the CGR environment for the user to experience. Figure 2 Figure 3 In some implementations, the object classifications and / or relationship estimates produced in the method 400 and other techniques disclosed herein are used to facilitate the provision of a computer-generated reality (CGR) environment. For example, during rendering of a CGR environment that includes objects classified by the method 400 and other techniques disclosed herein and estimated relationships, input can be received to position a virtual object in the CGR environment. For example, a user can provide a voice command to “place a virtual gyroscope on the table next to the wall.” A CGR unit (e.g., the CGR unit 246 of FIG. 2 or the CGR unit 346 of FIG. 3) providing the CGR environment can determine a location for the virtual object in the CGR environment based on the input and relationships between objects provided by the method 400 or other techniques disclosed herein. For example, it can identify the table Tl, rather than the table T2, as the table on which to automatically place the virtual gyroscope based on determining that the relationship “table T1 next to wall Wl” best matches the input “table next to the wall.” The virtual object (e.g., the virtual gyroscope) can then be automatically positioned at the appropriate location in the CGR environment for the user to experience.​

[0074] In some implementations, a semantic grid or a graph representing the semantic grid is simplified to have fewer vertices / nodes. Figure 5 is a block diagram illustrating an exemplary method for creating a simplified version of a diagram. In some implementations, method 500 is performed by a device (e.g., Figures 1-3 The method 500 may be performed by a controller 100 or an electronic device 120 of a computer. The method 500 may be performed at a mobile device, an HMD, a desktop computer, a laptop computer, a server device, or by multiple devices communicating with each other. In some implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 500 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

[0075] At box 502, method 500 involves determining that two nodes are connected by an edge. In some specific implementations, this can be achieved by checking or traversing on a semantic grid. At box 504, method 500 determines that two nodes have the same semantic label, for example, both are labeled "chair" or the like. At box 506, method 500 generates a combined node by combining the two nodes. Any other node previously connected to either of the two source nodes is connected to the combined node instead. In addition, the combined node can store the average position of the first node and the second node and / or the positions of the respective source nodes (for example, the first position of the first node and the second position of the second node) or otherwise be associated with these positions. At box 508, method 500 merges duplicate edges due to the combined node.

[0076] Figures 6 to 9 A simplified exemplary semantic grid is shown. Figure 6 is a block diagram illustrating an exemplary semantic grid 600. The semantic grid 600 includes nodes 610a-h, 620a-r, 630a-i connected to each other by edges represented by dashed lines between the nodes 610a-h, 620a-r, 630a-i. In this example, the nodes 610a-h are semantically labeled "cup", the nodes 620a-r are semantically labeled "table", and the nodes 630a-i are semantically labeled "floor". In typical use cases in real physical settings, additional objects and nodes will typically be involved, and the semantic grid will occupy a 3D coordinate space. Figure 6 In the example of , for ease of illustration, the semantic grid 600 includes only a limited number of nodes and edges depicted in two dimensions.

[0077] Figure 7 It shows Figure 6 A simplified version of the semantic grid 600 is shown in FIG. Figure 6Nodes 610a-h in the source node semantically labeled "cup" are combined into a single combined node 710a. These nodes 610a-h are combined based on determining that the nodes 610a-h are connected to each other and have the same label. Therefore, the edge between the nodes 610a-h is eliminated. Combined node 710a is assigned the semantic label "cup" based on the source node's label and is assigned a position based on the source node's position (e.g., average position).

[0078] Figure 8 This is a further simplification Figure 7 A block diagram of the semantic grid of . In this example, Figure 6 and Figure 7 Nodes 630a-i in the source node semantically labeled "floor" are combined into a single combined node 730a. These nodes 630a-i are combined based on determining that the nodes 630a-i are connected to each other and have the same label. Therefore, the edges between the nodes 630a-i are eliminated. The combined node 730a is assigned the semantic label "floor" based on the label of the source node and is assigned a position based on the source node position (e.g., average position).

[0079] Figure 9 This is a further simplification Figure 8 A block diagram of the semantic grid of . In this example, Figures 6 to 8 Nodes 620a-r in the source node semantically labeled "table" are combined into a single combined node 720a. These nodes 620a-r are combined based on determining that the nodes 620a-r are connected to each other and have the same label. Therefore, the edge between the nodes 620a-r is eliminated. The combined node 720a is assigned the semantic label "table" based on the label of the source node and is assigned a position based on the source node position (e.g., average position).

[0080] Figure 10 is a block diagram illustrating an exemplary method 900 for determining relationships between objects in a physical environment using a simplified version 905 of a semantic grid. In this example, the semantic grid is represented by a simplified version of the semantic grid that includes Figure 9 70. The combined nodes 710a, 720a, 730a of the semantic grid and the additional combined node 740a included for illustrative purposes. In this example, a simplified version 905 of the semantic grid is input to a machine learning model 910 that generates an object relationship graph 920. In some implementations, the machine learning model 910 is a trained neural network to generate an object relationship graph given a simplified semantic grid input. In some implementations, the machine learning model 910 is a convolutional neural network (CNN). In some implementations, the machine learning model 910 involves recursive learning via a feedback loop, for example, including a recursive neural network (RNN).

[0081] In this example, the object relationship graph 920 includes a first object 930, a second object 940, a third object 950, and a fourth object 950. The first object 930 has a type label of “cup” and an instance label of “Cl”. The second object 940 has a type label of “table” and an instance label of “Tl”. The third object 950 has a type label of “chair” and an instance label of “CHl”. The fourth object 960 has a type label of “floor” and an instance label of “Fl”. In addition, the object relationship graph 920 identifies relationships 935, 945, 955, 965 between the objects 930, 940, 950, 960. The first relationship 935 identifies that the first object 930 is “on” the second object 940. The second relationship 945 identifies that the second object 940 is “next to” the third object 950. The third relationship 955 identifies that the second object 940 is “on” the fourth object 960. The fourth relationship 965 identifies that the third object 950 is “on” the fourth object 960. It should be noted that the edges and nodes shown in the object relationship graph 920 need not be limited to holding only one label - each node and edge can produce two, three, or more labels.

[0082] Figure 11 is a representation of adding a virtual object to a CGR environment represented by Figure 10 a diagram of a representation of a CGR environment by an object relationship graph. In this example, a user provides a textual or verbal command to “add a virtual chair next to the table”. Based on this command, the second object 940 (table Tl) is identified as the table referenced in the command. A virtual chair is automatically added in the CGR environment based on the command, and the object relationship graph 1100 is updated to reflect the addition, including relationships determined based on the command. In particular, the command identifies that the virtual chair should be positioned “next to” the table. Based on this, an automatic position in the CGR environment is selected (e.g., using rules defining, for example, that a “next to” trigger places an object 6 inches away from the table, etc.). In addition, the object relationship graph 1100 is extended to include a new object 1130 (e.g., the virtual chair with a type label of “chair” and an instance label of “VCHl”). New relationships are also identified and included in the object relationship graph 1100. In particular, a fifth relationship 1135 identifies that the new object 1130 is next to the second object 940, and a sixth relationship identifies that the new object 1130 is “on” the fourth object 960. Such relationships can be determined based on the command (e.g., the fifth relationship 1135 is determined based on the “next to” in the command). Such relationships can be determined based on algorithms, rules, or machine learning. In this example, the sixth relationship 1145 is determined based on a rule specifying that if a virtual object is added next to an object that is “on” the floor, then by default the virtual object will also be considered to be “on” the floor.

[0083] Figure 12is a block diagram illustrating a method 1200 of generating synthetic data and training a machine learning model using the synthetic data. In some implementations, the method 1200 is performed by a controller 100 or electronic device 120 of FIG. 1. The method 1200 can be performed at a television, set-top box, mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices in communication with one another. In some implementations, the method 1200 is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method 1200 is performed by a processor executing code stored in a non-transitory computer-readable medium, such as a memory. Figures 1-3

[0084] At block 1202, the method 1200 models meshes for independent objects of a synthetic environment. The independent objects are associated with object types, and the meshes are associated with semantic labels. In some implementations, a user manually creates an example room by creating a floor, chairs, a table, walls, etc. for each such room, respectively.

[0085] At block 1204, the method 1200 determines a volumetric representation based on the plurality of meshes. For example, this can involve converting the individual meshes of the independent objects to form a single density cloud or point set.

[0086] At block 1206, the method 1200 determines a combined mesh based on the volumetric representation. For example, this can involve converting the density cloud back to a single mesh.

[0087] At block 1208, the method 1200 adds semantic labels to nodes of the combined mesh based on labels of the meshes for the independent objects. For example, labels from nodes of chair Cl can be used to label similarly positioned (e.g., based on location in 3D space) nodes in the combined mesh.

[0088] At block 1210, the method determines relationships between the independent objects. In some implementations, this involves determining whether a first object of the independent objects is in contact with a second object of the independent objects (e.g., to determine whether the objects are touching, contacting, etc.). In some implementations, this involves determining whether a first location of a first object of the independent objects satisfies a threshold condition based on distance relative to a second location of a second object of the independent objects (e.g., to determine whether the objects are attached, close, next to, etc.). In some implementations, this involves determining whether a first location of a first object of the independent objects satisfies a threshold condition based on orientation relative to a second location of a second object of the independent objects (e.g., to determine whether the objects are facing, above, below, etc.).

[0089] ​In some implementations, determining the relationship involves determining a first bounding box associated with a first object of the independent objects, determining a second bounding box associated with a second object of the independent objects, and determining the relationship by comparing the first bounding box and the second bounding box (e.g., to determine whether one object encloses another object or is inside another object, etc.).

[0090] In some implementations, determining the relationship involves generating a ray based on a location of the first object, determining an intersection of the ray with a second object of the independent objects, and determining the relationship based on the intersection.

[0091] At block 1212, the method 1200 trains a machine learning algorithm using the combined mesh and the relationships. For example, the combined mesh and the relationships can provide synthetic data that can be used to train a machine learning model to classify objects and estimate relationships from semantic meshes using the techniques disclosed herein. In some implementations, the combined mesh and the object relationship graph are used as input and ground truth, respectively, to train a machine learning model. The combined mesh and the estimated relationships can be used for alternative or additional purposes.

[0092] Numerous specific details are set forth herein to provide a thorough understanding of the claimed subject matter. However, those of ordinary skill in the art will understand that the claimed subject matter can be practiced without

[0093] Unless specifically stated otherwise, it will be appreciated that throughout this specification discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

[0094] The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result based at least in part on the input of one or more components. A suitable computing device includes a computer system based on a multiprocessor microcomputer that accesses stored software instructions that program or configure the computing system from a general-purpose computing device into a special-purpose computing device implementing one or more implementations of the subject innovation. The teachings contained herein can be implemented in software, scripting languages or other similar media that is used to program or configure a computing device for a specific purpose. Any suitable programming, scripting, or other type of language or combinations of languages can be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0095] Implementation of the methods disclosed herein can be performed in an operating environment of such computing devices. The order of the blocks presented in the above examples can vary, e.g., blocks can be reordered, combined or separated into sub-blocks. Certain blocks or processes can be performed in parallel.

[0096] The use of “adapted to” or “configured to” herein means open and inclusive language that is not limited to devices that operate in only one particular manner. Additionally, the use of “based on” means open and inclusive language that is not limited to only a single condition or value. The headings, lists, and numbering included herein are for ease of explanation only and are not intended to be limiting.

[0097] It will also be understood that, although the terms “first,” “second,” etc. can be used herein to describe various objects, these

[0098] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, objects, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0099] As used herein, the term "if' can be construed to mean "when" or "in response to determining" or "in response to detecting" that a stated condition precedent has been met, depending on the context. Similarly, the phrase "if it is determined [that a stated condition precedent is true]" or "if [a stated condition precedent is true]" or "when [a stated condition precedent is true]" is construed to mean "in response to determining" or "in response to determining" or "in response to determining" that the stated condition precedent has been met or "in response to detecting" or "in response to detecting" that the stated condition precedent is true, depending on the context.

[0100] The foregoing description and overview of the application should be understood to be illustrative and exemplary, but not limiting, and the scope of the application disclosed herein is not to be determined solely from the detailed description of the illustrative specific embodiments, but according to the full breadth permitted by the patent laws. It is to be understood that the embodiments shown and described herein are merely illustrative of the principles of the present application and that various modifications can be implemented by those skilled in the art without departing from the scope and spirit of the application.

Claims

1. A method for estimating object relationships, comprising: at an electronic device having a processor: generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment, and at least some of the vertices having a semantic label identifying an object type; converting the semantic mesh into a graph representing the semantic mesh, wherein based on determining that a plurality of vertices of the semantic mesh are connected to each other and have a same semantic label, the plurality of vertices of the semantic mesh are combined into one of the nodes of the graph during the conversion; and using the representation of the graph of the semantic mesh to identify relationships between the objects.

2. The method of claim 1, further comprising determining the representation of the graph of the semantic mesh by removing nodes in the graph.

3. The method of claim 2, wherein nodes are removed based on removing edges between nodes having a same semantic label.

4. The method of claim 2, wherein nodes are removed by: determining that a first node and a second node are connected by an edge; determining that the first node and the second node have a same semantic label; generating a combined node by combining the first node and the second node; and merging duplicate edges resulting from combining nodes.

5. The method of claim 4, wherein the combined node identifies an average position of the first node and the second node.

6. The method of claim 4, wherein the combined node identifies a first position of the first node and a second position of the second node.

7. The method of claim 1, wherein at least some of the nodes are semantically labeled as a floor, a table, a chair, a wall, or a ceiling.

8. The method of claim 1, wherein the graph includes edges connecting nodes associated with a same semantic label and edges connecting nodes associated with different semantic labels.

9. The method of claim 1, wherein identifying relationships includes identifying a probability that the objects are associated by the relationship.

10. The method of claim 1, wherein one of the relationships identifies: a first object being on top of a second object; the first object being next to the second object; the first object facing the second object; or the first object being attached to the second object.

11. The method of claim 1, wherein the representation of the graph of the semantic mesh is input to a machine learning model to identify relationships between the objects, and the machine learning model also uses as input: an image of the physical environment; or a pose associated with a viewpoint in the physical environment.

12. The method of claim 1, further comprising providing a graph representing the objects and the relationships.

13. The method of claim 1, further comprising: receiving input to position a virtual object in a computer generated reality (CGR) environment that includes the objects; determining a position of the virtual object in the CGR environment based on the relationships between the objects and the input; and providing the CGR environment.

14. The method of claim 1, further comprising updating object classification labels of nodes of the representation of the graph using a machine learning model.

15. The method of claim 11, wherein the machine learning model is trained using training data, the training data generated by: modeling a plurality of meshes of independent objects for a synthetic environment, the independent objects associated with an object type and the meshes associated with a semantic label; determining a volumetric representation based on the plurality of meshes; determining a combined mesh based on the volumetric representation; determining relationships between the independent objects.

16. A system for estimating object relationships, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment and at least some of the vertices having a semantic label identifying an object type; converting the semantic mesh into a graph representing the semantic mesh, wherein based on determining that a plurality of vertices of the semantic mesh are connected to each other and have a same semantic label, the plurality of vertices of the semantic mesh are combined into one of the nodes of the graph during the conversion; and using the representation of the graph of the semantic mesh to identify relationships between the objects.

17. The system of claim 16, wherein the operations further comprise determining the representation of the graph of the semantic mesh by removing nodes in the graph.

18. The system of claim 17, wherein nodes are removed based on removing edges between nodes having a same semantic label.

19. The system of claim 17, wherein nodes are removed by: determining that a first node and a second node are connected by an edge; determining that the first node and the second node have a same semantic label; generating a combined node by combining the first node and the second node; and merging duplicate edges resulting from combining nodes.

20. A non-transitory computer-readable storage medium storing program instructions executable on a computer by a computer to perform operations comprising: generating a semantic mesh of vertices and faces, the faces representing surfaces of objects of a physical environment and at least some of the vertices having a semantic label identifying an object type; converting the semantic mesh into a graph representing the semantic mesh, wherein based on determining that a plurality of vertices of the semantic mesh are connected to each other and have a same semantic label, the plurality of vertices of the semantic mesh are combined into one of the nodes of the graph during the conversion; and using the representation of the graph of the semantic mesh to identify relationships between the objects. ​

Images