Object relationship estimation from 3D semantic mesh

Abstract

The invention relates to object relationship estimation from a 3D semantic mesh. Implementations disclosed herein provide systems and methods for determining relationships between objects based on an original semantic grid of vertices and faces representing a 3D geometry of a physical environment. Such an original semantic grid may be generated and used to provide input to a machine learning model that estimates relationships between objects in a physical environment. For example, the machine learning model may output a representation of nodes and edges that indicates that the vase is above the table top or a particular instance V1 of the vase is above a particular instance T1 of the table.

Description

[0001] This application is a divisional application of the invention patent application No. 202010939015.X entitled "Estimation of Object Relationships Based on 3D Semantic Mesh", filed on September 9, 2020. Technical Field

[0002] This disclosure relates in general to identifying objects in a physical environment and the relationships between those objects, and more particularly to systems, methods, and apparatuses for using machine learning to estimate relationships and classify objects in a physical environment. Background Technology

[0003] Various computer vision techniques are used to identify physical objects in a physical environment. For example, images captured by a camera can be automatically evaluated to determine if the physical environment includes a table, chair, and vase. However, such techniques can provide little or no information about the relationships between objects. For instance, these techniques may not be able to determine whether the vase is on the tabletop or whether the chair is next to and facing the table. Summary of the Invention

[0004] The specific implementations disclosed herein provide a system and method for determining relationships between objects based on primitive semantic meshes of vertices and faces representing a 3D geometry of a physical environment. Such primitive semantic meshes can be generated and used as input to a machine learning model that estimates relationships between objects in a physical environment. For example, the machine learning model can output a graph of nodes and edges indicating whether a vase is on a table or a specific instance V1 of a vase is on a specific instance T1 of a table.

[0005] Some specific implementations provide a method for estimating or otherwise determining relationships between objects in a physical environment. This exemplary method can be implemented using a computing device that uses a processor to execute instructions. The method generates a semantic mesh of vertices and faces. The 3D shape of the semantic mesh represents the 3D geometry of object surfaces in the physical environment. Furthermore, at least a portion of the vertices has a semantic label identifying the object type (e.g., table, chair, vase, etc.). For example, in such a semantic mesh, a vertex on a floor surface could be labeled “floor,” and a vertex on a chair surface could be labeled “chair.” Adjacent vertices in the semantic mesh form faces; for example, three “floor” vertices could define a face representing a portion of the floor surface. Each vertex can store a location (e.g., x, y, z position information) or otherwise associate with that location.

[0006] One or more machine learning algorithms can be used to generate semantic meshes. In some specific implementations, semantic meshes can be generated based on a set of one or more images of a physical environment, such as using 3D reconstruction techniques (e.g., algorithms or machine learning models) that provide a three-dimensional (3D) triangular mesh representing the surfaces of the physical environment and / or semantic image segmentation techniques (e.g., algorithms or machine learning models) for annotating or classifying objects in the physical environment.

[0007] This method can transform a raw semantic mesh into a graph with nodes and edges determined based on the vertices and faces of the semantic mesh. This graph can be represented in full or simplified form. For example, the method can perform simplification techniques to reduce the complexity of the graph by reducing the number of nodes.

[0008] This method uses one or more machine learning models to estimate relationships between objects. In some implementations, the method inputs a representation of a graph (e.g., a full graph or a simplified version of a graph) into one or more machine learning models (e.g., a neural network). The method can also classify objects or update previously determined object classification labels based on the graph representation and / or estimated relationships using one or more machine learning models. In some implementations, the method classifies objects or updates classification labels. Using a simplified representation of the original semantic grid (e.g., a simplified graph) as input to the machine learning model can improve the accuracy and efficiency of the machine learning model. Using a simplified version as input allows for accurate results using a representation of the semantic grid (e.g., a graph or a simplified graph) instead of using random or all initially connected nodes. In some implementations, the machine learning model outputs a graph representing the objects and their relationships. In some implementations, the machine learning model outputs pairwise relationships between objects.

[0009] In some specific implementations, the relationships between objects generated by the machine learning model are based on probability estimates. For example, it is 99% likely that table T1 is on floor F1, 90% likely that container C1 is on table T1, 75% likely that chair C1 is next to table T1, 75% likely that chair C2 is facing TV1, 75% likely that handle H1 is attached to door D1, and so on.

[0010] Some specific implementations disclosed herein provide systems and methods for generating synthetic data such as semantic grids, which resemble scanned grids of a real physical environment and contain objects with known / labeled relationships. Such synthetic data can be used to train machine learning models to classify objects and determine relationships based on the semantic grids using the techniques disclosed herein and for various other purposes. One exemplary method can be implemented using a computing device that uses a processor to execute instructions. This method models multiple grids (e.g., individual grids for each of chairs, tables, floors, etc.) for independent objects in a synthetic environment. Each of the independent objects is associated with an object type and has an independent grid including semantic labels (e.g., the chair grid includes all nodes labeled "chair"). In some specific implementations, a user creates a synthetic room by individually creating or inserting floors, chairs, tables, walls, etc.

[0011] This method determines a volumetric representation based on multiple meshes, such as all the meshes a user creates for a synthetic room. For example, this could involve transforming the individual meshes of independent objects to form a single density cloud or point set representing all the meshes. The method then determines the composite mesh based on this volumetric representation. For instance, a density cloud generated from multiple individual meshes can be transformed back into a single mesh representing all the objects.

[0012] This method also determines relationships between independent objects within a single mesh. For example, relationships can be determined based on manual input from the creator of the synthetic scene, or automatically based on evaluating individual meshes or a single mesh. Individual meshes, the identification of known synthetic objects represented by the meshes, and / or the relationships between such objects can then be provided to train a machine learning model to classify objects and estimate relationships based on the semantic mesh using the techniques disclosed herein and for various other purposes. In some specific implementations, the machine learning model is trained to receive a semantic mesh (or a simplified version of a semantic mesh) and produce outputs identifying relationships between objects represented in the semantic mesh (or a simplified version of a semantic mesh).

[0013] In some embodiments, a non-transitory computer-readable storage medium stores instructions that are computer-executable to perform or cause to perform any of the methods described herein. In some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. Attached Figure Description

[0014] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0015] Figure 1 It is a block diagram based on some specific implementations of exemplary operating environments.

[0016] Figure 2 It is a block diagram of an exemplary controller based on some specific implementations.

[0017] Figure 3 It is a block diagram of an exemplary electronic device based on some specific implementations.

[0018] Figure 4 This is a flowchart illustrating an exemplary method, according to some specific implementation, for determining relationships between objects in a physical environment using a semantic grid.

[0019] Figure 5 This is a block diagram illustrating a simplified version of an exemplary method for creating a semantic grid.

[0020] Figure 6 This is a block diagram illustrating an exemplary semantic grid.

[0021] Figure 7 This shows, according to some specific implementations Figure 6 A simplified version of the semantic grid diagram.

[0022] Figure 8 This shows, according to some specific implementations Figure 7 A simplified version of the semantic grid diagram.

[0023] Figure 9 This shows, according to some specific implementations Figure 8 A simplified version of the semantic grid diagram.

[0024] Figure 10 This is a block diagram illustrating an exemplary method for determining relationships between objects in a physical environment using a simplified version of a semantic grid, based on some specific implementations.

[0025] Figure 11 Virtual objects are added to the system based on representations of specific implementations. Figure 10 The object relationship diagram is a block diagram representing the CGR environment.

[0026] Figure 12 This is a block diagram illustrating the generation of synthetic data and the use of that synthetic data to train a machine learning model.

[0027] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0028] Numerous details have been described to provide a thorough understanding of the exemplary embodiments shown in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein. Although Figures 1 to 3 Exemplary embodiments involving handheld electronic devices are shown, but other embodiments may not necessarily involve handheld devices and may involve other types of devices, including but not limited to televisions, set-top boxes, laptops, desktop computers, gaming devices, home automation devices, watches, head-mounted devices (HMDs) and other wearable electronic devices, as well as other devices that process or display content.

[0029] Figure 1 This is a block diagram of an exemplary operating environment 100 according to some specific implementation. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary specific implementations disclosed herein. Therefore, as a non-limiting example, operating environment 100 includes a controller 110 and an electronic device 120, one or both of which may be located in a physical environment.

[0030] Electronic device 120 is configured to process or display content. In some implementations, electronic device 120 includes a suitable combination of software, firmware, or hardware. Content may be provided from a recording source or a live streaming source for display on electronic device 120. For example, the content may be stored in memory on electronic device 120, controller 110, or another location. In another example, the content may be a frame stream captured or processed in real time by a camera on electronic device 120, controller 110, or another location. References below Figure 3 The electronic device 120 is described in more detail. In some specific implementations, the functionality of the controller 110 is provided by or combined with the electronic device 120, for example, in the case of the electronic device being used as a stand-alone unit.

[0031] In some implementations, controller 110 is a computing device located locally or remotely relative to physical environment 105. In one example, controller 110 is a local server located within physical environment 105. In another example, controller 110 is a remote server (e.g., a cloud server, central server, etc.) located outside physical environment 105. In some implementations, controller 110 is communicatively coupled to electronic device 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0032] Figure 2 This is a block diagram of an example controller 110 according to some specific implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing cores, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0033] In some embodiments, the one or more communication buses 204 include circuitry for communication between interconnecting system components and control system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image capture devices or other sensors, one or more displays, etc.

[0034] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subsets thereof, including optional operating system 230 and computer vision (CV) and computer generative reality (CGR) modules 240.

[0035] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks.

[0036] In some specific implementations, the CV and CGR modules 240 include a semantic mesh unit 242, a relation classification unit 244, a CGR unit 246, and a training unit 248. The semantic mesh unit 242 can be configured to generate a semantic mesh, for example, using 3D reconstruction techniques (e.g., algorithms or machine learning models) that provide a 3D triangular mesh representing a surface of the physical environment and / or semantic image segmentation techniques (e.g., algorithms or machine learning models) for annotating or classifying objects in the physical environment. The semantic mesh unit 242 can be configured to reduce the number of vertices / nodes in the semantic mesh, for example, by reducing the number of vertices / nodes in the semantic mesh via techniques disclosed herein.

[0037] The relation classification unit 244 can be configured to classify objects and estimate the relationships between objects, for example, using a machine learning model (e.g., a neural network), which uses a representation of the original semantic grid provided by the semantic grid unit 242.

[0038] CGR unit 246 can be configured to render a CGR environment that includes a depiction of a physical environment, virtual content, or both. CGR unit 246 can be configured to utilize objects and / or relationships classified or estimated by relation classification unit 244. For example, CGR unit 246 can respond to a user's voice command to place a virtual vase on a "table next to the window" based on a defined relationship between table T3 and window W2; for example, it knows to place the virtual vase on table T3 rather than on another table not next to the window.

[0039] Training unit 248 can be configured to generate synthetic data, such as semantic grids, which resemble scanned grids of a real physical environment and contain objects with known / labeled relationships. Such synthetic data can be used to train a machine learning model for relationship classification unit 244 to classify objects and estimate relationships based on a semantic grid generated by semantic grid unit 242 from images of the physical environment.

[0040] Although these modules and units are shown residing on a single device (e.g., controller 110), it should be understood that in other specific implementations, any combination of these modules and units may reside in a separate computing device. Furthermore, Figure 2 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware chosen for that particular implementation.

[0041] Figure 3 This is a block diagram of an example of an electronic device 120 according to some specific embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, electronic device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, SPI, I2C, or similar interfaces), one or more programming (e.g., I / O) interfaces 310, one or more displays 312, one or more internal or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0042] In some implementations, one or more communication buses 304 include circuitry for interconnecting and communicating between control system components. In some implementations, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0043] In some embodiments, one or more displays 312 are configured to present a CGR experience to a user. In some embodiments, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS), or similar display types. In some embodiments, one or more displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, electronic device 120 includes a single display. As another example, electronic device 120 includes displays for each of the user's eyes.

[0044] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures, or subsets thereof, including optional operating system 330 and computer vision (CV) and computer generative reality (CGR) modules 340.

[0045] Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks.

[0046] In some specific implementations, the CV and CGR modules 340 include a semantic mesh unit 342, a relation classification unit 344, a CGR unit 346, and a training unit 248. The semantic mesh unit 342 can be configured to generate a semantic mesh, for example, using 3D reconstruction techniques (e.g., algorithms or machine learning models) that provide a 3D triangular mesh representing a surface of the physical environment and / or semantic image segmentation techniques (e.g., algorithms or machine learning models) for annotating or classifying objects in the physical environment. The semantic mesh unit 342 can be configured to reduce the number of vertices / nodes in the semantic mesh, for example, by reducing the number of vertices / nodes in the semantic mesh via techniques disclosed herein.

[0047] The relation classification unit 344 can be configured to classify objects and estimate the relationships between them, for example, using a machine learning model (e.g., a neural network), which uses a representation of the original semantic grid provided by the semantic grid unit 342.

[0048] CGR unit 346 can be configured to render a CGR environment that includes a depiction of the physical environment and / or virtual content. CGR unit 346 can be configured to utilize objects and / or relationships determined by relationship classification unit 344. For example, CGR unit 346 can respond to a user's voice command to place a virtual vase on the "table next to the window" based on a defined relationship between table T3 and window W2.

[0049] Training unit 348 can be configured to generate synthetic data, such as semantic grids, which resemble scanned grids of a real physical environment and contain objects with known / labeled relationships. This synthetic data can be used to train a machine learning model for relationship classification unit 344 to classify objects and estimate relationships based on a semantic grid generated by semantic grid unit 342 from an image of the physical environment.

[0050] also, Figure 3 This is used more as a functional description of various features present in a specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3 Some functional modules shown individually can be implemented in a single module, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of modules and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, or firmware chosen for that particular implementation.

[0051] Figure 4This is a flowchart illustrating an exemplary method for determining relationships between objects in a physical environment using a semantic grid, according to some specific implementations. In some specific implementations, method 400 is performed by a device (e.g., Figures 1 to 3 The method 400 may be executed by a controller 100 or electronic device 120. It may be executed at a television, set-top box, mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices communicating with each other. In some embodiments, the method 400 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 400 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0052] At box 402, method 400 generates a semantic mesh. The 3D shape of the semantic mesh represents the surface of objects in the physical environment, and at least a portion of the vertices of the original semantic mesh have semantic labels that identify the object type (e.g., table, chair, vase, etc.). For example, in such a mesh, vertices on the floor surface can be labeled "floor," and vertices on the chair surface can be labeled "chair."

[0053] In some implementations, generating a semantic mesh involves estimating the depth of different portions (e.g., pixels) of multiple images of the physical environment, associating relationships (e.g., transformations) between the poses of the cameras used to capture the images, and generating a 3D mapping or other reconstruction of surfaces in the physical environment. Such mapping / reconstruction may include the positions of different surface points in a 3D coordinate system. In some implementations, a 3D triangle-based mesh is generated, and each vertex of this mesh stores or otherwise associates with a position (e.g., x, y, z position information). One or more machine learning algorithms may be used to generate the 3D reconstruction.

[0054] In some implementations, generating semantic meshes involves techniques that do not utilize machine learning models. For example, one exemplary method uses an algorithm to project the vertices of a 3D reconstructed mesh onto a semantically segmented image from different viewpoints, and then uses a majority voting technique to determine the final vertex labels.

[0055] In some implementations, semantic information is included or added to the 3D reconstruction of the physical environment. One or more machine learning models can be used to classify objects, including semantic information or adding semantic information to the 3D reconstruction. In some implementations, machine learning models are used to generate semantic labels for pixels of one or more images of the physical environment. Such semantic labels can be associated with corresponding parts of the 3D reconstruction based on determining the association between the image and the 3D reconstruction. In some implementations, the machine learning model uses the 3D reconstruction and semantically labeled images (e.g., automatically labeled images) as input to generate a semantically labeled mesh, such as a semantic mesh with at least some vertices associated with semantic labels.

[0056] At box 404, method 400 converts the semantic mesh into a graph. This may involve converting the vertices of the semantic mesh into nodes and determining the edges that connect nearby nodes to each other. The edges of the graph may represent the sides of the faces of the semantic mesh. Nodes may store information about coordinates (e.g., x, y, z coordinates), semantic labels, and other information extracted from the semantic mesh.

[0057] At box 406, method 400 performs graph simplification and instance detection. Graph simplification can reduce the complexity of the original semantic grid or the graph representing the original semantic grid, for example, by reducing the number of nodes in such a graph. Figure 5 Exemplary diagram simplification techniques are provided. Instance detection involves identifying or estimating individual instances of objects within a semantic mesh or a diagram representing that semantic mesh. In some implementations, a machine learning model is used to perform instance detection, which leverages the semantic and 3D locations of nodes in the diagram to identify instances.

[0058] At box 408, method 400 uses a machine learning model to estimate relationships between objects in a semantic grid or a diagram representing that semantic grid. Examples of relationships include, but are not limited to, estimating that a first object is on top of a second object, a first object is below a second object, a first object is adjacent to a second object, a first object is in contact with a second object, a first object is attached to a second object, a first object leans against a second object, a first object (e.g., a person) holds a second object, a first object is inside a second object, a first object is suspended from a second object, a first object is close to a second object, a first object faces a second object, a first object is away from a second object, a first object is in front of a second object, a first object is behind a second object, and a first object and a second object are aligned with each other.

[0059] In some specific implementations, the relationships between objects generated by the machine learning model are estimated as probabilities. For example, the probability that table T1 is on floor F1 is 99%, the probability that container C1 is on table T1 is 90%, the probability that chair C1 is next to table T1 is 75%, the probability that chair C2 is facing TV1 is 75%, the probability that handle H1 is attached to door D1 is 75%, and so on.

[0060] Additionally, images of the physical environment and / or pose information about the position and orientation of the camera capturing those images can be fed into the machine learning model to further improve its accuracy or efficiency. Camera pose information can be used to generate viewpoint-specific relationships. For example, if a computer-generated reality (CGR) environment is generated using real-time images, the pose (e.g., position and orientation) of the user's current device can be used to facilitate the determination of viewpoint-specific relationships, such as recognizing a chair in front of a table from the user's current viewpoint.

[0061] In some specific implementations, machine learning models output diagrams representing objects and their relationships. Figure 9 An object relationship graph 920, which can be output by method 400, is shown. In some specific implementations, the machine learning model outputs pairwise relationships between objects. The relationships between two or more objects can be represented in any suitable format for output and use.

[0062] At box 408, method 400 uses a machine learning model to update the object classification labels of the vertices of the semantic grid or the nodes representing the graph of the semantic grid. In some implementations, relationships between objects are used as input to update the labels of the vertices of the semantic grid or the nodes representing the graph of the semantic grid. In some implementations, the machine learning model used to update the classification labels is a classifier implemented using a neural network.

[0063] In some implementations, objects and / or relationships estimated in method 400 and other techniques disclosed herein are used to facilitate the provision of a computer-generated reality (CGR) environment. A CGR environment refers to a fully or partially simulated environment sensed and / or interacted with by people via electronic systems. In a CGR, a subset of a person's physical motion, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one physical law. For example, a CGR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of characteristics of virtual objects in the CGR environment may be done in response to a representation of physical motion (e.g., a voice command).

[0064] Humans can use any of their senses to sense and / or interact with CGR objects, including sight, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects that create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, audio objects can enable audio transparency, which selectively introduces ambient sound from the physical environment, with or without computer-generated audio. In some CGR environments, a person can sense and / or interact only with audio objects.

[0065] Examples of CGR include virtual reality and mixed reality. A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of a person's presence within the computer-generated environment, and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0066] Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments are simulated environments designed to incorporate sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between a purely physical environment as one end and a virtual reality environment as the other end, but not including either end.

[0067] In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present MR environments can track position and / or orientation relative to the physical environment, enabling virtual objects to interact with real objects (i.e., physical objects or representations of them from the physical environment). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0068] Examples of mixed reality include augmented reality and augmented virtual. An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are superimposed on the physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects superimposed on the physical environment using the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto a physical environment, such as as a hologram or on a physical surface, so that a person can use the system to perceive the virtual objects superimposed on the physical environment.

[0069] Augmented reality environments also refer to simulated environments where the representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, a system can transform images from one or more sensors to apply a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. Alternatively, the representation of the physical environment can be transformed by graphically modifying (e.g., magnifying) portions of it, such that the modified portion is a representative but not realistic version of the original captured image. Furthermore, the representation of the physical environment can be transformed by graphically removing or blurring portions of it.

[0070] Augmented virtual (AV) environments are simulated environments in which a virtual or computer-generated environment is combined with one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more features of the physical environment. For example, an AV park could have virtual trees and virtual buildings, but a person's face could be realistically reproduced from an image taken of a physical person. Similarly, virtual objects could adopt the shape or color of a physical object imaged by one or more imaging sensors. Furthermore, virtual objects could adopt shadows that correspond to the sun's position within the physical environment.

[0071] Many different types of electronic systems enable people to sense and / or interact with a variety of CGR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0072] In some implementations, object classification and / or relationship estimation generated in method 400 and other techniques disclosed herein are used to facilitate the provision of a computer-generated reality (CGR) environment. For example, during the presentation of the CGR environment, input can be received to position virtual objects within the CGR environment, which includes objects classified and estimated relationships using method 400 and other techniques disclosed herein. For example, a user can provide a voice command to “place the virtual gyroscope on a table against the wall.” The CGR unit providing the CGR environment (e.g., Figure 2 CGR unit 246 or Figure 3 The CGR unit 346 can determine the position of a virtual object in the CGR environment based on the input and the relationships between objects provided by method 400 or other techniques disclosed herein. For example, it can identify table T1, rather than table T2, for automatically placing a virtual gyroscope based on the existence of the relationship "table T1 is adjacent to wall W1," which best matches the input "table adjacent to wall." The virtual object (e.g., the virtual gyroscope) can then be automatically positioned in the appropriate location in the CGR environment for the user experience.

[0073] In some implementations, the semantic grid, or the diagram representing the semantic grid, is simplified to have fewer vertices / nodes. Figure 5 This is a block diagram illustrating a simplified version of an exemplary method for creating the illustration. In some specific implementations, method 500 is provided by a device (e.g., Figures 1-3 The method 500 may be executed by a controller 100 or electronic device 120. The method 500 may be executed at a mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices communicating with each other. In some embodiments, the method 500 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 500 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0074] At box 502, method 500 involves determining that two nodes are connected by an edge. In some implementations, this can be achieved by inspecting or traversing a semantic mesh. At box 504, method 500 determines that two nodes have the same semantic label, for example, both are labeled "chair," etc. At box 506, method 500 generates a combined node by combining the two nodes. Any other node previously connected to either of the two source nodes is instead connected to the combined node. Furthermore, the combined node may store the average position of the first and second nodes and / or the positions of the individual source nodes (e.g., the first position of the first node and the second position of the second node) or otherwise associate with these positions. At box 508, method 500 merges duplicate edges caused by the combined node.

[0075] Figures 6 to 9 A simplified exemplary semantic grid is shown. Figure 6 This is a block diagram illustrating an exemplary semantic mesh 600. The semantic mesh 600 includes nodes 610a-h, 620a-r, and 630a-i connected to each other by edges represented by dashed lines between nodes 610a-h, 620a-r, and 630a-i. In this example, node 610a-h is semantically labeled "cup," node 620a-r is semantically labeled "table," and node 630a-i is semantically labeled "floor." In typical use cases in actual physical settings, additional objects and nodes will typically be involved, and the semantic mesh will occupy 3D coordinate space. Figure 6 In the example, for ease of illustration, the semantic mesh 600 includes only a limited number of nodes and edges depicted in two dimensions.

[0076] Figure 7 It is shown Figure 6 A simplified block diagram of the semantic mesh 600. In this example, Figure 6Nodes 610a-h that are semantically labeled "cup" are combined into a single composite node 710a. These nodes 610a-h are combined based on the fact that they are connected to each other and have the same label. Thus, edges between nodes 610a-h are eliminated. The composite node 710a is assigned the semantic label "cup" based on the label of the source node and is assigned a position based on the position of the source node (e.g., average position).

[0077] Figure 8 This demonstrates further simplification. Figure 7 A block diagram of the semantic grid. In this example, Figure 6 and Figure 7 Nodes 630a-i that are semantically labeled as "floor" are combined into a single composite node 730a. These nodes 630a-i are combined based on the fact that they are connected to each other and have the same label. Thus, edges between nodes 630a-i are eliminated. The composite node 730a is assigned the semantic label "floor" based on the label of the source node and is assigned a position based on the position of the source node (e.g., average position).

[0078] Figure 9 This demonstrates further simplification. Figure 8 A block diagram of the semantic grid. In this example, Figures 6 to 8 Nodes 620a-r that are semantically labeled "table" are combined into a single composite node 720a. These nodes 620a-r are combined based on the fact that they are connected to each other and have the same label. Thus, edges between nodes 620a-r are eliminated. The composite node 720a is assigned the semantic label "table" based on the label of the source node and a position based on the position of the source node (e.g., average position).

[0079] Figure 10 This is a block diagram illustrating an exemplary method 900 for determining relationships between objects in a physical environment using a simplified version 905 of a semantic mesh. In this example, the semantic mesh is represented by a simplified version of the semantic mesh, which includes... Figure 9 The semantic mesh includes combination nodes 710a, 720a, and 730a, as well as an additional combination node 740a included for illustrative purposes. In this example, a simplified version 905 of the semantic mesh is input to a machine learning model 910 that generates an object relation graph 920. In some embodiments, the machine learning model 910 is a trained neural network to generate an object relation graph given a simplified semantic mesh input. In some embodiments, the machine learning model 910 is a convolutional neural network (CNN). In some embodiments, the machine learning model 910 involves recursive learning via feedback loops, for example, including a recurrent neural network (RNN).

[0080] In this example, object relationship diagram 920 includes a first object 930, a second object 940, a third object 950, and a fourth object 950. The first object 930 has a type label "cup" and an instance label "C1". The second object 940 has a type label "table" and an instance label "T1". The third object 950 has a type label "chair" and an instance label "CH1". The fourth object 960 has a type label "floor" and an instance label "F1". Furthermore, object relationship diagram 920 identifies relationships 935, 945, 955, and 965 between objects 930, 940, 950, and 960. The first relationship 935 identifies that the first object 930 is "above" the second object 940. The second relationship 945 identifies that the second object 940 is "adjacent" to the third object 950. The third relationship 955 identifies that the second object 940 is "above" the fourth object 960. The fourth relationship 965 identifies that the third object 950 is "above" the fourth object 960. It should be noted that the edges and nodes shown in the object relationship diagram 920 are not limited to holding only one label—each node and edge can generate two, three or more labels.

[0081] Figure 11 This indicates adding a virtual object to the... Figure 10 The object relationship graph is a block diagram illustrating the CGR environment. In this example, the user provides a text or verbal command to “add a virtual chair next to the table.” Based on this command, a second object 940 (table T1) is identified as the table mentioned in the command. The virtual chair is automatically added to the CGR environment based on this command, and the object relationship graph 1100 is updated to reflect this addition, including the relationships determined based on the command. Specifically, the command identifies that the virtual chair should be positioned “next to” the table. Based on this, an automatic position is selected in the CGR environment (e.g., using a rule that defines, for example, “next to” triggers a placement distance of 6 inches from the object). Furthermore, the object relationship graph 1100 is expanded to include a new object 1130 (e.g., a virtual chair with the type label “chair” and the instance label “VCH1”). New relationships are also identified and included in the object relationship graph 1100. Specifically, a fifth relationship 1135 identifies the new object 1130 as being next to the second object 940, and a sixth relationship identifies the new object 1130 as being “above” the fourth object 960. Such relationships can be determined based on commands (for example, the fifth relationship 1135 is determined based on the command "adjacent"). Such relationships can also be determined based on algorithms, rules, or machine learning. In this example, the sixth relationship 1145 is determined based on a rule specifying that if a virtual object is added adjacent to an object "above" the floor, then by default, that virtual object will also be considered "above" the floor.

[0082] Figure 12This is a block diagram illustrating a method 1200 for generating synthetic data and using that synthetic data to train a machine learning model. In some specific implementations, method 1200 is performed by a device (e.g., Figures 1-3 The method 1200 may be executed by a controller 100 or electronic device 120. It may be executed at a television, set-top box, mobile device, HMD, desktop computer, laptop computer, server device, or by multiple devices communicating with each other. In some embodiments, the method 1200 is executed by processing logic components (including hardware, firmware, software, or a combination thereof). In some embodiments, the method 1200 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0083] At box 1202, method 1200 models a mesh of independent objects for a synthetic environment. Independent objects are associated with object types, and the mesh is associated with semantic labels. In some implementations, users manually create exemplary rooms by creating floors, chairs, tables, walls, etc., separately for each such room.

[0084] At box 1204, method 1200 determines the volume representation based on multiple grids. For example, this might involve transforming the individual grids of an independent object to form a single density cloud or point set.

[0085] At box 1206, method 1200 determines the composite mesh based on the volume representation. For example, this might involve converting density clouds back into a single mesh.

[0086] At box 1208, method 1200 adds semantic labels to nodes of the composite mesh based on the labels of the meshes used for individual objects. For example, the label of a node from chair C1 can be used to mark nodes in the composite mesh that are similarly positioned (e.g., based on their location in 3D space).

[0087] At box 1210, the method determines relationships between independent objects. In some implementations, this involves determining whether a first object in the independent objects is in contact with a second object in the independent objects (e.g., to determine whether the objects are touching, contacting, etc.). In some implementations, this involves determining whether a first position of the first object in the independent objects satisfies a distance-based threshold condition relative to a second position of the second object in the independent objects (e.g., to determine whether the objects are attached, close, adjacent, etc.). In some implementations, this involves determining whether a first position of the first object in the independent objects satisfies an orientation-based threshold condition relative to a second position of the second object in the independent objects (e.g., to determine whether the object is facing, above, below, etc.).

[0088] In some specific implementations, the relationship is determined by the following operations: determining a first bounding box associated with a first object in an independent object, determining a second bounding box associated with a second object in an independent object, and determining the relationship by comparing the first bounding box and the second bounding box (e.g., to determine whether one object surrounds another object or is inside another object, etc.).

[0089] In some implementations, determining the relationship involves generating a ray based on the position of a first object, determining the intersection of the ray with a second object in the independent objects, and determining the relationship based on the intersection.

[0090] At box 1212, method 1200 uses labeled composite meshes and relations to train a machine learning algorithm. For example, the composite meshes and relations can provide synthetic data that can be used to train a machine learning model to classify objects based on semantic meshes and estimate relations using the techniques disclosed herein. In some implementations, the composite meshes and object relation graphs are used as input and ground reality, respectively, to train the machine learning model. The composite meshes and estimated relations can be used for alternative or additional purposes.

[0091] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill in the art have not been described in detail so as not to obscure the claimed subject matter.

[0092] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “computing,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0093] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein can be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0094] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the above examples can be changed; for example, the boxes can be reordered, grouped, or divided into sub-boxes. Some boxes or procedures can be executed in parallel.

[0095] The use of “applies to” or “configured to” in this document implies open and inclusive language, which does not exclude applicability to or configuration to devices performing additional tasks or steps. Similarly, the use of “based on” implies openness and inclusivity, as processes, steps, calculations, or other actions “based on” one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included in this document are for illustrative purposes only and are not intended to be restrictive.

[0096] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various objects, these objects should not be limited by these terms. These terms are merely used to distinguish one object from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, objects, or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0098] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when the prerequisite is detected" or "in response to detection" that the prerequisite is true, depending on the context.

[0099] The foregoing description and summary of the present invention should be understood as illustrative and exemplary in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the illustrative specific embodiments, but also by the full extent permitted by patent law. It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and various modifications can be made by those skilled in the art without departing from the scope and spirit of the invention.

Claims

1. A method for estimating object relationships, comprising: at an electronic device having a processor: generating a mesh of vertices and faces, the faces representing surfaces of objects in a physical environment, and at least some of the vertices having semantic labels identifying object types; converting the mesh into a relationship model representing the mesh, wherein based on determining that multiple vertices of the mesh are associated with a same semantic label, the multiple vertices of the mesh are combined during the conversion; and using the relationship model to identify relationships between the objects.

2. The method of claim 1, wherein individual nodes in the relationship model represent multiple vertices of the mesh.

3. The method of claim 2, wherein the individual nodes identify average positions of multiple vertices of the mesh.

4. The method of claim 2, wherein the individual nodes identify individual identified positions of multiple vertices of the mesh.

5. The method of claim 2, wherein at least some of the individual nodes are semantically labeled as a floor, a wall, or a ceiling.

6. The method of claim 2, wherein at least some of the individual nodes are semantically labeled as a table or a chair.

7. The method of claim 1, wherein the relationship model includes edges connecting nodes associated with a same semantic label and edges connecting nodes associated with different semantic labels.

8. The method of claim 1, wherein identifying relationships includes identifying probabilities that the objects are associated through the relationships.

9. The method of claim 1, wherein one of the relationships identifies: a first object being on top of a second object; or the first object being next to the second object.

10. The method of claim 1, wherein one of the relationships identifies: a first object facing the second object; or the first object being attached to the second object.

11. The method of claim 1, wherein a representation of the relationship model is input to a machine learning model to identify relationships between the objects, and the machine learning model also uses as input: an image of the physical environment; or a pose associated with a viewpoint in the physical environment.

12. The method of claim 1, further comprising providing a relationship model representing the objects and the relationships.

13. The method of claim 1, further comprising: receiving input to position a virtual object in a computer-generated reality (CGR) environment including the objects; determining a position of the virtual object in the CGR environment based on the relationships between the objects and the input; and providing the CGR environment.

14. The method of claim 1, further comprising updating object classification labels for portions of the relationship model using a machine learning model.

15. The method of claim 11, wherein the machine learning model is trained using training data, the training data generated by: modeling a plurality of meshes for independent objects of a synthetic environment, the independent objects being associated with an object type and the meshes being associated with a semantic label; determining a volumetric representation based on the plurality of meshes; determining a combined mesh based on the volumetric representation; determining relationships between the independent objects.

16. A system for estimating object relationships, comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: generating a mesh of vertices and faces, the faces representing surfaces of objects in a physical environment and at least some of the vertices having a semantic label identifying an object type; converting the mesh into a relationship model representing the mesh, wherein a plurality of vertices of the mesh that are determined to be associated with a same semantic label are combined during the conversion; and using the relationship model to identify relationships between the objects.

17. The system of claim 16, wherein individual nodes in the relationship model: represent a plurality of vertices of the mesh; identify an average position of a plurality of vertices of the mesh or individual positions of a plurality of vertices of the mesh; and are semantically labeled.

18. The system of claim 16, wherein identifying a relationship comprises identifying a probability that the objects are associated through the relationship.

19. The system of claim 16, wherein one of the relationships identifies: a first object being on top of a second object; the first object being next to the second object; the first object facing the second object; or the first object being attached to the second object.

20. A non-transitory computer-readable storage medium storing program instructions executable on a computer by a computer to perform operations comprising: generating a mesh of vertices and faces, the faces representing surfaces of objects in a physical environment and at least some of the vertices having a semantic label identifying an object type; converting the mesh into a relationship model representing the mesh, wherein a plurality of vertices of the mesh that are determined to be associated with a same semantic label are combined during the conversion; and using the relationship model to identify relationships between the objects.

Images