Identifying objects using spatial ontology

Abstract

In an example process, a representation of a setting is obtained. A first object is detected based on the representation of the setting, and a set of identifiers is retrieved. In response to detecting a change in the representation of the setting, the set of identifiers is modified based on a first set of definitions. User input associated with the representation of the setting is received, and a user intent corresponding to the user input is determined based on the modified set of identifiers.

Description

Background Technology

[0001] 1. Cross-references to related applications

[0002] This application relates to U.S. Provisional Application No. 63 / 155,081, filed March 1, 2021, entitled “IDENTIFYING OBJECTS USING SPATIALONTOLOGY,” the contents of which are incorporated herein by reference in their entirety for all purposes. 2. Technical Field

[0003] This disclosure relates generally to extended reality, and more specifically to techniques for identifying objects using one or more spatial ontologies.

[0004] 3. Description of related fields

[0005] Traditional extended reality environments can include various representations of virtual and physical objects. Users observing this environment can interact with the objects in different ways. Summary of the Invention

[0006] This disclosure generally describes techniques for identifying various objects in an environment. Extended reality environments provide a platform that enables users to interact with the environment. For example, users can provide input related to physical or virtual objects depicted in the environment using methods including physical controls, voice commands, gaze-based operations, etc. In the case of voice commands, users can mention various objects depicted in the environment, such as furniture, walls, appliances, or other objects. However, traditional extended reality systems do not have the ability to efficiently identify objects in a scene in order to respond to user requests in an effective manner. For example, such systems do not include a consistent set of object identifiers, let alone other characteristics that define the relationships between objects and attached object attributes. Therefore, improved systems for object identification using one or more spatial ontology are desired.

[0007] According to some implementation schemes, a representation of a scene is obtained. A first object is detected based on the representation of the scene, and a set of identifiers is retrieved. In response to detecting a change in the representation of the scene, the identifier set is modified based on a first set of definitions. User input associated with the representation of the scene is received, and a user intent corresponding to the user input is determined based on the modified identifier set. Attached Figure Description

[0008] Figures 1A to 1B An exemplary system used in various extended reality technologies is shown.

[0009] Figures 2A to 2C An exemplary process for identifying objects using one or more spatial ontology is described.

[0010] Figures 3A to 3C An exemplary process for identifying objects using one or more spatial ontology is described.

[0011] Figure 4 An exemplary process for identifying objects using one or more spatial ontology is described. Detailed Implementation

[0012] People can perceive or interact with the physical environment or world without using electronic devices. Physical features, such as physical objects or surfaces, can be included within the physical environment. For example, a physical environment may correspond to a physical city with physical buildings, roads, and vehicles. People can directly perceive or interact with the physical environment through various means such as smell, sight, taste, hearing, and touch. This contrasts with extended reality (XR) environments, which can refer to partially or entirely simulated environments that people can perceive or interact with using electronic devices. XR environments can include virtual reality (VR) content, mixed reality (MR) content, augmented reality (AR) content, and so on. Using an XR system, physical movement of a person or a portion of their representation can be tracked, and in response, the properties of virtual objects in the XR environment can be altered in a manner that conforms to at least one law of nature. For example, an XR system can detect head movement of a user and adjust the auditory and graphical content presented to the user in a way that simulates how sound and view would change in a physical environment. In other examples, an XR system can detect movement of electronic devices (e.g., laptops, tablets, mobile phones, etc.) presenting the XR environment. Therefore, XR systems can adjust the auditory and graphical content presented to the user in a way that simulates how sound and view change in a physical environment. In some instances, other inputs, such as representations of physical motion (e.g., voice commands), can cause the XR system to adjust the properties of the graphical content.

[0013] Numerous types of electronic systems allow users to perceive or interact with an XR environment. An incomplete list of examples includes lenses with integrated display capabilities placed on the user's eyes (e.g., contact lenses), head-up displays (HUDs), projection-based systems, head-mounted systems, windows or windshields with integrated display technology, headsets / earpieces, input systems with or without haptic feedback (e.g., handheld or wearable controllers), smartphones, tablets, desktop / laptop computers, and speaker arrays. Head-mounted systems may include opaque displays and one or more speakers. Other head-mounted systems may be configured to receive from opaque external displays, such as the display of a smartphone. Head-mounted systems may use one or more image sensors to capture images / video of the physical environment, or one or more microphones to capture audio of the physical environment. Instead of opaque displays, some head-mounted systems may have transparent or semi-transparent displays. Transparent or semi-transparent displays can guide light representing an image toward the user's eyes through a medium such as a holographic medium, optical waveguide, optical combiner, optical reflector, other similar technologies, or combinations thereof. Various display technologies can be used, such as liquid crystal on silicon, LED, uLED, OLED, laser scanning light sources, digital light projection, or combinations thereof. In some examples, transparent or translucent displays can be selectively controlled to become opaque. Projection-based systems can utilize retinal projection techniques that project images onto a user's retina or project virtual content into a physical environment (such as onto a physical surface or as a hologram).

[0014] Figure 1A and Figure 1B An exemplary system 100 for various extended reality technologies is depicted.

[0015] like Figure 1A As shown, system 100 includes device 100a. Device 100a includes RF circuitry 104, processor 102, memory 106, image sensor 108, touch-sensitive surface 122, speaker 118, position sensor 116, microphone 112, orientation sensor 110, and display 120. These components optionally communicate using communication bus 150 of device 100a.

[0016] In some examples, a base station device (e.g., a computing device, such as a remote server, mobile device, or laptop) implements some components of system 100, and a second device (e.g., a head-mounted device) implements other components of system 100. In some examples, device 100a is implemented in either the base station device or the second device.

[0017] like Figure 1BAs shown, in some examples, system 100 includes two or more devices that communicate, for example, via a wired or wireless connection. A first device 100b (e.g., a base station device) includes a memory 106, RF circuitry 104, and a processor 102. Such components optionally communicate using a communication bus 150 of device 100b. A second device 100c (e.g., a head-mounted device) includes components such as RF circuitry 104, processor 102, memory 106, image sensor 108, touch-sensitive surface 122, speaker 118, position sensor 116, microphone 112, orientation sensor 110, and display 120. These components optionally communicate using a communication bus 150 of device 100c.

[0018] System 100 includes RF circuitry 104. RF circuitry 104 optionally includes circuitry for communicating with networks (e.g., the Internet, wireless networks (e.g., such as cellular networks and wireless local area networks (LANs)) and / or intranets) and / or electronic devices. RF circuitry 104 optionally includes circuitry for using near-field communication and / or short-range communication (e.g., ...). The circuit used for communication.

[0019] System 100 includes a processor 102 and a memory 106. The processor 102 includes one or more graphics processors, one or more general-purpose processors, and / or one or more digital signal processors. In some examples, the memory 106 is one or more non-transitory computer-readable storage media (e.g., random access memory, flash memory) configured to be executed by the processor 102 to perform computer-readable instructions described below.

[0020] System 100 includes an image sensor 108. Image sensor 108 optionally includes one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the physical environment. For example, an active IR sensor includes an IR emitter (e.g., an IR point emitter) for emitting infrared light into the physical environment. Image sensor 108 also optionally includes one or more visible light image sensors, such as complementary metal-oxide-semiconductor (CMOS) sensors and / or charge-coupled device (CCD) sensors, capable of acquiring images of physical elements from the physical environment. Image sensor 108 also optionally includes one or more event cameras configured to capture movement of physical elements in the physical environment. Image sensor 108 also optionally includes one or more depth sensors capable of detecting distances to physical elements of system 100. In some examples, system 100 uses the IR sensor, CCD sensor, event camera, and depth sensor together to detect the physical environment surrounding system 100. In some examples, image sensor 108 includes a first image sensor and a second image sensor. The first and second image sensors are optionally capable of capturing images of physical elements in the physical environment from two corresponding different perspectives. In some examples, system 100 uses image sensor 108 to detect the position and orientation of system 100 and / or display 120 in a physical environment. For example, system 100 uses image sensor 108 to track the position and orientation of display 120 relative to one or more fixed elements in the physical environment. In some examples, image sensor 108 is capable of receiving user input, such as gestures.

[0021] In some examples, system 100 includes a touch-sensitive surface 122 for receiving user input such as tap and swipe input. In some examples, the touch-sensitive surface 122 and display 120 are combined to form a touch-sensitive display.

[0022] In some examples, system 100 includes a microphone 112. System 100 uses microphone 112 to detect physical environment from a user or sound from a user. In some examples, microphone 112 includes a microphone array (e.g., including multiple microphones) that optionally operates together, for example, to locate spatial sound sources from the physical environment or identify ambient noise.

[0023] System 100 includes an orientation sensor 110 for detecting the orientation and / or movement of system 100 and / or display 120. For example, system 100 uses orientation sensor 110 to track changes in the position and / or orientation of system 100 and / or display 120 (such as relative to physical elements in the physical environment). Orientation sensor 110 optionally includes a gyroscope and / or accelerometer.

[0024] System 100 includes a display 120. Display 120 may operate in conjunction with a transparent or translucent display (and optionally with one or more imaging sensors). Display 120 may include an opaque display. Display 120 may allow a person to directly observe the physical environment through the display, and may also allow virtual content to be added to the person's field of view, for example, by overlaying virtual content onto the physical environment. Display 120 may implement display technologies such as digital light projectors, laser scanning light sources, LEDs, OLEDs, liquid crystal on silicon, or combinations thereof. Display 120 may include a substrate through which light transmits, such as optical reflectors and combiners, optical waveguides, holographic substrates, or combinations thereof. As a particular example, the transparent or translucent display may selectively transition between a transparent or translucent state and an opaque state. Other exemplary embodiments of display 120 include a display-capable lens, tablet computer, smartphone, desktop computer, laptop computer, head-up display, display-capable automotive windshield, or display-capable window. In some examples, system 100 is a projection-based system. For example, system 100 projects virtual objects onto a physical environment (e.g., projects a hologram onto a physical environment or projects an image onto a physical surface). As another example, system 100 uses retinal projection to project an image onto a person's eye (e.g., the retina). In some examples, system 100 may be configured to interface with an external display (e.g., a smartphone display).

[0025] System 100 may also include one or more speech-to-text (STT) processing modules, each of which includes one or more automatic speech recognition (ASR) systems for performing speech-to-text conversion on speech received from various microphones. Each ASR system may include one or more speech recognition models and may implement one or more speech recognition engines. Examples of speech recognition models may include, but are not limited to, deep neural network models, n-gram language models, hidden Markov models (HMMs), Gaussian mixture models, etc. The natural language processing module may also obtain candidate text representations of the speech input and associate each of these candidate text representations with one or more recognizable "actionable intentions". In some examples, natural language processing is based on the use of ontology. An ontology is a hierarchical structure containing many nodes, each node representing an actionable intention related to other actionable intentions. These actionable intentions may represent tasks that the system can perform. The ontology may also include attributes representing parameters associated with actionable intentions, sub-aspects of other attributes, etc. The links between actionable intention nodes and attribute nodes in the ontology may define how the parameters represented by the attribute nodes are related to the tasks represented by the actionable intention nodes.

[0026] For reference Figures 2A to 4This describes an exemplary technique for identifying objects using one or more spatial ontology. Figure 2A A representation 200a depicts the surrounding environment corresponding to an electronic device (such as, for example, device 100a). Representation 200a may be based on a user's perspective, for example, based on one or more cameras that receive image information and are attached to the user's headset or other user-controlled devices. Generally, the environment may correspond to a physical environment, a virtual environment, an AR environment with virtual objects superimposed on a physical environment, etc. The environment may include various physical objects, such as tables, shelves, sofas, walls, pictures, and lamps. Image information may include representations of physical objects and may also include representations of virtual objects. For example, representation 200a may include sofa representation 202, lamp representation 204, vase representation 206, and table representation 208. Sofa representation 202, vase representation 206, and table representation 208 may correspond to physical sofa, vase, and table objects in the surrounding device environment, respectively. In some examples, lamp representation 204 may correspond to a virtual object such that a corresponding physical vase does not exist in the surrounding device environment. In an alternative example, representation 200a may also correspond to a completely virtual scene, such that sofa representation 202, lamp representation 204, vase representation 206, and table representation 208 each correspond to a representation of a virtual object.

[0027] Initially, a set of definitions can be used to identify various objects in the representation of a scene. As discussed herein, this reactive form of object identification can be continuous and dynamic as the representation of the scene changes. For example, generally, when activating an electronic device (e.g., activating an image sensor on the device), loading a virtual or AR environment, etc., a set of identifiers is obtained based on the set of definitions and the representation of the corresponding scene. Specifically, as discussed herein, the set of definitions may include basic definitions of descriptive identifiers. These definitions may include object profiles for identifying corresponding objects in the environment, such as profiles that include general object characteristics. Object profiles typically semantically represent objects, specifically by including defined features for object identification of the corresponding object, such as general object shape, size, appearance, structure, color scheme, etc. Thus, sofa representation 202 can be identified using the set of definitions based on the general object profile of "sofa" contained in the set of definitions. Light fixture representation 204, vase representation 206, and table representation 208 can each be similarly identified using general object profiles within the set of definitions (such as those object profiles corresponding to "light fixture," "vase," and "table," respectively).

[0028] For reference Figure 2BChanges in the representation of the setting can be detected based on various factors. For example, one or more objects in the surrounding environment may move from a first position to a second position. Physical objects can be added to the environment (e.g., by the user of the electronic device or another user), such as placing one or more cushions on a sofa. Therefore, one or more objects can be detected based on this representation. In this example, in representation 200b (relative to...) Figure 2A In representation 200a), one or more cushion representations are detected, such as cushion representation 210 placed on top of sofa representation 202. One or more virtual objects may also be populated into the representation by the user, another user associated with the environment, and extended reality applications, etc.

[0029] For reference Figure 3A This describes a set of identifiers. Generally, identifier sets can be used to enhance the user's experience of an extended reality environment. The identifier set can represent various physical and virtual objects detected in the environment and can be dynamically updated based on changes in the scene's representation. The identifier set, which may be called a scene graph, can also be generated based on an underlying spatial ontology. As discussed in this paper, the spatial ontology representation acts as a definition set of an object feature library used to generate a scene graph based on the scene's representation. For example, a spatial ontology may include object names (e.g., chair, table, lamp) and also category definitions for such objects (e.g., furniture, structure, creature, etc.). Category definitions define the parent object category to which an object belongs. Specifically, the association "chair is furniture" indicates that the chair object is within the furniture object category.

[0030] The identifier set may include descriptive identifiers associated with the detected objects. Descriptive identifiers typically identify the type associated with the detected objects. For example, such identifiers may include one or more string constants included within a header file. Here, a sofa representation 202 is detected, such that identifier 302 is generated to include the descriptive identifier "sofa". A lamp representation 204 is detected, such that identifier 304 is generated to include the descriptive identifier "lamp". Where sofa representation 202 corresponds to a physical object in the environment, identifier 302 may also include reality-type characteristics such as "physical". Similarly, where lamp representation 204 corresponds to a virtual object, identifier 304 may also include reality-type characteristics such as "virtual". Identifiers 306 and 308 may correspond to the descriptive identifiers "vase" and "table", respectively. Identifiers 302 ("sofa") and 308 ("table") may include parent identifiers corresponding to object categories (such as "furniture"). Identifier 304 ("lamp") may include a parent identifier corresponding to "electronic device". For example, identifier 306 (“vase”) may include a parent identifier corresponding to “household”.

[0031] Relationship identifiers typically identify relationships between corresponding objects detected from a scene representation. A relationship identifier for a first detected object can identify a second detected object and the position of the first detected object relative to the second detected object. As an example, based on the user's perspective, sofa representation 202 can be positioned to the right of lighting fixture representation 204 based on the perspective of a device (e.g., one or more cameras or sensors on the device) and the resulting scene representation. Therefore, identifier 302 (“sofa”) can be associated with relationship identifier 312 (via identifier 304, “lighting fixture”) that identifies lighting fixture representation 204 and further identifies the position “to the right of”. Specifically, based on the device's perspective, this position indicates that sofa representation 202 is “to the right of” lighting fixture representation 204. Similarly, based on the scene representation, lighting fixture representation 204 is positioned to the left of sofa representation 202. Therefore, identifier 304 (“lighting fixture”) can be associated with relationship identifier 314 (via identifier 302, “sofa”) that identifies sofa representation 202 and further identifies the position “to the left of”. Specifically, based on the device's perspective, the position indicator 204 is "to the left" of the sofa 202. Additional relational identifiers associated with identifiers 302, 304, 306, and 308 (such as "in front of", "behind", "on top of") may be included within the identifier set.

[0032] In some examples, relation identifiers can define additional relationships between detected objects. Various objects in a virtual environment can be contained within other objects. For example, a virtual box in the environment can include various objects such as virtual books, virtual toys, virtual electronics, etc. A user can place virtual objects, such as virtual books, into a virtual box. Based on detected object placement (e.g., parameters of the virtual scene indicating that the user has previously placed virtual books into a virtual box), relation identifiers can be generated to identify the detected virtual book object as contained within the detected virtual box object, such as "the book is inside the box". Various objects can also be generated from other objects. For example, a user can participate in an extended reality game, causing a virtual chicken object to be included in the environment. Within the context of the extended reality game, the virtual chicken object can generate a virtual egg object. In this example, relation identifiers can be generated to identify the detected egg object as generated from the detected chicken object, such as "the egg comes from the chicken". In some examples, a user can move physical objects relative to another physical object, such as moving a book on top of a box. Here, the relation identifier can be generated to identify that the detected book object is placed on top of the box after being moved, such as "books 'on top of box'".

[0033] Generally, attributes associated with a detected object can be identified using various additional identifiers. Specifically, a material identifier can identify the type of material constituting the detected object. For example, identifier 302 (“sofa”) can be associated with a material identifier corresponding to “fabric”. Identifier 308 (“table”) can be associated with a material identifier corresponding to “wood”. Multiple material identifiers can also be associated with corresponding objects. For example, identifier 304 (“light fixture”) can include material identifiers corresponding to “metal”, “glass”, “wiring”, “plastic”, etc. Other attributes representing the appearance characteristics of an object can be defined using additional identifiers. One or more numerical attributes, such as those related to size, volume, weight, etc., can be used. Such attributes can also include coefficients associated with the object. For example, one or more coefficients associated with a detected object include specific values, for example, between 0 and 100. Illumination coefficients can indicate various light properties of the detected object, such as gloss, brilliance, luster, brightness, radiation, etc. Various other coefficients can be associated with object characteristics such as object state (e.g., liquid, solid, gas), flexibility, sharpness, roughness, temperature, odor, weight, fragility, etc.

[0034] Acoustic coefficients indicate various acoustic properties of objects in an environment. For example, acoustic coefficients can be used to provide a user with an audible experience consistent with the currently observed environment. Generally, the acoustic output provided to the user can be modified as the user moves through the corresponding environment. For instance, although the user may remain physically still while moving through a virtual environment, the sound provided to the user may change as the user moves through the virtual environment. Therefore, the sound provided to the user can be customized to provide an acoustic experience consistent with the virtual environment.

[0035] Come back for reference Figure 2B The device can detect changes in the representation of the scene. Specifically, an object previously not in the device's view may come into view, causing the representation to now include that object. For example, one or more physical cushion objects may come into view, causing representation 200b to include cushion representation 210. In response to detecting this change in the representation of the scene, the identifier set may be modified at least in part based on a set of definitions (e.g., a spatial ontology). Generally, a spatial ontology may include basic definitions of descriptive identifiers, relational identifiers, material identifiers, and other identifiers corresponding to additional object attributes. The basic definitions of descriptive identifiers may include object profiles for identifying the corresponding objects in the environment, such as profiles that include general object characteristics. Object profiles may include defining characteristics of the corresponding objects for object identification, such as general object shape, size, appearance, structure, color scheme, etc. The spatial ontology may be generated at least in part using an ontology (e.g., a natural language processing ontology) that has the definition of the object when it is typically mentioned by the user. For example, a flat, rectangular wooden object with four legs is typically referred to as a "table".

[0036] As an example, the spatial ontology may include a basic definition of a cushion object, including the descriptive identifier "cushion," and an object profile including defining characteristics of the "cushion" (such as square and / or rectangular shape, relative size of two cubic feet, appearance of fabric material, etc.). Using the spatial ontology and object profile, object identification is performed such that the object corresponding to cushion representation 210 is identified as an object with the descriptive identifier "cushion." Furthermore, based on the determination that the cushion object is not associated with any identifier in the identifier set, the identifier set is modified by adding identifiers corresponding to newly detected cushion objects. See now for reference. Figure 3B Based on the newly detected cushion object corresponding to cushion representation 210, identifier 310 is generated within the identifier set. Specifically, using a spatial ontology, identifier 310 is associated with the descriptive identifier “cushion”. Various relational identifiers (such as relational identifier 316) are also added to the identifier set regarding identifier 310 (“cushion”). Relational identifier 316 may (via identifier 302, “sofa”) identify sofa representation 202, and further identify the position “on top of”. Specifically, based on the device's perspective, this position indicates that cushion representation 210 is “on top of” sofa representation 202. Relational identifier 318 may (via identifier 304, “lamp”) identify lamp representation 204, and further identify the position “to the right of”. This position indicates that cushion representation 210 is “to the right of” lamp representation 204. Additional identifiers (such as material identifiers including “fabric” associated with identifier 310 (“cushion”)) may be added to the identifier set.

[0037] Certain events can cause a change in the representation of the scene to be detected. Such changes may be based on new objects being added to the scene (e.g., a physical object entering the field of view, a virtual object being generated, etc.), movement of previously detected objects, movement of the user and / or electronic devices (e.g., a user walking around a room while wearing a device), changes in the position and / or orientation of devices based on user head movement, etc. In response to detecting changes based on such movement, the relational identifiers associated with various objects in the identifier set can be modified. For example, such as Figure 2C As shown, the vase corresponding to vase number 206 can be moved from its position on the top of the table to its position on the shelf. Therefore, as... Figure 3CAs shown, the relational identifier associated with identifier 306 (“vase”) can be modified by changing the position value that defines a new position of the vase relative to other objects in the environment. Specifically, relational identifier 320 may (via identifier 308, “table”) identify table representation 208 and can be modified to identify a position “behind”. Specifically, based on the device’s perspective, this position indicates that vase representation 306 is now positioned “behind” table representation 308 (different from its previous position “on top of” table representation 308). Similarly, relational identifier 322 may (via identifier 302, “sofa”) identify sofa representation 202 and can be modified to identify a position “to the right”. Specifically, based on the device’s perspective, this position indicates that vase representation 306 is now positioned “to the right” sofa representation 302 (different from its previous position “in front of” table representation 308). The relation identifier 324 can similarly (via identifier 302, "sofa") identify the sofa representation 202, and can be modified to identify the position "to the left of".

[0038] Various additional definition sets or spatial ontology can be received from third parties to facilitate object identification of customized or other additional objects. This proactive form of object identification allows developers or other users to create various environmental settings based on game environments, social network environments, user training environments, etc. These environments can be associated with additional spatial ontology that may include, for example, basic definitions (i.e., general definitions) included in a first definition set and additional definitions of user-generated objects (e.g., customized definitions of physical objects and / or virtual objects). Additional definitions can be shared across various applications, platforms, etc. For example, a second definition set obtained from a third party may be associated with a game environment corresponding to a racing game. In this environment, various game-specific virtual objects can be utilized, such as virtual cars and car parts including tires, steering wheels, lights, etc. The basic definitions in the first definition set may include objects with the descriptive identifier "chair". Therefore, the second definition set may also include objects with the descriptive identifier "chair". The second definition set may also include an additional descriptive identifier "racing chair". Here, the "racing chair" object may be associated with a parent identifier corresponding to "chair", such that the "chair" object corresponds to the parent object category of the "racing chair" object. In some examples, developers of additional definition sets can add additional "child" identifiers to existing identifiers, but can choose not to add new identifiers that are not associated with the corresponding "parent" identifier. For example, a developer can add an additional "furniture" type identifier (e.g., a five-legged chair is "a type" of furniture), but can choose not to add new object categories that do not exist in the first identifier set (e.g., "aircraft").

[0039] In some examples, an application may access an additional set of definitions when one or more conditions are met. For instance, in response to a user action that places a particular object within an extended reality setting (e.g., a racing chair), credentials or other access information may be provided to various applications associated with the extended reality setting in order to retrieve and utilize an additional set of definitions, including identifiers associated with the respective object.

[0040] Generally, a definition set (spatial ontology) can be stored locally on an electronic device and / or updated periodically. For example, an electronic device may obtain a first definition set from auxiliary electronic devices such as a server. The device may then store the first definition set for use by an extended reality system. Once an update to the first definition set becomes available, the updated definition set is received and then replaces the stored definition set on the electronic device. Specifically, additional objects or object categories may be defined in the updated definition set. Various objects or object categories may also be removed from the updated definition set. Updates to the definition set may also be performed periodically (e.g., daily, weekly, monthly), and / or may be performed when the electronic device is in a specific state (e.g., while charging) or at a specific time (e.g., at night).

[0041] Users of electronic devices can interact with an extended reality environment based on various input methods, such as controller and / or voice input. A set of identifiers can be used to interpret these user inputs. Therefore, the intent associated with the user's voice input can be determined, at least in part, based on the identifier set. For example, user input associated with a representation of a scene can be received, such as voice input including "move my vase to the shelf." Object references, such as "vase," can be identified from the voice input. One or more descriptive identifiers matching the identified object reference can be detected. For example, an application programming interface (API) can be used to query the identifier set to detect the corresponding descriptive identifier. Here, the descriptive identifier associated with identifier 306 ("vase") for "vase" can be identified based on its matching with the object reference "vase." Specifically, based on the identifier set, identifier 306 is located, which corresponds to... Figure 2A The vase in the image is represented by 206. Therefore, output is provided based on the detected descriptive identifiers, such as output depicting the corresponding object representation moved according to user requests. For example, as... Figures 2A to 2B As shown, vase representation 206 can be displayed as a position moved from table representation 208 to shelf representation.

[0042] In some examples, queries involving parent type objects can result in the detection of multiple descriptive identifiers. For example, a voice input such as "How many plants are in this room?" can be received. A single descriptive identifier corresponding to "plant" can be obtained (based on the match with the spoken word "plant"). In this example, the corresponding identifier for "plant" can be associated with several "child" identifiers. More specifically, the set of identifiers can include identifiers for "peace lily," "jade tree," and "rubber tree." Each identifier can be associated with a parent identifier corresponding to "plant." For example, the identifier "peace lily" can be represented together with its parent identifier as "peace lily is a type of 'plant.'" Therefore, the query results can include returning all identifiers associated with the parent identifier "plant," specifically the identifiers for "peace lily," "jade tree," and "rubber tree." Based on the returned identifiers, output can be provided to the user, including "There are three different types of plants in this room" or "There are three plants in this room: peace lily, jade tree, and rubber tree."

[0043] refer to Figure 4 This document describes a flowchart of an exemplary process 400 for displaying a virtual display in an extended reality setting. Process 400 can be performed using a user device (e.g., device 100a). For example, the user device can be a handheld mobile device or a head-mounted device. In some embodiments, process 400 is performed using two or more electronic devices (such as user devices communicatively coupled to another device). In various examples, the display of the user device can be transparent or opaque. Process 400 can be applied to, for example, extended reality applications, such as virtual reality, augmented reality, or mixed reality applications. Process 400 may also involve effects including visible features as well as invisible features (such as hearing, touch, etc.). One or more boxes of process 400 can be optional, and / or additional boxes can be performed. Furthermore, the boxes of process 400 are depicted in a specific order, but it should be understood that these boxes can be performed in other orders.

[0044] At box 402, a representation of the scene is obtained. At box 404, a first object is detected based on the representation of the scene. In some examples, the detected first object corresponds to either a physical object or a virtual object. At box 406, a set of identifiers is retrieved. In some examples, the identifier set includes a descriptive identifier associated with the detected first object, wherein the descriptive identifier identifies the type of the detected first object. In some examples, the identifier set includes a parent identifier associated with the descriptive identifier, wherein the parent identifier corresponds to an object category. In some examples, the identifier set includes a relation identifier associated with the detected first object, wherein the relation identifier identifies a detected second object and identifies the position of the detected first object relative to the detected second object. In some examples, the identification of the position of the detected first object relative to the detected second object depends on the viewpoint of the obtained representation of the scene. In some examples, the identifier set includes a relation identifier associated with the detected first object, wherein the relation identifier identifies the detected object as contained within the detected second object. In some examples, the identifier set includes a relation identifier associated with a detected first object, wherein the relation identifier identifies the detected first object as generated from a detected second object. In some examples, the identifier set includes a material identifier associated with the detected object, wherein the material identifier identifies the type of substance included within the detected first object. In some examples, the identifier set includes coefficients associated with the detected object, wherein the coefficients correspond to properties of the detected first object.

[0045] At box 408, in response to detecting a change in the representation of the scene, the identifier set is modified based on a first definition set. In some examples, the movement of the detected first object is detected from the representation of the scene, and the change in the representation of the scene is detected based on the movement of the detected first object. In some examples, modifying the identifier set based on the first definition set includes modifying the relation identifier associated with the detected first object by modifying the position value that defines the position of the detected first object relative to the detected second object. In some examples, the change in the representation of the scene is detected by detecting the second object from the representation of the scene. In some examples, based on determining that the second object is not associated with any identifier in the identifier set, the identifier set is modified by adding the following items to the identifier set: a descriptive identifier associated with the detected second object, a relation identifier associated with the detected second object, and a material identifier associated with the detected second object. In some examples, based on determining that the second object is not associated with any identifier in the identifier set, the identifier set is modified by obtaining the first definition set and identifying the detected second object as corresponding to the corresponding descriptive identifier contained in the first definition set. In some examples, identifying the detected second object as corresponding to a descriptive identifier contained in the first definition set includes identifying the detected second object as corresponding to an object profile, wherein the corresponding descriptive identifier corresponds to the object profile.

[0046] In some examples, a second definition set is obtained from a third party, and the identifier set is modified based on the second definition set in response to a detected change in the representation of the scene. In some examples, the second definition set is obtained from a third party, wherein the first definition set includes a first corresponding descriptive identifier, and the second definition set includes a parent identifier matching the corresponding descriptive identifier and a second corresponding descriptive identifier associated with the parent identifier. In some examples, the first definition set is obtained from an auxiliary electronic device and stored on the electronic device. In some examples, an updated definition set is obtained from the auxiliary electronic device, and the stored first definition set is replaced with the updated definition set.

[0047] At box 410, user input associated with a representation of the scene is received. At box 412, a user intent corresponding to the user input is determined based on a modified set of identifiers. In some examples, determining the user intent corresponding to the user input based on the modified set of identifiers includes: identifying an object reference from the received user input; detecting a descriptive identifier from the modified set of identifiers that matches the object reference; and providing output based on the detected descriptive identifier.

[0048] As described above, one aspect of this disclosure is the collection and use of data available from various sources to improve the placement of virtual objects based on referential expressions. This disclosure anticipates that, in some instances, such collected data may include personal information data that uniquely identifies or can be used to contact or locate specific individuals. Such personal information data may include demographic data, location-based data, telephone numbers, email addresses, Twitter IDs, home addresses, data or records relating to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other identifying or personal information.

[0049] This disclosure recognizes that the use of such personal information data in the art disclosed herein can benefit users. For example, personal information data can be used to enhance the accuracy of object identification using spatial ontology. Thus, the use of such personal information data enables users to control object identification in a planned manner. Furthermore, this disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used to provide insights into a user's overall health status or as positive feedback for individuals using technology to pursue health goals.

[0050] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are generally recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as data collection and / or use change. Personal information from users should be collected for the entity's lawful and reasonable purposes and not shared or sold outside of these lawful uses. Furthermore, such collection / sharing should be conducted only after obtaining informed consent from users. Additionally, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and processes. Furthermore, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Additionally, policies and practices should be adapted to the specific types of personal information data collected and / or accessed, and to applicable laws and standards, including specific considerations regarding jurisdiction. For example, in the United States, the collection or acquisition of certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while in other countries, health data may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained for different types of personal data in each country.

[0051] Regardless of the foregoing, this disclosure also envisions examples of users selectively blocking the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, with respect to object identification using spatial ontology, the technology of this disclosure can be configured to allow a user to opt-in or opt-out to participate in the collection of personal information data during or at any time after registering for the service. In another example, a user may choose not to provide information about specific objects in the device environment. In yet another example, a user may choose to limit the duration for which environment-specific data is maintained, or to completely prohibit the collection of certain environment-specific data. In addition to providing "opt-in" and "opt-out" options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, a user may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0052] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods.

[0053] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed examples, it also envisions that various examples can be implemented without access to such personal information data. That is, various examples of the techniques disclosed herein will not be unable to function properly due to the lack of all or part of such personal information data. For example, if a user is observing a completely virtual environment, personal information (such as a user's preference for frequently visiting a "virtual kitchen") may be restricted in its use (e.g., the arrangement of virtual kitchen appliances and other objects could otherwise reveal information about the user). As another example, preferences can be inferred based on non-personal information data or only a minimal amount of personal information (such as content requested by a device associated with the user), other non-personal information available to the system for identifying objects using spatial ontology, or publicly available information, thereby selecting content and delivering it to the user.

Claims

1. A method comprising: Obtain a representation of a scene, wherein the representation of the scene includes at least one representation of a physical object and at least one representation of a virtual object; The first object is detected based on the representation of the scene; Retrieve identifier set; Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; In response to detecting a change in the representation of the scene: The identifier set is modified based on a first definition set, wherein the first definition set includes multiple physical object definitions and multiple virtual object definitions; Based on the determination that the second object is not associated with any identifier in the identifier set: The identifier set is modified by adding the following items to the identifier set: A descriptive identifier associated with the detected second object; The relation identifier associated with the detected second object; as well as Material identifier associated with the detected second object; Receive user input associated with the representation of the scene; as well as The user intent corresponding to the user input is determined based on the modified set of identifiers.

2. The method according to claim 1, wherein, The identifier set includes descriptive identifiers associated with the detected first object, wherein the descriptive identifiers identify the type of the detected first object.

3. The method according to claim 2, wherein, The identifier set includes parent identifiers associated with the descriptive identifiers, wherein The parent identifier corresponds to the object category.

4. The method according to any one of claims 1 to 2, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected second object, and The relation identifier identifies the position of the first detected object relative to the second detected object.

5. The method according to claim 4, wherein, The position of the first object detected by the identifier relative to the second object detected depends on the viewpoint of the obtained representation of the scene.

6. The method according to any one of claims 1 to 2, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected object as being included within the detected second object.

7. The method according to any one of claims 1 to 2, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the first detected object as generated from the second detected object.

8. The method according to any one of claims 1 to 2, wherein, The identifier set includes material identifiers associated with the detected objects, wherein The material identifier identifies the type of substance included within the first detected object.

9. The method according to any one of claims 1 to 2, wherein, The identifier set includes the numerical attributes of the first detected object.

10. The method according to any one of claims 1 to 2, comprising: The movement of the first object detected is detected from the representation of the scene; as well as Detecting the change in the representation of the scene based on the movement of the detected first object, wherein modifying the identifier set based on the first definition set includes: The relation identifier associated with the first detected object is modified by changing the position value of the first detected object relative to the second detected object.

11. The method according to any one of claims 1 to 2, comprising: Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; as well as Based on the determination that the second object is not associated with any identifier in the identifier set, the identifier set is modified in the following way: Obtain the first definition set; as well as The detected second object is identified as a corresponding descriptive identifier contained in the first definition set.

12. The method according to claim 11, wherein, Identifying the detected second object as a descriptive identifier corresponding to one contained in the first definition set includes: The detected second object is identified as corresponding to an object configuration file, wherein the corresponding descriptive identifier corresponds to the object configuration file.

13. The method according to any one of claims 1 to 2, comprising: Obtain the second set of definitions from a third party; as well as In response to detecting a change in the representation of the scene: The identifier set is modified based on the second definition set.

14. The method according to any one of claims 1 to 2, comprising: Obtain the second set of definitions from a third party, where The first definition set includes a first corresponding descriptive identifier; as well as The second definition set includes The parent identifier that matches the corresponding descriptive identifier, and A second corresponding descriptive identifier associated with the parent identifier.

15. The method according to any one of claims 1 to 2, comprising: Obtain the first definition set from the auxiliary electronic device; Store the first definition set in the electronic device; Obtain an updated definition set from the auxiliary electronic device; as well as Replace the stored first definition set with the updated definition set.

16. The method according to any one of claims 1 to 2, wherein, Determining the user intent corresponding to the user input based on the modified identifier set includes: Refer to the received user input identifier object; Detect descriptive identifiers that match the object reference from the modified identifier set; and Output is provided based on the detected descriptive identifiers.

17. The method according to any one of claims 1 to 2, wherein, The first object detected corresponds to either a physical object or a virtual object.

18. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device, the one or more programs including instructions for performing the following operations: Obtain a representation of a scene, wherein the representation of the scene includes at least one representation of a physical object and at least one representation of a virtual object; The first object is detected based on the representation of the scene; Retrieve identifier set; Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; In response to detecting a change in the representation of the scene: The identifier set is modified based on a first definition set, wherein the first definition set includes multiple physical object definitions and multiple virtual object definitions; Based on the determination that the second object is not associated with any identifier in the identifier set: The identifier set is modified by adding the following items to the identifier set: A descriptive identifier associated with the detected second object; The relation identifier associated with the detected second object; as well as Material identifier associated with the detected second object; Receive user input associated with the representation of the scene; as well as The user intent corresponding to the user input is determined based on the modified set of identifiers.

19. The computer-readable storage medium of claim 18, wherein, The identifier set includes descriptive identifiers associated with the detected first object, wherein the descriptive identifiers identify the type of the detected first object.

20. The computer-readable storage medium according to claim 19, wherein, The identifier set includes parent identifiers associated with the descriptive identifiers, wherein The parent identifier corresponds to the object category.

21. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected second object, and The relation identifier identifies the position of the first detected object relative to the second detected object.

22. The computer-readable storage medium of claim 21, wherein, The position of the first object detected by the identifier relative to the second object detected depends on the viewpoint of the obtained representation of the scene.

23. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected object as being included within the detected second object.

24. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the first detected object as generated from the second detected object.

25. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The identifier set includes material identifiers associated with the detected objects, wherein The material identifier identifies the type of substance included within the first detected object.

26. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The identifier set includes the numerical attributes of the first detected object.

27. The computer-readable storage medium according to any one of claims 18 to 19, wherein the one or more programs include instructions for: The movement of the first object detected from the representation of the scene; and The change in the representation of the scene is detected based on the movement of the detected first object, wherein, Modifying the identifier set based on the first definition set includes: The relation identifier associated with the first detected object is modified by changing the position value of the first detected object relative to the second detected object.

28. The computer-readable storage medium according to any one of claims 18 to 19, wherein the one or more programs include instructions for: Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; as well as Based on the determination that the second object is not associated with any identifier in the identifier set, the identifier set is modified in the following way: Obtain the first definition set; as well as The detected second object is identified as a corresponding descriptive identifier contained in the first definition set.

29. The computer-readable storage medium according to claim 28, wherein, Identifying the detected second object as a descriptive identifier corresponding to one contained in the first definition set includes: The detected second object is identified as corresponding to an object configuration file, wherein the corresponding descriptive identifier corresponds to the object configuration file.

30. The computer-readable storage medium according to any one of claims 18 to 19, wherein the one or more programs include instructions for: Obtain the second set of definitions from a third party; and In response to detecting a change in the representation of the scene: The identifier set is modified based on the second definition set.

31. The computer-readable storage medium according to any one of claims 18 to 19, wherein the one or more programs include instructions for: Obtain the second set of definitions from a third party, where The first definition set includes a first corresponding descriptive identifier; and The second definition set includes The parent identifier that matches the corresponding descriptive identifier, and A second corresponding descriptive identifier associated with the parent identifier.

32. The computer-readable storage medium according to any one of claims 18 to 19, wherein the one or more programs include instructions for: Obtain the first definition set from the auxiliary electronic device; Store the first definition set in the electronic device; Obtain an updated definition set from the auxiliary electronic device; as well as Replace the stored first definition set with the updated definition set.

33. The computer-readable storage medium according to any one of claims 18 to 19, wherein, Determining the user intent corresponding to the user input based on the modified identifier set includes: Refer to the received user input identifier object; Detect descriptive identifiers that match the object reference from the modified identifier set; and Output is provided based on the detected descriptive identifiers.

34. The computer-readable storage medium according to any one of claims 18 to 19, wherein, The first object detected corresponds to either a physical object or a virtual object.

35. An electronic device comprising: One or more processors; as well as The memory stores one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: Obtain a representation of a scene, wherein the representation of the scene includes at least one representation of a physical object and at least one representation of a virtual object; The first object is detected based on the representation of the scene; Retrieve identifier set; Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; In response to detecting a change in the representation of the scene: The identifier set is modified based on a first definition set, wherein the first definition set includes multiple physical object definitions and multiple virtual object definitions; Based on the determination that the second object is not associated with any identifier in the identifier set: The identifier set is modified by adding the following items to the identifier set: A descriptive identifier associated with the detected second object; The relation identifier associated with the detected second object; as well as Material identifier associated with the detected second object; Receive user input associated with the representation of the scene; as well as The user intent corresponding to the user input is determined based on the modified set of identifiers.

36. The device according to claim 35, wherein, The identifier set includes descriptive identifiers associated with the detected first object, wherein the descriptive identifiers identify the type of the detected first object.

37. The device according to claim 36, wherein, The identifier set includes parent identifiers associated with the descriptive identifiers, wherein The parent identifier corresponds to the object category.

38. The device according to any one of claims 35 to 36, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected second object, and The relation identifier identifies the position of the first detected object relative to the second detected object.

39. The device according to claim 38, wherein, The position of the first object detected by the identifier relative to the second object detected depends on the viewpoint of the obtained representation of the scene.

40. The device according to any one of claims 35 to 36, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the detected object as being included within the detected second object.

41. The device according to any one of claims 35 to 36, wherein, The identifier set includes relation identifiers associated with the detected first object, wherein The relation identifier identifies the first detected object as generated from the second detected object.

42. The device according to any one of claims 35 to 36, wherein, The identifier set includes material identifiers associated with the detected objects, wherein The material identifier identifies the type of substance included within the first detected object.

43. The device according to any one of claims 35 to 36, wherein, The identifier set includes the numerical attributes of the first detected object.

44. The device according to any one of claims 35 to 36, wherein the one or more programs include instructions for: The movement of the first object detected from the representation of the scene; and The change in the representation of the scene is detected based on the movement of the detected first object, wherein, Modifying the identifier set based on the first definition set includes: The relation identifier associated with the first detected object is modified by changing the position value of the first detected object relative to the second detected object.

45. The device according to any one of claims 35 to 36, wherein the one or more programs include instructions for: Changes in the representation of the scene are detected by detecting a second object from the representation of the scene; as well as Based on the determination that the second object is not associated with any identifier in the identifier set, the identifier set is modified in the following way: Obtain the first definition set; as well as The detected second object is identified as a corresponding descriptive identifier contained in the first definition set.

46. ​​The device according to claim 45, wherein, Identifying the detected second object as a descriptive identifier corresponding to one contained in the first definition set includes: The detected second object is identified as corresponding to an object configuration file, wherein the corresponding descriptive identifier corresponds to the object configuration file.

47. The device according to any one of claims 35 to 36, wherein the one or more programs include instructions for: Obtain the second set of definitions from a third party; and In response to detecting a change in the representation of the scene: The identifier set is modified based on the second definition set.

48. The device according to any one of claims 35 to 36, wherein the one or more programs include instructions for: Obtain the second set of definitions from a third party, where The first definition set includes a first corresponding descriptive identifier; and The second definition set includes The parent identifier that matches the corresponding descriptive identifier, and A second corresponding descriptive identifier associated with the parent identifier.

49. The device according to any one of claims 35 to 36, wherein the one or more programs include instructions for: Obtain the first definition set from the auxiliary electronic device; Store the first definition set in the electronic device; Obtain an updated definition set from the auxiliary electronic device; as well as Replace the stored first definition set with the updated definition set.

50. The device according to any one of claims 35 to 36, wherein, Determining the user intent corresponding to the user input based on the modified identifier set includes: Refer to the received user input identifier object; Detect descriptive identifiers that match the object reference from the modified identifier set; and Output is provided based on the detected descriptive identifiers.

51. The device according to any one of claims 35 to 36, wherein, The first object detected corresponds to either a physical object or a virtual object.

52. An electronic device comprising means for performing the method according to any one of claims 1 to 2.

53. A computer program product comprising one or more programs, said one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method according to any one of claims 1 to 2.

Images