Scene information access for electronic device applications

Abstract

Aspects of the subject technology provide constrained access to scenario information by an application running on an electronic device. A system process of an electronic device may assign a region of a physical environment to an application. A user interface of the application may be displayed in the allocated area. In one or more implementations, the system process may provide scenario information and / or user information to the application only when the scenario information and / or user information occurs and / or is initiated within the allocated area.

Description

[0001] This application is a divisional application of the Chinese invention patent application entering the Chinese national phase of the PCT application with an international filing date of November 10, 2022, national application number 202280076589.6, and invention name “Scene Information Access for Electronic Device Applications”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 281,600, filed on November 19, 2021, and entitled “Scene Information Access for Electronic Device Applications,” the disclosure of which is hereby incorporated herein in its entirety. Technical Field

[0004] The present specification generally relates to electronic devices, including, for example, context information access for electronic device applications. Background Art

[0005] Extended reality technology aims to bridge the gap between virtual and physical environments by providing a view of the physical environment that is extended with electronic information. Thus, the electronic information appears to be part of the physical environment perceived by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Certain features of the subject technology are set forth in the appended claims. However, for illustrative purposes, several specific implementations of the subject technology are set forth in the following figures.

[0007] Figure 1 An exemplary system architecture including various electronic devices that may implement the subject system is shown according to one or more specific embodiments.

[0008] Figure 2 An example of an extended reality environment including multiple user interfaces displayed by an electronic device as appearing at multiple corresponding locations in a physical environment in accordance with aspects of the subject technology is shown.

[0009] Figure 3 FIG. 1 shows an example of a method for assigning various user interfaces to be displayed as appearing in a physical environment according to one or more implementations. Figure 2 various parts of the physical environment.

[0010] Figure 4 An exemplary electronic device that provides constrained access to context information according to one or more implementations is shown.

[0011] Figure 5A flow chart illustrating an exemplary process for constraining context information access in accordance with aspects of the subject technology is shown.

[0012] Figure 6 An exemplary computing device is shown that can be used to implement aspects of the subject technology. DETAILED DESCRIPTION

[0013] The specific description shown below is intended to be a description of various configurations of the subject technology and is not intended to represent the only configuration that the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details described herein, and one or more other specific implementations can be used to practice. In one or more specific implementations, structures and components are shown in block diagram form to avoid blurring the concepts of the subject technology.

[0014] The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment may include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park including physical trees, physical buildings, and physical people. People can directly sense the physical environment and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell. In contrast, the extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, the XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As an example, the XR system can detect head movement, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in the physical environment. As another example, an XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) presenting an XR environment, and in response, adjust the graphical content and sound fields presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust characteristics of graphical content in an XR environment in response to representations of physical movement (e.g., voice commands).

[0015] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0016] Specific implementations of the subject technology described herein provide constrained context information to electronic device applications that operate based on context information. For example, in order to allow applications to operate based on context information of a physical environment (e.g., information about the physical contents of the physical environment) without allowing the application to access the entire context information, the system process of the electronic device may only provide a subset of the context information to the application. This may help, for example, protect the privacy of users of the electronic device, and / or other people in and / or associated with the physical environment.

[0017] The subset of scene information may be a subset of scene information corresponding to a portion of the physical environment in which the user interface (UI) of the application is displayed (e.g., a three-dimensional area around the apparent location of the UI). For example, scene information corresponding to other portions of the physical environment may be obtained by the system process and not provided to the application. The system process may also constrain user information provided to the application to a subset of user information such as that occurring within the portion of the physical environment in which the UI appears to be displayed. For example, gesture input and / or gaze positions occurring within the portion of the physical environment may be provided to the application, and the application may be prevented from receiving gesture input and / or gaze positions occurring outside of the portion of the physical environment. In one or more specific implementations, if the allocated volume of the application changes, the scene information and / or user information available to the application may shrink or expand accordingly, so that the application can access all scene information, for example in a use case in which the user interface of the application is expanded to full screen mode.

[0018] Figure 1 An exemplary system architecture 100 including various electronic devices that may implement the subject system according to one or more implementations is shown. However, not all of the depicted components may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes in the arrangement and types of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0019] System architecture 100 includes electronic device 105, electronic device 110, electronic device 115, and server 120. For the purpose of explanation, system architecture 100 is shown in FIG. Figure 1 1 is shown as including electronic device 105, electronic device 110, electronic device 115, and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0020] The electronic device 105 may be a smartphone, a tablet device, or a wearable device (such as a head-mounted portable system) that includes a display system capable of presenting a visualization of an extended reality environment to the user 101. The electronic device 105 may be powered by a battery and / or any other power source. In one example, the display system of the electronic device 105 provides a stereoscopic presentation of the extended reality environment to the user, enabling a three-dimensional visual display of a particular scene rendering to be provided. In one or more specific implementations, instead of or in addition to using the electronic device 105 to access the extended reality environment, the user may use an electronic device 104, such as a tablet, a watch, a mobile device, etc.

[0021] The electronic device 105 may include one or more cameras, such as a camera 150 (e.g., a visible light camera, an infrared camera, etc.). In addition, the electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near field communication sensors, etc. In addition, the electronic device 105 may include hardware elements that can receive user input, such as hardware buttons or switches. The user input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed within a given extended reality environment. For example, such input modalities may include but are not limited to facial tracking, eye tracking (e.g., gaze direction), hand tracking, gesture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, breathing, temperature, electroencephalogram, smell), recognition of voice or audio (e.g., specific hot words), and activation of buttons or switches, etc. The electronic device 105 may also detect and / or classify physical objects in the physical environment of the electronic device 105.

[0022] The electronic device 105 may be communicatively coupled to a base device, such as the electronic device 110 and / or the electronic device 115. Generally, such base devices may include more computing resources and / or available power than the electronic device 105. In one example, the electronic device 105 may operate in various modes. For example, the electronic device 105 may operate in a standalone mode independent of any base device. When the electronic device 105 operates in a standalone mode, the number of input modalities may be constrained by the power limitations of the electronic device 105, such as the available battery power of the device. In response to the power limitations, the electronic device 105 may deactivate certain sensors within the device itself to preserve battery power.

[0023] The electronic device 105 may also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection) to work in conjunction with a given base device. The electronic device 105 may also operate in a connection mode in which the electronic device 105 is physically connected to the base device (e.g., via a cable or some other physical connector) and may utilize power resources provided by the base device (e.g., where the base device charges the electronic device 105 when physically connected).

[0024] When the electronic device 105 is operated in a wireless connection mode or a connection mode, at least a portion of processing user input and / or rendering an extended reality environment can be offloaded to the base device, thereby reducing the processing burden on the electronic device 105. For example, in one embodiment, the electronic device 105 works in conjunction with the electronic device 110 or the electronic device 115 to generate an extended reality environment, which includes physical objects and / or virtual objects that implement different forms of interaction (e.g., visual, auditory and / or physical or tactile interaction) between the user and the extended reality environment in real time. In one example, the electronic device 105 provides a rendering of a scene corresponding to the extended reality environment, which can be perceived by the user and interacted in real time. In addition, as part of presenting the rendered scene, the electronic device 105 may provide sound and / or tactile or haptic feedback to the user. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and current system workload.

[0025] The electronic device 105 may also detect an event that has occurred within the scene of the extended reality environment. Examples of such events include detecting the presence of a creature such as a person or pet, a particular person, entity, or object in the scene. The detected physical object may be classified by the electronic device 105, the electronic device 110, and / or the electronic device 115, and the location, position, size, dimension, shape, and / or other characteristics of the physical object may be used to provide a physical anchor object for the XR application, thereby generating virtual content (such as a UI of the application) for display within the XR environment.

[0026] It should also be understood that electronic device 110 and / or electronic device 115 may also work in conjunction with electronic device 105 or independently of electronic device 105 to generate such an extended reality environment.

[0027] The network 106 may communicatively couple (directly or indirectly) electronic device 105, electronic device 110, and / or electronic device 115 to server 120 and / or one or more electronic devices of one or more other users, for example. In one or more specific implementations, the network 106 may be an interconnected network that may include the Internet or devices communicatively coupled to the Internet.

[0028] The electronic device 110 may include a touch screen and may be, for example, a smart phone including a touch screen, a portable computing device such as a laptop computer including a touch screen, a peripheral device including a touch screen (e.g., a digital camera, headphones), a tablet device including a touch screen, a wearable device including a touch screen (such as a watch, wristband, etc.), any other suitable device including, for example, a touch screen, or any electronic device with a touchpad. In one or more specific implementations, the electronic device 110 may not include a touch screen, but may support touch screen-like gestures, such as in an extended reality environment. In one or more specific implementations, the electronic device 110 may include a touchpad. In Figure 1 In the embodiment, by way of example, electronic device 110 is depicted as a mobile smart phone device having a touch screen. In one or more specific implementations, electronic device 110, electronic device 104, and / or electronic device 105 may be and / or may include the following with respect to Figure 6 All or part of the electronic system discussed. In one or more specific implementations, the electronic device 110 can be another device, such as an Internet Protocol (IP) camera, a tablet computer, or a peripheral device such as an electronic stylus.

[0029] The electronic device 115 may be, for example, a desktop computer, a portable computing device such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera, a headset), a tablet device, a wearable device such as a watch, a wristband, etc. Figure 1 In the embodiment, the electronic device 115 is depicted as a desktop computer by way of example. The electronic device 115 may be and / or may include the following with respect to Figure 6 All or part of the electronic system in question.

[0030] The server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, the server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors and other dedicated or custom processors) for rendering and generating content such as graphics, images, video, audio, and multimedia files of an extended reality environment. In one implementation, the server 120 may be used as a cloud storage server that stores any of the aforementioned extended reality content generated by the above-mentioned devices and / or server 120.

[0031] Figure 2 An example of a physical environment 200 is shown in which an electronic device 105 may operate. Figure 2In the example of FIG. 1 , the electronic device 105 (e.g., the display 230 of the electronic device 105) displays virtual content to be perceived by a user viewing the display 230 of the electronic device 105 at various locations in the physical environment 200 (e.g., at anchor locations determined by the electronic device). When the electronic device 105 that causes the virtual content to appear to the user displays the virtual content in the physical environment 200, the combined physical environment and the virtual content may form an XR environment.

[0032] exist Figure 2 In the example of FIG. 1 , the display 230 of the electronic device 105 displays a user interface (UI) 204 of an application running on the electronic device 105. In this example, the display position of the UI 204 on the display 230 is anchored by the electronic device 105 to the physical anchor position 205, so that the UI 204 displayed in the visible area 207 of the display 230 appears to the user as if it is set on the physical wall 201 in the physical environment 200. Figure 2 As shown, UI 204 may include one or more elements 206. Elements 206 may include text input fields, buttons, selectable tools, scroll bars, menus, drop-down menus, links, plug-ins, image viewers, media players, sliders, game characters, other virtual content, and the like.

[0033] exist Figure 2 In the example of FIG. 1 , the UI 204 is displayed in the viewable area 207 of the display 230 of the electronic device 105 to appear attached to the physical wall 201 in the physical environment 200 in the extended reality environment generated in part by the electronic device 105. Figure 2 In the example of , the electronic device 105 also displays a UI 214 to be perceived by the user at another location in the physical environment (e.g., a UI of another application running on the electronic device and displayed at another location within the visible area 207 of the display 230). In this example, the UI 214 is anchored to a virtual anchor position 215 (e.g., an anchor position in the physical environment that is not associated with a physical object) and appears as a floating UI in the physical environment 200.

[0034] exist Figure 2In the example of , a physical table 212 also exists in the physical environment 200. In this example, the electronic device 105 also displays a UI 226 to be perceived by the user on the surface 220 of the physical table 212 in the physical environment 200 (e.g., a UI of another application running on the electronic device and displayed at another location within the visible area 207 of the display 230). For example, the display location of the UI 226 on the display 230 can be anchored by the electronic device 105 to a physical anchor point location 228 on the surface 220 of the physical table 212. In one or more specific implementations, the UI 226 can include a virtual game board (e.g., a virtual chess board, a virtual chess board, or other virtual board game setting), a virtual keyboard, a virtual character (e.g., a virtual animal, person, or fantasy character), or any other virtual object.

[0035] In various implementations, the physical anchor positions 205 and 228 and / or the virtual anchor position 215 may be generated by each application for its corresponding UI, or may be generated by a system process of the electronic device 105 on behalf of the application based on scene information obtained using the electronic device's camera 150 and / or sensor 152. However, in many use cases, it may not be desirable to allow applications underlying the UI 204, UI 214, and / or UI 226 to access some or all of the scene information describing the content of the physical environment 200 and / or access some or all of the user information such as user gestures performed in the physical environment.

[0036] For example, it may not be generally desirable to allow an application to obtain image information, sensor information, and / or scene information based on image information and / or sensor information that describes the entire physical environment 200. This is because the physical environment 200 may include user-specific information and / or objects. As another example, it may be desirable to prevent one application from receiving gesture input information and / or voice input information that is being provided to the UI of another application. For example, in one illustrative use case, UI 204 may be a UI for a banking application, and UI 214 may be a UI for a social media application. In this illustrative use case, when a user performs gestures or other inputs to a banking application (e.g., to enter private and / or sensitive banking information such as an account number and / or password), it may be desirable to prevent the social media application from receiving gesture information associated with those gestures.

[0037] However, because in an extended reality environment, the user may not physically touch any physical sensors (e.g., touch sensors) that are co-located with display pixels that display the UI, it may be difficult to determine which UI(s) (e.g., and which underlying application(s)) should receive user input and / or other scene information associated with the physical environment 200.

[0038] Aspects of the subject technology can facilitate concurrent operation of multiple applications based on context information and / or user information, such as gesture information, of the physical environment 200 while protecting the privacy of personally identifiable context information and / or user information. For example, the electronic device 105 can allow each application running on the device to obtain only context information and / or user information (e.g., gesture information and / or gaze information) that occurs and / or originates within a subset of the physical environment that has been assigned to the application.

[0039] For example, Figure 3 The following example is shown in which, for each UI (e.g., and each underlying application), the electronic device 105 (e.g., a system process of the electronic device 105) allocates a portion of the physical environment 200 that includes a location remote from the electronic device 105 at which the UI appears to be displayed (e.g., at which the display 230 causes the user of the electronic device 105 to perceive the UI even though no physical display may occur at the perceived / apparent location).

[0040] exist Figure 3 In the example of FIG. 1 , the display 230 of the electronic device 105 displays a representation 304 of the UI 204, which is displayed on the display 230 so that the UI 204 appears to the user viewing the display 230 at a location on the physical wall 201 behind the display 230. As shown, the display 230 of the electronic device 105 also displays: a representation 314 of the UI 214, which is displayed on the display 230 at a location so that the UI 214 appears to be a floating UI; and a representation 326 of the UI 226, which is displayed on the display 230 so that the UI 226 appears to be located at a location on the physical table 212. In various implementations as described herein, the display 230 can be a transparent or semi-transparent display that allows a user to view the physical environment directly through the viewing area of ​​the display, or the display can be a pass-through video display that captures an image of the physical environment and displays representations of physical environment objects (e.g., including representations 304, 314, and 326 of the UIs 204, 214, and 226 in this example) on the display 230 along with device-generated content.

[0041] like Figure 3As shown, the electronic device may assign a portion 344 of the physical environment 200, including the location where the UI 204 is displayed as appearing, to the application that provides the UI 204. In this example, the electronic device 105 also assigns a portion 354 of the physical environment 200, including the location where the UI 214 is displayed as appearing, to the application that provides the UI 214, and assigns a portion 366 of the physical environment 200, including the location where the UI 226 is displayed as appearing, to the application that provides the UI 226. Figure 3 In the example of , portions 344, 354, and 366 of the physical environment 200 are rectilinear three-dimensional volumes within the physical environment. However, it should also be understood that the portion of the physical environment allocated to the application may have any suitable size or shape, and may also be adjustable and / or changeable based on the size, shape, position, and / or content of the corresponding UI and / or based on scene information of the physical environment and / or the XR environment near the allocated portion of the physical environment (e.g., based on the presence of physical objects and / or virtual objects (including other UIs)).

[0042] The electronic device 105 may obtain (e.g., using the camera 150 and / or the sensor 152) scene information corresponding to the physical environment 200 of the electronic device 105. For example, the system process may detect and / or identify physical objects in the physical environment 200, generate a three-dimensional map of the physical environment 200, and / or obtain other scene information describing the physical characteristics of the physical environment 200. The electronic device 105 may determine, for each of the UI 204, the UI 214, and the UI 226, a respective subset of the scene information corresponding to the portion 344, the portion 354, and the portion 366 of the physical environment 200.

[0043] The electronic device 105 may provide a subset of the scene information for each application to the application by a system process without providing the remaining portion of the scene information to the application. The subset of scene information corresponding to a particular portion of the physical environment 200 (e.g., portion 344, portion 354, and portion 366) may include the location of an anchor point (e.g., a physical anchor point and / or a virtual anchor point), and / or the location, type, image, and / or other characteristics of one or more physical objects within the portion of the physical environment. For example, an application that generates UI 204 may be notified of the presence of physical wall 201 or a portion thereof, but not of the presence of physical table 212. As another example, an application that generates UI 226 may be notified of the presence of physical table 212 or a portion thereof (e.g., a surface), but not of the presence of physical wall 201.

[0044] As discussed herein, the electronic device 105 may also provide only a subset of the user information obtained by the electronic device to each application. For example, while operating the electronic device 105, the user may look around the physical environment 200, including viewing the UI 204, UI 214, and / or UI 226 at various times. The user may also move within the physical environment 200, including performing hand gestures corresponding to gesture inputs to the UI 204, UI 214, and / or UI 226 at various times, and / or performing hand movements that are not associated with gesture inputs to the electronic device 105. In one or more specific implementations, the electronic device 105 may only provide gesture information to the application corresponding to the UI 204, UI 214, and / or UI 226 when the gesture input occurs within the corresponding portion 344, 354, and / or 366 for that UI.

[0045] As another example, Figure 3 204. Figure 3 204 , the user's gaze position 305 is located within the portion 344 of the physical environment and provides the gaze information (e.g., the gaze position) to the application corresponding to UI 204. In this example, no gaze information is provided to the application corresponding to UI 214 or UI 226 (e.g., until the user's gaze moves to position the gaze position 305 within the portion 354 or portion 366 of the physical environment). In this way, the electronic device 105 provides the application with only scene information and / or user information that the application uses for its own operation.

[0046] Figure 4 1 shows how the system process of the electronic device 105 can control (e.g., restrict) the access of various applications to scene information and / or user information. For example, Figure 4 1 shows an example architecture that can be implemented by an electronic device 105 according to one or more specific implementations of the subject technology. For the purpose of explanation, Figure 4 Part of the architecture is described by Figure 1The electronic device 105 of the present invention is implemented, such as by a processor and / or memory of the electronic device; however, appropriate parts of the architecture may be implemented by any other electronic device, including the electronic device 110, the electronic device 115 and / or the server 120. However, not all of the components depicted may be used in all specific implementations, and one or more specific implementations may include additional or different components than those shown in the figure. Changes in the arrangement and type of these components may be made without departing from the spirit or scope of the claims listed herein. Additional components, different components, or fewer components may be provided.

[0047] Figure 4 Various parts of the architecture of may be implemented in software or hardware, including by one or more processors and memory devices containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. Figure 4 , the trapezoidal box may indicate that the sensor 152, the camera 150, and the display 230 may be hardware components, and the rectangular box may indicate that the OS service 400, the application 402, the rendering engine 423, and the synthesis engine 427 may be implemented in software, including by one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein.

[0048] exist Figure 4 In the example of FIG. 4 , an application such as application 402 (eg, an application having UI 204, such as Figure 2 and Figure 3 ) provides application data to a rendering engine 423 for rendering the application data, such as for rendering a UI 204 for the application. The application 402 may be a gaming application, a media player application, a content editor application, a training application, a simulator application, a social media application, or generally any application that provides a UI or other content for display at a location that depends on the physical environment, such as by anchoring the UI or other content to an anchor point in the physical environment. The application data may include application-generated content (e.g., windows, buttons, tools, characters, images, audio, etc.) and / or user-generated content (e.g., text, images, etc.) and information for rendering the content in the UI. In one or more specific implementations, the rendering engine 423 renders the UI 204 for display by a display (such as the display 230 of the electronic device 105). In one or more specific implementations, the OS service 400 may display a portion of the physical environment of the electronic device (e.g., portion 344 of the physical environment 200, as shown in Figure 3 ) is assigned to application 402 (e.g., when application 402 is running on electronic device 105 and when UI 204 is displayed by display 230).

[0049] like Figure 4 As shown in , additional information may be provided for the UI of the application 402, such as in a two-dimensional or three-dimensional (e.g., XR) scene (e.g., as in Figure 2 and Figure 3 In the example of Figure 4 In the example of , sensor 152 may provide physical environment information (e.g., depth information from one or more depth sensors, motion information from one or more motion sensors) and / or user information to OS service 400. Camera 150 may also provide images of the physical environment and / or one or more parts of the user (e.g., the user's eyes, hands, face, etc.) to OS service 400. OS service 400 may use the environmental information (e.g., depth information and / or images) from sensor 152 and camera 150 to generate scene information, such as a three-dimensional mapping, of some or all of the physical environment of electronic device 105. OS service 400 may also determine a gaze location based on images and / or other sensor data representing the position and / or orientation of the user's eyes, such as Figure 3 The OS service 400 may also identify gestures (eg, hand gestures) performed by a user of the electronic device 105 based on images and / or other sensor data representing the position and / or orientation of the user's hands and / or arms.

[0050] like Figure 4 As shown, in one or more specific implementations, application 402 can provide a request to OS service 400. For example, the request can be a request for scene information (e.g., information describing the content of the physical environment), and / or a request for user information, such as a request for gaze position and / or user gesture information. In one example, the request can be an anchor request for a physical anchor point (e.g., a horizontal surface, a vertical surface, a floor, a table, a wall, etc.).

[0051] like Figure 4 As indicated, OS service 400 can provide constraint information to application 402. In various implementations, constraint information can be provided in response to a request from application 402, or can be provided without an explicit request from application 402. Constraint information can include context information and / or user information corresponding to portion 344 of the physical environment that has been allocated to application 402 (e.g., by OS service 400).

[0052] The application 402 may include code that, when executed by one or more processors of the electronic device 105, generates application data for displaying the UI 204 at, near, attached to, or otherwise associated with an anchor location corresponding to an anchor identified by an identifier provided from the OS service 400. The application 402 may include code that, when executed by one or more processors of the electronic device 105, modifies and / or updates the application data based on user information (e.g., gaze location and / or gesture input) provided by the OS service 400.

[0053] Once the application data has been generated, the application data may be provided to the OS service 400 and / or the rendering engine 423, such as Figure 4 As shown. As shown, scene information may also be provided to rendering engine 423. As an example, scene information provided to rendering engine 423 from OS service 400 may include or be based on environmental information (such as a depth map of the physical environment) and / or object information of objects detected in the physical environment. Rendering engine 423 may then render application data from application 402 for displaying display 230 of electronic device 105 as appearing at a desired location in physical environment 200. For example, application data (which may be based on constraint scene information corresponding to portion 344 of the physical environment) and scene information from OS service 400 (which may include scene information of other portions of the physical environment) may be used to render representation 304 of UI 204 for display at an appropriate location on display 230 to appear to be located at a desired location in physical environment 200. Display 230 may be, for example, an opaque display, and camera 150 may be configured to provide a pass-through video feed to the opaque display. UI 204 may be rendered for display at a location on the display corresponding to a display location of a physical anchor object in the pass-through video. As another example, display 230 may be a transparent or semi-transparent display. UI 204 may be rendered for display at a location on the display that corresponds to a direct view of physical environment 200 through the transparent or semi-transparent display.

[0054] As shown, in one or more specific implementations, the electronic device 105 may also include a composition engine 427 that combines a video image of the physical environment based on the image from the camera 150 for display with the UI 204 from the rendering engine 423. For example, the composition engine 427 may be provided in an electronic device 105 that includes an opaque display to provide a transparent video to the display. In an electronic device 105 implemented with a transparent or semi-transparent display that allows a user to directly view the physical environment, the composition engine 427 may be omitted or not used in some cases or may be incorporated into the rendering engine 423. Although Figure 4The example of FIG. 4 shows the rendering engine 423 being separate from the OS service 400, but it should be understood that the OS service 400 and the rendering engine 423 may form a common service and / or the rendering operation for rendering content for display may be performed by the OS service 400. Figure 4 The example of 402 shows a rendering engine 423 separate from the application 402, but it should be understood that in some implementations, the application 402 can render content for display by the display 230 without using a separate rendering engine. Figure 4 A single application 402 is depicted in FIG. 1 , but it should be understood that multiple applications may be concurrently executed on the electronic device 105 to receive constraint information corresponding to respective portions of the physical environment and generate information for rendering respective UIs for display 230 (e.g., as shown in FIG. 1 ). Figure 2 and Figure 3 In the example above, application data is displayed concurrently.

[0055] Figure 5 A flowchart of an example process 500 for providing constrained access to scene information according to a specific implementation of the subject technology is shown. Figure 1 The process 500 is described with reference to the electronic device 105. However, the process 500 is not limited to Figure 1 The electronic device 105 of the embodiment of the present invention, and one or more blocks (or operations) of process 500 may be performed by one or more other components of other suitable devices (including electronic device 104, electronic device 110 and / or electronic device 115). Further for the purpose of explanation, some blocks of process 500 are described herein as occurring sequentially or linearly. However, multiple blocks of process 500 may occur in parallel. In addition, the blocks of process 500 do not have to be performed in the order shown, and / or one or more blocks of process 500 do not have to be performed and / or may be replaced by other operations.

[0056] like Figure 5 As shown, at block 502, a display (e.g., display 230) of an electronic device (e.g., electronic device 105) may display a user interface (e.g., UI 204) of an application (e.g., application 402) as appearing at a perceived location (e.g., physical anchor location 205) away from the display within a physical environment (e.g., physical environment 200). For example, the electronic device may render a representation 304 of UI 204 and display the representation 304 on the display 230 as appearing to be located at an expected three-dimensional location in the physical environment (e.g., as described herein in conjunction with Figure 3For example, by adjusting the relative position of the right eye portion of the representation 304 and the left eye portion of the representation 304, the electronic device 105 can control the depth at which the UI 204 appears to the user in the physical environment away from the electronic device 105. In one or more specific implementations, displaying the user interface of the application as appearing at a perceived location within the physical environment may include: receiving user interface information of the user interface from the application in the system unit (e.g., the non-physical digital unit) at the system process (e.g., as described herein in conjunction with Figure 4 The system process may determine the scaling between the system unit and the physical unit in the physical environment based in part on the context information of the physical environment. For example, the system process may determine the scaling between the system unit and the physical unit based on the location for displaying the UI 204 and / or other physical and / or virtual content near the UI 204 to display the UI 204 at a desired perceived location, orientation, and / or size. Such use of the system unit by the application may facilitate, for example, the OS service 400 allowing the user to pull the UI 204 closer or move the UI 204 further away without requiring processing operations by the application.

[0057] At block 504, a system process of the electronic device (e.g., OS service 400) may assign a portion of the physical environment (e.g., portion 344), including the perceived location, to the application. For example, the portion of the physical environment may be a three-dimensional area within the physical environment, such as Figure 3 As an example, a subset of scene information may include the surface of a physical object (e.g., the surface of a physical wall such as Figure 2 and Figure 3 The physical wall 201, the surface 220 of the physical table such as Figure 2 and Figure 3 The subset of scene information may include a location of a physical table 212, floor surface, desktop, window, wall, etc. within the portion of the physical environment. As another example, the subset of scene information may include a physical anchor point (e.g., physical anchor point location 205 or physical anchor point location 228) for virtual content generated by the application (e.g., included in UI 204), which is located within the portion of the physical environment.

[0058] At box 506, the system process may obtain scene information corresponding to the physical environment of the electronic device. For example, the system process may obtain one or more images of the physical environment (e.g., using camera 150) and / or other sensor information (e.g., from one or more sensors 152). The scene information may include the image and / or sensor information, and / or information derived from the image and / or sensor information, such as information describing the content of the physical environment. For example, the scene information may include a depth map of the physical environment, identifiers, types, and / or locations of one or more objects (e.g., physical walls such as physical wall 201, physical tables such as physical table 212, and / or other physical objects), identifiers, types, and / or locations of one or more creatures (e.g., other people and / or pets in the physical environment), and / or any other information describing aspects of the physical environment.

[0059] At block 508, the system process may determine a subset of context information corresponding to the portion (e.g., portion 344) of the physical environment. In one or more implementations, prior to this determination, the system process may receive a request for context information from an application (e.g., as described herein in conjunction with Figure 4 For example, the request may include a request for a physical anchor point for virtual content (e.g., UI 204) generated by the application, and process 500 may also include: using the scene information, by the system process, to determine that the physical anchor point exists in the physical environment outside of the portion of the physical environment; using the scene information, by the system process, to identify a different anchor point for the virtual content within the portion of the physical environment; and providing the different anchor point as a subset of the scene information to the application.

[0060] For example, in one illustrative use case, an application 402 corresponding to UI 214 may request a vertical wall for anchoring UI 214. A system process (e.g., OS service 400) may determine that the wall exists but is occupied by UI 204, and identify and provide a virtual anchor location 215 to the application instead of providing the physical anchor location on the wall to the application. In this way, context information within portion 354 of the physical environment (e.g., virtual anchor location 215) may be provided to application 402 corresponding to UI 214 without providing any context information outside portion 354 to application 402 corresponding to UI 214 (e.g., without informing the application that physical wall 201 exists but is unavailable) or any information associated with UI 204 or its underlying applications. In this way, application 402 corresponding to UI 214 receives sufficient context information to operate the application without exposing the full context information or information associated with other applications to the application.

[0061] At block 510, the system process may provide a subset of the context information to the application without providing the remaining portion of the context information to the application. In one or more implementations, one or more subsets of the remaining portion of the context information (e.g., subsets corresponding to portions 354 and 366 of the physical environment) may be provided to one or more other applications (e.g., applications corresponding to UI 214 and UI 226) running on the electronic device. In one or more implementations, process 500 may also include operating the application using the portion of the context information (e.g., to generate an update to the UI).

[0062] In one or more specific implementations, obtaining scene information corresponding to the physical environment of the electronic device may include: obtaining scene information in a global coordinate system of the physical environment, and providing a subset of the scene information to an application includes: providing a subset of the scene information in an application-specific coordinate system that is different from the global coordinate system. For example, a system process may convert global coordinates of a physical object or location in the physical environment 200 (e.g., relative to an origin in the physical environment) into application-specific coordinates (e.g., relative to a different origin in the application space), and provide the converted application-specific coordinates to the application. In this way, each application running on the electronic device can use scene information from the physical environment to operate in its own local coordinate system. The system process may then convert application data received from the application in application-specific coordinates back to global coordinates for rendering and displaying the application data.

[0063] In one or more specific implementations, the system process may also constrain the user information provided to the application. For example, the system process may also detect a gesture performed by a user of the electronic device within the portion of the physical environment and provide gesture information of the gesture to the application. The application may then process the gesture information as a gesture input from the user for controlling the application. The system process may also detect another gesture performed by the user of the electronic device outside of the portion of the physical environment (e.g., a non-input gesture as part of the user's normal body movement, or a gesture input to another application such as a gesture input within portion 354 or portion 366 of the physical environment 200). In this example, the system process does not provide gesture information associated with the other gesture to the application.

[0064] As another example, the system process may detect a gaze position (e.g., gaze position 305) within a portion of the physical environment (e.g., portion 344) and provide the gaze position to the application. In one or more specific implementations, the gaze position may be masked before providing the gaze position to the application, even if the gaze position is within the portion 344 of the physical environment corresponding to the application. For example, a gaze area with lower accuracy than the gaze position may be provided, or an indication that the gaze position is within portion 344 or intersects with UI 204 or a portion thereof may be provided to the application. The system process may also detect another gaze position outside of the portion of the physical environment and not provide the other gaze position to the application. For example, the user of the electronic device 105 may move his eyes to look from UI 204 toward UI 214. The system process may stop providing gaze information to the application corresponding to UI 204 when the gaze position moves away from portion 344 of the physical environment 200 (e.g., and start providing gaze information to the application corresponding to UI 214 when the gaze position moves to portion 354 of the physical environment).

[0065] In one or more specific implementations, the process may also include detecting, by the system process, a creature within a portion of the physical environment, and temporarily pausing providing the subset of scene information to the application while the creature is within the portion of the physical environment. Figure 2 When UI 204 of physical environment 200 is displayed as appearing on physical wall 201 and at a perceived location within portion 344 of physical environment 200, a person or pet may be walking through portion 344 of physical environment 200. For example, to protect the privacy of the person and / or user, the system process may temporarily suspend the provision of context information (even if the context information is from within application portion 344 of the physical environment) to the application. The provision of context information may be suspended without providing the application with a reason for the suspension. For example, the application may be notified that context information has been temporarily suspended without including a reason for the suspension in the notification. In this way, the application is notified of the suspension without providing the application with any information about the person or pet.

[0066] In one or more specific implementations, process 500 may also include: receiving, by the system process, a request for an additional portion of the physical environment from an application; and determining, by the system process, whether the additional portion of the physical environment is available to the application. For example, while an application may be assigned a portion of the physical environment surrounding a location at which the application's UI is displayed as appearing, in some use cases, the application may attempt to display additional application content as appearing at another location in the physical environment. For example, an application having a UI corresponding to a virtual fish tank may attempt to display virtual content that modifies the apparent color of a physical wall disposed behind the virtual fish tank (e.g., to enhance the appearance of the virtual fish tank). In this example, the application may provide a request to the system process (e.g., OS service 400) for a vertical wall or planar surface disposed behind an apparent location of the application's UI in the physical environment.

[0067] In an example use case, determining whether an additional portion of the physical environment is available for use by an application may include determining that the additional portion of the physical environment is available for use by the application, and the system process may display additional user interface information for the application (e.g., modifying the wall color of the virtual content in this example) as appearing in the additional portion of the physical environment (e.g., on the physical wall). For example, the system process may determine that the additional portion of the physical environment is available for use by the application by detecting (e.g., using camera 150 and / or sensor 152) that a physical wall exists behind the apparent location of the application's UI in the physical environment and that the physical wall is not occupied by the apparent display of another application's UI. In this way, an application may be provided with the ability to use other portions of the physical environment to decorate the application's UI.

[0068] In another example use case, determining whether an additional portion of the physical environment is available for use by an application may include determining that the additional portion of the physical environment is not available for use by the application, and the system process may provide a notification to the application denying the request for the additional portion of the physical environment. For example, the system process may determine that the additional portion of the physical environment is not available for use by the application by determining that no physical wall exists behind the apparent location of the application's UI in the physical environment and / or that a physical wall exists but is occupied by the apparent display of another application's UI. The system process may notify the application that the requested additional portion of the physical environment is not available without including a reason as to why the additional portion of the physical environment is not available in the notification. In this way, the application is provided with information to continue operating without being provided with information associated with the physical environment outside of the allocated portion of the physical environment or any information associated with any other application.

[0069] In one or more specific implementations, process 500 may also include: receiving, by the system process, a request from an application to modify an additional portion of the physical environment outside of the portion of the physical environment based on the content of the user interface, and displaying, by the system process, the modification to the additional portion of the physical environment outside of the portion of the physical environment based in part on the content of the user interface. For example, displaying the modification to the additional portion of the physical environment outside of the portion of the physical environment includes: generating, by the system process, modification content for the modification to the additional portion of the physical environment based in part on the content of the user interface and in part on context information corresponding to the additional portion of the physical environment of the electronic device, and without providing context information corresponding to the additional portion of the physical environment.

[0070] In one example use case, a UI (such as UI 204) of an application with an assigned portion (e.g., portion 344) of the physical environment 201 may be a light generating UI. The light generating UI may be or may include a virtual light or a virtual movie screen, or another UI representing a physical object that generates light. In one or more specific implementations, the light generating UI may request that a system process display projected light that appears to be generated by a UI within the assigned portion 344 of the physical environment 200 onto other portions of the physical environment 200 (e.g., in the form of a halo around the light generating UI, or in the manner in which physical light from a physical light generating device such as a physical light, a physical television screen, or a physical movie screen would project light onto the physical environment).

[0071] In one or more embodiments, a system process (e.g., OS service 400) may use scene information (e.g., a scene map) (including scene information for portions of the physical environment outside of portion 344) together with application data indicating light generated by a light generating UI (e.g., content displayed on a virtual TV) and generate and display additional virtual content that creates the appearance of the generated light shining on physical objects in the physical environment from the apparent location of the light generating UI rather than from the location of the electronic device 105 (e.g., including shadows cast by physical objects in the path of virtual light from the apparent location of the light generating UI).

[0072] As described above, aspects of the subject technology may include data collection. The present disclosure contemplates that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include demographic data, location-based data, online identifiers, phone numbers, email addresses, home address data, image data, audio data, environmental data, or records related to a user's health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, or any other personal information.

[0073] The present disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, the personal information data can be used to provide constrained access to scene information and / or user information through applications running on electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users. For example, health and fitness data can be used according to the user's preferences to provide insights into their overall health, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0074] The disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices that are generally recognized as meeting or exceeding the industry or government requirements for maintaining user privacy. Such information about the use of personal data should be prominently accessible to users and should be updated as the collection and / or use of data changes. The user's personal information should be collected only for legal use. In addition, such collection / sharing should only occur after receiving user consent or other legal basis specified in applicable law. Additionally, such entities should consider taking any necessary steps to defend and safeguard access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted for specific types of personal information data collected and / or accessed, and applied to applicable laws and standards, including specific considerations that are exclusive to jurisdictions that can be used to impose higher standards. For example, in the United States, collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0075] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, in the case where restricted access to scene information and / or user information is provided by an application running on an electronic device, the present technology may be configured to allow users to choose to "opt in" or "opt out" to participate in the collection of personal information data at any time during or after registration for the service. In addition to providing "opt-in" and "opt-out" options, the present disclosure also contemplates providing notifications related to access or use of personal information. For example, a user may be notified that their personal information data will be accessed when downloading an application, and then reminded again just before the personal information data is accessed by the application.

[0076] In addition, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. In addition, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. De-identification can be facilitated by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods such as differential privacy, when appropriate.

[0077] Therefore, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or part of such personal information data.

[0078] Figure 6 An example computing device that can be used to implement various aspects of the subject technology according to one or more specific implementations is shown. Computing device 600 can be any computing device or server for generating the features and processes described above and / or can be a part of any computing device or server for generating the features and processes described above, including but not limited to laptop computers, smart phones, tablet devices, wearable devices such as goggles or glasses, etc. Computing device 600 may include various types of computer-readable media, and interfaces for various other types of computer-readable media. Computing device 600 includes permanent storage device 602, system memory 604 (and / or buffer), input device interface 606, output device interface 608, bus 610, ROM 612, one or more processing units 614, one or more network interfaces 616, and / or their subsets and variations.

[0079] The bus 610 generally represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of the computing device 600. In one or more implementations, the bus 610 communicatively connects the one or more processing units 614 with the ROM 612, the system memory 604, and the permanent storage device 602. The one or more processing units 614 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed herein. In different implementations, the one or more processing units 614 can be a single processor or a multi-core processor.

[0080] ROM 612 stores static data and instructions needed by one or more processing units 614 and other modules of computing device 600. On the other hand, permanent storage device 602 can be a read-write memory device. Permanent storage device 602 can be a non-volatile memory unit that stores instructions and data even when computing device 600 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic or optical disk and its corresponding disk drive) can be used as permanent storage device 602.

[0081] In one or more implementations, a removable storage device (such as a floppy disk, a flash drive, and its corresponding disk drive) can be used as the permanent storage device 602. Like the permanent storage device 602, the system memory 604 can be a read-write memory device. However, unlike the permanent storage device 602, the system memory 604 can be a volatile read-write memory, such as a random access memory. The system memory 604 can store any of the instructions and data that one or more processing units 614 may need at runtime. In one or more implementations, the processes disclosed in the subject matter are stored in the system memory 604, the permanent storage device 602, and / or the ROM 612. The one or more processing units 614 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes of one or more implementations.

[0082] Bus 610 is also connected to input device interface 606 and output device interface 608. Input device interface 606 enables a user to transmit information to computing device 600 and select commands. Input devices that can be used with input device interface 606 may include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). Output device interface 608 may, for example, enable the display of images generated by computing device 600. Output devices that can be used with output device interface 608 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0083] One or more implementations may include a device that acts as both an input device and an output device, such as a touch screen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0084] Finally, if Figure 6As shown, bus 610 also couples computing device 600 to one or more networks and / or to one or more network nodes via one or more network interfaces 616. In this manner, computing device 600 may be part of a computer network, such as a LAN, a wide area network ("WAN"), or an intranet, or may be part of a network of networks, such as the Internet. Any or all components of computing device 600 may be used in conjunction with the subject disclosure.

[0085] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) programmed with one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0086] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0087] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more embodiments, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0088] Instructions can be directly executable, or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as high-level language instructions that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data, or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As recognized by those skilled in the art, the details including but not limited to the number, structure, sequence and organization of instructions can be significantly different without changing the underlying logic, function, processing and output.

[0089] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more embodiments are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0090] Those skilled in the art will recognize that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as electronic hardware, computer software or a combination of the two. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above according to functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders, or divided in different ways) without departing from the scope of the subject technology.

[0091] It should be understood that the specific order or the hierarchical structure of the frame in the process disclosed by the present invention are the illustrations of exemplary methods. Based on the design preferred requirements, it should be understood that the specific order or the hierarchical structure of the frame in the process can be rearranged or all the frames shown are executed. Any frame in these frames can be executed simultaneously. In one or more specific implementations, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the specific implementation described above should not be understood as requiring such division in all specific implementations, and it should be understood that the program component (e.g., computer program product) and system can be generally integrated together in a single software product or encapsulated in multiple software products.

[0092] As used in this specification and any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor" and "memory" all refer to electronic devices or other technical devices. These terms exclude people or groups of people. For the purpose of this specification, the term "display" or "displaying" means displaying on an electronic device.

[0093] As used herein, the phrase "at least one of" following a list of items, any of which is separated by the terms "and" or "or", modifies the list as a whole, rather than each member (i.e., each item) of the list. The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" each refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0094] The predicate words "configured to," "operable to," and "programmed to" do not imply any particular tangible or intangible modification of a subject matter but are intended to be used interchangeably. In one or more embodiments, a processor configured to monitor and control an operation or component may also mean that the processor is programmed to monitor and control the operation or that the processor is operable to monitor and control the operation. Likewise, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0095] Phrases such as aspect, this aspect, on the other hand, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, configuration, this configuration, other configurations, some configurations, one or more configurations, subject technology, disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such one or more phrases is essential to the subject technology, nor does it mean that such disclosure applies to all configurations of the subject technology. Disclosures involving such one or more phrases may apply to all configurations or one or more configurations. Disclosures involving such one or more phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects and vice versa, and this applies similarly to the other aforementioned phrases.

[0096] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" or as an "example" is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, to the extent that the terms "including," "having," and the like are used in the specification or claims, such terms are intended to be inclusive, in a manner similar to the way the term "comprising" is interpreted when used as a transitional word in a claim.

[0097] All structural and functional equivalents to the elements of various aspects described throughout this disclosure that are known or later become known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be made available to the public regardless of whether the disclosure is explicitly stated in the claims. No claim element shall be interpreted under the provisions of 35 U.S.C. §112(f) unless the element is explicitly stated using the phrase “means for…” or, in the case of a method claim, the element is stated using the phrase “step for…”

[0098] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications of these aspects are apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are intended to make the full scope consistent with the language claims, wherein reference to elements in singular values ​​is not intended to mean "only one", but refers to "one or more", unless specifically noted. Unless otherwise specifically stated, the term "some" refers to one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used only for convenience and do not limit the subject disclosure.

Claims

1. A method, comprising: Obtaining, by a system process of the electronic device, scene information corresponding to a physical environment of the electronic device; determining, by the system process, a subset of the context information corresponding to a portion of the physical environment that has been assigned by the system process to an application at the electronic device; as well as The subset of the context information is provided to the application by the system process, while a remaining portion of the context information is not provided to the application. 2 . The method of claim 1 , wherein the portion of the physical environment comprises a perceived location of a user interface of the application.

3. The method according to claim 2, further comprising: The user interface of the application is displayed by a display of the electronic device as appearing at the perceived location within the portion of the physical environment remote from the display.

4. The method of claim 3, wherein displaying the user interface of the application as appearing at the perceived location within the physical environment comprises: receiving, at the system process, user interface information of the user interface from the application in the system unit; as well as A scaling between the system unit and a physical unit in the physical environment is determined based in part on the scene information of the physical environment. The method of claim 1 , wherein the portion of the physical environment comprises a three-dimensional area within the physical environment.

6. The method according to claim 1, further comprising: detecting, by the system process, a living being within the portion of the physical environment; as well as While the creature is within the portion of the physical environment, provision of the subset of the scene information to the application is temporarily suspended.

7. The method according to claim 1, further comprising: receiving, by the system process, a request from the application for an additional portion of the physical environment; as well as A determination is made by the system process whether the additional portion of the physical environment is available for use by the application.

8. The method of claim 7, wherein determining whether the additional portion of the physical environment is available for use by the application comprises: Determining that the additional portion of the physical environment is available for use by the application, the method further comprising: Additional user interface information of the application is displayed to appear at additional perceived locations in the additional portion of the physical environment.

9. The method of claim 7, wherein determining whether the additional portion of the physical environment is available for use by the application comprises: Determining that the additional portion of the physical environment is unavailable for use by the application, the method further comprising: Notification is provided from the system process to the application that the request for the additional portion of the physical environment is denied.

10. The method according to claim 1, further comprising: receiving, by the system process, from the application to modify an additional portion of the physical environment other than the portion of the physical environment based on content of the user interface of the application; as well as The modification to the additional portion of the physical environment outside of the portion of the physical environment is displayed by the system process based in part on the content of the user interface.

11. The method of claim 10, wherein displaying the modification to the additional portion of the physical environment outside of the portion of the physical environment comprises: Modification content is generated by the system process for the modification of the additional portion of the physical environment based in part on the content of the user interface and in part on scene information corresponding to the additional portion of the physical environment of the electronic device, and without providing the scene information corresponding to the additional portion of the physical environment.

12. A method comprising: Providing, by an application at an electronic device, to a system process at the electronic device, a request for scene information corresponding to a physical environment of the electronic device; as well as receiving, by the application, a subset of the context information from the system process, and not receiving a remaining portion of the context information from the system process, The subset of the context information corresponds to a portion of the physical environment that has been assigned by the system process to the application, the portion including a perceived location of a user interface of the application.

13. The method of claim 12, wherein the request comprises a request for a physical anchor point for virtual content generated by the application, wherein the physical anchor point exists in the physical environment outside of the portion of the physical environment, and wherein the method further comprises: Different anchor points within the portion of the physical environment are received by the application from the system process as the subset of the scene information.

14. The method of claim 12, wherein the subset of the scene information comprises locations of surfaces of physical objects within the portion of the physical environment.

15. The method of claim 12, wherein the subset of the scene information includes a physical anchor point for virtual content generated by the application, the physical anchor point being located within the portion of the physical environment.

16. A method according to claim 12, wherein the subset of the scene information includes gesture information of a gesture performed by a user of the electronic device within the portion of the physical environment, and wherein the remaining portion of the scene information includes another gesture performed by the user of the electronic device outside the portion of the physical environment.

17. The method of claim 12, wherein the subset of the scene information comprises a gaze location within the portion of the physical environment, and wherein the remaining portion of the scene information comprises another gaze location outside of the portion of the physical environment.

18. The method of claim 12, wherein receiving the subset of the context information comprises receiving the subset of the context information in an application-specific coordinate system that is different than a global coordinate system of the system process.

19. The method of claim 12, further comprising using the portion of the context information to operate the application.

20. An apparatus comprising: Memory; as well as One or more processors configured to: Obtaining, by a system process of the device, scene information corresponding to a physical environment of the device; determining, by the system process, a subset of the context information corresponding to a portion of the physical environment that has been assigned by the system process to an application at the device, the portion including a perceived location of a user interface of the application; as well as The subset of the context information is provided to the application by the system process, while a remaining portion of the context information is not provided to the application.

Images