Intent-based user interface control for electronic devices

Abstract

Aspects of the subject technology provide for intent-based user interface control for electronic devices. For example, an electronic device can utilize a plurality of indirect engagement indicators performed by a user of the electronic device to ascertain which of several displayed user interfaces the user intends to engage. Once the electronic device determines which of the plurality of user interfaces the user intends to engage, the electronic device can provide user input to an application or other process underlying the user interface. The user input can be based in whole or in part on one or more of the plurality of indirect engagement indicators.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 281,599, filed November 19, 2021, entitled “Intention-Based User Interface Control for Electronic Devices,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This specification relates in its entirety to electronic devices, including, for example, intent-based user interface control for electronic devices. Background Technology

[0004] Extended reality technology aims to bridge the gap between virtual and physical environments by providing an enhanced view of the physical environment that utilizes electronic information. Thus, the electronic information appears to be part of the physical environment as perceived by the user. Attached Figure Description

[0005] Some features of this subject matter are set forth in the appended claims. However, for illustrative purposes, several specific embodiments of this subject matter are illustrated in the following figures.

[0006] Figure 1 An example system architecture is shown, comprising various electronic devices that can implement the system of this subject, according to one or more specific implementations.

[0007] Figure 2 An example of an extended reality environment comprising multiple user interfaces, which are displayed by a display of an electronic device and perceived at multiple corresponding locations in a physical environment, is shown according to various aspects of the subject matter.

[0008] Figure 3 It shows the relationship with one or more specific implementations. Figure 2 The user interface is associated with various direct and indirect participation indicators.

[0009] Figure 4 An example electronic device is shown that provides intent-based user interface control according to one or more specific implementations.

[0010] Figure 5 A flowchart illustrating an example process for intent-based user interface control, based on various aspects of the subject matter, is shown.

[0011] Figure 6 Example computing devices are shown that can be used to implement various aspects of the techniques in this subject matter. Detailed Implementation

[0012] The specific descriptions shown below are intended to illustrate various configurations of the subject matter and are not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject matter. However, the subject matter is not limited to the specific details described herein and can be practiced using one or more other specific embodiments. In one or more embodiments, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject matter.

[0013] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. As an example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system may adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., audio commands).

[0014] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0015] The specific implementation of the subject matter technology described herein provides confirmation of user engagement with the application's user interface displayed in an extended reality environment before providing user input to the application. Conversely, for example, in a physical device display, the user directly touches or moves the device's controllers to a location on the display of the application's user interface to be controlled, while in an extended reality environment, the user interface (UI) can be displayed at a location physically distant from the device itself. Therefore, user input to the UI can be performed at a location physically separate from the device and also physically separate from the UI's apparent location.

[0016] In addition, users may move their hands and / or direct their gaze toward or near the displayed application user interface (UI) without intending to interact with the application UI (e.g., due to normal movement of the user in the physical environment, such as during a conversation with another person unrelated to the electronic device), and / or may perform gestures to interact with a displayed application UI that are not intended to be provided to an application associated with another displayed application UI.

[0017] In one or more specific embodiments of the subject matter, an electronic device may receive two or more engagement indicators (e.g., including gaze data, gestures or other user postures, and / or voice input). These two or more engagement indicators can be coordinated to identify user engagement with the UI (e.g., to confirm the user's intention to engage with the UI) before user input based on one or more of the engagement indicators is provided to the UI's application. For example, even if a user gazes at a UI displayed in an XR environment, the underlying application of that UI may not receive gaze information unless the user has expressed their intention to engage with the UI through at least one auxiliary signal (e.g., a voice command or gesture). Confirming user intent / engagement in this way can contribute to efficient device operation (e.g., ensuring that user input is correctly directed to the intended application). In some use cases, confirming user intent / engagement in this way also helps protect user privacy. For example, a user entering a password in an application's UI via gesture may not want that gesture to be incorrectly transmitted to another application running on the same device.

[0018] Figure 1 An example system architecture 100 comprising various electronic devices capable of implementing the system of this subject matter is shown according to one or more embodiments. However, not all depicted components are usable in all embodiments, and one or more embodiments may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0019] System architecture 100 includes electronic devices 105, 110, 115, and server 120. For illustrative purposes, system architecture 100 is described in... Figure 1 The system is shown as including electronic devices 105, 110, 115 and server 120; however, system architecture 100 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0020] Electronic device 105 may be a smartphone, tablet, or wearable device (such as a head-mounted portable system) that includes a display system capable of presenting a visualization of an extended reality environment to user 101. Electronic device 105 may be powered by a battery and / or any other power source. In one example, the display system of electronic device 105 provides a stereoscopic representation of the extended reality environment to the user, enabling the provision of a three-dimensional visual display rendering a specific scene. In one or more embodiments, instead of using electronic device 105 to access the extended reality environment, or otherwise, the user may use electronic device 104, such as a tablet, watch, mobile device, etc.

[0021] Electronic device 105 may include one or more cameras, such as camera 150 (e.g., visible light camera, infrared camera, etc.). Furthermore, electronic device 105 may include various sensors 152, including but not limited to cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, lidar sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communication sensors, etc. Additionally, electronic device 105 may include hardware elements capable of receiving user input, such as hardware buttons or switches. User input detected by such sensors and / or hardware elements corresponds to various input modalities for interacting with virtual content displayed within a given extended reality environment. For example, such input modalities may include, but are not limited to, face tracking, eye tracking (e.g., gaze direction), hand tracking, posture tracking, biometric readings (e.g., heart rate, pulse, pupil dilation, respiration, temperature, electroencephalogram, olfaction), speech or audio recognition (e.g., specific hot words), and activation of buttons or switches. Electronic device 105 may also detect and / or classify physical objects in its physical environment.

[0022] Electronic device 105 may be communicatively coupled to infrastructure devices, such as electronic device 110 and / or electronic device 115. Generally, such infrastructure devices may include more computing resources and / or available power compared to electronic device 105. In one example, electronic device 105 may operate in various modes. For example, electronic device 105 may operate in an independent mode, independent of any infrastructure device. When electronic device 105 operates in independent mode, the number of input modes may be constrained by the power limitations of electronic device 105 (such as the device's available battery power). In response to power limitations, electronic device 105 may deactivate certain sensors within the device itself to conserve battery power.

[0023] Electronic device 105 can also operate in a wireless connection mode (e.g., connected to a base device via a wireless connection), thereby working in conjunction with a given base device. Electronic device 105 can also operate in a connection mode where electronic device 105 is physically connected to a base device (e.g., via a cable or some other physical connector), and can utilize power resources provided by the base device (e.g., when the base device is charging electronic device 105 while physically connected).

[0024] When electronic device 105 operates in wireless or connected mode, at least a portion of processing user input and / or rendering the extended reality environment can be offloaded to the base device, thereby reducing the processing burden on electronic device 105. For example, in one implementation, electronic device 105 works in conjunction with electronic device 110 or 115 to generate an extended reality environment comprising physical and / or virtual objects that enable various forms of interaction (e.g., visual, auditory, and / or physical or tactile interaction) between the user and the extended reality environment in real time. In one example, electronic device 105 provides a rendering of a scene corresponding to the extended reality environment, which can be perceived and interacted with by the user in real time. Additionally, as part of presenting the rendered scene, electronic device 105 may provide the user with auditory and / or tactile or haptic feedback. The content of a given rendered scene may depend on available processing power, network availability and capacity, available battery power, and the current system workload.

[0025] Electronic device 105 can also detect events that have occurred within a scene of the extended reality environment. Examples of such events include detecting the presence of living beings (e.g., people or pets, specific people, entities, or objects) in the scene. Detected physical objects can be categorized by electronic device 105, electronic device 110, and / or electronic device 115, and the location, position, size, dimensions, shape, and / or other characteristics of the physical objects can be used to provide physical anchor objects for XR applications, thereby generating virtual content (such as the application's UI) for display within the XR environment.

[0026] It should also be understood that electronic devices 110 and / or 115 may also work in conjunction with electronic device 105 or independently of electronic device 105 to generate such extended reality environments.

[0027] Network 106 may communicatively (directly or indirectly) couple electronic devices 105, 110, and / or 115 to server 120 and / or one or more electronic devices of one or more other users. In one or more embodiments, network 106 may be an interconnection network of devices that may include the Internet or be communicatively coupled to the Internet.

[0028] Electronic device 110 may include a touchscreen and may be, for example, a smartphone including a touchscreen, a portable computing device such as a laptop computer including a touchscreen, a peripheral device including a touchscreen (e.g., a digital camera, headphones), a tablet device including a touchscreen, a wearable device including a touchscreen (e.g., a watch, wristband, etc.), any other suitable device including, for example, a touchscreen, or any electronic device with a touchpad. In one or more embodiments, electronic device 110 may not include a touchscreen but may support touchscreen-like gestures, such as in extended reality environments. In one or more embodiments, electronic device 110 may include a touchpad. Figure 1 In this embodiment, by way of example, electronic device 110 is depicted as a mobile smartphone device with a touchscreen. In one or more embodiments, electronic device 110, electronic device 104, and / or electronic device 105 may be and / or may include the following description. Figure 6 The electronic system under discussion may be all or part of it. In one or more embodiments, electronic device 110 may be another device, such as an Internet Protocol (IP) camera, a tablet computer, or a peripheral device such as an electronic stylus.

[0029] Electronic device 115 may be, for example, a desktop computer, portable computing devices such as laptop computers, smartphones, peripherals (e.g., digital cameras, headphones), tablet devices, wearable devices such as watches, wristbands, etc. Figure 1 In this context, by way of example, electronic device 115 is described as a desktop computer. Electronic device 115 may be and / or may include, as described below in relation to... Figure 6 The electronic system under discussion, in whole or in part.

[0030] Server 120 may form all or part of a computer network or server group 130, such as in a cloud computing or data center implementation. For example, server 120 stores data and software and includes specific hardware (e.g., processors, graphics processors, and other dedicated or custom processors) for rendering and generating extended reality content such as graphics, images, videos, audio, and multimedia files. In one implementation, server 120 may function as a cloud storage server that stores any of the aforementioned extended reality content generated by the aforementioned devices and / or server 120.

[0031] Figure 2 An example of a physical environment 200 is shown, in which electronic device 105 can operate. Figure 2In the example, the electronic device 105 (e.g., the display 230 of the electronic device 105) displays virtual content in such a way that the virtual content displayed at the electronic device 105 appears to be located at various positions in the view of the physical environment 200 that are far from the electronic device 105 (e.g., at anchor points determined by the electronic device). When the virtual content is displayed as if it appears in the view of the physical environment 200, the combined physical environment and virtual content can form an XR environment.

[0032] exist Figure 2 In this example, the display 230 of electronic device 105 displays a user interface (UI) 204 of an application running on electronic device 105. In this example, the display position of UI 204 on display 230 is anchored by electronic device 105 to a physical anchor point 205, such that UI 204 displayed in the visible area 207 of display 230 appears to the user as if it were placed on a physical wall 201 in a physical environment 200 physically separate from electronic device 105. Figure 2 As shown, UI 204 can include one or more elements 206. Elements 206 can include text input fields, buttons, optional tools, scrollbars, menus, drop-down menus, links, plugins, image viewers, media players, sliders, game characters, and other virtual content.

[0033] exist Figure 2 In the example, UI 204 is displayed in the visible area 207 of the display 230 of the electronic device 105 so that it appears as if attached to the physical walls 201 in the physical environment 200 within the extended reality environment partially generated by the electronic device 105. Figure 2 In this example, electronic device 105 also displays UI 214 in the physical environment (e.g., the UI of another application running on the electronic device and displayed at another location within the visible area 207 of display 230) so that the user can perceive it from another obvious location away from electronic device 105. In this example, UI 214 is anchored to a virtual anchor point 215 (e.g., an anchor point in the physical environment not associated with a physical object) and appears to the user viewing display 230 as a floating UI in the physical environment 200.

[0034] exist Figure 2In this example, the physical table 212 is also present in the physical environment 200. In this example, the electronic device 105 also displays a UI 226 (e.g., the UI of another application running on the electronic device and displayed at another location within the visible area 207 of the display 230) on the surface 220 of the physical table 212 in the physical environment 200, for the user to perceive from a tangible location away from the electronic device 105. For example, the location where the UI 226 is displayed on the display 230 may be anchored by the electronic device 105 to a physical anchor point location 228 on the surface 220 of the physical table 212. In one or more embodiments, the UI 226 may include a virtual game board (e.g., a virtual chess board, a virtual checkers board, or other virtual board game setup), a virtual keyboard, a virtual character (e.g., a virtual animal, person, or fantasy character), or any other virtual object.

[0035] In various specific implementations, based on scene information obtained using camera 150 and / or sensor 152 of the electronic device, physical anchor positions 205 and 228 and / or virtual anchor position 215 can be generated by each application for its corresponding UI, or can be generated by the system process of the electronic device 105 on behalf of the application. To interact with the underlying applications of UI 204, UI 214, and / or UI 226, the user can, for example, perform gestures or other body postures, and / or speak voice commands intended to be directed to one of UI 204, UI 214, or UI 226. However, because in extended reality environments, users may not have physical contact with any physical sensors (e.g., touch sensors) located in the same position as the display pixels of the displayed UI, it can be difficult to determine in some use cases which UI(s) (e.g., which underlying application(s)) should receive user input associated with gestures and / or voice commands.

[0036] For example, Figure 3 This illustrates an example use case where a user is viewing (e.g., gazing at) UI 204 and performing a gesture 307 on UI 204. Figure 3In the example, the display 230 of electronic device 105 displays a representation 304 of UI 204, which is displayed at a location on the display 230 such that, to a user viewing the display 230, UI 204 appears behind the display 230 and on the physical wall 201 away from electronic device 105 and the display 230. As shown, the display 230 of electronic device 105 also displays: a representation 314 of UI 214, which is displayed at a location on the display 230 such that UI 214 appears to be a floating UI physically separated from the positions of electronic device 105 and the display 230; and a representation 326 of UI 226, which is displayed at a location on the display 230 such that UI 226 appears to be located on a physical table 212 and away from electronic device 105 and the display 230.

[0037] In the various specific embodiments described herein, display 230 may be a transparent or semi-transparent display that allows a user to view the physical environment directly through the visible area of ​​the display, or the display may be a pass-through video display that captures an image of the physical environment and displays a representation of objects in the physical environment and device-generated content on display 230 (e.g., Tables 304, 314, and 326 of UIs 204, 214, and 226 in this example).

[0038] exist Figure 3 In the example, gesture 307 may be expected by the user of electronic device 105 as a gesture input to UI 204. However, because the position of gesture 307 is separate from the position of electronic device 105 and from the apparent position of the displayed UI 204, gesture 307 may be an indirect indicator of user engagement with UI 204 (e.g., an indirect engagement indicator). Gesture 307 can be determined to be an indirect engagement indicator of UI 204 by, for example, determining a ray 309 extending from the user's hand along the direction defined by gesture 307. As an example, the direction of ray 309 may be determined by the direction the user's fingers are pointing or a direction perpendicular to the plane defined by the user's bent fingers and thumb (e.g., in a partial pinch gesture). The ray may be determined by electronic device 105 to intersect and / or pass through user interface 214. Figure 3 In the example, ray 309 intersects UI 204 at intersection point 311, which is located within the boundary of UI 204. In one or more specific implementations, ray 309 may traverse multiple UIs that partially overlap in the depth dimension of the physical environment, and electronics 105 may (e.g., based on previous user behavior and / or machine learning models and / or one or more other indirect engagement indicators) resolve ambiguity about which UI among the multiple UIs is associated with gesture 307.

[0039] For example, in order to confirm the user's intention to engage with UI 204 before providing gesture input to the underlying application of UI 204 based on gesture 307, electronic device 105 may detect a second indirect engagement indicator performed by the user of electronic device 105. For example, Figure 3 This illustrates how a user's eye 301 can have a line of sight (e.g., gaze direction 303) extending along a ray intersecting with UI 204. Although Figure 3 Only a single gaze direction 303 of a single eye 301 is shown, but it can be understood that by obtaining the gaze directions of the user's two eyes, the electronic device 105 can determine the gaze position 305 (e.g., a three-dimensional gaze position) where the user's gaze falls at a given time during operation of the electronic device 105. In this example, the electronic device 105 can determine that the gaze position 305 is located within the boundaries of the UI 204. However, because the user may look around the XR environment during use of the electronic device 105 without always engaging with the entity the user is gazing at, the gaze position 305 can also be an indirect engagement indicator.

[0040] In one or more specific implementations, the electronic device may determine a first indirect participation indicator (e.g., Figure 3 In the example gesture 307) and the second indirect participation indicator (e.g., Figure 3 In the example, gaze position 305) is associated with UI 204 to confirm the user's intention to engage with UI 204, and in response, user input is provided to the application underlying UI 204 based on gesture 307. For example, an electronic device may identify gestural input based on gesture 307 and provide gestural input to the application underlying UI 204. The application can then interpret the gestural input based on UI content displayed as being perceived at the location of the gesture. For example, if the UI includes buttons or a keyboard, the application may determine that the gestural input is a button-pushing gesture intended for interaction with a key on the button or keyboard, and at the gestural location, process the button-pushing gesture indicator. However, this is merely illustrative, and gestural input can be a pinch gesture, a grasping gesture, a swiping gesture, a pushing gesture, a pulling gesture, a rotating gesture, or any other gesture that can be detected by an electronic device (e.g., using camera 150 and / or sensor 152) and used as input to the application. In one or more embodiments, the system process of electronic device 105 may identify gesture input and spatial coordinates corresponding to the gesture input (e.g., coordinates corresponding to the location within the displayed application content that intersects with a ray from the user's finger), and may provide indications of gesture and / or spatial coordinates to the application in response to the gesture input.

[0041] exist Figure 3In one example, gesture 307 is the first indirect participation indicator, and gaze position 305 is the second indirect participation indicator. In other examples, gaze position 305 may be the first indirect participation indicator, and a gesture or voice command may be the second indirect participation indicator. In still other examples, a voice command may be the first indirect participation indicator, and a gesture or gaze position may be the second indirect participation indicator. In these various examples, once the electronic device 105 determines that both the first and second indirect participation indicators are associated with a particular UI, the electronic device 105 can provide user input to the application underlying that particular UI (e.g., based on at least one of the first and second indirect participation indicators).

[0042] Figure 3 Examples of direct engagement with UI 214 are also shown (e.g., at a different time than when gesture 307 is performed and gaze position 305 is associated with UI 204, or at the same time when gesture 307 is performed and gaze position 305 is associated with UI 204). For example, a user of electronic device 105 can perform direct interaction with UI 214, for example, by performing gesture 315 at a visible location of UI 214. For example, a user can tap or touch a UI element of UI 214 with their finger at a visible location perceived by the user. In one or more implementations, direct engagement with one UI may override indirect engagement with another UI, or may be processed in parallel with indirect engagement with another UI. In this way, a user can, for example, view UI 226, UI 204, or any other location away from UI 214 while reaching out and directly grasping UI 214 and moving it to a new visible location (e.g., even if the user's gaze position 305 is at the location of UI 226, UI 204, or any other location away from UI 214).

[0043] Figure 4 This illustrates how a system process of an electronic device 105, according to one or more specific embodiments, can determine the user intent of the user input before providing it to an application. For example, Figure 4 An example architecture, which can be implemented by an electronic device 105, is shown according to one or more specific embodiments of the subject matter. For illustrative purposes, Figure 4 The architecture is described as being composed of Figure 1The architecture is implemented by electronic device 105, such as by a processor and / or memory of the electronic device; however, suitable portions of the architecture may be implemented by any other electronic device, including electronic device 110, electronic device 115, and / or server 120. However, not all depicted components are usable in all embodiments, and one or more embodiments may include additional or different components compared to those shown in the figures. Variations in the arrangement and type of these components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0044] Figure 4 The various parts of the architecture can be implemented in software or hardware, including via one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein. For example, in Figure 4 In the document, the trapezoidal box may indicate that the sensor 152, camera 150, and display 230 are hardware components, and the rectangular box may indicate that the OS service 400 (e.g., an XR service), application 402, rendering engine 423, and compositing engine 427 are implemented in software, including by one or more processors and a memory device containing instructions that, when executed by the processor, cause the processor to perform the operations described herein.

[0045] exist Figure 4 In the examples, applications such as application 402 (e.g., applications with UI 204, such as...) Figure 2 and Figure 3 (As shown) Rendering engine 423 provides application data for rendering application data, such as UI 204 for rendering the application. Application 402 can be a game application, media player application, content editor application, training application, simulator application, social media application, or generally any application that provides UI or other content to be displayed in a visible location away from electronic device 105, such as by anchoring the UI or other content to anchor points in the physical environment. Application data can include application-generated content (e.g., windows, buttons, tools, characters, images, videos, etc.) and / or user-generated content (e.g., text, images, etc.) and information for rendering content in the UI. In one or more specific implementations, rendering engine 423 renders UI 204 for display by a display (such as display 230 of electronic device 105).

[0046] exist Figure 4In the example, sensor 152 may provide physical environment information (e.g., depth information from one or more depth sensors, motion information from one or more motion sensors) and / or user information to system processes (e.g., OS service 400). Camera 150 may also provide OS service 400 with images of the physical environment and / or one or more parts of the user (e.g., the user's eyes, hands, face, etc.). OS service 400 may use the environmental information (e.g., depth information and / or images) from sensor 152 and camera 150 to generate some or all of the scene information of the physical environment of electronic device 105, such as a 3D mapping. OS service 400 may also determine the gaze position based on images and / or other sensor data representing the position and / or orientation of the user's eyes, for example, Figure 3 The gaze position 305. OS service 400 can also identify gestures (e.g., gestures) performed by the user of electronic device 105 based on image and / or other sensor data representing the position and / or orientation of the user's hand and / or arm. OS service 400 can also determine rays extending from a part of the user, for example, extending from the user's hand in a direction determined by the user's hand. OS service 400 can also determine the location of gesture input (e.g., spatial coordinates) based on rays extending from a part of the user and / or partially based on the gaze position. In one or more embodiments, sensor 152 includes one or more microphones. OS service 400 can identify one or more voice commands in voice input received by the microphones.

[0047] like Figure 4 As shown, in one or more embodiments, OS service 400 may confirm user intent (e.g., user intent to interact with the UI of an application such as application 402) before providing user input to application 402 based on sensor signals and / or camera images. Application 402 may include code that, when executed by one or more processors of electronic device 105, generates application data for displaying UI 204 at, near, attached to, or otherwise associated with an anchor location corresponding to an anchor location identified by an identifier provided from OS service 400. Application 402 may include code that, when executed by one or more processors of electronic device 105, modifies and / or updates application data based on user input (e.g., gaze position, voice command, and / or gesture input) provided by OS service 400 after OS service 400 has confirmed engagement / intent.

[0048] Once the application data has been generated, it can be provided to OS service 400 and / or rendering engine 423, such as Figure 4As shown. Scene information can also be provided to the rendering engine 423. As an example, scene information provided to the rendering engine 423 from the OS service 400 may include or be based on environmental information such as a depth map of the physical environment and / or object information of objects detected in the physical environment. The rendering engine 423 may then render application data from the application 402 for display by the display 230 of the electronic device 105, thus appearing at a desired location in the physical environment 200. For example, a representation 304 of the UI 204 can be rendered using application data and scene information from the OS service 400 to be displayed on the display 230 at a position suitable for each eye of the user, thus appearing at a desired location in the physical environment 200. The display 230 may be, for example, an opaque display, and the camera 150 may be configured to provide a pass-through video feed to the opaque display. The UI 204 may be rendered for display on the display at a location corresponding to the display position of a physical anchor object in the pass-through video. As another example, the display 230 may be a transparent or semi-transparent display. UI 204 can be rendered to be displayed on the screen at a location corresponding to the direct view of the physical environment 200 through the transparent or semi-transparent screen.

[0049] As shown in the figure, in one or more specific embodiments, the electronic device 105 may also include a compositing engine 427 that synthesizes video images of the physical environment based on images from the camera 150 for display together with the UI 204 from the rendering engine 423. For example, the compositing engine 427 may be provided in the electronic device 105 including an opaque display to provide transparent video to the display. In electronic devices 105 implemented with transparent or semi-transparent displays that allow users to directly view the physical environment, the compositing engine 427 may be omitted or not used in some cases, or it may be incorporated into the rendering engine 423. Although Figure 4 The example illustrates a rendering engine 423 separate from OS service 400, but it should be understood that OS service 400 and rendering engine 423 may form common services and / or rendering operations for rendering content for display may be performed by OS service 400. Although Figure 4 The example illustrates a rendering engine 423 separate from application 402, but it should be understood that in some specific implementations, application 402 may render content for display by display 230 without using a separate rendering engine. Although in Figure 4 The image depicts a single application 402, but it should be understood that multiple applications can run simultaneously on the electronic device 105, thereby generating the corresponding UI for rendering for display on the monitor 230 (e.g., such as...). Figure 2 and Figure 3In the example, application data (multiple UIs are displayed simultaneously) is displayed, and each application receives confirmed user input from the OS service 400 when the OS service 400 determines the user's intent to engage with the application's UI.

[0050] Figure 5 A flowchart illustrating an example process 500 for providing intent-based user interface control according to a specific implementation of the subject matter is shown. For illustrative purposes, this document primarily refers to... Figure 1 The process 500 is described using electronic device 105. However, process 500 is not limited to... Figure 1 The electronic device 105, and one or more blocks (or operations) of process 500 may be performed by one or more other components of other suitable devices (including electronic devices 104, 110, and / or 115). Further for illustrative purposes, some blocks of process 500 are described herein as occurring sequentially or linearly. However, multiple blocks of process 500 may occur in parallel. Furthermore, the blocks of process 500 need not be performed in the order shown, and / or one or more blocks of process 500 need not be performed and / or may be replaced by other operations.

[0051] like Figure 5 As shown, at box 502, an electronic device (e.g., electronic device 105) displays a representation (e.g., representation 304) of the user interface (e.g., UI 204) of an application (e.g., application 402) on the display (e.g., display 230) of the electronic device, the application causing the user interface to appear in a prominent location (e.g., physical anchor point 205) within the physical environment (e.g., physical environment 200) away from the electronic device.

[0052] At box 504, the electronic device's system processes (e.g., OS service 400) can detect (e.g., using camera 150 and / or sensor 152) a first indirect engagement indicator performed by the user of the electronic device. As an example, the first indirect engagement indicator may include a gaze direction intersecting the user interface (e.g., gaze direction 303), a gesture performed at a location between the display of the electronic device and a visible position of the user interface (e.g., gesture 307), or voice input to a voice-assisted application of the electronic device.

[0053] At box 506, the system processes of the electronic device can detect (e.g., using camera 150 and / or sensor 152) a second indirect engagement indicator performed by a user of the electronic device. As an example, the second indirect engagement indicator may include a gaze direction intersecting the user interface (e.g., gaze direction 303), a gesture performed at a location between the display of the electronic device and a visible position of the user interface (e.g., gesture 307), or voice input to a voice-assisted application of the electronic device. In one or more embodiments, detecting the first indirect engagement indicator may include detecting the first indirect engagement indicator simultaneously with detecting the second indirect engagement indicator.

[0054] At box 508, the system process may determine that both the first indirect engagement indicator and the second indirect engagement indicator are associated with the application's user interface. As an example, the system process may determine that the user is simultaneously viewing and pointing at the user interface, simultaneously viewing and talking about the user interface, or simultaneously pointing at and talking about the user interface. In one or more embodiments, determining that both the first and second indirect engagement indicators are associated with the application's user interface may include determining the association of the first indirect engagement indicator with the user interface in part based on a ray extending from a portion of the user through the user interface. For example, the electronic device may identify a direction corresponding to the gaze (e.g., Figure 3 The electronic device can identify rays (303) extending along a direction partially or entirely determined by the user's hand, and determine the intersection of the gaze direction with a specific user interface. As another example, the electronic device can identify rays (e.g., rays extending along a direction partially or entirely determined by the user's hand) extending along a direction determined by the user's hand. Figure 3 The ray (309) is determined, and it is determined that the ray defined by the user's hand intersects with the same specific user interface. In one or more embodiments, the electronic device can disambiguate between multiple UIs traversed by one or more rays in the ray.

[0055] At box 510, the system process can, based on this determination, provide user input to the application based on at least one of a first indirect engagement indicator and a second indirect engagement indicator. As an example, user input includes gesture-based user gesture input or voice commands based on voice input. In one example use case, the first indirect engagement indicator includes a gaze direction intersecting the user interface, and the second indirect engagement indicator includes a gesture performed at a location between the user and the user interface. In this example use case, the user input can be gesture-based user gesture input.

[0056] In another example use case, the first indirect engagement indicator includes a gaze direction intersecting the user interface, and the second indirect engagement indicator includes voice input to a voice-assisted application of the electronic device. In this example use case, the user input can be a voice command based on voice input. In another example use case, the first indirect engagement indicator can include a gesture performed at a location between the user and the user interface, and the second indirect engagement indicator can include voice input to a voice-assisted application of the electronic device. In this example use case, the user input can include a voice command based on voice input or a gesture-based user gesture input.

[0057] exist Figure 5 In the example, process 500 may confirm the user's intent to interact with or engage with the user interface before (e.g., generating and) providing user input to the application providing the user interface. In one or more embodiments, process 500 may also include not providing user input to the application if user intent is not confirmed. For example, in one or more embodiments, process 500 may include the system process detecting a third indirect engagement indicator (e.g., one of gaze position, gesture, or voice command) performed by the user; the system process detecting a fourth indirect engagement indicator (e.g., a different one of gaze position, gesture, or voice command) performed by the user; the system process determining that the third indirect engagement indicator is associated with the application's user interface; the system process determining that the fourth indirect engagement indicator is not associated with the application's user interface; and proceeding without providing the application with any user input associated with the third or fourth indirect engagement indicator. For example, in an exemplary use case, electronic device 105 may identify a user gesture pointing towards UI 226 while the user is viewing UI 214 (e.g., when the user's gaze position 305 is at a prominent display location of UI 214), and may not provide user input to either UI 226 or UI 214. In one or more specific embodiments, continuing without providing the application with any user input associated with a third or fourth indirect engagement indicator may include continuing to detect gaze position, user gesture, and / or voice commands as potential input to various displayed user interfaces.

[0058] In one or more embodiments, process 500 may further include direct engagement with one UI that overrides indirect engagement with another UI displayed simultaneously. For example, in one or more embodiments, process 500 may include a system process detecting a third indirect engagement indicator (e.g., a gaze position, gesture, or voice command) performed by a user and associated with the user interface of an application; the system process detecting, while detecting the third indirect engagement indicator, a direct engagement gesture performed by the user and associated with another user interface of another application running on the electronic device (e.g., a gesture performed at a prominent display location of the user interface); and providing additional user input to the other application based on the direct engagement gesture, without providing the application with any information associated with the third indirect engagement indicator. For example, in an exemplary use case, electronic device 105 may determine that a user is viewing UI 214 while performing a grasping gesture at a prominent location of UI 226 in the physical environment. In this use case, electronic device 105 (e.g., OS service 400) may provide grasping gesture input to the application providing UI 226, but not to the application providing UI 214.

[0059] In one or more use cases, direct engagement with the UI can also be performed by the user without receiving indirect engagement with another UI. In these and / or other use cases, the electronic device 105 can be detected by system processes of direct engagement gestures (e.g., gestures, etc.) performed by the user at a visible location on the user interface (e.g., where the user interface appears to be displayed in the physical environment). Figure 3 Gestures 315), and provide additional user input to the application based on directly engaged gestures.

[0060] As described above, various aspects of this subject matter can include data sets. This disclosure envisions that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include contextual information, sensor data, data associated with a user's eyes, hands, etc., demographic data, location-based data, online identifiers, telephone numbers, email addresses, home address data, image data, audio data, environmental data, or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, or any other personal information.

[0061] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be beneficial to users. For example, personal information data can be used to provide intent-based user interface control. Furthermore, this disclosure also contemplates other uses of personal information data that are beneficial to users. For example, health and fitness data can be used according to user preferences to provide insights into their overall health status, or it can be used as positive feedback to individuals using the technology to pursue health goals.

[0062] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be prominently displayed and easily accessible to the user, and should be updated as the collection and / or use of data changes. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and processes. Additionally, such entities may be subject to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including specific jurisdictional considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0063] Regardless of the foregoing, this disclosure also envisions specific implementations allowing users to selectively block the use or access to their personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, in the case of providing intent-based user interface control, the inventive technology can be configured to allow users to opt-in or opt-out at any time during or after service registration to participate in the collection of personal information data. In addition to providing "opt-in" and "opt-out" options, this disclosure also envisions providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0064] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differentiated privacy.

[0065] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be unable to function properly due to the absence of all or part of such personal information data.

[0066] Figure 6 Example computing devices are illustrated according to one or more specific embodiments that can be used to implement aspects of the subject matter. Computing device 600 may be any computing device or server for generating the features and processes described above and / or may be part of any computing device or server for generating the features and processes described above, including but not limited to laptop computers, smartphones, tablet devices, wearable devices such as goggles or glasses. Computing device 600 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Computing device 600 includes persistent storage device 602, system memory 604 (and / or buffers), input device interface 606, output device interface 608, bus 610, ROM 612, one or more processing units 614, one or more network interfaces 616, and / or subsets and variations thereof.

[0067] Bus 610 generally represents all system, peripheral, and chipset buses that communicatively connect numerous internal devices of computing device 600. In one or more embodiments, bus 610 communicatively connects one or more processing units 614 to ROM 612, system memory 604, and persistent storage device 602. One or more processing units 614 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in this subject matter. In different embodiments, one or more processing units 614 may be a single processor or a multi-core processor.

[0068] ROM 612 stores static data and instructions required by one or more processing units 614 and other modules of computing device 600. On the other hand, persistent storage device 602 can be a read-write memory device. Persistent storage device 602 can be a non-volatile memory cell that stores instructions and data even when computing device 600 is turned off. In one or more embodiments, mass storage devices (such as magnetic disks or optical disks and their corresponding disk drives) can be used as persistent storage device 602.

[0069] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 602. Like persistent storage device 602, system memory 604 may be a read-write memory device. However, unlike persistent storage device 602, system memory 604 may be volatile read-write memory, such as random access memory. System memory 604 may store any instructions and data that one or more processing units 614 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 604, persistent storage device 602, and / or ROM 612. One or more processing units 614 retrieve instructions to be executed and data to be processed from these various memory units to execute the processes of one or more embodiments.

[0070] Bus 610 is also connected to input device interface 606 and output device interface 608. Input device interface 606 enables a user to communicate information and select commands to computing device 600. Input devices that can be used with input device interface 606 may include, for example, an alphanumeric keypad and pointing devices (also known as "cursor control devices"). Output device interface 608 may, for example, enable the display of images generated by computing device 600. Output devices that can be used with output device interface 608 may include, for example, printers and display devices such as liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, flexible displays, flat panel displays, solid-state displays, projectors, or any other device for outputting information.

[0071] One or more embodiments may include a device that acts as both an input device and an output device, such as a touchscreen. In these embodiments, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input.

[0072] Finally, as Figure 6As shown, bus 610 also couples computing device 600 to one or more networks and / or to one or more network nodes via one or more network interfaces 616. In this way, computing device 600 can be part of a computer network (such as a LAN, a wide area network (“WAN”), or an intranet, or a network of networks (such as the Internet)). Any or all components of computing device 600 may be used in conjunction with the subject matter disclosed herein.

[0073] Specific embodiments within the scope of this disclosure may be implemented in part or in whole using tangible computer-readable storage media (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. The tangible computer-readable storage media may also be substantially non-transitory.

[0074] Computer-readable storage media can be any storage medium that can be read, written, or otherwise accessed by general-purpose or special-purpose computing devices, including any processing electronics and / or processing circuits capable of executing instructions. For example, without limitation, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, track memory, FJG, and Millipede memory.

[0075] Furthermore, computer-readable storage media may include any non-semiconductor memory, such as optical disc storage devices, magnetic disk storage devices, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more embodiments, the tangible computer-readable storage medium may be directly coupled to a computing device, while in other embodiments, the tangible computer-readable storage medium may be indirectly coupled to a computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0076] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as high-level language instructions that can be compiled to produce executable or non-executable machine code. Furthermore, instructions can also be implemented as data, or may include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, etc. As those skilled in the art will recognize, details including, but not limited to, the number, structure, sequence, and organization of instructions can vary significantly without altering the underlying logic, functionality, processing, and output.

[0077] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, one or more specific implementations are executed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0078] Those skilled in the art will recognize that the various exemplary blocks, modules, elements, components, methods, and algorithms described herein can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability between hardware and software, the various exemplary blocks, modules, elements, components, methods, and algorithms have been generally described above in terms of functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the described functionality in different ways for each specific application. Various components and blocks can be arranged differently (e.g., in different orders or divided in different ways) without departing from the scope of the subject matter.

[0079] It should be understood that the specific order or hierarchy of the boxes in the process disclosed in this invention is an illustration of the exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of the boxes in the process may be rearranged or all illustrated boxes may be executed. Any box within these boxes may be executed simultaneously. In one or more embodiments, multitasking and parallel processing may be advantageous. Furthermore, the division of the various system components in the embodiments described above should not be construed as requiring such division in all embodiments, and it should be understood that the program components (e.g., computer program products) and the system may generally be integrated together in a single software product or packaged into multiple software products.

[0080] As used in this specification and any claim of this patent application, the terms "base station," "receiver," "computer," "server," "processor," and "memory" refer to electronic devices or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "display" or "being displayed" mean displaying on an electronic device.

[0081] As used herein, the phrase "at least one of" following a series of items separated by the terms "and" or "or" modifies the list as a whole, not each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each of the listed items; rather, it allows for the inclusion of at least one of any one item and / or at least one of any combination of items and / or at least one of each item. For example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" respectively refer to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0082] The predicates “configured to,” “operable to,” and “programmed to” do not imply any specific tangible or intangible modification to a particular subject matter but are intended to be used interchangeably. In one or more embodiments, a processor configured to monitor and control operations or components may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code may be interpreted as a processor programmed to execute code or operable to execute code.

[0083] Phrases such as aspect, that aspect, on the other hand, some aspects, one or more aspects, implementation, that implementation, another implementation, some implementations, one or more implementations, implementation scheme, that implementation scheme, another implementation scheme, some implementations, one or more implementations, configuration, that configuration, another configuration, some configurations, one or more configurations, subject matter technology, disclosure, this disclosure, other variations thereof, etc., are for convenience only and do not imply that disclosures involving such phrases are essential to the subject matter technology, nor do they imply that such disclosures apply to all configurations of the subject matter technology. Disclosures involving one or more such phrases may apply to all configurations or one or more configurations. Disclosures involving one or more such phrases may provide one or more examples. Phrases such as aspect or some aspects may refer to one or more aspects, and this applies similarly to other foregoing phrases.

[0084] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” or “example” is not necessarily to be construed as preferred or superior to other specific embodiments. Furthermore, the terms “comprising,” “having,” etc., as used in the specification or claims, are intended to be inclusive, similar to how “comprising” is interpreted when used as a transitional word in the claims.

[0085] All structural and functional equivalents of elements throughout the various aspects described herein that are known or later become apparent to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly stated in the claims. No claim element should be interpreted in accordance with 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “component for…” or, in the case of a method claim, using the phrase “step for…”.

[0086] The preceding descriptions are provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein references to elements in singular values ​​are not intended to mean “only one,” but rather “one or more,” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used for convenience only and do not limit the disclosure of this subject matter.

Claims

1. A method, the method comprising: Displaying a representation of the user interface of an application and another user interface of another application on the display of an electronic device, the application making the user interface appear to be displayed at a first location in the physical environment away from the electronic device, and the other application making the other user interface appear to be displayed at a second location in the physical environment away from the electronic device; The system process of the electronic device detects a first indirect participation indicator performed by the user of the electronic device, the first indirect participation indicator indicating a first user action occurring away from the first location where the user interface is displayed; The system process detects a second indirect participation indicator performed by the user of the electronic device, the second indirect participation indicator corresponding to a second user action occurring away from the first location where the user interface is displayed; Before determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application, no user input corresponding to at least one of the first indirect participation indicator or the second indirect participation indicator is provided to the application; The system process determines that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application; as well as In response to determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application: When a direct engagement gesture is detected simultaneously with the first and second indirect engagement indicators, and the direct engagement gesture occurs at the second location, the system process provides additional user input to the other application based on the direct engagement gesture, without providing the application with any information associated with the first or second indirect engagement indicator; otherwise... The system process provides the user input to the application based on at least one of the first indirect participation indicator or the second indirect participation indicator.

2. The method according to claim 1, wherein, The first indirect engagement indicator includes a gaze direction intersecting the user interface, and the second indirect engagement indicator includes a gesture performed at a location between the display of the electronic device and the first location of the user interface.

3. The method according to claim 2, wherein, The user input includes user gesture input based on the gesture.

4. The method according to claim 1, wherein, The first indirect engagement indicator includes a gaze direction intersecting the user interface, and the second indirect engagement indicator includes voice input to the voice assistance application of the electronic device.

5. The method according to claim 4, wherein, The user input includes voice commands based on the voice input.

6. The method according to claim 1, wherein, The first indirect participation indicator includes a gesture performed at a location between the display of the electronic device and the first location of the user interface, and the second indirect participation indicator includes voice input to a voice assistance application of the electronic device.

7. The method according to claim 6, wherein, The user input includes voice commands based on the voice input.

8. The method according to claim 1, wherein, Determining that both the first indirect engagement indicator and the second indirect engagement indicator are associated with the user interface of the application includes: determining that the first indirect engagement indicator is associated with the user interface in part based on a ray extending through the user interface from a portion of the user.

9. The method according to claim 1, wherein, Detecting the first indirect participation indicator includes detecting the first indirect participation indicator while detecting the second indirect participation indicator.

10. The method according to claim 1, further comprising: The system process detects a third indirect participation indicator executed by the user; The system process detects the fourth indirect participation indicator executed by the user; The system process determines that the third indirect participation indicator is associated with the user interface of the application; The system process determines that the fourth indirect participation indicator is not associated with the user interface of the application. as well as Continue without providing the application with any user input associated with the third or fourth indirect participation indicator.

11. The method of claim 1, wherein the direct engagement gesture includes a gesture of tapping or touching the other user interface within the physical environment.

12. An electronic device, the electronic device comprising: Display components; Memory; as well as One or more processors, said one or more processors being configured to: The display component displays a representation of the user interface of an application and another user interface of another application, wherein the application causes the user interface to appear in a first location in the physical environment away from the display component of the electronic device, and the other application causes the other user interface to appear to be displayed in a second location in the physical environment away from the electronic device. The system process of the electronic device detects a first indirect participation indicator performed by the user of the electronic device, the first indirect participation indicator indicating a first user action occurring away from the first location where the user interface is displayed; The system process detects a second indirect participation indicator performed by the user of the electronic device, the second indirect participation indicator corresponding to a second user action occurring away from the first location where the user interface is displayed; Before determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application, no user input corresponding to at least one of the first indirect participation indicator or the second indirect participation indicator is provided to the application; The system process determines that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application; as well as In response to determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application: When a direct engagement gesture is detected simultaneously with the first and second indirect engagement indicators, and the direct engagement gesture occurs at the second location, the system process provides additional user input to the other application based on the direct engagement gesture, without providing the application with any information associated with the first or second indirect engagement indicator; otherwise... The system process provides the user input to the application based on at least one of the first indirect participation indicator or the second indirect participation indicator.

13. The electronic device according to claim 12, wherein, Detecting the first indirect participation indicator includes detecting the first indirect participation indicator while detecting the second indirect participation indicator.

14. The electronic device according to claim 12, wherein, The one or more processors are configured to determine, in part, that the first indirect engagement indicator is associated with the user interface based on a ray extending through the user interface from a portion of the user, thereby partially determining that both the first indirect engagement indicator and the second indirect engagement indicator are associated with the user interface of the application.

15. The electronic device according to claim 12, wherein, The one or more processors are further configured to: The system process detects a third indirect participation indicator executed by the user; The system process detects the fourth indirect participation indicator executed by the user; The system process determines that the third indirect participation indicator is associated with the user interface of the application; The system process determines that the fourth indirect participation indicator is not associated with the user interface of the application. as well as Continue without providing the application with any user input associated with the third or fourth indirect participation indicator.

16. The electronic device of claim 12, wherein the direct engagement gesture includes a gesture of tapping or touching the other user interface within the physical environment.

17. The electronic device according to claim 12, wherein, The first indirect engagement indicator includes a gaze position, and the second indirect engagement indicator includes a gesture or voice command.

18. The electronic device according to claim 17, wherein, The user input includes gesture input based on the gesture.

19. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations including the following: Displaying a representation of the user interface of an application and another user interface of another application on the display of an electronic device, the application causing the user interface to appear in a first location in the physical environment away from the electronic device, and the other application causing the other user interface to appear to appear to be displayed in a second location in the physical environment away from the electronic device; The system process of the electronic device detects a first indirect participation indicator performed by the user of the electronic device, the first indirect participation indicator corresponding to a first user action occurring away from the first location where the user interface is displayed; The system process detects a second indirect participation indicator performed by the user of the electronic device, the second indirect participation indicator corresponding to a second user action occurring away from the first location where the user interface is displayed; Before determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application, no user input corresponding to at least one of the first indirect participation indicator or the second indirect participation indicator is provided to the application; The system process determines that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application; and In response to determining that both the first indirect participation indicator and the second indirect participation indicator are associated with the user interface of the application: When a direct engagement gesture is detected simultaneously with the first and second indirect engagement indicators, and the direct engagement gesture occurs at the second location, the system process provides additional user input to the other application based on the direct engagement gesture, without providing the application with any information associated with the first or second indirect engagement indicator; otherwise... The system process provides the user input to the application based on at least one of the first indirect participation indicator or the second indirect participation indicator.

Images