Device and method for presenting virtual object in virtual environment, and graphical user interface

Abstract

To provide a method for facilitating display and interaction of a user interface of an electronic device in a three-dimensional environment, an electronic device, and a program.SOLUTION: A method to be executed by a first electronic device 401 having a display and an input device includes: displaying a three-dimensional environment 450 on the display; presenting a visual representation 415' of a second electronic device in the three-dimensional environment while in a communication session with the second electronic device; receiving, from the second electronic device, a request to share content with the first electronic device; displaying a first representation 411' of a first user interface corresponding to the content, in the visual representation of the second device in the three-dimensional environment; detecting an input instruction 473B to the first user interface; and foregoing performing an operation directed to the first representation of the first user interface.SELECTED DRAWING: Figure 4C

Description

Cross-reference to related applications

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 498,239, filed April 25, 2023, U.S. Patent Application No. 18 / 429,294, filed January 31, 2024, and U.S. Patent Application No. 18 / 429,301, filed January 31, 2024, the contents of which are incorporated herein by reference in their entirety for all purposes. [Technical Field]

[0002] The present invention generally relates to systems and methods for displaying a representation of a user interface of an electronic device and facilitating interaction with the representation of the user interface of the electronic device. [Background technology]

[0003] Many types of computing systems are available, such as mobile phones, tablets, touchpads, portable or desktop computers, portable media players, and wearable devices. Computing systems may include displays or touchscreens to allow user interaction. However, these computing systems are limited by the physical constraints of the displays or touchscreens. Summary of the Invention

[0004] Some examples of the present disclosure are directed to systems and methods for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment. In some examples, the representation of the user interface is displayed within a virtual augmented display of the electronic device within the three-dimensional environment. In some examples, a first electronic device may be in communication with a display of the first electronic device and one or more input devices. In some examples, the first electronic device captures at least a portion of a real-world environment surrounding the first electronic device via the one or more input devices, and the real-world environment includes a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. In some examples, the second electronic device is in communication with the first electronic device. In some examples, the first electronic device presents, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. In some examples, the first affordance is displayed in a predetermined area of the display of the first electronic device. In some examples, while presenting a computer-generated environment including a representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, an input corresponding to a selection of the first affordance. In some examples, in response to receiving the input, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of a second electronic device that differs from the first representation of the first user interface. In some examples, the second representation of the second user interface is displayed on a virtual augmented display in the computer-generated environment.

[0005] In some examples, while presenting a computer-generated environment including a second representation of a second user interface, the first electronic device detects a first indication of a first individual input received via one or more input devices of the second electronic device. In some examples, in response to detecting the first indication, the second electronic device performs a first action directed to the first user interface in accordance with the first individual input in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface. In some examples, the second electronic device performs a second action directed to the second representation of the second user interface in accordance with the first individual input in accordance with a determination that the gaze is directed to the second representation of the second user interface.

[0006] A full description of these examples is set forth in the Drawings and Detailed Description, and it should be understood that this Summary is not intended to limit the scope of the present disclosure in any way. [Brief explanation of the drawings]

[0007] For a better understanding of the various examples described herein, reference should be made to the following detailed description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0008] [Figure 1] 1 illustrates an electronic device presenting an augmented reality environment, according to some examples of the present disclosure.

[0009] [Figure 2] 1 illustrates a block diagram of an exemplary architecture of a system or device according to an embodiment of the present disclosure.

[0010] [Figure 3A] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3B]1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3C] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3D] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3E] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3F] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3G] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3H] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3I] 1 illustrates an exemplary interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0011] [Figure 4A] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4B] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4C]1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4D] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4E] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4F] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4G] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4H] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4I] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4J] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4K] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4L] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4M]1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4N] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4O] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0012] [Figure 5A] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5B] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5C] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5D] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5E] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5F] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5G] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure.

[0013] [Figure 6A]1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6B] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6C] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6D] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6E] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6F] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6G] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6H] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6I] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6J]1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6K] 1 illustrates an example of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure.

[0014] [Figure 7A] 1 is a flow diagram illustrating an example process for facilitating displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 7B] 1 is a flow diagram illustrating an example process for facilitating displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0015] [Figure 8] 1 is a flow diagram illustrating an example process for facilitating display and interaction of representations of a user interface between two electronic devices in a three-dimensional environment, according to some examples of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] Some examples of the present disclosure are directed to systems and methods for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment. In some examples, the representation of the user interface is displayed within a virtual augmented display of the electronic device within the three-dimensional environment. In some examples, a first electronic device may be in communication with a display of the first electronic device and one or more input devices. In some examples, the first electronic device captures at least a portion of a real-world environment surrounding the first electronic device via the one or more input devices, and the real-world environment includes a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. In some examples, the second electronic device is in communication with the first electronic device. In some examples, the first electronic device presents, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. In some examples, the first affordance is displayed in a predetermined area of the display of the first electronic device. In some examples, while presenting a computer-generated environment including a representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, an input corresponding to a selection of the first affordance. In some examples, in response to receiving the input, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of a second electronic device that differs from the first representation of the first user interface. In some examples, the second representation of the second user interface is displayed on a virtual augmented display in the computer-generated environment.

[0017] In some examples, while presenting a computer-generated environment including a second representation of a second user interface, the first electronic device detects a first indication of a first individual input received via one or more input devices of the second electronic device. In some examples, in response to detecting the first indication, the second electronic device performs a first action directed to the first user interface in accordance with the first individual input in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface. In some examples, the second electronic device performs a second action directed to the second representation of the second user interface in accordance with the first individual input in accordance with a determination that the gaze is directed to the second representation of the second user interface.

[0018] In some examples, displaying a virtual augmented display in a three-dimensional (e.g., computer-generated) environment configured to display a user interface of a real-world electronic device may include interaction with one or more user interface objects in the three-dimensional environment. For example, initiating the display of the virtual augmented display in the three-dimensional environment may include interaction with one or more virtual options / affordances displayed in the three-dimensional environment. In some examples, a user's gaze may be tracked by the electronic device as input to identify one or more virtual options / affordances for selection when initiating the display of the virtual augmented display in the three-dimensional environment. For example, the gaze may be used to identify one or more virtual options / affordances for selection using another selection input. In some examples, the virtual options / affordances may be selected using hand-tracking input detected via an input device in communication with the electronic device. In some examples, objects displayed in the three-dimensional environment may be moved and / or reoriented within the three-dimensional environment according to movement input detected via the input device.

[0019] FIG. 1 illustrates an electronic device 101 presenting an augmented reality (XR) environment (e.g., a computer-generated environment) according to some examples of the present disclosure. In some examples, the electronic device 101 is a handheld or mobile device such as a tablet computer, a laptop computer, a smartphone, or a head-mounted display. Examples of the electronic device 101 are described below with reference to the architectural block diagram of FIG. 2. As shown in FIG. 1, the electronic device 101, a table 106, a computer 115, and a floor lamp 132 are located in a physical environment 100. The physical environment may include physical features such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lamps, etc.). In some examples, the electronic device 101 may be configured to capture an image of the physical environment 100 including the table 106, the computer 115, and the floor lamp 132 (shown within the field of view of the electronic device 101). In some examples, in response to a trigger, electronic device 101 may be configured to display within the computer-generated environment (e.g., represented by the rectangle shown in FIG. 1 ) a virtual object 104 (e.g., two-dimensional virtual content) that is not present within physical environment 100 but is displayed within the computer-generated environment positioned on top of (e.g., fixed to) a computer-generated representation 106′ of a real-world table 106. For example, in response to detecting a flat surface of table 106 within physical environment 100 (or some other trigger), virtual object 104 may be displayed on a surface of computer-generated representation 115′ of a table in the computer-generated environment next to computer-generated representation 106′ of a real-world table 115 displayed via electronic device 101.

[0020] It should be understood that virtual object 104 is exemplary of a virtual object, and that one or more different virtual objects (e.g., of various dimensions, such as two-dimensional or three-dimensional objects) may be included and rendered in a two-dimensional or three-dimensional computer-generated environment. For example, the virtual object may represent an application or a user interface displayed within the computer-generated environment. In some examples, the virtual object may represent content corresponding to an application and / or displayed via a user interface in the computer-generated environment. In some examples, the user interface is also configured to be displayed via (e.g., a display of) computer 115 in physical environment 100. In some examples, virtual object 104 is interactive and optionally configured to respond to user input, such that a user may virtually touch, tap, move, rotate, or otherwise interact with virtual object 104. Additionally, it should be understood that the 3D environment (or 3D virtual object) described herein may be a representation of a 3D environment (or 3D virtual object) projected or presented on an electronic device.

[0021] In the following description, an electronic device is described that communicates with display generating components and one or more input devices. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand-tracking device, an eye-tracking device, a stylus, etc. It should also be understood that, as noted above, the described electronic device, display, and touch-sensitive surface are optionally distributed across two or more devices. Thus, as used in this disclosure, information displayed on or by an electronic device may optionally be used to describe information output by the electronic device for display on another display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received at an electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device or touch input received on the surface of a stylus) is optionally used to describe input received at a separate input device from which the electronic device receives input information.

[0022] The device typically supports a variety of applications, such as one or more of a drawing application, a presentation application, a word processing application, a website creation application, a disc authoring application, a spreadsheet application, a gaming application, a telephony application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, a web browsing application, a digital music playback application, a television channel browsing application, and / or a digital video playback application.

[0023] 2 shows a block diagram of an exemplary architecture of a system or device 200 according to examples of the present disclosure. In some examples, system 200 is a mobile device, such as a mobile phone (e.g., a smartphone), a tablet computer, a laptop computer, a desktop computer, a head-mounted display, an auxiliary device that communicates with another device, etc. System 200 optionally includes various sensors (e.g., one or more hand-tracking sensor(s), one or more location sensor(s), one or more image sensor(s), one or more touch-sensitive surface(s), one or more movement and / or orientation sensor(s), one or more eye-tracking sensor(s), one or more microphone(s) or other audio sensor(s), etc.), one or more display generation component(s), one or more speaker(s), one or more processor(s), one or more memories, and / or communications circuitry. One or more communications buses are optionally used for communication between the above-mentioned components of system 200.

[0024] 2, system / device 200 may be divided among multiple devices. For example, first device 230 optionally includes processor(s) 218A, one or more memories 220A, communications circuitry 222A, touch-sensitive surface(s) 216A, and display generating component(s) or display(s) 214A, which optionally communicate via communications bus 208A. Second device 240 (e.g., corresponding to electronic device 101) optionally includes various sensors (e.g., one or more hand tracking sensor(s) 202, one or more location sensor(s) 204, one or more image sensor(s) 206, one or more touch-sensitive surface(s) 216B, one or more movement and / or orientation sensor(s) 210, one or more eye tracking sensor(s) 212, one or more microphone(s) 213 or other audio sensor(s), etc.), one or more display generating component(s) 214B, one or more speaker(s) 216, one or more processors 218B, one or more memories 220B, and / or communications circuitry 222B. One or more communications buses 208B are optionally used for communication between the above-mentioned components of device 240. First device 230 and second device 240 optionally communicate via a wired or wireless connection between the two devices (eg, via communication circuitry 222A-222B).

[0025] Communications circuitry 222A, 222B optionally includes circuitry for communicating with electronic devices, networks such as the Internet, an intranet, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). Communications circuitry 222A, 222B optionally includes circuitry for communicating using short-range communications such as near-field communications (NFC) and / or Bluetooth.

[0026] The processor(s) 218A, 218B include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, the memory 220A, 220B is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by the processor(s) 218A, 218B to perform the techniques, processes, and / or methods described below. In some examples, the memory 220A, 220B can include two or more non-transitory computer-readable storage media. A non-transitory computer-readable storage medium can be any medium (other than a signal) that can tangibly store or carry computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a transient computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media may include, but are not limited to, magnetic, optical, and / or semiconductor storage devices, such as magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory, such as flash and solid-state drives.

[0027] In some examples, display generating component(s) 214A, 214B include a single display (e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED), or other type of display). In some examples, display generating component(s) 214A, 214B include multiple displays. In some examples, display generating component(s) 214A, 214B can include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retina projector, etc. In some examples, device 240 includes touch-sensitive surface(s) 216B for receiving user inputs such as tap and swipe inputs or other gestures. In some examples, display generating component(s) 214A, 214B and touch-sensitive surfaces 216A, 216B form a touch-sensitive display (e.g., a touchscreen integrated with devices 230, 240 or a touchscreen external to devices 230, 240 that communicates with devices 230, 240).

[0028] Device 240 optionally includes image sensor(s) 206. Image sensor(s) 206 optionally include one or more visible light image sensors, such as a charge-coupled device (CCD) sensor and / or a complementary metal-oxide semiconductor (CMOS) sensor, operable to acquire images of physical objects from the real-world environment. Image sensor(s) 206 also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor(s) 206 also optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. Image sensor(s) 206 optionally include one or more depth sensors configured to detect the distance of physical objects from device 240. In some examples, information from one or more depth sensors may enable the device to identify an object in the real-world environment and distinguish it from other objects in the real-world environment. In some examples, the one or more depth sensors may enable the device to determine the texture and / or topography of an object in the real-world environment.

[0029] In some examples, device 240 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around device 240. In some examples, image sensor(s) 206 includes a first image sensor and a second image sensor. The first image sensor and the second image sensor cooperate and are optionally configured to capture different information of physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, device 240 uses image sensor(s) 206 to detect the position and orientation of device 240 and / or display generating component(s) 214B within the real-world environment. For example, device 240 uses image sensor(s) 206 to track the position and orientation of display generating component(s) 214B relative to one or more fixed objects in the real-world environment.

[0030] In some examples, device 240 includes microphone(s) 213 or other audio sensors. Device 240 uses microphone(s) 213 to detect sounds from the user and / or the user's real-world environment. In some examples, microphone(s) 213 optionally include an array of microphones (including multiple microphones) working in concert, such as to identify ambient noise or to identify the location of a sound source within the space of the real-world environment.

[0031] Device 240 optionally includes location sensor(s) 204 configured to detect the location of device 240 and / or display generating component(s) 214B. For example, location sensor(s) 204 may include a GPS receiver that receives data from one or more satellites, allowing device 240 to determine the absolute position of the device in the physical world.

[0032] Device 240 includes orientation sensor(s) 210 that detect the orientation and / or movement of device 240 and / or display generating component(s) 214B. For example, device 240 uses orientation sensor 210 to track changes in the position and / or orientation of device 240 and / or display generating component(s) 214B with respect to physical objects in a real-world environment, etc. Orientation sensor(s) 210 optionally include one or more gyroscopes and / or one or more accelerometers.

[0033] In some examples, device 240 optionally includes hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212. The hand tracking sensor(s) 202 are configured to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of the user's hand relative to the augmented reality environment, relative to display generating component(s) 214B, and / or relative to another defined coordinate system. The eye tracking sensor(s) 212 are configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or the augmented reality environment and / or relative to display generating component(s) 214B. In some examples, hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212 are implemented together with display generating component(s) 214B. In some examples, the hand tracking sensor(s) 202 and / or the eye tracking sensor(s) 212 are implemented separately from the display generation component(s) 214B.

[0034] In some examples, the hand tracking sensor(s) 202 can use image sensor(s) 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real world, including one or more hands (e.g., of a human user). In some examples, the hand can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some examples, the one or more image sensor(s) 206 are positioned relative to the user such that the field of view of the image sensor(s) 206 and an interaction space in which the position, orientation, and / or movement of the fingers / hands captured by the image sensor(s) are used as input (e.g., to distinguish from the user's stationary hand or other hands of other people in the real-world environment) are used as input. Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) can be advantageous in that it does not require the user to touch, hold, or wear any kind of beacon, sensor, or other marker.

[0035] In some examples, the eye tracking sensor(s) 212 include at least one eye tracking camera (e.g., infrared (IR) camera) and / or illumination source (e.g., IR light source, LED, etc.) that emits light toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR light from the light source directly or indirectly from the eyes. In some examples, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and focus / gaze can be determined from tracking of both eyes. In some examples, one eye (e.g., dominant eye) is tracked by a separate eye tracking camera / illumination source(s).

[0036] System 200 is not limited to the components and configuration of FIG. 2 and may include fewer, alternative, or additional components in multiple configurations. In some examples, system 200 may be implemented in a single device. A person using system 200 is optionally referred to herein as a user of the system (e.g., and / or a user of the device). Attention is now directed to an exemplary interaction involving initiating the display of a virtual augmented display associated with a physical electronic device visible in a three-dimensional environment. As described below, the three-dimensional environment may be displayed using a first electronic device, and a second electronic device different from the first electronic device may be located within the physical environment included in the three-dimensional environment, with the first electronic device configured to present a virtual augmented display of the second electronic device. In some examples, processes for initiating the display of the virtual augmented display and facilitating interaction with the virtual augmented display in the three-dimensional environment, as described below, may be performed by processors 218A, 218B of devices 230 and 240.

[0037] 3A-3I illustrate exemplary interactions for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, a first electronic device 301 may present a three-dimensional environment 350. The first electronic device 301 may be similar to the electronic devices 101 or 240 described above and / or may be a head-mounted system / device and / or a projection-based system / device (including a hologram-based system / device) configured to generate and present a three-dimensional environment, such as, for example, a head-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, or a display formed as a lens designed to be placed on a person's eye (e.g., similar to contact lenses). In the examples of FIGS. 3A-3I, a user is optionally wearing the first electronic device 301 such that the three-dimensional environment 350 may be defined by an X-axis, a Y-axis, and a Z-axis as seen from the perspective of the first electronic device (e.g., a perspective associated with the user of the electronic device 301). Thus, as used herein, the first electronic device 301 is configured to be movable in six degrees of freedom based on movement of a user (e.g., the user's head) such that the first electronic device 301 can be moved in a roll direction, a pitch direction, and / or a yaw direction.

[0038] 3A , a first electronic device 301 may be placed in a physical environment including a table 306 (e.g., corresponding to the real-world table 106 of FIG. 1 ), a second electronic device 315, and one or more physical input devices 331 / 332. Thus, a three-dimensional environment 350 presented using the first electronic device 301 optionally includes captured portions of the physical environment surrounding the electronic device 301, such as a representation of the table 306′, a representation of the second electronic device 315′, and one or more representations of the one or more physical input devices 331′ / 332′. In some examples, the second electronic device 315′ may be similar to the device 230 described above and / or may be a desktop computer, a laptop computer, a tablet computer, or other mobile computing device. In some examples, the one or more physical input devices may include a keyboard 331′ and a mouse 332′ (or trackpad) that communicate with the second electronic device 315′ (e.g., integrated with the second electronic device 315′ or communicating with the second electronic device 315′ via wired or wireless communication means). It should be understood that the input devices in FIG. 3A (e.g., keyboard 331′ and mouse 332′) are merely representative, and that additional, fewer, or different input devices may communicate with the second electronic device 315′. In some examples, the representation may include portions of the physical environment as seen through a transparent or translucent display of the first electronic device 301.

[0039] In some examples, as shown in FIG. 3A , a second electronic device 315′ within the three-dimensional environment 350 is configured to display one or more user interfaces. For example, as shown in FIG. 3A , the second electronic device 315′ includes a display that is visible within the three-dimensional environment 350 as a representation of display 314′. In some examples, the display 314′ displays a first user interface (e.g., “Window A”), which is visible in the three-dimensional environment 350 as a representation of first user interface 311′. In some examples, the first user interface 311′ is a user interface of a web browsing application (e.g., “Browser”) that includes a search bar that is visible in the three-dimensional environment 350 as a representation of search bar 313′, as shown in FIG. 3A . Additionally, as shown in FIG. 3A , the first user interface 311′ may be displayed along with a dock that is visible in the three-dimensional environment 350 as a representation of dock 309′ that includes a plurality of icons corresponding to applications, images, or other files local to the second electronic device 315′. Finally, as shown in FIG. 3A , in some examples, the second electronic device 315′ may display a cursor (e.g., or other pointer object) visible in the three-dimensional environment 350 as a representation of a cursor 312′ controllable via a mouse 332′ to provide input directed to user interface elements of the display 314′ (e.g., movable via and / or controllable via the mouse 332′ to provide selection and associated input directed to user interface elements).

[0040] In some examples, it may be advantageous to provide a fully functional work environment in three dimensions that is representative of customary interaction with a second electronic device in a real-world environment. Accordingly, providing a virtual augmented display configured to include a representation of one or more user interfaces of a second electronic device in a three-dimensional environment may be particularly useful for interacting with one or more user interfaces of a second electronic device without the constraints of the second electronic device's physical display (e.g., display 314′). For example, as described herein, the virtual augmented display may provide for displaying one or more user interfaces of a second electronic device at a size that exceeds the dimensions of the second electronic device's physical display (e.g., enlarging the user interface or providing multiple areas for display of the user interface beyond the area corresponding to the second electronic device's display). In some examples, it may be advantageous to present one or more objects of the second electronic device (e.g., objects corresponding to three-dimensional graphical data) in three dimensions. For example, two-dimensional content can be viewed or edited in three dimensions, while the display of the second electronic device is limited to two dimensions. In such examples, a user creating content for display in a three-dimensional environment may view the content in three dimensions via the virtual augmented display. In some examples, as described in more detail herein, it may be advantageous to facilitate interaction with content of the virtual augmented display based on input detected via a physical input device in communication with the second electronic device.

[0041] In some examples, the first electronic device 301 may provide pairing with the second electronic device 315′ to initiate display of a virtual augmented display associated with the second electronic device 315′ within the three-dimensional environment 350. For example, as shown in FIG. 3B , the first electronic device 301 may display a user interface object 317 associated with a separate application (e.g., a pairing application or a settings application running on the first electronic device 301) that provides pairing the first electronic device 301 with one or more other electronic devices. In some examples, as shown in FIG. 3B , the user interface object 317 includes a first indication 317-1 corresponding to “Device 1.” In some examples, Device 1 corresponds to the second electronic device 301′ that is not currently paired with the first electronic device 315 in FIG. 3B . In some examples, the first instruction 317-1 is included in the user interface object 317 because the second electronic device 301′ is within a threshold distance (e.g., 0.1, 0.25, 0.5, 1, 2, 3, 5, 10, 15, 20 m, etc.) of the first electronic device 315. In some examples, the first instruction 317-1 is included in the user interface object 317 because the second electronic device 315′ is visible within a field of view of the first electronic device 301 or a user of the first electronic device 301 (e.g., the second electronic device 315′ is located in a portion of the physical environment visible within the three-dimensional environment 350 from the user's current viewpoint). In some examples, the first instruction 317-1 is selectable to initiate a process for pairing the second electronic device 315′ with the first electronic device 301 to enable the first electronic device 301 to present a virtual augmented display associated with the second electronic device 315′.

[0042] 3B , the first electronic device 301 detects a selection input 371A directed toward the first instruction 317-1. For example, in FIG. 3B , the first electronic device 301 detects an air pinch gesture (e.g., the index finger and thumb of the user's hand touching together), a tap or touch input (e.g., provided by the index finger of the hand), a verbal command, or other input, optionally while the user's gaze is directed toward the first instruction 317-1. In some examples, as described below, in response to detecting the selection of the first instruction 317-1, the first electronic device 301 initiates a process for pairing the second electronic device 315′ with the first electronic device 301.

[0043] In some examples, initiating the process for pairing the second electronic device 315′ with the first electronic device 301 includes sending a pairing request to the second electronic device 315′ (e.g., directly to the second electronic device 315′ or indirectly via a server (e.g., a wireless communication terminal in communication with the second electronic device 315′)). In some examples, in response to receiving the pairing request, the second electronic device 315′ transmits / outputs data to the first electronic device 301 (e.g., directly or indirectly) to pair the first electronic device 301 with the second electronic device 315′. For example, as shown in FIG. 3B , the second electronic device 315′ may transmit a representation of content displayed on the display 314′, thereby allowing the first electronic device 301 to visually identify the second electronic device 315′ by matching the received representation with a portion of an image captured by the first electronic device 301. 3C , the second electronic device 315′ displays a unique identifier via the display 314′, which is associated with the second electronic device 315′ and is visible within the three-dimensional environment 350 as a representation of a unique identifier 321′ that is detectable by the first electronic device 301. As shown in FIG. 3C , the unique identifier 321′ is, optionally, a machine-readable image, such as a QR code, barcode, or other machine-readable code, that stores identification data of the second electronic device 301′ that is detectable via one or more image sensors or cameras of the first electronic device 315. In some examples, the unique identifier 321′ is optionally displayed at a frequency that allows the unique identifier 321′ to be detectable by the first electronic device 301 but invisible to a user of the first electronic device 301. In some examples, as shown in FIG. 3C , when the second electronic device 315′ displays the unique identifier 321′, the second electronic device 315′ sends instructions to the first electronic device 301 to prompt the user of the first electronic device 301 to identify the unique identifier 321′.3C , the instruction may cause the first electronic device 301 to display a message element 320 prompting the user of the first electronic device 301 to direct their attention (e.g., including their gaze) to the unique identifier 321′ displayed on the second electronic device 315′ (e.g., “Scan code on device 1”) so that the first electronic device 301 may scan / access the identification data contained in the unique identifier 321′. In some examples, once the user's attention is directed to the unique identifier 321′ (and / or the unique identifier 321′ remains within the user's field of view for more than a threshold amount of time, such as 1, 2, 3, 5, 10, 30, 45, 60 seconds), such that the first electronic device 301 scans the unique identifier 321′, the first electronic device 301 transmits confirmation data to the second electronic device 315′ that pairs the first electronic device 315 with the second electronic device 301′.

[0044] Additionally or alternatively, in some examples, the second electronic device 315′ outputs a sound detectable by the first electronic device 301 to pair the first electronic device 301 with the second electronic device 315′. For example, as shown in FIG. 3C , the second electronic device 315′ outputs a sound 322, which may be output at a frequency detectable by the first electronic device 301 (e.g., which may or may not be audible to a user of the first electronic device 301). In some examples, once the sound 322 is detected by the first electronic device 301, the first electronic device 301 transmits confirmation data to the second electronic device 315′, which pairs the first electronic device 301 with the second electronic device 315′. Additionally or alternatively, in some examples, the second electronic device 315′ emits infrared light 326, such as via a display 314′, that is detectable by the first electronic device 301 to pair the first electronic device 301 with the second electronic device 315′. In some examples, the infrared light 326 may be invisible to a user of the first electronic device 301. In some examples, once the infrared light 326 is detected by the first electronic device 301 (e.g., via one or more infrared sensors or other image sensors), the first electronic device 301 transmits confirmation data to the second electronic device 315′ that pairs the first electronic device 301 with the second electronic device 315′.

[0045] In some examples, as shown in FIG. 3D , the second electronic device 315′ sends a request to the first electronic device 315 prompting the user of the first electronic device 301 to provide an input detectable by the second electronic device 315′ to pair the first electronic device 301 with the second electronic device 301′. For example, as shown in FIG. 3D , the request causes the first electronic device 301 to display an instruction element 320A and / or an instruction element 320B prompting the user of the first electronic device 301 to provide an input via one or more input devices 331′ / 332′. As an example, in FIG. 3D , the instruction element 320A includes an instruction (e.g., “press a key on the keyboard”) prompting the user of the first electronic device 301 to provide a selection input directed to a keyboard 331′ in communication with the second electronic device 315′. In some examples, the user is prompted to select a particular key on the keyboard 331′. In some examples, as shown in FIG. 3D , the instruction element 320B includes an instruction (e.g., “click mouse”) that prompts the user of the first electronic device 301 to provide a selection input directed to the mouse 332′ that is in communication with the second electronic device 315′.

[0046] 3D, the second electronic device 315′ detects a selection input 373A directed to the keyboard 331′ and / or a selection input 373B directed to the mouse 332′. For example, the second electronic device 315′ detects a key press on the keyboard 331′ and / or a button press on the mouse 332′ (or a tap on the touch-sensitive surface of the mouse 332′) provided by a user of the first electronic device 301. In some examples, as shown in FIG. 3E, in response to detecting the selection input 373A and / or the selection input 373B, the second electronic device 315′ provides confirmation that the selection input 373A and / or the selection input 373B was detected. For example, as shown in FIG. 3E , the second electronic device 315′ displays a confirmation message via display 314′, which can be seen in the three-dimensional environment 350 as confirmation message 323′, indicating that the above-mentioned input has been successfully detected (e.g., “input detected”), thereby allowing the first electronic device 301 to be paired with the second electronic device 315′.

[0047] In some examples, the first electronic device 301 initiates a process for pairing the second electronic device 315′ with the first electronic device 301, according to the above examples, in accordance with a determination that one or more criteria are met. In some examples, the one or more criteria include a criterion that is met in accordance with a determination that the second electronic device 315′ is in a state for displaying a user interface via a virtual augmented display. In some examples, the one or more criteria include a criterion that is met in accordance with a determination that the second electronic device 315′ is detected by one or more sensors of the first electronic device 301 (e.g., when the second electronic device is captured by one or more image sensors or cameras of the first electronic device). In some examples, the one or more criteria include a criterion that is met in accordance with a determination that the second electronic device 315′ is configured to be able to communicate with the first electronic device 301 (e.g., when a wired or wireless communication channel is established). In some examples, the one or more criteria include criteria that are met pursuant to a determination that the second electronic device 315′ is an eligible electronic device (e.g., some electronic devices may have the capability to operate in the context of an augmented virtual display, while others may not). In some examples, the one or more criteria include criteria that are met pursuant to a determination that the first and second electronic devices are permitted to share information (e.g., privacy settings, user information, user data, etc.), such as when the first electronic device 301 and the second electronic device 315′ belong to the same user and / or are associated with the same user account logged into the electronic devices. In some examples, some or all of the above criteria must be met in order for the first electronic device 301 to be paired with the second electronic device 315′.

[0048] In some examples, the first electronic device 301 may be paired with the second electronic device 315′ without following the exemplary process described above. For example, in FIG. 3B , in response to detecting selection of the first instruction 317-1, the first electronic device 301 sends a request to pair with the second electronic device 315′, and if one or more criteria described above are met, the first electronic device 301 automatically pairs with the second electronic device 315′. In some examples, the first electronic device 301 and the second electronic device 315′ may be paired via any suitable communication means, such as Bluetooth, Wi-Fi, etc.

[0049] 3F , when the first electronic device 301 is paired with the second electronic device 315′ as described above, the first electronic device 301 displays a message element 320 (e.g., a notification) alerting the user of the first electronic device 301 that the first electronic device 301 is paired with the second electronic device 315′ (e.g., “Device 1 Paired”). In some examples, after the first electronic device 301 is paired with the second electronic device 315′, the first electronic device 301 initiates a process to display a virtual extended display associated with the second electronic device 315′. 3G , while the first electronic device 301 is in communication with the second electronic device 315′ and the second electronic device 315′ is within a field of view of a user of the first electronic device 301, the first electronic device 301 displays a first affordance 319 within the three-dimensional environment 350, which is selectable to display a virtual augmented display of the second electronic device 315′. In some examples, as shown in FIG. 3G , the first affordance 319 may be presented in proximity to an edge of the display of the second electronic device 315′ (e.g., within a threshold distance (e.g., 1, 2, 3, 5, 8, 10, 15, 20, 30 cm, etc.) of the edge of the display). 3G , the first affordance 319 is presented above the second electronic device 315′ (e.g., within a threshold distance of a corner of the display 314′ of the second electronic device 315′) within the three-dimensional environment 350. In some examples, the first affordance 319 may be presented at a depth within the three-dimensional environment 350 so that it appears in the same plane as the display 314′ of the second electronic device 315′.

[0050] In some examples, the first affordance 319 may be positioned differently within the three-dimensional environment 350. For example, the first affordance 319 may be presented above, near, below, or in front of the second electronic device 315′, and / or at another location within the three-dimensional environment 350 (e.g., a specified distance away from the second electronic device). Additionally or alternatively, in some examples, the first affordance may be displayed in a specified area of the display of the first electronic device 301 (e.g., regardless of the location of the second electronic device 315′ in the three-dimensional environment 350). For example, as shown in FIG. 3G , the first affordance may be displayed as a first system option 318A within the plurality of system options 318. In some examples, as shown in FIG. 3G , the plurality of system options 318 are displayed centrally along the top edge of the display of the first electronic device 301. In some examples, multiple system options 318 other than first system option 318A are selectable to access one or more settings of first electronic device 301, such as display settings (e.g., brightness level and / or immersive content level) and / or audio settings (e.g., volume level), and / or to access one or more applications of first electronic device 301, such as via a home user interface.

[0051] In some examples, the second electronic device 315′ may present, via the display 314′, a selectable affordance (e.g., using the keyboard 331′, the mouse 332′, input from the first electronic device 301, etc.) to initiate display of a virtual extended display by the second electronic device 315. For example, the second electronic device 315′ may provide an affordance different from, similar to, or identical to the first affordance 319 within a selectable setting or other application to cause the first electronic device 301 to send a request to the second electronic device 315′ to display a virtual extended display.

[0052] In some examples, as shown in FIG. 3G , the physical environment surrounding the first electronic device 301 may include a third electronic device 325 positioned on a coffee table 308. For example, a representation of the third electronic device 325′ and a representation of the coffee table 308′ are visible in a three-dimensional environment 350, as shown in FIG. 3G . In some examples, the third electronic device 325 may be a mobile electronic device, such as a cellular electronic device or a tablet computer. As shown in FIG. 3G , the third electronic device 325 is optionally not paired with the first electronic device 301. Alternatively, in FIG. 3G , the third electronic device 325 is optionally paired with the first electronic device 301 but may not have the capability to operate in the context of an augmented virtual display. Thus, as shown in FIG. 3G, the representation of the third electronic device 325′ is not displayed with a selectable affordance (e.g., adjacent to the representation of the third electronic device 325′ or among multiple system options 318) to display an extended virtual display of the third electronic device 325.

[0053] 3H, the first electronic device 301 detects a selection input 371B directed toward the first system option 318A described above. For example, the first electronic device 301 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell, etc. corresponding to the selection of the first system option 318A. In some examples, as shown in FIG. 3I, in response to detecting a selection of the first system option 318A, the first affordance 319, or another affordance for initiating a virtual augmented display, the first electronic device 301 presents a virtual augmented display 304 associated with the second electronic device 315′ within the three-dimensional environment 350 (e.g., not associated with the third electronic device 325′ described above).

[0054] In some examples, as shown in FIG. 3I, the virtual extended display 304 functions as a second (e.g., virtual) display for the second electronic device 315′. For example, as shown in FIG. 3I, the virtual extended display includes a representation of the dock 309″ displayed on the display 314′ of the second electronic device 315′. In some examples, as described herein, the virtual extended display 304 is configured to display one or more user interfaces (e.g., application windows), content, and / or other user interface elements of the second electronic device 315′, which may or may not be displayed on the display 314′. Additionally, in some examples, the cursor 312′ is controllable via the mouse 332′ such that the cursor 312′ is moveable between the display 314′ of the second electronic device 315′ and the virtual extended display 304. In some examples, as described later herein, user input detected via keyboard 331′ and / or mouse 332′ may be directed to a user interface on display 314′ or a user interface on virtual augmented display 304 depending on the location of the user's gaze on first electronic device 301. Attention is now directed to exemplary interactions with a virtual augmented display for an electronic device in a three-dimensional environment.

[0055] In some examples, when the first electronic device 301 presents the virtual augmented display 304 in the three-dimensional environment 350, the display 314' of the second electronic device 315' may be turned off (e.g., the second electronic device 315' enters a low-power mode or sleep state, or is powered off, so that the second electronic device 315' no longer displays a user interface via the display 314'). For example, after the first electronic device 301 is paired with the second electronic device 315' and after the first electronic device 301 generates and displays the virtual augmented display 304 in the three-dimensional environment 350, the second electronic device 315' (e.g., automatically) powers off the display 314'. In some examples, the display 314' of the second electronic device 315' may be turned off when the first electronic device 301 initiates the above-described process for pairing the first electronic device 301 and the second electronic device 315'. In such a scenario, the display 314' of the second electronic device 315' may be powered on for the pairing process (eg, to display the unique identifier 321' and / or the confirmation message 323').

[0056] 4A-4O illustrate examples of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, as shown in FIG. 4A, a first electronic device 401 may be used to present a three-dimensional environment 450. In some examples, the first electronic device 401 optionally corresponds to the first electronic device 301 described above. In some examples, the three-dimensional environment 450 includes a captured portion of a physical environment in which the first electronic device 401 is located. For example, the three-dimensional environment 450 optionally includes a table (e.g., a representation of table 406′), as shown in FIG. 4A. In addition, the three-dimensional environment 450 includes a second electronic device (e.g., a representation of second electronic device 415′) and one or more physical input devices (e.g., one or more representations of one or more input devices 431′ / 432′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, three-dimensional environment 450 optionally corresponds to three-dimensional environment 350 described above. In some examples, second electronic device 415' corresponds to second electronic device 315' described above. In some examples, one or more input devices 431' / 432' correspond to one or more input devices 331' / 332' described above. In some examples, the representation of the physical environment may include a portion of the physical environment that is seen through a transparent or translucent display of first electronic device 401.

[0057] As also described above, in some examples, the first electronic device 401 communicates with the second electronic device 415′. Additionally, in some examples, as shown in FIG. 4A , the first electronic device 401 presents a virtual extended display 404 associated with the second electronic device 415′. For example, FIG. 4A . The virtual extended display 404 serves as a second (e.g., virtual) display for the physical display (e.g., a representation of the display 414′) of the second electronic device 415′. In some examples, the virtual extended display 404 corresponds to the virtual extended display 304 described above. In some examples, as shown in FIG. 4A , the display 414′ of the second electronic device 415′ displays a first user interface (e.g., a representation of the first user interface 411′), and the virtual extended display 404 displays a second user interface 425 configured to be displayed on the display 414′ of the second electronic device 415′. In some examples, the first user interface 411′ and the second user interface 425 may be associated with separate applications (e.g., a web browsing application (“browser”) running on the second electronic device 514′). For example, the first user interface 411′ is a first window (e.g., “Window A”) of the separate application, and the second user interface 425 is a second window (e.g., “Window B”) of the separate application. Additionally, as shown in FIG. 4A , the display 414′ of the second electronic device 415′ is displaying a dock (e.g., a representation of dock 409′), and the virtual extended display 404 is displaying a representation of dock 409″. In some examples, the dock 409′ corresponds to the dock 309′ described above. In some examples, as shown in FIG. 4A , the display 414′ of the second electronic device 415′ includes a cursor (e.g., a representation of cursor 412′) that is controllable using a mouse 432′. In some examples, cursor 412' corresponds to cursor 312' described above.

[0058] In some examples, as shown in FIG. 4A , first user interface 411′ includes a selectable text entry field (e.g., a representation of text entry field 413′) for entering text to provide a search query to a web browser. Additionally, as shown in FIG. 4A , second user interface 425 includes multiple search results provided by the web browser (e.g., associated with the URL “www.Search.com”). For example, in FIG. 4A , second user interface 425 includes a first search result 426A (e.g., “Search Result A”) associated with a first website (e.g., “www.URL1.com”), a second search result 426B (e.g., “Search Result B”) associated with a second website (e.g., “www.URL2.com”), and a third search result 426C (e.g., “Search Result C”) associated with a third website. In some examples, the multiple search results 426 are selectable for accessing content (e.g., text, images, video, music, etc.) via the respective sources (e.g., websites).

[0059] In some examples, the first electronic device 401 displays the second user interface 425 within the virtual augmented display 404 based on image data (e.g., pixel or display data) transmitted to the first electronic device 401 by the second electronic device 415′ (e.g., directly or indirectly via a server). For example, as described above, the second user interface 425 is configured to be displayed on the display 414′ of the second electronic device 415′ (e.g., the second user interface 425 is a user interface of the second electronic device 415′). Thus, in some examples, the first electronic device 401 renders (e.g., generates) the second user interface 425, including content for the second user interface 425 described above, based on image data provided by the second electronic device 415′. For example, the image data includes information corresponding to the appearance of the second user interface 425, characteristics of the content of the second user interface 425 (e.g., whether the content is interactive, selectable, scrollable, etc.), and / or the quality of the second user interface 425 and its content (e.g., image quality, such as definition). In some examples, the image data transmitted by the second electronic device 401′ to the first electronic device 415 includes an image generated by the second electronic device 401′ that is displayed on the virtual extended display 404 by the first electronic device 415.

[0060] In some examples, user input detected via one or more input devices 431′ / 432′ may be selectively routed by the first electronic device 401 based on the location of the gaze of the user of the first electronic device 401 in the three-dimensional environment 450. As used herein, the user of the first electronic device 401 may also be the user of the second electronic device 415′. In FIG. 4B , while the user's gaze 427 is directed toward the first user interface 411′ of the display 414′, the second electronic device 415′ detects input 473A via a mouse 432′ in communication with the second electronic device 415′. For example, as shown in FIG. 4B , the second electronic device 415′ detects the movement of the mouse 432′ over the surface of the table 406′.

[0061] In some examples, as described above, the first electronic device 401 controls the applicability of user input based on the location of the line of sight 427. For example, in response to detecting the input 473A, the second electronic device 415′ transmits input data (e.g., directly or indirectly via a server) to the first electronic device 401, including information corresponding to movement of the mouse 432′ (e.g., corresponding to a request to move the cursor 412′). In some examples, when the first electronic device 401 receives the input data provided by the second electronic device 415′, the first electronic device 401 determines the location of the user's line of sight 427 in the three-dimensional environment 450. As described above and shown in FIG. 4B , when the input 473A is provided via the mouse 432′, the line of sight 427 is directed toward the first user interface 411′ displayed on the display 414′. Thus, the first electronic device 401 routes the input 473A detected via the mouse 432′ to the second electronic device 415′. 4B, the first electronic device 401 sends output data to the second electronic device 415' that includes one or more commands / instructions for moving a cursor 412' within a display 414' of the second electronic device 415' in accordance with the input 473A. Thus, in FIG. 4C, the second electronic device 415' moves the cursor 412' within the display 414' to a new location in the first user interface 411' in accordance with the movement of the mouse 432'.

[0062] 4C , the second electronic device 415′ detects a selection input 473B via the mouse 432′ while the user's gaze 427 is directed toward a text input field 413′ in the first user interface 411′. For example, as shown in FIG. 4C , the second electronic device 415′ detects a click or press of a button on the mouse 432′ (or a tap on the touch-sensitive surface of the mouse 432′) while the cursor 412′ is positioned over the text input field 413′. In some examples, as also described above, in response to detecting the selection input 473B, the second electronic device 415′ sends input data to the first electronic device 401 so that the first electronic device 401 can determine where to route the selection input 473B. As also described above, the first electronic device 401 determines where to direct the selection input 473B based on the location of the gaze 427 in the three-dimensional environment. 4C , because the gaze 427 is directed toward the text entry field 413′ in the first user interface 411′ (or, in some examples, because the gaze 427 is directed toward the first user interface 411′, the display 414′, etc.), the first electronic device 401 causes the second electronic device 415′ to direct a selection input 473B toward the first user interface 411′, particularly toward the location of a cursor 412′ controllable via the mouse 432′, as described above. Accordingly, as shown in FIG. 4D , the second electronic device 415′ performs a selection at the location of the cursor 412′, which includes selecting the text entry field 413′ in the first user interface 411′. As shown in FIG. 4D , in some examples, once the text entry field 413′ is selected, the second electronic device 415′ displays a text cursor (e.g., a representation of a text cursor 428′) in the text entry field 413′ indicating that text may be entered into the text entry field 413′.

[0063] 4C as gaze 427i, alternatively, is directed neither toward the first user interface 411′ (or, in some examples, the display 414′) nor toward the second user interface 425 (or, in some examples, the virtual augmented display 404), the first electronic device 401 and the second electronic device 415′ optionally do not perform any operations in response to the selection input 473B. For example, when the first electronic device 401 receives input data provided by the second electronic device 415′ corresponding to a selection detected on the mouse 432′, the first electronic device 401 determines that the user's gaze 427i is directed away from the first user interface 411′ and the second user interface 425 and sends a command to the second electronic device 415′ to refrain from performing any operations in response to the selection. 4C is detected by the second electronic device 401′, the first electronic device 415 implements an input latching operation that causes the input to be routed to the currently active user interface within the three-dimensional environment 450. For example, the input is automatically routed to the last user interface that the gaze 427 was directed at before the selection input 473B was detected, while the gaze 427i is directed away from the first user interface 411′ and the second user interface 425. According to this example, because the gaze 427 was last directed at the first user interface 411′ in FIG. 4B before the selection input 473B was detected in FIG. 4C, the input is latched to the first user interface 411′, causing the first electronic device 401 to route the selection input 473B to the first user interface 411′ (e.g., select the text input field 413′) as shown in FIG. 4D and similarly described above.

[0064] 4D , while the text cursor 428′ is displayed in the text entry field 413′ and while the user's gaze 427 is directed toward the first user interface 411′, the second electronic device 415′ detects a sequence of one or more inputs 473C via a keyboard 431′ in communication with the second electronic device 415′. For example, in FIG. 4D , the second electronic device 415′ detects the selection of one or more keys on the keyboard 431′. In some examples, as also described above, in response to detecting the sequence of one or more inputs 473C, the second electronic device 415′ transmits input data to the first electronic device 401, the input data including information corresponding to the selection of one or more keys on the keyboard 431′. In some examples, as also described above, when the first electronic device 401 receives the input data provided by the second electronic device 415′, the first electronic device 401 determines where to route the keyboard input based on the location of the gaze 427 within the three-dimensional environment 450. 4D , because gaze 427 is directed toward the first user interface 411′ when a series of one or more inputs 473C is detected by the second electronic device 415′, the first electronic device 401 causes (e.g., via output data including one or more commands / instructions) the second electronic device 415′ to direct the series of one or more inputs 473C to the first user interface 411′, and in particular to the text entry field 413′. Thus, as shown in FIG. 4E , because text cursor 428′ is active within the text entry field 413′ and the series of one or more inputs 473C is directed toward the first user interface 411′, the second electronic device 415′ displays text (e.g., a representation of text 429′ spelling “apartmen”) at the location of text cursor 428′ in the text entry field 413′ in accordance with the selection of one or more keys on keyboard 431′.

[0065] In some examples, the cursor 412′ can be configured to be movable between the physical display 414′ of the second electronic device 415′ and the virtual augmented display 404 within the three-dimensional environment 450. In some examples, the cursor 412′ may be moved between the physical display 414′ of the second electronic device 415′ and the virtual augmented display 404 based on movement of a mouse 432′ controlling the cursor 412′. For example, in FIG. 4F , the second electronic device 415′ detects movement input 473D corresponding to movement of the mouse 432′ (e.g., movement rightward across the top of the table 406′). Additionally, in some examples, the cursor 412′ may be moved between the physical display 414′ of the second electronic device 415′ and the virtual augmented display 404 based on the location of the user's line of sight 427. For example, in FIG. 4F , the line of sight 427 is directed toward the second user interface 425 while movement of the mouse 432′ is detected by the second electronic device 415′.

[0066] In some examples, the cursor 412′ is moved between the display 414′ of the second electronic device 415′ and the virtual extended display 404 in accordance with a determination that the movement of the mouse 432′ has exceeded a movement threshold. For example, the movement of the mouse 432′ exceeds the movement threshold if the velocity or acceleration of the movement of the mouse 432′ exceeds a velocity or acceleration threshold. In some examples, the movement of the mouse 432′ exceeds the movement threshold if the movement of the mouse 432′ corresponds to movement of the cursor 412′ beyond a threshold distance (e.g., 0.5, 1, 1.5, 2, 3, 5, 10, 15, 25 cm, etc.) of the boundary of the display, optionally in the direction of the virtual extended display 404. For example, in FIG. 4F , the movement of the mouse 432′ detected by the second electronic device 415′ corresponds to movement of the cursor 412′ less than a threshold distance of the boundary of the display 414′ of the second electronic device 415′. 4G , the display of the cursor 412′ is maintained within the display 414′ of the second electronic device 415′ (e.g., at or near the right boundary of the display 414′). For example, the second electronic device 415′ may send input data to the first electronic device 401 including information corresponding to movement of the cursor 412′ less than a threshold distance of the boundary of the display 414′, which causes the first electronic device 401 to send output data to the second electronic device 415′, which causes the second electronic device 415′ to maintain the display of the cursor 412′ on the display 414′ (e.g., despite moving the cursor 412′ rightward in accordance with movement of the mouse 432′). In another example, the second electronic device 415′ may cease sending input data to the first electronic device 401 when it is determined that the movement of the cursor 412′ does not exceed the movement threshold, and may send input data only when the movement of the cursor 412′ exceeds the movement threshold.

[0067] 4G , the second electronic device 415′ detects further movement of the mouse 432′ while the cursor 412′ is displayed on the display 414′ of the second electronic device 415′. For example, as shown in FIG. 4G , the second electronic device 415′ optionally detects movement input 473E corresponding to movement of the cursor 412′ to the right on the display 414′, beyond the aforementioned threshold distance of the boundary of the display 414′, while the user's gaze 427 is still directed toward the second user interface 425 of the virtual augmented display 404. Alternatively, in FIG. 4G , the second electronic device 415′ detects a selection input via the mouse 432′ while the gaze 427 is directed toward the second user interface 425. For example, the second electronic device 415′ detects a click or press of a button on the mouse 432′ (e.g., or a tap on the touch-sensitive surface of the mouse 432′).

[0068] 4H , in response to detecting movement of the mouse 432′ beyond a threshold distance of a boundary of the display 414′, optionally corresponding to movement of the cursor 412′ toward the virtual extended display 404, the first electronic device 401 displays the cursor 412′ within the virtual extended display 404 within the three-dimensional environment 450. For example, the first electronic device 401 causes the second electronic device 415′ to stop displaying the cursor 412′ on the display 414′ and re-display the cursor 412′ within the virtual extended display 404 at a location based on the movement of the mouse 432′ (e.g., the first electronic device 401 displays the cursor 412′ within the second user interface 425 based on input data provided by the second electronic device 415′, the input data including information corresponding to the movement of the mouse 432′). In some examples, the first electronic device 301 displays the cursor 412′ within the virtual extended display 404 based on image data (e.g., as described above) received from the second electronic device 415′, which shows the cursor 412′ at a location within the virtual extended display 404 based on movement of the mouse 432′ determined by the second electronic device 415′.

[0069] 4H , in response to the second electronic device 415′ detecting a selection input via the mouse 432′ while the user's gaze 427 is directed toward the second user interface 425, the first electronic device 401 moves the cursor 412′ to the second user interface 425. In another example, in response to the second electronic device 415′ detecting the movement input 473D of FIG. 4F while the gaze 427 is directed toward a different interface (e.g., the second user interface 425) or display (e.g., the virtual augmented display 404) than the one currently targeted by the cursor 412′, or while the gaze 427 is directed toward a location (in the same or a different interface, display, etc.) that is a threshold distance from the cursor 412′, the first electronic device 401 may move the cursor 412′ to a location based on the gaze 427. For example, as also described above, the first electronic device 401 causes the second electronic device 415′ to stop displaying the cursor 412′ on the display 414′ and re-display the cursor 412′ in the virtual extended display 404 at a location based on the location of the line of sight 427. As shown in FIG. 4H , the first electronic device 401 optionally displays the cursor 412′ in the second user interface 425 at the location of the line of sight 427 (e.g., over the first search result 426A). In some examples, as also described above, the first electronic device 401 renders and displays the cursor 412′ in the virtual extended display 404 based on image data corresponding to the cursor 412′ provided by the second electronic device 415′.

[0070] In some examples, input detected via keyboard 431′ and / or mouse 432′ in communication with second electronic device 401′ may be routed by first electronic device 401′ to interact with virtual augmented display 404 displayed in three-dimensional environment 450 at first electronic device 415. In some examples, as also described above, input is routed based on the location of the user's gaze 427 in three-dimensional environment 450. In FIG. 4H , while cursor 412′ is displayed in second user interface 425 of first electronic device 415, second electronic device 401′ detects selection input 473F via mouse 432′. For example, in FIG. 4H , second electronic device 415′ detects a click or press of a button on mouse 432′ (or a tap on the touch-sensitive surface of mouse 432′).

[0071] 4I , in response to detecting a selection input 473F on the mouse 432′, the first electronic device 401 performs a selection operation based on the location of the cursor 412′ in the second user interface 425. For example, as shown in FIG. 4I , because the cursor 412′ is positioned over the first search result 426A in the second user interface 425 and the line of sight 427 is directed toward the second user interface 425 when the selection input 473F is detected, the first electronic device 401 selects the first search result 426A, which includes displaying content associated with the first search result 426A (e.g., “Website 1 Content” accessible from the website “www.URL1.com”). Specifically, as described above herein, the second electronic device 415′ transmits input data including information corresponding to the selection input 473F to the first electronic device 401 and determines the location of the line of sight 427 in the three-dimensional environment 450 in response to detecting the input data. 4I , because the gaze 427 was directed toward the second user interface 425 when the selection input 473F was detected in FIG. 4H , the first electronic device 401 directs the selection input toward the virtual extended display 404, i.e., performs the selection at the location of the cursor 412′ in the second user interface 425. In other examples in which the second electronic device 415 generates the content for the virtual extended display 404, the second electronic device 415 may forgo sending input data to the first electronic device and instead perform the selection at the location of the cursor 412′ in the second user interface 425. The first electronic device 401 may display any resulting updates to the content of the virtual extended display 404 based on image data (e.g., as described above) received from the second electronic device 415′ that indicates changes to the content of the virtual extended display 404 based on that determined by the second electronic device 415′ based on the selection input 473F.

[0072] 4I , the second electronic device 415′ detects a movement input 473G via the mouse 432′. For example, in FIG. 4I , the second electronic device 415′ detects movement of the mouse 432′ rightward across the surface of the table 406′ while the cursor 412′ is displayed within the second user interface 425 of the first electronic device 401. In some examples, as shown in FIG. 4J , in response to detecting the movement of the mouse 432′ at the second electronic device 415′, the first electronic device 415 moves the cursor 412′ within the second user interface 425 according to the movement of the mouse 432′ detected at the second electronic device 401′. For example, in response to detecting the movement input 473G, the second electronic device 415′ transmits input data to the first electronic device 401, the input data including information corresponding to the movement of the mouse 432′ (e.g., the direction of the movement and / or the distance or speed of the movement). In some examples, as also discussed herein, when the first electronic device 401 receives input data provided by the second electronic device 415′, the first electronic device 401 determines the location of the user's gaze 427 when movement input 473G is detected at the second electronic device 415′. In FIG. 4J , because the gaze 427 was directed toward the second user interface 425 in the virtual extended display 404, the first electronic device 401 moves the cursor 412′ within the second user interface 425 in accordance with the input data. In some examples, the first electronic device 401 displays the movement of the cursor 412′ within the second user interface 425 in the virtual extended display 404 (e.g., changes the position at which the cursor 412′ is displayed) based on image data received from the second electronic device 415′ (e.g., described above), which shows the cursor 412′ with an updated position within the virtual extended display 404 based on the movement of the mouse 432′ determined by the second electronic device 415′.

[0073] 4J , the second electronic device 415′ detects a selection input 473H via the mouse 432′ while the first electronic device 401 displays the cursor 412′ in the second user interface 425. For example, as also described above, the second electronic device 415′ detects a click or press of a button on the mouse 432′ (or a tap on the touch-sensitive surface of the mouse 432′). In some examples, as shown in FIG. 4K , in response to the second electronic device 415′ detecting the selection input 473H, the first electronic device 401 performs a selection operation in the second user interface 425 based on the location of the cursor 412′. 4J , the first electronic device 401 may direct the selection input to the second user interface 425 in accordance with the detected selection input 473H, and in particular, perform a selection of the text input field 416 of the second user interface 425. In some examples, also as described above, the second electronic device 415′ may instead perform a selection of the text input field 416 of the second user interface 425 and provide updated image data for the virtual augmented display 404 to the first electronic device 401. As shown in FIG. 4K , in some examples, selecting the text input field 416 includes displaying a text cursor 428 in the text input field 416 indicating that text can be entered into the text input field in response to keyboard input.

[0074] 4K, the second electronic device 415′ detects a series of one or more selection inputs 473I via the keyboard 431′ while the first electronic device 401 displays a text cursor 428 in the text input field 416. For example, in FIG. 4K, the second electronic device 415′ detects one or more selections or presses of one or more keys on the keyboard 431′. In some examples, as shown in FIG. 4L, in response to the second electronic device 415′ detecting the series of one or more selection inputs 473I, the first electronic device 401 inputs text 449 into the text input field 416 according to the series of one or more selection inputs 473I detected via the keyboard 431′. For example, in FIG. 4L, the first electronic device 401 displays text 449 spelling "restaura" at the location of the text cursor 428 in the text input field 416 according to the selected keys of the keyboard 431′. In particular, similar to what has been described herein above, because the gaze 427 was directed to the second user interface 425 when a series of one or more selection inputs 473I was detected by the second electronic device 415′, the first electronic device 401 displays text 449 according to input data including information corresponding to selected keys of a keyboard 431′ provided by the second electronic device 415′. In some examples, also as described above, the second electronic device 415′ may instead enter text 449 into the text input field 416 according to a series of one or more selection inputs 473I detected via the keyboard 431′. In these examples, the first electronic device 415′ may display text 449 spelling out “restaura” in the text input field 416 based on image data indicating “restaura” received from the second electronic device 415′.

[0075] 4L, while the first electronic device 401 displays text 449 in the text entry field 416 of the second user interface 425, the second electronic device 415′ detects a further selection input 473J via the keyboard 431′. For example, in FIG. 4L, the second electronic device 415′ detects the selection of one or more additional keys on the keyboard 431′. As shown in FIG. 4L, the second electronic device 415′ detects the selection input 473J via the keyboard 431′ while the user's gaze 427 is directed toward the first user interface 411′ on the display 414′ of the second electronic device 415′.

[0076] In some examples, as shown in FIG. 4M , in response to the second electronic device 415′ detecting a selection input 473J via the keyboard 431′, the first electronic device 401 routes the selection input 473J to the first user interface 411′ of the second electronic device 415′ based on the location of the gaze 427. For example, similar to what has been described herein above, the first electronic device 401 determines that the gaze 427 is directed toward the display 414′ when the selection input 473J is detected on the keyboard 431′ of FIG. 4L and sends output data to the second electronic device 415′ that causes the second electronic device 415′ to update the text 429′ in the text input field 413′ according to the selected key of the keyboard 431′. As shown in FIG. 4M , the text 429′ in the first user interface 411′ is updated to include the text “apartments fo” at the location of the text cursor 428′ in the text input field 413′. Additionally, as shown in FIG. 4M, the first electronic device 401 ceases updating the text 449 in the text input field 416 in the second user interface 425 according to the selected key of the keyboard 431′.

[0077] 4M, when the second electronic device 415′ updates the text 429′ in the first user interface 411′ according to the selection input 473J, the cursor 412′ is re-displayed in the first user interface 411′, optionally based on the location of the line of sight 427 in the three-dimensional environment 450. For example, in FIG. 4M, the first electronic device 401 stops displaying the cursor 412′ in the second user interface 411 and sends data (e.g., output data) to the second electronic device 415′ to cause the second electronic device 415′ to re-display the cursor 412′ at a location in the first user interface 425′ that corresponds to the location of the line of sight 427 in the three-dimensional environment 450 (e.g., on the text input field 413′). In some examples, as described above in this specification, while the cursor 412′ is located (e.g., displayed) within the first user interface 411′ on the display 414′ and the line of sight 427 is directed toward the display 414′ of the second electronic device 415′, the cursor 412′ is controllable via the mouse 432′ to interact with user interface elements of the display 414′ (e.g., the first user interface 411′ and / or the dock 409′) in one or more of the ways described above.

[0078] In some examples, disassociation of the first electronic device 401 from the user of the first electronic device 401 causes the display 414′ of the second electronic device 415′ to be reconfigured as the sole display for the first user interface 411′ and the second user interface 425. For example, in FIG. 4M , the first electronic device 401 detects the removal of the first electronic device 401 from the user's head (e.g., via the orientation sensor 210 of FIG. 2 ), as represented by arrow 475A. In some examples, disassociation of the first electronic device 401 from the user causes the physical environment surrounding the user to no longer be visible through the display of the first electronic device 401 (e.g., such that the display of the first electronic device 401 is no longer positioned above or in front of the user's eyes). In some examples, disassociation of the first electronic device 401 corresponds to the first electronic device 401 being powered off or entering a sleep state, which optionally causes the display of the first electronic device 401 to stop or pause operation.

[0079] 4N , in response to detecting disassociation of the first electronic device 401 from the user of the first electronic device 401, the first electronic device 401 stops presenting the three-dimensional environment 450. For example, as shown in FIG. 4N , all virtual objects stop being displayed by the first electronic device 401, i.e., the virtual augmented display 404. Additionally, as shown in FIG. 4N , in some examples, when the first electronic device 401 is no longer being used by the user, the display 414 of the second electronic device 415 in the real-world environment 400 is reconfigured as the only available display for all windows, applications, files, and other user interface elements previously displayed across the display 414 and the virtual augmented display 404. 4N , the first user interface 411 and the second user interface 425 are simultaneously displayed on the display 414 of the second electronic device 415 because the virtual augmented display 404 is no longer available as a display means for the second user interface 425 as outlined above. In some examples, in response to detecting disassociation of the first electronic device 401 from a user of the first electronic device 401, interfaces associated with applications executing on the first electronic device 401 (and not on the second electronic device 415) may not be displayed on the display 414. In other examples, interfaces associated with applications executing on the first electronic device 401 (but not on the second electronic device 415) may be displayed on the display 414 in response to detecting disassociation of the first electronic device 401 from a user of the first electronic device 401 (e.g., based on the display capabilities of the second electronic device 415′).

[0080] In particular, in some examples, when the first electronic device 401 is disassociated from the user, the first electronic device 401 transmits context data to the second electronic device 415, the context data including information corresponding to the display state of the second user interface 425. For example, in FIG. 4N , the information corresponding to the display state of the second user interface 425 includes details about the appearance of the second user interface 425, content included in the second user interface 425, such as the text “restaura” as shown, and / or the display mode of the second user interface 425 (e.g., whether the second user interface 425 is displayed in an extended full-screen mode on the virtual extended display 404). 4N, when the second user interface 425 is displayed on the display 414 of the second electronic device 415 after the first electronic device 401 is no longer in use, the second user interface 425 is displayed according to the context data provided by the first electronic device 401 (e.g., such that the text "restaura" is still displayed in the second user interface 425 when the display 414 is reconfigured as the only display). In other examples in which the second electronic device 415 generates the content of the virtual augmented display 404 and manages the context data including information corresponding to the state of the display of the second user interface 425, the first electronic device 401 may cease sending the context data to the second electronic device 415 and may instead send an indication that the first electronic device 401 has been disassociated from the user.

[0081] In some examples, when the first electronic device 401 is reassociated with the user such that the display of the first electronic device 401 is operational (e.g., powered on or woken up from a sleep state), the first electronic device 401 redisplays the virtual augmented display 404 based on the previous display of the virtual augmented display 404 on the first electronic device 401. In FIG. 4N , the user returns the first electronic device 401 to its head, as represented by arrow 475B. For example, the user positions the display of the first electronic device 401 above and / or in front of the user's eyes so that the real-world environment 400 is once again visible through the display of the first electronic device 401.

[0082] In some examples, as shown in Figure 4O, when the first electronic device 401 is reassociated with the user, the first electronic device 401 redisplays the three-dimensional environment 450. Additionally, as shown in Figure 4O, the first electronic device 401 redisplays the virtual augmented display 404, including the second user interface 425, because the virtual augmented display 404 is restored as a second (e.g., virtual) display for the second electronic device 415'. For example, as shown in Figure 4O, the second electronic device 415' stops displaying the second user interface 425, and the first electronic device displays the second user interface 425 within the virtual augmented display 404 according to the display state of the second user interface 425 before the first electronic device 401 was disassociated from the user in Figure 4M.

[0083] Additionally, in some examples, as shown in FIG. 4O, when second user interface 425 re-displays on virtual extended display 404 at first electronic device 401, second user interface 425 includes any updates to the presentation of second user interface 425, including updates to the content of second user interface 425, since second user interface 425 was last displayed at first electronic device 401. For example, in FIG. 4O, the text "restaurants" has been updated in second user interface 425 since second user interface 425 was last displayed at first electronic device 401 in FIG. 4M (e.g., in response to detecting a selection input via keyboard 431′). Thus, as shown in FIG. 4O, when second user interface 425 re-displays on virtual extended display 404 at first electronic device 401, second user interface 425 includes the updated text described above.

[0084] In some examples, the virtual augmented display 404 is reconfigured as the second display of the second electronic device 401′ when the first electronic device 415 is reassociated with the user, if the reassociation occurs within a threshold amount of time of disassociation. For example, the virtual augmented display 404 remains configured to display the content (e.g., a user interface) of the second electronic device 415′ if the user initially removes the first electronic device 401 from their head (e.g., and / or away from their eye), and then places the first electronic device 401 on their head (e.g., above or in front of their eye) within 1, 2, 5, 10, 15, 20, 30, 60, 80, 100, 120 minutes, etc., after removing the first electronic device 401. In some examples, if reassociation does not occur within a threshold amount of time of disassociation of the first electronic device 401 from the user, the first electronic device 401 ceases to re-display the second user interface 425 within the virtual augmented display 404. In some examples, the first electronic device 401 also ceases to re-display the virtual extended display 404. In some such examples, the virtual extended display 404 may be re-displayed by re-activating the virtual extended display 404, such as by using an affordance similar to the first affordance 319 of FIG.

[0085] Note that in some examples, an input initially detected by the second electronic device 415 may not be transmitted to the first electronic device 401 pursuant to a determination that the input should be directed to the second electronic device 415 (e.g., based on the line of sight 427 being directed toward the display 414′, the interface of the display 414′, the virtual extended display 404, or the interface of the virtual extended display 404). In these examples, the first electronic device 401 may intermittently, periodically, or at any other desired interval, transmit information indicating the current destination of the user input to the second electronic device 415′. If the current destination is the second electronic device 415′, the second electronic device 415′ may refrain from transmitting the detected input to the first electronic device 401 and may instead process the input locally. If the current destination is the first electronic device 401 (or any device other than the second electronic device 415′), the second electronic device 415′ may send input data (e.g., directly or indirectly via a server) to the first electronic device 401, including information corresponding to the detected input.

[0086] It should be noted that in some examples, the first user interface 411′ and / or the second user interface 425 may additionally or alternatively be interacted with via input detected by the first electronic device 401. For example, individual elements of the first user interface 411′ and / or the second user interface 425 (e.g., text entry fields 413′ / 416, etc.) may be selected via voice or hand-based input, such as an air pinch input (e.g., the index finger and thumb of a user's hand contacting together) or an air tap or touch gesture (e.g., provided by the index finger of a hand), in accordance with the gaze-based rules described above for routing input. Additionally, in some examples, the virtual extended display 404 itself may be interacted with via hand-based input detected by the first electronic device 401. For example, the first electronic device 401 may detect a selection of the virtual extended display 404 (e.g., via an air pinch input, etc.) and subsequently reposition the virtual extended display 404 within the three-dimensional environment 450 in response to detecting a movement of the user's hand. As another example, the first electronic device 401 may scale the virtual augmented display 404 within the three-dimensional environment 450, such that the size of the second user interface 425 may be increased or decreased in the three-dimensional environment 450 and / or such that additional user interfaces may be displayed on the virtual augmented display 404. In such examples where input is directed to the virtual augmented display 404 itself rather than the content of the virtual augmented display 404 (e.g., the second user interface 425), the input is responded to by the first electronic device 415 rather than the second electronic device 301′ (e.g., because the hand-based input described above is detected by the first electronic device 301 independently of the second electronic device 415′).

[0087] It should also be noted that in some examples, one or more physical input devices (e.g., physical input devices such as mouse 332′, 432′ and keyboard 331′, 431′) may additionally or alternatively be communicatively coupled (e.g., wired or wirelessly) with the first electronic device (e.g., first electronic device 301 or 401). In these examples, input from the physical input devices may be routed to an appropriate device or interface in the same manner as described above. However, in these examples, input from the physical input devices need not first be detected by and sent to the second electronic device (e.g., second electronic device 315′ or 415′) before being routed to the appropriate device or interface.

[0088] Thus, as outlined above, providing an augmented virtual display for an electronic device within a three-dimensional environment that can be interacted with via input detected on physical input devices in communication with the electronic device advantageously facilitates seamless and efficient interaction with the display of the electronic device and the augmented virtual display without requiring a user to utilize separate input means. Another advantage of the above systems and methods is the seamless integration of a set of input devices (e.g., keyboard, mouse, trackpad, etc.) of a first physical electronic device with a set of input devices (e.g., eye tracking sensors, hand tracking sensors, orientation sensors, etc.) of a second physical electronic device, which allows input received at both sets of input devices to be directed to the same one or more user interfaces within the three-dimensional environment. Attention is now directed to exemplary interactions with images of a virtual augmented display within a real-time multi-user communication session.

[0089] 5A-5G illustrate exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. In some examples, as shown in FIG. 5A, a first electronic device 501 may be used to present a three-dimensional environment 550. In some examples, the first electronic device 501 optionally corresponds to the first electronic device 401 / 301 described above. In some examples, the three-dimensional environment 550 includes a captured portion of a physical environment in which the first electronic device 501 is located. For example, the three-dimensional environment 550 optionally includes a table (e.g., a representation of table 506′), as shown in FIG. 5A. In addition, the three-dimensional environment 550 includes a second electronic device (e.g., a representation of second electronic device 515′) and one or more physical input devices (e.g., one or more representations of one or more input devices 531′ / 532′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, three-dimensional environment 550 optionally corresponds to three-dimensional environment 450 / 350 described above. In some examples, second electronic device 515' corresponds to second electronic device 415' / 315' described above. In some examples, one or more input devices 531' / 532' correspond to one or more input devices 431' / 432' and / or 331' / 332' described above. In some examples, the representation of the physical environment may include a portion of the physical environment that is seen through a transparent or translucent display of first electronic device 501.

[0090] In some examples, similar to those described herein above, the first electronic device 501 may communicate with the second electronic device 515′ such that the first electronic device 501 may be configurable to display a virtual augmented display for the second electronic device 515′. For example, as shown in FIG. 5A , the three-dimensional environment 550 includes a first affordance 519 that is displayed at the second electronic device 515′. In some examples, the first affordance 519 corresponds to the first affordance 319 described above. In FIG. 5A , the first electronic device 501 is not displaying a virtual augmented display for the second electronic device 515′ because the first affordance 519 has not been selected by the user of the first electronic device 501.

[0091] 5A , the three-dimensional environment 550 includes an invitation element 535 corresponding to an invitation to join a third electronic device, different from the first electronic device 515 and the second electronic device 501′, in the multi-user communication session. For example, the first electronic device 501 displays the invitation element 535 in response to detecting an indication from a third electronic device (not shown) inviting the user of the first electronic device 501 to join the user of the third electronic device in the multi-user communication session. In some examples, as shown in FIG. 5A , the invitation element 535 includes a first option 537-1 selectable to accept the invitation to join the multi-user communication session and a second option 537-2 selectable to decline the invitation to join the multi-user communication session (e.g., including ceasing to display the invitation element 535 in the three-dimensional environment 550).

[0092] 5A , while displaying invitation element 535 in three-dimensional environment 550, first electronic device 501 detects selection input 571A directed toward first option 537-1 of invitation element 535. For example, first electronic device 501 detects, via one or more sensors of first electronic device 501, an air pinch gesture performed by the user's hand, a tap or touch gesture performed by the hand, a verbal command, a gaze dwell directed toward first option 537-1, etc. In some examples, as shown in FIG. 5B , in response to detecting selection of first option 537-1, first electronic device 501 joins a third electronic device in a multi-user communication session, as described below.

[0093] In some examples, when a first electronic device 501 participates in a multi-user communication session with a third electronic device (not shown), the first electronic device 501 and the third electronic device are configured to present a shared three-dimensional environment including one or more shared virtual objects (e.g., content such as images, video, audio, etc., a representation of an application's user interface, etc.). As used herein, the term "shared three-dimensional environment" refers to a three-dimensional environment independently presented, displayed, and / or visible on two or more electronic devices, in which content, applications, data, etc. may be shared and / or presented to users of the two or more electronic devices. In some examples, while the first electronic device 501 is in a multi-user communication session with a second electronic device, an avatar corresponding to a user of one electronic device is optionally displayed within the three-dimensional environment displayed via the other electronic device. For example, as shown in FIG. 5B , on the first electronic device 501, an avatar 536 corresponding to a user of the third electronic device is displayed in the three-dimensional environment 550.

[0094] In some examples, the presentation of the avatar 536 as part of the shared three-dimensional environment is optionally accompanied by audio effects corresponding to the voice of the user of the third electronic device. For example, the avatar 536 displayed within the three-dimensional environment 550 using the first electronic device 501 is optionally accompanied by audio effects corresponding to the voice of the user of the third electronic device. In some such examples, when the user of the third electronic device speaks, the user's voice may be detected by the third electronic device (e.g., via a microphone) and transmitted to the first electronic device 501 (e.g., directly or via a server (e.g., a wireless communication terminal)), whereby the detected voice of the user of the third electronic device may be presented as audio (e.g., using the speaker(s) 216) to the user of the first electronic device 501 within the three-dimensional environment 550. In some examples, audio effects corresponding to the voice of the user of the third electronic device may be spatialized so that they appear to the user of the first electronic device 501 to emanate from the location of the avatar 536 within the three-dimensional environment 550 (e.g., despite being output from speakers of the first electronic device 501). In some examples, audio effects corresponding to the voice of the user of the third electronic device may alternatively be presented in mono or stereo at the first electronic device 501.

[0095] In some examples, while the first electronic device 501 and the third electronic device are in a multi-user communication session, the avatar 536 is displayed within the three-dimensional environment 550 with an individual orientation that corresponds to and / or is based on the orientation of the third electronic device (and / or the user of the third electronic device) in the physical environment surrounding the third electronic device. For example, in FIG. 5B , in the three-dimensional environment 550, the avatar 536 optionally faces toward the viewpoint of the user of the first electronic device 501. As a particular user moves their electronic device (and / or themselves) within the physical environment within the multi-user communication session, the user's viewpoint may change accordingly, and therefore the orientation of the user's avatar within the three-dimensional environment may also change. For example, referring to FIG. 5B , if a user of a third electronic device looks leftward in the three-dimensional environment such that the third electronic device is rotated to the left (e.g., counterclockwise) (e.g., a corresponding amount), the user of the first electronic device 501 will see the avatar 536 corresponding to the user of the third electronic device rotate to the right (e.g., clockwise) relative to the perspective of the user of the first electronic device 501 as the third electronic device moves.

[0096] Additionally, in some examples, while the first electronic device 501 and the third electronic device are in a multi-user communication session, the field of view of the shared three-dimensional environment and / or the location of the user's viewpoint within the shared three-dimensional environment optionally changes according to movement of the electronic devices (e.g., by the user of the electronic device). For example, if during the communication session the first electronic device 501 is moved closer to the representation of the table 506′ and / or the avatar 536 (e.g., because the user of the first electronic device 501 moves forward in the physical environment surrounding the first electronic device 501), the field of view of the three-dimensional environment 550 changes accordingly, causing the representation of the table 506′, the representation of the second electronic device 515′, and the avatar 536 to appear larger in the field of view. In some examples, each user may interact independently with the shared three-dimensional environment, such that changes in viewpoint within the three-dimensional environment 550 and / or interactions with virtual objects within the three-dimensional environment 550 by the first electronic device 501 optionally do not affect what is shown in the three-dimensional environment at the third electronic device, and vice versa.

[0097] In some examples, avatar 536 is a representation (e.g., a full-body rendering) of a user of the third electronic device. In some examples, avatar 536 is a representation of a portion of a user of the third electronic device (e.g., a rendering of the head, face, head and torso, etc.). In some examples, avatar 536 is a user-personalized, user-selected, and / or user-created representation of a user of the third electronic device that is displayed in three-dimensional environment 550. While avatar 536 shown in FIG. 5B corresponds to a full-body representation of a user of the third electronic device, it should be understood that alternative avatars, such as one of those described above, may be provided.

[0098] In some examples, while the first electronic device 501 and a third electronic device are in a multi-user communication session, content viewed by one user on one electronic device may be shared with another user on another electronic device in the multi-user communication session. In some such examples, the content may be experienced (e.g., viewed and / or interacted with) by both users (e.g., via their respective electronic devices) in a shared three-dimensional environment (e.g., the content is content shared in the three-dimensional environment). For example, in FIG. 5C , the first electronic device 501 displays visual indication 539 corresponding to a request from a user of a third electronic device (e.g., “User 1”) to share content (e.g., “Content A”) with the user of the first electronic device 501. As shown in FIG. 5C , the visual indication 539 optionally includes a selectable option 539-1 selectable to approve the request from the user of the third electronic device to share content with the user of the first electronic device 501. In some examples, the first electronic device 501 displays a visual indication 539 within the three-dimensional environment 550 in response to detecting data corresponding to the sharing request described above.

[0099] Additionally, in some examples, the shared three-dimensional environment includes non-shared content that is private to one user in the multi-user communication session. For example, in FIGS. 5B-5C , the third electronic device may be displaying a private application window within the three-dimensional environment at the third electronic device, which is, optionally, an object that is not shared between the first electronic device 501 and the third electronic device in the multi-user communication session. In some examples, the private application window may be associated with a separate application (e.g., a virtual augmented display application, a media player application, a web browsing application, a messaging application, etc.) running on the third electronic device. Because the private application window is not shared with the first electronic device 501, the first electronic device 501 optionally displays a representation of the private application window 504″ within the three-dimensional environment 550, as shown in FIGS. 5B-5C . As shown in FIGS. 5B-5C, in some examples, the representation of the private application window 504'' may be a faded, obscured, discolored, and / or semi-transparent representation of the private application window that prevents a user of the first electronic device 501 from viewing the content of the private application window.

[0100] 5C , while displaying visual indication 539 within three-dimensional environment 550, first electronic device 501 detects selection input 571B directed toward selectable option 539-1 of visual indication 539. For example, as also described above, first electronic device 501 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell directed toward selectable option 539-1, etc. In some examples, such as in FIG. 5C , the request to share content with first electronic device 501 corresponds to a request to share private application window 504″ (visible in three-dimensional environment 550) with a user of first electronic device 501.

[0101] In some examples, as shown in FIG. 5D , in response to detecting a selection of selectable option 539-1, first electronic device 501 approves a request from the third electronic device to share content with first electronic device 501. In some examples, because the content shared with first electronic device 501 corresponds to the content of the private application window described above, first electronic device 501 stops displaying a representation of private application window 504″ in three-dimensional environment 550. Also, as shown in FIG. 5D , when content is shared with first electronic device 501, first electronic device 501 displays image 504 corresponding to the shared content in three-dimensional environment 550. In some examples, as shown in FIG. 5D , when first electronic device 501 displays image 504 in three-dimensional environment 550, the position of avatar 536 corresponding to the user of the third electronic device optionally changes in three-dimensional environment 550. For example, the avatar 536 is moved to a location adjacent to (e.g., to the right of) the user's viewpoint of the first electronic device 501 so that the avatar 536 faces the content of the image 504 in the three-dimensional environment 550, as shown in FIG. 5D.

[0102] In some examples, the image 504 corresponds to an image of a virtual augmented display of a fourth electronic device that is different from the first electronic device 501 and the second electronic device 515′. For example, the virtual augmented display is a second display for the fourth electronic device that is communicating with a third electronic device, similar to the communication between the electronic devices 401 and 415 described above. In some examples, the third electronic device is displaying a virtual augmented display within a three-dimensional environment at the third electronic device, and in response to detecting user input from a user of the third electronic device, has sent a request (e.g., via an application sharing application) to share an image of the virtual augmented display with a user of the first electronic device 501, as described above. As shown in FIG. 5D , the virtual augmented display image 504 includes a user interface 540. For example, before the image of the virtual augmented display was shared with the first electronic device 501, the user interface 540 was displayed on the virtual augmented display at the third electronic device. In some examples, in FIG. 5D , the user interface 540 is a user interface of a music player application. For example, as shown in FIG. 5D , user interface 540 includes a first representation 541-1 of a first music album and a second representation 541-2 of a second music album, each including selectable indications of music tracks available for consumption on a fourth electronic device and / or a third electronic device.

[0103] In some examples, the user of the first electronic device 501 can view the content of the image 504 shared between the first electronic device 501 and the third electronic device, but the first electronic device 501 optionally limits and / or prevents direct user interaction with the content of the image 504, such as interaction with the user interface 540. Notably, in some examples, the user of the first electronic device 501 cannot provide input to interact with the content of the image 504 because only the image of the virtual augmented display is shared with the first electronic device 501, rather than the virtual augmented display itself being a true shared experience. Thus, as described above, because the virtual augmented display is optionally shared by the third electronic device, the user of the third electronic device may interact with the user interface 540 at the third electronic device, thereby causing the image 504 to update in the three-dimensional environment 550, but the user of the first electronic device 501 may not be able to do so, as described below.

[0104] 5E , while the virtual augmented display image 504 is displayed in the three-dimensional environment 550 at the first electronic device 515, the second electronic device 501′ detects a selection input 573 via a mouse 532′ in communication with the second electronic device 515′. For example, the second electronic device 514′ detects a button press or click on the mouse 532′ (or a tap on the touch-sensitive surface of the mouse 532′). In addition, as shown in FIG. 5E , the second electronic device 514′ detects the selection input 573 while the gaze 527 of the user of the first electronic device 501 is directed toward the user interface 540 of the image 504 in the three-dimensional environment 550.

[0105] 5F , in response to the second electronic device 514′ detecting a selection input 573 via the mouse 532′ while the first electronic device 501 detects a line of sight 427 directed toward the user interface 540 of the image 504, the first electronic device 501 ceases performing an action directed toward the image 504. For example, because the virtual extended display of the image 504 is not an extended display of the second electronic device 514′ as described above, the input detected via the mouse 532′ is not interpreted by the first electronic device 501 as being directed toward the image 504. Rather, in some examples, the selection input 573 detected by the second electronic device 514′ remains local to the second electronic device 514′ (e.g., the second electronic device 514′ performs the selection action in accordance with the selection input 573 without transmitting the input data to the first electronic device 501 as described above).

[0106] 5F , while displaying an image 504 of a virtual augmented display in a three-dimensional environment 550, the first electronic device 501 detects a selection input 571C directed toward a first representation 541-1 of a user interface 540 within the image 504. For example, similar to that described herein, the first electronic device 501 detects an air pinch gesture, an air tap or touch gesture, a verbal command, a gaze dwell directed toward the first representation 541-1, or the like. In some examples, in response to detecting a selection of the first representation 541-1 within the image 504, the first electronic device 501 ceases performing the selection action directed toward the first representation 541-1, as shown in FIG. 5G . For example, as shown in FIG. 5G , in response to the selection input 571C, the image 504 is not updated such that music tracks associated with a first album (e.g., “Album A”) are displayed in the user interface 540.

[0107] In some examples, as also described above, because the virtual augmented display of the image 504 is not a true shared experience between the first electronic device 501 and the third electronic device (e.g., because the virtual augmented display is not shared with the user of the first electronic device 501, but rather is an image of the virtual augmented display), user input detected by the first electronic device 501 directed at the image 504, including the user interface 540, does not cause the image 504 to update based on the user input. Thus, as shown in FIG. 5G , the image 504 remains unchanged in the three-dimensional environment 550. Thus, as outlined above, if an image of the virtual augmented display displayed at the third electronic device is shared with the first electronic device 501 while the first electronic device 501 and the third electronic device are communicatively linked in a multi-user communication session, input provided by the user of the first electronic device 501 may not cause the content of the image (e.g., the image 504) to update in accordance with the input.

[0108] In some examples, interaction may be provided that is directed toward the image 504 itself, rather than the content of the image 504. For example, if a user of the first electronic device 501 provides input to move the image 504 within the three-dimensional environment 550 (as opposed to, e.g., input to move the user interface 540 within the image 504), the first electronic device 501 may move the image 504 according to the input. Similarly, if a user of the first electronic device 501 provides input to scale the image 504 within the three-dimensional environment 550 (as opposed to, e.g., resizing the user interface 540 within the image 504), the first electronic device 501 may scale the image 504 according to the input. In some examples, if a user of a third electronic device shares the content of the image 504 itself, such shared content may be configured to be interacted with by the user of the first electronic device 501. For example, in FIG. 5C, if the content being shared with the user of the first electronic device 501 is the user interface 540 itself, the first electronic device 501 displays a shared application window corresponding to the user interface 540 of FIG. 5D, and input directed to the content of the shared application window (e.g., first representation 541-1 and / or second representation 541-2) is acted upon by the first electronic device 501.

[0109] It is understood that the examples shown and described herein are merely illustrative, and that additional and / or alternative elements may be provided within the three-dimensional environment for interacting with the virtual objects and elements. It should be understood that the appearance, shape, form, and size of each of the various user interface elements and objects shown and described herein are exemplary, and that alternative appearances, shapes, forms, and / or sizes may be provided. For example, virtual objects representing a virtual augmented display (e.g., virtual augmented display 304 / 404) and / or a user interface (e.g., user interfaces 411, 425, and / or 540) may be provided in shapes other than a rectangle, such as a circle, a triangle, etc. In some examples, the various selectable affordances described herein (e.g., first affordance 319 / 519 and / or options 537-1 / 537-2 or 539-1) may be selected verbally via a user verbal command (e.g., a “select option” or a “select virtual object” verbal command). Additionally or alternatively, in some examples, various options, user interface elements, controls, etc. described herein may be selected and / or manipulated via user input received via one or more separate input devices in communication with the electronic device(s). For example, selection input may be received via a physical input device, such as a mouse, trackpad, keyboard, etc., in communication with the electronic device(s).

[0110] Additionally, while only a single interface is shown in the physical and virtual extended displays, it should be understood that any number of interfaces may be included in each display. Further, while only a single virtual extended display (e.g., virtual extended display 304 / 404) is shown for a single electronic device (e.g., second electronic device 315 / 415), it should be understood that any number of virtual extended displays may be presented for each of any number of other electronic devices. In these examples, input detected by any of the electronic devices may be routed to the appropriate electronic device or interface in the same manner as described above.

[0111] In some examples, in addition to displaying the virtual extended display, the first electronic device may display one or more additional virtual objects or windows corresponding to applications running on the first electronic device, as described in more detail with reference to Figures 6A-6K. For example, the first electronic device 415, in addition to displaying the virtual extended display 404 including the second user interface 425 for the web browsing application of the second electronic device 401' in Figure 4A, may further display a virtual window corresponding to a text document editing application running on the first electronic device 415 (but not the second electronic device 401'). In these examples, input detected by the first electronic device 401, the second electronic device 415', or other coupled electronic devices may be routed to the appropriate device, application, and / or interface in the same manner as described above.

[0112] 6A-6K illustrate examples of a first electronic device that facilitates interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. In some examples, as shown in FIG. 6A, a three-dimensional environment 650 may be presented using the first electronic device 601. In some examples, the first electronic device 601 optionally corresponds to the first electronic device 301 / 401 / 501 described above. In some examples, the three-dimensional environment 650 includes a captured portion of a physical environment in which the first electronic device 401 is located. For example, the three-dimensional environment 650 optionally includes a table (e.g., a representation of table 606′), as shown in FIG. 6A. Additionally, the three-dimensional environment 650 includes a second electronic device (e.g., a representation of the second electronic device 615′) and one or more physical input devices (e.g., one or more representations of one or more input devices 631′ / 632′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, the three-dimensional environment 650 optionally corresponds to the three-dimensional environments 350 / 450 / 550 described above. In some examples, the second electronic device 615′ corresponds to the second electronic device 315′ / 415′ / 515′ described above. In some examples, the one or more input devices 631′ / 632′ correspond to one or more input devices 331′ / 332′, 431′ / 432′, 531′ / 532′ described above. In some examples, the representation of the physical environment may include a portion of the physical environment that is viewed through a transparent or translucent display of the first electronic device 601.

[0113] As also described above, in some examples, the first electronic device 601 communicates with the second electronic device 615′. Additionally, in some examples, as shown in FIG. 6A , the first electronic device 601 presents a virtual display 604 associated with the second electronic device 615′. For example, FIG. 6A . The virtual display 604 serves as a (e.g., virtual) display for the physical display (e.g., a representation of the display 614′) of the second electronic device 615′. In some examples, the virtual display 604 has one or more characteristics of the virtual augmented display 304 / 404 / 504 described above. In some examples, as shown in FIG. 6A , while the virtual display 604 is displayed in the three-dimensional environment 650, the display 614′ of the second electronic device 615′ is optionally powered off (e.g., the second electronic device 615′ is in a low-power mode or sleep state, or is powered off).

[0114] In some examples, as shown in FIG. 6A , virtual display 604 optionally includes a user interface 625 that is a representation of a user interface configured to be displayed (e.g., or currently displayed) on (e.g., physical) display 614′ of a second electronic device 615′. In some examples, as shown in FIG. 6A , user interface 625 includes multiple search results provided by a web browser (e.g., associated with the URL “www.Search.com”). For example, in FIG. 6A , user interface 625 includes a first search result 626A (e.g., “Search Result A”) associated with a first website (e.g., “www.URL1.com”), a second search result 626B (e.g., “Search Result B”) associated with a second website (e.g., “www.URL2.com”), and a third search result 626C (e.g., “Search Result C”) associated with a third website. In some examples, the multiple search results 626 are selectable to access content (e.g., text, images, video, music, etc.) via the respective sources (e.g., websites). In some examples, user interface 625 corresponds to second user interface 425 described above. Additionally, as shown in FIG. 6A , virtual display 604 includes a representation of dock 609″ (e.g., dock 309 / / 409′ / 509 / , etc., described above) that corresponds to a dock (not shown) configured to be displayed on (e.g., physical) display 614′ of second electronic device 615′. In some examples, as shown in FIG. 6A , virtual display 604 also includes a cursor (e.g., a representation of cursor 612′) configured to be displayed on display 614′ of second electronic device 615′ and controllable using mouse 632′. In some examples, cursor 612′ corresponds to cursor 312′ / 412′ / 512′ described above.

[0115] In some examples, as also described above, the first electronic device 601 displays a user interface 625 within the virtual display 604 based on image data transmitted to the first electronic device 601 by the second electronic device 615′ (e.g., directly or indirectly via a server). For example, as described above, the user interface 625 is configured to be displayed on the display 614′ of the second electronic device 615′ (e.g., the user interface 625 is the user interface of the second electronic device 615′). Thus, in some examples, the first electronic device 601 renders (e.g., generates) the user interface 625, including content for the user interface 625, based on image data provided by the second electronic device 615′. For example, the image data includes information corresponding to the appearance of the user interface 625, characteristics of the content of the user interface 625 (e.g., whether the content is interactive, such as selectable, scrollable, etc.), and / or the quality of the user interface 625 and its content (e.g., image quality, such as definition). In some examples, the image data transmitted by the second electronic device 601′ to the first electronic device 615 includes an image generated by the second electronic device 601′ that is displayed on the virtual display 604 by the first electronic device 615.

[0116] In some examples, the first electronic device 601 is configured to simultaneously display content associated with an application executing (e.g., locally) on the first electronic device 601 and the virtual display 604. In some examples, as described below, the first electronic device 601 is configured to facilitate interaction with the content of the virtual display 604 and the content of the application executing locally on the first electronic device 601. In FIG. 6A , the first electronic device 601 is displaying a user interface object 624 associated with an individual application within the three-dimensional environment 650. In some examples, as shown in FIG. 6A , the user interface object 624 includes a selectable option 623A that is selectable to display content (e.g., “Content A”) corresponding to the individual application within the three-dimensional environment 650. In some examples, the content includes a user interface, a video, an image, a three-dimensional object (e.g., a model), etc.

[0117] 6A, the first electronic device 601 detects a selection input 672A directed at the selectable option 623A. For example, in FIG. 6A, the first electronic device 601 detects an air pinch gesture, an air tap, or a touch gesture (e.g., via one or more sensors such as a camera or other hand tracking sensor), optionally a verbal command, a gaze dwell, etc. while the user's gaze is directed at the selectable option 623A.

[0118] In some examples, as shown in FIG. 6B , in response to detecting selection input 672A, first electronic device 601 displays video content (content A shown in FIG. 6A ) associated with the respective application (e.g., a video playback application executing on first electronic device 601) described above. For example, as shown in FIG. 6B , first electronic device 601 displays virtual application window 656 (e.g., a video playback user interface) displaying video content (e.g., a movie, a television program, a video clip, a music video, etc.) in three-dimensional environment 650. As shown in FIG. 6B , virtual application window 656 includes playback controls 645 (e.g., including a play / pause button, a rewind button, and a fast-forward button, among other possibilities) for controlling playback of the video content displayed in virtual application window 656. In some examples, as shown in FIG. 6B , virtual application window 656 is displayed with a grabber affordance 635 (e.g., a handlebar affordance) selectable via user input to initiate movement of virtual application window 656 within three-dimensional environment 650.

[0119] In some examples, similar to those described herein above, the first electronic device 601 communicates with the second electronic device 615′ to facilitate and coordinate interaction with the content of the virtual display 604 and / or the content of the virtual application window 656 in response to user input based on the location of the user's gaze of the first electronic device 601. For example, as shown in FIG. 6C , the first electronic device 601 detects (e.g., via an eye tracking sensor and / or an image sensor) that the user's gaze 627 is directed toward the virtual display 604. In particular, in FIG. 6C , the first electronic device 601 detects that the gaze 627 is directed toward a particular location corresponding to the content of the virtual display 604.

[0120] 6C , the second electronic device 615′ detects a selection input 673A on a mouse 632′ that is in communication with the second electronic device 615′ while the first electronic device 601 detects that the user's gaze 627 is directed toward the virtual display 604 within the three dimensional environment 650. For example, in FIG. 6C , the second electronic device 615′ detects a click, tap, or touch on a button or touch-sensitive surface of the mouse 632′. As also described above, in response to detecting the selection input 673A, the second electronic device 615′ maintains the selection input 673A local to the second electronic device 615′ because the gaze 627 is directed toward the virtual display 604 within the three dimensional environment 650. For example, the first electronic device 601 communicates (e.g., transmits) gaze data including information corresponding to the location of the line of sight 627 in the three-dimensional environment 602 to the second electronic device 615′ (e.g., continuously, periodically, and / or upon request from the second electronic device 615′).

[0121] In some examples, in accordance with determining that the line of sight 627 is directed toward the virtual display 604 (e.g., the user interface 625 in particular), the first electronic device 601 causes the second electronic device 615′ to perform an operation in accordance with the selection input 673A. For example, in FIG. 6C , the second electronic device 615′ performs a selection at the location of the cursor 612′ in the user interface 625. In particular, as shown in FIG. 6C , the second electronic device 615′ selects (e.g., activates) the second search result 626B based on the location of the cursor 612′ in the user interface 625. It should be understood that, although the user interface 625 is not displayed on the (e.g., physical) display of the second electronic device 615′ in the example of FIG. 6C , performing the selection operation includes updating image data representing the user interface 625 (e.g., as if the user interface 625 were displayed on the display of the second electronic device 615′). In some examples, when the second electronic device 615′ performs the selection operation, the second electronic device 615′ transmits updated image data to the first electronic device 601, which causes the first electronic device 601 to update the representation of the virtual display 604 in the three-dimensional environment 650 based on the performance of the selection operation. For example, as shown in FIG. 6D , the first electronic device 601 updates the display of the virtual display 604 to include an updated representation of the user interface 625. In particular, as shown in FIG. 6D , the user interface 625 is updated to include the results of selecting the second search result 626B in FIG. 6C , i.e., to display website content (e.g., “Website Content” in the text input field 616) from the website (e.g., “www.URL2.com”) associated with the second search result 626B.

[0122] In Figure 6E, the first electronic device 601 detects that the user's gaze 627 is directed toward a virtual application window 656 within the three-dimensional environment 650. In particular, as shown in Figure 6E, the first electronic device 601 detects that the gaze 627 is directed toward a playback control 645 (e.g., a pause button on the playback control 645). In Figure 6E, while the first electronic device 601 detects the gaze 627 directed toward the virtual application window 656, the second electronic device 615' detects a selection input 673B via a mouse 632' in communication with the second electronic device 615', in a similar manner as described above.

[0123] In some examples, similar to those described above, in response to detecting the selection input 673B, the second electronic device 615′ transmits input data corresponding to the selection input 673B to the first electronic device 601, because the line of sight 627 is directed toward the virtual application window 656 within the three-dimensional environment 650. For example, the second electronic device 615′ transmits the input data to the first electronic device 601 based on the line of sight data communicated (e.g., transmitted) by the first electronic device 601 to the second electronic device 615′, also as described above.

[0124] In some examples, in accordance with a determination that the line of sight 627 is directed toward the virtual application window 656 (e.g., the playback control 645 in particular), the first electronic device 601 performs an action in accordance with the selection input 673B. For example, in FIG. 6E , the first electronic device 601 performs a selection at the location of the line of sight 627 within the virtual application window 656. Specifically, as shown in FIG. 6E , the first electronic device 601 selects (e.g., activates) the pause button of the playback control 645 based on the location of the line of sight 627 within the virtual application window 656 (e.g., rather than based on the location of the cursor 612′ within the user interface 625, as described herein above). In other examples, the selection input 673B may be performed in accordance with the location of the cursor 612′. In some examples, as shown in FIG. 6F , when the first electronic device 601 selects the pause button of the playback control 645, the first electronic device 601 pauses playback of the video content within the virtual application window 656 within the three-dimensional environment 650. 6F , the first electronic device 601 optionally updates the pause button in the playback controls 645 in the virtual application window 656 to a play button. In some examples, when the first electronic device 601 performs a selection action directed at the virtual application window 656, the first electronic device 601 ceases sending output data corresponding to the selection of the pause button of the playback controls 645 in the virtual application window 656 to the second electronic device 615′.

[0125] 6G , while a line of sight 627 is directed toward a virtual display 604 within three-dimensional environment 650, first electronic device 601 detects a selection input 672B directed toward content on virtual display 604 via one or more input devices of first electronic device 601. For example, as shown in FIG. 6G , first electronic device 601 detects an air pinch gesture, a tap or touch gesture, a verbal command, etc. while a line of sight 627 is directed toward a text input field 616 of user interface 625 within virtual display 604.

[0126] In some examples, in response to detecting the selection input 672B while the line of sight 627 is directed toward the virtual display 604, the first electronic device 601 transmits input data corresponding to the selection input 672B to the second electronic device 615′. In some examples, the second electronic device 615′ performs an operation directed toward the user interface 625 based on the input data corresponding to the selection input 672B. For example, the first electronic device 601 causes the second electronic device 615′ to perform a selection operation directed toward the text entry field 616 based on the location of the line of sight 627 in FIG. 6G . In some examples, similar to those described above, when the second electronic device 615′ performs the selection operation, the second electronic device 615′ transmits updated image data to the first electronic device 601, causing the first electronic device 601 to update the representation of the virtual display 604 in the three-dimensional environment 650 based on the performance of the selection operation. For example, as shown in Figure 6H, first electronic device 601 updates the display of virtual display 604 to include an updated representation of user interface 625. In particular, as shown in Figure 6H, text input field 616 includes text cursor 628 and no longer contains the text "www.URL2.com" from Figure 6G. Additionally, as shown in Figure 6H, the location of cursor 612' is optionally updated to correspond to the location of gaze 627 within user interface 625, as also discussed herein.

[0127] 6H , the second electronic device 615′ detects a selection input 673C via one or more keys of a keyboard 631′ in communication with the second electronic device 615′ while the gaze 627 remains directed toward the virtual display 604. For example, as also described herein, the second electronic device 615′ detects one or more presses or taps of one or more keys of the keyboard 631′ while the first electronic device 601 detects the gaze 627 directed toward the user interface 625 in the virtual display 604.

[0128] In some examples, as also described above, because the line of sight 627 is directed toward the virtual display 604 in the three-dimensional environment 650, in response to detecting the selection input 673C, the second electronic device 615′ maintains the selection input 673C local to the second electronic device 615′ (e.g., without transmitting input data corresponding to the selection input 673C to the first electronic device 601). For example, in FIG. 6H , the second electronic device 615′ performs an action directed toward the user interface 625 in accordance with the selection input 673C, as also described above. In some examples, when the second electronic device 615′ performs an action in accordance with the selection input 673C, the second electronic device 615′ updates image data (e.g., pixel or display data) representing the user interface 625 based on the action and transmits the updated image data to the first electronic device 601, causing the first electronic device 601 to update the representation of the user interface 625 in the virtual display 604. 6I, the first electronic device 601 updates the user interface 625 in the virtual display 604 such that the text entry field 616 is updated to include entered text 649 (e.g., "www.URL4.com") at the location of the text cursor 628 based on the selected key of the keyboard 631'. In FIG. 6I, while the gaze 627 is directed toward the virtual display 604 in the three-dimensional environment 650, the first electronic device 601 detects a selection input 672C provided by a user of the first electronic device 601. For example, as shown in FIG. 6I, while the gaze 627 is directed toward an option 651 (e.g., an enter or search option) in the text entry field 616, the first electronic device 601 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell, or the like, as similarly described herein, via one or more sensors or input devices of the first electronic device 601.In some examples, as also described above, because the line of sight 627 is directed toward the virtual display 604 when the selection input 672C is detected, the first electronic device 601 transmits input data corresponding to the selection input 672C to the second electronic device 615′, which causes the second electronic device 615′ to perform an action directed toward the user interface 625 based on the location of the line of sight 627. In particular, in FIG. 6I , the second electronic device 615′ performs a selection action directed toward the option 651 based on the location of the line of sight 627 (e.g., in accordance with the line of sight data described above that includes information corresponding to the location of the line of sight 627 within the user interface 625).

[0129] In some examples, as also described herein, when the second electronic device 615′ performs a selection action targeting option 651, the second electronic device 615′ updates image data representing the user interface 625 and transmits the updated image data to the first electronic device 601. In some examples, as shown in FIG. 6J , in response to receiving the updated image data from the second electronic device 615′, the first electronic device 601 updates the display of the user interface 625 displayed on the virtual display 604 within the three-dimensional environment 650. Specifically, as shown in FIG. 6J , selecting option 651 in FIG. 6I causes the second electronic device 615′ to navigate to the website “www.URL4.com” and display content from the website (e.g., “URL4 Website Content”), which is reflected in an updated representation of the user interface 625 within the virtual display 604 within the three-dimensional environment 650. Additionally, in some examples, as described above in this specification, the first electronic device 601 updates the location of the cursor 612' in the user interface 625 based on the updated image data, such that the location of the cursor 612' is moved to the location of the line of sight 627 when the selection input 672C is detected in FIG. 6I.

[0130] 6J , the first electronic device 601 detects a selection input 672D in the same manner as described above while the line of sight 627 is directed toward a virtual application window 656 within the three-dimensional environment 650. Specifically, as shown in FIG. 6J , the first electronic device 601 detects a selection input 672D (e.g., an air pinch gesture, an air tap or touch gesture, a verbal command, etc.) while the line of sight 627 is directed toward a playback control 645 (e.g., a play button within the playback control 645) within the virtual application window 656.

[0131] 6J , when selection input 672D is detected, the first electronic device 601 performs an action directed to the virtual application window 656 in accordance with selection input 672D. For example, as shown in FIG. 6K , the first electronic device 601 selects the play button in playback controls 645, causing the video content displayed in virtual application window 656 to resume playing in the three-dimensional environment 650. Additionally, as shown in FIG. 6K , the first electronic device 601 replaces the play button with a pause button in playback controls 645, similar to the above.

[0132] In some examples, as also described herein, when the first electronic device 601 performs the above-described selection operation, the first electronic device 601 ceases transmitting output data corresponding to the selection operation to the second electronic device 615′. In particular, the input data corresponding to the selection input 672D of FIG. 6J is maintained local to the first electronic device 601, as described above, and the results of the selection operation responsive to the selection input 672D are also maintained local to the first electronic device 601. Thus, as described above, providing a virtual display for the second electronic device within a three-dimensional environment that may be interacted with via input detected on a physical input device in communication with the second electronic device or an input device in communication with the first electronic device advantageously facilitates seamless and efficient interaction with the virtual display of the second electronic device and applications running on the first electronic device without requiring a user to utilize a separate input means. Another advantage of the above systems and methods is seamless integration of a set of input devices (e.g., keyboard, mouse, trackpad, etc.) of a first physical electronic device with a set of input devices (e.g., eye tracking sensors, hand tracking sensors, orientation sensors, etc.) of a second physical electronic device, which allows input received at both sets of input devices to be directed to the same one or more user interfaces within the three-dimensional environment, which improves user-device interaction.

[0133] 7A-7B are flow diagrams illustrating an example process for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, process 700 begins at a first electronic device in communication with a display and one or more input devices of the first electronic device. In some examples, the first electronic device is optionally a head-mounted display similar to or corresponding to device 240 of FIG. 2. As shown in FIG. 7A, in some examples, at 702, the first electronic device captures, via one or more input devices, at least a portion of a real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. For example, as shown in FIG. 4A, first electronic device 401 captures a portion of a physical environment surrounding first electronic device 401 including second electronic device 415 having display 414, display 414 displaying first user interface 411. In some examples, the second electronic device is optionally a computer similar to or corresponding to device 230 of FIG. 2 (eg, a laptop computer, a desktop computer, or a tablet computer).

[0134] In some examples, at 704, the first electronic device presents, via a display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. For example, as shown in FIG. 3G , the first electronic device 301 presents a three-dimensional environment 350 including a representation of the second electronic device 315′, a representation of a table 306′ on which the second electronic device 315′ is placed, and a first affordance 319 and / or system options 318A associated with the second electronic device 315′. In some examples, at 706, while the first electronic device presents the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, input corresponding to a selection of the first affordance. For example, as shown in FIG. 3H, the first electronic device 301 detects a selection input 371B (e.g., an air pinch gesture) directed at a system option 318A displayed in a predetermined area of the display of the first electronic device 301.

[0135] In some examples, in response to receiving the input at 708, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of the second electronic device, which differs from the first representation of the first user interface. For example, as shown in FIG. 4A, the first electronic device 401 displays a virtual augmented display 404 that optionally includes a second user interface 425 configured to be displayed on a display 414' of the second electronic device 415' in a three-dimensional environment 450. In some examples, as shown in FIG. 7B, while presenting the computer-generated environment including the second representation of the second user interface, at 710, the first electronic device detects a first indication of a first discrete input received via one or more input devices of the second electronic device. For example, as shown in FIG. 4B, the first electronic device 401 receives input data including information corresponding to movement input 473A detected via a mouse 432' in communication with the second electronic device 415'.

[0136] In some examples, in response to detecting the first instruction at 712, in accordance with determining at 614 that a gaze of a user of the first electronic device is directed toward the first representation of the first user interface, the first electronic device causes the second electronic device to perform a first action directed toward the first user interface in accordance with the first discrete input. For example, in FIG. 4B , when movement input 473A is detected by the second electronic device 415′ via the mouse 432′, the gaze 427 of the user of the first electronic device 401 is directed toward the first user interface 411′, causing the first electronic device 401 to route the movement input 473A to the second electronic device 415′, which causes the second electronic device 415′ to move a cursor 412′ within the first user interface 411′, as shown in FIG. 4C . Alternatively, in some examples, before the movement input 473A is detected by the second electronic device 415′ via the mouse 432′, the first electronic device 401 may send an indication to the second electronic device 415′ that the input should be directed toward the first user interface 411′, causing the second electronic device 415′ to process the movement input 473A locally without sending information corresponding to the movement input 473A to the first electronic device 401. In some examples, in accordance with determining at 716 that the gaze is directed toward the second representation of the second user interface, the first electronic device performs a second action directed toward the second representation of the second user interface in accordance with the first discrete input. For example, in FIG. 4I, when movement input 473G is detected by the second electronic device 415′ via the mouse 432′, the user's line of sight 427 of the first electronic device 401 is directed toward the second user interface 425 of the virtual extended display 404, thereby causing the first electronic device 401 to move the cursor 412′ within the second user interface 425 in accordance with the movement input 473G, as shown in FIG. 4J.

[0137] It will be understood that process 700 is an example, and that more, fewer, or different operations may be performed in the same or a different order. Furthermore, the operations in process 700 described above are, optionally, implemented by executing one or more functional modules within an information processing device, such as a general-purpose processor (e.g., as described with respect to FIG. 2) or an application-specific chip, and / or by other components of FIG. 2.

[0138] FIG. 8 is a flow diagram illustrating an example process for facilitating display of and interaction between two electronic devices in a three-dimensional environment, according to some examples of the present disclosure. In some examples, process 800 begins at a first electronic device in communication with a display and one or more input devices of the first electronic device. In some examples, the first electronic device is, optionally, a head-mounted display similar to or corresponding to device 240 of FIG. 2. As shown in FIG. 8, in some examples, at 802, the first electronic device receives display data representing a second user interface of the second electronic device from a second electronic device while presenting, via the display, a computer-generated environment including a first user interface of an application executing on the first electronic device. For example, as also shown in FIG. 6A, the first electronic device 601 receives image data representing a user interface 625 from a second electronic device 615′ located within a field of view of a user of the first electronic device 601. In some examples, the second electronic device is optionally a computer similar to or corresponding to device 230 of FIG. 2 (eg, a laptop computer, a desktop computer, or a tablet computer).

[0139] In some examples, in response to receiving the display data at 804, the first electronic device presents, via a display, a representation of a second user interface of the second electronic device based on the display data. For example, as shown in FIG. 6C , the first electronic device 601 simultaneously displays a virtual display 604 including a representation of a user interface 625 along with a virtual application window 656 associated with an application running on the first electronic device 601. In some examples, while presenting the representations of the first user interface of the application and the second user interface of the second electronic device at 806, the first electronic device 601 detects a separate event corresponding to a user input. For example, as shown in FIG. 6E , the second electronic device 615′ detects a selection input 673B via a physical mouse 632′ in communication with the second electronic device 601′, or as shown in FIG. 6G , the first electronic device 601 detects a selection input 672B via one or more sensors of the first electronic device 615.

[0140] In some examples, in response to detecting the distinct event at 808, and in accordance with determining at 810 that a gaze of a user of the first electronic device is directed toward a representation of a second user interface of the second electronic device when the distinct event is detected, the first electronic device causes the second electronic device to perform a first action directed toward the second user interface of the second electronic device based on the distinct event. For example, in FIG. 6G , when a selection input 672B is detected by the first electronic device 601, the first electronic device 601 detects a gaze 627 directed toward the user interface 625 in the virtual display 604. Accordingly, as shown in FIG. 6H , the first electronic device 601 causes the second electronic device 615′ to select the text input field 616 of the user interface 625 based on the location of the gaze 627 shown in the representation of the user interface 625 displayed in the virtual display 604 within the three-dimensional environment 650.

[0141] In some examples, in accordance with a determination at 812 that the gaze is directed to a first user interface of the application running on the first electronic device when the respective event is detected, the first electronic device performs a second action (e.g., different from the first action described above) directed to the first user interface based on the respective event. For example, in FIG. 6E , when a selection input 673B is detected by the second electronic device 601′ via the mouse 632′, the first electronic device 615 detects a gaze 627 directed toward the virtual application window 656. Thus, as shown in FIG. 6F , the first electronic device 601 selects a pause option in the playback controls 645 of the virtual application window 656 based on the location of the gaze 627 in accordance with input data corresponding to the selection input 673B received from the second electronic device 615′ in FIG. 6E .

[0142] It will be understood that process 800 is an example, and that more, fewer, or different operations may be performed in the same or a different order. Furthermore, the operations in process 800 described above are, optionally, implemented by executing one or more functional modules within an information processing device, such as a general-purpose processor (e.g., as described with respect to FIG. 2) or an application-specific chip, and / or by other components of FIG. 2.

[0143] Thus, in accordance with the above, some examples of the present disclosure include capturing, at a first electronic device in communication with a display and one or more input devices of the first electronic device, at least a portion of a real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device via the one or more input devices; presenting, via the display, a computer-generated environment including the representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device; receiving, via the one or more input devices, an input corresponding to a selection of the first affordance while presenting the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance; and, in response to receiving the input, displaying a first user interface of the second electronic device in the computer-generated environment. presenting a second representation of a second user interface of a second electronic device, the second representation being different from the first representation of the first user interface; detecting a first indication of a first discrete input received via one or more input devices of the second electronic device while presenting a computer-generated environment including the second representation of the second user interface; and in response to detecting the first indication, causing the second electronic device to perform a first action directed to the first user interface in accordance with the first discrete input in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, and to perform a second action directed to the second representation of the second user interface in accordance with the first discrete input in accordance with a determination that the gaze is directed to the second representation of the second user interface.

[0144] Additionally or alternatively, in some examples, the first affordance is displayed in a predetermined area of the display separate from the representation of the second electronic device within the computer-generated environment. Additionally or alternatively, in some examples, the method further includes, in response to detecting the indication, ceasing to perform any operation on the first representation of the first user interface or the second representation of the second user interface in accordance with a determination that the user's gaze is directed to a distinct location within the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface. Additionally or alternatively, in some examples, the method includes detecting a second indication of a second distinct input received via one or more input devices of the second electronic device while presenting the computer-generated environment that includes the second representation of the second user interface, wherein after detecting the first indication, no indication of the input was detected prior to detecting the second indication; and, in response to detecting the second indication, ceasing to perform any operation on the first representation of the first user interface or the second representation of the second user interface in accordance with a determination that the user's gaze is directed to a distinct location within the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface. and, in accordance with a determination that the gaze was directed to the first representation of the first user interface in accordance with the second discrete input, causing the second electronic device to perform a third action directed to the first user interface in accordance with the second discrete input in accordance with a determination that the gaze was directed to the first representation of the first user interface when the first instruction was detected, and performing a fourth action directed to the second representation of the second user interface in accordance with the second discrete input in accordance with a determination that the gaze was directed to the second representation of the second user interface when the first instruction was detected.

[0145] Additionally or alternatively, in some examples, the method further includes detecting a distinct input via one or more input devices of the first electronic device while presenting a computer-generated environment including a second representation of the second user interface, and in response to detecting the distinct input, causing the second electronic device to perform a third action directed toward the first user interface in accordance with the distinct input in accordance with a determination that a gaze is directed toward the first representation of the first user interface, and a fourth action directed toward the second representation of the second user interface in accordance with the distinct input in accordance with a determination that a gaze is directed toward the second representation of the second user interface. Additionally or alternatively, in some examples, the first representation of the first user interface includes a cursor controllable via one or more of the one or more input devices of the second electronic device. Additionally or alternatively, in some examples, the first discrete input corresponds to a movement of a cursor, and in response to detecting the first instruction, performing a fourth action directed to the second representation of the second user interface in accordance with a determination that the gaze is directed to the second representation of the second user interface includes moving the cursor from the first representation of the first user interface to the second representation of the second user interface in accordance with the movement of the cursor. Additionally or alternatively, in some examples, one or more of the one or more input devices includes a first input device having a touch-sensitive surface, and the first discrete input includes a movement of a contact on the touch-sensitive surface that satisfies one or more criteria.

[0146] Additionally or alternatively, in some examples, the one or more criteria include a criterion that is met when the movement of the contact includes liftoff of the contact from the touch-sensitive surface. Additionally or alternatively, in some examples, the one or more criteria include a criterion that is met when the movement of the contact on the touch-sensitive surface exceeds a threshold movement. Additionally or alternatively, in some examples, the real-world environment further includes a third electronic device. In some examples, the method further includes, in response to detecting the first indication, ceasing to perform any action involving the third electronic device. Additionally or alternatively, in some examples, the method further includes pairing the first electronic device and the second electronic device such that the first electronic device communicates with the second electronic device before displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes scanning, via one or more input devices of the first electronic device, an image displayed via a display of the second electronic device that causes the first electronic device to communicate with the second electronic device. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes detecting, via one or more input devices of the first electronic device, a distinct sound output from the second electronic device that causes the first electronic device to communicate with the second electronic device. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes detecting an indication of an input received via one or more input devices of the second electronic device that causes the first electronic device to communicate with the second electronic device.

[0147] Additionally or alternatively, in some examples, the method further includes detecting disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment including the second representation of the second user interface, and in response to detecting disassociation of the first electronic device from the user, sending a disassociation instruction to the second electronic device including data indicating a state of the second representation of the second user interface, the instruction causing the second electronic device to simultaneously display the first user interface and the second user interface. Additionally or alternatively, in some examples, the method further includes detecting re-association of the first electronic device with a user of the first electronic device, presenting a computer-generated environment and sending an instruction to the second electronic device in response to detecting re-association of the first electronic device with the user, receiving data from the second electronic device while presenting the computer-generated environment indicating a current state of a second user interface, and re-displaying a second representation of the second user interface in the computer-generated environment based on the current state of the second user interface in response to receiving the data. Additionally or alternatively, in some examples, the first electronic device includes a head-mounted display, and the second electronic device is a laptop computer, a desktop computer, or a tablet computer. Additionally or alternatively, in some examples, the method further includes, while presenting the computer-generated environment, receiving an invitation to join a communication session with a user of a third electronic device, and, in response to receiving the invitation, entering the communication session with the user of the third electronic device in accordance with a determination that the invitation has been accepted by the user of the first electronic device, and simultaneously displaying an avatar corresponding to the user of the third electronic device and a first representation of the first user interface.

[0148] Additionally or alternatively, in some examples, the method further includes receiving, while presenting a computer-generated environment including an avatar corresponding to a user of the third electronic device and a first representation of the first user interface, an instruction from the third electronic device corresponding to a request to share content with the first electronic device, and, in response to receiving the instruction, displaying, in the computer-generated environment, a third representation of the third user interface corresponding to the content in accordance with a determination that the request is accepted by the user of the first electronic device, the third user interface being configured to be displayed on the third electronic device. Additionally or alternatively, in some examples, the method further includes, while presenting a computer-generated environment including the third representation of the third user interface, detecting, via one or more input devices of the first electronic device, an input directed to the third representation of the third user interface, and, in response to detecting the input, ceasing to perform an action directed to the third representation of the third user interface in accordance with the input.

[0149] Some examples of the present disclosure include receiving, from a second electronic device, display data representing a second user interface of the second electronic device while presenting, via a display at a first electronic device in communication with the display and one or more input devices of the first electronic device, a computer-generated environment including a first user interface of an application running on the first electronic device; and, in response to receiving the display data, presenting, via the display, a representation of the second user interface of the second electronic device based on the display data; and, while presenting the first user interface of the application and the representation of the second user interface of the second electronic device. detecting a discrete event corresponding to a user input; and in response to detecting the discrete event, causing the second electronic device to perform a first action directed to a second user interface of the second electronic device based on the discrete event in accordance with a determination that a gaze of a user of the first electronic device is directed to a representation of a second user interface of the second electronic device when the discrete event is detected, and causing the second electronic device to perform a second action directed to the first user interface based on the discrete event in accordance with a determination that the gaze is directed to a first user interface of an application running on the first electronic device.

[0150] Additionally or alternatively, in some examples, detecting the discrete event includes detecting an indication of a discrete input received via one or more input devices in communication with the second electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting the indication of the discrete input, transmitting gaze data related to a location of the gaze to the second electronic device in accordance with a determination that a gaze is directed toward a representation of a second user interface of the second electronic device when the indication is detected, wherein the second electronic device performs the first operation in accordance with the data related to the location of the gaze without transmitting input data corresponding to the discrete input to the first electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting an indication of the discrete input, transmitting gaze data related to a location of the gaze to the second electronic device in accordance with a determination that the gaze was directed toward a representation of a second user interface of the second electronic device when the indication was detected, the second electronic device performing a first operation in accordance with the data related to the location of the gaze, and receiving input data from the second electronic device corresponding to the discrete input, wherein the first operation is performed at the second electronic device accordingly. Additionally or alternatively, in some examples, the method further includes, in response to detecting an indication of the discrete input, in accordance with a determination that the gaze was directed toward a first user interface of an application running on the first electronic device when the indication was detected, performing a second operation directed toward the first user interface in accordance with the discrete input without transmitting input data corresponding to the discrete input to the second electronic device.

[0151] Additionally or alternatively, in some examples, detecting the discrete event includes detecting a discrete input via one or more input devices of the first electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting the discrete input, transmitting input data corresponding to the discrete input to the second electronic device in accordance with a determination that a gaze was directed toward a representation of a second user interface of the second electronic device when the discrete input was detected. Additionally or alternatively, in some examples, the method further includes, in response to detecting the discrete input, performing a second operation directed toward the first user interface in accordance with the discrete input in accordance with a determination that a gaze was directed toward a first user interface of the application running on the first electronic device when the discrete input was detected, without transmitting input data corresponding to the discrete input to the second electronic device. Additionally or alternatively, in some examples, causing the second electronic device to perform the first operation includes transmitting gaze data to the second electronic device including information indicating that a gaze was directed toward a representation of a second user interface of the second electronic device when the discrete event was detected. Additionally or alternatively, in some examples, the information includes position information corresponding to a location of the gaze within a representation of the second user interface. Additionally or alternatively, in some examples, the second electronic device is not displaying the second user interface.

[0152] Additionally or alternatively, in some examples, causing the second electronic device to perform the first operation directed to the second user interface includes updating an appearance of the second user interface. Additionally or alternatively, in some examples, the method further includes, after causing the second electronic device to perform the first operation directed to the second user interface, receiving updated display data from the second electronic device representing one or more updates to an appearance of the second user interface of the second electronic device, and in response to receiving the updated display data, updating a display of a representation of the second user interface based on the updated display data. Additionally or alternatively, in some examples, the second electronic device is displaying the second user interface. Additionally or alternatively, in some examples, the method further includes presenting, in the computer-generated environment via the display, a first affordance associated with a representation of the second electronic device before receiving display data representing a second user interface of the second electronic device from the second electronic device; receiving, while presenting the computer-generated environment including the first affordance, input via one or more input devices corresponding to a selection of the first affordance; and presenting, in the computer-generated environment, a representation of the second user interface of the second electronic device based on the display data in response to receiving the input. Additionally or alternatively, in some examples, the method further includes causing the second electronic device to stop displaying the second user interface in response to receiving the input. Additionally or alternatively, in some examples, the method further includes pairing the first electronic device and the second electronic device such that the first electronic device communicates with the second electronic device before displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment.

[0153] Additionally or alternatively, in some examples, the method further includes detecting disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment, and in response to detecting disassociation of the first electronic device from the user, sending a disassociation instruction to the second electronic device, the instruction causing the second electronic device to display a second user interface. Additionally or alternatively, in some examples, the method further includes detecting reassociation of the first electronic device with a user of the first electronic device, and in response to detecting reassociation of the first electronic device with the user, presenting the computer-generated environment and sending a reassociation instruction to the second electronic device, receiving updated display data from the second electronic device while presenting the computer-generated environment, the updated display data representing a current state of the second user interface, and in response to receiving the updated display data, re-displaying a representation of the second user interface in the computer-generated environment based on the current state of the second user interface. Additionally or alternatively, in some examples, the method further includes, while presenting the computer-generated environment, receiving an invitation to join a communication session with a user of a third electronic device, and, in response to receiving the invitation, entering the communication session with the user of the third electronic device in accordance with a determination that the invitation has been accepted by the user of the first electronic device and simultaneously displaying an avatar corresponding to the user of the third electronic device and a first representation of the first user interface. Additionally or alternatively, in some examples, the method further includes, while presenting the computer-generated environment including an avatar corresponding to the user of the third electronic device and a first user interface of an application running on the first electronic device, receiving an instruction from the third electronic device corresponding to a request to share content with the first electronic device, and, in response to receiving the instruction, displaying in the computer-generated environment a representation corresponding to content of the third user interface configured to be displayed on the third electronic device in accordance with a determination that the request has been accepted by the user of the first electronic device.

[0154] Some examples of the present disclosure are directed to electronic devices comprising one or more processors, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0155] Some examples of the present disclosure are directed to a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the methods described above.

[0156] Some examples of the present disclosure are directed to electronic devices comprising one or more processors, a memory, and means for performing any of the above methods.

[0157] Some examples of the present disclosure are directed to an information processing apparatus for use in an electronic device, the information processing apparatus comprising means for performing any of the above methods.

[0158] The foregoing description has been set forth with reference to specific embodiments for purposes of explanation. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular applications contemplated.

Claims

1. a first electronic device in communication with a display and one or more input devices of the first electronic device, receiving, from a second electronic device, display data representing a second user interface of the second electronic device while presenting, via the display, a computer-generated environment including a first user interface of an application running on the first electronic device; responsive to receiving the display data, presenting via the display a representation of the second user interface of the second electronic device based on the display data; Detecting a discrete event corresponding to a user input while presenting the representation of the first user interface of the application and the second user interface of the second electronic device; In response to detecting the individual event, in accordance with determining that a gaze of a user of the first electronic device is directed toward the representation of the second user interface of the second electronic device when the respective event is detected, causing the second electronic device to perform a first action directed toward the second user interface of the second electronic device based on the respective event; performing a second action directed to the first user interface based on the respective event in accordance with determining that the gaze is directed to the first user interface of the application running on the first electronic device; A method comprising:

2. The method of claim 1 , wherein detecting the discrete event comprises detecting an indication of a discrete input received via one or more input devices in communication with the second electronic device.

3. In response to detecting the individual input instruction, and further comprising: transmitting gaze data relating to a location of the gaze to the second electronic device in accordance with a determination that the gaze is directed toward the representation of the second user interface of the second electronic device when the instruction is detected, wherein the second electronic device performs the first operation in accordance with the data relating to the location of the gaze without transmitting input data corresponding to the individual input to the first electronic device. The method of claim 2.

4. In response to detecting the individual input instruction, in accordance with determining that the gaze is directed at the representation of the second user interface of the second electronic device when the instruction is detected, transmitting gaze data regarding a location of the gaze to the second electronic device, the second electronic device performing the first action in accordance with the gaze data regarding the location of the gaze; receiving input data corresponding to the respective input from the second electronic device according to which the first operation is performed at the second electronic device; The method of claim 2 further comprising:

5. In response to detecting the individual input instruction, performing the second action directed to the first user interface in accordance with the individual input without transmitting input data corresponding to the individual input to the second electronic device in accordance with determining that the gaze is directed to the first user interface of the application running on the first electronic device when the instruction is detected; The method of claim 2 further comprising:

6. The method of claim 1 , wherein detecting the discrete event comprises detecting the discrete input via the one or more input devices of the first electronic device.

7. In response to detecting the individual input, transmitting input data corresponding to the discrete input to the second electronic device in accordance with a determination that the gaze is directed toward the representation of the second user interface of the second electronic device when the discrete input is detected; The method of claim 6 further comprising:

8. In response to detecting the individual input, in accordance with determining that the gaze is directed toward the first user interface of the application running on the first electronic device when the discrete input is detected, performing the second action directed toward the first user interface in accordance with the discrete input without transmitting input data corresponding to the discrete input to the second electronic device; The method of claim 6 further comprising:

9. 2. The method of claim 1, wherein causing the second electronic device to perform the first action includes transmitting gaze data to the second electronic device including information indicating that the gaze was directed at the representation of the second user interface of the second electronic device when the discrete event was detected.

10. The method of claim 9 , wherein the information includes position information corresponding to a location of the gaze within the representation of the second user interface.

11. The method of claim 1 , wherein the second electronic device is not displaying the second user interface.

12. causing the second electronic device to perform the first action directed to the second user interface includes updating a visual appearance of the second user interface, the method comprising: receiving, from the second electronic device, updated display data representing one or more updates to the visual appearance of the second user interface of the second electronic device after causing the second electronic device to perform the first action directed to the second user interface; 10. The method of claim 1, further comprising: in response to receiving the updated display data, updating a display of the representation of the second user interface based on the updated display data.

13. 1. A method, the second electronic device displaying the second user interface, the method comprising: presenting, via the display, in the computer-generated environment, a first affordance associated with a representation of the second electronic device prior to receiving, from the second electronic device, the display data representing the second user interface of the second electronic device; receiving, via the one or more input devices, input corresponding to a selection of the first affordance while presenting the computer-generated environment including the first affordance; 10. The method of claim 1, further comprising: in response to receiving the input, presenting, in the computer-generated environment, the representation of the second user interface of the second electronic device based on the display data.

14. In response to receiving the input, causing the second electronic device to stop displaying a second user interface; The method of claim 13 further comprising:

15. pairing the first electronic device and the second electronic device such that the first electronic device communicates with the second electronic device prior to displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment; The method of claim 13 further comprising:

16. Detecting a disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment; In response to detecting a disassociation of the first electronic device from the user, sending to the second electronic device an instruction to disassociate, the instruction causing the second electronic device to display the second user interface; The method of claim 1 further comprising:

17. Detecting a re-association of the first electronic device with the user of the first electronic device; In response to detecting the re-association of the first electronic device with the user, presenting the computer-generated environment; and sending the reassociation indication to the second electronic device; receiving updated display data from the second electronic device while presenting the computer-generated environment, the updated display data representing a current state of the second user interface; In response to receiving the updated display data, redisplaying the representation of the second user interface in the computer-generated environment based on the current state of the second user interface; and 17. The method of claim 16, further comprising:

18. receiving an invitation to participate in a communication session with a user of a third electronic device while presenting the computer-generated environment; In response to receiving the invitation, entering the communication session with the user of the third electronic device, including simultaneously displaying an avatar corresponding to the user of the third electronic device and the first user interface in accordance with a determination that the invitation has been accepted by the user of the first electronic device; The method of claim 1 further comprising:

19. 1. An electronic device comprising: one or more processors; Memory and and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 18.

20. 19. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 1 to 18.

Images