DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR PRESENTING VIRTUAL OBJECTS IN A VIRTUAL ENVIRONMENT - Patent application

Abstract

To provide systems and methods for displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment.SOLUTION: While presenting a first user interface, a first electronic device presents a representation of a second user interface of a second electronic device based on display data representing the second user interface. In response to detecting an individual event corresponding to user input, if a gaze of a user of the first electronic device is directed to the representation of the second user interface, then the first electronic device causes the second electronic device to perform a first operation directed to the second user interface based on the individual event. Alternatively, if the gaze is directed to the first user interface, then the first electronic device performs a second operation directed to the first user interface based on the individual event.SELECTED DRAWING: None

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 498,239, filed April 25, 2023, U.S. Patent Application No. 18 / 429,294, filed January 31, 2024, and U.S. Patent Application No. 18 / 429,301, filed January 31, 2024, the contents of which are incorporated by reference in their entirety herein for all purposes. [Technical field]

[0002] The present invention relates generally to systems and methods for displaying a representation of a user interface of an electronic device and facilitating interaction with the representation of the user interface of an electronic device. [Background technology]

[0003] Many types of computing systems are available, such as mobile phones, tablets, touchpads, portable or desktop computers, portable media players, and wearable devices. The computing systems may include a display or touch screen to allow user interaction. However, these computing systems are limited by the physical constraints of the display or touch screen. Summary of the Invention

[0004] Some examples of the present disclosure are directed to systems and methods for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment. In some examples, the representation of the user interface is displayed within a virtual augmented display of the electronic device within the three-dimensional environment. In some examples, a first electronic device may be in communication with a display of the first electronic device and one or more input devices. In some examples, the first electronic device captures at least a portion of a real-world environment surrounding the first electronic device via the one or more input devices, the real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. In some examples, the second electronic device is in communication with the first electronic device. In some examples, the first electronic device presents, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. In some examples, the first affordance is displayed in a predetermined area of ​​the display of the first electronic device. In some examples, while presenting a computer-generated environment including a representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, an input corresponding to a selection of the first affordance. In some examples, in response to receiving the input, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of the second electronic device that differs from the first representation of the first user interface. In some examples, the second representation of the second user interface is displayed on a virtual augmented display in the computer-generated environment.

[0005] In some examples, while presenting a computer-generated environment including a second representation of a second user interface, the first electronic device detects a first indication of a first individual input received via one or more input devices of the second electronic device. In some examples, in response to detecting the first indication, in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, causes the second electronic device to perform a first action directed to the first user interface in accordance with the first individual input. In some examples, in accordance with a determination that the gaze is directed to the second representation of the second user interface, causes the second electronic device to perform a second action directed to the second representation of the second user interface in accordance with the first individual input.

[0006] A full description of these examples is provided in the Drawings and Detailed Description, and it should be understood that this Summary is not intended to limit the scope of the present disclosure in any way. [Brief description of the drawings]

[0007] For a better understanding of the various examples described herein, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0008] [Figure 1] 1 illustrates an electronic device presenting an augmented reality environment, according to some examples of the present disclosure.

[0009] [Diagram 2] FIG. 1 illustrates a block diagram of an exemplary architecture of a system or device according to an embodiment of the present disclosure.

[0010] [Figure 3A] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3B]1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3C] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3D] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3E] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3F] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3G] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3H] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 3I] 1 illustrates an example interaction for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0011] [Figure 4A] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4B] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4C]1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4D] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4E] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4F] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4G] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4H] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4I] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4J] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4K] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4L] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4M]1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4N] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 4O] 1 illustrates an example of a first electronic device that facilitates interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0012] [Figure 5A] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5B] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5C] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5D] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5E] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5F] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure. [Figure 5G] 1 illustrates exemplary interactions with representations of user interfaces of electronic devices within a multi-user communication session, according to some examples of the present disclosure.

[0013] [Figure 6A]1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6B] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6C] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6D] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6E] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6F] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6G] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6H] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6I] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6J]1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure. [Figure 6K] 1 illustrates an example of a first electronic device that facilitates a representation of a user interface of a second electronic device and an interaction with a user interface of the first electronic device, according to some examples of the present disclosure.

[0014] [Figure 7A] 1 is a flow diagram illustrating an example process for facilitating displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. [Figure 7B] 1 is a flow diagram illustrating an example process for facilitating displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure.

[0015] [Figure 8] 1 is a flow diagram illustrating an example process for facilitating display and interaction of representations of a user interface between two electronic devices in a three-dimensional environment, according to some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Some examples of the present disclosure are directed to systems and methods for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment. In some examples, the representation of the user interface is displayed within a virtual augmented display of the electronic device within the three-dimensional environment. In some examples, a first electronic device may be in communication with a display of the first electronic device and one or more input devices. In some examples, the first electronic device captures at least a portion of a real-world environment surrounding the first electronic device via the one or more input devices, the real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. In some examples, the second electronic device is in communication with the first electronic device. In some examples, the first electronic device presents, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. In some examples, the first affordance is displayed in a predetermined area of ​​the display of the first electronic device. In some examples, while presenting a computer-generated environment including a representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, an input corresponding to a selection of the first affordance. In some examples, in response to receiving the input, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of the second electronic device that differs from the first representation of the first user interface. In some examples, the second representation of the second user interface is displayed on a virtual augmented display in the computer-generated environment.

[0017] In some examples, while presenting a computer-generated environment including a second representation of a second user interface, the first electronic device detects a first indication of a first individual input received via one or more input devices of the second electronic device. In some examples, in response to detecting the first indication, in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface, causes the second electronic device to perform a first action directed to the first user interface in accordance with the first individual input. In some examples, in accordance with a determination that the gaze is directed to the second representation of the second user interface, causes the second electronic device to perform a second action directed to the second representation of the second user interface in accordance with the first individual input.

[0018] In some examples, displaying a virtual augmented display in a three-dimensional (e.g., computer-generated) environment configured to display a user interface of a real-world electronic device may include interacting with one or more user interface objects in the three-dimensional environment. For example, initiating the display of the virtual augmented display in the three-dimensional environment may include interacting with one or more virtual options / affordances displayed in the three-dimensional environment. In some examples, the user's gaze may be tracked by the electronic device as an input to identify one or more virtual options / affordances to be selected when initiating the display of the virtual augmented display in the three-dimensional environment. For example, the gaze may be used to identify one or more virtual options / affordances to be selected using another selection input. In some examples, the virtual options / affordances may be selected using hand tracking input detected via an input device in communication with the electronic device. In some examples, objects displayed in the three-dimensional environment may be moved and / or reoriented within the three-dimensional environment according to movement input detected via the input device.

[0019] FIG. 1 illustrates an electronic device 101 presenting an extended reality (XR) environment (e.g., a computer-generated environment) according to some examples of the present disclosure. In some examples, the electronic device 101 is a handheld or mobile device, such as a tablet computer, a laptop computer, a smartphone, or a head-mounted display. Examples of the electronic device 101 are described below with reference to the architecture block diagram of FIG. 2. As illustrated in FIG. 1, the electronic device 101, a table 106, a computer 115, and a floor lamp 132 are located in a physical environment 100. The physical environment may include physical features, such as physical surfaces (e.g., floors, walls) or physical objects (e.g., tables, lamps, etc.). In some examples, the electronic device 101 may be configured to capture an image of the physical environment 100, including the table 106, the computer 115, and the floor lamp 132 (shown within the field of view of the electronic device 101). In some examples, in response to a trigger, the electronic device 101 may be configured to display within the computer-generated environment (e.g., represented by the rectangle shown in FIG. 1 ) a virtual object 104 (e.g., two-dimensional virtual content) that is not present within the physical environment 100 but is displayed within the computer-generated environment positioned (e.g., fixed) on top of a computer-generated representation 106′ of a real-world table 106. For example, in response to detecting a flat surface of the table 106 within the physical environment 100 (or some other trigger), the virtual object 104 may be displayed on a surface of the computer-generated representation 115′ of a table in the computer-generated environment next to the computer-generated representation 106′ of the real-world table 115 displayed via the electronic device 101.

[0020] It should be understood that the virtual object 104 is a representative virtual object, and that one or more different virtual objects (e.g., of various dimensions, such as two-dimensional or three-dimensional objects) may be included and rendered in a two-dimensional or three-dimensional computer-generated environment. For example, the virtual object may represent an application or a user interface displayed in the computer-generated environment. In some examples, the virtual object may represent content corresponding to an application and / or displayed via a user interface in the computer-generated environment. In some examples, the user interface is a user interface that is also configured to be displayed via (e.g., a display of) the computer 115 in the physical environment 100. In some examples, the virtual object 104 is interactive and optionally configured to respond to user input, such that a user may virtually touch, tap, move, rotate, or otherwise interact with the virtual object 104. In addition, it should be understood that the 3D environment (or 3D virtual object) described herein may be a representation of the 3D environment (or 3D virtual object) projected or presented in an electronic device.

[0021] In the following description, an electronic device is described that communicates with a display generating component and one or more input devices. It should be understood that the electronic device optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand tracking device, an eye tracking device, a stylus, etc. It should further be understood that, as described above, the described electronic device, display, and touch-sensitive surface are optionally distributed across two or more devices. Thus, as used in this disclosure, information displayed on or by an electronic device may optionally be used to describe information output by the electronic device for display on another display device (touch-sensitive or non-touch-sensitive). Similarly, as used in this disclosure, input received at an electronic device (e.g., touch input received on a touch-sensitive surface of the electronic device, or touch input received on the surface of a stylus) is optionally used to describe input received at a separate input device from which the electronic device receives the input information.

[0022] The device typically supports a variety of applications, such as one or more of drawing applications, presentation applications, word processing applications, website creation applications, disc authoring applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, training support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music playback applications, television channel browsing applications, and / or digital video playback applications.

[0023] FIG. 2 illustrates a block diagram of an exemplary architecture of a system or device 200 according to an example of the disclosure. In some examples, system 200 is a mobile device, such as a mobile phone (e.g., a smartphone), a tablet computer, a laptop computer, a desktop computer, a head-mounted display, an auxiliary device that communicates with another device, etc. System 200 optionally includes various sensors (e.g., one or more hand tracking sensor(s), one or more location sensor(s), one or more image sensor(s), one or more touch-sensitive surface(s), one or more motion and / or orientation sensor(s), one or more eye tracking sensor(s), one or more microphone(s) or other audio sensor(s), etc.), one or more display generating component(s), one or more speaker(s), one or more processor(s), one or more memories, and / or communication circuitry. One or more communication buses are optionally used for communication between the above-mentioned components of system 200.

[0024] In some examples, system / device 200 may be divided among multiple devices, as shown in Figure 2. For example, a first device 230 optionally includes a processor(s) 218A, one or more memories 220A, communications circuitry 222A, touch-sensitive surface(s) 216A, and display generating component(s) or display(s) 214A, which optionally communicate via a communications bus 208A. Second device 240 (e.g., corresponding to electronic device 101) optionally includes various sensors (e.g., one or more hand tracking sensor(s) 202, one or more location sensor(s) 204, one or more image sensor(s) 206, one or more touch-sensitive surface(s) 216B, one or more movement and / or orientation sensor(s) 210, one or more eye tracking sensor(s) 212, one or more microphone(s) 213 or other audio sensor(s), etc.), one or more display generating component(s) 214B, one or more speaker(s) 216, one or more processors 218B, one or more memories 220B, and / or communications circuitry 222B. One or more communications buses 208B are optionally used for communication between the above-mentioned components of device 240. First device 230 and second device 240 optionally communicate (eg, via communications circuitry 222A-222B) via a wired or wireless connection between the two devices.

[0025] The communications circuitry 222A, 222B optionally includes circuitry for communicating with electronic devices, networks such as the Internet, an intranet, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). The communications circuitry 222A, 222B optionally includes circuitry for communicating using short-range communications such as near-field wireless communications (NFC) and / or Bluetooth.

[0026] The processor(s) 218A, 218B include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, the memory 220A, 220B is a non-transitory computer-readable storage medium (e.g., flash memory, random access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by the processor(s) 218A, 218B to perform the techniques, processes, and / or methods described below. In some examples, the memory 220A, 220B can include two or more non-transitory computer-readable storage media. A non-transitory computer-readable storage medium can be any medium (other than a signal) that can tangibly store or store computer-executable instructions used by or in connection with an instruction execution system, apparatus, or device. In some embodiments, the storage medium is a transitory computer-readable storage medium. In some embodiments, the storage medium is a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include, but is not limited to, magnetic, optical, and / or semiconductor storage devices. Examples of such storage devices include magnetic disks, optical disks based on CD, DVD, or Blu-ray technology, and persistent solid-state memory such as flash, solid-state drives, and the like.

[0027] In some examples, the display generating component(s) 214A, 214B include a single display (e.g., a liquid crystal display (LCD), organic light emitting diode (OLED), or other type of display). In some examples, the display generating component(s) 214A, 214B include multiple displays. In some examples, the display generating component(s) 214A, 214B can include a display with touch capabilities (e.g., a touch screen), a projector, a holographic projector, a retina projector, etc. In some examples, the device 240 includes a touch-sensitive surface(s) 216B for receiving user input, such as tap and swipe inputs or other gestures. In some examples, display generating component(s) 214A, 214B and touch-sensitive surfaces 216A, 216B form a touch-sensitive display (e.g., a touchscreen integrated with devices 230, 240 or a touchscreen external to devices 230, 240 that communicates with devices 230, 240).

[0028] Device 240 optionally includes image sensor(s) 206. Image sensor(s) 206 optionally include one or more visible light image sensors, such as charge-coupled device (CCD) sensors and / or complementary metal-oxide semiconductor (CMOS) sensors operable to acquire images of physical objects from the real-world environment. Image sensor(s) 206 also optionally include one or more infrared (IR) sensors, such as passive or active IR sensors, for detecting infrared light from the real-world environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real-world environment. Image sensor(s) 206 also optionally include one or more cameras configured to capture movement of physical objects in the real-world environment. Image sensor(s) 206 optionally include one or more depth sensors configured to detect a distance of a physical object from device 240. In some examples, information from one or more depth sensors may enable a device to identify an object in a real-world environment and distinguish it from other objects in the real-world environment. In some examples, one or more depth sensors may enable a device to determine the texture and / or topography of an object in the real-world environment.

[0029] In some examples, the device 240 uses a combination of a CCD sensor, an event camera, and a depth sensor to detect the physical environment around the device 240. In some examples, the image sensor(s) 206 include a first image sensor and a second image sensor. The first image sensor and the second image sensor cooperate and are optionally configured to capture different information of a physical object in the real-world environment. In some examples, the first image sensor is a visible light image sensor and the second image sensor is a depth sensor. In some examples, the device 240 uses the image sensor(s) 206 to detect the position and orientation of the device 240 and / or the display generating component(s) 214 in the real-world environment. For example, the device 240 uses the image sensor(s) 206 to track the position and orientation of the display generating component(s) 214B relative to one or more fixed objects in the real-world environment.

[0030] In some examples, device 240 includes microphone(s) 213 or other audio sensors. Device 240 uses microphone(s) 213 to detect sounds from the user and / or the user's real-world environment. In some examples, microphone(s) 213 optionally include an array of microphones (including multiple microphones) working in concert to identify ambient noise, to identify the location of a sound source within the space of the real-world environment, etc.

[0031] Device 240 optionally includes location sensor(s) 204 configured to detect the location of device 240 and / or display generating component(s) 214B. For example, location sensor(s) 204 may include a GPS receiver that receives data from one or more satellites, allowing device 240 to determine the absolute position of the device in the physical world.

[0032] The device 240 includes orientation sensor(s) 210 that detects the orientation and / or movement of the device 240 and / or the display generating component(s) 214B. For example, the device 240 uses the orientation sensor 210 to track changes in position and / or orientation of the device 240 and / or the display generating component(s) 214B with respect to physical objects in a real-world environment, etc. The orientation sensor(s) 210 optionally include one or more gyroscopes and / or one or more accelerometers.

[0033] In some examples, the device 240 optionally includes hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212. The hand tracking sensor(s) 202 are configured to track the position / location of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the augmented reality environment, relative to the display generating component(s) 214B, and / or relative to another defined coordinate system. The eye tracking sensor(s) 212 are configured to track the position and movement of the user's gaze (more generally, eyes, face, or head) relative to the real world or the augmented reality environment, and / or relative to the display generating component(s) 214B. In some examples, the hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212 are implemented together with the display generating component(s) 214B. In some examples, the hand tracking sensor(s) 202 and / or the eye tracking sensor(s) 212 are implemented separately from the display generation component(s) 214B.

[0034] In some examples, the hand tracking sensor(s) 202 can use image sensor(s) 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that capture three-dimensional information from the real world including one or more hands (e.g., of a human user). In some examples, the hand can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some examples, the one or more image sensor(s) 206 are positioned relative to the user such that they define the field of view of the image sensor(s) 206 and an interaction space in which the position, orientation, and / or movement of the fingers / hands captured by the image sensor(s) are used as input (e.g., to distinguish from a stationary hand of the user or other hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touches, taps, etc.) can be advantageous in that the user does not need to touch, hold, or wear any kind of beacon, sensor, or other marker.

[0035] In some examples, the eye tracking sensor(s) 212 includes at least one eye tracking camera (e.g., infrared (IR) camera) and / or illumination source (e.g., IR light source, LED, etc.) that emits light toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR light from the light source directly or indirectly from the eye. In some examples, both eyes are tracked separately by respective eye tracking cameras and illumination sources, and focus / gaze can be determined from tracking of both eyes. In some examples, one eye (e.g., dominant eye) is tracked by a separate eye tracking camera / illumination source(s).

[0036] System 200 is not limited to the components and configuration of FIG. 2 and may include fewer, alternative, or additional components in multiple configurations. In some examples, system 200 may be implemented in a single device. A person using system 200 is optionally referred to herein as a user of the system (e.g., and / or a user of the device). Attention is now directed to an exemplary interaction that includes initiating a display of a virtual augmented display associated with a physical electronic device seen in a three-dimensional environment. As described below, the three-dimensional environment may be displayed using a first electronic device, and a second electronic device different from the first electronic device may be located within a physical environment included in the three-dimensional environment, and the first electronic device is configured to present a virtual augmented display of the second electronic device. In some examples, the process of initiating the display of the virtual augmented display and facilitating an interaction with the virtual augmented display in the three-dimensional environment described below may be performed by processors 218A, 218B of devices 230 and 240.

[0037] 3A-3I illustrate example interactions for initiating the display of a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, a first electronic device 301 may present a three-dimensional environment 350. The first electronic device 301 may be similar to the electronic devices 101 or 240 described above, and / or may be a head-mounted system / device and / or a projection-based system / device (including a hologram-based system / device) configured to generate and present a three-dimensional environment, such as, for example, a head-up display (HUD), a head-mounted display (HMD), a window with integrated display capabilities, or a display formed as a lens designed to be placed on a person's eye (e.g., similar to a contact lens). In the examples of FIG. 3A-3I, a user is optionally wearing the first electronic device 301 such that the three-dimensional environment 350 may be defined by an X-axis, a Y-axis, and a Z-axis as viewed from the perspective of the first electronic device (e.g., a perspective associated with the user of the electronic device 301). Thus, as used herein, the first electronic device 301 is configured to be movable in six degrees of freedom based on movement of a user (e.g., the user's head) such that the first electronic device 301 can be moved in a roll direction, a pitch direction, and / or a yaw direction.

[0038] 3A, the first electronic device 301 may be placed in a physical environment including a table 306 (e.g., corresponding to the real-world table 106 of FIG. 1), a second electronic device 315, and one or more physical input devices 331 / 332. Thus, the three-dimensional environment 350 presented using the first electronic device 301 optionally includes captured portions of the physical environment surrounding the electronic device 301, such as a representation of the table 306', a representation of the second electronic device 315', and one or more representations of the one or more physical input devices 331' / 332'. In some examples, the second electronic device 315' may be similar to the device 230 described above and / or may be a desktop computer, a laptop computer, a tablet computer, or other mobile computing device. In some examples, the one or more physical input devices may include a keyboard 331′ and a mouse 332′ (or trackpad) that communicate with the second electronic device 315′ (e.g., integrated with the second electronic device 315′ or that communicates with the second electronic device 315′ via wired or wireless communication means). It should be understood that the input devices (e.g., keyboard 331′ and mouse 332′) of FIG. 3A are merely representative, and that additional, fewer, or different input devices may communicate with the second electronic device 315′. In some examples, the representation may include a portion of the physical environment as seen through a transparent or translucent display of the first electronic device 301.

[0039] In some examples, as shown in FIG. 3A, the second electronic device 315' in the three-dimensional environment 350 is configured to display one or more user interfaces. For example, as shown in FIG. 3A, the second electronic device 315' includes a display that is visible in the three-dimensional environment 350 as a representation of the display 314'. In some examples, the display 314' displays a first user interface (e.g., "Window A") that is visible in the three-dimensional environment 350 as a representation of the first user interface 311'. In some examples, the first user interface 311' is a user interface of a web browsing application (e.g., "browser") that includes a search bar that is visible in the three-dimensional environment 350 as a representation of the search bar 313', as shown in FIG. 3A. Additionally, as shown in FIG. 3A, the first user interface 311' may be displayed with a dock that is visible in the three-dimensional environment 350 as a representation of the dock 309' that includes a plurality of icons that correspond to applications, images, or other files local to the second electronic device 315'. Finally, as shown in FIG. 3A , in some examples, the second electronic device 315′ may display a cursor (e.g., or other pointer object) visible in the three dimensional environment 350 as a representation of a cursor 312′ controllable via a mouse 332′ (e.g., movable and / or controllable via the mouse 332′ to provide selection and associated inputs directed to user interface elements) on the display 314′.

[0040] In some examples, it may be advantageous to provide a fully functional working environment in three dimensions that is representative of customary interaction with a second electronic device in a real-world environment. Thus, providing a virtual augmented display configured to include a representation of one or more user interfaces of the second electronic device in a three-dimensional environment may be particularly useful for interacting with one or more user interfaces of the second electronic device without the constraints of the physical display (e.g., display 314') of the second electronic device. For example, as described herein, the virtual augmented display may provide for displaying one or more user interfaces of the second electronic device at a size that exceeds the dimensions of the physical display of the second electronic device (e.g., enlarging the user interface or providing multiple areas for display of the user interface beyond the area corresponding to the display of the second electronic device). In some examples, it may be advantageous to present one or more objects of the second electronic device (e.g., objects corresponding to three-dimensional graphical data) in three dimensions. For example, two-dimensional content may be viewed or edited in three dimensions, while the display of the second electronic device is limited to two dimensions. In such examples, a user creating content for display in a three-dimensional environment may view the content in three dimensions via the virtual augmented display. In some examples, it may be advantageous to facilitate interaction with content of a virtual augmented display based on input detected via a physical input device in communication with a second electronic device, as described in more detail herein.

[0041] In some examples, the first electronic device 301 may provide pairing with the second electronic device 315' to initiate display of a virtual augmented display associated with the second electronic device 315' in the three-dimensional environment 350. For example, as shown in FIG. 3B, the first electronic device 301 may display a user interface object 317 associated with a separate application (e.g., a pairing application or a setting application running on the first electronic device 301) that provides pairing the first electronic device 301 with one or more other electronic devices. In some examples, as shown in FIG. 3B, the user interface object 317 includes a first indication 317-1 corresponding to "Device 1." In some examples, device 1 corresponds to the second electronic device 301' that is not currently paired with the first electronic device 315 in FIG. 3B. In some examples, the first instruction 317-1 is included in the user interface object 317 because the second electronic device 301' is within a threshold distance (e.g., 0.1, 0.25, 0.5, 1, 2, 3, 5, 10, 15, 20 m, etc.) of the first electronic device 315. In some examples, the first instruction 317-1 is included in the user interface object 317 because the second electronic device 315' is visible within a field of view of the first electronic device 301 or a user of the first electronic device 301 (e.g., the second electronic device 315' is located in a portion of the physical environment visible in the three-dimensional environment 350 from the user's current viewpoint). In some examples, the first instruction 317-1 is selectable to initiate a process for pairing the second electronic device 315' with the first electronic device 301 to enable the first electronic device 301 to present a virtual augmented display associated with the second electronic device 315'.

[0042] In some examples, as shown in FIG. 3B, the first electronic device 301 detects a selection input 371A directed at the first instruction 317-1. For example, in FIG. 3B, the first electronic device 301 detects an air pinch gesture (e.g., the index finger and thumb of the user's hand touching together), a tap or touch input (e.g., provided by the index finger of the hand), a verbal command, or other input, optionally while the user's gaze is directed at the first instruction 317-1. In some examples, in response to detecting the selection of the first instruction 317-1, as described below, the first electronic device 301 initiates a process for pairing the second electronic device 315′ with the first electronic device 301.

[0043] In some examples, initiating a process for pairing the second electronic device 315' with the first electronic device 301 includes transmitting a pairing request to the second electronic device 315' (e.g., directly to the second electronic device 315' or indirectly via a server (e.g., a wireless communication terminal in communication with the second electronic device 315'). In some examples, in response to receiving the pairing request, the second electronic device 315' transmits / outputs data (e.g., directly or indirectly) to the first electronic device 301 to pair the first electronic device 301 with the second electronic device 315'. For example, as shown in FIG. 3B, the second electronic device 315' may transmit a representation of the content displayed on the display 314', so that the first electronic device 301 may visually identify the second electronic device 315' by matching the received representation with a portion of an image captured by the first electronic device 301. In another example, as shown in FIG. 3C, the second electronic device 315′ displays a unique identifier via the display 314′, which is associated with the second electronic device 315′ and is visible within the three-dimensional environment 350 as a representation of a unique identifier 321′ that is detectable by the first electronic device 301. As shown in FIG. 3C, the unique identifier 321′ is optionally a machine-readable image, such as a QR code, barcode, or other machine-readable code, that stores identification data of the second electronic device 301′ that is detectable via one or more image sensors or cameras of the first electronic device 315. In some examples, the unique identifier 321′ is optionally displayed at a frequency that allows the unique identifier 321′ to be detectable by the first electronic device 301 but invisible to a user of the first electronic device 301. In some examples, as shown in FIG. 3C , when the second electronic device 315′ displays the unique identifier 321′, the second electronic device 315′ sends instructions to the first electronic device 301 to prompt a user of the first electronic device 301 to identify the unique identifier 321′.3C , the instruction may cause the first electronic device 301 to display a message element 320 prompting a user of the first electronic device 301 to direct attention (e.g., including gaze) to the unique identifier 321′ displayed on the second electronic device 315′ (e.g., “Scan the code on device 1”) so that the first electronic device 301 may scan / access the identification data contained in the unique identifier 321′. In some examples, once the user's attention is directed to the unique identifier 321′ (and / or the unique identifier 321′ remains within the user's field of view for more than a threshold amount of time, such as 1, 2, 3, 5, 10, 30, 45, 60 seconds), such that the first electronic device 301 scans the unique identifier 321′, the first electronic device 301 transmits confirmation data to the second electronic device 315′ that pairs the first electronic device 315 with the second electronic device 301′.

[0044] Additionally or alternatively, in some examples, the second electronic device 315' outputs a sound detectable by the first electronic device 301 to pair the first electronic device 301 with the second electronic device 315'. For example, as shown in FIG. 3C, the second electronic device 315' outputs a sound 322, which may be output at a frequency detectable by the first electronic device 301 (e.g., may or may not be audible to a user of the first electronic device 301). In some examples, once the sound 322 is detected by the first electronic device 301, the first electronic device 301 transmits confirmation data to the second electronic device 315' that pairs the first electronic device 301 with the second electronic device 315'. Additionally or alternatively, in some examples, the second electronic device 315′ emits infrared light 326, such as via a display 314′, that is detectable by the first electronic device 301 to pair the first electronic device 301 with the second electronic device 315′. In some examples, the infrared light 326 may be invisible to a user of the first electronic device 301. In some examples, once the infrared light 326 is detected by the first electronic device 301 (e.g., via one or more infrared or other image sensors), the first electronic device 301 transmits confirmation data to the second electronic device 315′ that pairs the first electronic device 301 with the second electronic device 315′.

[0045] In some examples, as shown in FIG. 3D, the second electronic device 315′ sends a request to the first electronic device 315 prompting the user of the first electronic device 301 to provide an input detectable by the second electronic device 315′ to pair the first electronic device 301 with the second electronic device 301′. For example, as shown in FIG. 3D, the request causes the first electronic device 301 to display an instruction element 320A and / or an instruction element 320B prompting the user of the first electronic device 301 to provide an input via one or more input devices 331′ / 332′. As an example, in FIG. 3D, the instruction element 320A includes an instruction (e.g., “press a key on the keyboard”) prompting the user of the first electronic device 301 to provide a selection input directed to a keyboard 331′ in communication with the second electronic device 315′. In some examples, the user is prompted to select a particular key on the keyboard 331′. In some examples, as shown in FIG. 3D , the instruction element 320B includes an instruction (e.g., “click the mouse”) that prompts the user of the first electronic device 301 to provide a selection input directed to a mouse 332′ that is in communication with the second electronic device 315′.

[0046] In Figure 3D, the second electronic device 315' detects a selection input 373A directed to the keyboard 331' and / or a selection input 373B directed to the mouse 332'. For example, the second electronic device 315' detects a key press on the keyboard 331' and / or a button press on the mouse 332' (or a tap on the touch-sensitive surface of the mouse 332') provided by a user of the first electronic device 301. In some examples, as shown in Figure 3E, in response to detecting the selection input 373A and / or the selection input 373B, the second electronic device 315' provides a confirmation that the selection input 373A and / or the selection input 373B was detected. For example, as shown in FIG. 3E, the second electronic device 315′ displays a confirmation message via display 314′, which can be seen in the three-dimensional environment 350 as confirmation message 323′, indicating that the above-mentioned input has been successfully detected (e.g., “input detected”), thereby enabling the first electronic device 301 to be paired with the second electronic device 315′.

[0047] In some examples, the first electronic device 301 initiates a process for pairing the second electronic device 315' with the first electronic device 301 according to the above examples, following a determination that one or more criteria are met. In some examples, the one or more criteria include a criterion that is met following a determination that the second electronic device 315' is in a state for displaying a user interface via a virtual augmented display. In some examples, the one or more criteria include a criterion that is met following a determination that the second electronic device 315' is detected by one or more sensors of the first electronic device 301 (e.g., when the second electronic device is captured by one or more image sensors or cameras of the first electronic device). In some examples, the one or more criteria include a criterion that is met following a determination that the second electronic device 315' is configured to be able to communicate with the first electronic device 301 (e.g., when a wired or wireless communication channel is established). In some examples, the one or more criteria include criteria that are met pursuant to a determination that the second electronic device 315' is an eligible electronic device (e.g., some electronic devices may have the capability to operate in the context of an augmented virtual display, while others may not). In some examples, the one or more criteria include criteria that are met pursuant to a determination that the first and second electronic devices are permitted to share information (e.g., privacy settings, user information, user data, etc.), such as when the first electronic device 301 and the second electronic device 315' belong to the same user and / or are associated with the same user account logged into the electronic devices. In some examples, some or all of the above criteria must be met in order for the first electronic device 301 to be paired with the second electronic device 315'.

[0048] In some examples, the first electronic device 301 may be paired with the second electronic device 315' without following the exemplary process described above. For example, in FIG. 3B, in response to detecting the selection of the first instruction 317-1, the first electronic device 301 transmits a request to pair with the second electronic device 315', and if one or more criteria described above are met, the first electronic device 301 automatically pairs with the second electronic device 315'. In some examples, the first electronic device 301 and the second electronic device 315' may be paired via any suitable communication means, such as Bluetooth, Wi-Fi, etc.

[0049] 3F, when the first electronic device 301 is paired with the second electronic device 315′ as described above, the first electronic device 301 displays a message element 320 (e.g., a notification) that alerts a user of the first electronic device 301 that the first electronic device 301 is paired with the second electronic device 315′ (e.g., “Device 1 Paired”). In some examples, after the first electronic device 301 is paired with the second electronic device 315′, the first electronic device 301 initiates a process for displaying a virtual augmented display associated with the second electronic device 315′. 3G, while the first electronic device 301 is in communication with the second electronic device 315' and the second electronic device 315' is within a field of view of a user of the first electronic device 301, the first electronic device 301 displays a first affordance 319 within the three-dimensional environment 350, which is selectable to display a virtual augmented display of the second electronic device 315'. In some examples, as shown in FIG. 3G, the first affordance 319 may be presented in proximity to an edge of the display of the second electronic device 315' (e.g., within a threshold distance (e.g., 1, 2, 3, 5, 8, 10, 15, 20, 30 cm, etc.) of the edge of the display). 3G, the first affordance 319 is presented above the second electronic device 315' in the three-dimensional environment 350 (e.g., within a threshold distance of a corner of the display 314' of the second electronic device 315'). In some examples, the first affordance 319 may be presented at a depth within the three-dimensional environment 350 such that it appears in the same plane as the display 314' of the second electronic device 315'.

[0050] In some examples, the first affordance 319 may be positioned differently within the three-dimensional environment 350. For example, the first affordance 319 may be presented above, near, below, or in front of the second electronic device 315′, and / or at other locations within the three-dimensional environment 350 (e.g., at a predetermined distance away from the second electronic device). Additionally or alternatively, in some examples, the first affordance may be displayed in a predetermined area of ​​the display of the first electronic device 301 (e.g., regardless of the location of the second electronic device 315′ in the three-dimensional environment 350). For example, as shown in FIG. 3G, the first affordance may be displayed as a first system option 318A within the plurality of system options 318. In some examples, as shown in FIG. 3G, the plurality of system options 318 are displayed centrally along a top edge of the display of the first electronic device 301. In some examples, multiple system options 318 other than the first system option 318A are selectable to access one or more settings of the first electronic device 301, such as display settings (e.g., brightness levels and / or immersive content levels) and / or audio settings (e.g., volume levels), and / or to access one or more applications of the first electronic device 301, such as via a home user interface.

[0051] In some examples, the second electronic device 315' may present, via the display 314', a selectable affordance (e.g., using the keyboard 331', mouse 332', input of the first electronic device 301, etc.) to initiate the display of a virtual augmented display by the second electronic device 315. For example, the second electronic device 315' may provide an affordance different from, similar to, or identical to the first affordance 319 within a selectable setting or other application to cause the first electronic device 301 to send a request to the second electronic device 315' to display a virtual augmented display.

[0052] In some examples, as shown in FIG. 3G, the physical environment surrounding the first electronic device 301 may include a third electronic device 325 disposed on a coffee table 308. For example, a representation of the third electronic device 325' and a representation of the coffee table 308' are seen in a three-dimensional environment 350, as shown in FIG. 3G. In some examples, the third electronic device 325 may be a mobile electronic device, such as a cellular electronic device or a tablet computer. As shown in FIG. 3G, the third electronic device 325 is optionally not paired with the first electronic device 301. Alternatively, in FIG. 3G, the third electronic device 325 is optionally paired with the first electronic device 301, but may not have the ability to operate in the context of an augmented virtual display. Thus, as shown in FIG. 3G, the representation of the third electronic device 325' is not displayed with a selectable affordance (e.g., in proximity to the representation of the third electronic device 325' or among multiple system options 318) to display an extended virtual display of the third electronic device 325.

[0053] In FIG. 3H, the first electronic device 301 detects a selection input 371B directed to the first system option 318A described above. For example, the first electronic device 301 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell, etc., corresponding to a selection of the first system option 318A. In some examples, as shown in FIG. 3I, in response to detecting a selection of the first system option 318A, the first affordance 319, or other affordance for initiating a virtual augmented display, the first electronic device 301 presents a virtual augmented display 304 associated with the second electronic device 315′ within the three-dimensional environment 350 (e.g., not associated with the third electronic device 325′ described above).

[0054] In some examples, as shown in FIG. 3I, the virtual extended display 304 functions as a second (e.g., virtual) display for the second electronic device 315′. For example, as shown in FIG. 3I, the virtual extended display includes a representation of a dock 309″ displayed on a display 314′ of the second electronic device 315′. In some examples, as described herein, the virtual extended display 304 is configured to display one or more user interfaces (e.g., application windows), content, and / or other user interface elements of the second electronic device 315′ that may or may not be displayed on the display 314′. Additionally, in some examples, the cursor 312′ is controllable via a mouse 332′ such that the cursor 312′ is moveable between the display 314′ of the second electronic device 315′ and the virtual extended display 304. In some examples, as described later herein, user input detected via keyboard 331′ and / or mouse 332′ may be directed to a user interface of display 314′ or a user interface of virtual augmented display 304 depending on the location of the gaze of the user of first electronic device 301. Attention is now directed to exemplary interactions with a virtual augmented display for an electronic device in a three-dimensional environment.

[0055] In some examples, when the first electronic device 301 presents the virtual augmented display 304 in the three-dimensional environment 350, the display 314' of the second electronic device 315' may be turned off (e.g., the second electronic device 315' enters a low power mode or sleep state or is powered off such that the second electronic device 315' no longer displays a user interface via the display 314'). For example, after the first electronic device 301 is paired with the second electronic device 315' and after the first electronic device 301 generates and displays the virtual augmented display 304 in the three-dimensional environment 350, the second electronic device 315' (e.g., automatically) powers off the display 314'. In some examples, the display 314' of the second electronic device 315' may be turned off when the first electronic device 301 initiates the above-described process for pairing the first electronic device 301 and the second electronic device 315'. In such a scenario, the display 314' of the second electronic device 315' may be powered on for the pairing process (eg, to display the unique identifier 321' and / or the confirmation message 323').

[0056] 4A-4O show examples of a first electronic device facilitating interaction with a representation of a user interface of a second electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, the first electronic device 401 may be used to present the three-dimensional environment 450, as shown in FIG. 4A. In some examples, the first electronic device 401 optionally corresponds to the first electronic device 301 described above. In some examples, the three-dimensional environment 450 includes a captured portion of a physical environment in which the first electronic device 401 is located. For example, the three-dimensional environment 450 optionally includes a table (e.g., a representation of a table 406′), as shown in FIG. 4A. In addition, the three-dimensional environment 450 includes a second electronic device (e.g., a representation of a second electronic device 415′) and one or more physical input devices (e.g., one or more representations of one or more input devices 431′ / 432′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, the three dimensional environment 450 optionally corresponds to the three dimensional environment 350 described above. In some examples, the second electronic device 415' corresponds to the second electronic device 315' described above. In some examples, the one or more input devices 431' / 432' correspond to the one or more input devices 331' / 332' described above. In some examples, the representation of the physical environment may include a portion of the physical environment that is viewed through a transparent or semi-transparent display of the first electronic device 401.

[0057] As also described above, in some examples, the first electronic device 401 communicates with the second electronic device 415'. In addition, in some examples, as shown in FIG. 4A, the first electronic device 401 presents a virtual extended display 404 associated with the second electronic device 415'. For example, FIG. 4A. The virtual extended display 404 serves as a second (e.g., virtual) display for the physical display (e.g., a representation of the display 414') of the second electronic device 415'. In some examples, the virtual extended display 404 corresponds to the virtual extended display 304 described above. In some examples, as shown in FIG. 4A, the display 414' of the second electronic device 415' displays a first user interface (e.g., a representation of the first user interface 411'), and the virtual extended display 404 displays a second user interface 425 configured to be displayed on the display 414' of the second electronic device 415'. In some examples, the first user interface 411' and the second user interface 425 may be associated with separate applications (e.g., a web browsing application ("browser") running on the second electronic device 514'). For example, the first user interface 411' is a first window (e.g., "Window A") of the separate application, and the second user interface 425 is a second window (e.g., "Window B") of the separate application. Additionally, as shown in FIG. 4A, the display 414' of the second electronic device 415' displays a dock (e.g., a representation of the dock 409') and the virtual extended display 404 displays a representation of the dock 409''. In some examples, the dock 409' corresponds to the dock 309' described above. In some examples, as shown in FIG. 4A, the display 414' of the second electronic device 415' includes a cursor (e.g., a representation of the cursor 412') that is controllable using the mouse 432'. In some examples, cursor 412' corresponds to cursor 312' described above.

[0058] In some examples, as shown in FIG. 4A, the first user interface 411′ includes a selectable text entry field (e.g., a representation of the text entry field 413′) for entering text to provide a search query to a web browser. Additionally, as shown in FIG. 4A, the second user interface 425 includes a plurality of search results (e.g., associated with the URL “www.Search.com”) provided by the web browser. For example, in FIG. 4A, the second user interface 425 includes a first search result 426A (e.g., “Search Result A”) associated with a first website (e.g., “www.URL1.com”), a second search result 426B (e.g., “Search Result B”) associated with a second website (e.g., “www.URL2.com”), and a third search result 426C (e.g., “Search Result C”) associated with a third website. In some examples, the plurality of search results 426 are selectable for accessing content (e.g., text, images, videos, music, etc.) via the respective sources (e.g., websites).

[0059] In some examples, the first electronic device 401 displays the second user interface 425 in the virtual augmented display 404 based on image data (e.g., pixel or display data) transmitted to the first electronic device 401 by the second electronic device 415′ (e.g., directly or indirectly via a server). For example, as described above, the second user interface 425 is configured to be displayed on the display 414′ of the second electronic device 415′ (e.g., the second user interface 425 is a user interface of the second electronic device 415′). Thus, in some examples, the first electronic device 401 renders (e.g., generates) the second user interface 425 including the content of the second user interface 425 described above based on image data provided by the second electronic device 415′. For example, the image data includes information corresponding to the appearance of the second user interface 425, characteristics of the content of the second user interface 425 (e.g., whether the content is interactive, selectable, scrollable, etc.), and / or the quality of the second user interface 425 and its content (e.g., image quality, such as definition). In some examples, the image data transmitted by the second electronic device 401′ to the first electronic device 415 includes an image generated by the second electronic device 401′ that is displayed by the first electronic device 415 on the virtual augmented display 404.

[0060] In some examples, user input detected via one or more input devices 431' / 432' may be selectively routed by the first electronic device 401 based on the location of the gaze of the user of the first electronic device 401 in the three-dimensional environment 450. As used herein, the user of the first electronic device 401 may be the user of the second electronic device 415'. In FIG. 4B, while the user's gaze 427 is directed at the first user interface 411' of the display 414', the second electronic device 415' detects input 473A via a mouse 432' in communication with the second electronic device 415'. For example, as shown in FIG. 4B, the second electronic device 415' detects the movement of the mouse 432' on the surface of the table 406'.

[0061] In some examples, as described above, the first electronic device 401 controls the applicability of the user input based on the location of the line of sight 427. For example, in response to detecting the input 473A, the second electronic device 415' transmits (e.g., directly or indirectly via a server) input data to the first electronic device 401, including information corresponding to a movement of the mouse 432' (e.g., corresponding to a request to move the cursor 412'). In some examples, when the first electronic device 401 receives the input data provided by the second electronic device 415', the first electronic device 401 determines the location of the user's line of sight 427 in the three-dimensional environment 450. As described above and shown in FIG. 4B, when the input 473A is provided via the mouse 432', the line of sight 427 is directed to the first user interface 411' displayed on the display 414'. Thus, the first electronic device 401 routes the input 473A detected via the mouse 432' to the second electronic device 415'. For example, in Figure 4B, the first electronic device 401 sends output data to the second electronic device 415' that includes one or more commands / instructions for moving a cursor 412' within a display 414' of the second electronic device 415' in accordance with the input 473A. Thus, in Figure 4C, the second electronic device 415' moves the cursor 412' within the display 414' to a new location in the first user interface 411' in accordance with the movement of the mouse 432'.

[0062] In FIG. 4C, the second electronic device 415′ detects a selection input 473B via the mouse 432′ while the user's gaze 427 is directed toward a text input field 413′ in the first user interface 411′. For example, as shown in FIG. 4C, the second electronic device 415′ detects a click or press of a button of the mouse 432′ (or a tap of the touch-sensitive surface of the mouse 432′) while the cursor 412′ is positioned over the text input field 413′. In some examples, as also described above, in response to detecting the selection input 473B, the second electronic device 415′ transmits input data to the first electronic device 401 such that the first electronic device 401 may determine where to route the selection input 473B. As also described above, the first electronic device 401 determines where to direct the selection input 473B based on the location of the gaze 427 in the three-dimensional environment. In FIG. 4C , because the gaze 427 is directed to the text input field 413′ in the first user interface 411′ (or, in some examples, because the gaze 427 is directed to the first user interface 411′, the display 414′, etc.), the first electronic device 401 causes the second electronic device 415′ to direct a selection input 473B to the first user interface 411′, in particular to the location of the cursor 412′ controllable via the mouse 432′, as described above. Thus, as shown in FIG. 4D , the second electronic device 415′ performs a selection at the location of the cursor 412′, including selecting the text input field 413′ in the first user interface 411′. As shown in FIG. 4D , in some examples, once the text input field 413′ is selected, the second electronic device 415′ displays a text cursor (e.g., a representation of the text cursor 428′) in the text input field 413′ indicating that text may be entered into the text input field 413′.

[0063] In some examples, if the user's gaze, alternatively represented in FIG. 4C as gaze 427i, is alternatively directed neither to the first user interface 411′ (or, in some examples, the display 414′) nor to the second user interface 425 (or, in some examples, the virtual augmented display 404), the first electronic device 401 and the second electronic device 415′ optionally do not perform an action in response to the selection input 473B. For example, when the first electronic device 401 receives input data provided by the second electronic device 415′ corresponding to a selection detected on the mouse 432′, the first electronic device 401 determines that the user's gaze 427i is directed away from the first user interface 411′ and the second user interface 425 and transmits an instruction to the second electronic device 415′ to refrain from performing any action in response to the selection. Alternatively, in some examples, as shown in Figure 4D, if the user's gaze 427i is not directed at either the first user interface 411' or the second user interface 425 when the selection input 473B of Figure 4C is detected by the second electronic device 401', the first electronic device 415 implements an input latching operation that causes the input to be routed to a currently active user interface in the three-dimensional environment 450. For example, the input is automatically routed to the last user interface that the gaze 427 was directed to before the selection input 473B was detected while the gaze 427i is directed away from the first user interface 411' and the second user interface 425. According to this example, because the gaze 427 was last directed at the first user interface 411' in FIG. 4B before the selection input 473B was detected in FIG. 4C, the input is latched to the first user interface 411', causing the first electronic device 401 to route the selection input 473B to the first user interface 411' (e.g., selecting the text input field 413') as shown in FIG. 4D and similarly described above.

[0064] In FIG. 4D , while the text cursor 428′ is displayed in the text entry field 413′ and while the user's gaze 427 is directed toward the first user interface 411′, the second electronic device 415′ detects a sequence of one or more inputs 473C via a keyboard 431′ in communication with the second electronic device 415′. For example, in FIG. 4D , the second electronic device 415′ detects a selection of one or more keys on the keyboard 431′. In some examples, also as described above, in response to detecting the sequence of one or more inputs 473C, the second electronic device 415′ transmits input data to the first electronic device 401, the input data including information corresponding to the selection of one or more keys on the keyboard 431′. In some examples, also as described above, when the first electronic device 401 receives the input data provided by the second electronic device 415′, the first electronic device 401 determines where to route the keyboard input based on the location of the gaze 427 within the three-dimensional environment 450. In Figure 4D, because the line of sight 427 is directed to the first user interface 411' when a series of one or more inputs 473C is detected by the second electronic device 415', the first electronic device 401 causes the second electronic device 415' to direct (e.g., via output data including one or more commands / instructions) a series of one or more inputs 473C to the first user interface 411', and in particular to the text input field 413'. Thus, as shown in Figure 4E, because the text cursor 428' is active within the text input field 413' and the series of one or more inputs 473C is directed to the first user interface 411', the second electronic device 415' displays text (e.g., a representation of text 429' spelling out "apartmen") at the location of the text cursor 428' in the text input field 413' according to a selection of one or more keys of the keyboard 431'.

[0065] In some examples, the cursor 412' can be configured to be movable between the physical display 414' of the second electronic device 415' and the virtual augmented display 404 in the three-dimensional environment 450. In some examples, the cursor 412' may be moved between the physical display 414' of the second electronic device 415' and the virtual augmented display 404 based on the movement of a mouse 432' controlling the cursor 412'. For example, in FIG. 4F, the second electronic device 415' detects a movement input 473D corresponding to the movement of the mouse 432' (e.g., a movement rightward across the top of the table 406'). Additionally, in some examples, the cursor 412' may be moved between the physical display 414' of the second electronic device 415' and the virtual augmented display 404 based on the location of the user's line of sight 427. For example, in FIG. 4F, the line of sight 427 is directed toward the second user interface 425 while the movement of the mouse 432' is detected by the second electronic device 415'.

[0066] In some examples, the cursor 412' is moved between the display 414' of the second electronic device 415' and the virtual extended display 404 in accordance with a determination that the movement of the mouse 432' has exceeded a movement threshold. For example, the movement of the mouse 432' exceeds the movement threshold if a velocity or acceleration of the movement of the mouse 432' exceeds a velocity or acceleration threshold. In some examples, the movement of the mouse 432' exceeds the movement threshold if the movement of the mouse 432' corresponds to a movement of the cursor 412', optionally in the direction of the virtual extended display 404, that exceeds a threshold distance (e.g., 0.5, 1, 1.5, 2, 3, 5, 10, 15, 25 cm, etc.) of a boundary of the display. For example, in FIG. 4F, the movement of the mouse 432' detected by the second electronic device 415' corresponds to a movement of the cursor 412' that is less than a threshold distance of a boundary of the display 414' of the second electronic device 415'. Thus, as shown in FIG. 4G, the display of the cursor 412′ is maintained within the display 414′ of the second electronic device 415′ (e.g., at or near the right boundary of the display 414′). For example, the second electronic device 415′ may transmit input data to the first electronic device 401 including information corresponding to movement of the cursor 412′ less than a threshold distance of the boundary of the display 414′, which may cause the first electronic device 401 to transmit output data to the second electronic device 415′, which may cause the second electronic device 415′ to maintain the display of the cursor 412′ on the display 414′ (e.g., despite moving the cursor 412′ rightward in accordance with the movement of the mouse 432′). In other examples, the second electronic device 415′ may cease transmitting input data to the first electronic device 401 when it is determined that the movement of the cursor 412′ does not exceed the movement threshold, and may transmit the input data only when the movement of the cursor 412′ exceeds the movement threshold.

[0067] In FIG. 4G, the second electronic device 415' detects further movement of the mouse 432' while the cursor 412' is displayed on the display 414' of the second electronic device 415'. For example, as shown in FIG. 4G, the second electronic device 415' optionally detects a movement input 473E corresponding to a movement of the cursor 412' to the right on the display 414' beyond the threshold distance of the boundary of the display 414' mentioned above while the user's gaze 427 is still directed at the second user interface 425 of the virtual augmented display 404. Alternatively, in FIG. 4G, the second electronic device 415' detects a selection input via the mouse 432' while the gaze 427 is directed at the second user interface 425. For example, the second electronic device 415' detects a click or press of a button on the mouse 432' (e.g., or a tap on the touch-sensitive surface of the mouse 432').

[0068] 4H , in response to detecting movement of the mouse 432′ beyond a threshold distance of a boundary of the display 414′, optionally corresponding to movement of the cursor 412′ toward the virtual augmented display 404, the first electronic device 401 displays the cursor 412′ within the virtual augmented display 404 in the three-dimensional environment 450. For example, the first electronic device 401 causes the second electronic device 415′ to stop displaying the cursor 412′ on the display 414′ and re-display the cursor 412′ within the virtual augmented display 404 at a location based on the movement of the mouse 432′ (e.g., the first electronic device 401 displays the cursor 412′ within the second user interface 425 based on input data provided by the second electronic device 415′ including information corresponding to the movement of the mouse 432′). In some examples, the first electronic device 301 displays the cursor 412' within the virtual extended display 404 based on image data received from the second electronic device 415' (e.g., as described above) that indicates the cursor 412' at a location within the virtual extended display 404 based on movement of the mouse 432' determined by the second electronic device 415'.

[0069] 4H, in response to the second electronic device 415′ detecting a selection input via the mouse 432′ while the user's gaze 427 is directed toward the second user interface 425, the first electronic device 401 moves the cursor 412′ to the second user interface 425. In another example, in response to the second electronic device 415′ detecting a movement input 473D of FIG. 4F while the gaze 427 is directed toward a different interface (e.g., the second user interface 425) or display (e.g., the virtual augmented display 404) than the one currently targeted by the cursor 412′, or while the gaze 427 is directed toward a location (in the same or a different interface, display, etc.) that is a threshold distance from the cursor 412′, the first electronic device 401 may move the cursor 412′ to a location based on the gaze 427. For example, as also described above, the first electronic device 401 causes the second electronic device 415' to stop displaying the cursor 412' on the display 414' and re-display the cursor 412' in the virtual extended display 404 at a location based on the location of the line of sight 427. As shown in FIG. 4H, the first electronic device 401 optionally displays the cursor 412' in the second user interface 425 at the location of the line of sight 427 (e.g., over the first search result 426A). In some examples, as also described above, the first electronic device 401 renders and displays the cursor 412' in the virtual extended display 404 based on image data corresponding to the cursor 412' provided by the second electronic device 415'.

[0070] In some examples, input detected via keyboard 431' and / or mouse 432' in communication with the second electronic device 401' may be routed by the first electronic device 401' to interact with the virtual augmented display 404 displayed in the three-dimensional environment 450 at the first electronic device 415. In some examples, the input is routed based on the location of the user's gaze 427 in the three-dimensional environment 450, as also described above. In FIG. 4H, the second electronic device 401' detects a selection input 473F via mouse 432' while cursor 412' is displayed in the second user interface 425 of the first electronic device 415. For example, in FIG. 4H, the second electronic device 415' detects a click or press of a button on mouse 432' (or a tap on the touch-sensitive surface of mouse 432').

[0071] In some examples, as shown in FIG. 4I, in response to detecting the selection input 473F on the mouse 432′, the first electronic device 401 performs a selection operation based on the location of the cursor 412′ in the second user interface 425. For example, as shown in FIG. 4I, because the cursor 412′ is located on the first search result 426A in the second user interface 425 when the selection input 473F is detected and the line of sight 427 is directed toward the second user interface 425, the first electronic device 401 selects the first search result 426A, which includes displaying content associated with the first search result 426A (e.g., “Website 1 Content” accessible from the website “www.URL1.com”). Specifically, as described above in this specification, the second electronic device 415′ transmits input data including information corresponding to the selection input 473F to the first electronic device 401, and in response to detecting the input data, determines the location of the line of sight 427 in the three-dimensional environment 450. In FIG. 4I, because the gaze 427 was directed to the second user interface 425 when the selection input 473F was detected in FIG. 4H, the first electronic device 401 directs the selection input to the virtual extended display 404, i.e., performs the selection at the location of the cursor 412′ in the second user interface 425. In other examples in which the second electronic device 415 generates the content for the virtual extended display 404, the second electronic device 415 may forgo sending input data to the first electronic device and instead perform the selection at the location of the cursor 412′ in the second user interface 425. The first electronic device 401 may display any resulting updates to the content of the virtual extended display 404 based on image data (e.g., as described above) received from the second electronic device 415′ that indicate changes to the content of the virtual extended display 404 based on what was determined by the second electronic device 415′ based on the selection input 473F.

[0072] In FIG. 4I, the second electronic device 415' detects a movement input 473G via the mouse 432'. For example, in FIG. 4I, the second electronic device 415' detects a movement of the mouse 432' rightward across the surface of the table 406' while the cursor 412' is displayed in the second user interface 425 of the first electronic device 401. In some examples, as shown in FIG. 4J, in response to detecting the movement of the mouse 432' at the second electronic device 415', the first electronic device 415 moves the cursor 412' in the second user interface 425 according to the movement of the mouse 432' detected at the second electronic device 401'. For example, in response to detecting the movement input 473G, the second electronic device 415' transmits input data to the first electronic device 401, the input data including information corresponding to the movement of the mouse 432' (e.g., the direction of the movement and / or the distance or speed of the movement). In some examples, as also discussed herein, when the first electronic device 401 receives input data provided by the second electronic device 415′, the first electronic device 401 determines the location of the user's gaze 427 when the movement input 473G is detected at the second electronic device 415′. In FIG. 4J, since the gaze 427 was directed to the second user interface 425 in the virtual extended display 404, the first electronic device 401 moves the cursor 412′ in the second user interface 425 according to the input data. In some examples, the first electronic device 301 displays the movement of the cursor 412′ in the second user interface 425 in the virtual extended display 404 (e.g., changes the position at which the cursor 412′ is displayed) based on image data received from the second electronic device 415′ (e.g., described above) showing the cursor 412′ with an updated position in the virtual extended display 404 based on the movement of the mouse 432′ determined by the second electronic device 415′.

[0073] 4J, the second electronic device 415' detects a selection input 473H via the mouse 432' while the first electronic device 401 displays the cursor 412' in the second user interface 425. For example, as also described above, the second electronic device 415' detects a click or press of a button on the mouse 432' (or a tap on a touch-sensitive surface of the mouse 432'). In some examples, as shown in FIG. 4K, in response to the second electronic device 415' detecting the selection input 473H, the first electronic device 401 performs a selection operation in the second user interface 425 based on the location of the cursor 412'. For example, similar to that described above, because the user's gaze 427 is directed to the second user interface 425 when the second electronic device 415′ detects the selection input 473H of FIG. 4J, the first electronic device 401 directs the selection input to the second user interface 425 in accordance with the detected selection input 473H, and in particular performs a selection of the text input field 416 of the second user interface 425. In some examples, also as described above, the second electronic device 415′ may instead perform a selection of the text input field 416 of the second user interface 425 and provide updated image data for the virtual augmented display 404 to the first electronic device 401. As shown in FIG. 4K, in some examples, selecting the text input field 416 includes displaying a text cursor 428 in the text input field 416 indicating that text may be entered into the text input field in response to keyboard input.

[0074] In FIG. 4K, the second electronic device 415′ detects a series of one or more selection inputs 473I via the keyboard 431′ while the first electronic device 401 displays the text cursor 428 in the text input field 416. For example, in FIG. 4K, the second electronic device 415′ detects one or more selections or presses of one or more keys of the keyboard 431′. In some examples, as shown in FIG. 4L, in response to the second electronic device 415′ detecting the series of one or more selection inputs 473I, the first electronic device 401 enters text 449 into the text input field 416 according to the series of one or more selection inputs 473I detected via the keyboard 431′. For example, in FIG. 4L, the first electronic device 401 displays text 449 spelling out “restaura” at the location of the text cursor 428 in the text input field 416 according to the selected keys of the keyboard 431′. In particular, similar to that described herein above, because the gaze 427 was directed to the second user interface 425 when a series of one or more selection inputs 473I was detected by the second electronic device 415′, the first electronic device 401 displays text 449 according to input data provided by the second electronic device 415′, including information corresponding to a selected key of the keyboard 431′. In some examples, also as described above, the second electronic device 415′ may instead input text 449 into the text input field 416 according to a series of one or more selection inputs 473I detected via the keyboard 431′. In these examples, the first electronic device 415′ may display text 449 spelling out “restaura” in the text input field 416 based on image data indicative of “restaura” received from the second electronic device 415′.

[0075] In Fig. 4L, the second electronic device 415' detects a further selection input 473J via the keyboard 431' while the first electronic device 401 displays text 449 in the text input field 416 of the second user interface 425. For example, in Fig. 4L, the second electronic device 415' detects a selection of one or more additional keys on the keyboard 431'. As shown in Fig. 4L, the second electronic device 415' detects a selection input 473J via the keyboard 431' while the user's gaze 427 is directed at the first user interface 411' on the display 414' of the second electronic device 415'.

[0076] In some examples, as shown in FIG. 4M, in response to the second electronic device 415′ detecting the selection input 473J via the keyboard 431′, the first electronic device 401 routes the selection input 473J to the first user interface 411′ of the second electronic device 415′ based on the location of the gaze 427. For example, similar to that described herein above, the first electronic device 401 determines that the gaze 427 is directed toward the display 414′ when the selection input 473J is detected on the keyboard 431′ of FIG. 4L and transmits output data to the second electronic device 415′ that causes the second electronic device 415′ to update the text 429′ in the text input field 413′ according to the selected key of the keyboard 431′. As shown in FIG. 4M, the text 429′ in the first user interface 411′ is updated to include the text “apartments fo” at the location of the text cursor 428′ in the text input field 413′. Further, as shown in FIG. 4M, the first electronic device 401 ceases updating the text 449 in the text input field 416 in the second user interface 425 according to the selected key of the keyboard 431′.

[0077] Additionally, in some examples, as shown in FIG. 4M, when the second electronic device 415′ updates the text 429′ in the first user interface 411′ according to the selection input 473J, the cursor 412′ is redisplayed in the first user interface 411′, optionally based on the location of the line of sight 427 in the three-dimensional environment 450. For example, in FIG. 4M, the first electronic device 401 stops displaying the cursor 412′ in the second user interface 411 and sends data (e.g., output data) to the second electronic device 415′ to cause the second electronic device 415′ to redisplay the cursor 412′ at a location in the first user interface 425′ that corresponds to the location of the line of sight 427 in the three-dimensional environment 450 (e.g., on the text input field 413′). In some examples, as described above in this specification, while the cursor 412' is located (e.g., displayed) within the first user interface 411' on the display 414' and the line of sight 427 is directed toward the display 414' of the second electronic device 415', the cursor 412' is controllable via the mouse 432' to interact with user interface elements of the display 414' (e.g., the first user interface 411' and / or the dock 409') in one or more of the manners described above.

[0078] In some examples, disassociation of the first electronic device 401 from the user of the first electronic device 401 causes the display 414' of the second electronic device 415' to be reconfigured as the sole display for the first user interface 411' and the second user interface 425. For example, in FIG. 4M, the first electronic device 401 detects removal of the first electronic device 401 from the user's head (e.g., via the orientation sensor 210 of FIG. 2), as represented by arrow 475A. In some examples, disassociation of the first electronic device 401 from the user causes the physical environment surrounding the user to no longer be visible through the display of the first electronic device 401 (e.g., such that the display of the first electronic device 401 is no longer positioned on or in front of the user's eyes). In some examples, disassociation of the first electronic device 401 corresponds to the first electronic device 401 being powered off or entering a sleep state, which optionally causes the display of the first electronic device 401 to stop or pause operation.

[0079] In some examples, as shown in FIG. 4N, in response to detecting disassociation of the first electronic device 401 from the user of the first electronic device 401, the first electronic device 401 stops presenting the three-dimensional environment 450. For example, as shown in FIG. 4N, all virtual objects stop being displayed by the first electronic device 401, i.e., the virtual augmented display 404. Additionally, as shown in FIG. 4N, in some examples, when the first electronic device 401 is no longer being used by the user, the display 414 of the second electronic device 415 in the real-world environment 400 is reconfigured as the only available display for all windows, applications, files, and other user interface elements previously displayed across the display 414 and the virtual augmented display 404. For example, as shown in FIG. 4N, the first user interface 411 and the second user interface 425 are simultaneously displayed on the display 414 of the second electronic device 415 since the virtual augmented display 404 is no longer available as a display means for the second user interface 425 as outlined above. In some examples, in response to detecting disassociation of the first electronic device 401 from a user of the first electronic device 401, interfaces associated with applications executing on the first electronic device 401 (and not on the second electronic device 415) may not be displayed on the display 414. In other examples, interfaces associated with applications executing on the first electronic device 401 (and not on the second electronic device 415) may be displayed on the display 414 in response to detecting disassociation of the first electronic device 401 from a user of the first electronic device 401 (e.g., based on the display capabilities of the second electronic device 415′).

[0080] In particular, in some examples, when the first electronic device 401 is disassociated from the user, the first electronic device 401 transmits context data to the second electronic device 415, the context data including information corresponding to a display state of the second user interface 425. For example, in FIG. 4N, the information corresponding to a display state of the second user interface 425 includes details regarding the appearance of the second user interface 425, content included in the second user interface 425, such as the text "restaura" as shown, and / or a display mode of the second user interface 425 (e.g., whether the second user interface 425 is displayed in an extended full-screen mode on the virtual extended display 404). 4N, when the second user interface 425 is displayed on the display 414 of the second electronic device 415 after the first electronic device 401 is no longer in use, the second user interface 425 is displayed according to the context data provided by the first electronic device 401 (e.g., such that the text "restaura" is still displayed in the second user interface 425 when the display 414 is reconfigured as the only display). In other examples in which the second electronic device 415 generates the content of the virtual augmented display 404 and manages the context data including information corresponding to the state of the display of the second user interface 425, the first electronic device 401 may cease sending the context data to the second electronic device 415 and may instead send an indication that the first electronic device 401 has been disassociated from the user.

[0081] In some examples, when the first electronic device 401 is reassociated with the user such that the display of the first electronic device 401 is operational (e.g., powered on or woken up from a sleep state), the first electronic device 401 redisplays the virtual augmented display 404 based on a previous display of the virtual augmented display 404 on the first electronic device 401. In FIG. 4N , the user moves the first electronic device 401 back onto the user's head, as represented by arrow 475B. For example, the user positions the display of the first electronic device 401 above and / or in front of the user's eyes so that the real-world environment 400 is once again visible through the display of the first electronic device 401.

[0082] In some examples, as shown in FIG 4O, when the first electronic device 401 is reassociated with the user, the first electronic device 401 redisplays the three-dimensional environment 450. In addition, as shown in FIG 4O, the first electronic device 401 redisplays the virtual augmented display 404 including the second user interface 425 because the virtual augmented display 404 is restored as a second (e.g., virtual) display for the second electronic device 415'. For example, as shown in FIG 4O, the second electronic device 415' stops displaying the second user interface 425, and the first electronic device displays the second user interface 425 within the virtual augmented display 404 according to the display state of the second user interface 425 before the first electronic device 401 was disassociated from the user in FIG 4M.

[0083] Additionally, in some examples, as shown in FIG. 4O, when the second user interface 425 re-displays on the virtual augmented display 404 at the first electronic device 401, the second user interface 425 includes any updates to the presentation of the second user interface 425, including updates to the content of the second user interface 425, since the second user interface 425 was last displayed at the first electronic device 401. For example, in FIG. 4O, the text "restaurants" has been updated in the second user interface 425 since the second user interface 425 was last displayed at the first electronic device 401 in FIG. 4M (e.g., in response to detecting a selection input via keyboard 431'). Thus, as shown in FIG. 4O, when the second user interface 425 re-displays on the virtual augmented display 404 at the first electronic device 401, the second user interface 425 includes the updated text described above.

[0084] In some examples, the virtual augmented display 404 is reconfigured as the second display of the second electronic device 401' when the first electronic device 415 is reassociated with the user, if the reassociation occurs within a threshold amount of time of disassociation. For example, the virtual augmented display 404 remains configured to display the content (e.g., user interface) of the second electronic device 415' if the user initially removes the first electronic device 401 from his / her head (e.g., and / or away from the eye) and then places the first electronic device 401 on his / her head (e.g., on or in front of the eye) within 1, 2, 5, 10, 15, 20, 30, 60, 80, 100, 120, etc. minutes after removing the first electronic device 401. In some examples, if reassociation does not occur within a threshold amount of time of disassociation of the first electronic device 401 from the user, the first electronic device 401 ceases to redisplay the second user interface 425 in the virtual augmented display 404. In some examples, the first electronic device 401 also ceases to redisplay the virtual extended display 404. In some such examples, the virtual extended display 404 may be redisplayed by reactivating the virtual extended display 404, such as by using an affordance similar to the first affordance 319 of FIG.

[0085] It should be noted that in some examples, an input initially detected by the second electronic device 415 may not be transmitted to the first electronic device 401 pursuant to a determination that the input should be directed to the second electronic device 415 (e.g., based on the line of sight 427 being directed to the display 414′, the interface of the display 414′, the virtual extended display 404, or the interface of the virtual extended display 404). In these examples, the first electronic device 401 may transmit information indicating the current destination of the user input to the second electronic device 415′ intermittently, periodically, or at any other desired interval. If the current destination is the second electronic device 415′, the second electronic device 415′ may cease transmitting the detected input to the first electronic device 401 and may instead process the input locally. If the current destination is the first electronic device 401 (or any device other than the second electronic device 415'), the second electronic device 415' may send input data including information corresponding to the detected input to the first electronic device 401 (e.g., directly or indirectly via a server).

[0086] It should be noted that in some examples, the first user interface 411′ and / or the second user interface 425 may additionally or alternatively be interacted with via input detected by the first electronic device 401. For example, individual elements of the first user interface 411′ and / or the second user interface 425 (e.g., text input fields 413′ / 416, etc.) may be selected via voice or hand-based input, such as air pinch input (e.g., the index finger and thumb of the user's hand touching together) or air tap or touch gesture (e.g., provided by the index finger of the hand), in accordance with the above-mentioned gaze-based rules for routing input. In addition, in some examples, the virtual extended display 404 itself may be interacted with via hand-based input detected by the first electronic device 401. For example, the first electronic device 401 may reposition the virtual extended display 404 within the three-dimensional environment 450 in response to detecting a selection of the virtual extended display 404 (e.g., via air pinch input, etc.) and subsequently detecting a movement of the user's hand. As another example, the first electronic device 401 may scale the virtual augmented display 404 within the three dimensional environment 450 such that the size of the second user interface 425 may be increased or decreased in the three dimensional environment 450 and / or additional user interfaces may be displayed on the virtual augmented display 404. In such examples where input is directed to the virtual augmented display 404 itself rather than the content of the virtual augmented display 404 (e.g., the second user interface 425), the input is responded to by the first electronic device 415 rather than the second electronic device 301′ (e.g., because the hand-based input described above is detected by the first electronic device 301 independently of the second electronic device 415′).

[0087] It should also be noted that in some examples, one or more physical input devices (e.g., physical input devices such as mouse 332′, 432′ and keyboard 331′, 431′) may additionally or alternatively be communicatively coupled (e.g., wired or wirelessly) to the first electronic device (e.g., first electronic device 301 or 401). In these examples, input from the physical input devices may be routed to an appropriate device or interface in the same manner as described above. However, in these examples, input from the physical input devices need not first be detected by and sent to the second electronic device (e.g., second electronic device 315′ or 415′) before being routed to the appropriate device or interface.

[0088] Thus, as outlined above, providing an augmented virtual display for an electronic device in a three-dimensional environment that can be interacted with via input detected on a physical input device in communication with the electronic device advantageously facilitates seamless and efficient interaction with the display of the electronic device and the augmented virtual display without requiring the user to utilize separate input means. Another advantage of the above system and method is the seamless integration of a set of input devices (e.g., keyboard, mouse, trackpad, etc.) of a first physical electronic device with a set of input devices (e.g., eye tracking sensors, hand tracking sensors, orientation sensors, etc.) of a second physical electronic device, which allows input received at both sets of input devices to be directed to the same one or more user interfaces in the three-dimensional environment. Attention is now directed to an exemplary interaction with an image of a virtual augmented display in a real-time multi-user communication session.

[0089] 5A-5G illustrate exemplary interactions with representations of user interfaces of electronic devices in a multi-user communication session, according to some examples of the present disclosure. In some examples, a first electronic device 501 may be used to present a three-dimensional environment 550, as shown in FIG. 5A. In some examples, the first electronic device 501 optionally corresponds to the first electronic device 401 / 301 described above. In some examples, the three-dimensional environment 550 includes a captured portion of a physical environment in which the first electronic device 501 is located. For example, the three-dimensional environment 550 optionally includes a table (e.g., a representation of a table 506′), as shown in FIG. 5A. In addition, the three-dimensional environment 550 includes a second electronic device (e.g., a representation of a second electronic device 515′) and one or more physical input devices (e.g., one or more representations of one or more input devices 531′ / 532′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, the three dimensional environment 550 optionally corresponds to the three dimensional environment 450 / 350 described above. In some examples, the second electronic device 515' corresponds to the second electronic device 415' / 315' described above. In some examples, the one or more input devices 531' / 532' correspond to the one or more input devices 431' / 432' and / or 331' / 332' described above. In some examples, the representation of the physical environment may include a portion of the physical environment that is viewed through a transparent or semi-transparent display of the first electronic device 501.

[0090] In some examples, the first electronic device 501 may communicate with the second electronic device 515' such that the first electronic device 501 may be configurable to display a virtual augmented display for the second electronic device 515', similar to that described herein above. For example, as shown in FIG. 5A, the three-dimensional environment 550 includes a first affordance 519 that is displayed at the second electronic device 515'. In some examples, the first affordance 519 corresponds to the first affordance 319 described above. In FIG. 5A, the first electronic device 501 is not displaying a virtual augmented display for the second electronic device 515' because the first affordance 519 has not been selected by a user of the first electronic device 501.

[0091] In addition, in some examples, as shown in FIG. 5A, the three-dimensional environment 550 includes an invitation element 535 corresponding to an invitation to join a third electronic device, different from the first electronic device 515 and the second electronic device 501′, in the multi-user communication session. For example, the first electronic device 501 displays the invitation element 535 in response to detecting an indication from a third electronic device (not shown) inviting the user of the first electronic device 501 to join the user of the third electronic device in the multi-user communication session. In some examples, as shown in FIG. 5A, the invitation element 535 includes a first option 537-1 selectable to accept the invitation to join the multi-user communication session and a second option 537-2 selectable to decline the invitation to join the multi-user communication session (e.g., including ceasing to display the invitation element 535 in the three-dimensional environment 550).

[0092] 5A, while displaying the invitation element 535 in the three-dimensional environment 550, the first electronic device 501 detects a selection input 571A directed to a first option 537-1 of the invitation element 535. For example, the first electronic device 501 detects, via one or more sensors of the first electronic device 501, an air pinch gesture performed by the user's hand, a tap or touch gesture performed by the hand, a verbal command, a gaze dwell directed to the first option 537-1, etc. In some examples, as shown in FIG. 5B, in response to detecting a selection of the first option 537-1, the first electronic device 501 joins a third electronic device in a multi-user communication session, as described below.

[0093] In some examples, when the first electronic device 501 participates in a multi-user communication session with a third electronic device (not shown), the first electronic device 501 and the third electronic device are configured to present a shared three-dimensional environment including one or more shared virtual objects (e.g., content such as images, video, audio, a representation of a user interface of an application, etc.). As used herein, the term "shared three-dimensional environment" refers to a three-dimensional environment that is independently presented, displayed, and / or viewed at two or more electronic devices, in which content, applications, data, etc. may be shared and / or presented to users of the two or more electronic devices. In some examples, while the first electronic device 501 is in a multi-user communication session with the second electronic device, an avatar corresponding to a user of one electronic device is optionally displayed within the three-dimensional environment displayed via the other electronic device. For example, as shown in FIG. 5B, at the first electronic device 501, an avatar 536 corresponding to a user of the third electronic device is displayed in the three-dimensional environment 550.

[0094] In some examples, the presentation of the avatar 536 as part of the shared three-dimensional environment is optionally accompanied by audio effects corresponding to the voice of the user of the third electronic device. For example, the avatar 536 displayed in the three-dimensional environment 550 using the first electronic device 501 is optionally accompanied by audio effects corresponding to the voice of the user of the third electronic device. In some such examples, when the user of the third electronic device speaks, the user's voice may be detected by the third electronic device (e.g., via a microphone) and transmitted to the first electronic device 501 (e.g., directly or via a server (e.g., a wireless communication terminal)), so that the detected voice of the user of the third electronic device may be presented as audio (e.g., using the speaker(s) 216) to the user of the first electronic device 501 in the three-dimensional environment 550. In some examples, audio effects corresponding to the voice of a user of the third electronic device may be spatialized such that they appear to the user of the first electronic device 501 to emanate from the location of the avatar 536 within the three-dimensional environment 550 (e.g., despite being output from speakers of the first electronic device 501). In some examples, audio effects corresponding to the voice of a user of the third electronic device may alternatively be presented in mono or stereo at the first electronic device 501.

[0095] In some examples, while the first electronic device 501 and the third electronic device are in a multi-user communication session, the avatar 536 is displayed in the three-dimensional environment 550 with an individual orientation that corresponds to and / or is based on an orientation of the third electronic device (and / or a user of the third electronic device) in the physical environment surrounding the third electronic device. For example, in FIG. 5B , in the three-dimensional environment 550, the avatar 536 is, optionally, oriented toward the viewpoint of the user of the first electronic device 501. Within the multi-user communication session, as a particular user moves the electronic device (and / or the user itself) in the physical environment, the user's viewpoint may change accordingly, and thus the orientation of the user's avatar in the three-dimensional environment may also change. For example, referring to FIG. 5B , if a user of the third electronic device looks leftward in the three-dimensional environment such that the third electronic device is rotated to the left (e.g., counterclockwise) (e.g., a corresponding amount), the user of the first electronic device 501 will see the avatar 536 corresponding to the user of the third electronic device rotate to the right (e.g., clockwise) relative to the perspective of the user of the first electronic device 501 as the third electronic device moves.

[0096] Additionally, in some examples, while the first electronic device 501 and the third electronic device are in a multi-user communication session, the field of view of the shared three-dimensional environment and / or the location of the user's viewpoint within the shared three-dimensional environment optionally changes in accordance with movement of the electronic devices (e.g., by the user of the electronic device). For example, if during the communication session the first electronic device 501 is moved closer to the representation of the table 506' and / or the avatar 536 (e.g., because the user of the first electronic device 501 moves forward in the physical environment surrounding the first electronic device 501), the field of view of the three-dimensional environment 550 changes accordingly, causing the representation of the table 506', the representation of the second electronic device 515', and the avatar 536 to appear larger in the field of view. In some examples, each user may interact with the shared three-dimensional environment independently, such that changes in viewpoint within the three-dimensional environment 550 and / or interactions with virtual objects within the three-dimensional environment 550 by the first electronic device 501 optionally do not affect what is shown in the three-dimensional environment at the third electronic device, and vice versa.

[0097] In some examples, the avatar 536 is a representation (e.g., a full-body rendering) of the user of the third electronic device. In some examples, the avatar 536 is a representation of a portion of the user of the third electronic device (e.g., a rendering of the head, face, head and torso, etc.). In some examples, the avatar 536 is a user-personalized, user-selected, and / or user-created representation representing the user of the third electronic device that is displayed in the three-dimensional environment 550. While the avatar 536 shown in FIG. 5B corresponds to a full-body representation of the user of the third electronic device, it should be understood that alternative avatars, such as one of those described above, may be provided.

[0098] In some examples, while the first electronic device 501 and the third electronic device are in a multi-user communication session, content viewed by one user at one electronic device may be shared with another user at another electronic device in the multi-user communication session. In some such examples, the content may be experienced (e.g., viewed and / or interacted with) by both users (e.g., via their respective electronic devices) in a shared three-dimensional environment (e.g., the content is content shared in the three-dimensional environment). For example, in FIG. 5C, the first electronic device 501 displays visual indications 539 corresponding to a request from a user of the third electronic device (e.g., “User 1”) to share content (e.g., “Content A”) with the user of the first electronic device 501. As shown in FIG. 5C, the visual indications 539 optionally include a selectable option 539-1 selectable to approve the request from the user of the third electronic device to share content with the user of the first electronic device 501. In some examples, the first electronic device 501 displays a visual indication 539 within the three-dimensional environment 550 in response to detecting data corresponding to the sharing request described above.

[0099] In addition, in some examples, the shared three-dimensional environment includes non-shared content that is private to one user in the multi-user communication session. For example, in FIG. 5B-5C, the third electronic device may display a private application window within the three-dimensional environment at the third electronic device, which is, optionally, an object that is not shared between the first electronic device 501 and the third electronic device in the multi-user communication session. In some examples, the private application window may be associated with an individual application (e.g., a virtual augmented display application, a media player application, a web browsing application, a messaging application, etc.) running on the third electronic device. Because the private application window is not shared with the first electronic device 501, the first electronic device 501 optionally displays a representation of a private application window 504'' within the three-dimensional environment 550, as shown in FIG. 5B-5C. As shown in FIGS. 5B-5C, in some examples, the representation of the private application window 504'' may be a faded, obscured, discolored, and / or semi-transparent representation of the private application window that prevents a user of the first electronic device 501 from viewing the content of the private application window.

[0100] In FIG. 5C , while displaying visual indications 539 within the three-dimensional environment 550, the first electronic device 501 detects a selection input 571B directed toward a selectable option 539-1 of the visual indications 539. For example, as also described above, the first electronic device 501 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell directed toward the selectable option 539-1, etc. In some examples, such as FIG. 5C , the request to share content with the first electronic device 501 corresponds to a request to share a private application window 504″ (visible in the three-dimensional environment 550) with a user of the first electronic device 501.

[0101] In some examples, as shown in FIG. 5D , in response to detecting a selection of the selectable option 539-1, the first electronic device 501 approves the request from the third electronic device to share content with the first electronic device 501. In some examples, since the content shared with the first electronic device 501 corresponds to the content of the private application window described above, the first electronic device 501 stops displaying a representation of the private application window 504″ in the three-dimensional environment 550. Also, as shown in FIG. 5D , when the content is shared with the first electronic device 501, the first electronic device 501 displays an image 504 corresponding to the shared content in the three-dimensional environment 550. In some examples, as shown in FIG. 5D , when the first electronic device 501 displays the image 504 in the three-dimensional environment 550, the position of the avatar 536 corresponding to the user of the third electronic device optionally changes in the three-dimensional environment 550. For example, the avatar 536 is moved to a location adjacent to (e.g., to the right of) the user's viewpoint of the first electronic device 501 so that the avatar 536 faces toward the content of the image 504 in the three-dimensional environment 550, as shown in FIG. 5D.

[0102] In some examples, the image 504 corresponds to an image of a virtual augmented display of a fourth electronic device that is different from the first electronic device 501 and the second electronic device 515'. For example, the virtual augmented display is a second display for the fourth electronic device that is in communication with the third electronic device, similar to the communication between the electronic devices 401 and 415 described above. In some examples, the third electronic device is displaying a virtual augmented display in a three-dimensional environment at the third electronic device, and in response to detecting a user input from a user of the third electronic device, has sent a request (e.g., via an application sharing application) to share an image of the virtual augmented display with a user of the first electronic device 501, as described above. As shown in FIG. 5D, the image 504 of the virtual augmented display includes a user interface 540. For example, the user interface 540 was displayed on the virtual augmented display at the third electronic device before the image of the virtual augmented display was shared with the first electronic device 501. In some examples, in FIG. 5D, the user interface 540 is a user interface of a music player application. For example, as shown in FIG. 5D , user interface 540 includes a first representation 541-1 of a first music album and a second representation 541-2 of a second music album, each including selectable indications of music tracks available for consumption on a fourth electronic device and / or a third electronic device.

[0103] In some examples, the user of the first electronic device 501 can see the content of the image 504 shared between the first electronic device 501 and the third electronic device, but the first electronic device 501 optionally limits and / or prevents direct user interaction with the content of the image 504, such as interaction with the user interface 540. In particular, in some examples, the user of the first electronic device 501 cannot provide input to interact with the content of the image 504 because only the image of the virtual augmented display is shared with the first electronic device 501, rather than the virtual augmented display itself being a true shared experience. Thus, as described above, because the virtual augmented display is optionally shared by the third electronic device, the user of the third electronic device may interact with the user interface 540 at the third electronic device, which causes the image 504 to be updated in the three-dimensional environment 550, but the user of the first electronic device 501 may not be able to do so, as described below.

[0104] In Figure 5E, the second electronic device 501' detects a selection input 573 via a mouse 532' in communication with the second electronic device 515' while the virtual augmented display image 504 is displayed in the three-dimensional environment 550 at the first electronic device 515. For example, the second electronic device 514' detects a button press or click on the mouse 532' (or a tap on the touch-sensitive surface of the mouse 532'). In addition, as shown in Figure 5E, the second electronic device 514' detects the selection input 573 while the gaze 527 of the user of the first electronic device 501 is directed toward the user interface 540 of the image 504 in the three-dimensional environment 550.

[0105] In some examples, as shown in FIG. 5F, in response to the second electronic device 514′ detecting a selection input 573 via the mouse 532′ while the first electronic device 501 detects a line of sight 427 directed toward the user interface 540 of the image 504, the first electronic device 501 ceases to perform an action directed toward the image 504. For example, because the virtual extended display of the image 504 is not an extended display of the second electronic device 514′ as described above, the input detected via the mouse 532′ is not interpreted by the first electronic device 501 as being directed toward the image 504. Rather, in some examples, the selection input 573 detected by the second electronic device 514′ remains local to the second electronic device 514′ (e.g., the second electronic device 514′ performs the selection action according to the selection input 573 without transmitting the input data to the first electronic device 501 as described above).

[0106] In FIG. 5F, while displaying an image 504 of a virtual augmented display in a three-dimensional environment 550, the first electronic device 501 detects a selection input 571C directed to a first representation 541-1 of the user interface 540 in the image 504. For example, similar to that described herein above, the first electronic device 501 detects an air pinch gesture, an air tap or touch gesture, a verbal command, a gaze dwell directed to the first representation 541-1, etc. In some examples, in response to detecting a selection of the first representation 541-1 in the image 504, the first electronic device 501 ceases to perform a selection action directed to the first representation 541-1, as shown in FIG. 5G. For example, as shown in FIG. 5G, in response to the selection input 571C, the image 504 is not updated such that music tracks associated with the first album (e.g., "Album A") are displayed in the user interface 540.

[0107] In some examples, as also described above, because the virtual augmented display of the image 504 is not a true shared experience between the first electronic device 501 and the third electronic device (e.g., because the virtual augmented display is not shared with a user of the first electronic device 501, but rather is an image of the virtual augmented display), user input detected by the first electronic device 501 directed at the image 504 including the user interface 540 does not cause the image 504 to update based on the user input. Thus, as shown in FIG. 5G, the image 504 remains unchanged in the three-dimensional environment 550. Thus, as outlined above, when an image of the virtual augmented display displayed at the third electronic device is shared with the first electronic device 501 while the first electronic device 501 and the third electronic device are communicatively linked in a multi-user communication session, input provided by the user of the first electronic device 501 may not cause the content of the image (e.g., the image 504) to update in accordance with the input.

[0108] In some examples, interaction may be provided that is directed to the image 504 itself, rather than the content of the image 504. For example, if a user of the first electronic device 501 provides an input to move the image 504 within the three-dimensional environment 550 (as opposed to, e.g., an input to move the user interface 540 within the image 504), the first electronic device 501 may move the image 504 according to the input. Similarly, if a user of the first electronic device 501 provides an input to scale the image 504 within the three-dimensional environment 550 (as opposed to, e.g., resizing the user interface 540 within the image 504), the first electronic device 501 may scale the image 504 according to the input. In some examples, if a user of a third electronic device shares the content of the image 504 itself, such shared content may be configured to be interacted with by the user of the first electronic device 501. For example, in FIG. 5C, if the content being shared with the user of the first electronic device 501 is the user interface 540 itself, the first electronic device 501 displays a shared application window corresponding to the user interface 540 of FIG. 5D, and input directed to the content of the shared application window (e.g., first representation 541-1 and / or second representation 541-2) is acted upon by the first electronic device 501.

[0109] It is understood that the examples shown and described herein are merely illustrative and that additional and / or alternative elements may be provided within the three-dimensional environment for interacting with the virtual objects and elements. It is understood that the appearance, shape, form, and size of each of the various user interface elements and objects shown and described herein are exemplary and that alternative appearances, shapes, forms, and / or sizes may be provided. For example, virtual objects representing a virtual augmented display (e.g., virtual augmented display 304 / 404) and / or a user interface (e.g., user interfaces 411, 425, and / or 540) may be provided in shapes other than a rectangle, such as a circle, a triangle, etc. In some examples, the various selectable affordances described herein (e.g., first affordance 319 / 519 and / or options 537-1 / 537-2 or 539-1) may be selected verbally via a user's verbal command (e.g., a "select option" or a "select virtual object" verbal command). Additionally or alternatively, in some examples, various options, user interface elements, controls, etc. described herein may be selected and / or manipulated via user input received via one or more separate input devices in communication with the electronic device(s). For example, selection input may be received via a physical input device, such as a mouse, trackpad, keyboard, etc., in communication with the electronic device(s).

[0110] In addition, while only a single interface is shown in the physical and virtual extended displays, it should be understood that any number of interfaces may be included in each display. Further, while only a single virtual extended display (e.g., virtual extended display 304 / 404) is shown for a single electronic device (e.g., second electronic device 315 / 415), it should be understood that any number of virtual extended displays may be presented for each of any number of other electronic devices. In these examples, input detected by any of the electronic devices may be routed to the appropriate electronic device or interface in the same manner as described above.

[0111] In some examples, in addition to displaying the virtual extended display, the first electronic device may display one or more additional virtual objects or windows corresponding to applications running on the first electronic device, as described in more detail with reference to Figures 6A-6K. For example, the first electronic device 415, in addition to displaying the virtual extended display 404 including the second user interface 425 for the web browsing application of the second electronic device 401' in Figure 4A, may further display a virtual window corresponding to a text document editing application running on the first electronic device 415 (but not the second electronic device 401'). In these examples, input detected by the first electronic device 401, the second electronic device 415', or other coupled electronic devices may be routed to the appropriate device, application, and / or interface in the same manner as described above.

[0112] 6A-6K show examples of a first electronic device facilitating interaction between a representation of a user interface of a second electronic device and a user interface of the first electronic device, according to some examples of the present disclosure. In some examples, as shown in FIG. 6A, a three-dimensional environment 650 may be presented using the first electronic device 601. In some examples, the first electronic device 601 optionally corresponds to the first electronic device 301 / 401 / 501 described above. In some examples, the three-dimensional environment 650 includes a captured portion of a physical environment in which the first electronic device 401 is located. For example, the three-dimensional environment 650 optionally includes a table (e.g., a representation of a table 606′), as shown in FIG. 6A. In addition, the three-dimensional environment 650 includes a second electronic device (e.g., a representation of the second electronic device 615′) and one or more physical input devices (e.g., one or more representations of one or more input devices 631′ / 632′) in communication with the second electronic device (e.g., positioned on a table in the physical environment). In some examples, the three-dimensional environment 650 optionally corresponds to the three-dimensional environment 350 / 450 / 550 described above. In some examples, the second electronic device 615′ corresponds to the second electronic device 315′ / 415′ / 515′ described above. In some examples, the one or more input devices 631′ / 632′ correspond to one or more input devices 331′ / 332′, 431′ / 432′, 531′ / 532′ described above. In some examples, the representation of the physical environment may include a portion of the physical environment viewed through a transparent or semi-transparent display of the first electronic device 601.

[0113] As also described above, in some examples, the first electronic device 601 communicates with the second electronic device 615'. In addition, in some examples, as shown in FIG. 6A, the first electronic device 601 presents a virtual display 604 associated with the second electronic device 615'. For example, FIG. 6A. The virtual display 604 serves as a (e.g., virtual) display for the physical display (e.g., a representation of the display 614') of the second electronic device 615'. In some examples, the virtual display 604 has one or more characteristics of the virtual augmented display 304 / 404 / 504 described above. In some examples, as shown in FIG. 6A, the display 614' of the second electronic device 615' is optionally powered off (e.g., the second electronic device 615' is in a low power mode or sleep state, or is powered off) while the virtual display 604 is displayed in the three-dimensional environment 650.

[0114] In some examples, as shown in FIG. 6A, the virtual display 604 optionally includes a user interface 625 that is a representation of a user interface configured (e.g., or currently displayed) to be displayed on the (e.g., physical) display 614′ of the second electronic device 615′. In some examples, as shown in FIG. 6A, the user interface 625 includes a plurality of search results (e.g., associated with the URL “www.Search.com”) provided by a web browser. For example, in FIG. 6A, the user interface 625 includes a first search result 626A (e.g., “Search Result A”) associated with a first website (e.g., “www.URL1.com”), a second search result 626B (e.g., “Search Result B”) associated with a second website (e.g., “www.URL2.com”), and a third search result 626C (e.g., “Search Result C”) associated with a third website. In some examples, the plurality of search results 626 are selectable to access content (e.g., text, images, videos, music, etc.) via the respective sources (e.g., websites). In some examples, the user interface 625 corresponds to the second user interface 425 described above. Additionally, as shown in FIG. 6A, the virtual display 604 includes a representation of a dock 609″ (e.g., dock 309 / / 409′ / 509 / , etc., described above) that corresponds to a dock (not shown) configured to be displayed on a (e.g., physical) display 614′ of the second electronic device 615′. In some examples, as shown in FIG. 6A, the virtual display 604 also includes a cursor (e.g., a representation of cursor 612′) configured to be displayed on a display 614′ of the second electronic device 615′ and controllable using a mouse 632′. In some examples, the cursor 612′ corresponds to cursor 312′ / 412′ / 512′ described above.

[0115] In some examples, as also described above, the first electronic device 601 displays the user interface 625 within the virtual display 604 based on image data transmitted to the first electronic device 601 by the second electronic device 615′ (e.g., directly or indirectly via a server). For example, as described above, the user interface 625 is configured to be displayed on the display 614′ of the second electronic device 615′ (e.g., the user interface 625 is a user interface of the second electronic device 615′). Thus, in some examples, the first electronic device 601 renders (e.g., generates) the user interface 625 including the content of the user interface 625 described above based on the image data provided by the second electronic device 615′. For example, the image data includes information corresponding to an appearance of the user interface 625, characteristics of the content of the user interface 625 (e.g., whether the content is interactive, such as selectable, scrollable, etc.), and / or a quality of the user interface 625 and its content (e.g., image quality, such as definition). In some examples, the image data transmitted by the second electronic device 601′ to the first electronic device 615 includes an image generated by the second electronic device 601′ that is displayed on the virtual display 604 by the first electronic device 615.

[0116] In some examples, the first electronic device 601 is configured to display content associated with an application executing (e.g., locally) on the first electronic device 601 simultaneously with the virtual display 604. In some examples, as described below, the first electronic device 601 is configured to facilitate interaction with the content of the virtual display 604 and the content of the application executing locally on the first electronic device 601. In FIG. 6A, the first electronic device 601 displays a user interface object 624 associated with an individual application in the three-dimensional environment 650. In some examples, as shown in FIG. 6A, the user interface object 624 includes a selectable option 623A that is selectable to display content (e.g., "Content A") corresponding to the individual application in the three-dimensional environment 650. In some examples, the content includes a user interface, a video, an image, a three-dimensional object (e.g., a model), and the like.

[0117] In some examples, as shown in Figure 6A, the first electronic device 601 detects a selection input 672A directed at the selectable option 623A. For example, in Figure 6A, the first electronic device 601 detects an air pinch gesture, an air tap or a touch gesture (e.g., via one or more sensors such as a camera or other hand tracking sensor), optionally a verbal command, a gaze dwell, etc. while the user's gaze is directed at the selectable option 623A.

[0118] In some examples, as shown in FIG. 6B, in response to detecting the selection input 672A, the first electronic device 601 displays video content (content A shown in FIG. 6A) associated with the respective application (e.g., a video playback application executing on the first electronic device 601) described above. For example, as shown in FIG. 6B, the first electronic device 601 displays a virtual application window 656 (e.g., a video playback user interface) displaying video content (e.g., a movie, a television program, a video clip, a music video, etc.) in a three-dimensional environment 650. As shown in FIG. 6B, the virtual application window 656 includes playback controls 645 (e.g., including a play / pause button, a rewind button, and a fast-forward button, among other possibilities) for controlling playback of the video content displayed in the virtual application window 656. In some examples, as shown in FIG. 6B, the virtual application window 656 is displayed with a grabber affordance 635 (e.g., a handlebar affordance) selectable via a user input to initiate movement of the virtual application window 656 within the three-dimensional environment 650.

[0119] In some examples, similar to those described herein above, the first electronic device 601 communicates with the second electronic device 615′ to facilitate and coordinate interaction with the content of the virtual display 604 and / or the content of the virtual application window 656 in response to user input based on the location of the user's gaze of the first electronic device 601. For example, as shown in FIG. 6C, the first electronic device 601 detects (e.g., via an eye tracking sensor and / or an image sensor) that the user's gaze 627 is directed toward the virtual display 604. In particular, in FIG. 6C, the first electronic device 601 detects that the gaze 627 is directed toward a particular location that corresponds to the content of the virtual display 604.

[0120] In Fig. 6C, the second electronic device 615' detects a selection input 673A on a mouse 632' in communication with the second electronic device 615' while the first electronic device 601 detects that the user's gaze 627 is directed toward the virtual display 604 in the three dimensional environment 650. For example, in Fig. 6C, the second electronic device 615' detects a click, tap, or touch of a button or touch-sensitive surface of the mouse 632'. As also described above, in response to detecting the selection input 673A, the second electronic device 615' maintains the selection input 673A local to the second electronic device 615' as the gaze 627 is directed toward the virtual display 604 in the three dimensional environment 650. For example, the first electronic device 601 communicates (e.g., transmits) gaze data including information corresponding to the location of the line of sight 627 in the three-dimensional environment 602 to the second electronic device 615′ (e.g., continuously, periodically, and / or upon request from the second electronic device 615′).

[0121] In some examples, in accordance with a determination that the line of sight 627 is directed toward the virtual display 604 (e.g., the user interface 625 in particular), the first electronic device 601 causes the second electronic device 615' to perform an operation in accordance with the selection input 673A. For example, in FIG. 6C, the second electronic device 615' performs a selection at the location of the cursor 612' in the user interface 625. In particular, as shown in FIG. 6C, the second electronic device 615' selects (e.g., activates) the second search result 626B based on the location of the cursor 612' in the user interface 625. It should be understood that, in the example of FIG. 6C, the user interface 625 is not displayed on the (e.g., physical) display of the second electronic device 615', performing the selection operation includes updating image data representing the user interface 625 (e.g., as if the user interface 625 was displayed on the display of the second electronic device 615'). In some examples, when the second electronic device 615′ performs the selection operation, the second electronic device 615′ transmits updated image data to the first electronic device 601, which causes the first electronic device 601 to update the representation of the virtual display 604 in the three-dimensional environment 650 based on the performance of the selection operation. For example, as shown in FIG. 6D, the first electronic device 601 updates the display of the virtual display 604 to include an updated representation of the user interface 625. In particular, as shown in FIG. 6D, the user interface 625 is updated to include the result of the selection of the second search result 626B in FIG. 6C, i.e., to display website content (e.g., “Website Content” in the text input field 616) from the website (e.g., “www.URL2.com”) associated with the second search result 626B.

[0122] In Fig. 6E, the first electronic device 601 detects that the user's gaze 627 is directed toward a virtual application window 656 in the three-dimensional environment 650. In particular, as shown in Fig. 6E, the first electronic device 601 detects that the gaze 627 is directed toward a playback control 645 (e.g., a pause button of the playback control 645). In Fig. 6E, while the first electronic device 601 detects the gaze 627 directed toward the virtual application window 656, the second electronic device 615' detects a selection input 673B via a mouse 632' in communication with the second electronic device 615' in a similar manner as described above.

[0123] In some examples, similar to those described above, in response to detecting the selection input 673B, the second electronic device 615' transmits input data corresponding to the selection input 673B to the first electronic device 601, as the line of sight 627 is directed toward the virtual application window 656 in the three-dimensional environment 650. For example, the second electronic device 615' transmits the input data to the first electronic device 601 based on the line of sight data communicated (e.g., transmitted) by the first electronic device 601 to the second electronic device 615', also as described above.

[0124] In some examples, in accordance with a determination that the line of sight 627 is directed toward the virtual application window 656 (e.g., the playback control 645 in particular), the first electronic device 601 performs an action in accordance with the selection input 673B. For example, in FIG. 6E, the first electronic device 601 performs a selection at a location of the line of sight 627 within the virtual application window 656. Specifically, as shown in FIG. 6E, the first electronic device 601 selects (e.g., activates) a pause button of the playback control 645 based on a location of the line of sight 627 within the virtual application window 656 (e.g., rather than based on a location of the cursor 612′ within the user interface 625, as described herein above). In other examples, the selection input 673B may be performed in accordance with a location of the cursor 612′. In some examples, as shown in FIG. 6F, when the first electronic device 601 selects the pause button of the playback control 645, the first electronic device 601 pauses playback of the video content within the virtual application window 656 within the three-dimensional environment 650. 6F , the first electronic device 601 optionally updates a pause button in the playback controls 645 in the virtual application window 656 to a play button. In some examples, when the first electronic device 601 performs a selection action directed to the virtual application window 656, the first electronic device 601 ceases sending output data corresponding to the selection of the pause button of the playback controls 645 in the virtual application window 656 to the second electronic device 615′.

[0125] In Figure 6G, while a line of sight 627 is directed toward a virtual display 604 in the three-dimensional environment 650, the first electronic device 601 detects a selection input 672B directed toward the content of the virtual display 604 via one or more input devices of the first electronic device 601. For example, as shown in Figure 6G, the first electronic device 601 detects an air pinch gesture, a tap or touch gesture, a verbal command, etc., while a line of sight 627 is directed toward a text input field 616 of a user interface 625 in the virtual display 604.

[0126] In some examples, in response to detecting the selection input 672B while the line of sight 627 is directed toward the virtual display 604, the first electronic device 601 transmits input data corresponding to the selection input 672B to the second electronic device 615'. In some examples, the second electronic device 615' performs an operation directed toward the user interface 625 based on the input data corresponding to the selection input 672B. For example, the first electronic device 601 causes the second electronic device 615' to perform a selection operation directed toward the text input field 616 based on the location of the line of sight 627 of FIG. 6G. In some examples, similar to those described above, when the second electronic device 615' performs the selection operation, the second electronic device 615' transmits updated image data to the first electronic device 601, causing the first electronic device 601 to update the representation of the virtual display 604 in the three-dimensional environment 650 based on the performance of the selection operation. For example, as shown in Figure 6H, first electronic device 601 updates the display of virtual display 604 to include an updated representation of user interface 625. In particular, as shown in Figure 6H, text input field 616 includes text cursor 628 and no longer contains the text "www.URL2.com" from Figure 6G. Additionally, as shown in Figure 6H, the location of cursor 612' is optionally updated to correspond to the location of gaze 627 within user interface 625, as also discussed herein.

[0127] 6H , the second electronic device 615′ detects a selection input 673C via one or more keys of a keyboard 631′ in communication with the second electronic device 615′ while the gaze 627 remains directed at the virtual display 604. For example, the second electronic device 615′ detects one or more presses or taps of one or more keys of the keyboard 631′ while the first electronic device 601 detects the gaze 627 directed at the user interface 625 in the virtual display 604, as also described herein.

[0128] In some examples, as also described above, in response to detecting the selection input 673C, the second electronic device 615′ maintains the selection input 673C local to the second electronic device 615′ (e.g., without transmitting input data corresponding to the selection input 673C to the first electronic device 601) as the line of sight 627 is directed toward the virtual display 604 in the three-dimensional environment 650. For example, in FIG. 6H , the second electronic device 615′ performs an action directed toward the user interface 625 in accordance with the selection input 673C, as also described above. In some examples, as the second electronic device 615′ performs an action in accordance with the selection input 673C, the second electronic device 615′ updates image data (e.g., pixel or display data) representing the user interface 625 based on the action and transmits the updated image data to the first electronic device 601 to cause the first electronic device 601 to update the representation of the user interface 625 in the virtual display 604. For example, as shown in FIG. 6I, the first electronic device 601 updates the user interface 625 in the virtual display 604 such that the text entry field 616 is updated to include entered text 649 (e.g., “www.URL4.com”) at the location of the text cursor 628 based on the selected key of the keyboard 631′. In FIG. 6I, while the gaze 627 is directed at the virtual display 604 in the three-dimensional environment 650, the first electronic device 601 detects a selection input 672C provided by a user of the first electronic device 601. For example, as shown in FIG. 6I, while the gaze 627 is directed at an option 651 (e.g., an enter or search option) in the text entry field 616, the first electronic device 601 detects an air pinch gesture, a tap or touch gesture, a verbal command, a gaze dwell, or the like, as also described herein, via one or more sensors or input devices of the first electronic device 601.In some examples, as also described above, because the line of sight 627 is directed toward the virtual display 604 when the selection input 672C is detected, the first electronic device 601 transmits input data corresponding to the selection input 672C to the second electronic device 615′, which causes the second electronic device 615′ to perform an action directed toward the user interface 625 based on the location of the line of sight 627. In particular, in FIG. 6I , the second electronic device 615′ performs a selection action directed toward option 651 based on the location of the line of sight 627 (e.g., in accordance with the line of sight data described above that includes information corresponding to the location of the line of sight 627 within the user interface 625).

[0129] In some examples, as also described herein, when the second electronic device 615′ performs a selection action directed to the option 651, the second electronic device 615′ updates image data representing the user interface 625 and transmits the updated image data to the first electronic device 601. In some examples, as shown in FIG. 6J, in response to receiving the updated image data from the second electronic device 615′, the first electronic device 601 updates the display of the user interface 625 displayed on the virtual display 604 in the three-dimensional environment 650. Specifically, as shown in FIG. 6J, upon selecting the option 651 in FIG. 6I, the second electronic device 615′ navigates to the website “www.URL4.com” and displays content from the website (e.g., “URL4 website content”), which is reflected in an updated representation of the user interface 625 in the virtual display 604 in the three-dimensional environment 650. Additionally, in some examples, as described above in this specification, the first electronic device 601 updates the location of the cursor 612' in the user interface 625 based on the updated image data, such that the location of the cursor 612' is moved to the location of the line of sight 627 when the selection input 672C is detected in FIG. 6I.

[0130] In Figure 6J, the first electronic device 601 detects a selection input 672D in a similar manner as described above while the line of sight 627 is directed at a virtual application window 656 in the three-dimensional environment 650. Specifically, as shown in Figure 6J, the first electronic device 601 detects a selection input 672D (e.g., an air pinch gesture, an air tap or touch gesture, a verbal command, etc.) while the line of sight 627 is directed at a playback control 645 (e.g., a play button in the playback control 645) in the virtual application window 656.

[0131] In some examples, as shown in Figure 6K, because the gaze 627 was directed to the virtual application window 656 when the selection input 672D was detected in Figure 6J, the first electronic device 601 performs an action directed to the virtual application window 656 according to the selection input 672D. For example, as shown in Figure 6K, the first electronic device 601 selects the play button of the playback control 645, which causes the video content displayed in the virtual application window 656 to resume playing in the three-dimensional environment 650. In addition, as shown in Figure 6K, the first electronic device 601 replaces the play button with a pause button in the playback control 645, similar to the above.

[0132] In some examples, as also described herein, when the first electronic device 601 performs the selection operation described above, the first electronic device 601 ceases to transmit output data corresponding to the selection operation to the second electronic device 615'. In particular, the input data corresponding to the selection input 672D of FIG. 6J is maintained local to the first electronic device 601 as described above, and the result of the selection operation in response to the selection input 672D is also maintained local to the first electronic device 601. Thus, as described above, providing a virtual display for the second electronic device in a three-dimensional environment that may be interacted with via input detected on a physical input device in communication with the second electronic device or an input device in communication with the first electronic device advantageously facilitates seamless and efficient interaction with the virtual display of the second electronic device and applications running on the first electronic device without requiring a user to utilize a separate input means. Another advantage of the above systems and methods is seamless integration of a set of input devices (e.g., keyboard, mouse, trackpad, etc.) of a first physical electronic device with a set of input devices (e.g., eye tracking sensors, hand tracking sensors, orientation sensors, etc.) of a second physical electronic device, which enables input received at both sets of input devices to be directed to the same one or more user interfaces within the three-dimensional environment, which enhances user-device interaction.

[0133] 7A-7B are flow diagrams illustrating an example process for facilitating display of and interaction with a representation of a user interface of an electronic device in a three-dimensional environment, according to some examples of the present disclosure. In some examples, the process 700 begins at a first electronic device in communication with a display and one or more input devices of the first electronic device. In some examples, the first electronic device is, optionally, a head-mounted display similar to or corresponding to device 240 of FIG. 2. As shown in FIG. 7A, in some examples, at 702, the first electronic device captures, via one or more input devices, at least a portion of a real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device. For example, as shown in FIG. 4A, the first electronic device 401 captures a portion of a physical environment surrounding the first electronic device 401 including a second electronic device 415 having a display 414, the display 414 displaying a first user interface 411. In some examples, the second electronic device is, optionally, a computer similar to or corresponding to device 230 of FIG. 2 (e.g., a laptop computer, a desktop computer, or a tablet computer).

[0134] In some examples, at 704, the first electronic device presents, via a display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device. For example, as shown in FIG. 3G, the first electronic device 301 presents a three-dimensional environment 350 including a representation of the second electronic device 315′, a representation of a table 306′ on which the second electronic device 315′ is positioned, and a first affordance 319 and / or system options 318A associated with the second electronic device 315′. In some examples, at 706, while the first electronic device presents the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance, the first electronic device receives, via one or more input devices, an input corresponding to a selection of the first affordance. For example, as shown in FIG. 3H, the first electronic device 301 detects a selection input 371B (e.g., an air pinch gesture) directed at a system option 318A displayed in a predetermined area of ​​the display of the first electronic device 301.

[0135] In some examples, in response to receiving the input, at 708, the first electronic device presents, in the computer-generated environment, a second representation of a second user interface of the second electronic device, different from the first representation of the first user interface. For example, as shown in FIG. 4A, the first electronic device 401 displays a virtual augmented display 404, optionally including a second user interface 425 configured to be displayed on a display 414' of the second electronic device 415' in a three-dimensional environment 450. In some examples, as shown in FIG. 7B, while presenting the computer-generated environment including the second representation of the second user interface, at 710, the first electronic device detects a first indication of a first individual input received via one or more input devices of the second electronic device. For example, as shown in FIG. 4B, the first electronic device 401 receives input data including information corresponding to a movement input 473A detected via a mouse 432' in communication with the second electronic device 415'.

[0136] In some examples, in response to detecting the first indication at 712, in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface at 614, the first electronic device causes the second electronic device to perform a first action directed to the first user interface in accordance with the first discrete input. For example, in FIG. 4B, when a movement input 473A is detected by the second electronic device 415′ via the mouse 432′, the gaze 427 of the user of the first electronic device 401 is directed to the first user interface 411′, which causes the first electronic device 401 to route the movement input 473A to the second electronic device 415′, which causes the second electronic device 415′ to move a cursor 412′ in the first user interface 411′, as shown in FIG. 4C. Alternatively, in some examples, before the movement input 473A is detected by the second electronic device 415′ via the mouse 432′, the first electronic device 401 may send an indication to the second electronic device 415′ that the input should be directed to the first user interface 411′, causing the second electronic device 415′ to process the movement input 473A locally without sending information corresponding to the movement input 473A to the first electronic device 401. In some examples, in accordance with determining at 716 that the gaze is directed to the second representation of the second user interface, the first electronic device performs a second action directed to the second representation of the second user interface in accordance with the first discrete input. For example, in FIG. 4I, when a movement input 473G is detected by the second electronic device 415' via the mouse 432', the user's gaze 427 of the first electronic device 401 is directed to the second user interface 425 of the virtual augmented display 404, which causes the first electronic device 401 to move the cursor 412' within the second user interface 425 in accordance with the movement input 473G, as shown in FIG. 4J.

[0137] It will be understood that process 700 is an example and that more, fewer, or different operations may be performed in the same or different order. Furthermore, the operations in process 700 described above are, optionally, implemented by executing one or more functional modules within an information processing device, such as a general-purpose processor (e.g., as described with respect to FIG. 2) or an application-specific chip, and / or by other components of FIG.

[0138] FIG. 8 is a flow diagram illustrating an example process for facilitating display and interaction of a representation of a user interface between two electronic devices in a three-dimensional environment, according to some examples of the present disclosure. In some examples, the process 800 begins at a first electronic device in communication with a display and one or more input devices of the first electronic device. In some examples, the first electronic device is, optionally, a head-mounted display similar to or corresponding to device 240 of FIG. 2. As shown in FIG. 8, in some examples, at 802, the first electronic device receives display data representing a second user interface of the second electronic device from a second electronic device while presenting, via the display, a computer-generated environment including a first user interface of an application executed on the first electronic device. For example, as also shown in FIG. 6A, the first electronic device 601 receives image data representing a user interface 625 from a second electronic device 615′ located within a field of view of a user of the first electronic device 601. In some examples, the second electronic device is, optionally, a computer similar to or corresponding to device 230 of FIG. 2 (e.g., a laptop computer, a desktop computer, or a tablet computer).

[0139] In some examples, in response to receiving the display data, at 804, the first electronic device presents, via a display, a representation of a second user interface of the second electronic device based on the display data. For example, as shown in FIG. 6C, the first electronic device 601 simultaneously displays a virtual display 604 including a representation of a user interface 625 with a virtual application window 656 associated with an application running on the first electronic device 601. In some examples, while presenting, at 806, a representation of the first user interface of the application and the second user interface of the second electronic device, the first electronic device 601 detects a separate event corresponding to a user input. For example, as shown in FIG. 6E, the second electronic device 615′ detects a selection input 673B via a physical mouse 632′ in communication with the second electronic device 601′, or as shown in FIG. 6G, the first electronic device 601 detects a selection input 672B via one or more sensors of the first electronic device 615.

[0140] In some examples, in response to detecting the distinct event at 808, in accordance with a determination at 810 that a gaze of the user of the first electronic device is directed to a representation of the second user interface of the second electronic device when the distinct event is detected, the first electronic device causes the second electronic device to perform a first action directed to the second user interface of the second electronic device based on the distinct event. For example, in FIG. 6G, when a selection input 672B is detected by the first electronic device 601, the first electronic device 601 detects a gaze 627 directed to the user interface 625 in the virtual display 604. Thus, as shown in FIG. 6H, the first electronic device 601 causes the second electronic device 615′ to select a text input field 616 of the user interface 625 based on a location of the gaze 627 shown in a representation of the user interface 625 displayed in the virtual display 604 in the three-dimensional environment 650.

[0141] In some examples, in accordance with a determination at 812 that the gaze is directed to a first user interface of the application running on the first electronic device when the respective event is detected, the first electronic device performs a second action (e.g., different from the first action described above) directed to the first user interface based on the respective event. For example, in FIG. 6E, when a selection input 673B is detected by the second electronic device 601′ via the mouse 632′, the first electronic device 615 detects a gaze 627 directed to the virtual application window 656. Thus, as shown in FIG. 6F, the first electronic device 601 selects a pause option in the playback control 645 of the virtual application window 656 based on the location of the gaze 627 according to the input data corresponding to the selection input 673B received from the second electronic device 615′ in FIG. 6E.

[0142] It will be understood that process 800 is an example and that more, fewer, or different operations may be performed in the same or different order. Furthermore, the operations in process 800 described above are, optionally, implemented by executing one or more functional modules within an information processing device, such as a general-purpose processor (e.g., as described with respect to FIG. 2) or an application-specific chip, and / or by other components of FIG.

[0143] Thus, in accordance with the above, some examples of the present disclosure include capturing, at a first electronic device in communication with a display and one or more input devices of the first electronic device, at least a portion of a real-world environment including a second electronic device having a display configured to display a first representation of a first user interface of the second electronic device via the one or more input devices; presenting, via the display, a computer-generated environment including a representation of the captured portion of the real-world environment and a first affordance associated with the representation of the second electronic device; receiving, via the one or more input devices, an input corresponding to a selection of the first affordance while presenting the computer-generated environment including the representation of the captured portion of the real-world environment and the first affordance; and, in response to receiving the input, displaying a first affordance in the computer-generated environment. presenting a second representation of a second user interface of a second electronic device, the second representation of the first user interface being different from the first representation of the first user interface; detecting a first indication of a first discrete input received via one or more input devices of the second electronic device while presenting a computer-generated environment including the second representation of the second user interface; and in response to detecting the first indication, causing the second electronic device to perform a first action directed to the first user interface in accordance with the first discrete input in accordance with a determination that a gaze of a user of the first electronic device is directed to the first representation of the first user interface and a second action directed to the second representation of the second user interface in accordance with the first discrete input in accordance with a determination that the gaze is directed to the second representation of the second user interface.

[0144] Additionally or alternatively, in some examples, the first affordance is displayed in a predetermined area of ​​the display separate from the representation of the second electronic device in the computer-generated environment. Additionally or alternatively, in some examples, the method further includes ceasing to perform any operation on the first representation of the first user interface or the second representation of the second user interface in response to detecting the indication in accordance with a determination that the user's gaze is directed to a distinct location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface. Additionally or alternatively, in some examples, the method includes detecting a second indication of a second distinct input received via one or more input devices of the second electronic device while presenting the computer-generated environment including the second representation of the second user interface, where after detecting the first indication, no indication of the input was detected prior to detecting the second indication, and in response to detecting the second indication, ceasing to perform any operation on the first representation of the first user interface or the second representation of the second user interface in accordance with a determination that the user's gaze is directed to a distinct location in the computer-generated environment that does not include the first representation of the first user interface or the second representation of the second user interface. and, in accordance with a determination that the gaze was directed to the first representation of the first user interface in accordance with the second discrete input, causing the second electronic device to perform a third action directed to the first user interface in accordance with the second discrete input in accordance with a determination that the gaze was directed to the first representation of the first user interface when the first instruction was detected, and performing a fourth action directed to the second representation of the second user interface in accordance with the second discrete input in accordance with a determination that the gaze was directed to the second representation of the second user interface when the first instruction was detected.

[0145] Additionally or alternatively, in some examples, the method further includes detecting a distinct input via one or more input devices of the first electronic device while presenting the computer-generated environment including the second representation of the second user interface, and in response to detecting the distinct input, causing the second electronic device to perform a third action directed to the first user interface in accordance with the distinct input in accordance with a determination that the gaze is directed to the first representation of the first user interface, and a fourth action directed to the second representation of the second user interface in accordance with the distinct input in accordance with a determination that the gaze is directed to the second representation of the second user interface. Additionally or alternatively, in some examples, the first representation of the first user interface includes a cursor controllable via one or more of the one or more input devices of the second electronic device. Additionally or alternatively, in some examples, the first discrete input corresponds to a movement of a cursor, and in response to detecting the first indication, performing a fourth action directed to the second representation of the second user interface in accordance with a determination that the gaze is directed to the second representation of the second user interface includes moving the cursor from the first representation of the first user interface to the second representation of the second user interface in accordance with the movement of the cursor. Additionally or alternatively, in some examples, one or more of the one or more input devices includes a first input device having a touch-sensitive surface, and the first discrete input includes a movement of a contact on the touch-sensitive surface that satisfies one or more criteria.

[0146] Additionally or alternatively, in some examples, the one or more criteria include a criterion that is met when the movement of the contact includes a lift-off of the contact from the touch-sensitive surface. Additionally or alternatively, in some examples, the one or more criteria include a criterion that is met when the movement of the contact on the touch-sensitive surface exceeds a threshold movement. Additionally or alternatively, in some examples, the real-world environment further includes a third electronic device. In some examples, the method further includes ceasing to perform any operation involving the third electronic device in response to detecting the first indication. Additionally or alternatively, in some examples, the method further includes pairing the first electronic device and the second electronic device such that the first electronic device communicates with the second electronic device prior to displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes scanning, via one or more input devices of the first electronic device, an image displayed via a display of the second electronic device that causes the first electronic device to communicate with the second electronic device. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes detecting, via one or more input devices of the first electronic device, a distinct sound output from the second electronic device that causes the first electronic device to communicate with the second electronic device. Additionally or alternatively, in some examples, pairing the first electronic device with the second electronic device includes detecting an indication of an input received via one or more input devices of the second electronic device that causes the first electronic device to communicate with the second electronic device.

[0147] Additionally or alternatively, in some examples, the method further includes detecting a disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment including the second representation of the second user interface, and in response to detecting a disassociation of the first electronic device from the user, sending a disassociation instruction to the second electronic device including data indicative of a state of the second representation of the second user interface, the instruction causing the second electronic device to simultaneously display the first user interface and the second user interface. Additionally or alternatively, in some examples, the method further includes detecting a reassociation of the first electronic device with a user of the first electronic device, presenting a computer-generated environment and sending an indication of the reassociation to the second electronic device in response to detecting the reassociation of the first electronic device with the user, receiving data from the second electronic device indicating a current state of the second user interface while presenting the computer-generated environment, and redisplaying a second representation of the second user interface in the computer-generated environment based on the current state of the second user interface in response to receiving the data. Additionally or alternatively, in some examples, the first electronic device includes a head mounted display and the second electronic device is a laptop computer, a desktop computer, or a tablet computer. Additionally or alternatively, in some examples, the method further includes receiving an invitation to join a communications session with a user of a third electronic device while presenting the computer-generated environment, and in response to receiving the invitation, entering the communications session with the user of the third electronic device in accordance with a determination that the invitation has been accepted by the user of the first electronic device, and simultaneously displaying an avatar corresponding to the user of the third electronic device and a first representation of the first user interface.

[0148] Additionally or alternatively, in some examples, the method further includes receiving an instruction from the third electronic device corresponding to a request to share content with the first electronic device while presenting a computer-generated environment including an avatar corresponding to a user of the third electronic device and a first representation of the first user interface, and in response to receiving the instruction, displaying in the computer-generated environment a third representation of the third user interface corresponding to the content according to a determination that the request is accepted by the user of the first electronic device, the third user interface being configured to be displayed on the third electronic device. Additionally or alternatively, in some examples, the method further includes detecting an input directed to the third representation of the third user interface via one or more input devices of the first electronic device while presenting a computer-generated environment including the third representation of the third user interface, and in response to detecting the input, ceasing to perform an action directed to the third representation of the third user interface according to the input.

[0149] Some examples of the disclosure include a method for presenting a computer-generated environment including a first user interface of an application executing on the first electronic device via a display at a first electronic device in communication with the display and one or more input devices of the first electronic device, receiving display data from a second electronic device representing a second user interface of the second electronic device, and in response to receiving the display data, presenting via the display a representation of the second user interface of the second electronic device based on the display data, while presenting the representation of the first user interface of the application and the second user interface of the second electronic device. detecting a discrete event corresponding to a user input; and in response to detecting the discrete event, causing the second electronic device to perform a first action directed to a second user interface of the second electronic device based on the discrete event in accordance with a determination that a gaze of a user of the first electronic device is directed to a representation of a second user interface of the second electronic device when the discrete event is detected, and causing the second electronic device to perform a second action directed to the first user interface based on the discrete event in accordance with a determination that the gaze is directed to a first user interface of an application running on the first electronic device.

[0150] Additionally or alternatively, in some examples, detecting the discrete event includes detecting an indication of a discrete input received via one or more input devices in communication with the second electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting the indication of the discrete input, transmitting gaze data related to a location of the gaze to the second electronic device in accordance with a determination that a gaze is directed at a representation of a second user interface of the second electronic device when the indication is detected, wherein the second electronic device performs the first operation in accordance with the data related to the location of the gaze without transmitting input data corresponding to the discrete input to the first electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting an indication of the discrete input, transmitting gaze data relating to a location of the gaze to a second electronic device in accordance with a determination that the gaze is directed to a representation of a second user interface of the second electronic device when the indication is detected, the second electronic device performing a first operation in accordance with the data relating to the location of the gaze, and receiving input data from the second electronic device corresponding to the discrete input, whereby the first operation is performed at the second electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting an indication of the discrete input, in accordance with a determination that the gaze is directed to a first user interface of the application running on the first electronic device when the indication is detected, performing a second operation directed to the first user interface in accordance with the discrete input without transmitting input data corresponding to the discrete input to the second electronic device.

[0151] Additionally or alternatively, in some examples, detecting the discrete event includes detecting a discrete input via one or more input devices of the first electronic device. Additionally or alternatively, in some examples, the method further includes, in response to detecting the discrete input, transmitting input data corresponding to the discrete input to the second electronic device in accordance with a determination that a gaze is directed to a representation of a second user interface of the second electronic device when the discrete input is detected. Additionally or alternatively, in some examples, the method further includes, in response to detecting the discrete input, performing a second operation directed to the first user interface in accordance with the discrete input without transmitting input data corresponding to the discrete input to the second electronic device in accordance with a determination that a gaze is directed to a first user interface of the application running on the first electronic device when the discrete input is detected. Additionally or alternatively, in some examples, causing the second electronic device to perform the first operation includes transmitting gaze data to the second electronic device including information indicating that a gaze was directed to a representation of a second user interface of the second electronic device when the discrete event was detected. Additionally or alternatively, in some examples, the information includes position information corresponding to a location of the gaze within a representation of the second user interface. Additionally or alternatively, in some examples, the second electronic device is not displaying the second user interface.

[0152] Additionally or alternatively, in some examples, causing the second electronic device to perform the first operation directed to the second user interface includes updating an appearance of the second user interface. Additionally or alternatively, in some examples, the method further includes, after causing the second electronic device to perform the first operation directed to the second user interface, receiving updated display data from the second electronic device representing one or more updates to an appearance of the second user interface of the second electronic device, and in response to receiving the updated display data, updating a display of a representation of the second user interface based on the updated display data. Additionally or alternatively, in some examples, the second electronic device is displaying the second user interface. Additionally or alternatively, in some examples, the method further includes presenting, in the computer-generated environment via the display, a first affordance associated with a representation of the second electronic device prior to receiving display data from the second electronic device representing a second user interface of the second electronic device, receiving, while presenting the computer-generated environment including the first affordance, an input via one or more input devices corresponding to a selection of the first affordance, and presenting, in the computer-generated environment, a representation of the second user interface of the second electronic device based on the display data in response to receiving the input. Additionally or alternatively, in some examples, the method further includes causing the second electronic device to stop displaying the second user interface in response to receiving the input. Additionally or alternatively, in some examples, the method further includes pairing the first electronic device and the second electronic device such that the first electronic device communicates with the second electronic device prior to displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment.

[0153] Additionally or alternatively, in some examples, the method further includes detecting disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment, and in response to detecting disassociation of the first electronic device from the user, sending a disassociation indication to the second electronic device, the indication causing the second electronic device to display a second user interface. Additionally or alternatively, in some examples, the method further includes detecting reassociation of the first electronic device with a user of the first electronic device, and in response to detecting reassociation of the first electronic device with the user, presenting the computer-generated environment and sending a reassociation indication to the second electronic device, receiving updated display data from the second electronic device while presenting the computer-generated environment representing a current state of the second user interface, and in response to receiving the updated display data, redisplaying a representation of the second user interface in the computer-generated environment based on the current state of the second user interface. Additionally or alternatively, in some examples, the method further includes receiving an invitation to join a communication session with a user of a third electronic device while presenting the computer-generated environment, and in response to receiving the invitation, in accordance with a determination that the invitation has been accepted by the user of the first electronic device, entering into a communication session with the user of the third electronic device and simultaneously displaying an avatar corresponding to the user of the third electronic device and a first representation of the first user interface. Additionally or alternatively, in some examples, the method further includes receiving an indication from the third electronic device while presenting a computer-generated environment including an avatar corresponding to the user of the third electronic device and a first user interface of the application running on the first electronic device, corresponding to a request to share content with the first electronic device, and in response to receiving the indication, in accordance with a determination that the request has been accepted by the user of the first electronic device, displaying in the computer-generated environment a representation corresponding to content of the third user interface configured to be displayed on the third electronic device.

[0154] Some examples of the present disclosure are directed to electronic devices comprising one or more processors, a memory, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0155] Some examples of the present disclosure are directed to a non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform any of the methods described above.

[0156] Some examples of the present disclosure are directed to electronic devices comprising one or more processors, a memory, and means for performing any of the above methods.

[0157] Some examples of the present disclosure are directed to an information processing apparatus for use in an electronic device, the information processing apparatus comprising means for performing any of the above methods.

[0158] The foregoing description has been given for purposes of illustration and with reference to specific embodiments. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments have been chosen and described to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular applications contemplated.

Claims

1. A first electronic device in communication with a display and one or more input devices of the first electronic device, receiving, while presenting via the display a computer-generated environment including a first user interface of an application executing on the first electronic device, display data from a second electronic device representative of a second user interface of the second electronic device; in response to receiving the display data, presenting via the display a representation of the second user interface of the second electronic device based on the display data; Detecting a discrete event corresponding to a user input while presenting the representation of the first user interface of the application and the second user interface of the second electronic device; In response to detecting the individual event, in response to a determination that a gaze of a user of the first electronic device is directed toward the representation of the second user interface of the second electronic device when the respective event is detected, causing the second electronic device to perform a first action directed toward the representation of the second user interface of the second electronic device based on the respective event; In response to determining that the gaze is directed toward the first user interface of the application executing on the first electronic device, performing a second action directed toward the first user interface based on the respective event; and A method comprising:

2. The method of claim 1 , wherein detecting the discrete event comprises detecting an indication of a discrete input received via one or more input devices in communication with the second electronic device.

3. In response to detecting the individual input instruction, and transmitting gaze data relating to a location of the gaze to the second electronic device in response to a determination that the gaze is directed at the representation of the second user interface of the second electronic device when the instruction is detected, the second electronic device performing the first operation in accordance with the gaze data relating to the location of the gaze without transmitting input data corresponding to the individual input to the first electronic device. The method of claim 2.

4. In response to detecting the individual input instruction, in accordance with a determination that the gaze is directed at the representation of the second user interface of the second electronic device when the instruction is detected, transmitting gaze data relating to a location of the gaze to the second electronic device, the second electronic device performing the first operation in accordance with the gaze data relating to the location of the gaze; receiving input data corresponding to the individual input from the second electronic device according to which the first operation is performed at the second electronic device; The method of claim 2 further comprising:

5. In response to detecting the individual input instruction, performing, in accordance with a determination that the gaze is directed toward the first user interface of the application running on the first electronic device when the instruction is detected, the second action directed toward the first user interface in accordance with the individual input without transmitting input data corresponding to the individual input to the second electronic device; The method of claim 2 further comprising:

6. The method of claim 1 , wherein detecting the discrete event comprises detecting the discrete input via the one or more input devices of the first electronic device.

7. In response to detecting the individual input, transmitting input data corresponding to the discrete input to the second electronic device in accordance with a determination that the gaze is directed toward the representation of the second user interface of the second electronic device when the discrete input is detected; The method of claim 6 further comprising:

8. In response to detecting the individual input, performing, in accordance with a determination that the gaze is directed toward the first user interface of the application executing on the first electronic device when the discrete input is detected, the second action directed toward the first user interface in accordance with the discrete input without transmitting input data corresponding to the discrete input to the second electronic device; The method of claim 6 further comprising:

9. 2. The method of claim 1, wherein causing the second electronic device to perform the first action includes transmitting gaze data to the second electronic device including information indicating that the gaze was directed at the representation of the second user interface of the second electronic device when the discrete event was detected.

10. The method of claim 9 , wherein the information includes position information corresponding to a location of the gaze within the representation of the second user interface.

11. The method of claim 1 , wherein the second electronic device is not displaying the second user interface.

12. causing the second electronic device to perform the first action directed to the second user interface includes updating a visual appearance of the second user interface, the method further comprising: receiving, after causing the second electronic device to perform the first action directed to the second user interface, updated display data from the second electronic device representing one or more updates to the visual appearance of the second user interface of the second electronic device; The method of claim 1 , further comprising: in response to receiving the updated display data, updating a display of the representation of the second user interface based on the updated display data.

13. 13. A method, the second electronic device displaying the second user interface, the method comprising: presenting, via the display, in the computer-generated environment a first affordance associated with a representation of the second electronic device prior to receiving, from the second electronic device, the display data representing the second user interface of the second electronic device; receiving, via the one or more input devices, input corresponding to a selection of the first affordance while presenting the computer-generated environment including the first affordance; 10. The method of claim 1 , further comprising: in response to receiving the input, presenting, in the computer-generated environment, the representation of the second user interface of the second electronic device based on the display data.

14. in response to receiving the input, causing the second electronic device to stop displaying a second user interface; The method of claim 13 further comprising:

15. pairing the first electronic device with the second electronic device such that the first electronic device is in communication with the second electronic device prior to displaying the first affordance associated with the representation of the second electronic device in the computer-generated environment; The method of claim 13 further comprising:

16. detecting a disassociation of the first electronic device from a user of the first electronic device while presenting the computer-generated environment; in response to detecting a disassociation of the first electronic device from the user; sending to the second electronic device an indication of the disassociation, the indication causing the second electronic device to display the second user interface; The method of claim 1 further comprising:

17. Detecting a re-association of the first electronic device with the user of the first electronic device; in response to detecting the re-association of the first electronic device with the user, presenting the computer-generated environment; and sending an indication of the reassociation to the second electronic device; receiving updated display data from the second electronic device while presenting the computer-generated environment, the updated display data representing a current state of the second user interface; In response to receiving the updated display data, redisplaying the representation of the second user interface in the computer-generated environment based on the current state of the second user interface; and The method of claim 16 further comprising:

18. receiving an invitation to participate in a communications session with a user of a third electronic device while presenting the computer-generated environment; In response to receiving the invitation, entering the communication session with the user of the third electronic device, including simultaneously displaying an avatar corresponding to the user of the third electronic device and the first user interface in accordance with a determination that the invitation has been accepted by the user of the first electronic device; The method of claim 1 further comprising:

19. 1. An electronic device comprising: one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 18.

20. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform a method according to any one of claims 1 to 18.

Images