Method for manipulating objects in environment

Abstract

The invention relates to a method for manipulating objects in an environment. The method includes: displaying a three-dimensional environment via a display generation component; receiving a mobile input via one or more input devices; and in response to receiving the movement input: in accordance with a determination that the movement input corresponds to a request to move the representation of the first application from the first position to a third position in the three-dimensional environment different from the first position: displaying the representation of the first application at the third position in the three-dimensional environment via a display generation component; and updating an orientation of the representation of the first application from the first orientation to a second orientation, different from the first orientation, relative to the three-dimensional environment based on the third location; and in accordance with a determination that the mobile input corresponds to a request to associate a fourth location different from the second location with the first user, maintaining display of the representation of the first application at the first location in the three-dimensional environment without updating an orientation of the representation of the first application relative to the three-dimensional environment.

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of September 17, 2021, national application number 202180066188.8, and invention name “Method for Manipulating Objects in an Environment”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 083,720, filed September 25, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present invention generally relates to a computer system having a display generation component and one or more input devices for presenting a graphical user interface, including but not limited to an electronic device that presents a three-dimensional environment including virtual objects via the display generation component. Background Art

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices for computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects (including digital images, videos, text, icons, control elements (such as buttons), and other graphics).

[0006] However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems where virtual object manipulation is complex, cumbersome, and error-prone can place a significant cognitive burden on users and detract from the experience of the virtual / augmented reality environment. Furthermore, these methods take longer than necessary, wasting energy. This latter consideration is particularly important in battery-powered devices. Summary of the Invention

[0007] Therefore, there is a need for computer systems with improved methods and interfaces for providing computer-generated experiences to users, thereby making user interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces optionally supplement or replace conventional methods for providing computer-generated reality experiences to users. Such methods and interfaces reduce the amount, extent, and / or nature of input from the user by helping the user understand the connection between the input provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0008] The disclosed system reduces or eliminates the above-mentioned defects and other problems associated with user interfaces for computer systems having display generation components and one or more input devices. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to the display generation component, the computer system also has one or more output devices, including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contacts and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI or the user's body (as captured by a camera and other motion sensors), and voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with objects in a three-dimensional environment. Such methods and interfaces can supplement or replace conventional methods for interacting with objects in a three-dimensional environment. Such methods and interfaces reduce the amount, extent, and / or nature of input from a user and produce a more efficient human-computer interface.

[0010] In some embodiments, the electronic device automatically updates the orientation of a virtual object in a three-dimensional environment based on the user's viewpoint in the three-dimensional environment. In some embodiments, the electronic device automatically updates the orientation of a virtual object in the three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment. In some embodiments, the electronic device modifies the appearance of a real object between the virtual object and the user's viewpoint in the three-dimensional environment. In some embodiments, the electronic device automatically selects the user's position in a three-dimensional environment that includes one or more virtual objects and / or other users.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described in this specification are not comprehensive. In particular, many additional features and advantages will be apparent to those skilled in the art from the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification has been selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0013] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing a CGR experience according to some embodiments.

[0014] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience according to some embodiments.

[0015] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of a CGR experience to a user according to some embodiments.

[0016] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0017] Figure 5is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input from a user according to some embodiments.

[0018] Figure 6 is a flowchart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0019] 7A to 7C Examples are shown of how an electronic device according to some embodiments automatically updates the orientation of a virtual object in a three-dimensional environment based on a user's viewpoint in the three-dimensional environment.

[0020] Figures 8A to 8P is a flowchart illustrating a method for automatically updating the orientation of a virtual object in a three-dimensional environment based on a user's viewpoint, according to some embodiments.

[0021] 9A to 9D Examples are shown of how an electronic device according to some embodiments automatically updates the orientation of a virtual object in a three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment.

[0022] 10A to 10T is a flowchart illustrating a method for automatically updating the orientation of a virtual object in a three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment, according to some embodiments.

[0023] Figures 11A to 11C Examples are shown of how an electronic device according to some embodiments modifies the appearance of a real object between a virtual object and a user's viewpoint in a three-dimensional environment.

[0024] 12A to 12H is a flow chart illustrating a method of modifying the appearance of a real object between a virtual object and a user's viewpoint in a three-dimensional environment, according to some embodiments.

[0025] 13A to 13C Examples are shown of how an electronic device according to some embodiments automatically selects a user's position in a three-dimensional environment that includes one or more virtual objects and / or other users.

[0026] Figures 14A to 14L is a flowchart illustrating a method of automatically selecting a user's position in a three-dimensional environment including one or more virtual objects and / or other users, according to some embodiments. DETAILED DESCRIPTION

[0027] According to some embodiments, the present disclosure relates to a user interface for providing a computer-generated reality (CGR) experience to a user.

[0028] The systems, methods, and GUIs described herein provide improved ways for electronic devices to interact with and manipulate objects in a three-dimensional environment. The three-dimensional environment optionally includes one or more virtual objects, one or more representations of real objects located in the physical environment of the electronic device (e.g., displayed as image-realistic (e.g., "see-through") representations of the real objects, or visible to the user through transparent portions of display generation components), and / or a representation of a user in the three-dimensional environment.

[0029] In some embodiments, the electronic device automatically updates the orientation of the virtual object in the three-dimensional environment based on the user's viewpoint in the three-dimensional environment. In some embodiments, the electronic device moves the virtual object according to the user input and displays the object at the updated position in response to the termination of the user input. In some embodiments, the electronic device automatically updates the orientation of the virtual object at the updated position (e.g., and / or when the virtual object moves to the updated position) so that the virtual object is oriented toward the user's viewpoint in the three-dimensional environment (e.g., throughout its movement and / or at the end of its movement). Automatically updating the orientation of the virtual object in the three-dimensional environment enables the user to view and interact with the virtual object more naturally and effectively, without the need for the user to manually adjust the orientation of the object.

[0030] In some embodiments, the electronic device automatically updates the orientation of the virtual object in the three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment. In some embodiments, the electronic device moves the virtual object according to the user input and displays the object at the updated position in response to the termination of the user input. In some embodiments, the electronic device automatically updates the orientation of the virtual object at the updated position (e.g., and / or when the virtual object moves to the updated position) so that the virtual object is oriented toward the viewpoints of the multiple users in the three-dimensional environment (e.g., throughout its movement and / or at the end of its movement). Automatically updating the orientation of the virtual object in the three-dimensional environment enables the user to view and interact with the virtual object more naturally and effectively, without the need for the user to manually adjust the orientation of the object.

[0031] In some embodiments, the electronic device modifies the appearance of a real object between the virtual object and the user's viewpoint in the three-dimensional environment. The electronic device optionally blurs, darkens, or otherwise modifies a portion of the real object between the user's viewpoint in the three-dimensional environment and the virtual object (e.g., displayed as an image-realistic (e.g., "see-through") representation of the real object, or visible to the user through a transparent portion of a display generating component). In some embodiments, the electronic device modifies the portion of the real object that is within a threshold distance (e.g., 5 cm, 10 cm, 30 cm, 50 cm, 100 cm, etc.) of the boundary of the virtual object, without modifying the portion of the real object that is more than the threshold distance from the boundary of the virtual object. Modifying the appearance of the real object allows the user to view and interact with the virtual object more naturally and efficiently. In addition, modifying the appearance of the real object reduces the cognitive burden on the user.

[0032] In some embodiments, the electronic device automatically selects the position of the user in a three-dimensional environment that includes one or more virtual objects and / or other users. In some embodiments, the user obtains access to a three-dimensional environment that already includes one or more other users and one or more virtual objects. In some embodiments, the electronic device automatically selects a position to be associated with the user (e.g., a position to place the user's viewpoint) based on the position and orientation of the virtual objects and other users in the three-dimensional environment. In some embodiments, the electronic device will select the user's position so that the user can view other users and virtual objects in the three-dimensional environment without blocking other users' view of the user and virtual objects. Automatically placing the user in the three-dimensional environment based on the position and orientation of the virtual objects and other users in the three-dimensional environment enables the user to effectively view and interact with virtual objects and other users in the three-dimensional environment, without requiring the user to manually select a position in the three-dimensional environment to be associated with it.

[0033] Figures 1 to 6 A description of an exemplary computer system for providing a CGR experience to a user, such as described below with respect to methods 800, 1000, 1200, and 1400, is provided. In some embodiments, Figure 1As shown in FIG, a CGR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted device or a handheld device).

[0034] In describing a CGR experience, various terms are used to distinctly refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the CGR experience, which inputs cause the computer system generating the CGR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0035] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

[0036] Computer-generated reality: In contrast, a computer-generated reality (CGR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In CGR, a subset of a person's physical movements, or representations thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the CGR environment are adjusted in a manner consistent with at least one law of physics. For example, a CGR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in a CGR environment can be made in response to representations of physical movement (e.g., voice commands). People can sense and / or interact with CGR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, a person can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency, which selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some CGR environments, people can sense and / or interact only with audio objects.

[0037] Examples of CGR include virtual reality and mixed reality.

[0038] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of the person's presence within the computer-generated environment and / or through the simulation of a subset of the person's physical movement within the computer-generated environment.

[0039] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause movement so that virtual trees appear stationary relative to the physical ground.

[0040] Examples of mixed reality include augmented reality and augmented virtuality.

[0041] Augmented Reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with the virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as holograms or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0042] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but people's faces are realistically reproduced from images taken of physical people. In another example, a virtual object can adopt the shape or color of a physical object imaged by one or more imaging sensors. In another example, a virtual object can adopt a shadow that conforms to the position of the sun in the physical environment.

[0043] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection technology that projects graphic images onto a person's retina. The projection system may also be configured to project virtual objects into a physical environment, for example as a hologram, or onto a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, the controller 110 comprises a suitable combination of software, firmware, and / or hardware. References below Figure 2Controller 110 is described in more detail. In some embodiments, controller 110 is a computing device that is located locally or remotely relative to scene 105 (e.g., physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside of scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of the input devices 125, one or more output devices of the output devices 155, one or more sensors of the sensors 190, and / or one or more peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0044] In some embodiments, the display generation component 120 is configured to provide a CGR experience (e.g., at least the visual component of the CGR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0045] According to some embodiments, display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within scene 105 .

[0046] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present CGR content, and the user holds the device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR room, housing, or room configured to present CGR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) can be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface that illustrates interactions with CGR content that are triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device can similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and the responses to the CGR content are displayed via the HMD. Similarly, a user interface that illustrates interactions with CGR content that are triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) can similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0047] Despite Figure 1 Relevant features of the operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

[0048] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0049] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a trackpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0050] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and a CGR experience module 240.

[0051] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR experience module 240 is configured to manage and coordinate single or multiple CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for corresponding groups of one or more users). To this end, in various embodiments, the CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0052] In some embodiments, the data acquisition unit 242 is configured to Figure 1 1 and / or peripherals 195. The data acquisition unit 242 may be configured to acquire data (e.g., presentation data, interaction data, sensor data, position data, etc.) from at least the display generation component 120 of the display generation component 120, and optionally from one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0053] In some embodiments, the tracking unit 244 is configured to map the scene 105 and track at least the display generation component 120 relative to the scene 105. Figure 1 105, and optionally tracks the position of one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For this purpose, in various embodiments, the tracking unit 244 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 244 includes a hand tracking unit 243 and / or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the position of one or more parts of the user's hand and / or the position of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to the coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 243 is described in more detail. In some embodiments, the eye tracking unit 245 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to CGR content displayed via the display generation component 120. Figure 5 The eye tracking unit 245 is described in more detail.

[0054] In some embodiments, the coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by the display generation component 120, and optionally by one or more of the output devices 155 and / or peripheral devices 195. To this end, in various embodiments, the coordination unit 246 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0055] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, position data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transmission unit 248 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0056] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0057] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how features are distributed among them will vary depending on the specific implementation and, in some embodiments, will depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0058] Figure 3is a block diagram of an example of a display generation component 120 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For the purposes described, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0059] In some embodiments, the one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, the one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0060] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to the user. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, HMD 120 includes a single CGR display. For another example, HMD 120 includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR and VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content.

[0061] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see in the absence of the HMD 120 (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), one or more infrared (IR) cameras, and / or one or more event-based cameras, among others.

[0062] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR rendering module 340.

[0063] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. For this purpose, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transfer unit 348.

[0064] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0065] In some embodiments, the CGR rendering unit 344 is configured to render CGR content via one or more CGR displays 312. To this end, in various embodiments, the CGR rendering unit 344 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0066] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate a computer-generated reality) based on the media content data. For this purpose, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0067] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0068] Although the data acquisition unit 342, CGR rendering unit 344, CGR map generation unit 346, and data transmission unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR rendering unit 344, the CGR map generation unit 346 and the data transmission unit 348 may be located in a separate computing device.

[0069] also, Figure 3 It serves more as a functional description of various features that may be present in a particular embodiment, rather than as a structural schematic diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how features are distributed among them will vary depending on the specific implementation and, in some embodiments, will depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0070] Figure 4 is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 243 ( Figure 2 ) to track the position of one or more parts of the user's hand, and / or one or more parts of the user's hand relative to Figure 1The hand tracking device 140 is configured to monitor movement of the scene 105 relative to the user's surroundings (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hands). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0071] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information, including at least a human user's hand 406. The image sensor 404 captures images of the hand at a sufficient resolution to enable the fingers and their respective positions to be distinguished. The image sensor 404 typically captures images of other parts of the user's body, or may also capture images of all parts of the body, and may have zoom capabilities or specialized sensors with increased magnification to capture images of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or serves as an image sensor for capturing the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment in such a way that the field of view of the image sensor 404, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are treated as input to the controller 110.

[0072] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and possibly color image data in addition) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application program interface (API) to an application running on the controller, which in turn drives the display generation component 120. For example, a user can interact with the software running on the controller 110 by moving his hand 408 and changing his hand posture.

[0073] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offsets of the spots in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of a point in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of a point in the scene correspond to the z component measured by the image sensor. Alternatively, the hand tracking device 440 may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on a single or multiple cameras or other types of sensors.

[0074] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand from these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's hand joints and fingertips.

[0075] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0076] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on tangible, non-transitory media such as optical, magnetic, or electronic memory media. In some embodiments, the database 408 is also stored in memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 44, but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the hand tracking device 402 or other devices associated with the image sensor 404, for example. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or integrated with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0077] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where shades of gray become darker with increasing depth. The controller 110 processes these depth values ​​in order to identify and segment components of the image (i.e., a group of adjacent pixels) that have characteristics of a human hand. These characteristics may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0078] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 , a skeleton 414 is superimposed on a hand background 416 that has been segmented from the original depth map. In some embodiments, key feature points of the hand and, optionally, on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, the center of the palm, the end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine a gesture performed by the hand or the current state of the hand according to some embodiments.

[0079] Figure 5 An eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is comprised of an eye tracking unit 245 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the CGR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmet, goggles or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating CGR content for the user to view and components for tracking the user's gaze relative to the CGR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or a CGR room, the eye tracking device 130 is optionally a device separate from the handheld device or the CGR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of the head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally used in conjunction with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally part of a non-head-mounted display generation component.

[0080] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display and display virtual objects on the transparent or translucent display, through which the user can directly view the physical environment. In some embodiments, the display generation component projects the virtual objects into the physical environment. The virtual objects may, for example, be projected onto a physical surface or projected as a hologram, so that the individual using the system observes the virtual objects superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0081] like Figure 5As shown in , in some embodiments, the gaze tracking device 130 includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye-tracking camera can be pointed at the user's eyes to receive IR or NIR light that the light source reflects directly from the eyes, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect IR or NIR light from the eyes to the eye-tracking camera while allowing visible light to pass through. The gaze tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, both eyes of the user are tracked separately by corresponding eye-tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by corresponding eye-tracking camera and illumination source.

[0082] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometry and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process can include an estimation of eye parameters of a specific user, such as pupil position, fovea position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0083] like Figure 5As shown in FIG, an eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system that includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 may be directed toward a mirror 550 (which reflects the IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.). Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 (as shown in the bottom portion of the ).

[0084] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0085] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal region, determined based on the user's current gaze direction, than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content within the view based at least in part on the user's current gaze direction. As another example use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of a CGR experience to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment on the display 510 that the user is currently looking at. As another example use case, the eye lens 520 may be a focusable lens, and the controller may use gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object the user is currently looking at has the appropriate vergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0086] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light sources can be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 can be used, and other arrangements and positions of the light sources 530 can be used.

[0087] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0088] like Figure 5 Embodiments of the gaze tracking system shown in can, for example, be used in computer-generated reality, virtual reality, and / or mixed reality applications to provide a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience to a user.

[0089] Figure 6 A flash-assisted gaze tracking pipeline according to some embodiments is shown. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and glint in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0090] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0091] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0092] At 640, if proceeding from element 410, the current frame is analyzed to track the pupil and glint based in part on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupil and glint detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result can be trusted. For example, the result can be checked to determine whether the pupil and a sufficient number of glints were successfully tracked or detected in the current frame to perform gaze estimation. At 650, if the result is not likely to be trusted, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is trustworthy, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0093] Figure 6 This is intended to be used as an example of an eye tracking technology that may be used for a particular implementation. As one of ordinary skill in the art will recognize, according to various embodiments, other eye tracking technologies currently existing or developed in the future may be used in place of or in combination with the flash-assisted eye tracking technology described herein in the computer system 101 for providing a CGR experience to a user.

[0094] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., a CGR environment) that includes representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively via a camera and display of an electronic device, or passively via a transparent or translucent display of the electronic device). As previously described, the three-dimensional environment is optionally a mixed reality system in which the three-dimensional environment is based on a physical environment captured by one or more sensors of the device and displayed via a display generation component. As a mixed reality system, the device is optionally capable of selectively displaying portions and / or objects of the physical environment so that the corresponding portions and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the electronic device. Similarly, the device is optionally capable of displaying virtual objects in the three-dimensional environment so that they appear as if the virtual objects exist in the real world (e.g., a physical environment) by placing the virtual objects at corresponding locations in the three-dimensional environment that have corresponding locations in the real world. For example, the device optionally displays a vase so that it appears as if a real vase is placed on top of a table in the physical environment. In some embodiments, each location in the three-dimensional environment has a corresponding location in the physical environment. Thus, when a device is described as displaying a virtual object at a corresponding location relative to a physical object (e.g., such as at or near a user's hand or at or near a physical table), the device displays the virtual object at a particular location in the three-dimensional environment such that the virtual object appears as if the virtual object is at or near the physical object in the physical world (e.g., the virtual object is displayed at a location in the three-dimensional environment that corresponds to the location in the physical environment where the virtual object would be displayed if the virtual object were a real object at that particular location).

[0095] In some embodiments, real-world objects that exist in the physical environment and are displayed in the three-dimensional environment can interact with virtual objects that exist only in the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment and the vase is a virtual object.

[0096] Similarly, the user is optionally able to use one or more hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the device optionally capture one or more hands of the user and display representations of the user's hands in the three-dimensional environment (e.g., in a manner similar to the above-described display of real-world objects in the three-dimensional environment), or in some embodiments, due to the transparency / translucency of the portion of the display generation component that is displaying the user interface or the projection of the user interface onto a transparent / translucent surface or the projection of the user interface onto the user's eyes or into the field of view of the user's eyes, the user's hands are visible via the display generation component via the ability to see the physical environment through the user interface. Thus, in some embodiments, the user's hands are displayed at corresponding locations in the three-dimensional environment and are treated as if they were objects in the three-dimensional environment, which can interact with virtual objects in the three-dimensional environment as if the virtual objects were real physical objects in the physical environment. In some embodiments, the user is able to move his or her hands so that the representation of the hands in the three-dimensional environment moves in conjunction with the movement of the user's hands.

[0097] In some embodiments described below, the device is optionally capable of determining an "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is interacting with the virtual object (e.g., whether the hand is touching, grabbing, holding, etc., or is within a threshold distance from the virtual object). For example, the device determines the distance between the user's hand and the virtual object when determining whether the user is interacting with the virtual object and / or how the user is interacting with the virtual object. In some embodiments, the device determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, one or more hands of the user are located at a specific location in the physical world, and the device optionally captures the one or more hands of the user and displays the one or more hands of the user at a specific corresponding location in the three-dimensional environment (e.g., the location of the hand will be displayed in the three-dimensional environment if the hand is a virtual hand rather than a physical hand). The position of the hand in the three-dimensional environment is optionally compared with the position of the virtual object of interest in the three-dimensional environment to determine the distance between the one or more hands of the user and the virtual object. In some embodiments, the device optionally determines the distance between a physical object and a virtual object by comparing positions in the physical world (e.g., rather than comparing positions in a three-dimensional environment). For example, when determining the distance between one or more hands of a user and a virtual object, the device optionally determines the corresponding position of the virtual object in the physical world (e.g., the position where the virtual object would be located in the physical world if the virtual object were a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and the one or more hands of the user. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the device optionally performs any of the above techniques to map the position of the physical object to the three-dimensional environment and / or map the position of the virtual object to the physical world.

[0098] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed at and / or where and what the physical stylus held by the user is pointed at. For example, if the user's gaze is directed at a particular location in the physical environment, the device optionally determines the corresponding location in the three-dimensional environment, and if a virtual object is located at the corresponding virtual location, the device optionally determines that the user's gaze is directed at the virtual object. Similarly, the device is optionally able to determine the location in the physical world that the stylus is pointing to based on the orientation of the physical stylus. In some embodiments, based on this determination, the device determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical world that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0099] Similarly, the embodiments described herein may relate to the position of a user (e.g., a user of a device) in a three-dimensional environment and / or the position of a device in a three-dimensional environment. In some embodiments, the user of the device is holding, wearing, or otherwise located at or near the electronic device. Thus, in some embodiments, the position of the device is used as a proxy for the user's position. In some embodiments, the position of the device and / or user in the physical environment corresponds to a corresponding position in the three-dimensional environment. In some embodiments, the corresponding position is the position from which the "camera" or "field of view" of the three-dimensional environment extends. For example, the position of the device will be a position in the physical environment (and the corresponding position of the device in the three-dimensional environment) that, if the user were standing at that position facing the corresponding portion of the physical environment displayed by the display generation component, would see objects in the physical environment at the same position, orientation, and / or size (e.g., in absolute terms and / or relative to each other) as when they were displayed by the display generation component of the device. Similarly, if the virtual objects displayed in the three-dimensional environment are physical objects in the physical environment (e.g., placed at the same location in the physical environment as they are in the three-dimensional environment, and have the same size and orientation in the physical environment as they are in the three-dimensional environment), then the position of the device and / or user is the position at which the user will see the virtual objects in the physical environment in the same location, orientation and / or size (e.g., in an absolute sense and / or relative to each other and real-world objects) as they are displayed by the display generation components of the device.

[0100] In this disclosure, various input methods are described with respect to interaction with a computer system. When one input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with and optionally utilizes the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. When one output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with and optionally utilizes the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment through a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, the present disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0101] In addition, in the method described herein where one or more steps depend on having met one or more conditions, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions of the steps in the method of determining the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then those of ordinary skill will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on having met one or more conditions can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require a system or computer-readable medium to declare that the system or computer-readable medium includes instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether a possible situation has been met without explicitly repeating the steps of the method until all conditions of the steps in the method of determining the method have been met. Those of ordinary skill in the art will also understand that, similar to the method with a contingent step, a system or computer-readable storage medium can repeat the steps of the method as needed multiple times to ensure that all contingent steps have been performed.

[0102] User interface and associated processes

[0103] Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system (such as a portable multifunction device or a head-mounted device) having display generating components, one or more input devices, and (optionally) one or more cameras.

[0104] 7A to 7C Examples are shown of how an electronic device according to some embodiments automatically updates the orientation of a virtual object in a three-dimensional environment based on a user's viewpoint in the three-dimensional environment.

[0105] Figure 7A The electronic device 101 displays a three-dimensional environment 704 on the user interface via the display generation component 120. Figures 1 to 6 As described, the electronic device 101 optionally includes a display generation component 120 (e.g., a touch screen) and a plurality of image sensors 314. The image sensor optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the electronic device 101 will be able to use to capture one or more images of the user or a portion of the user when the user interacts with the electronic device 101. In some embodiments, the display generation component 120 is a touch screen that can detect gestures and movements of the user's hands. In some embodiments, the user interface shown below can also be implemented on a head-mounted display that includes a display generation component that displays the user interface to the user and sensors that detect the physical environment and / or movements of the user's hands (e.g., external sensors facing outward relative to the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0106] like Figure 7A , a three-dimensional environment 704 includes a plurality of virtual objects, including a representation 706 of application A, a representation 708 of application B, and a virtual object 710. In some embodiments, the three-dimensional environment 704 is presented relative to a fixed reference frame (e.g., gravity, one or more objects in the physical environment of the electronic device 101). In some embodiments, the three-dimensional environment 704 is a three-dimensional environment based on and / or includes representations of objects in the physical environment of the electronic device, and the reference frame relative to which the three-dimensional environment 704 is presented is the physical environment of the electronic device 101 or one or more objects in the physical environment. Figure 7A Included is a legend 712 depicting a bird's-eye view of the three-dimensional environment 704. The legend 712 shows the position and orientation of the representation 706 of application A, the representation 708 of application B, the virtual object 710, and the user 714 in the three-dimensional environment. Figure 7A The electronic device 101 is also shown displaying the three-dimensional environment from the user's viewpoint in the three-dimensional environment 704 .

[0107] As shown in legend 712, representation 706 of application A and representation 708 of application B have orientations in three-dimensional environment 704 that are tilted toward user 714. For example, representation 706 of application A is to the left of user 714, and therefore, the orientation of representation 706 of application A is tilted to the right, toward user 714. For another example, representation 708 of application B is to the right of user 714, and therefore, the orientation of representation 708 of application B is tilted to the left, toward user 714. Therefore, the orientations of representation 706 of application A and representation 708 of application B are different from each other. In some embodiments, a representation of an application includes content of the application on one side of the representation. For example, the side of representation 706 of application A that faces user 714 in three-dimensional environment 704 includes content of the user interface of application A. Similarly, for example, the side of representation 708 of application B that faces user 714 in three-dimensional environment 704 includes content of the user interface of application B.

[0108] In some embodiments, the electronic device 101 automatically orients the representation of the application toward the user 714 in the three-dimensional environment 712 (e.g., in response to input that displays the representation of the application, after moving the representation of the application in response to user input, etc.). In some embodiments, automatically orienting the representation of the application toward the user includes tilting the representation toward the user and positioning the side of the representation that includes the content of the representation toward the user. In some embodiments, the electronic device 101 constructs a vector field superimposed on the three-dimensional environment 704 and / or has access to the vector field, wherein the vector points to the position of the user 714 in the three-dimensional environment 704, and orients the virtual object (e.g., the representation of the application, etc.) according to the vector field. The results of this orientation setting are described below.

[0109] For example, in Figure 7A , electronic device 101 (e.g., via eye tracking device 314) detects a user's gaze 716 on representation 706 of application A. In response to detecting the user's gaze on representation 706 of application A, electronic device 101 displays a user interface element 718 indicating that the user's gaze was detected on representation 706 of application A. Figure 7A User interface element 718 is shown as an outline around representation 706 of application A, but it should be understood that in some embodiments, user interface element 718 has a different shape / appearance / position. For example, in response to detecting the user's gaze on representation 706 of application A, electronic device 101 optionally displays a user interface element, such as a bar or line along one edge of representation 706 of application A.

[0110] When the user's gaze 716 is detected on the representation 706 of application A, the electronic device 101 detects (e.g., via a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., trackpad or touch screen)) that the user is performing a gesture with their hand 720. For example, the gesture is the user touching their thumb to another finger on the hand (e.g., index finger, middle finger, ring finger, pinky finger). In response to detecting the user's gaze 716 on the representation 706 of application A while detecting the gesture being performed with the user's hand 720, the electronic device 101 initiates a process of moving the representation 706 in the three-dimensional environment according to the movement of the user's hand 720 while the gesture is maintained. As Figure 7A As shown in , the user moves their hand 720 towards the user's body and to the right while keeping their other fingers touching their thumb.

[0111] In response to Figure 7A 7 (eg, toward user 714 and to the right), the electronic device moves the representation 706 of application A in the three-dimensional environment 704 in accordance with the movement of the hand 720 shown in FIG. Figure 7B As shown in .

[0112] Figure 7B Shown including according to Figure 7A 704 of a representation 706 of application A at an updated position (e.g., closer to user 704 and now positioned between object 710 and representation 708 of application B, slightly to the right of user 714) in response to the user input shown in FIG. Figure 7A In addition to updating the position of the representation 706 of application A based on the movement of the user's hand 720 in the application, the electronic device 101 also updates the orientation of the representation 706 of application A. Figure 7A As shown in FIG, the electronic device 101 detects the user input while the representation 706 of application A is tilted to the right so as to be oriented toward the user. Figure 7B , in response to moving the representation 706 of application A to a position slightly to the right of the user 714 in the three-dimensional environment 704, the electronic device 101 updates the orientation of the representation 706 of application A to be tilted slightly to the left so as to continue to be oriented toward the user 714 in the three-dimensional environment. The representation 706 of application A continues to be oriented so that the side of the representation 706 facing the user 714 in the three-dimensional environment 704 is the side of the representation 706 that includes the content of the user interface of application A. In some embodiments, the orientation of the representation 706 of application A is defined by a vector field that points to the user 714 in the three-dimensional environment 704. For example, the representation 706 of application A in Figure 7BThe orientation of the three-dimensional environment 704 is in the representation 706 of the application A located at Figure 7B In some embodiments, the orientation of the representation 706 of application A is continuously updated according to the vector field as the representation 706 of application A moves in the three-dimensional environment 704.

[0113] In addition to changing the orientation and position of the representation 706 of application A, the electronic device 101 optionally also updates the size of the representation 706 of application A in response to user input that moves the representation 706 of application A in the three-dimensional environment. Figure 7A As shown in , when the electronic device 101 detects the input of the representation 706 of the mobile application A, the representation 706 of the application A is at a corresponding distance from the user 714 in the three-dimensional environment 704, as shown in the legend 712. Figure 7B As shown in the legend 712 of FIG, the updated position of the representation 706 of application A is closer to the user 714 in the three-dimensional environment 704 than the position of the representation 706 of application A before the electronic device 101 moved the representation 706 of application A. Although the representation 706 of application A appears to be displayed by the electronic device 101 at the same or similar size (e.g., via the display generation component 120) before and after the input to move the representation 706 of application A is detected, as shown in the legend 712, the size of the representation 706 is optionally reduced in response to moving the representation 706 closer to the user. Similarly, in some embodiments, if the user 714 were to move the representation 706 of application A further away from the user 714 in the three-dimensional environment 704, the electronic device 101 will optionally increase the size of the representation 706. In this way, the electronic device 101 is able to maintain the amount of display area of ​​the display generation component 120 used to display the representation 706 even if the distance of the representation 706 to the user 714 in the three-dimensional environment 704 changes.

[0114] In some embodiments, as will be referred to below 8A to 13C As described in more detail, the three-dimensional environment 704 includes multiple users. For example, the three-dimensional environment 704 includes users other than the user 714. In some embodiments, one or more objects (e.g., object 710, representation 706, representation 708) are accessible to only one of the users in the three-dimensional environment 704 (e.g., user 714 or one of the other users). In some embodiments, an object accessible only to one user is displayed only by the user's electronic device 101, and only the user can provide user input for the object. For example, an object accessible only to the user 714 is displayed by the user's electronic device 101. 7A to 7B , but not displayed by other users' electronic devices. 7A to 7BThe electronic device 101 shown in does not display objects that are accessible only to other users.

[0115] In some embodiments, the orientation of a virtual object is updated based on the location of the user who has access to the object and not based on the locations of other users (e.g., when initially displayed in the three-dimensional environment 704, in response to a request to move the object, etc.). For example, if the three-dimensional environment 704 includes additional users who do not have access to the representation 706 of application A, the electronic device 101 will update the virtual object in the same manner as described above with reference to FIG. 7A to 7B The same manner is described for updating the orientation of representation 706 of application A. As another example, if three-dimensional environment 704 includes another user who has access to representation 706 of application A and user 714 does not have access to representation 706 of application A, then when the position of representation 706 of application A is updated, the orientation of representation 706 of application A will be updated based on the position of the other user rather than the position of user 714.

[0116] The following reference 8A to 9D As described in more detail, if an object is accessible to multiple (e.g., all) users in the three-dimensional environment 704, the electronic device 101 optionally updates the orientation of the object based on the locations of multiple (e.g., all) users who have access to the object (e.g., when initially displayed in the three-dimensional environment 704, in response to a request to move the object, etc.). For example, if the three-dimensional environment 704 includes additional users, then in response to input to move the representation 706 of application A, the electronic device 101 will update the orientation of the representation 706 of application A based on both the location of user A 714 and the location of the other user. Thus, in this example, even if the representation 706 of application A is moved to Figure 7B The orientation of the representation 706 of application A may also be different from the position shown in FIG. Figure 7B In some embodiments, an object (e.g., a virtual television, a user interface for an application, etc.) includes content on only one side of the object. 9A to 9D 706, 708, and 724) are oriented so that the side with the content faces multiple (e.g., all) users in the three-dimensional environment 704. As another example, an object (e.g., a virtual object, a virtual board game, such as 9A to 9D The representation 922 in is oriented between users, as will be referenced 9A to 10T Described in more detail.

[0117] In some embodiments, in response to input that increases the number of users with access to the virtual object (e.g., updating the virtual object from being accessible to only one user to being accessible to multiple or all users in the three-dimensional environment), the electronic device makes the object evaluable to the increased number of users and updates one or more of the orientation, size, and position of the object based on the positions of the users with access to the object. For example, if the three-dimensional environment 704 includes multiple users with access to the representation 706, the orientation and size of the representation 706 will be based on Figure 7B In some embodiments, the orientation of an object is updated to be oriented toward the multiple users with access to the object (e.g., according to a vector field based on the positions of all users with access to the object). In some embodiments, the size of an object is updated based on the distance between the object and the users with access to the object to ensure that the object is legible to all users with access to the object (e.g., the size is at least a minimum size based on the distance to the user farthest from the object). In some embodiments, a position (e.g., an initial position in response to initiating a request for display of an object accessible to multiple users) is selected based on the positions of the multiple users with access to the object so that all users are able to view the object.

[0118] In some embodiments, if the object is shared after being prompted by the electronic device 101 to share with other users, the electronic device 101 only updates the orientation, size, and / or position of the object (e.g., in response to a request to move the object, such as Figure 7A In some embodiments, the electronic device 101 does not update the position, size, or orientation of the object in response to an increase in the number of users having access to the object. For example, if the electronic device 101 displays Figure 7A When the electronic device 101 receives an instruction from another user to join the three-dimensional environment 704, the electronic device 101 continues to use the three-dimensional environment 704 even if the other user has access to the representation 706 of the application A. Figure 7A The representation 706 of application A is displayed at the size, position, and orientation shown in FIG. 7 until a request to move the representation 706 is received, such as by Figure 7A In some embodiments, once an object is accessible to multiple users, in response to an input to move the object (such as moving Figure 7A 706 in the representation), the electronic device 101 moves the object based on the input and updates the size and / or orientation of the object based on the positions of all users with access to the object to maintain the legibility of the object for users with access to the object.

[0119] Therefore, if 7A to 7B As shown in , in some embodiments, in response to user input that updates the position of the application's representation 706 in the three-dimensional environment 704, the electronic device 101 automatically updates the orientation and / or size of the application's representation 706. In some embodiments, the electronic device 101 will similarly update the orientation and / or size of the representation 708 and object 710 in response to a request to move any of these virtual objects. In some embodiments, as will be described below, Figure 7C As described in further detail, in response to the user 714 updating the position of the user 714 in the three-dimensional environment 704, the electronic device 101 forgoes updating the orientation and / or size of the representation of the application (e.g., and / or other virtual objects in the three-dimensional environment 704).

[0120] Figure 7C Shows that user 714 has been Figure 7B In some embodiments, the electronic device updates the position of the user 714 in the three-dimensional environment 704 after the position of the user 714 in the three-dimensional environment 704 is updated. In some embodiments, the electronic device updates the position of the user 714 in response to detecting (e.g., via one or more gyroscopes, accelerometers, cameras, depth sensors, etc.) movement of the electronic device 101 in the physical environment of the electronic device 101. For example, the electronic device 101 updates the position of the user 714 in the three-dimensional environment 704 by moving the user 714 in the direction of the detected movement of the electronic device 101. In some embodiments, the electronic device 101 updates the position of the user 714 in response to user input requesting movement of the position of the user 714 in the three-dimensional environment 704.

[0121] User 714 is Figure 7C The position in the three-dimensional environment 704 is greater than that of the user in Figure 7B The position of the user 714 in the three-dimensional environment 704 is closer to the representation 706 of the application A and the object 710, as indicated by the legend 712. In response to detecting the change in the position of the user 714, the electronic device 101 updates the display of the three-dimensional environment 704. Figure 7C As shown in FIG, the representation 706 of application A and the object 710 are Figure 7C Compared with them in Figure 7B A larger size is displayed by display 120, and as the user approaches representation 706 of application A and object 710, portions of each of object 710 and representation 706 of application A are outside the field of view of electronic device 101. Figure 7B and Figure 7CAs shown in legend 712 of , representations 706 and 708 and the size of object 710 in three-dimensional environment 704 remain constant in response to updating the position of user 714 in three-dimensional environment 704 (and, in some embodiments, the size of the object remains the same in three-dimensional environment 704 as movement of user 714 is detected). Figure 7C As shown in FIG, the representation 708 of application B is no longer Figure 7C User 714 is visible at the location shown in .

[0122] therefore, 7A to 7C The electronic device 101 is shown updating the position, size, and / or orientation of the representation 706 of application A in response to user input moving the representation 706 of application A, and forgoing updating the position, size, and / or orientation of the representation 706 of application A, the representation 708 of application B, and the virtual object 710 in response to detecting a change in the position of the user 714 in the three-dimensional environment 704. In some embodiments, the electronic device 101 will also update the position, size, and / or orientation of the representation 708 or the object 710 in response to user input moving one of these objects.

[0123] In some embodiments, in response to user input, the electronic device 101 initiates display of a representation 706 of application A, a representation 708 of application B, and / or an object 710. The electronic device optionally initially displays the virtual object at a distance that is legible and convenient for the user 714 (e.g., close enough for the user to view and interact with the object, and far enough to view other objects in the three-dimensional environment 704). In some embodiments, if there are additional users in the three-dimensional environment 704 in addition to the user 714, the electronic device 101 will initially display the virtual object to which both users have access at a location that is legible to both users. However, if only the user 714 has access to the object, in some embodiments, the electronic device 101 selects the initial location of the object based on the location of the user 714 and not based on the location of the other user. In some embodiments, when the virtual object is initially displayed in the three-dimensional environment 704 (e.g., in response to user input), if the user 714 is the only user in the three-dimensional environment 704, or if the user 714 is the only user in the three-dimensional environment 704 with access to the object, the electronic device 101 selects the location of the object to be a distance from the user 714 that is between a first threshold value (e.g., 30 centimeters, 40 centimeters, 50 centimeters, 100 centimeters, etc.) and a second threshold value (e.g., 200 centimeters, 300 centimeters, 400 centimeters, 500 centimeters, etc.). For example, if the user 714 is the only user in the three-dimensional environment 704, or if the user 714 is the only user in the three-dimensional environment 704 with access to the object, the electronic device 101 selects the location of the object to be a distance from the user 714 that is between a first threshold value (e.g., 30 centimeters, 40 centimeters, 500 centimeters, etc.). Figure 7A, and the representation 706 of application A is accessible to user 714 but not to another user, the electronic device 101 will initially display the representation 706 of application A at a position, size, and orientation based on the position of user 714 in the three-dimensional environment 704 and not based on the position of the other user (because the other user cannot see or interact with the representation 706). In some embodiments, if there are multiple users with access to the object in the three-dimensional environment 704, the electronic device 101 initially displays the object at a position that is at least a third threshold distance (e.g., 30 cm, 40 cm, 50 cm, 100 cm, etc.) from all users but less than a fourth threshold distance (e.g., 200 cm, 300 cm, 400 cm, 500 cm, etc.) from all users. For example, if in Figure 7A If there are additional users in the three-dimensional environment 704 shown in FIG, and the representation 706 of application A is accessible to the user 714 and the other user, the electronic device 101 will initially display the representation 706 of application A at a position, size, and orientation based on the positions of the user 714 and the additional users in the three-dimensional environment 704 (e.g., to make the representation 706 legible to the user 714 and the other user).

[0124] The following reference Figures 13A to 14L Described in more detail, in some embodiments, when the three-dimensional environment 704 includes at least one user and at least one virtual object, the electronic device 101 detects that another user has been added to the three-dimensional environment 704. In some embodiments, in response to the new user being added to the three-dimensional environment 704, the electronic device 101 updates the orientation of the virtual object in the three-dimensional environment 704 to an orientation based on the positions of all users (including the user who has just joined the three-dimensional environment 704) in the three-dimensional environment 704, to ensure that the virtual object is legible to all users who have access to the virtual object. In some embodiments, the orientation of the object in the three-dimensional environment is updated in response to the user joining the three-dimensional environment 704 without detecting an input to move the virtual object. In some embodiments, the orientation of the object in the three-dimensional environment 704 is not updated until an input to move the object is detected. In some embodiments, one or more virtual objects that are accessible only to a subset of users in the three-dimensional environment 704 (e.g., one user) and not to the user who has just been added to the three-dimensional environment 704 do not change orientation in response to the new user being added to the three-dimensional environment 704. In some embodiments, when a virtual object is inaccessible to the user, the user can see an indication of the location of the virtual object in the three-dimensional environment 704, but cannot see the content of the object, as described below with reference to Figure 9AIn some embodiments, the electronic device 101 does not display indications of virtual objects that are not accessible to a user of the electronic device.

[0125] In some embodiments, the electronic device 101 updates the virtual object from being accessible to multiple (or all) users in the three-dimensional environment 704 to being accessible to fewer than multiple (or all) users in the three-dimensional environment 704 (e.g., accessible to one user in the three-dimensional environment 704). For example, if Figure 7A , includes an additional user and the representation 706 of application A is accessible to the additional user and user 714, the representation 706 of application A can be updated to be accessible to user 714 and inaccessible to the additional user. In some embodiments, in response to input to move an object to a location between users in the three-dimensional environment 704, the electronic device 101 reduces the number of users that have access to the object. For example, if Figure 7A , wherein the three-dimensional environment 704 shown in FIG includes an additional user and the representation 706 of application A is accessible to the additional user and user 714, then in response to detecting an input to move the representation 706 of application A to a position between the position of user 714 and the position of another user in the three-dimensional environment 704, the electronic device 101 updates the representation 706 of application A to be accessible to user 714 and inaccessible to the other user. For another example, in response to an input to move an object to within a threshold distance (e.g., 0.5 meters, 1 meter, 2 meters, etc.) of user 714 in the three-dimensional environment 704, the electronic device 101 reduces the number of users that have access to the object. For example, if Figure 7A , includes an additional user and the representation 706 of application A is accessible to the additional user and user 714, then, in response to detecting input to move the representation 706 of application A to a position within a threshold distance of user 714 in the three-dimensional environment 704, the electronic device 101 updates the representation 706 of application A to be accessible to user 714 and inaccessible to the other user. In some embodiments, when a virtual object is accessible to a first user in the three-dimensional environment 704, the electronic device 101 detects input (e.g., from the first user) to move the virtual object to a position within a threshold distance (e.g., 0.5 meters, 1 meter, 2 meters, etc.) of a second user. In some embodiments, in response to the input, the electronic device 101 updates the virtual object from being accessible to the first user to being accessible to the second user. For example, if Figure 7AThe three-dimensional environment 704 shown in the figure includes additional users and the representation 706 of application A is accessible to user 714 and inaccessible to the other user, then in response to detecting input that moves the representation 706 of application A to within a threshold of the position of the other user in the three-dimensional environment 704, the electronic device 101 updates the representation 706 of application A to be accessible to the other user and inaccessible to user 714.

[0126] In some embodiments, reference will be made to 9A to 10T Described in more detail, in response to user input (eg, user input to share a virtual object), the electronic device 101 updates the virtual object from being accessible to only one user to being accessible to multiple users.

[0127] Figures 8A to 8P is a flow chart illustrating a method for automatically updating the orientation of a virtual object in a three-dimensional environment based on a user's viewpoint according to some embodiments. In some embodiments, method 800 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4 In some embodiments, the method 800 is performed by one or more processors of a computer system, such as one or more processors 202 of the computer system 101 (e.g., a computer readable medium, a display generating component 120 in the computer system 101) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward toward the user's hand (e.g., a color sensor, an infrared sensor, and other depth sensing cameras) or a camera pointing forward from the user's head). Figure 1 Some operations in method 800 may be optionally combined, and / or the order of some operations may be optionally changed.

[0128] In some embodiments, such as Figure 7AIn some embodiments, method 800 is performed at an electronic device (e.g., a mobile device (e.g., a tablet, smart phone, media player, or wearable device) or a computer) that communicates with a display generation component and one or more input devices. In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touch screen display), an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users. In some embodiments, one or more input devices include an electronic device or a component capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor), etc.

[0129] In some embodiments, such as Figure 7A In some embodiments, the electronic device 100 (e.g., 101) displays (802) a three-dimensional environment (e.g., 704) via a display generation component, the three-dimensional environment including a representation (e.g., 706) of a first application displayed at a first position in the three-dimensional environment (e.g., 704). In some embodiments, the three-dimensional environment is generated, displayed, or otherwise enabled to be viewed by the device (e.g., the three-dimensional environment is a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment). In some embodiments, the electronic device corresponds to a position of the electronic device within a physical environment corresponding to the three-dimensional environment from a viewpoint of its display environment. In some embodiments, the electronic device corresponds to a virtual position of a user of the electronic device within the three-dimensional environment from a viewpoint of its display environment.

[0130] In some embodiments, such as Figure 7AIn some embodiments, when the representation of the first application is displayed at the first location in the three-dimensional environment, the representation of the first application is displayed as being oriented toward a corresponding fixed reference frame within the three-dimensional environment. For example, in response to detecting an input to launch the first application, the electronic device displays the representation of the first application at the first location in an orientation facing toward a location associated with a user from whom the input to launch the first application was received.

[0131] In some embodiments, such as Figure 7A In some embodiments, the electronic device displays the three-dimensional environment from a viewpoint located at the second position in the three-dimensional environment. In some embodiments, the second position in the three-dimensional environment corresponds to a physical position of the electronic device in the physical surroundings of the electronic device. For example, the electronic device displays a virtual environment corresponding to the physical environment of the electronic device from a vantage point corresponding to the physical position of the electronic device in the physical environment via a display generation component.

[0132] In some embodiments, such as Figure 7A In some embodiments, while displaying a three-dimensional environment (e.g., 704) including a representation of a first application (e.g., 706), the electronic device (e.g., 101) receives (808) movement input via one or more input devices. In some embodiments, receiving the movement input includes detecting movement of the electronic device from a first physical location corresponding to a second location in the three-dimensional environment to a second physical location via one or more sensors (e.g., accelerometers, gyroscopes, cameras, depth sensors, etc.) in communication with the electronic device. In some embodiments, receiving the movement input includes detecting movement of a user's gaze via an eye tracking device (e.g., in conjunction with selection input, such as detecting execution of a predetermined gesture with a user's hand via a hand tracking device). In some embodiments, receiving the movement input includes detecting movement of a user's hand via a hand tracking device, optionally in conjunction with detecting execution of a predetermined gesture via a hand tracking device. In some embodiments, the movement input is detected via an input device in communication with the electronic device, such as detecting activation of a key of a keyboard or virtual keyboard, movement of a mouse, movement of a contact on a touch-sensitive surface (e.g., a touch screen, trackpad, etc.), or another directional input.

[0133] In some embodiments, such as Figure 7BIn some embodiments, in response to receiving movement input (810), based on a determination (812) that the movement input corresponds to a request to move a representation of a first application (e.g., 706) from a first location to a third location in a three-dimensional environment (e.g., 704) that is different from the first location (e.g., without changing a location associated with the first user), the electronic device (e.g., 101), via display generation component 120, displays (814) the representation of the first application (e.g., 706) at the third location in the three-dimensional environment (e.g., 704). In some embodiments, receiving movement input corresponding to the request to move the representation of the first application from the first location to the third location includes detecting a selection of a user interface element associated with the representation of the first application. In some embodiments, receiving movement input corresponding to the request to move the representation of the first application from the first location to the third location includes detecting movement of a hand of the user, via a hand tracking device, optionally in combination with detecting a selection input. In some embodiments, detecting the selection input includes detecting, via the hand tracking device, the user performing a predetermined gesture. In some embodiments, detecting the selection input includes: detecting via an eye tracking device that the user is looking at the user interface element for a predetermined time (e.g., 0.1 seconds, 0.2 seconds, 0.3 seconds, 0.4 seconds, 0.5 seconds, 0.75 seconds, 1 second, etc.). In some embodiments, receiving a movement input corresponding to a request to move the representation of the first application from a first position to a third position includes: detecting input using an input device (such as a keyboard, trackpad, mouse, or touch screen) that communicates with the electronic device. In some embodiments, the third position is determined based on an orientation input detected as part of detecting the movement input. Examples of detecting directional input include: detecting movement of a user's hand in a corresponding direction using a hand tracking device; detecting movement of a user's gaze in a corresponding direction via an eye tracking device; detecting movement of a contact or hovering object via a touch-sensitive surface; detecting selection of a directional key (e.g., an arrow key or another key assigned a directional input) via a keyboard; and detecting movement of an input device such as a mouse. In some embodiments, the directional input includes a magnitude and a direction. In some embodiments, the distance that the electronic device moves the representation of the first application corresponds to the magnitude of the directional input. In some embodiments, the direction in which the electronic device moves the representation of the first application corresponds to the direction of the directional input.

[0134] In some embodiments, such as Figure 7BIn some embodiments, in response to receiving movement input (810), based on a determination (812) that the movement input corresponds to a request to move a representation of a first application (e.g., 706) from a first location to a third location in a three-dimensional environment that is different from the first location, the electronic device (e.g., 101) updates (816) the orientation of the representation of the first application (e.g., 706) from the first orientation to a second orientation relative to the three-dimensional environment (e.g., 704) that is different from the first orientation based on the third location (and optionally based on a second location associated with the first user). In some embodiments, the orientation of the representation of the first application is relative to a fixed reference frame and is updated relative to the fixed reference frame. In some embodiments, the orientation of the representation of the first application gradually changes as the representation of the first application moves in response to the movement input, such that the representation of the first application faces the user who provided the input to move the representation of the first application. For example, the representation of the first application is a virtual object having a corresponding face on which information related to the first application is displayed, and in response to moving the virtual object from the first location to the third location, the electronic device rotates the representation of the first object so that the corresponding face is positioned toward the location associated with the first user. In some embodiments, the electronic device changes the orientation of the representation of the first application without receiving input to change the orientation of the representation of the first application but only receives input to move the representation of the first application. In some embodiments, moving the representation of the first application to a fourth position (different from the third position) causes the electronic device to display the representation of the first application at the fourth position relative to a fixed reference system in a third appearance (e.g., orientation) that is different from the second appearance (e.g., orientation).

[0135] In some embodiments, such as Figure 7CIn some embodiments, in response to receiving movement input (810), based on a determination that the movement input corresponds to a request to associate a fourth location different from the second location with the first user (e.g., 714) (e.g., without changing the location of the representation of the first application in the three-dimensional environment), the electronic device (e.g., 101) maintains (818) display of the representation (e.g., 706) of the first application at the first location in the three-dimensional environment (e.g., 704) without updating the orientation of the representation of the first application relative to the three-dimensional environment (e.g., 704). In some embodiments, receiving the movement input corresponding to the request to associate the fourth location with the first user includes detecting, via one or more sensors (e.g., accelerometers, gyroscopes, cameras, range sensors, etc.) in communication with the electronic device, movement of the electronic device from a corresponding location in the physical surroundings of the electronic device that is associated with the first location in the three-dimensional environment to a different corresponding location in the physical surroundings of the electronic device. In some embodiments, receiving movement input corresponding to a request to associate a fourth position with the first user includes: detecting movement of a user's hand in a corresponding direction using a hand tracking device; detecting movement of the user's gaze in a corresponding direction via an eye tracking device; detecting movement of a contact or hovering object via a touch-sensitive surface; detecting selection of a directional key (e.g., an arrow key or another key assigned a directional input) via a keyboard; and detecting movement of an input device such as a mouse. In some embodiments, the fourth position is selected based on the magnitude and direction of movement of the movement input. In some embodiments, moving the representation of the application causes the orientation of the application to change based on the representation of the application and the position of the user, but changing the position of the user does not cause the orientation of the representation of the application to change. In some embodiments, if there are two or more users in the three-dimensional environment, then in response to detecting movement of the representation of the application, the electronic device updates the orientation of the representation in accordance with one or more steps of method 1000.

[0136] The above-described manner of moving the representation of the first application and updating the orientation of the representation of the first application or changing the location associated with the first user without updating the orientation of the representation of the first application provides an efficient manner of orienting the representation of the application toward the location associated with the user when moving the representation of the first application (for example, by automatically changing the orientation of the representation of the first application) and an efficient manner of updating the location associated with the user (for example, by forgoing updating the orientation of the representation of the first application to save power), which further reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0137] In some embodiments, such as Figure 11AIn the embodiment of the present invention, the three-dimensional environment (e.g., 1104) includes a representation (820) of the physical environment of the electronic device (e.g., 101) displayed via the display generation component 120. In some embodiments, the physical environment includes a pass-through video (e.g., a virtual representation of the physical environment of the electronic device). In some embodiments, the physical environment around the electronic device is captured by one or more visible light sensors (e.g., cameras) in communication with the electronic device and displayed as an image-realistic representation of the physical environment via the display generation component. In some embodiments, the physical environment around the electronic device is passively presented to the user, for example, via a transparent or semi-transparent display generation component.

[0138] The above-described manner of including a representation of the physical environment of an electronic device in a three-dimensional environment provides an efficient manner of orienting virtual content of the three-dimensional environment relative to physical content of the physical environment of the electronic device, which in turn reduces power usage and improves battery life of the electronic device by enabling a user to use the electronic device more quickly and efficiently (e.g., by reducing conflicts between virtual objects and physical objects and the time spent resolving such conflicts, and by reducing user errors such as a user physically bumping into a physical object in the environment of the electronic device), while reducing errors in use.

[0139] In some embodiments, such as Figure 7A In some embodiments, the three-dimensional environment (e.g., 704) includes a virtual environment (822) displayed via the display generation component 120. In some embodiments, the virtual environment includes virtual objects that do not correspond to objects in the physical environment of the electronic device and / or the physical environment of the electronic device. For example, the virtual environment includes virtual objects that create a virtual setting in which a representation of an application is displayed.

[0140] The above-described manner of adjusting the orientation of the representation of the first application within the virtual environment provides an effective manner of automatically orienting the first application toward a location associated with the user in the virtual environment, which additionally reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0141] In some embodiments, such as Figure 7A In some embodiments, such as in Figure 7BIn some embodiments, in response to receiving a movement input (826), based on a determination that the movement input corresponds to a request to move a representation of a first application (e.g., 706) from a first location to a third location in a three-dimensional environment that is different from the first location, the electronic device (e.g., 101) updates (828) the representation of the first application to be displayed at a second size that is different from the first size when the representation of the first application is displayed at the third location. In some embodiments, updating the size of the representation includes updating the amount of display area of ​​the display generation component that is occupied by the representation (e.g., and therefore displayed to the user). For example, in response to the input to move the representation to a location closer to a virtual location associated with the user, the electronic device increases the size of the representation to convey that the representation is closer to the virtual location associated with the user than it was before receiving the input. In some embodiments, updating the size of the representation includes updating the virtual size of the representation (e.g., the size of the representation relative to the three-dimensional environment). For example, in response to the input to move the representation to a virtual location closer to the location associated with the user, the electronic device forgoes modifying the display area in which the representation is displayed, or modifies the display area in which the representation is displayed by an amount that is different from the amount corresponding to moving the representation to a different virtual distance, without changing the size of the representation. In some embodiments, when the representation is moving from a first position to a third position, the electronic device displays an animation in which the representation is moving and gradually changes size as the representation moves.

[0142] The above-described manner of updating the size of a representation in response to input corresponding to a request to move an object provides an efficient manner of displaying a representation at an appropriate size at a third location (e.g., without requiring a separate input to adjust the size of the representation), which further reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0143] In some embodiments, such as Figure 9AIn some embodiments, updating (830) the orientation of the representation (e.g., 908) of the first application includes: based on a determination that the representation (e.g., 908) of the first application is accessible to the first user (e.g., 914A) but not to the second user 914B associated with a fifth location in the three-dimensional environment (e.g., 904), updating the orientation (832) of the representation (e.g., 908) of the first application based on the second location associated with the first user (e.g., 914A) and not based on the fifth location associated with the second user (e.g., 914B). In some embodiments, the electronic device displays the representation of the second user at the fifth location in the three-dimensional environment, and the electronic device of the second user displays the virtual environment from a viewpoint of the fifth location. In some embodiments, one or more applications in the virtual environment are displayed in a private mode in which not all users (e.g., one or more, but less than all) of the users in the virtual environment are able to interact with and / or view the applications in the private mode. For example, if the first application is displayed in private mode and is accessible only to the first user, in response to input to move the first application, the electronic device updates the orientation of the representation of the first application to be oriented toward the first user when moving the representation (e.g., without regard to the position of the second user in the three-dimensional environment). In some embodiments, orienting the first application toward the first user is independent of the position of the second user. In some embodiments, such as in Figure 9C In some embodiments, updating the orientation of the representation of the first application (e.g., 906) includes (830): in accordance with a determination that the representation of the first application (e.g., 906) is accessible to the first user (e.g., 914A) and the second user (e.g., 914B) associated with a fifth location in the three-dimensional environment, updating the orientation of the representation of the first application (e.g., 906) based on the second location associated with the first user (e.g., 914A) and the fifth location associated with the second user (e.g., 914B) (834). In some embodiments, one or more applications in the virtual environment are displayed in a public mode in which all users in the virtual environment can interact with and / or view the applications in the public mode, or in a shared mode in which the first user and the second user (and potentially other users, but fewer than all users) can interact with and / or view the applications in the shared mode that are accessible to the first user and the second user. For example, if a first application is displayed in a shared mode and is accessible to both a first user and a second user, then in response to an input to move the first application, the electronic device updates the orientation of the representation of the first application to be oriented toward the shared viewpoint of the first user and the second user. In some embodiments, the orientation of the application is updated according to the positions of the two users in accordance with one or more steps of method 1000.

[0144] The above-described manner of updating the orientation of an application's representation based solely on the location of users able to interact with the application provides an efficient manner of orienting an application so that the application's representation can be easily viewed by all users with access to the application, which in turn reduces power usage and improves battery life of electronic devices by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0145] In some embodiments, the electronic device (e.g., 101) detects (836) a corresponding input to display a representation of a second application in a three-dimensional environment (e.g., 704), wherein the representation of the second application is not displayed in the three-dimensional environment (e.g., 704) when the corresponding input is detected. In some embodiments, the input is an input to launch an application that is not running on the electronic device when the input is received, an input to view a hidden user interface when the input is received, or an input to display an additional representation of an application running on the electronic device without displaying the additional representation when the input is received. In some embodiments, in response to detecting the corresponding input, the electronic device (e.g., 101) displays (838) the representation of the second application at a corresponding location in the three-dimensional environment based on the number of users in the three-dimensional environment (e.g., 704). In some embodiments, if there is one user in the three-dimensional environment, the electronic device displays the representation at a first corresponding location based on a location associated with the user. In some embodiments, if there are two users in the three-dimensional environment, the electronic device displays the representation at a second corresponding position different from the first corresponding position based on the positions associated with the two users (e.g., oriented toward a position between the two users where the two users are on the same side of the representation, or located between the two users where the two users are on different sides of the representation). In some embodiments, the position within the three-dimensional environment at which the representation is initially presented is based on the position of the users who can interact with and / or view the representation. For example, if there are three users in the virtual environment and all users can interact with and / or view the representation, the electronic device displays the representation at a corresponding position in the three-dimensional environment based on the position associated with each of the three users. In some embodiments, the initial position of the representation is selected according to one or more steps of method 1000.

[0146] The above-described manner of displaying a representation of a second application at a location in a three-dimensional environment based on the number of users in the three-dimensional environment provides an efficient manner of initially presenting the second representation at a location suitable for all relevant users in the virtual environment, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to move the representation to a location suitable for all users in the three-dimensional environment), while reducing errors in use.

[0147] In some embodiments, such as Figure 7A In some embodiments, the corresponding position is in front of the first user (e.g., 714) based on the determination that the three-dimensional environment (e.g., 704) includes the first user (e.g., 714) and no other users (840). In some embodiments, the users are associated with corresponding positions and orientations within the three-dimensional environment and are positioned at positions in the user's line of sight with an orientation toward the position associated with the user. In some embodiments, such as in Figure 9A In some embodiments, a corresponding position is located based on a determination that a three-dimensional environment (e.g., 904) includes a first user (e.g., 914A) and a second user (e.g., 914B), and a corresponding position is located based on the position of the first user (e.g., 914A) and the position of the second user (e.g., 914B) (842). In some embodiments, the corresponding position is between the position of the first user and the position of the second user. In some embodiments, the corresponding position is equidistant from the position of the first user and the position of the second user. In some embodiments, the first user and the second user are each associated with a position within the three-dimensional environment, and a representation is presented at a position between the positions associated with the users (e.g., such that a first side of the representation is oriented toward the first user and / or away from the second user, and a second side of the representation is oriented toward the second user and / or away from the first user). In some embodiments, the electronic device presents the representation in a first orientation based on the position associated with the first user, and the second user's electronic device presents the representation in a second orientation based on the position associated with the second user.

[0148] The above-described manner of placing applications differently when there is one user than when there are two users provides an efficient manner of presenting representations in a manner that is easily viewable by users who have access to the representations, which in addition reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to move the representation to a location where it can be easily viewed by the user), while reducing errors in use.

[0149] In some embodiments, such as Figure 7AIn some embodiments, based on a determination that the three-dimensional environment (e.g., 704) includes the first user (e.g., 714) and no other users, a representation of the second application is displayed as oriented toward the first user (844). In some embodiments, the users are associated with corresponding positions and orientations within the three-dimensional environment, and the representation is placed at a position in the user's line of sight with an orientation toward the position associated with the user. In some embodiments, such as in Figure 9A In some embodiments, the representation of the second application is displayed as oriented toward a location between the first user and the second user (846) based on a determination that the three-dimensional environment (e.g., 904) includes a first user (e.g., 914A) and a second user (e.g., 914B). In some embodiments, the representation of the second application is displayed at a location that is not between the locations associated with the users. In some embodiments, the position and orientation of the representation of the second application are such that a corresponding face / side of the representation of the application is oriented toward a location associated with the users (e.g., a location between the users, such as a "shared" location). In some embodiments, both users are on the same side of the representation of the second application (e.g., both are on the front side of the representation of the second application). In some examples, the position of the representation of the second application is selected according to one or more steps of method 1000.

[0150] The above-described manner of placing applications differently when there is one user than when there are two users provides an efficient manner of presenting representations in a manner that is easily viewable by users who have access to the representations, which in addition reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to move the representation to a location where it can be easily viewed by the user), while reducing errors in use.

[0151] In some embodiments, such as Figure 7A In some embodiments, the corresponding location is a first distance from a location associated with the user in the three-dimensional environment. In some embodiments, the first distance is determined based on the location associated with the user and the location of one or more other objects (e.g., virtual objects, real objects) in the three-dimensional environment. In some embodiments, such as in Figure 9AIn some embodiments, the corresponding location is a second distance from the location associated with the first user in the three-dimensional environment. In some embodiments, the distance is selected so that both the first user and the second user can view the representation of the first application. The corresponding location, the second distance, and the third distance are optionally selected based on the locations associated with the first user and the second user and / or the locations of one or more other objects (e.g., virtual objects, real objects) in the three-dimensional environment. In some embodiments, the corresponding location is selected according to one or more steps of method 1000. In some embodiments, when there is one user in the three-dimensional environment, the electronic device displays a representation at a first distance from the user, and when there are more than one user in the three-dimensional environment, the electronic device displays a representation at a second distance (e.g., greater than the first distance) from the users' shared points of interest (e.g., locations between all users, areas of the environment including all users, etc.).

[0152] The above-described manner of placing the application at different distances from the users when there is one user and when there are two users provides an efficient manner of presenting a representation in a manner that is easily viewable by a user who has access to the representation, which in addition reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to move the representation to a location where it can be easily viewed by the user), while reducing errors in use.

[0153] In some embodiments, such as Figure 13A , when a representation 1306 of a first application is displayed at a first location in the three-dimensional environment 1304, the electronic device (e.g., 101) detects (852) that a second user has been added to the three-dimensional environment 1304 (e.g., detects indication 1330a or 1330b), where the second user is associated with a fifth location in the three-dimensional environment. In some embodiments, the electronic device detects input corresponding to a request to add the second user to the three-dimensional environment, or the electronic device receives an indication that the second user is joining the three-dimensional environment (e.g., via a network connection). In some embodiments, prior to detecting that the second user has been added to the three-dimensional environment, the first user is the only user in the three-dimensional environment. In some embodiments, such as in Figure 13BIn some embodiments, in response to detecting that the second user 1314C has been added to the three-dimensional environment 1304 (854), based on a determination that the representation 1306 of the first application is accessible to the first user 1314A and the second user 1314C, the electronic device (e.g., 101) moves (856) the representation 1306 of the first application from the first position to a sixth position in the three-dimensional environment based on the second position and the fifth position (e.g., the electronic device (e.g., 101a) or (e.g., 101b) is capable of moving the representation 1306 in response to user C 1314C joining the three-dimensional environment 1304). In some embodiments, the sixth position is selected according to one or more steps of method 1000. In some embodiments, in response to detecting that the second user has been added to the three-dimensional environment, the electronic device updates the representation of the first application to be displayed at a position and / or in an orientation that makes the representation visible to both users. For example, the representation includes respective sides on which content is displayed, and the sixth position is a position at which the electronic device is capable of orienting the respective sides of the representation toward positions associated with the two users. In some embodiments, only the position of the representation of the first application changes. In some embodiments, only the orientation of the representation of the first application changes. In some embodiments, both the position and orientation of the representation of the first application change.

[0154] The above-described manner of moving the representation of the first application based on the second position and the fifth position provides an efficient manner of updating the three-dimensional environment so that both the first user and the second user can view the representation of the first application, which further reduces power usage and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by reducing the input required to place the representation at a location where both users can see the representation), while reducing errors in use.

[0155] In some embodiments, such as Figure 13A, when a representation 1306 of a first application is displayed at a first location in the three-dimensional environment 1304, the electronic device (e.g., 101) detects (858) that a second user has been added to the three-dimensional environment 1304 (e.g., detects indication 1330a or 1330b), where the second user is associated with a fifth location in the three-dimensional environment. In some embodiments, the electronic device detects input corresponding to a request to add the second user to the three-dimensional environment, or the electronic device receives an indication that the second user is joining the three-dimensional environment (e.g., via a network connection). In some embodiments, before detecting that the second user is added to the three-dimensional environment, the first user is the only user in the three-dimensional environment. In some embodiments, in response to detecting that the second user has been added to the three-dimensional environment (860), based on a determination that the representation of the first application (e.g., 908) is accessible to the first user (e.g., 914A) but not to the second user (e.g., 914B), such as in Figure 9B In some embodiments, the electronic device (e.g., 101) maintains (862) display of a representation (e.g., 908) of a first application at a first location, where the representation of the first application is not visible to a second user (e.g., the electronic device (e.g., 101a) maintains display of the representation (e.g., 908) in response to another user joining the three-dimensional environment (e.g., 904) because the representation (e.g., 908) is accessible only to user A (e.g., 914A)). In some embodiments, the representation of the first application is displayed in a private mode in which the first user is able to interact with and / or view the first application, but other users in the three-dimensional environment are unable to interact with and / or view the first application. For example, the second user's electronic device abandons displaying the representation of the first application because the second user does not have access to the first application. In some embodiments, after the second user joins the three-dimensional space, the electronic device continues to display the representation of the first application in the same location and orientation that was used to display the representation of the first application before the second user joined the three-dimensional space.

[0156] The above-described manner of maintaining display of representations of applications that are not accessible to the second user after the second user is added to the three-dimensional space provides an efficient manner of displaying representations of applications that are not visible to other users in the three-dimensional environment, which additionally reduces power usage and improves battery life of the electronic device by forgoing calculating the updated position of the representation of the first application when the second user does not have access to the first application.

[0157] In some embodiments, such as Figure 9AIn some embodiments, the three-dimensional environment includes a first user (e.g., 914A) at a second location in the three-dimensional environment and a second user (e.g., 914B) at a fifth location in the three-dimensional environment (e.g., 904) (864). In some embodiments, the electronic device displays a representation of the second user at the fifth location in the three-dimensional environment, and the second user's electronic device displays the three-dimensional environment from a viewpoint at the fifth location in the three-dimensional environment. In some embodiments, the electronic device (e.g., 101) detects (866) a corresponding input to display a representation of a second application in the three-dimensional environment, wherein when the corresponding input is detected, the representation of the second application is not displayed in the three-dimensional environment, and the representation of the second application is not accessible to multiple users in the three-dimensional environment, such as Figure 9A In some embodiments, the second application is in a private mode in which the first user providing the input is able to access and / or view the second application, but other users in the three-dimensional environment are unable to access and / or view the second application. In some embodiments, the corresponding input corresponds to a request to launch the second application, a request to display a representation of the second application that has been previously hidden, or a request to display a representation of the second application in addition to the representation of the second application already displayed by the electronic device. In some embodiments, in response to receiving the corresponding input (868), based on a determination that the corresponding input is associated with the first user (e.g., 914A) (e.g., the electronic device detecting the corresponding input is provided by the first user), the electronic device (e.g., 101) displays (870) the representation of the second application (e.g., 908) at a sixth position in the three-dimensional environment (e.g., 904) based on the second position of the first user (e.g., 914A) rather than the fifth position of the second user (e.g., 914B), such as at Figure 9A In some embodiments, if the first user provides input corresponding to a request to display a representation of the second application, the electronic device displays the representation of the second application at a position and / or orientation based on the position associated with the first user. For example, the representation of the second application is displayed at a position adjacent to the second position and / or oriented toward the second position. In some embodiments, the sixth position is selected independently of the positions of users other than the first user in the three-dimensional environment. In some embodiments, in response to receiving the corresponding input (868), based on a determination that the corresponding input is associated with the second user (e.g., the electronic device detecting the corresponding input is provided by the second user), the electronic device (e.g., 101) displays the representation of the second application based on the fifth position of the second user rather than the second position of the first user (e.g., if the representation (e.g., 908) is for Figure 9AIn some embodiments, the electronic device displays (872) a representation of the second application at a seventh location in the three-dimensional environment, accessible to user B (e.g., 914B) but not user A (e.g., 914A). In some embodiments, if the second user provides input corresponding to a request to display a representation of the second application, the electronic device displays the representation of the second application at a location and / or orientation based on the location associated with the second user. For example, the representation of the second application is displayed at a location adjacent to the fifth location and / or oriented toward the fifth location. In some embodiments, the sixth location is selected independently of the locations of users other than the second user in the three-dimensional environment.

[0158] The above-described manner of placing a representation of a second application at a location based on a location associated with a user who provided input for displaying the representation of the second application provides an efficient manner of placing an application at a location visible to a user who requested display of the application, which further reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (e.g., by reducing the input required to place the application at a location readily visible to the user who requested display of the application), while reducing errors in use.

[0159] In some embodiments, such as Figure 9A In some embodiments, the three-dimensional environment (e.g., 904) includes a first user at a second position in the three-dimensional environment and a second user (874) at a fifth position in the three-dimensional environment. In some embodiments, the electronic device displays a representation of the second user at the fifth position in the three-dimensional environment, and the second user's electronic device displays the three-dimensional environment from a viewpoint at the fifth position in the three-dimensional environment. In some embodiments, the electronic device (e.g., 101) detects (876) a corresponding input to display a representation of the second application in the three-dimensional environment, wherein when the corresponding input is detected (e.g., received in Figure 9A In some embodiments, when the corresponding input is received by the electronic device (e.g., 101), the representation of the second application is not displayed in the three-dimensional environment. In some embodiments, the second application is in a private mode in which some but not all users in the three-dimensional environment are able to access and / or view the application. In some embodiments, the corresponding input corresponds to a request to launch the second application, a request to display a representation of the second application that was previously hidden, or a request to display a representation of the second application in addition to the representation of the second application already displayed by the electronic device. In some embodiments, in response to receiving the corresponding input, the electronic device (e.g., 101) launches the second application in the three-dimensional environment (e.g., Figure 9AA representation of a second application (e.g., Figure 9A In some embodiments, the representation of the second application is inaccessible to the first user (e.g., in a manner consistent with the representation (e.g., 908)). Figure 9A In some embodiments, the second application is accessible to the second user and, optionally, one or more additional users in the three-dimensional environment other than the first user. In some embodiments, if the second application is accessible only to the second user, the corresponding location is based on the fifth location. In some embodiments, if the second application is accessible to the second user and one or more additional users, the corresponding location is based on the locations of all users who have access to the application, and is not based on the locations of one or more users who do not have access to the application. In some embodiments, the electronic device places a representation of the second application according to the locations of multiple users in accordance with one or more steps of method 1000. In some embodiments, the representation of the second application is inaccessible to the second user (e.g., in a manner similar to that in which the representation (e.g., 908) is accessible to the second user). Figure 9A In some embodiments, the second application is accessible to the first user and, optionally, one or more additional users in the three-dimensional environment other than the second user. In some embodiments, if the second application is accessible only to the first user, the corresponding location is based on the second location. In some embodiments, if the second application is accessible to the first user and one or more additional users, the corresponding location is based on the locations of all users who have access to the application, and is not based on the locations of one or more users who do not have access to the application. In some embodiments, the electronic device places a representation of the second application based on the locations of multiple users in accordance with one or more steps of method 1000.

[0160] The above-described manner of not basing the corresponding location of the second application on the location of a user who does not have access to the second application provides an effective way of setting criteria for placing the representation of the second application, which further reduces power usage and improves battery life of the electronic device by simplifying the complexity of determining the location of the second application, while reducing errors in use.

[0161] In some embodiments, such as Figure 9AIn accordance with a determination that the representation of the second application (e.g., 908) is inaccessible to the first user (e.g., 914B), the representation of the second application (e.g., by Figure 9A A visual indication of a representation (e.g., 908) displayed by electronic device B (e.g., 101b) in is visible to the first user, while the content of the representation of the second application is not visible to the first user (884). In some embodiments, the visual indication is an object displayed at the location of the representation of the second application, indicating that an object that is inaccessible to the first user is being displayed. In some embodiments, the indication includes an indication that the object is an application user interface. In some embodiments, the indication does not include an indication that the object is an application user interface. In some embodiments, the indication includes an indication of the name of the second application. In some embodiments, the indication does not include an indication of the name of the second application. In some embodiments, the indication has the same size and / or shape as the representation of the second user interface. In some embodiments, the indication has a size and / or shape that is different from the size of the representation of the second user interface. In some embodiments, such as in Figure 9A In accordance with a determination that the representation of the second application (e.g., 908) is inaccessible to the second user, the representation of the second application (e.g., by Figure 9A A visual indication of the representation (e.g., 908) displayed by electronic device B (e.g., 101b) in FIG is visible to the second user, while the content of the representation of the second application is not visible to the second user (886). In some embodiments, the visual indication is an object displayed at the location of the representation of the second application, indicating that an object that is inaccessible to the second user is being displayed. In some embodiments, the indication includes an indication that the object is an application user interface. In some embodiments, the indication does not include an indication that the object is an application user interface. In some embodiments, the indication includes an indication of the name of the second application. In some embodiments, the indication does not include an indication of the name of the second application. In some embodiments, the indication has the same size and / or shape as the representation of the second user interface. In some embodiments, the indication has a size and / or shape that is different from the size of the representation of the second user interface.

[0162] The above-described manner of displaying a visual indication of the content of a second application that does not include a representation of the second application provides an effective manner of preventing users who do not have access to the second application from placing virtual objects at the location of the second application, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (e.g., such as by reducing the number of inputs required to avoid placing two objects at the same location in a three-dimensional environment), while reducing errors in use.

[0163] In some embodiments, such as Figure 9A In some embodiments, the three-dimensional environment (e.g., 904) includes a first user (e.g., 914A) at a second position in the three-dimensional environment and a second user (e.g., 914B) at a fifth position in the three-dimensional environment (888). In some embodiments, the electronic device displays a representation of the second user at the fifth position in the three-dimensional environment, and the second user's electronic device displays the three-dimensional environment from a viewpoint at the fifth position in the three-dimensional environment. In some embodiments, before receiving the movement input, the representations of the corresponding applications are accessible to and / or viewable by the first user and the second user. In some embodiments, in response to receiving the movement input (890), based on a determination (892) that the movement input corresponds to a request to move the representation of the first application from the first position to a third position in the three-dimensional environment, based on a determination that the third position is between the first user's second position and the second user's fifth position, the electronic device (e.g., 101) updates (894) the representation of the first application to be accessible to the first user but not to the second user (e.g., the representation (e.g., 908) is not accessible to the second user). Figure 9A In some embodiments, the second user's electronic device stops displaying a representation and / or content of the corresponding application. In some embodiments, in response to the movement input, the corresponding application is no longer accessible to one or more other users in the three-dimensional environment other than the first user and the second user. In some embodiments, in response to the movement input, the corresponding application is accessible only to the user who provided the movement input and is no longer accessible to any other users in the three-dimensional environment. In some embodiments, such as in Figure 9C In some embodiments, in response to receiving a movement input (890), based on a determination (892) that the movement input corresponds to a request to move a representation of a first application (e.g., 906) from a first location to a third location in a three-dimensional environment (e.g., 904), and based on a determination that the third location is not between a second location of the first user (e.g., 914A) and a fifth location of the second user (e.g., 914B), the electronic device (e.g., 101) maintains (896) the representation of the first application (e.g., 906) accessible to the first user (e.g., 914A) and the second user (e.g., 914B). In some embodiments, the electronic device of the second user updates the display of the representation of the corresponding application to display the representation at the third location in the three-dimensional environment. In some embodiments, the corresponding application continues to be accessible to one or more other users in the three-dimensional environment in addition to the first user and the second user.

[0164] The above-described manner of updating the representation of the first application to no longer be accessible to the second user based on a determination that the third position is between the second position and the fifth position provides an effective manner of removing other users' access to the representation of the application, which in addition reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0165] In some embodiments, in response to receiving the movement input (898), based on a determination (897) that the movement input corresponds to a request to move a representation of the first application from the first location to a third location in the three-dimensional environment, based on a determination that the third location is within a threshold distance of the first user's second location, the electronic device (e.g., 101) updates (895) the representation of the first application to be accessible to the first user but inaccessible to the second user in the three-dimensional environment (e.g., similar to how the representation (e.g., 908) is accessible to the first user). Figure 9A In some embodiments, the electronic device of the second user stops displaying the representation and / or content of the corresponding application. In some embodiments, in response to the movement input, the corresponding application is no longer accessible to one or more other users in the three-dimensional environment other than the first user (e.g., the user who provided the movement input). In some embodiments, the electronic device updates the first application to be accessible to only one first user based on (1) a determination that the third position is within a threshold distance of the second position and (2) the first user provided the movement input. For example, if the second user moves the representation of the application to a position within a threshold distance of the second position, the electronic device continues to present the first application to the second user and the first user. In some embodiments, such as in Figure 9C In some embodiments, in response to receiving movement input (898), based on a determination (897) that the movement input corresponds to a request to move a representation of a first application (e.g., 906) from a first location to a third location in the three-dimensional environment, and based on a determination that the third location is not within a threshold distance of a second location of the first user (e.g., 914A), the electronic device (e.g., 101) abandons (893) updating the representation of the first application to be accessible to the first user (e.g., 914A) but not to the second user (e.g., 914B). In some embodiments, the second user's electronic device updates the display of the representation of the corresponding application to display the representation at the third location in the three-dimensional environment. In some embodiments, the corresponding application continues to be accessible to one or more other users in the three-dimensional environment in addition to the first user and the second user.

[0166] The above-described manner of updating the representation of a first application to no longer be accessible to a second user based on a determination that a third location is within a predetermined threshold distance from a second location provides an effective means of removing other users' access to the representation of an application, which in turn reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0167] In some embodiments, in response to receiving the movement input (891), based on a determination (889) that the movement input corresponds to a request to move the representation of the first application from the first location to a third location in the three-dimensional environment, based on a determination that the third location is within a threshold distance of a fifth location associated with the second user, the electronic device (e.g., 101) updates (887) the representation of the first application to be accessible to the second user but inaccessible to the first user in the three-dimensional environment (e.g., to correspond to the representation (e.g., 906)). Figure 9A In a manner similar to that in which the first application is accessible to user A (e.g., 914A) but inaccessible to user B (e.g., 914B) in the three-dimensional environment). In some embodiments, the first user's electronic device stops displaying the representation of the second application, and the second user's electronic device displays the representation of the second application at a third location in the three-dimensional environment. In some embodiments, when the first application is accessible to the second user but inaccessible to the first user, the second user is able to view the first application, provide input to the first application, hear audio associated with the first application, move the representation of the first application within the three-dimensional environment, etc., while the first user is unable to view the first application, provide input to the first application, hear audio associated with the first application, move the representation of the first application within the three-dimensional environment, etc. In some embodiments, if (1) the third location is within a threshold of the fifth location and (2) the second user provides movement input, the electronic device only updates the first application to be accessible to the second user but inaccessible to the first user. In some embodiments, regardless of which user provides the movement input, if the third location is within a threshold of the fifth location, the electronic device updates the first application to be accessible to the second user but inaccessible to the first user. In some embodiments, when movement input is received (e.g., from a first user), the first application is accessible to the first user but inaccessible to the second user. In some embodiments, based on a determination that the third position is equidistantly spaced between the first user and the second user, the electronic device orients the representation of the first application toward whichever user provided the input to move the representation of the first application. In some embodiments, such as in Figure 9CIn some embodiments, in response to receiving movement input (891), based on a determination (889) that the movement input corresponds to a request to move a representation of a first application (e.g., 906) from a first location to a third location in the three-dimensional environment, based on a determination that the third location is not within a threshold distance of a fifth location associated with a second user (e.g., 914B), the electronic device (e.g., 101) abandons (885) updating the representation of the first application to be accessible to the second user (e.g., 914B) but not accessible to the first user (e.g., 914A). In some embodiments, based on a determination that the third location is not within a threshold distance of any location associated with users in the three-dimensional environment, the representation of the second application is accessible to all users in the three-dimensional environment. In some embodiments, based on a determination that the third location is within a threshold distance of a third corresponding user in the three-dimensional environment other than the second user and the first user, the electronic device updates the representation of the second application to be accessible to the third corresponding user and not accessible to the first user or the second user.

[0168] The above-described manner of updating the representation of a first application to be accessible to a second user but inaccessible to the first user based on a determination that a third location is within a threshold distance of a fifth location provides an efficient manner of updating an application to be visible to only one user, which additionally reduces power usage and improves battery life of an electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0169] In some embodiments, such as Figure 7A In some embodiments, an electronic device (e.g., 101) is in a three-dimensional environment (e.g., 704) and simultaneously displays (883) a representation of a first application (e.g., 706) at a first location in the three-dimensional environment and a representation of a second application (e.g., 708) at a fifth location in the three-dimensional environment, wherein the representation of the first application (e.g., 706) is displayed in the three-dimensional environment (e.g., 704) at a first corresponding orientation and the representation of the second application (e.g., 708) is displayed in the three-dimensional environment (e.g., 704) at a second corresponding orientation that is different from the first corresponding orientation. In some embodiments, when there are multiple representations of different applications in the three-dimensional environment, the orientations of the representations of the applications are different from each other. In some embodiments, the representations of the applications are oriented toward a second location associated with the user and the representations of the applications are displayed at different locations such that none of the representations obstructs the user's view of another representation. In some embodiments, in this arrangement, the orientations of the representations are different from each other.

[0170] The above-described manner of displaying representations of a first application and representations of a second application in different orientations provides an efficient manner of presenting these representations so that neither representation obstructs the user's view of the other representation, which in addition reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the number of inputs required to view the various representations), while reducing errors in use.

[0171] In some embodiments, such as Figure 7A A first orientation of a representation of a first application (e.g., 706) and such as Figure 7B The second orientation of the representation of the first application (e.g., 706) in the three-dimensional environment is based on an orientation field (881) associated with the first user. In some embodiments, the orientation field defines an orientation toward the location of the first user for each location in the three-dimensional environment. In some embodiments, an object displayed at a corresponding location in the three-dimensional environment is displayed with an orientation corresponding to the orientation field at the corresponding location. In some embodiments, the orientation field defines a direction of a surface including content that is normal to the corresponding object when the corresponding object is placed at the corresponding location within the orientation field. In some embodiments, if there is one user in the three-dimensional environment, the orientation field points to the location associated with the user. In some embodiments, if there are multiple users in the three-dimensional environment, the electronic device creates an orientation field based on (e.g., as) a mathematical combination (such as a vector sum) of the orientation fields associated with multiple users (e.g., each user) in the three-dimensional environment. In some embodiments, the orientation and placement of the object in the three-dimensional environment including the multiple users are set according to one or more steps of method 1000. In some embodiments, in response to detecting an input to move an object in the three-dimensional environment, the electronic device moves the object to a location corresponding to the movement input and updates the orientation of the object according to the attention field at the location corresponding to the movement input. In some embodiments, the field of attention is two-dimensional and is based on the user's two-dimensional position in the three-dimensional environment (e.g., the field of attention is the same for all vertical heights in the three-dimensional environment). In some embodiments, the field of attention is three-dimensional and is based on the user's three-dimensional position in the three-dimensional environment (e.g., the field of attention depends on the user's vertical position).

[0172] The above-described manner of orienting a representation of a first application and a representation of a second application based on an attention field provides an efficient manner of calculating the orientation of the representations, which additionally reduces power usage and improves the battery life of the electronic device by reducing the computational complexity of updating the orientation of an object when the electronic device initiates display of the object or when a user moves the object.

[0173] In some embodiments, such as Figure 7AIn some embodiments, a three-dimensional environment (e.g., 904) includes a first user (e.g., 914A) and a second user (e.g., 914B), and a representation of a first application (e.g., 908) is accessible to the first user but inaccessible to the second user (879). In some embodiments, the first user's electronic device presents the representation of the first application, but the second user's electronic device does not present the representation of the first application. In some embodiments, when the first application is accessible to the first user, the first user is able to view the first application, hear audio of the first application, provide input to the first application, and / or move the representation of the first application. In some embodiments, when the first application is inaccessible to the second user, the second user is unable to view the first application, hear audio of the first application, provide input to the first application, and / or move the representation of the first application. In some embodiments, such as in Figure 9B , while displaying a representation (e.g., 908) of a first application in a first appearance (e.g., a first size, a first position, and / or a first orientation), the electronic device (e.g., 101) detects (877) via one or more input devices a corresponding input corresponding to a request to update the representation (e.g., 908) of the first application to be accessible to a first user (e.g., 914A) and a second user (e.g., 914B). In some embodiments, the request is a request to share the first application with the corresponding second user and, optionally, any other users present in the three-dimensional environment. In some embodiments, once the first application is accessible to the second user, the second user's electronic device presents the representation of the first application, and the second user is optionally able to interact with the first application in the three-dimensional environment. In some embodiments, while the first application is accessible to the first user and the second user, the first user and the second user are able to view the first application, hear audio of the first application, provide input to the first application, and / or move the representation of the first application. In some embodiments, such as in Figure 9C In some embodiments, in response to detecting the corresponding input (875), the electronic device (e.g., 101) updates (873) the representation of the first application (e.g., 908) to be accessible to the first user (e.g., 914A) and the second user (e.g., 914B). In some embodiments, the second user's electronic device initiates display of the representation of the first application, and the first user's electronic device continues to display the representation of the first application. In some embodiments, in response to detecting the corresponding input (875), the electronic device (e.g., 101) displays (871) the representation of the first application in a second appearance that is different from the first appearance (e.g., if Figure 9C The appearance of the representation in (e.g., 908) is different from Figure 9BIn some embodiments, the first user's electronic device updates the appearance of the representation of the first application in response to the request to make the first application accessible to the second user. For example, the electronic device updates the size, orientation, and / or position of the representation of the first application, such as to orient the representation of the first application toward both the first user and the second user. In some embodiments, the representation of the first application is moved toward the second user. In some embodiments, the representation of the first application is displayed at a larger size (e.g., so that the two users can view the content more easily at a greater distance than before sharing the first application).

[0174] The above-described manner of updating the representation of a first application in response to a request to make the first application accessible to a second user provides an efficient manner of presenting the representation in an appearance that is more easily visible to both users, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to update the appearance of the representation of the first application after sharing the first application), while reducing errors in use.

[0175] In some embodiments, the corresponding input is associated with a suggestion to share the representation of the first application with the second user (869) (e.g., if the electronic device (e.g., 101a) receives Figure 9B ). In some embodiments, the electronic device presents a prompt to share the first application, and the input corresponding to the request to make the first application accessible to the second user is directed to the prompt. In some embodiments, the prompt is displayed in response to the second user entering the three-dimensional environment. In some embodiments, the prompt is presented in response to the initiation of a corresponding operation of the first application other than a request to display the prompt (e.g., a request to present content within the application). In some embodiments, the prompt is presented in response to a request from the second user. In some embodiments, the prompt is presented because the first application is a content display application and / or because the first user has previously shared access to the first application with the second user and / or other users. In some embodiments, such as in Figure 9B, while displaying a representation of a first application (e.g., 908) in a first appearance, the electronic device detects (867) via one or more input devices a second corresponding input corresponding to a request to update the representation of the first application (e.g., 908) to be accessible to a first user (e.g., 914A) and a second user (e.g., 914B), wherein the second corresponding input is not associated with a suggestion to share the representation of the first application (e.g., 908) with the second user (e.g., 914B). In some embodiments, the request to update the representation to be accessible to the second user is initiated by the first user. For example, the electronic device detects a selection of an option to share the first application displayed in a menu presented in response to a request to present the menu, rather than a selection or acceptance of a prompt suggesting sharing the first application. In some embodiments, when the first application is accessible to the first user and the second user, the first user and the second user are able to view the first application, hear audio of the first application, provide input to the first application, and / or move the representation of the first application. In some embodiments, such as in Figure 9C In some embodiments, in response to detecting the corresponding input (865), the electronic device (e.g., 101) updates (863) the representation (e.g., 908) of the first application to be accessible to the first user (e.g., 914A) and the second user (e.g., 914B), while maintaining the representation (e.g., 908) of the first application with the first appearance. In some embodiments, the electronic device of the first user and the electronic device of the second user both display the representation with the first appearance. In some embodiments, the orientation of the representation of the first application in the three-dimensional environment does not change. In some embodiments, the position of the representation of the first application in the three-dimensional environment does not change. In some embodiments, the size of the representation of the first application in the three-dimensional environment does not change.

[0176] The above-described manner of updating the appearance of a representation when a request to make a representation of a first application accessible to a second user is associated with a sharing suggestion provides an efficient manner of presenting a representation in an appearance that is easily visible to both the first user and the second user when a sharing suggestion is made for an application, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (e.g., by reducing the number of inputs required to update the appearance of the representation of the first application), while reducing errors in use.

[0177] In some embodiments, such as Figure 9AIn some embodiments, a three-dimensional environment (e.g., 904) includes a first user (e.g., 914A) and a second user (e.g., 914B), and a representation of a first application (e.g., 908) is accessible to the first user (e.g., 914A) but not to the second user (e.g., 914B) (861). In some embodiments, the second user's electronic device forgoes presenting the representation of the first application. In some embodiments, such as in Figure 9B , while displaying a representation of a first application (e.g., 908) in a first appearance, the electronic device (e.g., 101) detects (859) via one or more input devices a corresponding input corresponding to a request to update the representation of the first application (e.g., 908) to be accessible to the first user (e.g., 914A) and the second user (e.g., 914B). In some embodiments, such as in Figure 9C In some embodiments, in response to detecting the corresponding input (857), the electronic device (e.g., 101) updates (855) the representation (e.g., 908) of the first application to be accessible to the first user (e.g., 914A) and the second user (e.g., 914B), while maintaining the representation (e.g., 908) of the first application in the first appearance. In some embodiments, the electronic devices of the first user and the second user both display the representation of the first application in the first appearance. In some embodiments, the first appearance includes a corresponding size, position, and / or orientation relative to a fixed reference frame in the three-dimensional space. In some embodiments, the orientation of the representation of the first application in the three-dimensional environment does not change. In some embodiments, the position of the representation of the first application in the three-dimensional environment does not change. In some embodiments, the size of the representation of the first application in the three-dimensional environment does not change. In some embodiments, the electronic devices of the first user and the second user display the three-dimensional space from different perspectives and / or viewpoints. In some embodiments, when the first application is accessible to the first user and the second user, the first user and the second user can view the first application, hear audio of the first application, provide input to the first application, and / or move the representation of the first application.

[0178] The above-described manner of maintaining a first appearance of a representation in response to a request to make a representation of a first application accessible to a second user provides an efficient manner of sharing a representation without computing an updated appearance of the representation, which additionally reduces power usage and improves battery life of the electronic device by reducing the complexity of the operation of sharing the representation with a second user, while reducing errors in use.

[0179] In some embodiments, such as Figure 7AIn some embodiments, the three-dimensional environment (e.g., 704) includes a representation (e.g., 706) of a first corresponding object of a first type at a fifth location in the three-dimensional environment, the three-dimensional environment (e.g., 704) includes a representation (e.g., 710) of a second corresponding object of a second type different from the first type (e.g., a type of object that does not dynamically scale when moving around the three-dimensional environment (e.g., 704)) at a sixth location in the three-dimensional environment (e.g., 704), the fifth location being a first distance from a second location associated with a first user (e.g., 714A), and the sixth location being a second distance from the second location associated with the first user (e.g., 714A) (853). In some embodiments, the first object and the second object are representations of applications, content items, files, or other virtual objects. In some embodiments, the first object type is a type of object having a dynamic virtual size based on a distance between the object in the three-dimensional environment and the user. In some embodiments, the second object type is a type of object having a fixed virtual size that does not depend on the distance between the object in the virtual environment and the user. In some embodiments, such as in Figure 7A In some embodiments, such as in Figure 7B In some embodiments, in response to receiving the first corresponding input (849), the electronic device (e.g., 101) moves (847) the representation of the first corresponding object (e.g., 706) to a third distance from the second location of the first user (e.g., 714A). In some embodiments, in response to the first corresponding input, the electronic device updates the orientation of the first corresponding object, as described herein in method 800 and / or method 1000. In some embodiments, such as in Figure 7BIn some embodiments, in response to receiving the first corresponding input (849), the electronic device (e.g., 101) changes (845) the size of the representation (e.g., 706) of the first corresponding object based on the third distance (e.g., relative to the size of the representation when the representation is displayed at the fifth position). In some embodiments, in response to the input of moving the first corresponding object, the electronic device changes the virtual size of the first corresponding object. For example, if the first corresponding object is moved to a distance further away from the second position, the electronic device increases the virtual size of the first corresponding object to maintain the visibility of the first corresponding object (e.g., and the display area of ​​the first object remains the same, increases, or decreases). For another example, if the first corresponding object is moved to a distance closer to the second position, the electronic device reduces the virtual size of the first corresponding object to reserve space in the display area for other objects (e.g., and the display area of ​​the first object remains the same, increases, or decreases). In some embodiments, the electronic device receives (843) via one or more input devices a second corresponding input corresponding to a request to move the representation of the second corresponding object from a second distance from the second position to a fourth distance from the second position that is different from the second distance. Figure 7A In some embodiments, in response to receiving the second corresponding input (841), the electronic device (e.g., 101) moves (839) the representation of the second corresponding object to a fourth distance from the second position while maintaining the size of the representation of the second corresponding object (e.g., relative to the size of the representation when the representation is displayed at the fifth position) (e.g., in response to ...). Figure 7A In some embodiments, the electronic device moves an object (e.g., 710) in a three-dimensional environment (e.g., 704) to different distances from a user (e.g., 714) while maintaining the size of the object (e.g., 710) in the three-dimensional environment (e.g., 704)). In some embodiments, in response to the second corresponding input, the electronic device moves the second corresponding object and continues to display the second corresponding object at a fixed virtual size. In some embodiments, maintaining the fixed virtual size of the corresponding second object includes updating the display area used to display the second corresponding object. For example, if the second corresponding object is moved further away from the user in the three-dimensional environment, the display generation component uses less display area to display the second corresponding object. For another example, if the second corresponding object is moved closer to the user in the three-dimensional environment, the display generation component uses more display area to display the second corresponding object. In some embodiments, a user interface element associated with the second virtual object (e.g., a menu bar, an option that initiates the process of moving the second virtual object when selected and / or interacted with, such as described with reference to method 1000) is displayed with a variable virtual size that depends on the distance between the user and the second corresponding object in the three-dimensional environment, as described with reference to method 1000.

[0180] The above-described manner of changing the size of a first corresponding object and maintaining the size of a second corresponding object provides an effective way of scaling the size of an object to improve visibility and a way of displaying a real virtual object at a fixed virtual size, which further reduces power usage and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0181] 9A to 9D An example is shown of how electronic devices 101a and / or 101b according to some embodiments automatically update the orientation of a virtual object in a three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment.

[0182] Figure 9A The electronic devices 101a and 101b are shown displaying a three-dimensional environment 904 on the user interface via the display generation components 120a and 120b. Figures 1 to 6 As described, electronic devices 101a and 101b optionally include display generation components 120a and 120b (e.g., touch screens) and multiple image sensors 314a and 314b. The image sensors optionally include one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that electronic devices 101a and 101b can use to capture one or more images of the user or a portion of the user when the user interacts with the electronic devices 101a and 101b. In some embodiments, the display generation components 120a and / or 120b are touch screens that can detect gestures and movements of the user's hands. In some embodiments, the user interface shown below can also be implemented on a head-mounted display that includes a display generation component that displays the user interface to the user and sensors that detect the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward relative to the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0183] In some embodiments, the three-dimensional environment 904 is accessible to multiple users. Figure 9A In FIG, electronic device A 101a displays a three-dimensional environment 904 for user A (e.g., from the perspective of user A), and electronic device B 101b displays a three-dimensional environment 904 for user B (e.g., from the perspective of user B). The three-dimensional environment 904 includes a representation 914A of user A, a representation 914B of user B, a representation 908 of application B, a representation 906 of application A, a representation 922 of application C, and a representation 924 of application D. Figure 9A, representation 906 of application A, representation 908 of application B, and representation 924 of application D include content on one side of these representations, while representation 922 of application C includes content visible from multiple sides. For example, applications A, B, and D can be content applications (e.g., a video player, a music player, a photo gallery) or other applications that include a two-dimensional user interface (e.g., a web browser, an email application, a word processing application, a two-dimensional game), while application C can be a virtual board game or other virtual content (e.g., three-dimensional content) visible from multiple sides. Figure 9A In the example shown in , a representation 906 of application A, a representation 922 of application C, and a representation 924 of application D are accessible to user A 914A and user B 914B, and a representation 908 of application B is accessible to user A 914A but not to user B 914B. Thus, both user A 914A and user B 914B are able to view and interact with representation 906 of application A, representation 922 of application C, and representation 924 of application D, while only user A 914A is able to view and interact with representation 908 of application B.

[0184] Legend 912 indicates the position and orientation of objects and users in a three-dimensional environment 904, including a field of view 928a of user A 914A and a field of view 928b of user B 914B. Thus, electronic device A 101a displays a portion of the three-dimensional environment 904 in the field of view 928a of user A 914A, and electronic device B 101b displays a portion of the three-dimensional environment 904 in the field of view 928b of user B 914B. Electronic device A 101a displays a representation 906 of application A and a representation 908 of application B. Electronic device B 101b displays a representation of user A 914A (e.g., an avatar of user A 914A) and a representation 922 of application C. In some embodiments, electronic device B 101b displays an indication of the representation 908 of application B (e.g., its size and / or position) without displaying the content of the representation 908 of application B because user B 914B does not have access to the representation 908 of application B. In some embodiments, electronic device B 101b forgoes the indication to display the representation 908 of application B.

[0185] In some embodiments, the position and / or orientation of the virtual object in the three-dimensional environment 904 is automatically selected (e.g., in response to an input to move one of the objects, in response to an input to initiate display of the object, etc.) based on the position of the users 914A and / or 914B in the three-dimensional environment 904. For example, the electronic device 101a and / or 101b selects / updates the orientation, size, and / or position of a single-sided object, such as the representation 906 of application A and the representation 908 of application B, so that the side with the content of the representations 906 and 908 is oriented toward the users A 914A and B 914B in the three-dimensional environment 904. For another example, the electronic device 101a and / or 101b selects / updates the orientation, size, and / or position of a multi-sided object, such as the representation 922 of application C, to be positioned between the users A 914A and B 914B. In some embodiments, electronic devices 101a and 101b construct and / or have access to a vector field superimposed on the three-dimensional environment 904, wherein vectors point to a shared point of interest of users 914A and 914B. In some embodiments, the vector field is three-dimensional. In some embodiments, the vector field is the sum of a vector field pointing to the position of user A 914A in the three-dimensional environment 904 and a vector field pointing to the position of user B 914B in the three-dimensional environment 904 (or, more generally, the sum of vector fields each pointing to one of multiple users in the three-dimensional environment, including the vector fields of each user in the three-dimensional environment 904). Therefore, if a virtual object (e.g., a representation of an application) is placed at a height above the eye level of one or more users, electronic devices 101a and 101b orient the virtual object so that it faces downward toward the user's viewpoint. The users' shared point of interest is optionally a location in the three-dimensional environment 904 between the users, toward which the virtual object is oriented so that the virtual object is visible to both users in the three-dimensional environment 904.

[0186] In some embodiments, when determining the orientation of an object, the electronic device 101a and / or 101b prioritizes users who are closer to the corresponding virtual object over users who are farther away from the virtual object. For example, the representation 924 of application D is closer to user B 914B in the three-dimensional environment 904, so the orientation of the representation 924 of application D is more toward user B 914B than toward user A 914A. In some embodiments, when a first user of a corresponding virtual object is blocked (e.g., by another user or another virtual object), the orientation of the corresponding virtual object is based on the positions of the other users (whose view of the corresponding object is not blocked) in the three-dimensional environment 904, and is not based on the position of the first user (or any other user whose view of the corresponding virtual object is blocked). For example, if in Figure 9A, if there is a virtual object between user A 914A and the representation 906 of application A in the three-dimensional environment 904 in FIG, the orientation and / or size of the representation 905 of application A will be based on the position of user B 914B and not based on the position of user A 914A. In some embodiments, when a first user cannot view the corresponding virtual object because the distance between the first user and the corresponding virtual object exceeds a threshold value (e.g., 4 meters, 5 meters, 10 meters, etc.), the orientation of the corresponding virtual object is based on the positions of other users (who are within the threshold distance of the corresponding virtual object) in the three-dimensional environment 904, and not based on the position of the first user (or any other user who is farther than the threshold distance from the corresponding virtual object). For example, if Figure 9A If the three-dimensional environment 904 shown in FIG includes an additional object that is within a threshold distance of user A 914A but further than the threshold distance from user B 914B, the orientation and / or size of the additional object will be based on the position of user A 914A and not on the position of user B 914B.

[0187] In some embodiments, constructing a vector field corresponding to the automatic orientation of the virtual object includes establishing a region of the three-dimensional environment 904 beyond which the virtual object has an orientation based on the boundaries of the region and not based on the position of the user in the three-dimensional environment. Figure 9A In the example, user A 914A and user B 914B are located within the region, and the orientation of any virtual object (e.g., a representation of application C 922) within the region of the three-dimensional environment 904 is based on the location of one or more users, as described above and below. In this example, outside the region, electronic devices 101a and 101b orient virtual objects to positions closest to the location of the virtual object facing the region boundary. Thus, in some embodiments, regardless of the distance of the object from the region boundary, an object located closest to a corresponding point along the region boundary outside the region will have the same corresponding orientation, regardless of the distance of the object from the region boundary. In some embodiments, within the region, even if the object remains closest to a corresponding point along the region boundary, changing the distance of the object from the corresponding point along the region boundary also changes the object's orientation. In some embodiments, objects outside the region and within a threshold distance (e.g., 0.2 meters, 0.5 meters, 0.7 meters, 1 meter, etc.) of the region boundary are oriented based on the user's location in the three-dimensional environment 904 (e.g., a criterion for orienting objects within the region) and the region boundary (e.g., a criterion for orienting objects outside the region). Thus, in some embodiments, the electronic device 101a and / or 101b is able to "blend" different algorithms used inside and outside the area when placing an object outside the area and within a threshold distance of the area.

[0188] In some embodiments, electronic devices 101a and / or 101b are described above with reference to 9A to 10T The orientation of the virtual object is automatically updated in response to a user request to move the virtual object within the three-dimensional environment 904 and / or when the virtual object is initially displayed in the three-dimensional environment 904. For example, Figure 9A The orientation of the representation 906 of application A in the three-dimensional environment 904 is automatically selected based on the positions of user A 914A and user B 914B in the three-dimensional environment 904. In some embodiments, in response to a request to display a virtual object in the three-dimensional environment 904, the electronic device 101a and / or 101b selects an initial position for the virtual object and selects an orientation based on the positions of the users 914A and 914B and other objects in the three-dimensional environment 904. In some embodiments, the initial position of the virtual object is not based on the orientation of the users 914A and 914B (e.g., the users' fields of view 928a and 928b). For example, the initial position of the representation 906 of application A is Figure 9A Regardless of the location of user A 914A and user B 914B, Figure 9A , which directions the virtual objects are facing in the three-dimensional environment 904 when in the position shown in FIG. Thus, in some embodiments, the initial placement of virtual objects in the three-dimensional environment is based on the position of the user in the three-dimensional environment. For example, the representation 906 of application A and the representation 924 of application D are placed at a position where both users 914A and 914B can see one side of the representation, including the content of the representation, while the representation 922 of application C is placed between users 914A and 914B so that the users can view different sides of the representation (e.g., for a virtual board game).

[0189] Figure 9B Shown above reference Figure 9A Another example of a three-dimensional environment 904 is described. Figure 9B In the example, users 914A and 914B and other objects are positioned differently in the three-dimensional environment 904 than they were in Figure 9A In some embodiments, the initial positions and orientations of the representation 906 of application A, the representation 924 of application D, the representation 922 of application C, and the representation 908 of application B are in Figure 9B In and in Figure 9A , because the initial positions and orientations of these objects are based on the positions of users 914A and 914B in the three-dimensional environment, and the positions of users 914A and 914B are different in Figure 9B In and with them Figure 9A Different.

[0190] It should be understood that in some embodiments, Figure 9A and Figure 9BThe initial position and orientation of the object in the three-dimensional environment 904 or the automatically selected orientation of the object in response to user input for moving the object is shown. In some embodiments, the electronic devices 101a and / or 101b do not update the orientation and / or position of the virtual object in response to one or more users updating their position in the three-dimensional environment 904. In response to detecting that one or more of the users 914A or 914B change their position within the three-dimensional environment 904, the electronic devices 101a and 101b forgo updating the position and orientation of the virtual object, but update the vector field used to determine the initial position and orientation of new objects added to the three-dimensional environment 904 and the orientation of objects moving around the three-dimensional environment 904.

[0191] Figure 9B Legend 912 of FIG. 1 illustrates a field of view 928a of user A 914a and a field of view 928b of user B 914B in a three-dimensional environment 904. Electronic device 101a displays the three-dimensional environment 904 from the viewpoint of user A 914a, including a representation 906 of application A and a representation 908 of application B. Electronic device 101b displays the three-dimensional environment 904 from the viewpoint of user B 914b, including a representation 906 of application A, a representation 922 of application C, and a representation (e.g., an avatar) of user A 914A. Although user B 914B's field of view 928b includes representation 908 of application B as shown in legend 912, electronic device B 101b does not display representation 908 of application B because representation 908 of application B is not accessible to user B.

[0192] Furthermore, because user B 914B does not have access to the representation 908 of application B, in some embodiments, the orientation of the representation 908 of application B is based on the position of user A 914A and not on the position of user B 914B in the three-dimensional environment 904. For example, the representation 908 of application A and the representation 924 of application D are oriented toward both user A 914A and user B 914B, but the representation 908 of application B is oriented toward user A 914A and at an angle that would make it difficult for user B 914B to view the contents of the representation 908 if user B 914B did have access to the representation 908.

[0193] In some embodiments, the electronic device 101a and / or 101b initially displays a representation of an application in a private mode in which the user requesting the display has access to the representation of the application, while other users in the three-dimensional environment 904 do not have access to the representation. For example, in response to detecting a request from user A 914A to display a representation 908 of application B, electronic device A 101a displays the representation 908 in a private mode such that user A 914A has access to the representation 908 and user B 914B does not have access to the representation 908 until a request is received from user A 914A to make the representation 908 accessible to user B 914B. In some embodiments, if there are additional users in the three-dimensional environment 904 in addition to user A 914A and user B 914B, the request to make the representation 908 of application B accessible to user B 914B is a request to make the representation 908 accessible to all users in the three-dimensional environment 914 or a request to make the representation 908 accessible to user B 914B and inaccessible to other users (e.g., with or without maintaining the ability of user A 914A to access the representation 908).

[0194] Figure 9B 908 of application B is made accessible to user B 914b, and electronic device B 101b detects input moving the representation 906 of application A. For example, electronic device A 101a detects user A 914A's gaze 916a pointing to option 926 to share the representation 908 of application B, while simultaneously detecting user A 914A performing a gesture with their hand 920a. Electronic device B 101b detects user B 914B's gaze 916b pointing to the representation 906 of application A, while simultaneously detecting user B 914B performing a gesture with their hand 920b and moving their hand 920b while maintaining the gesture. In some embodiments, the gesture detected by the electronic device is the user touching their thumb to another finger (e.g., index finger, middle finger, ring finger, pinky finger) on the same hand as the thumb.

[0195] In some embodiments, electronic device A 101a displays an indication 918 around the representation 908 of application B that indicates the user's gaze 916a is on the representation 908. In some embodiments, the indication 918 is displayed in response to detecting the user's gaze 916a on the representation 908 of application B. In some embodiments, in response to detecting the gaze of user A 914A on the indication 918, electronic device A 101a displays an option 926 to make the representation 908 of application B accessible to user B 914B. In some embodiments, electronic device A 101a displays additional options (e.g., an option to move the representation 908, an option to stop displaying the representation 908, etc.) in response to detecting the user's gaze on the indication 918. Although the indication 918 is Figure 9B 908, but in some embodiments, the indication 918 is displayed adjacent to an edge of the representation 908 (e.g., the indication 918 is a bar along the bottom of the representation 908 or along another edge). Electronic device A 101a detects user A 914a's gaze 916a on the option 926 to share the representation 908 of application B and simultaneously detects user A 914a performing a gesture with their hand 920a (e.g., touching their thumb to another finger (e.g., index finger, middle finger, ring finger, pinky finger) on the same hand as the thumb). In response, electronic device A 101a makes the representation 908 of application B accessible to user B 914B, as will be referenced below. Figure 9C Described in more detail.

[0196] In some embodiments, Figure 9B In FIG. 1 , electronic device B 101 b displays an indication 918 around the representation 906 of application A indicating that the user's gaze 916 b is on the representation 906. In some embodiments, the indication 918 is displayed in response to detecting the user's gaze 916 b on the representation 906 of application A. Although the indication 918 is Figure 9B 906, but in some embodiments, the indication 918 is displayed adjacent to one edge of the representation 906 (e.g., the indication 918 is a bar along the bottom of the representation 906 or along another edge). Electronic device B 101b detects the gaze 916b of user B 914b on the indication 918 surrounding the representation 906 and simultaneously detects user B 914b performing a gesture with their hand 920b (e.g., touching their thumb to another finger on the same hand as the thumb (e.g., index finger, middle finger, ring finger, pinky finger)). In response, electronic device A 101b initiates a process of moving the representation 906 of application A in the three-dimensional environment 904 in accordance with the movement of the hand 920b (e.g., or the arm of hand 920b), as will be referenced below. Figure 9CIn some embodiments, any user with access to a corresponding virtual object in the three-dimensional environment 904 can similarly move the virtual object in the three-dimensional environment 904. For example, user A 914A can move the representation 906 of application A in a manner similar to how user B 914B moves the representation 906 of application A.

[0197] Figure 9C 906 of application B is received and the mobile Figure 9B 908 of application A. In response to user A 914A's request to make application B's representation 908 accessible to user B 914B, electronic device B 101b displays application B's representation 908 in the three-dimensional environment 904. In response to user B's 914B's request to move application A's representation 906, electronic devices 101a and 101b update the position, orientation, and size of application A's representation 906 in the three-dimensional environment 904.

[0198] In some embodiments, the electronic device prevents virtual objects from being placed within a threshold distance (e.g., 0.1 meters, 0.2 meters, 0.3 meters, 0.4 meters, 0.5 meters, etc.) of the user in the three-dimensional environment 904. For example, the electronic device avoids placing objects within the user's personal space so that the user's view of the rest of the three-dimensional environment is not obscured. Figure 9B The movement of the hand 920b of user B 914B shown in FIG may correspond to moving the representation 906 of application A to a position within a threshold distance of user A 914A, but in response to Figure 9BIn response to a request to move a representation 906 of application A provided by user B 914b in the three-dimensional environment, the electronic device displays the representation 906A of application A at a location in the three-dimensional environment 904 that is a threshold distance from user A 914A in the three-dimensional environment. In some embodiments, before the movement input is terminated (e.g., detecting that user B 914b stops performing a predetermined gesture associated with the movement input (e.g., touching a thumb to another finger (e.g., index finger, middle finger, ring finger, pinky finger) on the same hand as the thumb)), the electronic device 101a and / or 101b moves the representation 906 of application A in accordance with the movement of user B's hand 920b, including displaying the representation 906 of application A at a location within the threshold distance of user A 914A. In some embodiments, in response to detecting the termination of the movement input, the electronic device 101a and / or 101b displays an animation of the representation 906 of application A moving from the location within the threshold distance of user A 914A to a location at least the threshold distance from user A 914A. In some embodiments, when the movement input is being provided, the electronic device 101a and / or 101b does not display the representation 906 of application A at a location within a threshold distance of user A 914A, and displays the representation 906 at a threshold distance from user A 914A in response to the movement input corresponding to moving the representation 906 to a location within the threshold distance of user A 914A.

[0199] In some embodiments, the electronic device 101a and / or 101b does not allow users 914A and 914B to place a virtual object in a position where it is impossible for both users 914A and 914B to view the content of the virtual object. For example, the electronic device 101a and / or 101b also prevents one-way representations of applications (e.g., corresponding representations 906, 908, and 924 of application A, application B, and application D that include content on one side) from being placed between two users in the three-dimensional environment 904 (e.g., user A 914A and user B 914B) because if the one-way representations are between the users, the side of the representations with content cannot face both users. In some embodiments, in response to a movement input corresponding to moving the one-way representation of the application to a position between user A 914A and user B 914B (e.g., an input to move representation 906 or 908 to the position of representation 922 (e.g., when representation 922 is not displayed on the left side of the representation), the one-way representation of the application is moved to a position between user A 914A and user B 914B. Figure 9B), the electronic device 101a and / or 101b displays the representation at a location that is at least a threshold distance (e.g., 5 centimeters, 10 centimeters, 15 centimeters, 20 centimeters, etc.) from a line directly between the two users 914A and 914B. In some embodiments, while the movement input is being provided, the electronic device 101a and / or 101b displays the representation in accordance with the movement input (e.g., including displaying the representation between the two users 914A and 914B) and, in response to detecting termination of the movement input, moves the representation to a location that is at least a threshold distance (e.g., 5 centimeters, 10 centimeters, 15 centimeters, 20 centimeters, etc.) from a line directly between the two users 914A and 914B. In some embodiments, while the movement input is being provided, in response to an amount of movement corresponding to moving the representation to a position between the two users 914A and 914B while the movement input is being provided, the electronic device 101a and / or 101b displays the representation at least a threshold distance (e.g., 5 centimeters, 10 centimeters, 15 centimeters, 20 centimeters, etc.) from a line directly between the two users 914A and 914B. In some embodiments, in response to detecting the movement input corresponding to moving the representation across the line directly between the two users, the electronic device 101a and / or 101b moves the representation from one side of the line between the two users 914A and 914B to the other side of the line between the two users 914A and 914B.

[0200] In some embodiments, in response to a movement input such as Figure 9B The electronic device 101a and / or 101b adjusts the size of the representation 906 of application A based on the positions of user A 914A and user B 914B. Figure 9C It is in Figure 9B is closer to the two users, so the size of the representation 906 of application A in the three-dimensional environment 904 is Figure 9C It is in Figure 9B , as shown in the legend 912 of both figures. Similarly, in response to input that moves the representation of the application further away from users 914A and 914B in the three-dimensional environment 904, electronic devices 101a and / or 101b increase the size of the representation in the three-dimensional environment 904. In some embodiments, electronic devices 101a and / or 101b display the representation 906 of application A at a size that is large enough to be readable by a user (e.g., user B 914B) who is further away from the updated position of the representation 906 of application A. For example, in Figure 9C, the size of the representation 906 of application A is selected to be large enough to make it legible for user B 914B because user B 914B is farther from the representation 906 than user A 914A. In some embodiments, the electronic devices 101a and / or 101b do not update the size of some objects in response to a change in the distance between the object and the user. In some embodiments, even if the electronic devices 101a and / or 101b maintain the size of the object in response to a change in the distance between the object and the users 914A and 914B, the size of the indication 918 displayed is updated in response to detecting the user's gaze on the object (e.g., becoming smaller when closer to the user and becoming larger when farther away from the user).

[0201] Therefore, in some embodiments, as shown in reference Figure 9B and Figure 9C As described above, when a representation of an application is accessible to multiple users 914A and 914B in a three-dimensional environment 904, the orientation and / or size of the representation is updated based on the positions of the multiple users 914A and 914B in the three-dimensional environment 904. In some embodiments, when a representation of an application is accessible to only one user in the three-dimensional environment 904, the orientation and / or size of the representation is updated based on the positions of the multiple users 914A and 914B in the three-dimensional environment 904. Figures 7A to 8P In some embodiments, the orientation and / or position of the representation of the application is updated in response to input (e.g., Figure 9B The representation 908 of application B remains the same based on the input provided by user A 914A in the representation and is not automatically updated until the representation is repositioned when it is accessible to multiple users. For example, in response to being shared, the position, orientation and / or size of the representation 908 of application B is changed to Figure 9C In and with them Figure 9B 908 is the same as before the input to share the representation 908 is received by electronic device A 101a. In some embodiments, if one of users 914A or 914B moves the representation 908, the electronic devices 101a and / or 101b will move the representation 908 according to the input and automatically update the size and orientation of the representation 908 according to the position of the representation 906, user A 914A, and user B 914B, so that user A 914A and user B 914B can view the representation 906 after (and optionally, simultaneously with) receiving the movement input.

[0202] As above reference Figures 9A to 9CAs described above, in some embodiments, the electronic device 101a and / or 101b automatically updates the orientation and / or size of the virtual object based on the position of user A 914A and user B 914B in the three-dimensional environment 904 (e.g., in response to a request to move the virtual object when the virtual object is initially displayed). Therefore, in some embodiments, if the number of users in the three-dimensional environment 904 changes (e.g., by adding or removing users), the automatic orientation of the corresponding object at the corresponding position in the three-dimensional environment 904 will also change.

[0203] Figure 9D A display of a three-dimensional environment 904 at each of electronic devices A 101a, B 101b, and C 101c is shown. Electronic device A 101a shows a portion of the three-dimensional environment 904 within the field of view 928a of user A 914A, including a representation of application A 906 and a representation of application B 908. Electronic device B 101b shows a portion of the three-dimensional environment 904 within the field of view 928b of user B 914B, including a representation of application A 906, a representation of application B 908, a representation of application C 922, and a representation of user A 914A. Electronic device C 101c shows a portion of the three-dimensional environment 904 within the field of view 928c of user C 914c, including a representation of application A 906, a representation of application B 908, a representation of application C 922, and a representation of user A 914A.

[0204] In some embodiments, in response to user C 914C joining the three-dimensional environment 904, the electronic devices 101a, 101b, and / or 101c automatically select an initial position for user C 914C in the three-dimensional environment 904. Figures 13A to 14L An exemplary manner in which an electronic device selects an initial position of a user in a three-dimensional environment is described in more detail. In some embodiments, the electronic device automatically selects the user's position in response to the user joining the three-dimensional environment or in response to receiving user input to reset the user's view of the three-dimensional environment. For example, in response to detecting a request to reset the three-dimensional environment, the electronic device automatically selects the user's position according to one or more rules (e.g., as described below with reference to Figures 13A to 14L In this way, the electronic device can update the display of objects and other users in the three-dimensional environment for the user without updating the positions and / or orientations of the other users and objects in the three-dimensional environment.

[0205] Figure 9D The previous reference is shown in the case of adding user C 914C Figure 9C Described three-dimensional environment 904. Figure 9DRespective representations 906, 908, 922, and 924 of Application A, Application B, Application C, and Application D are shown, as they appear in the three-dimensional environment 904 with respect to these objects. Figure 9C The orientations of the respective representations 906, 908, 922, and 924 of Application A, Application B, Application C, and Application D are shown in FIG. Figure 9D In and with them Figure 9C is different because the users present in the three-dimensional environment 904 are arranged in Figure 9D In and in Figure 9C In some embodiments, electronic devices 101a, 101b, and / or 101c do not update the orientations of respective representations 906, 908, 922, and 924 of Application A, Application B, Application C, and Application D in response to User C 914C being added to the three-dimensional environment 904. Instead, Figure 9D Optionally, respective representations 906, 908, 922, and 924 of Application A, Application B, Application C, and Application D are shown where the objects are to be moved. Figure 9D In some embodiments, in response to user C 914C joining the three-dimensional environment 904, as shown in FIG. Figure 9D Updating the orientation and / or size of respective representations 906, 808, 922, and 924 of Application A, Application B, Application C, and Application D is shown in FIG.

[0206] For example, the corresponding representations 908 and 906 of Application B and Application A are located to the right of these objects due to user C 914C. Figure 9D China and Belgium Figure 9C 914C, so that representations 908 and 906 are at least partially visible to user C 914C (e.g., via electronic device C 101c). As another example, representation 922 of application C is rotated counterclockwise so that one side of representation 922 is oriented toward both user B 914B and user C 914C, thereby enabling user C 914C to view at least a portion of representation 922 of application C. Figure 9D , user A 914A is still able to view the other side of the representation 922 of application C because the representation 922 of application C includes content on multiple sides (e.g., the side facing the user, all sides, etc.).

[0207] 10A to 10T is a flow chart illustrating a method for automatically updating the orientation of a virtual object in a three-dimensional environment based on the viewpoints of multiple users in the three-dimensional environment according to some embodiments. In some embodiments, the method 1000 is executed on a computer system (e.g., Figure 1The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4 In some embodiments, method 1000 is performed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., a computer readable medium, a display generating component 120 in the computer system) (e.g., a heads-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward toward the user's hand (e.g., a color sensor, an infrared sensor, and other depth sensing cameras) or a camera pointing forward from the user's head). Figure 1 Some operations in method 1000 may be optionally combined, and / or the order of some operations may be optionally changed.

[0208] In some embodiments, method 1000 is performed at an electronic device (e.g., a mobile device (e.g., a tablet, smart phone, media player, or wearable device) or a computer that communicates with a display generating component and one or more input devices. In some embodiments, the display generating component is a display integrated with the electronic device (optionally a touch screen display), an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users, etc. In some embodiments, the one or more input devices include an electronic device or a component capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information associated with the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor), etc.

[0209] In some embodiments, such as Figure 9AIn some embodiments, an electronic device (e.g., 101) displays (1002) a three-dimensional environment (e.g., 904) shared by a first user (e.g., 914A) and a second user (e.g., 914B) via a display generation component 120. In some embodiments, the three-dimensional environment is generated, displayed, or otherwise enabled to be viewed by a device (e.g., the three-dimensional environment is a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment). In some embodiments, an electronic device associated with the first user displays the three-dimensional environment from a viewpoint corresponding to the position and orientation of the first user in the three-dimensional environment. In some embodiments, another electronic device associated with the second user (e.g., different from the electronic device) displays the three-dimensional environment from a viewpoint corresponding to the position and orientation of the second user in the three-dimensional environment. In some embodiments, the three-dimensional environment includes virtual objects representing physical objects in the environment in which the first user is located and / or virtual objects representing physical objects in the environment in which the second user is located. In some embodiments, the location in the three-dimensional environment corresponding to the first user is a location corresponding to the physical location of an electronic device associated with the first user within the physical environment of the electronic device associated with the first user. In some embodiments, the location in the three-dimensional environment corresponding to the second user is a location corresponding to the physical location of another electronic device associated with the second user within the physical environment of another electronic device associated with the second user. In some embodiments, the electronic device displays an avatar of the first user at a location associated with the first user in the three-dimensional environment, and / or displays an avatar of the second user at a location corresponding to the second user in the three-dimensional environment.

[0210] In some embodiments, the electronic device (e.g., 101) receives (1004) via one or more input devices an input corresponding to a request to display a corresponding representation of a corresponding application in a three-dimensional environment (e.g., Figure 9A In some embodiments, the input includes input associated with a request to launch a corresponding application and / or a request to display another representation of a corresponding application already running on the electronic device. In some embodiments, the corresponding representation of the corresponding application is a user interface of the corresponding application. In some embodiments, the corresponding representation of the corresponding application is a virtual object associated with the corresponding application.

[0211] In some embodiments, such as Figure 9AIn some embodiments, in response to receiving input (1006) corresponding to a request to display a corresponding representation of a corresponding application, based on a determination that a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B) are at a first arrangement (e.g., spatial arrangement) of viewpoints including a first plurality of positions in a three-dimensional environment (e.g., 904) (e.g., the viewpoints of the first user and the viewpoints of the second user are at a first plurality of positions (e.g., and orientations) in the three-dimensional environment), the electronic device (e.g., 101) displays (1008) a corresponding representation (e.g., 906) of the corresponding application at a first position in the three-dimensional environment (e.g., 904) based on the first arrangement of the viewpoints of the first user (e.g., 914A) and the viewpoints of the second user (e.g., 914B). In some embodiments, the first position is a position that satisfies one or more placement criteria. In some embodiments, the one or more placement criteria include criteria that are satisfied based at least in part on the positions and, optionally, orientations of the first and second users in the three-dimensional environment. In some embodiments, the one or more placement criteria include a criterion that is satisfied when both users are able to see the respective representations of the respective applications from their viewpoints in the three-dimensional environment. In some embodiments, the placement criteria include a criterion that is satisfied when the respective representations are oriented such that the content of the respective representations faces the positions of the first user and the second user in the three-dimensional environment.

[0212] In some embodiments, such as Figure 9BIn some embodiments, in response to receiving input (1006) corresponding to a request to display a corresponding representation of a corresponding application, based on a determination that a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B) are in a second arrangement (e.g., spatial arrangement) of viewpoints at a second plurality of positions in a three-dimensional environment (e.g., 904) that is different from the first plurality of positions in the three-dimensional environment (e.g., 904) (e.g., the viewpoints of the first user and the viewpoints of the second user are at a second plurality of positions (e.g., and orientations) in the three-dimensional environment), the electronic device (e.g., 101) displays (1010) a corresponding representation (e.g., 906) of the corresponding application at a second position in the three-dimensional environment (e.g., 904) based on the second arrangement of the viewpoint of the first user (e.g., 914A) and the viewpoint of the second user (e.g., 914B). In some embodiments, the difference between the first plurality of positions and the second plurality of positions is a difference in the position of the viewpoint of one of the users, while the positions of the viewpoints of one or more other users are the same. In some embodiments, if the viewpoints of the first and second users are swapped, the location at which the corresponding representation is displayed is the same. In some embodiments, the location at which the corresponding representation is displayed depends on the location of the user's viewpoint, regardless of the user's viewpoint orientation. In some embodiments, the location at which the corresponding representation is displayed depends on both the location and orientation of the user's viewpoint. In some embodiments, the electronic device displays the corresponding representation of the corresponding application at different locations depending on the location of the viewpoints of the first and second users in the three-dimensional environment. In some embodiments, the representation of the corresponding application is a bidirectional representation intended to be viewed by the first and second users from opposite sides of the corresponding representation of the corresponding application. For example, the corresponding representation is a virtual board game, and the electronic device is configured to display the corresponding representation at a location between the location of the first user's viewpoint and the location of the second user's viewpoint. As an example, if the first user's location is to the right of the second user's location relative to a reference frame, the electronic device displays the corresponding representation to the right of the second user and to the left of the first user relative to the reference frame. In this example, if the first user is to the left of the second user relative to the reference frame, the electronic device displays the corresponding representation to the left of the second user and to the left of the first user relative to the reference frame.

[0213] The above-described manner of displaying corresponding representations of corresponding applications at positions depending on the position of the user's viewpoint in the three-dimensional environment provides an efficient manner of displaying corresponding representations at positions convenient for the user, which in addition reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0214] In some embodiments, such as Figure 9BIn some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first location in a three-dimensional environment (e.g., 904), wherein the corresponding representation (e.g., 906) of the corresponding application is displayed in a first orientation relative to the three-dimensional environment (e.g., 904) based on a first arrangement of a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B), the electronic device (e.g., 101) receives (1012) via one or more input devices a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application to a third location in the three-dimensional environment (e.g., 904). In some embodiments, the input includes a selection of an option that, when selected, causes the electronic device to initiate a process of moving the corresponding representation. In some embodiments, detecting input includes detecting, via a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)), the user performing a gesture (e.g., tapping their thumb against a finger on the same hand as the thumb (e.g., index finger, middle finger, ring finger, pinky finger)), moving their hand while performing the gesture, and releasing the gesture. In some embodiments, the electronic device moves a representation based on the movement of the user's hand. In some embodiments, such as in Figure 9C In response to receiving the corresponding input (1014), the electronic device (e.g., 101) displays (1016) a corresponding representation (e.g., 906) of the corresponding application at a third location in the three-dimensional environment (e.g., 904). Figure 9C In some embodiments, in response to receiving the corresponding input (1014), the electronic device (e.g., 101) updates (1018) the corresponding representation (e.g., 906) of the corresponding application to be displayed in a second orientation different from the first orientation based on the third position and the viewpoint of the first user (e.g., 914A) and the viewpoint of the second user (e.g., 914B). In some embodiments, when the electronic device moves the corresponding representation according to the corresponding input, the electronic device displays an animation in which the corresponding representation moves and changes orientation as the corresponding representation moves from the first position to the third position. In some embodiments, the second orientation is an orientation facing the viewpoint of the first user and the viewpoint of the second user when the corresponding representation is in the third position. In some embodiments, the corresponding representation is updated according to one or more steps of methods 800 and / or 1200.

[0215] The above-described manner of updating the position of the corresponding representation based on the third position and the first arrangement of the viewpoint of the first user and the viewpoint of the second user provides an efficient manner of displaying the corresponding representation in an orientation visible to the first user and the second user, which further reduces power usage and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by reducing the input required to adjust the orientation of the representation based on changes in the position of the representation), while reducing errors in use.

[0216] In some embodiments, such as Figure 9C , when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first location in a three-dimensional environment (e.g., 904), wherein the corresponding representation (e.g., 906) of the corresponding application is displayed in a first orientation relative to the three-dimensional environment (e.g., 904) based on a first arrangement of a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B), the electronic device (e.g., 101) detects (1020) that the first arrangement of the viewpoint of the first user (e.g., 914A) and the viewpoint of the second user (e.g., 914B) has changed to a third arrangement of the viewpoint of the first user (e.g., 914A) and the viewpoint of the second user (e.g., 914B), wherein the third arrangement includes viewpoints at a third plurality of locations in the three-dimensional environment (e.g., 904) that are different from the first plurality of locations in the three-dimensional environment (e.g., if Figure 9C In some embodiments, user A (e.g., 914A) and / or user B (e.g., 914B) of the first user and the second user want to change positions within the three-dimensional environment (e.g., 904). In some embodiments, the position of the viewpoint of one or more of the first user and the second user has changed. For example, one or more of the users changes positions within the three-dimensional environment. In some embodiments, the electronic device detects input corresponding to a request to change the position of the first user of the electronic device. In some embodiments, the request to change the position of the first user of the electronic device is a movement of the electronic device within the physical environment of the electronic device detected via one or more accelerometers, gyroscopes, distance sensors, cameras, etc. In some embodiments, in response to detecting that the first arrangement of the viewpoint of the first user and the viewpoint of the second user has changed, the electronic device (e.g., 101) maintains (1022) a corresponding representation (e.g., 906) of the corresponding application at a first position in a first orientation relative to the three-dimensional environment (e.g., 904), such as in Figure 9C In some embodiments, the electronic device does not change the orientation or position of the corresponding representation in response to detecting a change in the position of one or more of the users in the three-dimensional environment. In some embodiments, the electronic device updates the three-dimensional environment from the viewpoint of the user moving in accordance with one or more steps of method 800.

[0217] The above-mentioned manner of maintaining the corresponding representations at the first position and the first orientation in the three-dimensional environment when the arrangement of the viewpoint of the first user and the viewpoint of the second user changes provides an efficient way to update the arrangement of the viewpoint of the first user and the viewpoint of the second user without calculating the new position or orientation for updating the corresponding representations, which further reduces power usage and improves the battery life of the electronic device by reducing the computational complexity of the electronic device.

[0218] In some embodiments, such as Figure 9B , when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first position in a first orientation relative to a three-dimensional environment (e.g., 904), the electronic device (e.g., 101) detects (1024) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application away from the first position in the three-dimensional environment (e.g., 904). In some embodiments, the electronic device detects a selection of an option that, when selected, causes the electronic device to initiate a process of moving the corresponding representation within the three-dimensional environment. In some embodiments, detecting the input includes: detecting, via a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)), a user performing a gesture (e.g., tapping their thumb against a finger on the same hand as the thumb (e.g., index finger, middle finger, ring finger, pinky finger)), moving their hand while performing the gesture, and releasing the gesture. In some embodiments, the electronic device moves the representation based on the movement of the user's hand. In some embodiments, such as in Figure 9C In some embodiments, in response to detecting a corresponding input, the electronic device (e.g., 101) moves a corresponding representation (e.g., 906) of the corresponding application from a first position (1026) according to the corresponding input, and updates the corresponding representation (e.g., 906) to a second orientation different from the first orientation in the three-dimensional environment (e.g., 906) based on a third arrangement of the viewpoint of the first user (e.g., 914A) and the viewpoint of the second user (e.g., 914B). In some embodiments, when an input to move the corresponding representation is received, the electronic device updates the orientation of the representation based on the arrangement of the user's viewpoint. For example, when the user's viewpoint is in the first arrangement, the arrangement of the viewpoint is changed to the updated arrangement, and the input to move the corresponding representation is received, the representation is initially displayed in the first position and orientation. In this example, the orientation of the corresponding representation changes based on the updated arrangement of the user's viewpoint and the updated position of the corresponding representation in response to the input to move the corresponding representation (e.g., but not before the input to move the corresponding representation is detected). In some embodiments, the second orientation of the corresponding representation based on the third arrangement of the viewpoint of the first user and the viewpoint of the second user is an orientation facing the viewpoint of the user at the current position of the corresponding representation. In some embodiments, the corresponding representation is updated according to one or more steps of methods 800 and / or 1200.

[0219] The above-described manner of updating the orientation of a corresponding representation based on a third arrangement of the viewpoint of the first user and the viewpoint of the second user provides an efficient manner of displaying a representation in an orientation visible to a user when moving the representation, which further reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the input required to adjust the orientation of the corresponding representation), while reducing errors in use.

[0220] In some embodiments, such as Figure 9B In some embodiments, the electronic device (e.g., 101) detects (1030) an input corresponding to a selection of the corresponding user interface element (e.g., 918) via one or more input devices (e.g., in the example of FIG. 1028). Figure 9B In some embodiments, if the user's gaze (e.g., 916b) is on the user interface element (e.g., 918) when the user performs a gesture with their hand (e.g., 920b). In some embodiments, in response to input corresponding to selection of the corresponding user interface element (e.g., 918), the electronic device (e.g., 101) initiates (1032) a process of moving the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904). In some embodiments, in response to a second input directed to the corresponding user interface element, the electronic device updates the corresponding user interface element to include a plurality of selectable options, the plurality of selectable options including a selectable option that, when selected, causes the electronic device to initiate the process of moving the corresponding representation. In some embodiments, the corresponding user interface element is a system object that is displayed in association with each of the other application representations and other objects in the user interface. In some embodiments, the corresponding user interface element is displayed in response to detecting, via an eye tracking device, that the user's gaze is directed to the corresponding representation of the corresponding application. In some embodiments, the electronic device moves the representation in response to detecting via a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)) that the user performs a gesture while looking at the corresponding user interface element (e.g., tapping their thumb against a finger on the same hand as the thumb (e.g., index finger, middle finger, ring finger, pinky finger)), moves their hand while performing the gesture, and releases the gesture. In some embodiments, the electronic device moves the representation based on the movement of the user's hand. In some embodiments, the corresponding user interface element is displayed adjacent to an edge (e.g., the bottom edge) of the representation of the corresponding application.

[0221] The above-described manner of displaying the corresponding representation with corresponding user interface elements that are interactive to move the corresponding representation in the three-dimensional environment provides an efficient manner of initiating the process of moving the corresponding representation, which in addition reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0222] In some embodiments, such as Figure 9B In some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first distance from a viewpoint of a first user 614A in a three-dimensional environment (e.g., 904), wherein the corresponding representation (e.g., 906) of the corresponding application is displayed at a first size from the viewpoint of the first user (e.g., 914A), the electronic device (e.g., 101) detects (1034) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application to a second distance from the viewpoint of the first user (e.g., 914A) that is greater than the first distance. Figure 9C In response to detecting the corresponding input (1036), the electronic device (e.g., 101) moves (1038) the corresponding representation (e.g., 906) of the corresponding application to a second distance from the viewpoint of the first user (e.g., 914A) (e.g., displays the corresponding representation of the corresponding application at an updated position that is the second distance from the viewpoint of the first user in the three-dimensional environment). In some embodiments, such as in Figure 9C In some embodiments, in response to detecting the corresponding input (1036), the electronic device (e.g., 101) displays (1040) the corresponding representation (e.g., 906) of the corresponding application from the viewpoint of the first user (e.g., 914A) at a second size that is larger than the first size. In some embodiments, the display generation component uses the same display area to display the corresponding representation at the second distance and the first distance, thereby making the virtual size of the corresponding representation larger. In some embodiments, the display generation component uses more display area to display the corresponding representation at the second distance than the display area used to display the corresponding representation at the first distance. In some embodiments, the size used to display the corresponding representation from the viewpoint of the second user is updated based on the updated distance between the corresponding representation and the viewpoint of the second user. In some embodiments, the size of the corresponding representation is the same from the viewpoint of the first user and the viewpoint of the second user, and the size of the corresponding representation is updated based on the distance of the corresponding representation from the viewpoint of the first user and the distance of the corresponding representation from the viewpoint of the second user. In some embodiments, based on a determination that the corresponding representation is moved closer to the viewpoint of the first user and / or the viewpoint of the second user, the electronic device updates the corresponding representation to be displayed at a smaller size.

[0223] The above-described manner of increasing the size of the corresponding representation in response to detecting that the corresponding representation has moved further away from the viewpoint of the first user provides an effective manner of maintaining the legibility of the corresponding representation, which in addition reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the input required to adjust the size of the corresponding representation), while reducing errors in use.

[0224] In some embodiments, such as Figure 9B In some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first distance from a viewpoint of a first user (e.g., 914A) in a three-dimensional environment (e.g., 904), wherein a corresponding user interface element (e.g., 918) is displayed at a second size from the viewpoint of the first user (e.g., 914A) (e.g., and the corresponding representation of the corresponding application is displayed at the first size from the viewpoint of the first user), the electronic device (e.g., 101) detects (1042) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application to a second distance from the viewpoint of the first user (e.g., 914A) that is greater than the first distance. Figure 9CIn some embodiments, in response to detecting the corresponding input (1044), the electronic device (e.g., 101) moves (1046) the corresponding representation (e.g., 906) of the corresponding application to a second distance from the viewpoint of the first user (e.g., 914A) (e.g., displaying the corresponding representation of the corresponding application at an updated position at the second distance from the viewpoint of the first user in the three-dimensional environment). In some embodiments, in response to detecting the corresponding input (1044), the electronic device (e.g., 101) displays (1048) the corresponding user interface element at a third size that is larger than the second size from the viewpoint of the first user (e.g., updating the size of the user interface element (e.g., 918) based on the distance of the changed representation (e.g., 906) from user A (e.g., 914A)). In some embodiments, the display generation component uses the same display area to display the corresponding user interface element at the second distance and the first distance, thereby making the virtual size of the corresponding user interface element larger. In some embodiments, the display generation component uses more display area to display the corresponding user interface element at the second distance than the display area used to display the corresponding user interface element at the first distance. In some embodiments, the size of the corresponding user interface element displayed from the second user's viewpoint is updated based on the updated distance between the corresponding user interface element and the second user's viewpoint. In some embodiments, the size of the corresponding user interface element is the same from the first user's viewpoint and the second user's viewpoint, and the size of the corresponding user interface element is updated based on the distance of the corresponding user interface element from the first user's viewpoint and the distance of the corresponding user interface element from the second user's viewpoint. In some embodiments, based on determining that the corresponding user interface element has moved closer to the first user's viewpoint and / or the second user's viewpoint, the electronic device updates the corresponding user interface element to be displayed at a smaller size. Thus, in some embodiments, the electronic device updates the size of the corresponding user interface element even if the size of the object remains the same. In some embodiments, the electronic device displays the corresponding representation of the corresponding application at a first size from the first user's viewpoint at a second distance from the user's viewpoint. In some embodiments, after moving the corresponding representation farther from the first user's viewpoint to maintain the virtual size of the corresponding representation, the display generation component uses less display area to display the corresponding representation of the corresponding application. In some embodiments, the electronic device maintains the size of the corresponding representation from the second user's viewpoint as the corresponding representation is moved around the three-dimensional environment. In some embodiments, the electronic device maintains the size of the corresponding representation as viewed from the first user's viewpoint and / or the second user's viewpoint, regardless of the distance between the corresponding representation and the first user's viewpoint and / or the second user's viewpoint. In some embodiments, the electronic device maintains the size of the corresponding representation in response to input to move the corresponding representation closer to the first user's viewpoint and / or the second user's viewpoint.In some embodiments, in response to input to move the corresponding representation closer to the viewpoint of the first user and / or the viewpoint of the second user, the electronic device increases the display area used to present the corresponding representation to maintain the virtual size of the corresponding representation at a distance closer to the viewpoint of the first user and / or the viewpoint of the second user.

[0225] The above-mentioned manner of displaying the corresponding representations at the same size and displaying the corresponding user interface elements at different sizes in response to detecting a change in the distance between the first user's viewpoint and the corresponding representation and the corresponding user interface elements provides an effective way to maintain the readability of the corresponding user interface elements displayed together with the virtual objects having a constant size, which further reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0226] In some embodiments, such as Figure 9B In some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first location in a three-dimensional environment (e.g., 904), an electronic device (e.g., 101) detects (1050) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904) via a hand tracking device in communication with the electronic device, wherein the corresponding input includes a gesture performed by a hand (e.g., 920b) of a user (e.g., 914B) of the electronic device 101b, followed by movement of the hand (e.g., 920b) while maintaining the gesture performed by the hand (e.g., 920b). In some embodiments, the gesture is a user touching their thumb to one of their other fingers (e.g., index finger, middle finger, ring finger, pinky finger) while the user is looking at a corresponding user interface element that can be selected to move the corresponding representation (e.g., detected via an eye tracking device), and the electronic device detects movement of the hand (e.g., using a hand tracking device) when the thumb touches the other finger. In some embodiments, such as in Figure 9C In some embodiments, in response to receiving the corresponding input, the electronic device (e.g., 101) moves (1052) the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904) according to the movement of the hand (e.g., 920b). In some embodiments, in response to detecting the user's gaze on the corresponding representation and simultaneously detecting the gesture, the electronic device moves the corresponding representation of the corresponding application according to the movement of the hand. In some embodiments, the corresponding representation is updated according to one or more steps of methods 800 and / or 1200.

[0227] The above-mentioned manner of moving the corresponding representation according to the movement of the hand in response to detecting a gesture performed by the user's hand provides an effective way of controlling the position of the corresponding representation, which further reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0228] In some embodiments, such as Figure 9B , when a corresponding representation (e.g., 906) of a corresponding application is displayed at a first position in a three-dimensional environment (e.g., 904), the electronic device (e.g., 101) detects (1054) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904). In some embodiments, the corresponding input includes a selection of an option that, when selected, causes the electronic device to initiate a process of moving the corresponding representation. In some embodiments, the process of moving the corresponding representation includes: detecting, via a hand tracking device, that a user's hand is performing a gesture and moving and moving the corresponding representation in accordance with the movement of the hand. In some embodiments, in response to receiving the corresponding input (1056), based on a determination that the corresponding input is associated with the first user (e.g., 914A), the electronic device (e.g., 101) moves (1058) the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904) in accordance with the corresponding input, such as in Figure 9C In some embodiments, the electronic device of the first user detects the corresponding input for moving the corresponding representation of the corresponding application. In some embodiments, in response to the electronic device of the first user detecting the corresponding input, the electronic device of the first user and the electronic device of the second user display the movement of the corresponding representation. In some embodiments, in response to receiving the corresponding input (1056), based on a determination that the corresponding input is associated with the second user (e.g., 914B), the electronic device (e.g., 101) moves (1060) the corresponding representation (e.g., 906) of the corresponding application in the three-dimensional environment (e.g., 904) based on the corresponding input, such as in Figure 9C In some embodiments, the second user's electronic device detects a corresponding input for moving the corresponding representation of the corresponding application. In some embodiments, in response to the second user's electronic device detecting the corresponding input, the first user's electronic device and the second user's electronic device display the movement of the corresponding representation. In some embodiments, the corresponding representation is updated according to one or more steps of methods 800 and / or 1200.

[0229] The above-described manner of moving the corresponding representation of the corresponding application in response to the corresponding input regardless of whether the corresponding input is associated with the first user or the second user provides an efficient manner of moving the corresponding representation of the corresponding application, which further reduces power usage and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently (for example, by enabling either user to move the representation, thereby saving time compared to one user having to ask another user to move the representation), while reducing errors in use.

[0230] In some embodiments, such as Figure 9A In some embodiments, when a corresponding representation (e.g., 922) of a corresponding application is displayed at a first location in a three-dimensional environment (e.g., 904), wherein the corresponding representation (e.g., 922) of the corresponding application is displayed at a first orientation relative to the three-dimensional environment (e.g., 904), at a second orientation relative to a viewpoint of a first user (e.g., 914A) and at a first corresponding location, and at a third orientation relative to a viewpoint of a second user (e.g., 914B), the electronic device (e.g., 101) receives (1062) via one or more input devices a corresponding input corresponding to a request to automatically change the viewpoint of the first user (e.g., 914A) in the three-dimensional environment (e.g., 904), wherein the corresponding input does not indicate a manner in which the viewpoint of the first user (e.g., 914A) in the three-dimensional environment (e.g., 904) is changed. In some embodiments, the input corresponds to a request to reset the three-dimensional environment from the viewpoint of the first user. In some embodiments, the input is a selection of an option displayed in the three-dimensional environment, the option displayed based on a determination that a location associated with the first user is at least a threshold distance from objects and other users in the three-dimensional environment. In some embodiments, in response to receiving the corresponding input, the electronic device (e.g., 101) updates (1064) the viewpoint of the first user (e.g., 914A) in the three-dimensional environment (e.g., 904) based on the viewpoint of the second user (e.g., 914B) and the corresponding representation (e.g., 922) of the corresponding application at the first position in the three-dimensional environment (e.g., 904). In some embodiments, the second user's electronic device continues to display the corresponding representation of the corresponding application in a third orientation and a second corresponding position relative to the second user's viewpoint. In some embodiments, in response to the corresponding input, the electronic device automatically updates the position and orientation of the first user's viewpoint according to one or more criteria based on the position of the other user's viewpoint in the three-dimensional environment, the representation of the application in the three-dimensional environment, and / or other objects in the three-dimensional environment in accordance with one or more steps of method 1400. For example, the first user's position is automatically updated based on user and virtual object placement rules.

[0231] The above-described manner of automatically updating the viewpoint of the first user in a three-dimensional environment in response to corresponding input provides an efficient manner of presenting the three-dimensional environment from the viewpoint of the first user according to one or more criteria without the need to manually rearrange objects in the three-dimensional environment or manually select the updated position of the first user, which further reduces power usage and improves the battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0232] In some embodiments, after updating the viewpoint of the first user in the three-dimensional environment, a corresponding representation of the corresponding application (e.g., 922) is displayed (1066) at a first position and in a first orientation (1068) relative to the three-dimensional environment (e.g., 904), such as in Figure 9A In some embodiments, the position of the corresponding representation of the corresponding application in the three-dimensional environment and the orientation of the corresponding representation relative to the fixed reference frame in the three-dimensional environment do not change in response to the corresponding input. In some embodiments, other users see the position of the representation of the first user updated in response to the input (for example, the representation of the first user "jumps" from the first corresponding position to the second corresponding position in the three-dimensional environment). In some embodiments, the system maintains a three-dimensional environment that includes position information of virtual objects (for example, representations of users, representations of applications, representations of virtual objects), and each user has a view of the three-dimensional environment from their own viewpoint. In some embodiments, each user's viewpoint includes representations of physical objects in the environment of the first user's electronic device (for example, a pass-through video of an image-realistic representation of the environment of the electronic device). For example, the first user's electronic device presents a three-dimensional environment with one or more physical objects in the physical environment of the first user's electronic device, and the second user's electronic device presents a three-dimensional environment with one or more physical objects in the physical environment of the second user. In some embodiments, in addition to physical objects, the three-dimensional environment also includes virtual objects. In some embodiments, after updating the viewpoint of a first user (e.g., 914A) in a three-dimensional environment (e.g., 904), the corresponding representation (e.g., 922) of the corresponding application is displayed (1066) at a fourth orientation and at a third corresponding position (1070) relative to the updated viewpoint of the first user (e.g., 914A). In some embodiments, when the viewpoint of the first user changes, the orientation and position of the corresponding representation of the corresponding application change relative to the viewpoint of the first user. In some embodiments, after updating the viewpoint of a first user (e.g., 914A) in a three-dimensional environment (e.g., 904), the corresponding representation (e.g., 922) of the corresponding application is displayed (1066) at a third orientation and at a second corresponding position (1072) relative to the viewpoint of a second user (e.g., 914B), such as in Figure 9AIn some embodiments, the orientation and position of the corresponding representation of the corresponding application relative to the second user's viewpoint do not change in response to the corresponding input. In some embodiments, the position of the first user in the three-dimensional environment and the position of the first user relative to the second user's viewpoint change in response to the corresponding input. In some embodiments, the second user's electronic device displays the movement of the representation of the first user from one position to another position in the three-dimensional environment (e.g., to a "reset" position automatically determined by the electronic device) in response to the corresponding input.

[0233] The above-described manner of maintaining the position of the corresponding representation in the three-dimensional environment after the first user's viewpoint changes provides an effective way to automatically locate the first user, which is more efficient than manually repositioning the first user and / or one or more objects or other users in the three-dimensional environment, which further reduces power usage and improves battery life of the electronic device by enabling the user to use the electronic device more quickly and efficiently (for example, by reducing the input required to reply to the second user's viewpoint), while reducing errors in use.

[0234] In some embodiments, such as Figure 9C In some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed at a corresponding position in a three-dimensional environment (e.g., 904) in a first corresponding orientation, the electronic device (e.g., 101) detects (1074) a change in the number of users with which the three-dimensional environment (e.g., 904) is shared (e.g., the first user or the second user leaves the three-dimensional environment and / or one or more additional users join the three-dimensional environment). In some embodiments, in response to detecting a change in the number of users with which the three-dimensional environment is shared, based on a determination that one or more criteria are met (e.g., the corresponding representation of the corresponding application is moved from the first corresponding position and returned to the first corresponding position after the number of users changes, or the representation of the application is able to change orientation based on users leaving or joining without having to move), the electronic device (e.g., 101) displays (1076) the corresponding representation (e.g., 906) at the corresponding position in a second corresponding orientation different from the first corresponding orientation based on the arrangement of the viewpoints of the changed number of users with which the three-dimensional environment (e.g., 904) is shared, such as in Figure 9DIn some embodiments, the orientation of the corresponding representation of the corresponding application is based on the arrangement of the viewpoints of one or more or all of the users in the three-dimensional environment, so if the number of users in the three-dimensional environment changes, the arrangement of the viewpoints will change, and the orientation of the corresponding representation will also change in response to moving the corresponding representation to the first position in the three-dimensional environment. For example, when there are two users in the three-dimensional environment, in response to an input to move the corresponding representation to the first corresponding position in the three-dimensional environment, the electronic device displays the corresponding representation at the first corresponding position in the three-dimensional environment with a first orientation. For another example, when there are three users in the three-dimensional environment, in response to an input to move the corresponding representation to the first corresponding position in the three-dimensional environment, the electronic device displays the corresponding representation at the first corresponding position in the three-dimensional environment with a second orientation different from the first orientation. In some embodiments, the electronic device does not update the orientation of the corresponding representation in response to a change in the number of users in the three-dimensional environment. In some embodiments, the electronic device updates the orientation of the corresponding representation in response to a change in the number of users in the three-dimensional environment (for example, to orient the corresponding representation toward one or more or all of the users in the three-dimensional environment).

[0235] The above-mentioned manner of displaying a corresponding representation in a second orientation at a corresponding position in response to detecting a change in the number of users in a three-dimensional environment provides an effective manner of orienting the corresponding representation toward one or more or all of the users in the three-dimensional environment, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0236] In some embodiments, such as Figure 9A In some embodiments, the electronic device (e.g., 101) displays (1078) a corresponding representation (e.g., 906) of a corresponding application at a first location in a three-dimensional environment (e.g., 904) based on a first arrangement of a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B). Figure 9A In some embodiments, based on a determination that a corresponding representation (e.g., 906) of a corresponding application is viewable from a single direction (e.g., an object (e.g., a virtual television) including virtual content such as video content on one side of the object), a first position in a three-dimensional environment (e.g., 904) is a first corresponding position in the three-dimensional environment (e.g., 904) such that the corresponding representation (e.g., 906) of the corresponding application is viewable from a single direction from a viewpoint of a first user (e.g., 914A) and a viewpoint of a second user (e.g., 914B) (1080). In some embodiments, the first position is selected such that the single direction of the corresponding representation is oriented toward the viewpoint of the first user and the viewpoint of the second user. In some embodiments, such as in Figure 9AIn some embodiments, based on a determination that a corresponding representation (e.g., 922) of a corresponding application is viewable from a first direction and a second direction different from the first direction (e.g., a virtual board game visible from multiple orientations around the virtual board game), a first position in the three-dimensional environment (e.g., 904) is a second corresponding position in the three-dimensional environment (e.g., 904) different from the first corresponding position, such that the corresponding representation (e.g., 922) of the corresponding application is viewable from the first direction from a viewpoint of a first user (e.g., 914A) and viewable from the second direction from a viewpoint of a second user (e.g., 914B) (1082). In some embodiments, the first position is between the viewpoints of the first user and the viewpoints of the second user, such that the first direction faces the viewpoint of the first user and the second direction faces the viewpoint of the second user.

[0237] The above-described manner of selecting different positions for corresponding representations depending on whether the corresponding representation is capable of being viewed from a single direction or from multiple directions provides an efficient manner of placing the corresponding representation in a position visible to both the first user and the second user, which in addition reduces power usage and improves the battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0238] In some embodiments, such as Figure 9BIn some embodiments, when a corresponding representation (e.g., 906) of a corresponding application is displayed in a three-dimensional environment (e.g., 904) at a first distance from a viewpoint of a first user (e.g., 914A) and at a second distance from a viewpoint of a second user (e.g., 914B), wherein the corresponding representation (e.g., 906) of the corresponding application is displayed at a first size from the viewpoint of the first user (e.g., 914A) and at a second size from the viewpoint of the second user (e.g., 914B), the electronic device (e.g., 101) detects (1084) a corresponding input for moving the corresponding representation (e.g., 906) of the corresponding application to a third distance from the viewpoint of the first user (e.g., 914A) and a fourth distance from the viewpoint of the second user (e.g., 914B), respectively, wherein the third distance is different from the first distance and the fourth distance is different from the second distance. In some embodiments, the corresponding representation moves to a position farther from both the viewpoint of the first user and the viewpoint of the second user. In some embodiments, the representation moves to a position closer to both the viewpoint of the first user and the viewpoint of the second user. In some embodiments, the corresponding representation moves to a position that is further away from the viewpoint of the first user and closer to the viewpoint of the second user. In some embodiments, the corresponding representation moves to a position that is closer to the viewpoint of the first user and further away from the viewpoint of the second user. In some embodiments, the first size and the second size are the same. In some embodiments, the first size and the second size are different. In some embodiments, the size of the corresponding representation is the virtual size of the representation in the three-dimensional environment, and the amount of display area used to display the representation depends on the size of the representation and the distance of the representation from the user's viewpoint. In some embodiments, in response to detecting the corresponding input (1086), the electronic device moves (1088) the corresponding representation of the corresponding application to a third distance from the viewpoint of the first user (for example, and a fourth distance from the viewpoint of the second user). In some embodiments, such as in Figure 9CIn some embodiments, in response to detecting the corresponding input (1086), the electronic device displays (1090) the corresponding representation of the corresponding application at a third size different from the first size from the viewpoint of the first user and at a fourth size different from the second size from the viewpoint of the second user based on the third distance and the fourth distance. In some embodiments, the third size and the fourth size are the same. In some embodiments, the third size and the fourth size are different. In some embodiments, the size of the corresponding representation is the virtual size of the representation in the three-dimensional environment, and the amount of display area used to display the representation depends on the size of the representation and the distance of the representation from the user's viewpoint. In some embodiments, in response to moving the corresponding representation away from the viewpoint of the first user, the electronic device increases the size of the corresponding representation. In some embodiments, in response to moving the corresponding representation toward the viewpoint of the first user, the electronic device decreases the size of the corresponding representation. In some embodiments, the size of the corresponding representation is updated according to an average or another combination of the distances to the viewpoint of the first user and the viewpoint of the second user, regardless of which user provides the input to move the representation of the application.

[0239] The above-mentioned manner of updating the size of a representation based on a third distance from the corresponding representation to the viewpoint of the first user and a fourth distance from the corresponding representation to the viewpoint of the second user, respectively, provides an effective way to maintain the readability of the corresponding representation when the corresponding representation is moved away from one or more viewpoints and to preserve display area for other objects when the corresponding representation is moved toward one or more viewpoints, which further reduces power usage and improves battery life of the electronic device by enabling users to use the electronic device more quickly and efficiently, while reducing errors in use.

[0240] In some embodiments, such as Figure 9B In some embodiments, such as in Figure 9B In some embodiments, displaying (1092) a corresponding representation of the corresponding application (e.g., 906) at a third size ...

Claims

1. A method, comprising: At an electronic device in communication with a display generating component and one or more input devices: displaying, via the display generating component, a three-dimensional environment, the three-dimensional environment including a representation of a first application displayed at a first location in the three-dimensional environment, wherein: The representation of the first application at the first location in the three-dimensional environment has a first orientation in the three-dimensional environment, and A second location in the three-dimensional environment is associated with a first user in the three-dimensional environment; receiving movement input via the one or more input devices while displaying the three-dimensional environment including the representation of the first application; and In response to receiving the movement input: Based on determining that the movement input corresponds to a request to move the representation of the first application from the first location to a third location in the three-dimensional environment that is different from the first location: displaying, via the display generation component, the representation of the first application at the third location in the three-dimensional environment; and updating the orientation of the representation of the first application from the first orientation to a second orientation relative to the three-dimensional environment that is different from the first orientation based on the third position; and Based on determining that the movement input corresponds to a request to associate a fourth position different from the second position with the first user, maintain display of the representation of the first application at the first position in the three-dimensional environment without updating the orientation of the representation of the first application relative to the three-dimensional environment. 2 . The method of claim 1 , wherein the three-dimensional environment comprises a representation of a physical environment of the electronic device displayed via the display generation component. 3 . The method of claim 1 , wherein the three-dimensional environment comprises a virtual environment displayed via the display generation component.

4. The method of claim 1 , wherein when the representation of the first application is displayed at the first position, the representation of the first application is displayed at a first size, the method further comprising: In response to receiving the movement input: Based on determining that the movement input corresponds to a request to move the representation of the first application from the first position to the third position in the three-dimensional environment that is different from the first position, the representation of the first application is updated to be displayed at a second size that is different from the first size when the representation of the first application is displayed at the third position.

5. The method of claim 1 , wherein updating the orientation of the representation of the first application comprises: updating the orientation of the first application based on the second position of the first user and not based on the fifth position of the second user in response to determining that the representation of the first application is accessible to the first user but not to a second user associated with a fifth position in the three-dimensional environment; as well as Based on determining that the representation of the first application is accessible to the first user and the second user associated with the fifth position in the three-dimensional environment, updating the orientation of the first application based on the second position associated with the first user and the fifth position associated with the second user.

6. The method according to claim 1, further comprising: detecting a corresponding input to display a representation of a second application in the three-dimensional environment, wherein the representation of the second application is not displayed in the three-dimensional environment when the corresponding input is detected; and In response to detecting the respective input, the representation of the second application is displayed at a respective location in the three-dimensional environment based on a number of users in the three-dimensional environment.

7. The method according to claim 6, wherein: Based on determining that the three-dimensional environment includes the first user and no other users, the corresponding position is in front of the first user, and According to determining that the three-dimensional environment includes the first user and the second user, the corresponding position is located based on the position of the first user and the position of the second user.

8. The method according to claim 6, wherein: Based on determining that the three-dimensional environment includes the first user and no other users, the representation of the second application is displayed as oriented toward the first user, and based on determining that the three-dimensional environment includes the first user and the second user, the representation of the second application is displayed as oriented toward a position between the first user and the second user.

9. The method according to claim 6, wherein: Based on determining that the three-dimensional environment includes the first user and no other users, the corresponding position is a first distance away from the first user, and According to determining that the three-dimensional environment includes the first user and the second user, the corresponding position is at a second distance from the first user that is different from the first distance.

10. The method according to claim 1, further comprising: while displaying the representation of the first application at the first location in the three-dimensional environment, detecting that a second user has been added to the three-dimensional environment, wherein the second user is associated with a fifth location in the three-dimensional environment; as well as In response to detecting that the second user has been added to the three-dimensional environment: Based on determining that the representation of the first application is accessible to the first user and the second user, the representation of the first application is moved from the first location to a sixth location in the three-dimensional environment based on the second location and the fifth location.

11. The method according to claim 1, further comprising: while displaying the representation of the first application at the first location in the three-dimensional environment, detecting that a second user has been added to the three-dimensional environment, wherein the second user is associated with a fifth location in the three-dimensional environment; as well as In response to detecting that the second user has been added to the three-dimensional environment: Based on determining that the representation of the first application is accessible to the first user but not accessible to the second user, maintaining display of the representation of the first application at the first location, wherein the representation of the first application is not visible to the second user.

12. The method of claim 1, wherein the three-dimensional environment includes the first user at the second position in the three-dimensional environment and a second user at a fifth position in the three-dimensional environment, the method further comprising: detecting a corresponding input to display a representation of a second application in the three-dimensional environment, wherein when the corresponding input is detected, the representation of the second application is not displayed in the three-dimensional environment and the representation of the second application is not accessible to a plurality of users in the three-dimensional environment; as well as In response to receiving the corresponding input: based on determining that the corresponding input is associated with the first user, displaying the representation of the second application at a sixth location in the three-dimensional environment based on the second location of the first user instead of the fifth location of the second user; as well as Based on determining that the corresponding input is associated with the second user, displaying the representation of the second application at a seventh position in the three-dimensional environment based on the fifth position of the second user instead of the second position of the first user.

13. The method of claim 1, wherein the three-dimensional environment includes the first user at the second position in the three-dimensional environment and a second user at a fifth position in the three-dimensional environment, the method further comprising: detecting a corresponding input to display a representation of a second application in the three-dimensional environment, wherein the representation of the second application is not displayed in the three-dimensional environment when the corresponding input is detected; and In response to receiving the corresponding input, displaying the representation of the second application at the corresponding location in the three-dimensional environment, wherein: Based on determining that the representation of the second application is not accessible to the first user, the corresponding location is not based on the second location of the first user, and Based on determining that the representation of the second application is not accessible to the second user, the corresponding location is not based on the fifth location of the second user.

14. The method according to claim 13, wherein: Based on determining that the representation of the second application is inaccessible to the first user, a visual indication of the representation of the second application is visible to the first user, while content of the representation of the second application is not visible to the first user, and Based on determining that the representation of the second application is inaccessible to the second user, the visual indication of the representation of the second application is visible to the second user, while the content of the representation of the second application is not visible to the second user.

15. The method of claim 1, wherein the three-dimensional environment includes the first user at the second position in the three-dimensional environment and a second user at a fifth position in the three-dimensional environment, the method further comprising: In response to receiving the movement input: Based on determining that the movement input corresponds to the request to move the representation of the first application from the first location to the third location in the three-dimensional environment: updating the representation of the first application to be accessible to the first user but inaccessible to the second user based on determining that the third location is between the second location of the first user and the fifth location of the second user; as well as Based on determining that the third location is not between the second location of the first user and the fifth location of the second user, maintaining the representation of the first application accessible to the first user and the second user.

16. The method according to claim 1, further comprising: In response to receiving the movement input: Based on determining that the movement input corresponds to the request to move the representation of the first application from the first location to the third location in the three-dimensional environment: updating the representation of the first application to be accessible to the first user but inaccessible to a second user in the three-dimensional environment based on determining that the third location is within a threshold distance of the second location of the first user; as well as Based on determining that the third location is not within the threshold distance of the second location of the first user, updating the representation of the first application to be accessible to the first user but not to the second user is abandoned.

17. The method according to claim 16, further comprising: In response to receiving the movement input: Based on determining that the movement input corresponds to the request to move the representation of the first application from the first location to the third location in the three-dimensional environment: updating the representation of the first application to be accessible to the second user but inaccessible to the first user in the three-dimensional environment based on determining that the third location is within a threshold distance of a fifth location associated with the second user; as well as Based on determining that the third location is not within the threshold distance of the fifth location associated with the second user, updating the representation of the first application to be accessible to the second user but not to the first user is abandoned.

18. The method of claim 1, further comprising: The representation of the first application is simultaneously displayed at the first position in the three-dimensional environment and the representation of the second application is simultaneously displayed at a fifth position in the three-dimensional environment, wherein the representation of the first application is displayed in the three-dimensional environment at a first corresponding orientation, and the representation of the second application is displayed in the three-dimensional environment at a second corresponding orientation different from the first corresponding orientation.

19. The method of claim 1, wherein the first orientation of the representation of the first application and the second orientation of the representation of the first application are based on an orientation field associated with the first user.

20. The method of claim 1, wherein the three-dimensional environment includes the first user and a second user, and the representation of the first application is accessible to the first user but not to the second user, the method further comprising: while displaying the representation of the first application in a first appearance, detecting, via the one or more input devices, a corresponding input corresponding to a request to update the representation of the first application to be accessible to the first user and the second user; and In response to detecting the corresponding input: updating the representation of the first application to be accessible to the first user and the second user; and The representation of the first application is displayed in a second appearance different from the first appearance.

21. The method of claim 20, wherein the corresponding input is associated with a suggestion to share the representation of the first application with the second user, the method further comprising: while displaying the representation of the first application in the first appearance, detecting, via the one or more input devices, a second corresponding input corresponding to a request to update the representation of the first application to be accessible to the first user and the second user, wherein the second corresponding input is not associated with a suggestion to share the representation of the first application with the second user; as well as In response to detecting the corresponding input: The representation of the first application is updated to be accessible to the first user and the second user while maintaining the representation of the first application having the first appearance.

22. The method of claim 1, wherein the three-dimensional environment includes the first user and a second user, and the representation of the first application is accessible to the first user but not to the second user, the method further comprising: while displaying the representation of the first application in a first appearance, detecting, via the one or more input devices, a corresponding input corresponding to a request to update the representation of the first application to be accessible to the first user and the second user; and In response to detecting the corresponding input: The representation of the first application is updated to be accessible to the first user and the second user while maintaining the representation of the first application having the first appearance.

23. The method of claim 1, wherein the three-dimensional environment includes a representation of a first corresponding object of a first type at a fifth location in the three-dimensional environment, the three-dimensional environment includes a representation of a second corresponding object of a second type different from the first type at a sixth location in the three-dimensional environment, the fifth location being a first distance from the second location associated with the first user, and the sixth location being a second distance from the second location associated with the first user, the method further comprising: receiving, via the one or more input devices, a first corresponding input corresponding to a request to move the representation of the first corresponding object from the first distance from the second location to a third distance from the second location that is different than the first distance; In response to receiving the first corresponding input: moving the representation of the first corresponding object to the third distance from the second location; and changing a size of the representation of the first corresponding object based on the third distance; receiving, via the one or more input devices, a second corresponding input corresponding to a request to move the representation of the second corresponding object from the second distance from the second location to a fourth distance from the second location that is different than the second distance; as well as In response to receiving the second corresponding input: The representation of the second corresponding object is moved to the fourth distance from the second position while maintaining a size of the representation of the second corresponding object.

24. An electronic device comprising: one or more processors; Memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing any one of the methods according to claims 1 to 23.

25. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to perform any one of the methods according to claims 1 to 23.

Images