Method for adjusting or controlling immersive sensation associated with user interface

Abstract

To solve a problem of necessity of a computer system having an improved method and interface for providing a computer generation experience which makes an interaction with the computer system more efficient and intuitive, to a user.SOLUTION: According to some embodiments, in an electronic device, a user interface element makes display of an individual person in a physical environment of an electronic device at least partially ambiguous via a display generation component. While a user interface is being displayed, display of the user interface element is updated, according to determination that the individual person is present farther than the threshold distance from the electronic device in the physical environment, and determination that attention of the individual person is directed to a user of the electronic device, so that the presence of the individual person in the physical environment at a position corresponding to the user interface element is displayed.SELECTED DRAWING: Figure 14A

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 083,792, filed September 25, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes. [Technical field]

[0002] It generally relates to a computer system having a display generating component and one or more input devices that present a graphical user interface, including but not limited to an electronic device that presents a three-dimensional environment including virtual objects, via the display generating component. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual objects, including digital images, videos, text, icons, and control elements such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback to perform actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulation of virtual objects is complex and error-prone cause significant cognitive burden to users and detract from the experience of the virtual / augmented reality environment. In addition, the methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Thus, there is a need for a computer system having improved methods and interfaces for providing a user with a computer-generated experience that makes interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can complement or replace conventional methods of providing a user with a computer-generated reality experience. Such methods and interfaces reduce the number, extent and / or type of inputs from the user by helping the user understand the connection between the inputs provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above deficiencies and other problems associated with user interfaces for computer systems having a display generating component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generating component, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, and / or playing digital videos. Executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices having improved methods and interfaces for adjusting and / or controlling the immersive experience associated with a user interface. Such methods and interfaces can complement or replace conventional methods for displaying user interfaces in three-dimensional environments. Such methods and interfaces reduce the number, extent, and / or type of inputs from a user, producing a more efficient human-machine interface.

[0008] In some embodiments, the electronic device highlights and / or de-highlights the user interface based on the user's gaze. In some embodiments, the electronic device defines immersion levels of different user interfaces independently of each other. In some embodiments, the electronic device resumes displaying the user interface at the previously displayed immersion level after (e.g., temporarily) reducing the immersion level associated with the user interface. In some embodiments, the electronic device causes objects, people, and / or parts of the environment to be visible through the user interface displayed by the electronic device. In some embodiments, the electronic device reduces the immersion level associated with the user interface based on characteristics of the electronic device and / or the physical environment of the electronic device.

[0009] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and description, and not to define or limit the subject matter of the present invention. [Brief description of the drawings]

[0010] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0012] [Diagram 2] FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0013] [Diagram 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of a CGR experience to a user, according to some embodiments.

[0014] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0015] [Diagram 5]FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0016] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0017] [Figure 7A] 1A-1C illustrate examples of how an electronic device can highlight and / or de-highlight a user interface based on a user's gaze, according to some embodiments. [Figure 7B] 1A-1C illustrate examples of how an electronic device can highlight and / or de-highlight a user interface based on a user's gaze, according to some embodiments.

[0018] [Figure 8A] 1 is a flowchart illustrating a method for highlighting and / or de-highlighting a user interface based on a user's gaze, according to some embodiments. [Figure 8B] 1 is a flowchart illustrating a method for highlighting and / or de-highlighting a user interface based on a user's gaze, according to some embodiments. [Figure 8C] 1 is a flowchart illustrating a method for highlighting and / or de-highlighting a user interface based on a user's gaze, according to some embodiments. [Figure 8D] 1 is a flowchart illustrating a method for highlighting and / or de-highlighting a user interface based on a user's gaze, according to some embodiments.

[0019] [Figure 9A] FIG. 1 illustrates an example of defining immersion levels for different user interfaces independently of each other, according to some embodiments. [Figure 9B]FIG. 1 illustrates an example of defining immersion levels for different user interfaces independently of each other, according to some embodiments. [Figure 9C] FIG. 1 illustrates an example of defining immersion levels for different user interfaces independently of each other, according to some embodiments.

[0020] [Figure 10A] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10B] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10C] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10D] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10E] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10F] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10G] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10H] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10I] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10J]1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments. [Figure 10K] 1 is a flowchart illustrating a method for defining immersion levels of different user interfaces independently of one another, according to some embodiments.

[0021] [Figure 11A] A diagram showing an example of resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing the immersion level associated with the user interface, according to some embodiments. [Figure 11B] A diagram showing an example of resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing the immersion level associated with the user interface, according to some embodiments.

[0022] [Figure 12A] 1 is a flowchart illustrating a method for resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing an immersion level associated with the user interface, according to some embodiments. [Figure 12B] 1 is a flowchart illustrating a method for resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing an immersion level associated with the user interface, according to some embodiments. [Figure 12C] 1 is a flowchart illustrating a method for resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing an immersion level associated with the user interface, according to some embodiments. [Figure 12D] 1 is a flowchart illustrating a method for resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing an immersion level associated with the user interface, according to some embodiments.

[0023] [Figure 13A] 1A-1C illustrate examples of making objects, people, and / or parts of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 13B] 1A-1C illustrate examples of making objects, people, and / or parts of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 13C] 1A-1C illustrate examples of making objects, people, and / or parts of an environment visible through a user interface displayed by an electronic device, according to some embodiments.

[0024] [Figure 14A] 1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 14B] 1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 14C] 1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 14D] 1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 14E] 1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments. [Figure 14F]1 is a flowchart illustrating a method for making objects, people, and / or portions of an environment visible through a user interface displayed by an electronic device, according to some embodiments.

[0025] [Figure 15A] FIG. 1 illustrates an example of reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or the physical environment of the electronic device, according to some embodiments. [Figure 15B] FIG. 1 illustrates an example of reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or the physical environment of the electronic device, according to some embodiments. [Figure 15C] FIG. 1 illustrates an example of reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or the physical environment of the electronic device, according to some embodiments.

[0026] [Figure 16A] 1 is a flowchart illustrating a method for reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or a physical environment of the electronic device, according to some embodiments. [Figure 16B] 1 is a flowchart illustrating a method for reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or a physical environment of the electronic device, according to some embodiments. [Figure 16C] 1 is a flowchart illustrating a method for reducing an immersion level associated with a user interface based on characteristics of an electronic device and / or a physical environment of the electronic device, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0028] The systems, methods, and GUIs described herein provide improved ways for electronic devices to adjust and / or control the level of immersion associated with a user interface.

[0029] In some embodiments, the computer system de-highlights the second user interface relative to the first user interface when the system detects that the user's gaze is directed toward the first user interface. In some embodiments, the system performs such de-highlighting when the first user interface is a user interface of a particular type of application (e.g., a media player application). In some embodiments, the second user interface includes a representation of one or more of the virtual elements displayed by the system or by parts of the system's physical environment. De-highlighting the second user interface allows the user to focus on the first user interface without being distracted by content outside of the first user interface.

[0030] In some embodiments, the computer system defines immersion levels for different user interfaces independently of each other. A change in the immersion level at which the system displays a first user interface (e.g., of an operating system, of a first application) optionally does not affect the immersion level at which the system displays a second user interface (e.g., of a second application). In some embodiments, the immersion is controlled via manipulation of a mechanical input element (e.g., a rotatable input element) associated with the computer system, and the direction and / or magnitude of the input on the mechanical input element defines the magnitude and / or direction of the change in the immersion level. The immersion level optionally defines the degree to which content other than the user interface in question (e.g., a representation of the system's physical environment, virtual elements outside the user interface, etc.) is visible through the display. Providing independently controlled immersion levels and / or doing so according to the magnitude and / or direction of the input provides the user with a consistent and expected display behavior for various user interfaces, resulting in reduced errors in interacting with such user interfaces.

[0031] In some embodiments, the computer system resumes displaying the user interface at a previously displayed immersion level after (e.g., temporarily) reducing an immersion level associated with the user interface. The computer system optionally detects an event for reducing the immersion level at which the individual user interface was displayed, and reduces the immersion level in response to the event. Then, in response to detecting an event corresponding to a request to redisplay the individual user interface at the previously displayed immersion level, the system optionally resumes displaying the individual user interface at the previously displayed immersion level. In some embodiments, the event for reducing the immersion level includes detecting a press input on a mechanical input element used to control immersion, and the event for resuming the previous immersion level includes detecting a release of the mechanical input element used to control immersion. Resuming displaying the user interface at its previous immersion level provides a quick and efficient way to return to a previously valid immersion level without requiring a user input defining a particular immersion level to return to, which also avoids erroneous user input defining an incorrect immersion level to return to.

[0032] In some embodiments, the computer system makes objects, people, and / or parts of the environment visible through a user interface displayed by the system. Representations of people in the system's environment are optionally made visible through the user interface based on the distance between the user and the people and / or the attention of the people (e.g., whether attention is directed at the user). Representations of objects in the system's environment are optionally made visible through the user interface based on the distance between the user and the objects and / or a determined risk level of the objects towards the user (e.g., whether the object(s) pose a risk to the user). Making a representation of the system's physical environment visible through the user interface helps the user to avoid hazards in the physical environment and facilitates interaction with people in the environment without requiring separate input from the user to do so.

[0033] In some embodiments, the computer system reduces an immersion level associated with a user interface based on the characteristics of the system and / or the system's physical environment. If the computer system determines that it is moving at a speed above a speed threshold, the system optionally reduces an immersion level at which the user interface(s) are displayed so that a user of the system can view the physical environment through the system. If the computer system determines that a sound associated with a potential hazard has been detected within the system's environment, the system optionally reduces an immersion level at which the user interface(s) are displayed so that a user of the system can view the physical environment through the system. Reducing the immersion level as described provides a quick and efficient way of allowing a user of the system to view the physical environment and does not require separate input from the user to do so.

[0034] 1-6 provide an illustration of an exemplary computer system (as described below with reference to methods 800, 1000, 1200, 1400, and 1600) for providing a CGR experience to a user. In some embodiments, the CGR experience is provided to a user via an operating environment 100 that includes a computer system 101, as shown in FIG. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, tactile sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensors 190, and the peripheral devices 195 are integrated with the display generation component 120 (eg, within a head-mounted or handheld device).

[0035] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or that the user can interact with (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following are a subset of these terms:

[0036] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and / or interact with the physical environment directly, such as through sight, touch, hearing, taste, and smell.

[0037] Computer-Generated Reality: In contrast, a Computer-Generated Reality (CGR) environment refers to a fully or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or a representation thereof is tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave with at least one law of physics. For example, a CGR system may detect a person's head rotation and in response adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to a property(s) of a virtual object(s) in a CGR environment may be made in response to a representation of a body movement (e.g., a voice command). A person may sense and / or interact with a CGR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create an audio environment with 3D or spatial breadth that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0038] Examples of CGR include virtual reality and mixed reality.

[0039] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0040] Mixed reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from a physical environment or a representation thereof in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0041] Examples of mixed reality include augmented reality and augmented virtuality.

[0042] Augmented reality: An augmented reality (AR) environment refers to a mimicked environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment that are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. The person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, a video of a physical environment shown on an opaque display is referred to as a "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a mimicked environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, such that the altered portion is a modified version that represents the original captured image but is non-photorealistic. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0043] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0044] Hardware: There are many different types of electronic systems that allow a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over the person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display rather than an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology that projects a graphical image onto the person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces.In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0045] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of a CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0046] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0047] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on one's head, on one's hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds a device with a display pointed at the user's field of view and a camera pointed at the scene 105. In some embodiments, the handheld device is optionally located in a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing an interaction with CGR content that is triggered based on an interaction that occurs in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the CGR content response is displayed via the HMD. Similarly, a user interface showing an interaction with CGR content that is triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD where the interaction is triggered by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0048] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for purposes of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0049] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communications interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communications buses 204 for interconnecting these and various other components.

[0050] In some embodiments, the one or more communications buses 204 include circuitry that interconnects and controls communications between the system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0051] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 comprises a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0052] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0053] 1, and optionally from one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0054] In some embodiments, the tracking unit 244 is configured to map the scene 105 and track the position of at least the display generation component 120 relative to the scene 105 of FIG. 1, and optionally the position of one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To that end, in various embodiments, the tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the tracking unit 244 includes a hand tracking unit 243 and / or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the position of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 243 is described in more detail below with respect to FIG. 4. In some embodiments, eye tracking unit 245 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with respect to FIG.

[0055] In some embodiments, the adjustment unit 246 is configured to manage and adjust the CGR experience presented to the user by the display generation component 120, and optionally by one or more of the output devices 155 and / or peripheral devices 195. To that end, in various embodiments, the adjustment unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0056] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0057] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0058] Moreover, FIG. 2 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, may vary from implementation to implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0059] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, and by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communications interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communications buses 304 for interconnecting these and various other components.

[0060] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communications between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0061] In some embodiments, the one or more CGR displays 312 are configured to provide a CGR experience to the user. In some embodiments, the one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more CGR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, the one or more CGR displays 312 can present MR or VR content. In some embodiments, the one or more CGR displays 312 can present MR or VR content.

[0062] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0063] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or the non-transitory computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0064] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0065] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0066] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0067] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0068] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0069] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as existing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0070] Moreover, FIG. 3 is intended more to illustrate the functionality of various features that may be present in a particular implementation, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in FIG. 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0071] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by a hand tracking unit 243 (FIG. 2) to track the position and / or movement of one or more parts of a user's hand relative to the scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a coordinate system defined relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to the user's hand. In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0072] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for distinguishing fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or an image of the entire body, and can have either a dedicated sensor with zoom capabilities or high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0073] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with software running on the controller 110 by moving their hands 408 and changing the posture of their hands.

[0074] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0075] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's wrist joints and fingertips.

[0076] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with the motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120, or perform other functions, in response to the pose and / or gesture information.

[0077] In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 440, by way of example, some or all of the processing functionality of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing functions of the image sensor 404 may similarly be integrated into a computer or other computerized device that is controlled by the sensor output.

[0078] FIG. 4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels with respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0079] 4 also illustrates, in accordance with some embodiments, a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406. In FIG. 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand that connects to the wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0080] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 245 (FIG. 2) to track the position and movement of the user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device, such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if display generation component is a handheld device or a CGR chamber, eye tracking device 130 is, optionally, a separate device from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a head-mounted or non-head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally part of a non-head-mounted display generation component.

[0081] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0082] As shown in FIG. 5, in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye tracking camera and lighting source.

[0083] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters have been determined for the eye tracking device 130, images captured by the eye tracking camera may be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0084] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0085] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's viewpoint on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0086] Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The autofocus mechanism of the external camera can then focus on an object or surface in the environment that the user is currently viewing on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

[0087] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0088] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the eye tracking system. Note that the positions and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0089] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0090] FIG. 6 illustrates a glint-assisted eye tracking pipeline according to some embodiments. In some embodiments, the eye tracking pipeline is realized by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in FIG. 1 and FIG. 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0091] As shown in FIG. 6, an eye tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0092] At 610, if the tracking status is yes for the current captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eye.

[0093] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0094] 6 is intended to serve as an example of eye-tracking technology that may be used in certain implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0095] Thus, the description herein includes several embodiments of a three-dimensional environment (e.g., a CGR environment) that includes representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment that is captured and displayed in the three-dimensional environment (e.g., actively via a camera and display of the electronic device or passively via a transparent or semi-transparent display of the electronic device). As mentioned above, the three-dimensional environment is optionally a mixed reality system based on a physical environment where the three-dimensional environment is captured by one or more sensors of the device and displayed via a display generation component. As a mixed reality system, the device can optionally display portions and / or objects of the physical environment such that each of the portions and / or objects of the physical environment appears to exist in the three-dimensional environment displayed by the electronic device. Similarly, the device can optionally display virtual objects in the three-dimensional environment such that each of the portions and / or objects of the physical environment appears to exist in the real world (e.g., the physical environment) by placing the virtual objects at respective positions in the three-dimensional environment that have corresponding positions in the real world. For example, the device optionally displays a vase such that it appears as if the real vase were placed on a table in the physical environment. In some embodiments, each location in the three-dimensional environment has a corresponding location in the physical environment. Thus, when a device is described as displaying a virtual object at a distinct location relative to a physical object (e.g., at or near the location of a user's hand, or at or near a physical table, etc.), the device displays the virtual object at a particular location in the three-dimensional environment such that it appears as if the virtual object were at or near the physical object in the physical world (e.g., the virtual object is displayed at a location in the three-dimensional environment that corresponds to the location in the physical environment where the virtual object would be displayed if the virtual object were a real object at that particular location).

[0096] In some embodiments, real-world objects present in the physical environment that are displayed in the three-dimensional environment can interact with virtual objects that exist only in the three-dimensional environment. For example, the three-dimensional environment can include a table and a vase placed on the table, where the table is a view (or representation) of the physical table in the physical environment and the vase is a virtual object.

[0097] Similarly, the user can optionally use one or more hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the device optionally capture one or more of the user's hands and display a representation of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or, in some embodiments, due to the transparency / semi-transparency of the user interface, or the projection of the user interface onto a transparent / semi-transparent surface, or the portion of the display generating component displaying the projection of the user interface to the user's eye or the field of view of the user's eye, the user's hands are visible through the display generating component with the ability to see the physical environment through the user interface. Thus, in some embodiments, the user's hands are displayed at discrete locations in the three-dimensional environment and are treated as if they were objects in the three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if they were actual physical objects in the physical environment. In some embodiments, the user can move his or her hands to move a representation of the hands in the three-dimensional environment in coordination with the movement of the user's hands.

[0098] In some of the embodiments described below, for example, to determine whether a physical object is interacting with a virtual object (e.g., whether a hand is touching, grabbing, holding, etc., or within a threshold distance from the virtual object), the device can optionally determine an "effective" distance between the physical object in the physical world and the virtual object in the three-dimensional environment. For example, when determining whether and / or how a user is interacting with a virtual object, the device determines the distance between the user's hand and the virtual object. In some embodiments, the device determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of a target virtual object in the three-dimensional environment. For example, the user's one or more hands are placed at a particular position in the physical world, which the device optionally captures and displays at a particular corresponding position in the three-dimensional environment (e.g., the position in the three-dimensional environment where the hand is displayed, in the case where the hand is a virtual hand rather than a physical hand). The position of the hand in the three-dimensional environment is optionally compared to the position of the target virtual object in the three-dimensional environment to determine the distance between the user's one or more hands and the virtual object. In some embodiments, the device optionally determines the distance between a physical object and a virtual object by comparing positions in the physical world (e.g., as opposed to comparing positions in a three-dimensional environment). For example, when determining the distance between one or more hands of a user and a virtual object, the device optionally determines a corresponding position in the physical world of the virtual object (e.g., a position where the virtual object would be located in the physical world if the virtual object were a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and the one or more hands of the user. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object.Thus, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the device optionally performs any of the techniques described above to map the position of the physical object to the three-dimensional environment and / or to map the position of the virtual object to the physical world.

[0099] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed at and / or where and what a physical stylus held by the user is directed at. For example, if the user's gaze is directed at a particular location in the physical environment, the device optionally determines a corresponding location in the three-dimensional environment, and if a virtual object is located at that corresponding virtual location, the device optionally determines that the user's gaze is directed at that virtual object. Similarly, the device can optionally determine where the stylus is pointing in the physical world based on the orientation of the physical stylus. In some embodiments, based on this determination, the device determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical world where the stylus is pointing, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0100] Similarly, embodiments described herein may refer to the location of a user (e.g., a user of a device) and / or the location of a device within a three-dimensional environment. In some embodiments, a user of a device is holding, wearing, or otherwise located at or near an electronic device. Thus, in some embodiments, the location of the device is used as a proxy for the location of the user. In some embodiments, the location of the device and / or the user within the physical environment corresponds to a distinct location within the three-dimensional environment. In some embodiments, the distinct location is a location from which a "camera" or "view" of the three-dimensional environment extends. For example, the location of the device is a location within the physical environment (and its corresponding location within the three-dimensional environment) where the user would see objects within the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other) as the objects are displayed by the display generation component of the device, if the user stood in a position facing a distinct portion of the physical environment displayed by the display generation component of the device. Similarly, if the virtual objects displayed in the three-dimensional environment were physical objects in the physical environment (e.g., the virtual objects were located in the same physical environment location and had the same physical environment size and orientation as in the three-dimensional environment), the position of the device and / or user is where the user would see the virtual objects in the physical environment in the same location, orientation, and / or size (e.g., absolutely and / or relative to each other and to real-world objects) as they were displayed by the display generation component of the device.

[0101] In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0102] Furthermore, in methods described herein in which one or more steps are contingent on one or more conditions being met, it should be understood that the described method can be repeated in multiple iterations such that over the course of the iterations, all of the conditions on which the steps of the method are contingent are met in different iterations of the method. For example, if a method requires performing a first step if a condition is met and performing a second step if the condition is not met, one skilled in the art would understand that the steps recited in the claim are repeated in a particular order until the conditions are met and are no longer met. Thus, a method described with one or more steps that are dependent on one or more conditions being met can be rewritten as a method that is repeated until each condition recited in the method is met. However, this is not required for system or computer readable medium claims in which the system or computer readable medium includes instructions for performing a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency is met without explicitly repeating the steps of the method until all conditions on which the steps of the method are contingent are met. Those skilled in the art will also appreciate that, as with methods having conditional steps, the system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed. User Interface and Related Processes

[0103] Attention now turns to embodiments of user interfaces ("UI") and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, with a display generating component, one or more input devices, and (optionally) one or more cameras.

[0104] 7A-7B show examples of how an electronic device may highlight and / or de-highlight portions of a user interface based on a user's gaze, according to some embodiments.

[0105] FIG. 7A illustrates an electronic device 101 displaying a three-dimensional environment 701 in a user interface via a display generating component (e.g., display generating component 120 of FIG. 1). As described above with reference to FIGS. 1-6, the electronic device 101 optionally includes a display generating component (e.g., a touch screen) and a number of image sensors (e.g., image sensor 314 of FIG. 3). The image sensors optionally include one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user while the user is interacting with the electronic device 101. In some embodiments, the user interface illustrated below may also be implemented on a head-mounted display that includes a display generating component that displays the user interface to the user and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0106] As shown in FIG. 7A , the device 101 captures one or more images of the physical environment 700 (e.g., operating environment 100) around the device 101, including one or more objects in the physical environment 700 around the device 101. In some embodiments, the device 101 displays a representation of the physical environment in a three-dimensional environment 701. For example, the three-dimensional environment 701 includes a representation 702b of a tree (e.g., corresponding to the tree 702a in the physical environment 700) and a representation 704b of a person (e.g., corresponding to the person 704a in the physical environment 700). In some embodiments, the device 101 also displays one or more virtual objects (e.g., objects not in the physical environment 700) in the three-dimensional environment 701. For example, in FIG. 7A , the device 101 displays a virtual object 710 (e.g., an ornament) on the tree representation 702b, a virtual object 706 (e.g., a hat) on the person representation 704b, and a representation 708 of the sun. In some embodiments, a representation of the sun 708 is displayed and processed as a light source within the three-dimensional environment 701, and lighting effects resulting from the representation of the sun 708 are applied by the device 101 to representations of physical and / or virtual objects displayed in the three-dimensional environment 701.

[0107] 7A , device 101 also displays a representation 712 of an application (e.g., app A) within three-dimensional environment 701. Representation 712 is, optionally, a user interface of the application (e.g., app A) and includes content being displayed by the application (e.g., a movie playing, a collection of photos, etc.). In some embodiments, representation 712 is displayed at a position within three-dimensional environment 701 such that it overlays one or more representations of physical and / or virtual objects within three-dimensional environment 701 (e.g., in front of and overlaying a portion of representation 704b of a person corresponding to person 704a in physical environment 700). In some embodiments, representation 712 is positioned within three-dimensional environment 701 such that one or more virtual objects are in front of and overlaying a portion of representation 712 and / or one or more representations of physical objects are in front of and overlaying a portion of representation 712.

[0108] In some embodiments, the device 101 displays the three-dimensional environment 701 and / or the user interface displayed via the display generation component 120 at a particular level of immersion. As described in more detail below with reference to FIGS. 9-16 , the immersion level optionally includes the degree to which the content displayed by the electronic device 101 obscures background content around / behind the user interface and / or virtual objects displayed by the device 101 (e.g., content corresponding to the physical environment, such as representations 702b and 704b, or content other than the user interface of interest), and optionally includes the number of items of background content displayed and the visual characteristics (e.g., color, contrast, opacity) at which the background content is displayed, and / or the angular range of content displayed via the display generation component (e.g., 60 degrees for content displayed with low immersion, 120 degrees for content displayed with medium immersion, 180 degrees for content displayed with high immersion), and / or the percentage of the field of view displayed via the display generation that is consumed by the content displayed by the electronic device (e.g., 33% of the field of view consumed by the content displayed by the electronic device with low immersion, 66% of the field of view consumed by the content displayed by the electronic device with medium immersion, 100% of the field of view consumed by the content displayed by the electronic device with high immersion). Additional details regarding displaying user interfaces at various immersion levels are described with reference to FIGS.

[0109] In FIG. 7A, the device 101 displays the three-dimensional environment 701 at a particular immersion level 718 (e.g., as shown on an immersion scale 716, the leftmost side of which corresponds to no immersion and the rightmost side of which corresponds to maximum immersion) at which the representations 702b and 704b of the physical objects are visible through the display generation component 120. In some embodiments, even when the device 101 does not change the immersion level at which the device 101 displays the three-dimensional environment 701, the device 101 highlights and / or de-highlights various portions of the three-dimensional environment 701 based on the gaze of the user of the device 101. For example, in FIG. 7A, the gaze 714 of the user of the device 101 is directed at the representation 702b of the physical tree. However, in FIG. 7B, the device 101 detects (e.g., using the sensor 314) that the gaze 714 of the user of the device 101 is directed at the representation 712 of app A. In response to detecting a user's gaze 714 directed at representation 712 of app A, without changing the particular immersion level 718 at which the three-dimensional environment 701 was displayed, device 101 optionally de-highlights portions of the three-dimensional environment outside of representation 712, relative to representation 712, as shown in Figure 7B. For example, device 101 optionally dims, blurs, or otherwise obscures the display of representation(s) of physical environment 700 (e.g., representations 702b and 704b and / or representations of the physical environment in which these representations are displayed) displayed via display generation component 120, as shown in Figure 7B. Device 101 may additionally or alternatively dim, blur, or otherwise obscure the display of virtual objects other than representation 712 (e.g., virtual objects toward which a gaze 714 of a user of device 101 is directed), e.g., device 101 optionally dims, blurs, or otherwise obscures the display of representation 710, representation 708, and representation 706, as shown in FIG. 7B. In some embodiments, device 101 dims, blurs, or otherwise obscures both the display of the representation(s) of physical environment 700 displayed via display generation component 120 and the display of virtual objects other than representation 712.In this way, device 101 can reduce the distraction(s) caused by representations other than the representation the user of device 101 is looking at. In some embodiments, if device 101 detects that the user's gaze 714 is directed away from representation 712 (e.g., back toward portions of three-dimensional environment 701 outside representation 712, as in FIG. 7A), device 101 optionally stops de-highlighting portions of three-dimensional environment 701 that are de-highlighted in FIG. 7B and reverts to displaying the various representations shown in FIG. 7A. As previously mentioned, in some embodiments, the de-highlighting described with reference to FIGS. 7A-7B is separate and independent of the change in immersion level at which device 101 displays various user interfaces, which will be described in more detail with reference to FIGS. 9-16.

[0110] In some embodiments, device 101 performs the above-described de-highlighting only when a user of device 101 is looking at a particular type of object (e.g., a representation of a media player application) and not at other types of objects (e.g., representations of other types of applications). For example, in FIGS. 7A-7B, app A is, optionally, a media player application (e.g., representation 712 displays media such as a movie or television program). If the user's gaze 714 were instead directed at representation 706 of a hat, or a representation of a non-media player application, device 101 would, optionally, not de-highlight any object or representation other than the one the user was looking at.

[0111] 8A-8D are flow charts illustrating a method 800 for highlighting and / or de-highlighting a user interface based on a user's gaze, according to some embodiments. In some embodiments, the method 800 is performed on a computer system (e.g., computer system 101 of FIG. 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward on a user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or a camera pointing forward from the user's head). In some embodiments, the method 800 is governed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0112] In method 800, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer) receives (802) user input corresponding to a two-dimensional drawing, such as user input drawn two-dimensional drawing 708 of FIG. 7A , via the one or more input devices (e.g., receives the user input drawing, generates, inserts, or otherwise causes the two-dimensional drawing to be displayed).

[0113] In method 800, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with the display generation component and the eye tracking device (e.g., a mobile device (e.g., tablet, smartphone, media player, or wearable device), or a computer) simultaneously displays (802) a first user interface (804) associated with a first application, such as user interface 712 of FIG. 7A , and a second user interface (806) surrounding at least a portion of the first user interface, such as three-dimensional environment 701 of FIG. 7A , via the display generation component. In some embodiments, the electronic device is a mobile device (e.g., tablet, smartphone, media player, or wearable device), or a computer. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the electronic device, an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting the user interface or making the user interface visible to one or more users, or the like. In some embodiments, the electronic device communicates with one or more input devices, including electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, a motion sensor (e.g., a hand tracking device, a hand motion sensor), and / or an eye tracking device, etc.).

[0114] In some embodiments, the electronic device displays content of the first application in an application window. For example, displays a video in a video content playback application, such as an application that can browse and / or play one or more types of content (e.g., music, songs, television episodes, movies, etc.). In some embodiments, the second user interface is a system user interface, such as a user interface of an operating system of the electronic device on which the first user interface is displayed. In some embodiments, the first user interface is overlaid on the second user interface. In some embodiments, the first user interface is displayed in response to an input to display the first user interface that is received while the electronic device is displaying the second user interface. In some embodiments, the second user interface is not specifically associated with any single application (e.g., because it is an operating system user interface rather than an application user interface). In some embodiments, the second user interface includes or is a three-dimensional environment in which the first user interface is displayed, such as displaying the first user interface within a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment, that is generated, displayed, or otherwise made viewable by an electronic device.

[0115] In some embodiments, while simultaneously displaying the first user interface and a portion of the second user interface surrounding at least a portion of the first user interface, the electronic device detects (808) via the eye tracking device that the user's gaze is directed toward the first user interface, such as gaze 714 directed toward representation 712 of FIG. 7B (e.g., the electronic device determines via the eye tracking device that the user is looking at an area within the first user interface or within an area / volume of the second user interface that contains the first user interface. In some embodiments, the user's gaze is directed toward the first user interface when the user's gaze coincides with a location in the second user interface that is within a threshold distance (e.g., 1 inch, 6 inches, 1 foot, 2 feet, 10 feet, etc.) of the first user interface).

[0116] In some embodiments, in response to detecting a user's gaze being directed toward the first user interface, according to a determination that the one or more criteria are met (e.g., the one or more criteria include a criterion that is met when the first application is a first type of application (e.g., a media player application for browsing and / or watching photos, videos, movies, etc.) but is not met when the first application is a second type of application different from the first type of application (e.g., a word processing application). In some embodiments, the one or more criteria include a criterion that is met when the first application is a first type of application (e.g., a media player application for browsing and / or watching photos, videos, movies, etc.) but is not met when the first application is a second type of application different from the first type of application (e.g., a word processing application). If the one or more criteria are met without any additional input from the user other than the user's gaze being directed at the face (e.g., the second user interface is automatically de-highlighted relative to the first user interface in response to gaze input). In some embodiments, the one or more criteria include a criterion that is met when the user's gaze is directed at the first user interface for longer than a time threshold, such as 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, etc., and is not met otherwise, the electronic device de-highlights (810) the second user interface relative to the first user interface, such as de-highlighting the content of the three-dimensional environment 701 relative to the representation 712 of FIG. 7A. For example, the first user interface is brightened and / or the second user interface is darkened / dimmed. In some embodiments, the first user interface is updated such that a respective visual characteristic of the first user interface changes from having a first value to having a second value different from the first value. In some embodiments, the second user interface is updated such that a respective visual characteristic of the second user interface changes from having a third value to having a fourth value different from the third value. In some embodiments, the second user interface is blurred (if it was not previously blurred) or blurred more (if it was already previously blurred).In some embodiments, the first user interface is expanded in size and thus occupies more and / or overlays more of the second user interface. In some embodiments, if one or more criteria are not met, the second user interface is not de-highlighted relative to the first user interface. In some embodiments, when the user's gaze moves from the first user interface to the second user interface, the de-highlighting of the second user interface relative to the first user interface is reversed (e.g., returning to the relative highlighting that existed before the user's gaze was directed to the first user interface). In some embodiments, when the user's gaze moves from the first user interface to the second user interface, the first user interface is de-highlighted relative to the second user interface (e.g., in the same or similar manner as described herein with respect to de-highlighting the second user interface relative to the first user interface). The above-described method of de-highlighting a second user interface relative to a first user interface provides an efficient way of indicating that the first user interface is the user interface that is in focus, which simplifies interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion with the second user interface and thus avoiding unnecessary interaction with the second user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0117] In some embodiments, the one or more criteria include (812) a criterion that is met when the first application is an application of a first type and that is not met when the first application is an application of a second type different from the first type (e.g., the criterion is met for an application that is an application for browsing and / or viewing content or media (e.g., an application in which movies, pictures, music, television programs, etc. can be viewed) and is not met for an application that is not an application for viewing content or media (e.g., a word processing application, a calendar application, a spreadsheet application, etc.)). In some embodiments, in response to detecting that the user's gaze is directed toward the first user interface, the electronic device refrains from de-highlighting (814) the second user interface relative to the first user interface in accordance with a determination that the one or more criteria are not met (e.g., because the first application is not a content / media browsing and / or viewing application). For example, the display of the first user interface and the second user interface remain as they were before the user's gaze was detected to be directed toward the first user interface, and optionally the relative highlighting of the first user interface relative to the second user interface, and vice versa, remains as they were before the user's gaze was detected to be directed toward the first user interface. Thus, the second user interface is optionally not dimmed or blurred in response to the user's gaze being directed toward the first user interface. In some embodiments, the gaze-based de-highlighting of the first user interface relative to the second user interface is done for all types of applications, including applications other than content and / or media viewing applications (e.g., the one or more criteria do not include the application type criteria described above).The above-described method of performing gaze-based de-highlighting of a user interface based on the type of application associated with the user interface provides a quick and efficient way of only highlighting / de-highlighting a user interface in situations in which such highlighting / de-highlighting is likely to be desired, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by avoiding erroneous highlighting / de-highlighting of the user interface that would require additional user input to correct), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0118] In some embodiments, after detecting that the user's gaze is directed toward the first user interface, and while displaying the second user interface as unhighlighted relative to the first user interface (e.g., the user's gaze is directed toward the first user interface, causing the second user interface to be unhighlighted relative to the first user interface), as in FIG. 7A , the electronic device detects (816) via the eye tracking device that the user's gaze is directed toward the second user interface, e.g., such that gaze 714 moves back outside of representation 712 (e.g., the electronic device determines via the eye tracking device that the user is looking at an area in the second user interface outside of the area / volume occupied by the first user interface. In some embodiments, the user's gaze is directed toward the second user interface when the user's gaze coincides with a location in the first user interface that is within a threshold distance (e.g., 1 inch, 6 inches, 1 foot, 2 feet, 10 feet, etc.) of the second user interface). In some embodiments, in response to detecting the user's gaze being directed toward the second user interface, pursuant to a determination that the one or more second criteria are met (in some embodiments, the one or more second criteria are met without any additional input from the user other than the user's gaze being directed toward the second user interface (e.g., the first user interface is automatically de-highlighted relative to the second user interface in response to gaze input). In some embodiments, the one or more second criteria include criteria that are met if the user's gaze is directed toward the second user interface for longer than a time threshold of 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, etc., and are not met otherwise), the electronic device highlights (818) the second user interface relative to the first user interface (e.g., and / or de-highlights the first user interface relative to the second user interface), such as resulting in the display of FIG. 7A .

[0119] In some embodiments, the display of the first and second user interfaces reverts to the way the two user interfaces were displayed (e.g., in an absolute sense and / or relative to each other) before the user's gaze was detected to be directed toward the first user interface. In some embodiments, the second user interface is brightened and / or blurred and / or otherwise less obscured. In some embodiments, the first user interface is darkened and / or blurred and / or otherwise more obscured. In some embodiments, the second user interface becomes more highlighted relative to the first user interface compared to the relative display of the first and second user interfaces before the user's gaze was directed toward the first user interface. In some embodiments, the second user interface becomes less highlighted relative to the first user interface compared to the relative display of the first and second user interfaces before the user's gaze was directed toward the first user interface. The above-described method of restoring the display of a user interface based on gaze provides a faster and more efficient way of returning to a previous display of a user interface, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not requiring additional or different types of user input to restore the display of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0120] In some embodiments, prior to detecting that the user's gaze is directed toward the first user interface, the first user interface and the second user interface are displayed (820) simultaneously with a representation of at least a portion of the physical environment of the electronic device, such as representations 702b and 704b of FIG. 7A (e.g., representations of parts and / or objects in the physical environment of the electronic device are passed through the display generation component to be displayed simultaneously with the first user interface and / or the second user interface). In some embodiments, the pass-through objects are actively displayed by the electronic device (e.g., by being captured via one or more sensors of the electronic device and generated for display via the display generation component). In some embodiments, the display of the pass-through objects is not obscured by the electronic device (e.g., in the case of a transparent or semi-transparent display generation component that allows a portion of the physical environment of the electronic device to be seen through the display generation component). In some embodiments, the second user interface includes the portion(s) of the physical environment seen through the display generation component. In some embodiments, the second user interface and the first user interface are displayed within a three-dimensional environment displayed via the display generation component that includes a portion or portions of the physical environment visible via the display generation component, in some embodiments, the portion of the physical environment is a physical table positioned on a physical floor of the physical environment of the electronic device.

[0121] In some embodiments, de-highlighting the second user interface relative to the first user interface includes de-highlighting a representation of the portion of the electronic device's physical environment relative to the first user interface (822), such as de-highlighting representations 702b and 704b shown in FIG. 7B. For example, dimming and / or blurring and / or otherwise obscuring the display of the representation of the portion of the electronic device's physical environment (e.g., dimming the representation of a physical table) and / or brightening and / or otherwise highlighting the display of the first user interface. The above-described method of treating the display of the portion of the electronic device's physical environment similarly to the second user interface maintains consistency in the electronic device's response to the user's gaze, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion with the portion of the electronic device's physical environment and thus avoiding unnecessary interactions with the portion of the electronic device's physical environment), which further reduces the electronic device's power usage and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0122] In some embodiments, prior to detecting that the user's gaze is directed toward the first user interface, the first user interface and the second user interface are simultaneously displayed (824) with a virtual object that is not within the physical environment of the electronic device, such as representations 706, 708, and 710 of FIG. 7A (e.g., displaying representations of objects, elements, applications, etc. that are virtual objects rather than representations of physical objects and / or portions of the physical environment of the electronic device. In some embodiments, the second user interface includes a virtual object. In some embodiments, the virtual object is displayed outside of the first and / or second user interface. For example, a representation of a virtual (e.g., non-physical, non-real) vase is displayed as if it were placed on a physical table that is placed on a physical floor of the physical environment of the electronic device. In some embodiments, de-highlighting the second user interface relative to the first user interface may be de-highlighting the representations 706, 708, and 710 shown in FIG. 7B . The method includes de-highlighting the virtual object relative to the first user interface (826), such as highlighting the virtual object. For example, dimming and / or blurring and / or otherwise obscuring the display of the virtual object while brightening and / or otherwise highlighting the display of the first user interface (e.g., dimming a virtual vase displayed as if placed on a physical table in the physical environment of the electronic device). The above-described method of handling the display of the virtual object in a similar manner to the second user interface maintains consistency in the electronic device's response to the user's gaze, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion with the virtual object and thus avoiding unnecessary interactions with the virtual object), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0123] In some embodiments, prior to detecting that the user's gaze is directed toward the first user interface, the first user interface and the second user interface are displayed (828) simultaneously with a representation of at least a portion of the physical environment of the electronic device, such as representations 702b and 704b in FIG. 7A (e.g., representations of parts and / or objects in the physical environment of the electronic device are passed through the display generation component to be displayed simultaneously with the first user interface and / or the second user interface). In some embodiments, the pass-through objects are actively displayed by the electronic device (e.g., by being captured via one or more sensors of the electronic device and generated for display via the display generation component). In some embodiments, the display of the pass-through objects is not obscured by the electronic device (e.g., in the case of a transparent or semi-transparent display generation component that allows a portion of the physical environment of the electronic device to be seen through the display generation component). In some embodiments, the second user interface includes the portion(s) of the physical environment seen through the display generation component. In some embodiments, the second user interface and the first user interface are displayed within a three-dimensional environment displayed via the display generation component that includes a portion or portions of the physical environment visible via the display generation component, in some embodiments, the portion of the physical environment is a physical table positioned on a physical floor of the physical environment of the electronic device.

[0124] In some embodiments, the first user interface and the second user interface are simultaneously displayed (828) with virtual objects that are not in the physical environment of the electronic device, such as representations 706, 708, and 710 of FIG. 7A (e.g., displaying representations of objects, elements, applications, etc. that are virtual objects rather than representations of physical objects and / or portions of the physical environment of the electronic device). In some embodiments, the second user interface includes a virtual object. In some embodiments, the virtual object is displayed outside of the first and / or second user interface. For example, a representation of a virtual (e.g., non-physical, non-real) vase is displayed as if it were placed on a physical table placed on a physical floor of the physical environment of the electronic device. In some embodiments, dehighlighting the second user interface relative to the first user interface (830) includes dehighlighting a representation of a portion of the physical environment of the electronic device relative to the first user interface (832) (e.g., dimming and / or blurring, and / or otherwise obscuring a representation of the portion of the physical environment of the electronic device (e.g., dimming a representation of a physical table) while brightening and / or otherwise highlighting the display of the first user interface), as shown in FIG. 7B , and dehighlighting a virtual object relative to the first user interface (834) (e.g., dimming and / or blurring, and / or otherwise obscuring a representation of a virtual object while brightening and / or otherwise highlighting the display of the first user interface (e.g., dimming a virtual vase displayed as if it were placed on a physical table in the physical environment of the electronic device)), as shown in FIG. 7B .The above-described method of processing the display of virtual objects and parts of the physical environment of the electronic device in a manner similar to a second user interface maintains consistency in the electronic device's response to the user's gaze, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion with the virtual objects and parts of the physical environment and thus avoiding unnecessary interactions with the virtual objects and parts of the physical environment), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0125] In some embodiments, the second user interface includes (836) virtual objects that are not in the physical environment of the electronic device, such as representations 706, 708, and 710 of FIG. 7A. For example, display representations of objects, elements, applications, etc. that are virtual objects in the second user interface (e.g., compared to representations of physical objects and / or portions of the physical environment of the electronic device). For example, a representation of a virtual (e.g., non-physical, non-real) vase is displayed in the second user interface as if it were placed on a physical table that is placed on a physical floor of the physical environment of the electronic device. In some embodiments, the second user interface is a background on and / or within which the first user interface is displayed (e.g., the first user interface is thus overlaid on an area / volume that includes the virtual objects). In some embodiments, the virtual objects are also de-highlighted relative to the first user interface (e.g., in the same manner as the second user interface) in response to the user's gaze being directed toward the first user interface. The above-described method of performing gaze-based highlighting / de-highlighting of a user interface provides a quick and efficient way of highlighting / de-highlighting a user interface in a display space including virtual objects, which simplifies the interaction between a user and an electronic device, improves usability of the electronic device, and makes the user-device interface more efficient, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0126] In some embodiments, prior to detecting the user's gaze being directed toward the first user interface, the first user interface is displayed (838) at a first immersion level, such as immersion level 718 of FIG. 7A (e.g., displayed at a discrete immersion level described with reference to methods 1000, 1200 and / or 1600), and de-highlighting the second user interface relative to the first user interface includes maintaining display of the first user interface (840) at the first immersion level, such as maintaining the same immersion level 718 of FIG. 7B (e.g., continuing to be displayed at a discrete immersion level described with reference to methods 1000, 1200 and / or 1600). For example, gaze-based de-highlighting of the second user interface relative to the first user interface described herein in method 800, optionally does not change the immersion level at which the displayed user interface(s) are displayed. In some embodiments, the de-highlighting and / or highlighting of one user interface relative to another user interface described herein in method 800 is independent of the change in display characteristics resulting from a change in immersion level, as described with reference to methods 1000, 1200, and / or 1600. In some embodiments, the de-highlighting and / or highlighting of one user interface relative to another user interface described herein in method 800 is additive to and / or combined with (e.g., whether contributing positively or negatively) the change in display characteristics resulting from a change in immersion level, as described with reference to methods 1000, 1200, and / or 1600.The above-described method of performing gaze-based highlighting / de-highlighting of a user interface provides a quick and efficient way of highlighting / de-highlighting a user interface without changing the immersion level of the displayed user interface, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding erroneous changes to the immersion level of the displayed user interface that would require additional user input to correct), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0127] In some embodiments, the first user interface is displayed (842) overlaid on the second user interface, such as representation 712 overlaid on the content of the three-dimensional environment 701 in Figure 7A. For example, the second user interface is a background upon which and / or within which the first user interface is displayed such that portions of the second user interface (e.g., portions of the second user interface that are behind the first user interface) are obscured (e.g., not visible) by the first user interface, and other portions of the second user interface (e.g., portions of the second user interface that surround and / or are not behind the first user interface) are not obscured (e.g., visible) by the first user interface. The above-described method of performing gaze-based highlighting / de-highlighting of a user interface overlaid on another user interface provides a quick and efficient way of highlighting / de-highlighting the overlaid user interface, which simplifies the interaction between a user and an electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion due to the background behind the overlaid user interface and thus avoiding unnecessary interaction with the background behind the overlaid user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0128] 9A-9C show examples of immersion level settings for an operating system user interface compared to an application user interface.

[0129] FIG. 9A illustrates an electronic device 101 displaying a three-dimensional environment 901 in a user interface via a display generating component (e.g., display generating component 120 of FIG. 1). As described above with reference to FIGS. 1-6, the electronic device 101 optionally includes a display generating component (e.g., a touch screen) and a number of image sensors (e.g., image sensor 314 of FIG. 3). The image sensors optionally include one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user while the user is interacting with the electronic device 101. In some embodiments, the user interface illustrated below may also be implemented on a head-mounted display that includes a display generating component that displays the user interface to the user and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0130] As shown in FIG. 9A , the device 101 captures one or more images of the physical environment 900 (e.g., operating environment 100) surrounding the device 101, including one or more objects in the physical environment 900 surrounding the device 101. In some embodiments, the device 101 displays a representation of the physical environment in the three-dimensional environment 901. For example, the three-dimensional environment 901 includes a representation of a tree 902b (e.g., corresponding to the tree 902a in the physical environment 900) and a representation of a person 904b (e.g., corresponding to the person 904a in the physical environment 900). In some embodiments, the device 101 also displays one or more virtual objects (e.g., objects not in the physical environment 900) in the three-dimensional environment 901. For example, in FIG. 9A , the device 101 displays a virtual object 910 (e.g., an ornament) on the tree representation 902b, a virtual object 906 (e.g., a hat) on the person representation 904b, and a representation of the sun 908. In some embodiments, a representation of the sun 908 is displayed and treated as a light source within the three-dimensional environment 901, and lighting effects resulting from the representation of the sun 908 are applied by the device 101 to representations of physical and / or virtual objects displayed in the three-dimensional environment 901.

[0131] In FIG. 9A , device 101 is also displaying a user interface 912 of the operating system of device 101. User interface 912 is, optionally, an application browsing user interface of the operating system of device 101 that may display / launch one or more applications accessible on device 101. In some embodiments, user interface 912 is any other user interface of the operating system of device 101 (e.g., as compared to being a user interface of an application on device 101). In FIG. 9A , user interface 912 includes icons corresponding to different applications accessible on device 101 (e.g., an app A icon, an app B icon, an app C icon, and an app D icon), which are selectable to display individually selected applications via display generation component 120 of device 101. In some embodiments, user interface 912 is displayed at a position within three-dimensional environment 901 such that it overlays one or more representations of physical and / or virtual objects within three-dimensional environment 901 (e.g., in front of and overlaying a portion of person representation 904b corresponding to person 904a in physical environment 900). In some embodiments, the user interface 912 is positioned within the three-dimensional environment 901 such that one or more virtual objects are in front of and overlay portions of the user interface 912 and / or such that one or more representations of physical objects are in front of and overlay portions of the user interface 912.

[0132] In some embodiments, the device 101 displays the three-dimensional environment 901 and / or the user interface displayed via the display generation component 120 at a particular immersion level. In FIG. 9A, the device 101 displays the three-dimensional environment 901 at a particular immersion level 918 (e.g., as shown on an immersion scale 916, the leftmost side of which corresponds to no immersion and the rightmost side of which corresponds to maximum immersion) where the representations of the physical objects 902b and 904b are visible via the display generation component 120. In some embodiments, the immersion level includes the degree to which the content displayed by the electronic device obscures background content (e.g., content other than the user interface 912) around / behind the user interface 912, optionally including the number of items of background content displayed and the visual characteristics (e.g., color, contrast, opacity) at which the background content is displayed. In some embodiments, the background content is included in the background against which the user interface 912 is displayed. In some embodiments, the background content includes additional user interfaces (e.g., user interfaces generated by the device 101 corresponding to applications, system user interfaces, etc.), virtual objects not related to or included in the user interface 912 (e.g., files generated by the device 101, representations of other users, etc.), and / or real objects (e.g., pass-through objects representing real objects in the physical environment of the electronic device 101, displayed by the device as seen through the display generation component). In some embodiments, at a first (e.g., low) immersion level, the background, virtual, and / or real objects are displayed in an obscured manner. For example, a separate user interface having a low immersion level is optionally displayed simultaneously with the background content, which is optionally displayed at full brightness, color, and / or translucency. In some embodiments, at a second (e.g., high) immersion level, the background, virtual, and / or real objects are displayed in an obscured manner (e.g., dimmed, blurred, removed from view, etc.).For example, a separate user interface having a high immersion level is displayed (e.g., in a full screen or fully immersive mode) without simultaneously displaying background content. As another example, a user interface displayed at an intermediate immersion level is displayed simultaneously with dimmed, blurred, or otherwise non-highlighted background content. In some embodiments, the visual characteristics of the background objects differ between the background objects. For example, at a particular immersion level, one or more first background objects are visually less highlighted (e.g., dimmed, blurred, or displayed with increased transparency) than one or more second background objects, and one or more third background objects cease to be displayed. Further details regarding immersion levels are described herein, including with reference to methods 1200, 1400, and 1600.

[0133] 9A , input is received on an input element 920 of device 101. Input element 920 is optionally a mechanical input element, such as a slider element or a rotary input element, that can be manipulated to change the immersion level of the user interface(s) currently being displayed by device 101, with the amount and direction of the change in immersion (e.g., increase or decrease) based on the magnitude and direction of manipulation of input element 920 (e.g., the direction and / or magnitude of movement of input element 920, or the direction and / or magnitude of rotation of a rotatable input element). In some embodiments, device 101 and / or input element 920 generate haptic feedback (e.g., a minor tactile output for each increase or decrease in an immersion stage, and a different major tactile output upon reaching a minimum or maximum immersion) as the immersion level changes.

[0134] In FIG. 9A , the input received on the input element 920 is to slide the input element 920 upward to increase the immersion level of the currently displayed user interface(s) by an amount based on the amount of upward movement of the element 920. In response to the input of FIG. 9A , the device 101 increases the immersion level at which the operating system user interface 912 is displayed to an intermediate immersion level, as shown in FIG. 9B and the immersion scale 916. Notably, in some embodiments, increasing the immersion level at which the user interface 912 is displayed, as shown in FIG. 9B , causes the device 101 to display the user interface at a larger size via the display generation component 120. Further, as discussed above, additionally or alternatively, content displayed around and / or behind the user interface 912 (e.g., representations of the physical objects 902b and 904b and the virtual objects 910 and 906) is dimmed and / or blurred in FIG. 9B , while the user interface 912 is not dimmed or blurred. In some embodiments, increasing the immersion level at which the currently displayed user interface(s) are displayed additionally or alternatively increases atmospheric visual and / or audio effects associated with the virtual or physical objects displayed via display generation component 120. For example, in Figure 9B, the light beam emanating from the sun representation 908 has become longer (e.g., corresponding to more virtual light emanating from representation 908), and the decoration 910 on the tree representation 902b has begun to glow as a result. Additional or alternative changes to atmospheric lighting, visual, audio, and other effects associated with changes in immersion level are also contemplated.

[0135] As mentioned above, in some embodiments, the immersion levels at which different user interfaces are displayed are optionally set independently of one another. For example, a change in the immersion level at which an operating system user interface was displayed optionally does not change the immersion level at which an application user interface was displayed, and in some embodiments a change in the immersion level at which a user interface of a first application was displayed does not change the immersion level at which a user interface of a second application was displayed. Thus, in some embodiments, in response to an input that switches from displaying one user interface to displaying another user interface, device 101 optionally displays the switched-to user interface at the immersion level at which that user interface was last displayed, regardless of any changes in immersion applied to the currently displayed user interface.

[0136] For example, in Figure 9B, device 101 detects an input to select an icon corresponding to app C in user interface 912. In response, device 101 displays user interface 934 for app C, as shown in Figure 9C. App C is, optionally, an application accessible via device 101 (e.g., installed on the device), and is, optionally, any type of application (e.g., a content viewing application, a word processing application, etc.). App C is, optionally, distinct from the operating system of device 101, and thus user interface 934 is, optionally, a user interface for app C, rather than a user interface for the operating system of device 101.

[0137] 9C, in response to the input to display app C, device 101 is displaying user interface 934 (e.g., as indicated by immersion scale 930 for app C) at a higher immersion level 932 than the immersion level 918 at which device 101 displayed user interface 912 in FIG. 9B. This is, optionally, because, as discussed above, the immersion level setting for an application is independent of the immersion level setting for the operating system of device 101. Thus, it is, optionally, the higher immersion level at which device 101 is displaying user interface 934 of app C in FIG. 9C is the immersion level at which user interface 934 was last displayed by device 101. If user interface 934 was last displayed at a lower immersive level than immersive level 918 of user interface 912 shown in Figure 9B, device 101 optionally displays user interface 934 of Figure 9C at that lower immersive level rather than the higher immersive level shown in Figure 9C. Also shown in Figure 9C is an increase in the size of user interface 934 on display generation component 120 (compared to the size of user interface 912 of Figure 9B) and an increase in the darkening and / or blurring of content outside / around user interface 934 (compared to the darkening and / or blurring of content outside / around user interface 912 of Figure 9B), these changes in display characteristics being, optionally, part of the changes that occur in response to increased immersion, as described above. Additional or alternative changes in display characteristics as described above are also contemplated.

[0138] As discussed above, a change in the immersion level at which a user interface in one application is displayed optionally does not change the immersion level at which the user interfaces of other applications and / or the user interface of the operating system are displayed. For example, if device 101 detects an input that changes the immersion level of user interface 934 of app C of FIG. 9C (e.g., via input element 920), device 101 changes the immersion level at which user interface 934 is displayed according to the input, as discussed above. In response to an input to redisplay user interface 912 (e.g., an input to stop displaying user interface 934) after changing the immersion level at which user interface 934 was displayed, the device optionally returns to the display of FIG. 9B where user interface 912 is displayed at immersion level 918, which is the immersion level at which user interface 912 was last displayed and is independent of the changed immersion level at which user interface 934 was displayed when the input to redisplay user interface 912 was detected.

[0139] In some embodiments, applications (e.g., app C) include controls in their user interfaces to change the immersion level at which those applications are displayed. For example, in some embodiments, app C's user interface 934 includes controls to increase or decrease the immersion level at which the user interface 934 is displayed, and the immersion at which app C is displayed is changed by device 101 in response to interaction with these controls, and additionally or alternatively, in response to interaction with input element 920. Further, in some embodiments, an immersion above a threshold (e.g., maximum immersion) of the operating system user interface is attainable in response to input detected at input element 920. However, in some embodiments, an immersion above that threshold for the application user interface is not attainable in response to input detected at input element 920, and in some embodiments, the immersion at which the application user interface is displayed can only be reached using input elements, and once that threshold is reached, a different type of input is required to increase the immersion at which the application user interface is displayed beyond that threshold. 9C , the immersion level 932 at which user interface 934 is displayed is, optionally, the highest immersion that can be reached for user interface 934 via input at input element 920. To increase the immersion beyond immersion level 932, device 101 optionally requires input selecting a user interface element shown in user interface 934. For example, in FIG. 9C , user interface 934 includes user interface element 950 that is selectable to increase the immersion at which user interface 934 is displayed beyond immersion level 932 (e.g., user input selecting user interface element 950 optionally causes device 101 to display user interface 934 at maximum immersion or full screen, where content surrounding user interface 934 is not displayed by device 101).In some embodiments, user interface 934 only includes element 950 once the immersion level at which user interface 934 is displayed reaches a threshold immersion level as discussed above. In some embodiments, user interface 934 includes element 950 regardless of whether the immersion level at which user interface 934 is displayed reaches a threshold immersion level as discussed above.

[0140] 10A-10K are flow charts illustrating a method 1000 for defining different user interface immersion levels independently of one another, according to some embodiments. In some embodiments, the method 1000 is performed on a computer system (e.g., computer system 101 of FIG. 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward on a user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras) or a camera pointing forward from the user's head). In some embodiments, the method 1000 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0141] In method 1000, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer) displays (1002) a first separate user interface of an operating system of the electronic device, such as user interface 912 of FIG. 9B . In some embodiments, the electronic device is a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the electronic device, an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting the user interface or making the user interface visible to one or more users, or the like. In some embodiments, the electronic device is in communication with one or more input devices, including electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.), and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, a motion sensor (e.g., a hand tracking device, a hand motion sensor), a physical mechanical input element included in the electronic device (e.g., a button, a rotating mechanical element, a switch, etc.), and / or an eye tracking device, etc.

[0142] In some embodiments, the first separate user interface is a user interface generated and / or displayed by an operating system of the electronic device, rather than by an application on (e.g., installed) the electronic device. For example, the first separate user interface is an application browsing and / or launching user interface that is, optionally, a system user interface, and optionally includes a plurality of selectable representations of different applications that, when selected, causes the electronic device to display the user interface of the selected application (e.g., launches the selected application). In some embodiments, the first separate user interface is displayed within a computer-generated reality (CGR) environment, such as a virtual reality (VR), mixed reality (MR), or augmented reality (AR) environment, that is generated, displayed, or otherwise made viewable by the electronic device.

[0143] In some embodiments, the first individual user interface is displayed (1004) at a first immersion level, such as immersion level 918 of FIG. 9B (e.g., in some embodiments, the immersion level includes the degree to which content displayed by the electronic device obscures background content (e.g., content other than the first individual user interface) around / behind the first individual user interface, and optionally includes the number of items of background content displayed and the visual characteristics (e.g., color, contrast, opacity) with which the background content is displayed and / or the visual characteristics (e.g., color, contrast, opacity) with which the background content is displayed via the display generation component). The range of angles of the content displayed via the display generation that is consumed by the content displayed by the electronic device (e.g., 60 degrees for content displayed at low immersion, 120 degrees for content displayed at medium immersion, 180 degrees for content displayed at high immersion), and / or the percentage of the field of view displayed via the display generation that is consumed by the content displayed by the electronic device (e.g., 33% of the field of view consumed by the content displayed by the electronic device at low immersion, 66% of the field of view consumed by the content displayed by the electronic device at medium immersion, 100% of the field of view consumed by the content displayed by the electronic device at high immersion). In some embodiments, the background content is included in the background against which the first separate user interface is displayed. In some embodiments, the background content includes additional user interfaces (e.g., user interfaces generated by the device corresponding to applications other than the application of the first individual user interface, system user interfaces), virtual objects not related to or included in the first individual user interface (e.g., files generated by the device, representations of other users, etc.), and / or real objects (e.g., pass-through objects representing real objects in the physical environment of the electronic device displayed by the device as the objects are seen through the display generating components and / or as seen through a transparent or translucent display generating component because the electronic device does not obscure / obstruct the visibility of the objects through the display generating components).In some embodiments, at a first (e.g., low) immersion level, the background, virtual, and / or real objects are displayed in an obscured manner. For example, a separate user interface with a low immersion level is optionally displayed simultaneously with background content, which is optionally displayed in full brightness, color, and / or translucency. In some embodiments, in a separate user interface displayed in a non-immersive (or very low) state, no virtual user interface elements are displayed (or are barely displayed), and only (or mostly only) the background content is displayed, optionally via a display generation component. In some embodiments, at a second (e.g., high) immersion level, the background, virtual, and / or real objects are displayed in an obscured manner (e.g., dimmed, blurred, removed from view, etc.). For example, a separate user interface with a high immersion level is displayed (e.g., in a full screen or fully immersive mode) without simultaneously displaying background content. As another example, a user interface displayed at an intermediate immersion level is displayed simultaneously with dimmed, blurred, or otherwise unhighlighted background content. In some embodiments, the visual characteristics of the background objects differ between the background objects. For example, at a particular immersion level, one or more first background objects are visually less highlighted (e.g., dimmed, blurred, or displayed with increased transparency) than one or more second background objects, and one or more third background objects cease to be displayed. Further details regarding immersion levels are described herein, including with reference to methods 1200, 1400, and 1600.

[0144] In some embodiments, while displaying the first individual user interface at the first immersion level, the electronic device receives (1006) a first user input via one or more input devices corresponding to a request to display a second individual user interface of the individual application, such as an input on an icon of app C in FIG. 9B (e.g., detecting a selection of an application icon of the individual application within an application browsing user interface). In some embodiments, the first user input is a voice input to display the individual application without selecting the individual application's icon, whether or not it is displayed in the first individual user interface. In some embodiments, in response to receiving the first user input, the electronic device, via the display generation component, displays (1008) a second individual user interface of the individual application, such as the user interface 934 shown in FIG. 9C (e.g., in some embodiments, the second individual user interface is displayed overlaid on the first individual user interface, both of which optionally continue to be displayed on the background and / or within the three-dimensional environment. In some embodiments, displaying the second individual user interface includes ceasing to display the first individual user interface, such that the second individual user interface (but not the first individual user interface) continues to be displayed on the background and / or within the three-dimensional environment), and the second individual user interface of the individual application is displayed (1010) at a second immersion level (e.g., an immersion level independent of the immersion level of the operating system user interface), such as the immersion level 932 shown in FIG. 9C. Thus, in some embodiments, the immersion level of the operating system user interface does not affect the immersion level of the application user interface. In some embodiments, the second immersion level at which the second individual user interface was displayed is the immersion level at which the second individual user interface was last displayed (prior to the current display of the second individual user interface).In some embodiments, the immersion level of the operating system user interface affects the immersion level of the application user interface (e.g., if the system user interface is set to a first immersion level, subsequently displaying the application user interface includes displaying the application user interface at that same first immersion level, even if the application user interface was last displayed at a different immersion level).

[0145] In some embodiments, while displaying the second individual user interface at the second immersion level, the electronic device receives (1012), via one or more input devices, a second user input corresponding to a request to change a current immersion level associated with the electronic device, such as an input on input element 920 of FIG. 9C (e.g., a voice input, a hand gesture, a selection of a displayed user interface element, and / or detection of an input (e.g., a rotational input) at a mechanical input element included in the electronic device corresponding to a request to increase or decrease the immersion level of the currently displayed user interface. In some embodiments, in response to receiving the second user input, the electronic device displays (1014), via the display generation component, the second individual user interface at a third immersion level different from the second immersion level (e.g., increasing or decreasing the immersion level of the second individual user interface in accordance with the second user input).

[0146] In some embodiments, while displaying the second individual user interface at the third immersion level, the electronic device receives (1016) a third user input via one or more input devices corresponding to a request to display the first individual user interface (e.g., an input to stop displaying the second individual user interface, an input to navigate backwards in the navigation hierarchy of the electronic device, an input to redisplay the application browsing user interface, etc.), such as an input to stop displaying the user interface 934 of FIG. 9C (e.g., without an explicit user input corresponding to a request to change the immersion level of the user interface). In some embodiments, in response to receiving the third user input, the electronic device displays (1018) the first individual user interface via the display generation component, and the first individual user interface is displayed (1020) at the first immersion level (e.g., the same immersion level that the first individual user interface had when the second individual user interface was displayed in response to the first user input), such as user interface 912 at immersion level 918 of FIG. Thus, in some embodiments, the immersion level of the first individual user interface did not change as a result of the change in the immersion level of the second individual user interface). Thus, in some embodiments, a change to the immersion level of the application's user interface, optionally, does not cause a corresponding change to the immersion level of the system user interface. In some embodiments, a change to the immersion level of the system user interface does not cause a change to the immersion level of the application's user interface. In some embodiments, a change to the immersion level of the application's user interface, optionally, causes a corresponding change to the immersion level of the system user interface (e.g., the immersion levels increase / decrease together) and / or vice versa.In some embodiments, the immersion level of the operating system user interface is alterable via a mechanical input element included in the electronic device, but the immersion level of the application user interface is not alterable via that mechanical input element, and in some embodiments, the immersion level of the application user interface is alterable via a control displayed via a display generating component. The above-described method of maintaining the immersion level of the system user interface despite changes in the immersion level of the application user interface provides a flexible architecture in which the immersion level may be defined and / or changed independently for different user interfaces, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user device interface more efficient (e.g., by avoiding erroneous immersion level changes in another user interface as a result of a change in the immersion level in a current user interface), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0147] In some embodiments, the one or more input devices include a rotatable input element, and receiving a second user input corresponding to a request to change a current immersion level associated with the electronic device includes detecting a rotation of the rotatable input element (1022), as described with reference to input element 920. For example, the electronic device includes or communicates with a mechanical input element that rotates in response to a rotation user input provided to the electronic device, and in some embodiments may be pressed in response to a press user input provided to the electronic device. In some embodiments, the rotation input of the mechanical input element corresponds to an input that changes a current immersion level of the user interface(s) currently displayed by the electronic device via a display generation component. In some embodiments, the direction of the rotation (e.g., clockwise or counterclockwise) defines whether the input is a request to increase (e.g., clockwise) or decrease (e.g., counterclockwise) the immersion level of the displayed user interface(s). In some embodiments, the magnitude of the rotation defines the amount by which the immersion level of the displayed user interface(s) is changed. The above-described method of changing the immersion level of a displayed user interface(s) provides a quick and efficient method of changing the immersion level of a displayed user interface(s), which simplifies the interaction between a user and an electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding the need to display user interface elements to change the immersion level), which further reduces power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0148] In some embodiments, while detecting rotation of the rotatable element (1024), in accordance with a determination that the current immersion level of the currently displayed user interface is a minimum immersion level, the electronic device generates (1026) via a tactile output generator (e.g., coupled to the rotatable input element and / or coupled to a housing of a device to which the rotatable input element is optionally coupled, etc.) a first tactile output having a distinct characteristic having a first value, as described with reference to FIGS. 9A-9C (e.g., if in response to the rotational input detected at the rotatable input element the displayed user interface(s) reaches a minimum immersion level (e.g., no immersion), the electronic device optionally generates a first type of tactile output (e.g., a major tactile output) at the rotatable input element to indicate that a minimum of a range of available immersion levels of the displayed user interface(s) has been reached). The first type of tactile output optionally has a first magnitude, a first texture, a first frequency, etc. In some embodiments, the electronic device generates a tactile output at a housing of the electronic device (eg, rather than at / directly to the rotatable input element).

[0149] In some embodiments, in accordance with a determination that the current immersion level of the currently displayed user interface is the maximum immersion level, the electronic device generates (1028), via the tactile output generator, a second tactile output having a distinct characteristic having a first value, as described with reference to FIGS. 9A-9C (e.g., in response to a rotational input detected at the rotatable input element, if the displayed user interface(s) reaches the maximum immersion level (e.g., full immersion), the electronic device optionally generates a first type of tactile output (e.g., a major tactile output) at the rotatable input element to indicate that a maximum value of a range of available immersion levels of the displayed user interface(s) has been reached). The first type of tactile output optionally has a first magnitude, a first texture, a first frequency, etc. In some embodiments, the electronic device generates a tactile output at a housing of the electronic device (e.g., rather than at / directly to the rotatable input element).

[0150] In some embodiments, in accordance with a determination that the current immersion level of the currently displayed user interface is an intermediate immersion level, the electronic device generates (1030) via the tactile output generator a third tactile output having a distinct characteristic having a second value different from the first value, as described with reference to FIGS. 9A-9C. For example, in response to a rotational input detected at the rotatable input element, when the displayed user interface(s) reaches a predefined intermediate immersion level (e.g., moves from one immersion stage to the next immersion stage), the electronic device optionally generates a second type of tactile output (e.g., a minor tactile output) at the rotatable input element to indicate reaching a predefined level in the range of available immersion levels of the displayed user interface(s). The second type of tactile output optionally has a second magnitude, a second texture, a second frequency, etc. In some embodiments, the electronic device generates a tactile output at the housing of the electronic device (e.g., not at / directly to the rotatable input element). In some embodiments, different applications and / or operating systems have different minimum, maximum, and / or default intermediate immersion levels for their respective user interfaces, and thus in some embodiments, the tactile output described above is optionally generated at different immersion levels depending on which user interface is currently being adjusted. In some embodiments, the electronic device can display the user interface(s) at (e.g., one of) multiple immersion stages (e.g., 2-1000 immersion stages). In some embodiments, the immersion stages progress monotonically toward more immersion (e.g., from no immersion to maximum immersion). As the electronic device moves from one immersion stage to the next, as described herein, the electronic device optionally changes display characteristics of one or more portions of the displayed user interface(s).For example, as immersion increases, the electronic device may perform one or more of: reducing an amount of the physical environment displayed in the user interface, blurring and / or decreasing the brightness or saturation of the physical environment displayed in the user interface, increasing an amount of the field of view of the display generation component occupied by one or more virtual objects, unblurring and / or increasing the brightness or saturation of one or more virtual objects, or any of the other effects described herein with reference to methods 1000, 1200, 1400, or 1600. In some embodiments, the electronic device optionally makes opposite changes as one progresses from one immersion stage to the previous immersion stage in descending order. In some embodiments, different immersion stages result in different changes in characteristics to what the electronic device displays. For example, in one set of immersion stages (e.g., a lower immersion stage level), the electronic device may initially move through stages of increasing blurring or obscuring portions of the physical environment displayed via the display generating components as it moves through the first set of immersion stages, in a subsequent second set of immersion stages, the electronic device may optionally move through stages of darkening (e.g., blurred) portions of the physical environment displayed via the display generating components as it moves through the second set of immersion stages, and in a subsequent third set of immersion stages, the electronic device may optionally move through stages of replacing portions of the physical environment displayed via the display generating components (e.g., blurred and / or darkened) with virtual objects as it moves through the third set of immersion stages. In some embodiments, the electronic device optionally makes the opposite changes as it progresses through descending sets of immersion stages.The above-described methods of generating a tactile output provide a quick and efficient way of indicating the current immersion level of the displayed user interface(s), which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding the need to display user interface elements to indicate the immersion level), which further reduces the power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0151] In some embodiments, the second user input includes a first portion and a second portion (e.g., a total of 360 degrees clockwise rotation, a first clockwise rotation of 180 degrees followed by a second clockwise rotation of 180 degrees) (1032). In some embodiments, while receiving the second user input (1034), in response to receiving the first portion of the second user input, the electronic device, via the display generation component, displays (1036) a second separate user interface at a fourth immersion level that is between the second and third immersion levels (e.g., increasing the immersion level as the rotatable input element rotates and / or in proportion to the amount of rotation of the rotatable input element in the first portion of the second user input), as described with reference to FIGS. In some embodiments, in response to receiving the second portion of the second user input, the electronic device, via the display generation component, displays (1038) the second separate user interface at a third immersion level, as described with reference to Figures 9A-9C (e.g., further increasing the immersion level as the rotatable input element rotates and / or proportional to the amount of rotation of the rotatable input element in the first portion of the second user input). Thus, in some embodiments, the electronic device gradually changes the immersion level of the display user interface(s) as the rotatable input element is rotated and / or proportional to and / or according to the rotation of the rotatable input element (e.g., based on the magnitude and / or direction of the user input). The above-described methods of changing the immersion level provide real-time feedback to the user while changing the immersion level, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, makes the user-device interface more efficient (e.g., by avoiding transitions to erroneous immersion levels, which avoids the need for further user input to reverse some or all of the change in immersion level), which further reduces power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0152] In some embodiments, receiving a second user input corresponding to a request to change a current immersion level associated with the electronic device (1040), as described with reference to FIGS. 9A-9C, includes: detecting an interaction with a first individual control displayed with the second individual user interface in accordance with a determination that the second individual user interface is a user interface of the first application (1042); and detecting an interaction with a second individual control displayed with the second user interface, different from the first individual control, in accordance with a determination that the second individual user interface is a user interface of a second application different from the first application (1044). For example, in some embodiments, the different applications display and / or have different controls for changing the immersion level of the application's user interface. For example, the user interface of the first application optionally has a control for changing the immersion level of the user interface by steps of five immersion stages (e.g., an up / down control for stepping up / down the immersion stage by increments of five from 0 to 100). The user interface of the second application optionally has a control for changing the immersion level of the user interface by steps of one immersion stage (e.g. a displayed dial rotatable in response to user input to step up / down the immersion stage by one stage from 0 to 100 according to rotation of the displayed dial). In some embodiments, some applications and / or user interfaces display a control for changing the immersion level (e.g. the immersion level is changed via interaction with the displayed control, optionally not changed via rotation of a rotatable input element, or optionally may be changed via rotation of a rotatable input element), while some applications and / or user interfaces do not display a control for changing the immersion level (e.g. the immersion level may only be changed via rotation of a rotatable input element).For example, in some embodiments, the immersion level of an operating system user interface is optionally changeable via rotation of a rotatable input element (e.g., not via a displayed user interface control), while the immersion level of an application running on the electronic device is optionally changeable via rotation of a rotatable input element and / or via a displayed user interface control. The above-described method with application-based immersion control provides a means for changing the immersion to better fit a displayed application and / or user interface, which simplifies the interaction between a user and the electronic device, improves the usability of the electronic device, makes the user device interface more efficient (e.g., by avoiding inputs to change the immersion level of a user interface that does not well fit the user interface, which avoids the need for further user inputs to correct such inputs), which further reduces the power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0153] In some embodiments, at a second immersion level, a first amount of the electronic device's physical environment is replaced (1046) with a separate user interface displayed by the electronic device via the display generation component as shown in Figures 9A-9B, and at a third immersion level, a second amount different from the first amount of the electronic device's physical environment is replaced (1048) with a separate user interface displayed by the electronic device via the display generation component as shown in Figures 9B-9C. For example, in some embodiments, the electronic device is displaying content / user interface(s) in addition to displaying a representation(s) of the electronic device's physical environment (e.g., objects in the physical environment, surfaces in the physical environment, etc.) as described with reference to methods 1200, 1400, and 1600. In some embodiments, the representation(s) of the physical environment are passed through the display generation component as described above. In some embodiments, at lower immersion levels, more representation(s) of the physical environment are passed through the display generation components while the electronic device is displaying the content / user interface(s) via the display generation components. In some embodiments, at higher immersion levels, fewer representation(s) of the physical environment are passed through the display generation components while the electronic device is displaying the content / user interface(s) via the display generation components. For example, at a first immersion level, the electronic device optionally passes through 80% of the representation(s) of the physical environment (e.g., representations of physical objects) via the display generation components, while at a second, higher immersion level, the electronic device optionally passes through 50% of the representation(s) of the physical environment (e.g., representations of physical objects) via the display generation components. In some embodiments, the lower and higher immersion levels optionally cause more or less of the representation(s) of the physical environment to be passed through the display generation components, respectively.In some embodiments, at higher immersion levels (e.g., because the content / user interface(s) occupy a large portion of the display area of ​​the display generation component), more of the representation(s) of the physical environment is replaced by the content / user interface(s) displayed by the electronic device, and at lower immersion levels (e.g., because the content / user interface(s) occupy a small portion of the display area of ​​the display generation component), less of the representation(s) of the physical environment is replaced by the content / user interface(s) displayed by the electronic device. The above-described methods of varying the amount of pass-through content (e.g., replaced by the electronic device) reduce interference between the user interface(s) displayed by the electronic device and the physical environment as the immersion level increases, which simplifies the interaction between the user and the electronic device and improves usability of the electronic device, making the user-device interface more efficient (e.g., by avoiding confusion caused by pass-through the physical environment at high immersion levels), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0154] In some embodiments, at a second immersion level, a first amount of the electronic device's physical environment is de-highlighted (1050) by the electronic device via the display generation component as shown in Figures 9A-9B, and at a third immersion level, a second amount different from the first amount of the electronic device's physical environment is de-highlighted (1052) by the electronic device via the display generation component as shown in Figures 9B-9C. For example, in some embodiments, the electronic device is displaying content / user interface(s) in addition to displaying a representation(s) of the electronic device's physical environment (e.g., objects in the physical environment, surfaces in the physical environment, etc.) as described with reference to methods 1200, 1400, and 1600. In some embodiments, the representation(s) of the physical environment are passed through the display generation component as described above. In some embodiments, at the lower immersion level, while the electronic device is displaying the content / user interface(s) via the display generation components, the representation(s) of the physical environment are displayed via the display generation components at a first highlight level (e.g., color, brightness, less blurred, etc.). In some embodiments, at the higher immersion level, while the electronic device is displaying the content / user interface(s) via the display generation components, the representation(s) of the physical environment are displayed via the display generation components at a second highlight level that is lower than the first highlight level (e.g., less saturated color or grayscale, less brightness, more blurred, etc.). For example, at a first immersion level, the electronic device optionally passes through 80% of the colors, brightness, etc. of the representation(s) of the physical environment (e.g., representations of physical objects) through the display generation components, while at a second, higher immersion level, the electronic device optionally passes through 50% of the colors, brightness, etc. of the representation(s) of the physical environment (e.g., representations of physical objects) through the display generation components.In some embodiments, lower and higher immersion levels optionally cause less or more de-highlighting, respectively, of the representation(s) of the physical environment passed through the display generation component. The above-described method of varying the highlighting of the pass-through content reduces interference between the user interface(s) displayed by the electronic device and the physical environment as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by passing through the physical environment at high immersion levels), which further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0155] In some embodiments, at the second immersion level, a first amount of the display area of ​​the display generating component is occupied by a separate user interface displayed by the electronic device (1054), and at the third immersion level, a second area different from the first area of ​​the display area of ​​the display generating component is occupied by the separate user interface displayed by the electronic device (1056), such as the area of ​​user interface 912 shown in FIGS. 9A-9B. In some embodiments, as described with reference to methods 1200, 1400, and 1600, at no immersion, the content / user interface(s) displayed by the electronic device do not occupy a portion of the display space of the display generating component, and optionally, the entirety of the display space of the display generating component is occupied by a representation(s) of the physical environment of the electronic device (e.g., objects in the physical environment, surfaces in the physical environment, etc.). In some embodiments, the representation(s) of the physical environment are passed through the display generating component, as described above. In some embodiments, in full immersion, the content / user interface(s) displayed by the electronic device occupy the entirety of the display space of the display generation component, and optionally, none of the display space of the display generation component is occupied by the representation(s) of the electronic device's physical environment (e.g., objects in the physical environment, surfaces in the physical environment, etc.) as described with reference to methods 1200, 1400, and 1600. In some embodiments, as the immersion level increases from no immersion / low immersion to full immersion / high immersion, the display space of the display generation component becomes increasingly occupied by the content / user interface(s) displayed by the electronic device and the display space of the display generation component becomes increasingly unoccupied by the representation(s) of the electronic device's physical environment, or vice versa.The above-described methods of varying the amount of display space occupied by content displayed by the electronic device, and the representation(s) of the electronic device's physical environment, reduce interference of the physical environment with the user interface(s) displayed by the electronic device as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by passing through the physical environment at high immersion levels), which further reduces the electronic device's power usage and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0156] In some embodiments, changing the second separate user interface from being displayed at the second immersive level to being displayed at the third immersive level includes modifying (1058) the atmospheric effect displayed via the display generation component, as shown in the transition from FIG. 9A to FIG. 9B on objects 908 and 910. For example, in some embodiments, the electronic device displays a representation(s) of the physical environment of the electronic device (e.g., objects in the physical environment, surfaces in the physical environment, etc.) as described with reference to methods 1200, 1400, and 1600. In some embodiments, the representation(s) of the physical environment are passed through the display generation component as previously described. In some embodiments, the electronic device modifies the display of these representation(s) of the device's physical environment to display these representations with one or more visual effects corresponding to the atmospheric effect generated by the electronic device. For example, in some embodiments, the atmospheric effect is a mimicking effect of twilight lights / sounds / etc. during the day. In some embodiments, at low / no immersion, the representation(s) of the physical environment are displayed via the display generation components as if they were displayed in daylight (e.g., when this is the case on the electronic device, such as being outside at noon on a sunny day). In some embodiments, as the immersion level increases, the electronic device modifies the display of the representation(s) of the physical environment to reduce the brightness of sunlight displayed via the display generation components, including reducing the brightness at which various objects, potentially including the sun, or physical objects on which the sunlight falls, are displayed via the display generation components, to mimic the look / feel of the physical environment if the current time was dusk instead of noon. In some embodiments, altering such atmosphere effects includes adding a different virtual light source to the three-dimensional environment (e.g., the moon, or the sun, which emits light as soon as it is set, or shortly thereafter) and displaying the representation(s) of the physical environment with the appearance they would have if illuminated by the new light source.In some embodiments, at maximum immersion, the midday lighting / atmosphere / look, etc. of the representation(s) of the physical environment is completely replaced by the twilight light / atmosphere / look, etc. of the representation(s) of the physical environment. The above-described method of varying the atmospheric effects at which the physical environment is displayed increases the level to which the user is drawn into the virtual experience as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by the physical environment at high immersion levels), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0157] In some embodiments, the second individual user interface includes individual user interface elements displayed by the electronic device, and changing the second individual user interface from being displayed at the second immersion level to being displayed at the third immersion level includes changing the display of the individual user interface elements (1060), such as changing the display of objects 906, 908, and / or 910 in the transition from FIG. 9A to FIG. 9B. As previously described and further described with reference to methods 1200, 1400, and 1600, in some embodiments, the electronic device displays one or more virtual objects via a display generation component. In some embodiments, the virtual object(s) is, for example, a user interface / content of an application. In some embodiments, the virtual object is a virtual object displayed in some relationship to a representation of a physical object in the physical environment of the electronic device (e.g., a virtual vase displayed on a representation of a physical table). In some embodiments, the virtual object is a virtual object displayed without relationship to a representation of a physical object (e.g., a virtual drawing drawn by a user in a spec). In some embodiments, changing the immersion level optionally causes the electronic device, via the display generation component, to change the display of such virtual objects. For example, in some embodiments, increasing the immersion level optionally causes the virtual object(s) to be enlarged (e.g., displayed in a larger display space via the display generation component), to be displayed with more emphasis (e.g., different color, different shading, different (e.g., more) lighting, higher brightness, higher definition, lower translucency, etc.), to be displayed closer to the user's viewpoint, etc.In some embodiments, decreasing the immersion level optionally causes the virtual object(s) to shrink (e.g., be displayed in a smaller display space via the display generation component), to be displayed less highlighted (e.g., different color, different shade, different (e.g., less) lighting, lower brightness, lower sharpness, higher translucency, etc.), to be displayed further away from the user's viewpoint, etc. The above-described methods of varying the display of the virtual objects increase the user's engagement with the virtual objects as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by other objects, such as physical objects, at high immersion levels), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0158] In some embodiments, changing the second distinct user interface from being displayed at the second immersion level to being displayed at the third immersion level includes modifying audio effects generated by the electronic device (1062). In some embodiments, the electronic device generates audio for playing through one or more speakers or audio output devices (e.g., headphones) in communication with the electronic device while displaying the content / user interface(s) (e.g., ambient sound effects, sounds corresponding to (e.g., generated to emanate from) virtual objects (e.g., content of an application), sounds corresponding to (e.g., generated to emanate from) physical objects in the physical environment of the electronic device, etc.). In some embodiments, changing the immersion level optionally modifies the audio generated by the electronic device. For example, in some embodiments, increasing the immersion level optionally causes sounds corresponding to physical objects to be de-highlighted (e.g., reducing volume, clarity, expansion, etc.) and / or sounds corresponding to virtual objects to be highlighted (e.g., increasing volume, clarity, expansion, etc.). In some embodiments, increasing the immersion level optionally highlights (e.g., increases volume, clarity, extensiveness) atmospheric sound effects (e.g., sounds corresponding to the atmospheric effects described above) and / or de-highlights (e.g., reduces volume, clarity, extensiveness, etc.) sounds corresponding to physical objects. In some embodiments, at high (e.g., maximum) immersion, sounds corresponding to physical objects are not generated or reduced to a low level (e.g., minimum level) and sounds corresponding to virtual objects are increased to a high level (e.g., maximum level), and in some embodiments, at low (e.g., minimum) immersion, sounds corresponding to physical objects are increased to a high level (e.g., maximum level) and sounds corresponding to virtual objects are not generated or reduced to a low level (e.g., minimum level).The above-described method of varying audio effects based on immersion reduces interference between the user interface(s) displayed by the electronic device and the physical environment as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by audio corresponding to the physical environment at higher immersion levels), which further reduces power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0159] In some embodiments, displaying the second separate user interface at the second immersion level includes simultaneously displaying content of the first application in a first portion of the display area of ​​the display generation component and one or more representations of a first separate portion of a physical environment of the electronic device in a second portion of the display area of ​​the display generation component (1064), as described with reference to user interface 934 of FIG. 9C and shown with respect to user interface 912 of FIGS. 9A-9B (e.g., in some embodiments, the electronic device is displaying the content / user interface(s) in addition to displaying a representation(s) of the physical environment of the electronic device (e.g., objects in the physical environment, surfaces in the physical environment, etc.), as described with reference to methods 1200, 1400, and 1600). In some embodiments, the representation(s) of the physical environment are passed through the display generation component, as previously described. In some embodiments, at the lower immersion level, the content / user interface(s) occupy a smaller portion of the field of view of the display generation component (e.g., not in the field of view or a small portion of the field of view) and the representation(s) of the physical environment occupy a larger portion of the field of view of the display generation component (e.g., all of the field of view or a majority of the field of view). In some embodiments, displaying the second separate user interface at the third immersion level includes simultaneously displaying (1066) content of the first application in a third portion of the display area of ​​the display generation component that is larger than the first portion and one or more representations of a second separate portion of the physical environment of the electronic device that is different from the first separate portion in a fourth portion of the display area of ​​the display generation component that is smaller than the second portion of the display generation component, as described with reference to user interface 934 of FIG. 9C and shown with respect to user interface 912 of FIGS. 9A-9B.In some embodiments, as the immersion level increases, the content / user interface(s) increasingly occupy more of the field of view of the display generation component (e.g., all or a majority of the field of view) and the representation(s) of the physical environment increasingly do not occupy the field of view of the display generation component (e.g., not in the field of view or only a small portion of the field of view). In some embodiments, the visual characteristics (e.g., brightness, sharpness, color, etc.) at which the representation(s) of the physical environment are displayed optionally do not change as the immersion level changes, but rather the amount of display space occupied by those representation(s) of the physical environment changes. Thus, in some embodiments, changing the immersion level changes the amount of the field of view of the display generation component that is occupied by the content / user interface(s) displayed by the electronic device. The above-described methods of varying the amount of display space / field of view occupied by content displayed by the electronic device, and the representation(s) of the electronic device's physical environment, reduce interference between the user interface(s) displayed by the electronic device and the physical environment as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by pass-through of the physical environment at higher immersion levels), which further reduces the electronic device's power usage and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0160] In some embodiments, as described with reference to methods 1200, 1400, and 1600, displaying the second separate user interface at the second immersion level as shown in FIGS. 9A-9C includes simultaneously displaying (1068) content of the first application (e.g., playing a video, displaying photos, etc.) and one or more representations of the physical environment (e.g., in some embodiments, the electronic device is displaying the content / user interface(s) in addition to displaying a representation(s) of the electronic device's physical environment (e.g., objects in the physical environment, surfaces in the physical environment, etc.). In some embodiments, the representation(s) of the physical environment are passed through a display generation component as described above). In some embodiments, displaying the second separate user interface at the third immersion level includes simultaneously displaying (1070) the content of the first application and one or more representations of the physical environment unhighlighted relative to the one or more representations of the physical environment displayed at the second immersion level, as described with reference to user interface 934 of FIG. 9C and shown with respect to user interface 912 of FIG. 9A-9B. In some embodiments, at the lower immersion level, the representation(s) of the physical environment are displayed at the first highlighting level (e.g., color, brightness, unblurred, etc.) via the display generation components while the electronic device is displaying the content / user interface(s) via the display generation components. In some embodiments, as the immersion level increases, while the electronic device displays the content / user interface(s) via the display generation component, the representation(s) of the physical environment are displayed via the display generation component at a second highlight level (e.g., less saturated colors or grayscale, less brightness, more dim, etc.) that is lower than the first highlight level.For example, at a first immersion level, the electronic device optionally passes through 80% of the colors, brightness, etc. of the representation(s) of the physical environment (e.g., representations of physical objects) through the display generation components, while at a second, higher immersion level, the electronic device optionally passes through 50% of the colors, brightness, etc. of the representation(s) of the physical environment (e.g., representations of physical objects) through the display generation components. In some embodiments, lower and higher immersion levels optionally cause less or more de-highlighting, respectively, of the representation(s) of the physical environment passed through the display generation components. In some embodiments, increasing the immersion level increases highlighting of the content of the first application, and decreasing the immersion level decreases highlighting of the content of the first application. In some embodiments, increasing or decreasing the immersion level causes the content of the first application to occupy a larger or smaller portion, respectively, of the display space of the display generation component (and thus, optionally, causes the representation(s) of the physical environment to occupy a larger or smaller portion, respectively, of the display space of the display generation component). In some embodiments, increasing or decreasing the immersion level causes the content of the first application to occupy a larger or smaller portion, respectively, of the display space of the display generation component (and thus, optionally, causes the representation(s) of the physical environment to occupy a smaller or larger portion, respectively, of the display space of the display generation component).The above-described method of varying the highlighting of pass-through content reduces interference between the user interface(s) displayed by the electronic device and the physical environment as the immersion level increases, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding confusion caused by pass-through the physical environment at high immersion levels), which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0161] In some embodiments, after displaying the second individual user interface at the third immersion level and displaying the first individual user interface at the first immersion level, without receiving any input that changes the immersion level of the first individual user interface from the first immersion level, and without receiving any input that changes the immersion level of the second individual user interface from the third immersion level (1072) (e.g., the last immersion level at which the second individual user interface was displayed was the third immersion level and the last immersion level at which the first individual user interface was displayed was the first immersion level, which After displaying the first and second user interfaces at these immersion levels, no input has been received to change these immersion levels, the electronic device receives (1074) a fourth user input via one or more input devices corresponding to a request to display a third individual user interface of a second individual application different from the individual application (e.g., the input is received while displaying the second individual user interface at the third immersion level, while displaying the first individual user interface at the first immersion level, or while displaying a different user interface at a different immersion level). In some embodiments, the fourth user input includes detecting a selection of an application icon for the second individual application within the application browsing user interface. In some embodiments, the second user input is a voice input to display the second individual application without selecting an icon for the second individual application.

[0162] In some embodiments, in response to receiving the fourth user input (1076), the electronic device, via the display generation component, displays a third separate user interface at a fourth immersion level (e.g., in some embodiments, the third separate user interface is displayed overlaid on the user interface displayed when the fourth user input was received, both of which optionally continue to be displayed on the background and / or within the three-dimensional environment). In some embodiments, displaying the third separate user interface includes ceasing to display the user interface displayed when the fourth user input was received. In some embodiments, the fourth immersion level is independent of the immersion levels of the operating system user interface and / or the individual application's user interfaces. Thus, in some embodiments, the immersion levels of the operating system user interface and / or the application do not affect the immersion levels of the other application user interfaces (e.g., each application has its own independent immersion level). In some embodiments, the fourth immersion level at which the third individual user interface is displayed is the immersion level at which the third individual user interface was last displayed (prior to the current display of the third individual user interface). In some embodiments, the immersion level of the operating system user interface affects the immersion level of the application user interface (e.g., if the system user interface is set to the first immersion level, subsequently displaying the application user interface includes displaying the application user interface at that same first immersion level, even if the application user interface was last displayed at a different immersion level).

[0163] In some embodiments, while displaying the third individual user interface at the fourth immersion level, the electronic device receives (1078) a fifth user input, via one or more input devices, corresponding to a request to modify a current immersion level associated with the electronic device, such as an input at input element 920 (e.g., a voice input, a hand gesture, a selection of a displayed user interface element, and / or detecting an input (e.g., a rotational input) at a mechanical input element included in the electronic device corresponding to a request to increase or decrease the immersion level of the currently displayed user interface). In some embodiments, in response to receiving the fifth user input, the electronic device displays (1080), via the display generation component, the third individual user interface at a fifth immersion level different from the fourth immersion level (e.g., increasing or decreasing the immersion level of the third individual user interface in accordance with the fifth user input). In some embodiments, after receiving the fifth user input (1082), the electronic device receives a sixth user input via one or more input devices corresponding to a request to display the first individual user interface (e.g., a request to redisplay the first individual user interface after receiving an input that changes the immersion level of the third individual user interface) (1084). In some embodiments, in response to receiving the sixth user input, the electronic device displays the first individual user interface via the display generation component at a first immersion level (1086) (e.g., displaying the first individual user interface at the immersion level at which the first individual user interface was last displayed), such as the immersion level shown in FIG. 9A. In some embodiments, the electronic device receives a seventh user input via one or more input devices corresponding to a request to display the second individual user interface (e.g., a request to redisplay the second individual user interface after receiving an input that changes the immersion level of the third individual user interface) (1088).In some embodiments, in response to receiving the seventh user input, the electronic device, via the display generation component, displays (1090) the second individual user interface at a third immersion level, such as the immersion level shown in FIG. 9C (e.g., displaying the second individual user interface at the immersion level at which the second individual user interface was last displayed). Thus, in some embodiments, changing the immersion level of the user interface of the first application does not affect the immersion level of other applications and / or the user interface of the operating system of the electronic device. The above-described method of setting immersion independently for different applications provides a flexible architecture that may independently define and / or change immersion levels for different user interfaces, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user device interface more efficient (e.g., by avoiding erroneous immersion level changes in another user interface as a result of a change in immersion level in a current user interface), which further reduces power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0164] In some embodiments, while displaying the first individual user interface at the first immersion level after receiving the third user input, the electronic device receives (1092) via one or more input devices a fourth user input corresponding to a request to display a second individual user interface (e.g., an input to redisplay the individual application after ceasing to display the individual application to display an operating system user interface, such as an application browsing user interface). In some embodiments, the fourth user input is received while displaying the application browsing user interface (e.g., a selection of an icon corresponding to the individual application). In some embodiments, in response to receiving the fourth user input, the electronic device displays (1094) via the display generation component the second individual user interface at a third immersion level, such as the immersion level shown in FIG. 9C (e.g., the immersion level at which the second individual user interface was last displayed). For example, in some embodiments, redisplaying the individual application after exiting the operating system causes the individual application to be displayed at the immersion level at which the individual application was last displayed, regardless of any changes to the immersion level of the operating system user interface. The above-described method of maintaining immersion for a previously terminated application provides a flexible architecture in which immersion levels may be defined and / or maintained independently for different user interfaces, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding erroneous changes to the immersion level in another user interface as a result of a change in the immersion level in a current user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0165] In some embodiments, the second user input is a user input of the first type corresponding to a request to increase a current immersion level associated with the electronic device (e.g., a clockwise rotation of a rotatable input element of the electronic device while displaying a user interface of an application (e.g., rather than a user interface of an operating system of the electronic device) to increase the immersion level of the currently displayed user interface), and the third immersion level is higher than the second immersion level (1096). In some embodiments, while displaying the second distinct user interface at the third immersion level, the electronic device receives a fourth user input of the first type corresponding to a request to increase a current immersion level associated with the electronic device, such as an input at input element 920 via one or more input devices (e.g., a further clockwise rotation of a rotatable input element of the electronic device while displaying the user interface of the distinct application) to increase the immersion level of the currently displayed user interface (1098). In some embodiments, in response to receiving a fourth user input of the first type, the electronic device maintains (1097) the display of the second separate user interface at a third immersion level, as described with reference to user interface 934 of FIG. 9C (e.g., the third immersion level is, optionally, the highest immersion level that can be reached through rotation of the rotatable input element while displaying the application's user interface). In some embodiments, at the third immersion level, the separate application's user interface does not occupy the entirety of the field of view of the display generation component and / or continues to be displayed along with at least some representation(s) of the electronic device's physical environment. In some embodiments, a higher immersion level than the third immersion level is achievable, but not achievable using the first type of input.In some embodiments, a higher level of immersion can be achieved using the first type of input (e.g., rotation of a rotatable input element), but only to the user interface of the operating system of the electronic device and not to the user interface of the application.

[0166] In some embodiments, while displaying the second individual user interface at the third immersion level, the electronic device receives (1095), via one or more input devices, a fifth user input of a second type different from the first type, corresponding to a request to increase a current immersion level associated with the electronic device, such as a selection of element 950 of FIG. 9C (e.g., a selection of a user interface element displayed by the individual application in the second individual user interface to increase the immersion level of the individual application; e.g., a selection of a “full screen” user interface element in the second individual user interface). In some embodiments, in response to receiving the fifth user input of the second type, the electronic device displays (1093), via the display generation component, the second individual user interface at a fourth immersion level higher than the third immersion level (e.g., a fully immersive or full screen display of the individual application), as described with reference to user interface 934 of FIG. 9C. Thus, in some embodiments, full immersion, or at least a level of immersion above the third level of immersion, is not achievable via the first type of input when displaying the application's user interface (e.g., as opposed to the operating system's user interface). In some embodiments, a different type of input (e.g., selection of a user interface element) is required to exceed the third level of immersion in the application user interface.The above-described method of requiring different types of input to achieve a high immersion level for an application ensures that a high immersion level for an application cannot be achieved without additional indication that the user wishes to achieve such a high immersion level, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by avoiding accidentally reaching a high immersion level for applications whose behavior at high immersion levels may be unknown or unpredictable), which further reduces power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0167] 11A-11B illustrate an example of reducing the immersion level at which a user interface is displayed by an electronic device and then resuming display of the user interface from that immersion level in a particular scenario according to some embodiments.

[0168] FIG. 11A illustrates an electronic device 101 displaying a three-dimensional environment 1101 in a user interface via a display generating component (e.g., display generating component 120 of FIG. 1). As described above with reference to FIGS. 1-6, the electronic device 101 optionally includes a display generating component (e.g., a touch screen) and a number of image sensors (e.g., image sensor 314 of FIG. 3). The image sensors optionally include one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that the electronic device 101 can use to capture one or more images of the user or a portion of the user while the user is interacting with the electronic device 101. In some embodiments, the user interface illustrated below may also be implemented on a head-mounted display that includes a display generating component that displays the user interface to the user and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0169] 11A , device 101 captures one or more images of physical environment 1100 (e.g., operating environment 100) surrounding device 101, including one or more objects within physical environment 1100 surrounding device 101. In some embodiments, device 101 displays a representation of the physical environment in three-dimensional environment 1101. For example, three-dimensional environment 1101 includes a representation of a tree 1102b (e.g., corresponding to tree 1102a in physical environment 1100) and a representation of a person 1104b (e.g., corresponding to person 1104a in physical environment 1100). In some embodiments, device 101 also displays one or more virtual objects (e.g., objects not within physical environment 1100) within three-dimensional environment 1101. 1A , the device 101 displays a virtual object 1110 (e.g., a decoration) on the representation of a tree 1102b, a virtual object 1106 (e.g., a hat) on the representation of the person 1104b, and a representation of a sun 1108. In some embodiments, the representation of the sun 1108 is displayed and treated as a light source within the three-dimensional environment 1101, and lighting effects resulting from the representation of the sun 1108 are applied by the device 101 to representations of physical and / or virtual objects displayed in the three-dimensional environment 1101.

[0170] 11A, device 101 also displays a user interface 1134 of an application (e.g., app C). User interface 1134 is, optionally, any user interface displayed by device 101 (e.g., an operating system user interface, a user interface of a content (e.g., music, movies, videos, photos, etc.) browsing and / or viewing browsing application, etc.). In some embodiments, user interface 1134 is displayed at a position within three-dimensional environment 1101 such that it overlays one or more representations of physical and / or virtual objects within three-dimensional environment 1101 (e.g., in front of and overlaying a portion of representation 1104b of person corresponding to person 1104a in physical environment 1100). In some embodiments, user interface 1134 is located within three-dimensional environment 1101 such that one or more virtual objects are in front of and overlay a portion of user interface 1112 and / or one or more representations of physical objects are in front of and overlay a portion of user interface 1112.

[0171] In some embodiments, device 101 displays three-dimensional environment 1101 and / or user interface displayed via display generation component 120 at a particular immersion level. For example, in FIG. 11A , device 101 displays user interface 1134 at a relatively high immersion level 1132 on immersion scale 1130. For example, the display of FIG. 11A is, optionally, a result of device 101 detecting an input, using input element 1120 of device 101, that increases the immersion level of user interface 1134 to immersion level 1132, as described with reference to FIG. 9A . Thus, in some embodiments, content surrounding user interface 1134 displayed by display generation component 120 (e.g., a representation of physical environment 1100 and / or objects within physical environment 1100, virtual objects, etc.) is obscured to a relatively high degree (e.g., as described in more detail with reference to method 1000). In some embodiments, details regarding displaying the user interface 1134 at the relatively high immersion level 1132 are as described with reference to the method 1000 .

[0172] 9A , the input element 1120 is optionally a mechanical input element, such as a slider element or a rotary input element, that can be manipulated to change the immersion level of the user interface(s) currently being displayed by the device 101, with the amount and direction of the change in immersion (e.g., increase or decrease) being based on the magnitude and direction of manipulation of the input element 1120 (e.g., the direction and / or magnitude of the movement of the input element 1120, or the direction and / or magnitude of the rotation of a rotatable input element). In some embodiments, the device 101 and / or the input element 1120 generate haptic feedback as the immersion level changes (e.g., a minor tactile output for each increase or decrease in an immersion stage, and a different major tactile output upon reaching a minimum or maximum immersion). In some embodiments, the input element 1120 is also pressable and / or pressure-sensitive (e.g., in addition to being movable and / or rotatable) such that providing a pressure input to the input element (e.g., a button click, pressure above a pressure threshold, etc.) is detectable as an input by the device 101.

[0173] In some embodiments, in response to detecting a press of the input element 1120, the device 101 reduces the immersion level at which the currently displayed user interface(s) was displayed to a predetermined immersion level (e.g., a minimum or relatively low immersion level), and in response to detecting a release of the press of the input element 1120, the device 101 resumes displaying the user interface(s) at the immersion level that was displayed when the press input was detected. In some embodiments, the device 101 requires that the press of the input element 1120 be longer than a time threshold (e.g., 0.1 seconds, 0.5 seconds, 1 second, 5 seconds, 10 seconds) before transitioning to a lower immersion level. For example, in FIG. 11A, the device 101 detects a press of the input element 1120 ("Input 1"). In response, the device reduces the immersion level 1132 to a predetermined relatively low immersion level (e.g., no immersion), as shown in FIG. 11B. 11B, the device 101 has stopped displaying all virtual objects in the three-dimensional environment 1101 (e.g., including the user interface 1134 and virtual objects 1110, 1108, and 1106) and has stopped obscuring (e.g., stopping dimming, blurring, or otherwise de-highlighting) the representations 1102b and 1104b of objects in the physical environment 1100 and / or portions of the physical environment 1100 itself. Additional or alternative details regarding the display characteristics of the virtual objects and / or the physical environment 1100 when transitioning to a lower or no immersion level are, optionally, as described with reference to the method 1000. Thus, in some embodiments, pressing the input element 1120 causes the device 101 to stop displaying the entire physical environment 1100 in the three-dimensional environment 1101 and / or to stop displaying any virtual objects in the three-dimensional environment 1101 entirely. In this manner, a user of the device 101 can quickly and easily transition between views of the physical environment 1100 in an unambiguous manner.

[0174] As mentioned above, in some embodiments, in response to detecting the release of the input element 1120, the device 101 optionally returns to a previous immersion level at which it was displaying the user interface(s) when the press of the input element 1120 was detected. For example, in FIG. 11B, the device 101 detects the release of the input element 1120 ("Input 2"). In response, the device returns to the immersion level 1132 shown in FIG. 11A. In some embodiments, in addition to or alternatively to returning to a previous immersion level in response to detecting the release of the input element 1120, the device 101 returns to a previous immersion level in response to detecting a movement of the input element 1120 corresponding to a request to increase the current immersion level (e.g., the same type of input for gradually changing the immersion level described with reference to FIG. 9A). In some embodiments, in response to such an input, the device 101 immediately resumes displaying the user interface 1134 at the previous immersion level (e.g., even if the amount of movement of the input element 1120 would not normally be sufficient to increase the immersion level from the level of FIG. 11B to the previous immersion level of FIG. 11A), or in some embodiments, the device 101 gradually increases the immersion level (e.g., as the movement of the input element 1120 continues) until the movement of the input element 1120 sufficiently reaches the previous immersion level of FIG. 11A. Other inputs for resuming display of the user interface 1134 at the previous immersion level are also contemplated, such as a particular hand gesture detected by the device 101, a particular gaze-based gesture detected by (e.g., the sensor 314 of) the device 101, etc.

[0175] 12A-12D are flow charts illustrating a method 1200 for resuming display of a user interface at a previously displayed immersion level after (e.g., temporarily) reducing the immersion level associated with the user interface, according to some embodiments. In some embodiments, the method 1200 is performed on a computer system (e.g., computer system 101 of FIG. 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward on a user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras), or a camera pointing forward from the user's head). In some embodiments, the method 1200 is governed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 1200 are optionally combined, and / or the order of some operations is optionally changed.

[0176] In method 1200, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer) detects (1202) a first input, via the one or more input devices, corresponding to a request to increase an immersion level associated with the electronic device, such as an input on element 920 described with reference to FIG. 9A , while displaying a first user interface at a first immersion level. In some embodiments, the electronic device is a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the electronic device, an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users, or the like. In some embodiments, the electronic device communicates with one or more input devices, including electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.) and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, a motion sensor (e.g., hand tracking device, hand motion sensor), a physical mechanical input element included in the electronic device (e.g., a button, a rotating mechanical element, a switch, etc.), and / or an eye tracking device, etc.

[0177] In some embodiments, the first immersion level of the first user interface optionally has one or more of the characteristics of the immersion levels described with reference to methods 1000, 1400 and 1600. The first input is optionally a user input detected via one or more input devices, such as a voice input, a hand gesture, a selection of a displayed user interface element, and / or detecting an input (e.g., a rotational input) on a mechanical input element included in the electronic device corresponding to a request to increase or decrease the immersion level of the currently displayed user interface. In some embodiments, the first user interface is displayed within a three-dimensional environment displayed by the electronic device via a display generation component, such as a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment. In some embodiments, in response to detecting the first input, the electronic device, via the display generation component, displays (1204) the first user interface at a discrete immersion level higher than the first immersion level, such as displaying user interface 1134 at immersion level 1132 in FIG. 11A (e.g., increasing the immersion level of the first user interface in accordance with the first input).

[0178] In some embodiments, while displaying the first user interface at the respective immersion level, the electronic device detects (1206) the occurrence of a first event, such as detecting an input 1 on the input element 1120 of FIG. 11A via one or more input devices (e.g., detecting an input corresponding to a request to decrease the respective predefined immersion level of the electronic device (e.g., a minimum immersion level of the electronic device, such as no immersion or minimal immersion) that is lower than the first immersion level). For example, the input is a press (rather than a rotation) of a mechanical input element that is optionally rotated to increase (or decrease) the immersion level of the first user interface. In some embodiments, the first event is detecting a characteristic associated with a physical environment of the electronic device, as described with reference to methods 1400 and / or 1600. In some embodiments, the event is the detection of any amount and / or input to reduce the immersion level at the electronic device (e.g., of a currently displayed user interface(s)) such that one or more portions of the physical environment that were previously not visible through the display generating component (e.g., because the display generating component was displaying content that blocked or otherwise obscured those portions of the physical environment) become visible through the display generating component (e.g., because the display generating component is no longer displaying content that blocks or otherwise obscures those portions of the physical environment). Thus, the first event optionally corresponds to an event that allows a physical object / environment to pass through and be displayed via the display generating component.

[0179] In some embodiments, in response to detecting the occurrence of the first event, the electronic device reduces (1208) the immersion level of the user interface displayed via the display generating component to a second immersion level lower than the individual immersion level, such as immersion level 1132 shown in FIG. 11B (e.g., in some embodiments, the second immersion level is non-immersive (e.g., none of the physical environment is obscured from passing through the display generating component). In some embodiments, the second immersion level is simply lower than the individual immersion level, such that the physical environment is at least completely unobscured from passing through the display generating component. In some embodiments, when the user interface displayed by the display generating component is displayed at the second immersion level, the first user interface is stopped from being displayed. In some embodiments, when the user interface displayed by the display generation component is displayed at the second immersion level, the representation(s) of the physical environment of the electronic device is displayed via the display generation component (e.g., when the representation of the physical environment of the electronic device was not displayed when the first user interface was displayed at the first immersion level and / or the individual immersion level). In some embodiments, the first user interface continues to be displayed via the display generation component, but at a lower immersion level.

[0180] In some embodiments, while displaying the user interface at the second immersion level, the electronic device detects (1210) the occurrence of a second event, via one or more input devices, corresponding to a request to resume displaying the first user interface, such as detection of input 2 on input element 1120 of FIG. 11B (e.g., the second event is not indicative of a particular immersion level of the electronic device). For example, if the first event is detection of pressing a mechanical input element, the second event is, optionally, detection of release of the mechanical input element. If the first event is detection of a characteristic associated with the physical environment of the electronic device, the second event is, optionally, detection that the characteristic is no longer valid in the physical environment of the electronic device (e.g., a sound corresponding to the first event has stopped). In some embodiments, the second event is not an input corresponding to a request to increase the immersion level of the electronic device to a particular immersion level. For example, the second event is, optionally, not detection of a rotation of a previously detected mechanical input element to increase (or decrease) the immersion level of the electronic device according to the rotation of the mechanical input element.

[0181] In some embodiments, in response to detecting the occurrence of the second event, the electronic device resumes displaying the first user interface via the display generation component (1212), such as returning to displaying the user interface 1134 of FIG. 11A. In some embodiments, in accordance with a determination that the individual immersion level is a third immersion level, the first user interface is displayed at the third immersion level (1214), and in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level, the first user interface is displayed at the fourth immersion level (1216). Thus, in some embodiments, the electronic device returns to displaying the first user interface at the immersion level at which the first user interface was displayed when the first event was detected, without detecting an input indicative of such immersion level. In some embodiments, the electronic device remembers / records the immersion level that was in effect when the first event was detected, and resumes at that immersion level when the second event is detected. The above-described method of resuming the display of the first user interface at the immersion level that was in effect when the first event was detected provides a quick and efficient way of returning to the previously active immersion level, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, makes the user-device interface more efficient (e.g., by not requiring user input to define a particular immersion level to return to, which also avoids erroneous user input defining an incorrect immersion level to return to), which further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0182] In some embodiments, the second immersion level is no immersion (1218), as shown in FIG. 11B. For example, detecting the occurrence of the first event transitions the electronic device to a lowest or minimum immersion level where most or all of the physical environment of the electronic device is viewable via the display generating component and little or no virtual elements (e.g., a user interface generated by the electronic device, virtual objects generated by the electronic device, etc.) are displayed via the display generating component. The above-described method of displaying the physical environment of the electronic device without including virtual elements or objects provides a fast and efficient way of providing a user with environmental awareness, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient, which further reduces power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors during use.

[0183] In some embodiments, the one or more input devices include (1220) a rotatable input element (e.g., a rotatable input element included on / in the electronic device or on another device in communication with the electronic device; in some embodiments, the rotatable input element is rotatable via user input), as described with reference to FIGS. 11A-11B. In some embodiments, while displaying the first user interface at the first immersion level, the electronic device detects (1222) a respective rotation of the rotatable input element (e.g., a clockwise or counterclockwise rotation of the rotatable input element by a user). In some embodiments, in response to detecting (1224) the individual rotations of the rotatable input elements, in accordance with a determination that the individual rotations are in a first direction (e.g., clockwise rotation), the electronic device displays (1226), via the display generation component, the first user interface at a fifth immersion level higher than the first immersion level, as described with reference to FIGS. 11A-11B (e.g., rotating the rotatable input elements in the first direction increases the immersion level of the user interface(s) displayed via the display generation component. The increasing immersion level optionally has one or more of the characteristics described with reference to method 1000). In some embodiments, in accordance with a determination that the respective orientation is a second orientation different from the first orientation (e.g., counterclockwise rotation), the electronic device displays (1228), via the display generation component, the first user interface at a sixth immersion level lower than the first immersion level, as described with reference to FIGS. 11A-11B (e.g., rotating the rotatable input element in a second orientation decreases the immersion level of the user interface(s) displayed via the display generation component; the decreasing immersion level optionally has one or more of the characteristics described with reference to method 1000).The above-described method of adjusting the immersion level of an electronic device provides a quick and efficient way of doing so, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient, which in turn reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0184] In some embodiments, detecting the occurrence of the first event includes detecting (1230) a press of a rotatable input element, such as input 1 on input element 1120 of FIG. 11A. For example, in some embodiments, the rotatable input element is rotatable to provide rotation-based input to the electronic device and is pressable to provide press-and / or release-based input to the electronic device. In some embodiments, detecting the press of the rotatable input element is, optionally, to cause the electronic device to display a user interface(s) at a minimum or minimal immersion level. The above-described method of responding to a press of the rotatable input element provides a fast and efficient way of providing a low immersion experience, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user device interface more efficient, which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0185] In some embodiments, detecting the occurrence of the second event includes detecting (1232) a rotation of the rotatable input element, as described with reference to FIGS. 11A-11B. In some embodiments, detecting any amount of rotation of the rotatable input element causes the electronic device to transition back to the individual immersion level after transitioning to the second immersion level, even if the amount of rotation is different from the amount of rotation normally required to increase the immersion level from the second immersion level to the individual immersion level. In some embodiments, the amount of rotation normally required to increase the immersion level from the second immersion level to the individual immersion level is, optionally, an amount of rotation required when the electronic device transitions to the second immersion level not through detection of the first event, but through detection of a different type of input, such as a rotation of the rotatable input element, to gradually decrease the immersion level to the second immersion level. For example, if the second immersion level is a level of 0 on a scale of 0 to 10, the individual immersion level is a level of 7, and the detected amount of rotation is an amount that would normally increase the immersion level by 3 (e.g., from 1 to 4, or from 2 to 5, etc.), then when the electronic device transitions to the second immersion level via detection of the first event, the detected amount of rotation optionally resumes the first user interface at immersion level 7 (e.g., the immersion level does not gradually increase from the second immersion level in accordance with the amount of rotation of the rotatable input element). However, if the electronic device transitions to the second immersion level not via detection of the first event (e.g., rather, via detecting rotation of the rotatable element and decreasing the immersion level to the second immersion level), the detected amount of rotation optionally displays the first user interface at immersion level 3 rather than immersion level 7. After an initial amount of rotation that causes the first user interface to be displayed at the respective immersion level, further rotation (clockwise or counterclockwise) to increase or decrease the immersion level optionally causes an increase or decrease in immersion from the respective immersion level.The above-described method of resuming a previous immersion level of a device provides a quick and efficient way of returning to a previously active immersion level while using the same input method normally used to change the immersion level of a device, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device and makes the user-device interface more efficient, which further reduces power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0186] In some embodiments, while displaying the user interface at the second immersion level, the electronic device detects (1234) a second individual rotation of the rotatable input element including a first portion followed by a second portion of the second individual rotation (e.g., detecting a first portion of a clockwise rotation of the rotatable input element followed consecutively by a second portion of the clockwise rotation of the rotatable input element). In some embodiments, in response to detecting the first portion of the second individual rotation (e.g., while detecting the second individual rotation of the rotatable input element), the electronic device displays (1236), via the display generation component, the first user interface at a seventh immersion level that is higher than the second immersion level but lower than the individual immersion level (e.g., gradually increasing the immersion level from the second immersion level according to the amount of rotation of the rotatable input element during the first portion of the second individual rotation). In some embodiments, in response to detecting a second portion of the second individual rotation (e.g., while detecting the second individual rotation of the rotatable input element), the electronic device increases the immersion level of the first user interface from the seventh immersion level to the individual immersion level (1238). For example, during the second portion of the second individual rotation, the immersion level continues to increase according to the successive rotation amounts of the rotatable input element. If the second individual rotation includes a total amount of rotation sufficient to increase the immersion level from the second immersion level to the individual immersion level, the first user interface is optionally displayed at the individual immersion level. If the total amount of rotation of the second individual rotation corresponds to an immersion level other than the individual immersion level, the first user interface is optionally displayed at the other immersion level. The above-described method of gradually returning to a previous level of immersion on the device provides a consistent response to rotation of the rotatable input element, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient, which further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0187] In some embodiments, detecting the occurrence of the second event includes detecting (1240) a distinct gesture performed by a hand of a user of the electronic device via a hand tracking device in communication with the display generation component (e.g., a particular pinch gesture between two or more fingers of the user's hand, such as the user's thumb and index finger. In some embodiments, the gesture includes a particular movement and / or rotation of the hand while maintaining the hand pinch gesture. In some embodiments, if a different gesture other than the distinct gesture is detected to be performed by the user's hand, the electronic device optionally does not resume displaying the first user interface at the distinct immersion level. In some embodiments, the distinct hand gesture is different from the first event that transitioned the electronic device to the second immersion level). The above-described method of resuming a device's previous immersion level provides a quick and efficient way of returning to a previously effective level of immersion while avoiding an accidental return to such immersion, which simplifies the interaction between a user and the electronic device, improves usability of the electronic device and makes the user-device interface more efficient, which further reduces the power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0188] In some embodiments, detecting the occurrence of the first event includes detecting a press of a mechanical input element that communicates with the electronic device (1242), similar to input 1 on input element 1120 of FIG. 11A. In some embodiments, the mechanical input element is a button included in / on the electronic device or on another device that communicates with the electronic device. In some embodiments, the mechanical input element can only be pressed or released (e.g., not rotated). In some embodiments, the mechanical input element can be pressed or released in addition to being rotated (e.g., to change the immersiveness level of the user interface(s) displayed by the electronic device, as described with reference to method 1000). In some embodiments, the first event is a press of the mechanical input element, not including a subsequent release of the mechanical input element. In some embodiments, the first event includes a press and a subsequent release of the mechanical input element. In some embodiments, the mechanical input element is pressure sensitive, and detecting a press of the mechanical input element includes detecting a force or pressure on the input element greater than a threshold force or pressure (e.g., as opposed to detecting a press based on actual movement of a button), and detecting a release of the mechanical input element includes detecting a force or pressure less than the threshold force or pressure (or the absence of force or pressure). The above-described method of responding to a press of an input element provides a fast and efficient way of providing a low immersive experience, which simplifies the interaction between a user and an electronic device, improves usability of the electronic device, and makes the user-device interface more efficient, which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0189] In some embodiments, detecting the occurrence of the first event includes detecting (1244) a press of a mechanical input element in communication with the electronic device, similar to input 1 on input element 1120 of FIG. 11A, longer than a time threshold. In some embodiments, the mechanical input element is a button included in / on the electronic device or on another device in communication with the electronic device. In some embodiments, the mechanical input element can only be pressed or released (e.g., not rotated). In some embodiments, the mechanical input element can be pressed or released in addition to being rotated (e.g., to change the immersiveness level of the user interface(s) displayed by the electronic device, as described with reference to method 1000). In some embodiments, the first event is a press of the mechanical input element longer than a time threshold (e.g., 0.1 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds), not including a subsequent release of the mechanical input element. In some embodiments, the first event includes a press and a subsequent release of the mechanical input element. In some embodiments, a press and release of the mechanical input element before the mechanical input element is pressed for the time threshold does not correspond to a first event and does not transition the electronic device to the second immersion level. In some embodiments, the mechanical input element is pressure sensitive, and detecting the press of the mechanical input element includes detecting a force or pressure on the input element that is greater than a threshold force or pressure (e.g., as opposed to detecting a press based on actual movement of a button), and detecting the release of the mechanical input element includes detecting a force or pressure (or the absence of force or pressure) that is less than the threshold force or pressure. The above-described method of responding to a press of an input element provides a fast and efficient way of providing a low immersion experience while avoiding accidental transitions to a low immersion experience, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user-device interface more efficient, which further reduces power usage of the electronic device and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0190] In some embodiments, detecting the occurrence of the second event includes detecting (1246) a release of a press of a mechanical input element in communication with the electronic device, such as input 2 detected in input element 1120 of FIG. 11B. In some embodiments, the mechanical input element is a button included in / on the electronic device or on another device in communication with the electronic device. In some embodiments, the mechanical input element can only be pressed or released (e.g., not rotated). In some embodiments, the mechanical input element can be pressed or released in addition to being rotated (e.g., to change an immersion level of a user interface(s) displayed by the electronic device, as described with reference to method 1000). In some embodiments, the electronic device transitions the first user interface at the first immersion level to the second immersion level in response to detecting a press of the mechanical input element, and transitions from the second immersion level back to the first user interface at the first immersion level in response to detecting a subsequent release of the mechanical input element. In some embodiments, the mechanical input element is pressure sensitive, and detecting a press of the mechanical input element includes detecting a force or pressure on the input element that is greater than a threshold force or pressure (as opposed to, for example, detecting a press based on the actual movement of a button), and detecting a release of the mechanical input element includes detecting a force or pressure (or the absence of force or pressure) that is less than the threshold force or pressure. Using mechanical input elements that rotate and are pressable to perform corresponding functions reduces the number of controls on the electronic device and reduces the complexity of manufacturing the device. Furthermore, the above-described method of responding to the release of an input element provides a quick and efficient way to return to a previously effective level of immersion, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, and makes the user-device interface more efficient, which further reduces the power usage of the electronic device and improves battery life by allowing the user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0191] 13A-13C show examples of displaying representations of people, objects, and / or portions of an electronic device's physical environment through a user interface(s) displayed by the electronic device, according to some embodiments.

[0192] FIG. 13A illustrates electronic device 101 displaying user interface 1334 via a display generating component (e.g., display generating component 120 of FIG. 1). As described above with reference to FIGS. 1-6, electronic device 101 optionally includes a display generating component (e.g., a touch screen) and a number of image sensors (e.g., image sensor 314 of FIG. 3). The image sensor optionally includes one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that electronic device 101 can use to capture one or more images of the user or a portion of the user while the user is interacting with electronic device 101. In some embodiments, the user interface illustrated below may also be implemented on a head-mounted display that includes a display generating component that displays the user interface to the user and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0193] The device 101 optionally captures one or more images of the physical environment 1300 (e.g., operating environment 100) around the device 101, including one or more objects in the physical environment 1300 around the device 101. For example, in Fig. 13A, the physical environment 1300 includes two objects (e.g., a tree 1304a and a ball 1306a) and two people (e.g., a person 1302a and a person 1308a). The legend 1303 provides a top-down view of the physical environment 1300, where element 101a corresponds to the location of the device 101 in the physical environment, elements 1302c and 1308c correspond to the locations of the people 1302a and 1308a, respectively, in the physical environment 1300, and elements 1304c and 1306c correspond to the locations of the tree 1304a and the ball 1306a, respectively, in the physical environment 1300. As indicated by legend 1303, each of the people 1302a and 1308a and objects 1304a and 1306a in Figure 13A is farther from the device 101 than distance 1340. The relevance of distance 1340 is explained in more detail below.

[0194] In Figure 13A, device 101 is displaying a user interface 1334 (e.g., of application App C) at a relatively high (e.g., maximum) immersion level 1332 on immersion scale 1330. For example, the display of Figure 13A is, optionally, the result of device 101 detecting an input, using an input element of device 101 (e.g., input element 920 of device 101, as described with reference to Figure 9A), that increases the immersion level of user interface 1334 to immersion level 1332. In some embodiments, user interface 1334 is a user interface of an operating system of device 101, as opposed to a user interface of an application. 13A , at an immersion level 1332 where the device 101 is displaying the user interface 1334, the user interface 1334 obscures (e.g., completely obscures) the display of the physical environment 1300 of the device 101 (e.g., objects, people, etc. in the physical environment 1300) via the display generation component 120 such that the content of the user interface 1334 is visible via the display generation component 120, but the representation(s) of the physical environment 1300 is not visible via the display generation component 120. In some embodiments, details regarding displaying the user interface 1334 at a relatively high (e.g., maximum) level of immersion level 1332 are as described with reference to method 1000. Although the description of Figures 13A-13C is provided in the context of a user interface 1334 having a relatively high immersion level 1332, it will be understood that the features and methods 1400 of Figures 13A-13C optionally apply to any situation in which a virtual element, object, etc. displayed by device 101 at least partially obscures the display of any portion of the physical environment 1300 via display generation component 120.

[0195] In some embodiments, device 101 may detect one or more characteristics of objects, people, one or more portions of physical environment 1300, etc., and in response to detecting a particular characteristic, device 101 optionally enables representations of one or more of those objects, people, or one or more portions of physical environment 1300 to “break through” user interface 1334 so that they become visible (e.g., when they were not visible prior to detecting those characteristics) via display generation component 120. In some embodiments, representations of those objects, people, or one or more portions of physical environment 1300 break through portions of user interface 1334 that correspond to the locations of those objects, people, or one or more portions of physical environment 1300, and the remaining portions of user interface 1334 continue to be displayed via display generation component 120.

[0196] For example, in FIG. 13A, both person 1302a and person 1308a are farther than a threshold distance 1340 (e.g., 1, 3, 5, 10, 15, 30, 50, etc.) from device 101. At these distances, if device 101 determines that the person's attention is directed to the user of device 101, device 101 optionally causes the person to break through user interface 1334. For example, in FIG. 13A, device 101 has determined that user 1302a's attention is not directed to the user of device 101, and thus a representation of person 1302a is not visible through user interface 1334. However, device 101 has determined that user 1308a's attention is directed to the user of device 101. Thus, device 101 displays representation 1308b via display generation component 120 through a portion of user interface 1334 (e.g., a portion of display generation component 120 that corresponds to the location of person 1308a in the three-dimensional environment including device 101, as described in this disclosure). In this manner, the user of device 101 can see people in physical environment 1300 through user interface 1334 when those people are paying attention to the user of device 101, even when those people are far away from the user of device 101. In some embodiments, when device 101 breaks through user interface 1334 to a person, object, portion, etc. of physical environment 1300, device 101 stops displaying the portion of user interface 1334 (e.g., or displays the portion of user interface 1334 with semi-transparency) and displays a representation of the person, object, portion, etc. of physical environment 1300 in the portion of user interface 1334. In some embodiments, the representation of the person, object, portion, etc. of the physical environment 1300 is displayed with a variable blur effect, such that portions of the user interface 1334 surrounding the representation of the person, object, portion, etc. of the physical environment 1300 are displayed with different amounts of blur that decrease depending on the distance from the representation of the person, object, portion, etc. of the physical environment 1300.

[0197] The device 101 optionally determines that the person's attention is directed to a user of the device 101 using one or more sensors on the device 101 that capture images, sounds, and / or otherwise detected characteristics of the physical environment 1300. For example, in some embodiments, the person's attention is optionally determined to be directed to a user of the electronic device if the person is looking towards the electronic device and / or towards the user of the electronic device. In some embodiments, the person's attention is optionally determined to be directed to the user of the electronic device if the person is speaking towards the electronic device and / or to the user of the electronic device. In some embodiments, an electronic device associated with a person detects the person's relevant attention using sensors in communication with the electronic device, transmits such attention information to the user's electronic device, and then responds accordingly. In some embodiments, the determination of a person's attention is based on one or more factors described with reference to method 1400, including based on a gesture from the person (e.g., a gesture directed toward the user of device 101), speech from the person (e.g., speech directed toward the user of device 101, speaking the name of the user of device 101, etc.), the speed at which the person is moving toward the user of device 101, the person's gaze (e.g., gaze directed toward the user of device 101), etc.

[0198] In some embodiments, in addition to or as an alternative to using factors about a person's attention in the physical environment 1300 to cause the person to break through the user interface 1334, the device 101 utilizes one or more factors about the interaction of the user of the device 101 with the person in the physical environment 1300 to cause the person to break through the user interface 1334. For example, in some embodiments, if the user of the device 101 is engaging with the person (e.g., gesturing toward the person, speaking toward the person, moving toward the person, etc.) and the device 101 detects such engagement of the user with the person, the device 101 causes the person to break through the user interface 1334. In some embodiments, the device 101 uses a combination of factors including the person's attention and the user of the device 101's engagement with the person that the person breaks through the user interface 1334.

[0199] In some embodiments, if the device 101 determines that objects that are farther than the threshold distance 1340 from the device 101 are high-risk objects, the device 101 similarly causes those objects to break through the user interface 1334, thus allowing the user of the device 101 to view those objects via the display generation component 120 (e.g., to avoid potential danger). For example, the device 101 optionally determines that the objects are high risk based on one or more factors described with reference to the method 1400 (e.g., based on the detected trajectories of those objects), including whether those objects will collide with the user of the device 101. In FIG. 13A, the device 101 determines that the ball 1306a is on a trajectory to collide with the user of the device 101. As a result, the device 101 allows the ball 1306a to break through the user interface 1334 and displays the representation 1306b of the ball 1306a via the display generation component 120 in a manner and / or location on the display generation component 120 similar to that described with respect to displaying the representation 1308b of the person 1308a through the user interface 1334. The device 101 of FIG. 13A , conversely, has determined that the tree 1304a is not a high-risk object and therefore does not allow the tree 1304a to break through the user interface 1334.

[0200] 13A , despite allowing certain representations of people, objects, portions, etc. of physical environment 1300 to break through user interface 1334, in some embodiments device 101 continues to display the remainder of user interface 1334 at a relatively high immersion level 1332. Thus, in some embodiments, breakthrough of people, objects, portions, etc. of physical environment 1300 through user interface 1334 optionally does not reduce the immersion level 1332 at which device 101 is or normally displays user interface(s).

[0201] In some embodiments, based on the confidence of determining that (e.g., in the case of objects) the object is a high risk object, (e.g., in the case of people) the person's attention is directed to the user of device 101, (e.g., in the case of people) the user of device 101 is engaged with the person, etc., device 101 enables the person, object, portion, etc. of physical environment 1300 to break through user interface 1334 to various degrees. For example, in some embodiments, the fewer the breakthrough factors described herein with reference to method 1400, the lower the confidence level associated with the breakthrough that a given object or person meets, and the more the breakthrough factors described herein with reference to method 1400, the higher the confidence level associated with the breakthrough that a given object or person meets. In some embodiments, as confidence in a given breakthrough increases, device 101 displays a representation of that person, object, etc., through user interface 1334 with less translucency, more color saturation, more brightness, etc., and / or displays portions of user interface 1334 in which the representation of that person, object, etc. is displayed with more translucency, less color saturation, less brightness, etc. For example, in FIG. 13B , device 101 has determined with a relatively high degree of confidence that person 1308a's attention is directed at the user of device 101, and thus device 101 optionally displays representation 1308b of person 1308a through user interface 1334 with relatively less translucency, more color saturation, more brightness, etc., and / or displays portions of user interface 1334 in which representation 1308b is displayed with relatively more translucency, less color saturation, less brightness, etc.In contrast, in FIG. 13B , the device 101 optionally determines that the object 1306a is a high-risk object with a relatively low degree of confidence, and therefore the device 101 optionally displays a representation 1306b of the object 1306a through the user interface 1334 with relatively high translucency, relatively low color saturation, relatively low brightness, etc., and / or displays a portion of the user interface 1334 in which the representation 1306b is displayed with relatively low translucency, relatively high color saturation, relatively high brightness, etc.

[0202] In some embodiments, the device 101 breaks through objects or people that are within the threshold distance 1340 of the device 101, regardless of whether those objects are high risk, whether those people's attention is directed to the user of the device 101, or whether the user of the device 101 is engaged with those people. For example, in FIG. 13C, the device 101 detects a tree 1304a within the threshold distance 1340 of the device 101. Although the tree 1304a is, optionally, not a high risk object (e.g., does not meet the high risk criteria described herein and / or with reference to the method 1400), because the tree 1304a is within the threshold distance 1340 of the device, the device 101 optionally displays a representation 1304b of the tree 1304a through the user interface 1334, as shown in FIG. 13C, the device 101 has determined that the ball 1306a is no longer or is not a high risk object, and therefore the device 101 does not display a representation of the ball 1306a through the user interface 1334 because the ball 1306a is farther than a threshold distance 1340 from the device 101. Finally, in contrast to FIG. 13A, in FIG. 13C, the device 101 has determined that the attention of the person 1302a is directed to the user of the device 101 and that the attention of the person 1308a is not directed to the user of the device 101, and therefore the device 101 displays a representation 1302b of the person 1302a through the user interface 1334 but does not display a representation of the person 1308a through the user interface 1334 because the people 1302a and 1308a are farther than the threshold distance 1340 from the device 101.

[0203] 14A-14F are flow charts illustrating a method 1400 for making objects, people, and / or parts of an environment visible through a user interface displayed by an electronic device, according to some embodiments. In some embodiments, the method 1400 is performed on a computer system (e.g., computer system 101 of FIG. 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera pointing downward on a user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras), or a camera pointing forward from a user's head). In some embodiments, the method 1400 is governed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 1400 are optionally combined, and / or the order of some operations is optionally changed.

[0204] In method 1400, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component and one or more input devices (e.g., a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer) displays (1402) a user interface including user interface elements, such as user interface 1334 of FIG. 13A . In some embodiments, the electronic device is a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer. In some embodiments, the display generation component is a display (optionally a touch screen display) integrated with the electronic device, an external display such as a monitor, projector, television, or a hardware component (optionally integrated or external) for projecting the user interface or making the user interface visible to one or more users, or the like. In some embodiments, the electronic device is in communication with one or more input devices, including electronic devices or components capable of receiving user input (e.g., capturing user input, detecting user input, etc.), and transmitting information related to the user input to the electronic device. Examples of input devices include a touch screen, a mouse (e.g., external), a track pad (optionally integrated or external), a touch pad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, a motion sensor (e.g., a hand tracking device, a hand motion sensor), a physical mechanical input element included in the electronic device (e.g., a button, a rotating mechanical element, a switch, etc.), and / or an eye tracking device, etc.

[0205] In some embodiments, the electronic device displays a three-dimensional environment, such as a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment, that is generated, displayed, or otherwise made viewable by the electronic device. In some embodiments, the user interface elements are virtual elements or objects (e.g., a user interface of an application on the electronic device, a virtual object displayed in the three-dimensional environment, etc.). In some embodiments, the electronic device is not displaying a three-dimensional environment, but is simply displaying content (e.g., a user interface having user interface elements) via a display generation component.

[0206] In some embodiments, the user interface elements, via the display generation component, at least partially obscure (1404) the display of the individual persons in the physical environment of the electronic device, such as obscuring the display of the people 1302a and 1308a in FIG. 13A (e.g., the display generation component optionally passes through the physical environment of the electronic device to an extent that the virtual content displayed by the electronic device is not displayed overlaid on portions of that physical environment). In some embodiments, the user interface elements displayed by the electronic device are displayed "in front" of or closer to the viewpoint of the electronic device than the individual persons in the physical environment of the electronic device. Thus, the user interface elements, optionally, via the display generation component, obscure / block the display of the representations of the individual persons (e.g., completely obscure the display of the representations of the individual persons (e.g., the user interface elements are completely opaque and / or completely cover the representations of the individual persons), or partially obscure the display of the representations of the individual persons (e.g., the user interface elements are semi-transparent and / or do not completely cover the representations of the individual persons)). In some embodiments, the user interface and / or its content occludes the entirety of the physical environment such that no portion of the physical environment is visible through the display generation component. In some embodiments, the user interface and / or its content occludes less than the entirety of the physical environment such that portions of the physical environment are visible through the display generation component while other portions (e.g., individual people) are not visible through the display generation component. In some embodiments, the electronic device displays the user interface at the individual immersion levels as described with reference to methods 1000, 1200, and 1600.

[0207] In some embodiments, while displaying (1406) the user interface, pursuant to a determination that an individual person, such as people 1302a and 1308a in FIG. 13A, is farther than a threshold distance from the electronic device in the physical environment (e.g., the person is farther than 1 foot, 3 feet, 5 feet, 10 feet, 15 feet, 50 feet, 100 feet, etc. from the electronic device. In some embodiments, if viewable via the display generation component, the individual person is displayed within the three-dimensional environment displayed by the electronic device as farther than 1 foot, 3 feet, 5 feet, 10 feet, 15 feet, 50 feet, 100 feet, etc. from the viewpoint of the electronic device in the three-dimensional environment), and pursuant to a determination that the attention of the individual person is directed to a user of the electronic device, such as person 1308a in FIG. 13A (e.g., the electronic device optionally includes one or more cameras, microphones, or other sensors that can detect an attention state of the individual person in the physical environment of the electronic device. In some embodiments, the individual person is directed toward the electronic device, and The attention of the individual person is directed to the user of the electronic device if the individual person is looking at the electronic device and / or looking towards the user of the electronic device. In some embodiments, the attention of the individual person is directed to the user of the electronic device if the individual person is speaking towards the electronic device and / or towards the user of the electronic device. In some embodiments, one electronic device (e.g., other than the electronic devices mentioned above) associated with the individual person detects the individual person's relevant attention using a sensor in communication with the electronic device, transmits such attention information to the electronic device, which then responds accordingly. ), the electronic device updates (1408) the display of the user interface element to indicate the presence of the individual person in the physical environment at a location corresponding to the user interface element, such as displaying the representation 1308b of FIG. 13A (e.g., ceasing to display the user interface element such that the user interface element is no longer "in front" of the individual person via the display generation component, and thus a representation of the individual person in the physical environment of the electronic device is visible via the display generation component).In some embodiments, the electronic device increases the translucency of the user interface element so that the individual person becomes visible (or more visible) through the user interface element. In some embodiments, the electronic device moves the user interface element in the user interface so that the user interface element is no longer "in front of" the individual person in the physical environment of the electronic device. In some embodiments, the electronic device reduces the degree to which the user interface element obscures the view of the individual person via a display generation component. In some embodiments, the electronic device visually indicates the presence and / or attention of the individual person. In some embodiments, the electronic device blurs or otherwise changes the appearance of the user interface element. In some embodiments, the electronic device updates the entirety of the displayed user interface (e.g., everything displayed in the user interface) as described above. In some embodiments, the electronic device updates only the portion of the user interface that obscures the individual person (e.g., objects displayed in that portion of the user interface) and does not modify the display of the remainder of the user interface.

[0208] In some embodiments, pursuant to a determination that the individual person is farther than a threshold distance from the electronic device in the physical environment and the individual person's attention is not directed to a user of the electronic device (e.g., the individual person is not looking at the electronic device and / or is not speaking at the electronic device and / or is not speaking at the user of the electronic device), the electronic device refrains from updating (1410) the display of user interface elements, such as not displaying a representation of the person 1302a in the user interface 1334 of FIG. 13A. For example, not modifying the display of the user interface and / or user interface elements. Thus, the user interface elements optionally continue to obscure the display of the individual person via the display generation component to the same extent as before. The above-described method of selectively altering the display of a user interface based on attention provides a fast and efficient way of allowing a user of an electronic device to view people in the electronic device's physical environment via a display generation component only when interaction with an individual person is likely, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not unnecessarily updating the display of the user interface, thus maintaining continuity of the displayed user interface and avoiding user input that would undo an erroneous update of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0209] In some embodiments, while displaying 1412 the user interface, pursuant to a determination that an individual person is closer than a threshold distance (e.g., 3, 5, 10, 15, etc.) from the electronic device in the physical environment, the electronic device updates 1414 the display of the user interface element to indicate the presence of the individual person in the physical environment at a location corresponding to the user interface element (e.g., whether or not the attention of the associated person is directed at the user of the electronic device). Thus, in some embodiments, the electronic device updates the display of the user interface element to indicate the presence of the individual person when the individual person is nearby (e.g., as described above), regardless of the individual person's attention state. The above-described method of modifying the display of a user interface when a person is near the electronic device provides a quick and efficient method of allowing a user of the electronic device to view nearby people in the electronic device's physical environment via a display generation component, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not requiring a separate user input to indicate the presence of an individual person in the physical environment), which further reduces the electronic device's power usage and improves battery life by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0210] In some embodiments, determining that the individual person's attention is directed to the user of the electronic device is based on one or more gestures performed by the individual person detected by one or more sensors (e.g., one or more optical or other cameras capturing the physical environment of the electronic device including the individual person) in communication with the electronic device (1416), as described with reference to Figures 13A-13C. For example, if the electronic device detects that the individual person is pointing, waving, or otherwise gesturing toward the user of the electronic device, e.g., with one or more hands of the individual person (e.g., in some embodiments, regardless of how long the individual person gestures toward the user, and in some embodiments, after gesturing toward the user for longer than a time threshold, such as 1 second, 3 seconds, 5 seconds, 10 seconds, 20 seconds, etc.), the electronic device optionally determines that the individual person's attention is directed to the user of the electronic device. In some embodiments, the electronic device determines that the individual person's attention is directed to the user of the electronic device based on the above-mentioned gesture in combination with one or more other factors described herein (e.g., the individual person's gaze, the individual person's speech, the individual person's speed and / or movement, etc.), and further determines that the individual person's attention is not directed to the user of the electronic device when the above-mentioned gesture is detected without detecting one or more of the other factors described herein.The above-described method of determining attention of an individual person based on gestures from the individual person provides a quick and efficient method of enabling a user of an electronic device to view people in the electronic device's physical environment via a display generation component when interaction with the individual person is likely to occur, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not unnecessarily updating the display of the user interface, thus maintaining continuity of the displayed user interface and avoiding user input that would undo an erroneous update of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0211] In some embodiments, determining that the individual person's attention is directed toward the user of the electronic device is based on the individual person's gaze detected by one or more sensors (e.g., one or more optical or other cameras capturing the physical environment of the electronic device, including the individual person) in communication with the electronic device (1418), as described with reference to Figures 13A-13C. For example, if the electronic device detects that the individual person is looking at or heading toward the user of the electronic device (e.g., after looking toward the user for more than a time threshold of 1 second, 3 seconds, 5 seconds, 10 seconds, 20 seconds, etc., in some embodiments, regardless of how long the individual person looks toward the user) (e.g., within a range of positions at or within a threshold distance of 0.5 feet, 1 foot, 3 feet, 5 feet, 10 feet, 20 feet, etc. surrounding the user of the electronic device), the electronic device optionally determines that the individual person's attention is directed toward the user of the electronic device. In some embodiments, the electronic device determines that the individual person's attention is directed to the user of the electronic device based on the above-mentioned gaze in conjunction with one or more other factors described herein (e.g., the individual person's gesture(s), the individual person's speech, the individual person's speed and / or movement, etc.), and further determines that the individual person's attention is not directed to the user of the electronic device when the above-mentioned gaze is detected without detecting one or more of the other factors described herein.The above-described method of determining attention of an individual person based on line of sight from the individual person provides a quick and efficient method of enabling a user of an electronic device to view people in the physical environment of the electronic device via a display generation component when interaction with the individual person is likely to occur, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not unnecessarily updating the display of the user interface, thus maintaining continuity of the displayed user interface and avoiding user input that would undo an erroneous update of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0212] In some embodiments, determining that the individual person's attention is directed to the user of the electronic device is based on detected speech from the individual person detected by one or more sensors (e.g., one or more microphones capturing sounds from the physical environment of the electronic device, including the individual person) in communication with the electronic device (1420), as described with reference to Figures 13A-13C. For example, if the electronic device detects that the individual person is speaking a greeting (e.g., "Hello everyone") or otherwise speaking to the user of the electronic device (e.g., after speaking to the user for more than a time threshold, in some embodiments 1 second, 3 seconds, 5 seconds, 10 seconds, 20 seconds, etc., regardless of how long the individual person speaks to the user), the electronic device optionally determines that the individual person's attention is directed to the user of the electronic device. In some embodiments, the electronic device determines that the individual person's attention is directed to the user of the electronic device based on the above-mentioned utterance in combination with one or more other factors described herein (e.g., the individual person's gesture(s), the individual person's gaze, the individual person's speed and / or movement, etc.), and determines that the individual person's attention is not directed to the user of the electronic device when the above-mentioned utterance is detected without detecting one or more of the other factors described herein.The above-described method of determining the attention of an individual person based on utterances from the individual person provides a quick and efficient method of enabling a user of an electronic device to view people in the physical environment of the electronic device via a display generation component when an interaction with the individual person is likely to occur, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not unnecessarily updating the display of the user interface, thus maintaining continuity of the displayed user interface and avoiding user input that would undo an erroneous update of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0213] In some embodiments, determining that the individual person's attention is directed to the user of the electronic device is based on determining that the electronic device detects that speech from the individual person includes the name of the user of the electronic device (1422), as described with reference to Figures 13A-13C. For example, if the individual person speaks the user's name, in some embodiments in any context (e.g., in a non-greeting statement such as "I saw Tom (user) at the store today"), and in some embodiments only in certain contexts (e.g., in a greeting statement such as "Hello Tom (user)"), the electronic device optionally determines that the individual person's attention is directed to the user of the electronic device. In some embodiments, the electronic device determines that the individual person's attention is directed to the user of the electronic device based on the above-mentioned utterance of the user's name in conjunction with one or more other factors described herein (e.g., the individual person's gesture(s), the individual person's gaze, the individual person's speed and / or movement, etc.), and further determines that the individual person's attention is not directed to the user of the electronic device when the above-mentioned utterance of the user's name is detected without detecting one or more of the other factors described herein. The above-described method of determining attention of an individual person based on the utterance of a user's name from the individual person provides a quick and efficient method of enabling a user of an electronic device to view people in the physical environment of the electronic device via a display generation component when interaction with the individual person is likely to occur, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user-device interface more efficient (e.g., by not unnecessarily updating the display of the user interface, thus maintaining continuity of the displayed user interface and avoiding user input that would undo an erroneous update of the user interface), which further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently, while reducing errors in use.

[0214] In some embodiments, determining that the individual person's attention is directed to the user of the electronic device is based on the movement (e.g., speed) of the individual person (e.g., toward the user of the electronic device) detected by one or more sensors (e.g., one or more optical or other cameras capturing the physical environment of the electronic device, including the individual person) in communication with the electronic device (1424), as described with reference to Figures 13A-13C. For example, if the electronic device detects that the individual person is moving toward the user of the electronic device (e.g., the individual person's trajectory based on the individual person's current movement brings the individual person within a distance of less than a threshold distance of 0.5 feet, 1 foot, 3 feet, 5 feet, 10 feet, 20 feet, etc., from the user of the electronic device), the electronic device optionally determines that the individual person's attention is directed to the user of the electronic device. In some embodiments, the electronic device optionally requires the speed of the movement of the individual person to meet certain criteria (e.g., a speed above a threshold, a speed below a threshold, such as 0.3 m / s, 0.5 m / s, 1 m / s, 3 m / s, 5 m / s, 10 m / s, 20 m / s, etc.) in addition to the individual person moving towards the user of the electronic device, at which point the electronic device optionally determines that the attention of the individual person is directed to the user of the electronic device (and, optionally, does not so determine, otherwise). In some embodiments, the electronic device determines that the attention of the individual person is directed to the user of the electronic device based on said speed and / or movement of the individual person in conjunction with one or more other factors described herein (e.g., the gesture(s) of the individual person, the gaze of the individual person, the speech of the individual person, etc.), and further determines that the attention of the individual person is not directed to the user of the electronic device when said speed and / or movement of the individual person is detected without detecting one or more of the other factors described herein.The above-described method of determining attention of an individual person based on the speed and / or movement of the individual person provides a fast and efficient method of enabling a...

Claims

1. 1. A method comprising: An electronic device in communication with the display generation component and the eye tracking device, via said display generation component, A first user interface associated with the first application; and a second user interface surrounding at least a portion of the first user interface; and detecting, via the eye tracking device, a user's gaze directed toward the first user interface while simultaneously displaying the first user interface and the portion of the second user interface that surrounds at least the portion of the first user interface; in response to detecting that the gaze of the user is directed toward the first user interface; de-highlighting the second user interface relative to the first user interface in accordance with a determination that one or more criteria are met; A method comprising:

2. The one or more criteria include a criterion that is met when the first application is an application of a first type and that is not met when the first application is an application of a second type different from the first type, and the method further comprises: in response to detecting that the gaze of the user is directed toward the first user interface; refraining from de-highlighting the second user interface relative to the first user interface in accordance with a determination that the one or more criteria are not met. The method of claim 1 further comprising:

3. after detecting that the gaze of the user is directed toward the first user interface, and while displaying the second user interface as unhighlighted relative to the first user interface, detecting via the eye tracking device that the gaze of the user is directed toward the second user interface; in response to detecting the user's gaze being directed toward the second user interface, highlighting the second user interface relative to the first user interface in accordance with a determination that one or more second criteria are met; The method of claim 1 or 2, further comprising:

4. 4. The method according to any one of claims 1 to 3, prior to detecting that the gaze of the user is directed toward the first user interface, the first user interface and the second user interface are displayed simultaneously with a representation of at least a portion of a physical environment of the electronic device; 4. The method of claim 1, wherein dehighlighting the second user interface relative to the first user interface comprises dehighlighting the representation of the portion of the physical environment of the electronic device relative to the first user interface.

5. prior to detecting that the gaze of the user is directed toward the first user interface, the first user interface and the second user interface are displayed simultaneously with a virtual object that is not within a physical environment of the electronic device; dehighlighting the second user interface relative to the first user interface dehighlighting the virtual object relative to the first user interface; The method of any one of claims 1 to 4, comprising:

6. prior to detecting that the gaze of the user is directed toward the first user interface, the first user interface and the second user interface are simultaneously displayed with a representation of at least a portion of a physical environment of the electronic device and a virtual object that is not within the physical environment of the electronic device; De-highlighting the second user interface relative to the first user interface. dehighlighting the representation of the portion of the physical environment of the electronic device for the first user interface; de-highlighting the virtual object with respect to the first user interface; The method of any one of claims 1 to 5, comprising:

7. The method of claim 1 , wherein the second user interface includes a virtual object that is not within a physical environment of the electronic device.

8. the first user interface is displayed at a first immersion level prior to detecting that the gaze of the user is directed toward the first user interface; dehighlighting the second user interface relative to the first user interface maintains a display of the first user interface at the first immersion level; The method of any one of claims 1 to 7, comprising:

9. The method of claim 1 , wherein the first user interface is displayed as an overlay on the second user interface.

10. one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: Through the display generation component, A first user interface associated with the first application; and a second user interface surrounding at least a portion of the first user interface; simultaneously, detecting, via an eye tracking device, a user's gaze directed toward the first user interface while simultaneously displaying the first user interface and the portion of the second user interface that surrounds at least the portion of the first user interface; in response to detecting that the gaze of the user is directed toward the first user interface; 11. The electronic device, comprising: instructions for de-highlighting the second user interface relative to the first user interface in accordance with a determination that one or more criteria are met.

11. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: Through the display generation component, A first user interface associated with the first application; and a second user interface surrounding at least a portion of the first user interface; and detecting, via an eye tracking device, that a user's gaze is directed toward the first user interface while simultaneously displaying the first user interface and the portion of the second user interface that surrounds at least the portion of the first user interface; in response to detecting that the gaze of the user is directed toward the first user interface; de-highlighting the second user interface relative to the first user interface in accordance with a determination that one or more criteria are met; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

12. 1. An electronic device comprising: one or more processors; Memory, Through the display generation component, A first user interface associated with the first application; and a second user interface surrounding at least a portion of the first user interface; and a means for simultaneously displaying means for detecting, via an eye tracking device, that a user's gaze is directed toward the first user interface while simultaneously displaying the first user interface and the portion of the second user interface that surrounds at least the portion of the first user interface; in response to detecting that the gaze of the user is directed toward the first user interface; means for de-highlighting the second user interface relative to the first user interface in accordance with a determination that one or more criteria are met; An electronic device comprising:

13. 1. An information processing apparatus for use in an electronic device, comprising: Through the display generation component, A first user interface associated with the first application; and a second user interface surrounding at least a portion of the first user interface; and a means for simultaneously displaying means for detecting, via an eye tracking device, that a user's gaze is directed toward the first user interface while simultaneously displaying the first user interface and the portion of the second user interface that surrounds at least the portion of the first user interface; in response to detecting that the gaze of the user is directed toward the first user interface; means for de-highlighting the second user interface relative to the first user interface in accordance with a determination that one or more criteria are met; An information processing device comprising:

14. 1. An electronic device comprising: one or more processors; Memory, One or more programs; An electronic device comprising:

10. An electronic device, comprising: one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform the method of any one of claims 1 to 9.

15. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 1 to 9.

16. one or more processors; Memory, Means for carrying out the method according to any one of claims 1 to 9; An electronic device comprising:

17. 1. An information processing apparatus for use in an electronic device, comprising: An information processing device comprising means for carrying out the method according to any one of claims 1 to 9.

18. 1. A method comprising: An electronic device in communication with a display generation component and one or more input devices, comprising: displaying, via the display generation component, a first separate user interface of an operating system of the electronic device, the first separate user interface being displayed at a first immersion level; receiving, while displaying the first distinct user interface at the first immersion level, a first user input via the one or more input devices corresponding to a request to display a second distinct user interface of a distinct application; displaying, via the display generation component, in response to receiving the first user input, a second individual user interface of the individual application, wherein the second individual user interface of the individual application is displayed at a second immersion level; receiving, while displaying the second distinct user interface at the second immersion level, a second user input via the one or more input devices corresponding to a request to change a current immersion level associated with the electronic device; displaying, via the display generation component, the second distinct user interface at a third immersion level different from the second immersion level in response to receiving the second user input; receiving, while displaying the second distinct user interface at the third immersion level, a third user input via the one or more input devices corresponding to a request to display the first distinct user interface; displaying, via the display generation component, the first distinct user interface in response to receiving the third user input, the first distinct user interface being displayed at the first immersion level; A method comprising:

19. 20. The method of claim 18, wherein the one or more input devices include a rotatable input element, and receiving the second user input corresponding to the request to change the current immersion level associated with the electronic device includes detecting a rotation of the rotatable input element.

20. While detecting the rotation of the rotatable element, generating, via the tactile output generator, a first tactile output having a distinct characteristic having a first value in accordance with a determination that a current immersion level of the currently displayed user interface is a minimum immersion level; generating, via the tactile output generator, a second tactile output having the distinct characteristic having the first value in accordance with a determination that the current immersion level of the currently displayed user interface is a maximum immersion level; generating, via the tactile output generator, a third tactile output having the distinct characteristic having a second value different from the first value in accordance with a determination that the current immersion level of the currently displayed user interface is an intermediate immersion level; 20. The method of claim 19, further comprising:

21. the second user input includes a first portion and a second portion, and the method further comprises: While receiving the second user input, displaying, via the display generation component, the second distinct user interface at a fourth immersion level that is between the second immersion level and the third immersion level in response to receiving the first portion of the second user input; displaying, via the display generation component, the second distinct user interface at the third immersion level in response to receiving the second portion of the second user input; 21. The method of any one of claims 18 to 20, further comprising:

22. receiving the second user input corresponding to the request to change the current immersion level associated with the electronic device; Detecting an interaction with a first discrete control displayed with the second discrete user interface in accordance with a determination that the second discrete user interface is a user interface of a first application; responsive to a determination that the second distinct user interface is a user interface of a second application distinct from the first application, detecting an interaction with a second distinct control displayed with the second user interface and distinct from the first distinct control; 22. The method of any one of claims 18 to 21, comprising:

23. at the second immersion level, a first amount of the physical environment of the electronic device is replaced with a distinct user interface displayed by the electronic device via the display generation component; at the third immersion level, a second amount, different from the first amount, of the physical environment of the electronic device is replaced with the individual user interface displayed by the electronic device via the display generation component.

23. The method according to any one of claims 18 to 22.

24. at the second immersion level, a first amount of a physical environment of the electronic device is de-highlighted by the electronic device via the display generation component; At the third immersion level, a second amount of the physical environment of the electronic device that is different from the first amount is de-highlighted by the electronic device via the display generation component.

24. The method according to any one of claims 18 to 23.

25. at the second immersion level, a first amount of a display area of ​​the display generation component is occupied by a distinct user interface displayed by the electronic device; and at the third immersion level, a second area of ​​the display area of ​​the display generation component that is different from the first area is occupied by the separate user interface displayed by the electronic device.

25. The method of any one of claims 18 to 24.

26. 26. The method of claim 18, wherein changing the second respective user interface from being displayed at the second immersive level to being displayed at the third immersive level comprises changing an atmospheric effect displayed via the display generation component.

27. 27. The method of claim 18, wherein the second individual user interface includes individual user interface elements displayed by the electronic device, and changing the second individual user interface from being displayed at the second immersive level to being displayed at the third immersive level includes changing a display of the individual user interface elements.

28. 28. The method of claim 18, wherein changing the second distinct user interface from being displayed at the second immersive level to being displayed at the third immersive level comprises changing an audio effect generated by the electronic device.

29. Displaying the second discrete user interface at the second immersive level includes simultaneously displaying content of a first application in a first portion of a display area of ​​the display generation component and one or more representations of a first discrete portion of a physical environment of the electronic device in a second portion of the display area of ​​the display generation component; displaying the second discrete user interface at the third immersion level includes simultaneously displaying the content of the first application in a third portion of the display area of ​​the display generation component that is larger than the first portion and one or more representations of a second discrete portion of the physical environment of the electronic device that is different from the first discrete portion in a fourth portion of the display area of ​​the display generation component that is smaller than the second portion; 29. The method of any one of claims 18 to 28, comprising:

30. displaying the second distinct user interface at the second immersion level includes simultaneously displaying content of a first application and one or more representations of a physical environment; displaying the second separate user interface at the third immersion level comprises simultaneously displaying the content of the first application and the one or more representations of the physical environment that are de-highlighted relative to the one or more representations of the physical environment displayed at the second immersion level; 30. The method of any one of claims 18 to 29, comprising:

31. displaying the second individual user interface at the third immersion level and, after displaying the first individual user interface at the first immersion level, without receiving an input to change the immersion level of the first individual user interface from the first immersion level, and without receiving an input to change the immersion level of the second individual user interface from the third immersion level; receiving a fourth user input via the one or more input devices corresponding to a request to display a third distinct user interface of a second distinct application, the second distinct application being different from the distinct application; displaying, via the display generation component, the third distinct user interface at a fourth immersion level in response to receiving the fourth user input; receiving, while displaying the third distinct user interface at the fourth immersion level, a fifth user input via the one or more input devices corresponding to a request to change the current immersion level associated with the electronic device; displaying, via the display generation component, the third distinct user interface at a fifth immersion level different from the fourth immersion level in response to receiving the fifth user input; after receiving the fifth user input; receiving a sixth user input via the one or more input devices corresponding to a request to display the first distinct user interface; displaying, via the display generation component, the first distinct user interface at the first immersion level in response to receiving the sixth user input; receiving a seventh user input via the one or more input devices corresponding to a request to display the second separate user interface; displaying, via the display generation component, the second distinct user interface at the third immersion level in response to receiving the seventh user input; 31. The method of any one of claims 18 to 30, further comprising:

32. receiving, after receiving the third user input, a fourth user input via the one or more input devices, corresponding to a request to display the second distinct user interface while displaying the first distinct user interface at the first immersion level; displaying, via the display generation component, the second distinct user interface at the third immersion level in response to receiving the fourth user input; 32. The method of any one of claims 18 to 31, further comprising:

33. the second user input is a first type of user input corresponding to a request to increase the current immersion level associated with the electronic device, the third immersion level being greater than the second immersion level, and the method further comprising: receiving, while displaying the second distinct user interface at the third immersion level, a fourth user input of the first type via the one or more input devices corresponding to a request to increase the current immersion level associated with the electronic device; maintaining display of the second separate user interface at the third immersion level in response to receiving the fourth user input of the first type; while displaying the second distinct user interface at the third immersion level, receiving, via the one or more input devices, a fifth user input of a second type different from the first type, the fifth user input corresponding to a request to increase the current immersion level associated with the electronic device; displaying, via the display generation component, the second distinct user interface at a fourth immersion level greater than the third immersion level in response to receiving the fifth user input of the second type; 33. The method of any one of claims 18 to 32, further comprising:

34. one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: displaying, via a display generation component, a first separate user interface of an operating system of the electronic device, the first separate user interface being displayed at a first immersion level; while displaying the first distinct user interface at the first immersion level, receiving a first user input via one or more input devices corresponding to a request to display a second distinct user interface of a distinct application; displaying, via the display generation component, in response to receiving the first user input, a second individual user interface of the individual application, wherein the second individual user interface of the individual application is displayed at a second immersion level; receiving, while displaying the second distinct user interface at the second immersion level, a second user input via the one or more input devices corresponding to a request to change a current immersion level associated with the electronic device; displaying, via the display generation component, the second distinct user interface at a third immersion level different from the second immersion level in response to receiving the second user input; receiving a third user input via the one or more input devices corresponding to a request to display the first distinct user interface while displaying the second distinct user interface at the third immersion level; in response to receiving the third user input, display, via the display generation component, the first distinct user interface, wherein the first distinct user interface is displayed at the first immersion level.

35. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: displaying, via a display generation component, a first separate user interface of an operating system of the electronic device, the first separate user interface being displayed at a first immersion level; While displaying the first distinct user interface at the first immersion level, receiving a first user input via one or more input devices corresponding to a request to display a second distinct user interface of a distinct application; displaying, via the display generation component, in response to receiving the first user input, a second individual user interface of the individual application, wherein the second individual user interface of the individual application is displayed at a second immersion level; receiving, while displaying the second distinct user interface at the second immersion level, a second user input via the one or more input devices corresponding to a request to change a current immersion level associated with the electronic device; displaying, via the display generation component, the second distinct user interface at a third immersion level different from the second immersion level in response to receiving the second user input; receiving, while displaying the second distinct user interface at the third immersion level, a third user input via the one or more input devices corresponding to a request to display the first distinct user interface; displaying, via the display generation component, the first distinct user interface in response to receiving the third user input, the first distinct user interface being displayed at the first immersion level; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

36. 1. An electronic device comprising: one or more processors; Memory, means for displaying, via a display generation component, a first separate user interface of an operating system of the electronic device, the first separate user interface being displayed at a first immersion level; means for receiving, while displaying the first distinct user interface at the first immersion level, a first user input via one or more input devices corresponding to a request to display a second distinct user interface of a distinct application; means for displaying, via the display generation component, in response to receiving the first user input, a second individual user interface of the individual application, wherein the second individual user interface of the individual application is displayed at a second immersion level; means for receiving, while displaying the second distinct user interface at the second immersion level, a second user input via the one or more input devices corresponding to a request to change a current immersion level associated with the electronic device; means for displaying, via the display generation component, the second distinct user interface at a third immersion level different from the second immersion level in response to receiving the second user input; means for receiving, while displaying the second distinct user interface at the third immersion level, a third user input via the one or more input devices, the third user input corresponding to a request to display the first distinct user interface; means for displaying, via the display generation component, the first individual user interface in response to receiving the third user input, the first individual user interface being displayed at the first immersion level; 1. An electronic device comprising:

37. 1. An information processing apparatus for use in an electronic device, comprising: means for displaying, via a display generation component, a first separate user interface of an operating system of the electronic device, the first separate user interface being displayed at a first immersion level; means for receiving, while displaying the first distinct user interface at the first immersion level, a first user input via one or more input devices corresponding to a request to display a second distinct user interface of a distinct application; means for displaying, via the display generation component, in response to receiving the first user input, a second individual user interface of the individual application, wherein the second individual user interface of the individual application is displayed at a second immersion level; means for receiving, while displaying the second distinct user interface at the second immersion level, a second user input via the one or more input devices corresponding to a request to change a current immersion level associated with the electronic device; means for displaying, via the display generation component, the second distinct user interface at a third immersion level different from the second immersion level in response to receiving the second user input; means for receiving, while displaying the second distinct user interface at the third immersion level, a third user input via the one or more input devices, the third user input corresponding to a request to display the first distinct user interface; means for displaying, via the display generation component, the first individual user interface in response to receiving the third user input, the first individual user interface being displayed at the first immersion level; An information processing device comprising:

38. one or more processors; Memory, One or more programs; An electronic device comprising:

34. An electronic device, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform the method of any one of claims 18 to 33.

39. 34. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 18 to 33.

40. one or more processors; Memory, Means for carrying out the method according to any one of claims 18 to 33; An electronic device comprising:

41. 1. An information processing apparatus for use in an electronic device, comprising:

34. An information processing device comprising means for carrying out the method according to any one of claims 18 to 33.

42. An electronic device in communication with a display generation component and one or more input devices, comprising: detecting, while displaying a first user interface at a first immersion level, a first input via the one or more input devices corresponding to a request to increase an immersion level associated with the electronic device; displaying, via the display generation component, the first user interface at a respective immersion level that is higher than the first immersion level in response to detecting the first input; detecting, while displaying the first user interface at the individual immersion level, an occurrence of a first event via the one or more input devices; responsive to detecting the occurrence of the first event, reducing an immersion level of a user interface displayed via the display generation component to a second immersion level that is lower than the individual immersion level; while displaying the user interface at the second immersion level, detecting, via the one or more input devices, the occurrence of a second event corresponding to a request to resume displaying the first user interface; In response to detecting the occurrence of the second event, the first user interface: the first user interface is displayed at a third immersion level in accordance with a determination that the individual immersion level is a third immersion level; resume display of the first user interface via the display generation component, the first user interface being displayed at the fourth immersion level in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level; A method comprising:

43. 43. The method of claim 42, wherein the second immersion level is non-immersive.

44. the one or more input devices include a rotatable input element, the method comprising: detecting a respective rotation of the rotatable input element while displaying the first user interface at the first immersion level; in response to detecting the respective rotations of the rotatable input elements; displaying, via the display generation component, the first user interface at a fifth immersion level greater than the first immersion level in accordance with a determination that the respective rotation is in a first direction; displaying, via the display generation component, the first user interface at a sixth immersion level lower than the first immersion level in accordance with a determination that the respective orientation is a second orientation different from the first orientation; 44. The method of claim 42 or 43, further comprising:

45. 45. The method of claim 44, wherein detecting the occurrence of the first event comprises detecting a press of the rotatable input element.

46. 46. ​​The method of claim 44 or 45, wherein detecting the occurrence of the second event comprises detecting a rotation of the rotatable input element.

47. detecting a second discrete rotation of the rotatable input element including a first portion followed by a second portion of a second discrete rotation while displaying the user interface at the second immersion level; in response to detecting the first portion of the second individual rotation, displaying, via the display generation component, the first user interface at a seventh immersion level that is higher than the second immersion level but lower than the individual immersion level; increasing the immersion level of the first user interface from the seventh immersion level to the individual immersion level in response to detecting the second portion of the second individual rotation; 47. The method of any one of claims 44 to 46, further comprising:

48. 48. The method of any one of claims 42 to 47, wherein detecting the occurrence of the second event comprises detecting a discrete gesture performed by a hand of a user of the electronic device via a hand tracking device in communication with the display generation component.

49. 49. The method of any one of claims 42 to 48, wherein detecting the occurrence of the first event comprises detecting a press of a mechanical input element in communication with the electronic device.

50. 50. The method of claim 42, wherein detecting the occurrence of the first event comprises detecting a press of a mechanical input element in communication with the electronic device for longer than a time threshold.

51. 51. The method of any one of claims 42 to 50, wherein detecting the occurrence of the second event comprises detecting a release of a press of a mechanical input element in communication with the electronic device.

52. one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: While displaying a first user interface at a first immersion level, detecting a first input via one or more input devices corresponding to a request to increase an immersion level associated with the electronic device; displaying, via a display generation component, the first user interface at a respective immersion level that is higher than the first immersion level in response to detecting the first input; detecting, while displaying the first user interface at the individual immersion level, an occurrence of a first event via the one or more input devices; responsive to detecting the occurrence of the first event, reducing an immersion level of a user interface displayed via the display generation component to a second immersion level that is lower than the individual immersion level; while displaying the user interface at the second immersion level, detecting, via the one or more input devices, the occurrence of a second event corresponding to a request to resume displaying the first user interface; In response to detecting the occurrence of the second event, the first user interface: the first user interface is displayed at a third immersion level in accordance with a determination that the individual immersion level is a third immersion level; the first user interface is displayed at a fourth immersion level in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level, and instructions for resuming display of the first user interface via the display generation component.

53. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: detecting, while displaying a first user interface at a first immersion level, a first input via one or more input devices corresponding to a request to increase an immersion level associated with the electronic device; displaying, via a display generation component, the first user interface at a respective immersion level that is higher than the first immersion level in response to detecting the first input; detecting, while displaying the first user interface at the individual immersion level, an occurrence of a first event via the one or more input devices; responsive to detecting the occurrence of the first event, reducing an immersion level of a user interface displayed via the display generation component to a second immersion level that is lower than the individual immersion level; while displaying the user interface at the second immersion level, detecting, via the one or more input devices, the occurrence of a second event corresponding to a request to resume displaying the first user interface; In response to detecting the occurrence of the second event, the first user interface: the first user interface is displayed at a third immersion level in accordance with a determination that the individual immersion level is a third immersion level; resume display of the first user interface via the display generation component, the first user interface being displayed at the fourth immersion level in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

54. 1. An electronic device comprising: one or more processors; Memory, means for detecting, while displaying a first user interface at a first immersion level, a first input via one or more input devices corresponding to a request to increase an immersion level associated with the electronic device; means for displaying, via a display generation component, the first user interface at a respective immersion level that is higher than the first immersion level in response to detecting the first input; means for detecting, via the one or more input devices, the occurrence of a first event while displaying the first user interface at the individual immersion level; means for reducing an immersion level of a user interface displayed via the display generation component to a second immersion level that is lower than the individual immersion level in response to detecting the occurrence of the first event; means for detecting, while displaying the user interface at the second immersion level, the occurrence of a second event via the one or more input devices, the second event corresponding to a request to resume displaying the first user interface; In response to detecting the occurrence of the second event, the first user interface: the first user interface is displayed at a third immersion level in accordance with a determination that the individual immersion level is a third immersion level; means for resuming display of the first user interface via the display generation component, the first user interface being displayed at the fourth immersion level in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level; An electronic device comprising:

55. 1. An information processing apparatus for use in an electronic device, comprising: means for detecting, while displaying a first user interface at a first immersion level, a first input via one or more input devices corresponding to a request to increase an immersion level associated with the electronic device; means for displaying, via a display generation component, the first user interface at a respective immersion level that is higher than the first immersion level in response to detecting the first input; means for detecting, via the one or more input devices, the occurrence of a first event while displaying the first user interface at the individual immersion level; means for reducing an immersion level of a user interface displayed via the display generation component to a second immersion level that is lower than the individual immersion level in response to detecting the occurrence of the first event; means for detecting, while displaying the user interface at the second immersion level, the occurrence of a second event via the one or more input devices, the second event corresponding to a request to resume displaying the first user interface; In response to detecting the occurrence of the second event, the first user interface: the first user interface is displayed at a third immersion level in accordance with a determination that the individual immersion level is a third immersion level; means for resuming display of the first user interface via the display generation component, the first user interface being displayed at the fourth immersion level in accordance with a determination that the individual immersion level is a fourth immersion level different from the first immersion level; An information processing device comprising:

56. one or more processors; Memory, One or more programs; An electronic device comprising:

52. An electronic device, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform the method of any one of claims 42 to 51.

57. 52. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 42 to 51.

58. one or more processors; Memory, Means for carrying out the method according to any one of claims 42 to 51; An electronic device comprising:

59. 1. An information processing apparatus for use in an electronic device, comprising:

52. An information processing device comprising means for carrying out a method according to any one of claims 42 to 51.

60. An electronic device in communication with a display generation component and one or more input devices, comprising: displaying, via the display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a display of an individual person within a physical environment of the electronic device; While displaying the user interface, updating a display of the user interface element to indicate a presence of the individual person in the physical environment at a location corresponding to the user interface element in accordance with a determination that the individual person is greater than a threshold distance from the electronic device in the physical environment and that the individual person's attention is directed to a user of the electronic device; refraining from updating the display of the user interface elements in accordance with a determination that the individual is farther than the threshold distance from the electronic device in the physical environment and the attention of the individual is not directed to the user of the electronic device; A method comprising:

61. While displaying the user interface, updating a display of the user interface element to indicate the presence of the individual person in the physical environment at the location corresponding to the user interface element in accordance with a determination that the individual person is closer than the threshold distance from the electronic device in the physical environment; 61. The method of claim 60, further comprising:

62. 62. The method of claim 60 or 61, wherein determining that the attention of the individual person is directed to the user of the electronic device is based on one or more gestures performed by the individual person detected by one or more sensors in communication with the electronic device.

63. 63. The method of any one of claims 60 to 62, wherein determining that the attention of the individual person is directed to the user of the electronic device is based on the individual person's line of sight detected by one or more sensors in communication with the electronic device.

64. 64. The method of any one of claims 60 to 63, wherein the determination that the attention of the individual person is directed to the user of the electronic device is based on detected speech from the individual person detected by one or more sensors in communication with the electronic device.

65. 65. The method of claim 64, wherein determining that the attention of the individual is directed to the user of the electronic device is based on determining that the electronic device detects that the speech from the individual includes a name of the user of the electronic device.

66. 66. The method of any one of claims 60 to 65, wherein determining that the attention of the individual person is directed to the user of the electronic device is based on movement of the individual person detected by one or more sensors in communication with the electronic device.

67. Detecting that the user of the electronic device is engaged in an interaction with the individual person while displaying the user interface; in response to detecting that the user of the electronic device is engaged in the interaction with the individual person, updating a display of the user interface element to indicate the presence of the individual person in the physical environment at the location corresponding to the user interface element; 67. The method of any one of claims 60 to 66, further comprising:

68. updating a display of the user interface element to indicate the presence of the individual person in the physical environment at the location corresponding to the user interface element; altering a display of the user interface element by a first amount in accordance with a determination that one or more first criteria are met; altering the display of the user interface element by a second amount different from the first amount in response to a determination that one or more second criteria different from the first criteria are satisfied; 68. The method of any one of claims 60 to 67, comprising:

69. the user interface includes individual user interface elements that, via the display generation component, at least partially obscure a display of individual objects in a physical environment of the electronic device, the method comprising: While displaying the user interface, updating a display of the individual user interface element to indicate the presence of the individual object in the physical environment at the location corresponding to the individual user interface element in accordance with a determination that the individual object is farther than a respective threshold distance from the electronic device in the physical environment and the individual object satisfies one or more first criteria; refraining from updating the display of the individual user interface element in accordance with a determination that the individual object is farther than the individual threshold distance from the electronic device in the physical environment and the individual object does not satisfy the one or more first criteria; updating a display of the individual user interface element to indicate the presence of the individual object in the physical environment at the location corresponding to the individual user interface element in accordance with a determination that the individual object is closer than the individual threshold distance to the electronic device in the physical environment.

69. The method of any one of claims 60 to 68, further comprising:

70. prior to updating the display of the user interface element to indicate the presence of the individual person in the physical environment at the location corresponding to the user interface element, a first distinct portion of the user interface surrounding a second distinct portion of the user interface corresponding to the location of the individual person in the physical environment is displayed with visual characteristics having a first value; 70. The method of claim 60, wherein after updating the display of the user interface element to indicate the presence of the individual person in the physical environment at the position corresponding to the user interface element, the first individual portion of the user interface surrounding the second individual portion of the user interface corresponding to the position of the individual person in the physical environment is displayed with the visual characteristic having a second value different from the first value.

71. one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: displaying, via a display generation component, a user interface including user interface elements that, via the display generation component, at least partially obscure a view of an individual person within a physical environment of the electronic device; While displaying the user interface, updating a display of the user interface element to indicate a presence of the individual person in the physical environment at a location corresponding to the user interface element in accordance with a determination that the individual person is greater than a threshold distance from the electronic device in the physical environment and that the individual person's attention is directed to a user of the electronic device; The electronic device includes instructions for refraining from updating the display of the user interface elements in accordance with a determination that the individual person is farther than the threshold distance from the electronic device in the physical environment and that the individual person's attention is not directed to the user of the electronic device.

72. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: displaying, via a display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a display of an individual person within a physical environment of the electronic device; While displaying the user interface, updating a display of the user interface element to indicate a presence of the individual person in the physical environment at a location corresponding to the user interface element in accordance with a determination that the individual person is greater than a threshold distance from the electronic device in the physical environment and that the individual person's attention is directed to a user of the electronic device; refraining from updating the display of the user interface elements in accordance with a determination that the individual is farther than the threshold distance from the electronic device in the physical environment and the attention of the individual is not directed to the user of the electronic device; A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

73. 1. An electronic device comprising: one or more processors; Memory, means for displaying, via a display generation component, a user interface including user interface elements, the user interface elements at least partially obscuring, via the display generation component, a display of individual persons within a physical environment of the electronic device; While displaying the user interface, updating a display of the user interface element to indicate a presence of the individual person in the physical environment at a location corresponding to the user interface element in accordance with a determination that the individual person is greater than a threshold distance from the electronic device in the physical environment and that the individual person's attention is directed to a user of the electronic device; means for refraining from updating the display of the user interface elements in accordance with a determination that the individual is farther than the threshold distance from the electronic device in the physical environment and the attention of the individual is not directed to the user of the electronic device; An electronic device comprising:

74. 1. An information processing apparatus for use in an electronic device, comprising: means for displaying, via a display generation component, a user interface including user interface elements, the user interface elements at least partially obscuring, via the display generation component, a display of individual persons within a physical environment of the electronic device; While displaying the user interface, updating a display of the user interface element to indicate a presence of the individual person in the physical environment at a location corresponding to the user interface element in accordance with a determination that the individual person is greater than a threshold distance from the electronic device in the physical environment and that the individual person's attention is directed to a user of the electronic device; means for refraining from updating the display of the user interface elements in accordance with a determination that the individual is farther than the threshold distance from the electronic device in the physical environment and the attention of the individual is not directed to the user of the electronic device; An information processing device comprising:

75. one or more processors; Memory, One or more programs; An electronic device comprising:

71. An electronic device, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform the method of any one of claims 60 to 70.

76. 71. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method of any one of claims 60 to 70.

77. one or more processors; Memory, Means for carrying out the method according to any one of claims 60 to 70; An electronic device comprising:

78. 1. An information processing apparatus for use in an electronic device, comprising:

71. An information processing device comprising means for carrying out a method according to any one of claims 60 to 70.

79. An electronic device in communication with a display generation component and one or more input devices, comprising: displaying, via the display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a view of a respective portion of a physical environment of the electronic device; While displaying the user interface, updating, in accordance with a determination that one or more criteria have been satisfied, including criteria that are satisfied based on a speed of movement of the electronic device, a display of the user interface element, via the display generation component, to reduce the extent to which the user interface element obscures the display of the respective portion of the physical environment; refraining from updating the display of the user interface element in accordance with a determination that the one or more criteria have not been satisfied; and A method comprising:

80. 80. The method of claim 79, wherein the one or more criteria include criteria that are met when the electronic device is not located in a vehicle and that are not met when the electronic device is located in the vehicle.

81. 81. The method of any one of claims 79 to 80, wherein the one or more criteria include a criterion that is met when the physical environment of the electronic device includes one or more detected hazards.

82. 82. The method of any one of claims 79 to 81, wherein the one or more criteria comprises a criterion that is met when the distinct portion of the physical environment of the electronic device includes one or more physical objects.

83. While displaying the user interface, pursuant to a determination that one or more second criteria are satisfied, including criteria satisfied based on one or more characteristics of one or more physical objects in the physical environment of the electronic device, updating, via the display generation component, a display of the user interface element to reduce the extent to which the user interface element obscures a display of a second discrete portion of the physical environment that is smaller than the discrete portion; 83. The method of any one of claims 79 to 82, further comprising:

84. While displaying the user interface, updating, in accordance with a determination that one or more second criteria are satisfied, wherein the criteria are satisfied based on sounds detected within the physical environment of the electronic device, a display of the user interface element, via the display generation component, to reduce the extent to which the user interface element obscures the display of the respective portion of the physical environment; 84. The method of any one of claims 79 to 83, further comprising:

85. one or more processors; Memory, and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: displaying, via a display generation component, a user interface including user interface elements, the user interface elements at least partially obscuring a view of a respective portion of a physical environment of the electronic device, via the display generation component; While displaying the user interface, pursuant to a determination that one or more criteria have been satisfied, including a criterion satisfied based on a speed of movement of the electronic device, updating, via the display generation component, a display of the user interface element to reduce an extent to which the user interface element obscures the display of the respective portion of the physical environment; 11. The electronic device, comprising: instructions for refraining from updating the display of the user interface element in accordance with a determination that the one or more criteria have not been met.

86. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: displaying, via a display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a view of a respective portion of a physical environment of the electronic device; While displaying the user interface, updating, in accordance with a determination that one or more criteria have been satisfied, including criteria that are satisfied based on a speed of movement of the electronic device, a display of the user interface element, via the display generation component, to reduce the extent to which the user interface element obscures the display of the respective portion of the physical environment; refraining from updating the display of the user interface element in accordance with a determination that the one or more criteria have not been satisfied; and A non-transitory computer-readable storage medium for causing a method to be performed, comprising:

87. 1. An electronic device comprising: one or more processors; Memory, means for displaying, via a display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a view of a respective portion of a physical environment of the electronic device; While displaying the user interface, pursuant to a determination that one or more criteria have been satisfied, including a criterion satisfied based on a speed of movement of the electronic device, updating, via the display generation component, a display of the user interface element to reduce an extent to which the user interface element obscures the display of the respective portion of the physical environment; means for refraining from updating the display of the user interface element in response to a determination that the one or more criteria have not been satisfied; An electronic device comprising:

88. 1. An information processing apparatus for use in an electronic device, comprising: means for displaying, via a display generation component, a user interface including a user interface element, the user interface element at least partially obscuring, via the display generation component, a view of a respective portion of a physical environment of the electronic device; While displaying the user interface, pursuant to a determination that one or more criteria have been satisfied, including a criterion satisfied based on a speed of movement of the electronic device, updating, via the display generation component, a display of the user interface element to reduce an extent to which the user interface element obscures the display of the respective portion of the physical environment; means for refraining from updating the display of the user interface element in response to a determination that the one or more criteria have not been satisfied; An information processing device comprising:

89. one or more processors; Memory, One or more programs; An electronic device comprising:

85. An electronic device, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions to perform the method of any one of claims 79 to 84.

90. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

85. A non-transitory computer readable storage medium causing execution of the method of any one of claims 79 to 84.

91. one or more processors; Memory, Means for carrying out the method according to any one of claims 79 to 84; An electronic device comprising:

92. 1. An information processing apparatus for use in an electronic device, comprising:

85. An information processing device comprising means for carrying out a method according to any one of claims 79 to 84.

Images