A method for optimizing virtual user interfaces in a three-dimensional environment.

Abstract

In some embodiments, the computer system modifies the level of detail in which individual environments are displayed. In some embodiments, the computer system applies neutralization adjustments to generate a representation of the physical environment. In some embodiments, the computer system displays user interface objects within a three-dimensional environment using three-dimensional stereoscopic effects. In some embodiments, the computer system facilitates light blending. In some embodiments, the computer system uses visual effects to transition between three-dimensional environments. In some embodiments, the computer system detects movement in the user's viewpoint while displaying a portal to a virtual environment and maintains or discontinues displaying the portal based on the amount of movement and / or the direction in which the portal opens. In some embodiments, the computer system outputs different sound effects when initiating the display of different virtual three-dimensional environments.

Description

Cross - reference to related applications

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 515,117, filed on July 23, 2023; U.S. Provisional Patent Application No. 63 / 506,093, filed on June 4, 2023; and U.S. Provisional Application No. 63 / 503,934, filed on May 23, 2023, the contents of which are hereby incorporated by reference in their entirety for all purposes.

Technical Field

[0002] This relates generally, without limitation, to computer systems that provide computer - generated experiences, including electronic devices that provide virtual and mixed - reality experiences via a display.

Background Art

[0003] The development of computer systems for augmented reality has increased significantly in recent years. Exemplary augmented - reality environments include at least some virtual elements that replace or enhance the physical world. Input devices such as cameras, controllers, joysticks, touch - sensitive surfaces, and touch - screen displays for computer systems and other electronic computing devices are used to interact with virtual / augmented - reality environments. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements such as buttons and other graphics.

Summary of the Invention

[0004] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and restrictive. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems where manipulating virtual objects is complex and error-prone impose a significant cognitive burden on the user and detract from the virtual / augmented reality experience. In addition, these methods are unnecessarily time-consuming, thereby wasting the energy of the computer system. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, there is a need for computer systems with improved methods and interfaces to provide users with computer-generated experiences that make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can optionally complement or replace conventional methods of providing users with extended reality experiences. Such methods and interfaces reduce the number, extent, and / or types of user input by helping the user understand the connection between the inputs provided and the device response to those inputs, thereby generating a more efficient human-machine interface.

[0006] The above-mentioned drawbacks and other problems associated with the user interface of a computer system are mitigated or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has (e.g., includes or communicates with) a display generating component (e.g., a display device such as a head-mounted device (HMD), a display, a projector, a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”), or other devices or components that present visual content to the user that is visible on or in the display generating component itself or generated from the display generating component and is visible elsewhere). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generation components, and the output devices include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions.In some embodiments, the user interacts with the GUI (and / or computer system) through stylus and / or finger touch and gestures on a touch-sensitive surface, the GUI as captured by a camera and other motion sensors, or the user's eye and hand movements in space relative to the user's body, and / or voice input as captured by one or more audio input devices. In some embodiments, the functions performed through the interaction optionally include image editing, drawing, presentation, word processing, spreadsheet creation, gameplay, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally include temporary computer-readable storage media and / or non-temporary computer-readable storage media, or other computer program products configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can complement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the number, degree, and / or type of user input, resulting in a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and increase the interval between battery charges.

[0008] In some embodiments, the computer system modifies the level of detail displayed in individual environments based on the number of application user interfaces displayed simultaneously with the individual environments. In some embodiments, the computer system applies neutralization adjustments to generate a representation of the physical environment. In some embodiments, the computer system displays user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of content associated with user interface objects. In some embodiments, the computer system facilitates light blending for one or more objects in the three-dimensional environment. In some embodiments, the computer system transitions from displaying one three-dimensional environment to displaying another using visual effects dependent on the type (singular or plural) of the environment. In some embodiments, the computer system detects movement in the user's viewpoint while displaying a portal to a virtual environment and maintains or discontinues displaying the portal based on the amount of movement and / or the direction in which the portal opens. In some embodiments, the computer system outputs different sound effects when initiating the display of different virtual three-dimensional environments. In some embodiments, the computer system displays simulated clouds in the environment. In some embodiments, the computer system displays background elements in the environment.

[0009] It should be noted that the various embodiments described herein can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, in particular, in light of the drawings, specification and claims. Furthermore, it should be noted that the language used herein has been selected solely for readability and explanatory purposes and not to define or limit the subject matter of the invention. [Brief explanation of the drawing]

[0010] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0011] [Figure 1A] This block diagram shows the operating environment of a computer system for providing an XR experience, according to several embodiments.

[0012] [Figure 1B] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1C] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1D] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1E] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1F] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1G] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1H] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1I] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1J] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1K] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1L] This is an example of a computer system for providing an XR experience in the operating environment shown in Figure 1A. [Figure 1M] An example of a computer system for providing an XR experience in the operating environment of FIG. 1A. [Figure 1N] An example of a computer system for providing an XR experience in the operating environment of FIG. 1A. [Figure 10] An example of a computer system for providing an XR experience in the operating environment of FIG. 1A. [Figure 1P] An example of a computer system for providing an XR experience in the operating environment of FIG. 1A.

[0013] [Figure 2] A block diagram showing a controller of a computer system configured to manage and adjust an XR experience for a user, according to some embodiments.

[0014] [Figure 3] A block diagram showing a display generation component of a computer system configured to provide visual components of an XR experience to a user, according to some embodiments.

[0015] [Figure 4] A block diagram showing a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0016] [Figure 5] A block diagram showing an eye tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0017] [Figure 6] A flowchart showing a glint-assisted gaze tracking pipeline, according to some embodiments.

[0018] [Figure 7A]The following are examples of computer systems in which the level of detail displayed for individual environments is changed based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments. [Figure 7A1] The following are examples of computer systems in which the level of detail displayed for individual environments is changed based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments. [Figure 7B] The following are examples of computer systems in which the level of detail displayed for individual environments is changed based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments. [Figure 7C] The following are examples of computer systems in which the level of detail displayed for individual environments is changed based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments. [Figure 7D] The following are examples of computer systems in which the level of detail displayed for individual environments is changed based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments.

[0019] [Figure 7E] Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments. [Figure 7F] Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments. [Figure 7G] Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments. [Figure 7H] Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments. [Figure 7I] Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments. [Figure 7J]Examples of displaying simulated clouds and / or background elements in an environment are shown using several embodiments.

[0020] [Figure 8A] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments. [Figure 8B] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments. [Figure 8C] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments. [Figure 8D] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments. [Figure 8E] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments. [Figure 8F] This flowchart shows how to change the level of detail displayed for individual environments based on the number of application user interfaces displayed simultaneously with the individual environments according to several embodiments.

[0021] [Figure 9A] Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment. [Figure 9A1]Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment. [Figure 9B] Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment. [Figure 9C] Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment. [Figure 9D] Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment. [Figure 9E] Several embodiments illustrate how a computer system applies neutralization adjustments to generate a representation of a physical environment.

[0022] [Figure 10A] This flowchart shows a method for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments. [Figure 10B] This flowchart shows a method for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments. [Figure 10C] This flowchart shows a method for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments. [Figure 10D] This flowchart shows a method for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments.

[0023] [Figure 11A] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11A1]This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11B] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11C] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11D] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11E] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11F] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11G] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11H]This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments. [Figure 11I] This document presents examples of computer systems that display user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface objects, according to several embodiments.

[0024] [Figure 12A] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12B] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12C] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12D] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12E] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12F]This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12G] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12H] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments. [Figure 12I] This flowchart shows how to display a user interface object in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of the content associated with the user interface object, according to several embodiments.

[0025] [Figure 13A] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13B] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13C] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13D] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13E] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13F] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13F1] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments. [Figure 13G] Examples of computer systems that facilitate optical blending techniques for one or more objects in a three-dimensional environment are presented, according to several embodiments.

[0026] [Figure 14A] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14B] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14C] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14D] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14E] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14F] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14G] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14H] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14I] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments. [Figure 14J] This flowchart shows a method for facilitating light blending techniques for one or more objects in a three-dimensional environment, according to several embodiments.

[0027] [Figure 15A] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15B] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15C] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15C1] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15D] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15E] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15F] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15G] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15H] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15I] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15J]This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15K] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15L] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15M] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15N] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments. [Figure 15O] This document presents examples of computer systems that transition between displays of different three-dimensional environments, based on several embodiments.

[0028] [Figure 16A] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16B] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16C] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16D] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16E] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16F] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16G]A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16H] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16I] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown. [Figure 16J] A flowchart illustrating a method for transitioning between displays in different three-dimensional environments, according to several embodiments, is shown.

[0029] [Figure 17A] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17B] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17C] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17D] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17E] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17F] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17G] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17H] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17I]Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17J] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17J1] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17K] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17L] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments. [Figure 17M] Examples of computer systems that display and then stop displaying a portal to a virtual environment are shown, according to several embodiments.

[0030] [Figure 18A] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments. [Figure 18B] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments. [Figure 18C] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments. [Figure 18D] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments. [Figure 18E] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments. [Figure 18F] A flowchart is shown illustrating how to display and stop displaying a portal to a virtual environment, according to several embodiments.

[0031] [Figure 19A] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19B] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19C] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19D] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19E] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19F] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19G] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19H] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19H1] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments. [Figure 19I] This document presents examples of computer systems that output different sound effects when initiating the display of different virtual three-dimensional environments, according to several embodiments.

[0032] [Figure 20A]A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20B] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20C] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20D] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20E] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20F] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20G] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20H] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20I] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20J] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20K] A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. [Figure 20L]A flowchart is shown illustrating how to output different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments.

[0033] [Figure 21] A flowchart illustrating how to display simulated clouds in an environment according to several embodiments is shown.

[0034] [Figure 22] A flowchart illustrating methods for displaying background elements within an environment, according to several embodiments, is shown. [Modes for carrying out the invention]

[0035] This disclosure relates to user interfaces that provide users with Extended Reality (XR) experiences, in several embodiments.

[0036] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in multiple ways.

[0037] In some embodiments, while displaying individual environments, the computer system detects changes in the number of application user interfaces displayed simultaneously with the individual environments. In some embodiments, in response to detecting changes in the number of application user interfaces displayed simultaneously with the individual environments, the computer system changes the level of detail at which the individual environments are displayed.

[0038] In some embodiments, while at least a portion of the physical environment of the computer system user is visible, the computer system receives a first input corresponding to a request to apply a first visual effect to a representation of the physical environment. In some embodiments, in response to receiving the first input, the computer system displays a representation of the physical environment. In some embodiments, according to a determination that at least a portion of the physical environment has a first visual appearance, the computer system applies a first visual adjustment to generate a representation of the physical environment. In some embodiments, according to a determination that at least a portion of the physical environment has a second visual appearance different from the first visual appearance, the computer system applies a second visual adjustment different from the first visual adjustment to generate a representation of the physical environment.

[0039] In some embodiments, the computer system displays a user interface object having a first visual appearance within an environment selectable for displaying content. In some embodiments, while displaying the user interface object having the first visual appearance, the computer system detects the attention of the computer system's user directed towards the user interface object. In some embodiments, in response to detecting the user's attention directed towards the user interface object, the computer system displays a user interface object having a second visual appearance different from the first visual appearance, the second visual appearance including a three-dimensional stereoscopic effect corresponding to multiple different views of the content corresponding to the user interface object. In some embodiments, the first visual appearance of the user interface object, displayed before the user's attention is directed towards the user interface object, does not include a three-dimensional stereoscopic effect.

[0040] In some embodiments, the computer system displays a three-dimensional environment that includes virtual objects and one or more physical objects. In some embodiments, the three-dimensional environment includes a portion of the physical environment surrounding the computer system that is visible in a first region of the three-dimensional environment, and a portion of the virtual environment that is displayed in a second region of the three-dimensional environment. In some embodiments, the computer system displays virtual objects using visual lighting effects based on one or more visual characteristics of at least a portion of the physical environment within the three-dimensional environment and one or more visual characteristics of at least a portion of the virtual environment. In some embodiments, the computer system displays physical objects among one or more physical objects using visual lighting effects based on one or more visual characteristics of at least a portion of the physical environment within the three-dimensional environment and one or more visual characteristics of at least a portion of the virtual environment.

[0041] In some embodiments, the computer system displays a first three-dimensional environment that optionally includes a virtual environment, a representation of a physical environment, an atmospheric environment, and / or a mixed environment. Upon detecting user input corresponding to a request to display a second (different) three-dimensional environment, the computer system transitions from displaying the first three-dimensional environment to displaying the second three-dimensional environment, using visual effects appropriate to the type of the first three-dimensional environment (and optionally, the type of the second three-dimensional environment). The visual effects optionally include, for example, crossfading the hues of the first three-dimensional environment with the hues of the second three-dimensional environment, fading out the first three-dimensional environment before fading in the second three-dimensional environment, and / or using other visual effects as described herein.

[0042] In some embodiments, the computer system displays a portal to a virtual environment within a three-dimensional environment, the portal having a first or second opening direction. For example, the portal optionally opens in a first direction (e.g., vertically, perpendicular to the floor or ceiling plane of the three-dimensional environment) or a second direction (e.g., horizontally, perpendicular to the walls or horizontal lines of the three-dimensional environment). For example, the portal optionally opens from above the user's viewpoint, or from below the user's viewpoint (e.g., vertically), or from in front of the user's viewpoint (e.g., horizontally). The computer system detects movement in the user's viewpoint of the computer system and, accordingly, maintains or discontinues displaying the portal depending on the amount of movement and / or the direction in which the portal opens.

[0043] In some embodiments, the computer system receives a first user input corresponding to a request to display an individual virtual three-dimensional environment. In some embodiments, in response to receiving the first user input, the computer system displays the individual virtual three-dimensional environment. In some embodiments, according to the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, the computer system outputs a first sound effect when it begins to display the first virtual three-dimensional environment. In some embodiments, according to the determination that the individual virtual three-dimensional environment is a second virtual three-dimensional environment different from the first virtual three-dimensional environment, the computer system outputs a second sound effect different from the first sound effect when it begins to display the second virtual three-dimensional environment.

[0044] Figures 1A to 6 provide a description of exemplary computer systems for providing users with an XR experience (as described below with respect to methods 800, 1000, 1200, 1400, 1600, 1800, and / or 2000). Figures 7A to 7D illustrate exemplary techniques, according to several embodiments, for changing the level of detail displayed in individual environments based on the number of application user interfaces displayed simultaneously with the individual environments. Figures 8A to 8F are flowcharts, according to several embodiments, for changing the level of detail displayed in individual environments based on the number of application user interfaces displayed simultaneously with the individual environments. The user interfaces in Figures 7A to 7D are used to illustrate the processes in Figures 8A to 8F. Figures 7E to 7J illustrate examples, according to several embodiments, for displaying simulated clouds and / or background elements within an environment. Figures 21 and 22 are flowcharts, according to several embodiments, for displaying simulated clouds and / or background elements within an environment. The user interfaces in Figures 7E to 7J are used to illustrate the processes in Figures 21 and 22. Figures 9A to 9E illustrate exemplary techniques for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments. Figures 10A to 10D are flowcharts of methods for applying neutralization adjustments to generate a representation of a physical environment, according to several embodiments. The user interfaces in Figures 9A to 9E are used to illustrate the processes in Figures 10A to 10D. Figures 11A to 11I illustrate exemplary techniques for displaying user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of content associated with user interface objects, according to several embodiments. Figures 12A to 12I are flowcharts of methods for displaying user interface objects in a three-dimensional environment using three-dimensional stereoscopic effects corresponding to different views of content associated with user interface objects, according to several embodiments. The user interfaces in Figures 11A to 11I are used to illustrate the processes in Figures 12A to 12I.Figures 13A to 13G illustrate exemplary techniques for facilitating optical blending techniques for one or more virtual objects in a three-dimensional environment, according to several embodiments. Figures 14A to 14J are flowcharts of methods for facilitating optical blending techniques for one or more virtual objects in a three-dimensional environment, according to several embodiments. The user interfaces in Figures 13A to 13G are used to illustrate the processes in Figures 14A to 14J. Figures 15A to 15O illustrate techniques for transitioning between displays of different three-dimensional environments. Figures 16A to 16J illustrate flowcharts of methods for transitioning between displays of different three-dimensional environments. The user interfaces in Figures 15A to 15O are used to illustrate the processes in Figures 16A to 16J. Figures 17A to 17M illustrate techniques for displaying and discontinuing portals to virtual environments. Figures 18A to 18D illustrate flowcharts of methods for displaying and discontinuing portals to virtual environments. The user interfaces in Figures 17A to 17M are used to illustrate the processes in Figures 18A to 18D. Figures 19A to 19I illustrate exemplary techniques for outputting different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. Figures 20A to 20L are flowcharts of methods for outputting different sound effects when starting to display different virtual three-dimensional environments, according to several embodiments. The user interfaces in Figures 19A to 19I are used to illustrate the processes in Figures 20A to 20L.

[0045] The processes described below enhance the usability of the device and make the user device interface more efficient (for example, by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various technologies, including providing the user with improved visual feedback, reducing the number of inputs required to perform actions, providing additional control options without cluttering the user interface with additional displayed controls, performing actions without requiring further user input when a set of conditions is met, improving privacy and / or security, providing a more diverse, detailed, and / or realistic user experience while saving memory space, and / or additional technologies. These technologies also reduce power consumption and improve the battery life of the device by enabling the user to use the device more quickly and efficiently. Saving battery power, and therefore weight, improves the ergonomics of the device. These technologies also enable real-time communication, allow the use of fewer and / or less accurate sensors, resulting in more compact, lighter, and less expensive devices, and enabling the device to be used in a variety of lighting conditions. These technologies reduce energy consumption and thereby reduce the heat emitted by the device, which is especially important for wearable devices that can become uncomfortable for the user to wear if they generate excessive heat, even if the device is well within the operating parameters for its components.

[0046] Furthermore, in any method described herein that is conditional on one or more conditions being met in one or more steps, it should be understood that the method described can be repeated in multiple iterations such that all the conditions that the steps of the method are conditional on are met in different iterations of the method. For example, if a method requires that a first step be performed if a condition is met, and a second step be performed if the condition is not met, a person skilled in the art will understand that the steps described in the claim are repeated in an unspecified order until the conditions are met and then not met. Thus, a method described in one or more steps that depends on one or more conditions being met can be rewritten as a method that is repeated until each of the conditions described in the method is met. However, this is not required for a claim of a system or computer-readable medium that includes instructions that perform a conditional action based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency has been met without explicitly repeating the steps of the method until all the conditions that the steps of the method are conditional on are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0047] In some embodiments, as shown in Figure 1A, the XR experience is provided to the user via an operating environment 100 which includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or remote server), display generation components 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a home appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (for example, within a head-mounted device or handheld device).

[0048] When describing an XR experience, various terms are used to refer individually to several related but distinct environments that the user can perceive and / or interact with (for example, using inputs detected by the computer system 101 that generates the XR experience, causing the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following is a subset of these terms.

[0049] Physical Environment: The physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic systems. Examples of physical environments, such as a physical park, include physical objects such as physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses of sight, touch, hearing, taste, and smell.

[0050] Extended reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through an electronic system. In XR, a subset of a person's bodily movements or their representation is tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. For example, an XR system may detect a person's head rotation and, accordingly, adjust the graphical content and sound field presented to the person in a similar way to how such views and sounds would change in a physical environment. In some situations (e.g., for reasons of accessibility), adjustments to the properties of one or more virtual objects within the XR environment may be made in response to a representation of physical movement (e.g., a voice command). A person may perceive and / or interact with an XR object using any one of their senses, including sight, sound, touch, taste, and smell. For example, a person can perceive and / or interact with audio objects that create a 3D or spatial audio environment, providing the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may perceive and / or interact with only audio objects.

[0051] Examples of XR include virtual reality and mixed reality.

[0052] Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment and / or through a simulation of a subset of their physical movement within the computer-generated environment.

[0053] Mixed Reality: A mixed reality (MR) environment is a simulated environment designed to incorporate sensory input or its representation from a physical environment, in addition to including computer-generated sensory input (e.g., virtual objects), in contrast to a virtual reality (VR) environment designed to rely entirely on computer-generated sensory input. On a virtual continuum, a mixed reality environment is any location between, but not encompassing, the complete physical environment at one end and the virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Also, some electronic systems for presenting an MR environment may track location and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles or their representations from the physical environment). For example, the system may take movement into account so that a virtual tree appears stationary relative to the physical ground.

[0054] Examples of mixed reality include augmented reality and augmented virtual reality.

[0055] Augmented Reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on or onto a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display that allows a person to directly view the physical environment. The system may also be configured to present virtual objects on the transparent or translucent display, thereby allowing a person to use the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture an image or video of the physical environment, which is a representation of the physical environment. The system composites the image or video with the virtual objects and presents the composite on the opaque display. A person uses this system to perceive the virtual objects superimposed on the physical environment by indirectly viewing the physical environment through the image or video of the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example, as holograms, into or onto the physical environment, thereby allowing a person to perceive the virtual objects superimposed on the physical environment using the system. An augmented reality environment also refers to a simulated environment in which the representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may transform one or more sensor images to plane a selected perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of it, thereby making the modified portion a non-photorealistic altered version of the original captured image. As yet another example, the representation of the physical environment may be transformed by graphically removing or obscuring a portion of it.

[0056] Augmented Virtuality (AV) refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. These sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park might have virtual trees and buildings, but people with faces might be realistically reproduced from images of real people. Another example is that a virtual object might adopt the shape or color of a physical article captured by one or more imaging sensors. A further example is that a virtual object might adopt shadows that correspond to the position of the sun in the physical environment.

[0057] In augmented reality, mixed reality, or virtual reality environments, a view of a three-dimensional environment is visible to the user. Typically, the view of the three-dimensional environment is visible to the user through one or more display-generating components (e.g., a display or a pair of display modules providing stereoscopic content to different eyes of the same user) via a virtual viewport having a viewport boundary that defines the extent of the three-dimensional environment visible to the user through one or more display-generating components. In some embodiments, the area defined by the viewport boundary is smaller than the user's field of view in one or more dimensions (e.g., based on the user's field of view, the size, optical properties, or other physical characteristics of one or more display-generating components, and / or the location and / or orientation of one or more display-generating components relative to the user's eyes). In some embodiments, the area defined by the viewport boundary is larger than the user's field of view in one or more dimensions (e.g., based on the user's field of view, the size, optical properties, or other physical characteristics of one or more display-generating components, and / or the location and / or orientation of one or more display-generating components relative to the user's eyes). Viewports and viewport boundaries typically move as one or more display-generating components move (for example, with the user's head in the case of a head-mounted device, or with the user's hand in the case of a handheld device such as a tablet or smartphone). The user's viewpoint determines which content is visible within the viewport, and the viewpoint generally specifies the location and orientation of the three-dimensional environment, so that as the viewpoint shifts, the view of the three-dimensional environment also shifts within the viewport. In the case of head-mounted devices, the viewpoint is typically based on the location and orientation of the user's head, face, and / or eyes to provide a view of the three-dimensional environment that is perceptually accurate and provides an immersive experience when the user is using the head-mounted device.For handheld or stationary devices, the viewpoint shifts as the handheld or stationary device moves and / or as the user's position relative to the handheld or stationary device changes (e.g., as the user moves toward or away from the device, above or below the device, to the right of the device, and / or to the left of the device). For devices that include display-generating components with virtual passthrough, the portion of the physical environment visible (e.g., displayed and / or projected) through one or more display-generating components is based on the field of view of one or more cameras communicating with the display-generating components, which typically moves with the display-generating components (e.g., moves with the user's head in a head-mounted device, or moves with the user's hand in a handheld device such as a tablet or smartphone), because the user's viewpoint moves as the field of view of one or more cameras moves (and the appearance of one or more virtual objects displayed through one or more display-generating components is updated based on the user's viewpoint (e.g., the displayed position and orientation of the virtual objects are updated based on the user's viewpoint)). In the case of a display generation component with optical passthrough, parts of the physical environment that are visible through one or more display generation components (for example, optically visible through one or more partially or completely transparent parts of the display generation component) are based on the user's field of view through the partially or completely transparent parts of the display generation component (for example, moving with the user's head in the case of a head-mounted device, or moving with the user's hand in the case of a handheld device such as a tablet or smartphone), because the user's viewpoint moves as the user's field of view moves through the partially or completely transparent parts of the display generation component (one or more), and the appearance of one or more virtual objects is updated based on the user's viewpoint.

[0058] In some embodiments, the representation of the physical environment (e.g., displayed via virtual passthrough or optical passthrough) can be partially or completely obscured by the virtual environment. In some embodiments, the amount of the virtual environment displayed (e.g., the amount of the physical environment not displayed) is based on the level of immersion of the virtual environment (e.g., relative to the representation of the physical environment). For example, increasing the immersion level optionally displays more of the virtual environment and replaces and / or obscures more of the physical environment, while decreasing the immersion level optionally displays less of the virtual environment and reveals portions of the physical environment that were not previously displayed and / or obscured. In some embodiments, at a certain level of immersion, one or more first background objects (e.g., in the representation of the physical environment) are visually less emphasized than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects are discontinued from being displayed. In some embodiments, the immersion level includes the relevant degree to which the virtual content displayed by the computer system (e.g., a virtual environment and / or virtual content) obscures the background content surrounding / behind the virtual content (e.g., content other than the virtual environment and / or virtual content), and optionally includes the number of items of the background content displayed and / or the visual characteristics of the background content on which it is displayed (e.g., color, contrast, and / or opacity), the angular range of the virtual content displayed through the display-generating components (e.g., 60-degree content displayed at low immersion, 120-degree content displayed at medium immersion, or 180-degree content displayed at high immersion), and / or the percentage of the field of view displayed through the display-generating components consumed by the virtual content (e.g., 33% of the field of view consumed by the virtual content at low immersion, 66% of the field of view consumed by the virtual content at medium immersion, or 100% of the field of view consumed by the virtual content at high immersion). In some embodiments, the background content is included in the background on which the virtual content is displayed (e.g., background content within a representation of a physical environment).In some embodiments, background content includes a user interface (e.g., a user interface generated by a computer system corresponding to the application), virtual objects not associated with or included in the virtual environment and / or virtual content (e.g., files or representations of other users generated by the computer system), and / or real objects (e.g., pass-through objects representing real objects in the physical environment around the user, which are visible so as to be displayed through the display generation components and / or are visible through transparent or translucent components of the display generation components so as not to obscure / hinder their visibility through the display generation components by the computer system). In some embodiments, at low immersion levels (e.g., a first immersion level), the background, virtual and / or real objects are displayed in a non-obscuring manner. For example, a low-immersion virtual environment is optionally displayed simultaneously with the background content, and the background content is optionally displayed with full brightness, color, and / or translucency. In some embodiments, at higher immersion levels (e.g., a second immersion level higher than a first immersion level), backgrounds, virtual and / or real objects are displayed in an obscured manner (e.g., dimmed, blurred, or removed from the display). For example, a separate virtual environment with a high immersion level is displayed without simultaneously displaying background content (e.g., in full-screen or fully immersive mode). As another example, a virtual environment displayed at an intermediate immersion level is displayed simultaneously with background content that is dimmed, blurred, or otherwise de-emphasized. In some embodiments, the visual characteristics of background objects differ among them. For example, at a particular immersion level, one or more first background objects are visually de-emphasized more than one or more second background objects (e.g., dimmed, blurred, and / or displayed with increased transparency), and one or more third background objects are not displayed at all.In some embodiments, a null or zero immersion level corresponds to the discontinuation of the display of the virtual environment, and instead, the representation of the physical environment is displayed (optionally together with one or more virtual objects such as applications, windows, or virtual three-dimensional objects) without the representation of the physical environment being obscured by the virtual environment. Adjusting the immersion level using physical input elements provides a quick and efficient way to adjust immersion, improving the usability of the computer system and making the user-device interface more efficient.

[0059] Viewpoint-locked virtual objects: A virtual object is viewpoint-locked when the computer system displays the virtual object in the same location and / or position within the user's view, even if the user's view shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's view is locked in the forward direction of the user's head (e.g., the user's view is at least a portion of the user's field of vision when the user is looking straight ahead). Thus, the user's view remains fixed even if the user's gaze shifts without moving the user's head. In embodiments where the computer system has a display-generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's view is the augmented reality view presented to the user on the display-generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper-left corner of the user's view when the user's view is in a first orientation (e.g., the user's head is facing north) will continue to be displayed in the upper-left corner of the user's view even if the user's view changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position in which a viewpoint-locked virtual object is displayed from the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so that the virtual object is also referred to as a “head-locked virtual object”.

[0060] Environment-Locked Virtual Objects: A virtual object is environment-locked (or "world-locked") when a computer system displays it at a location and / or position in the user's viewpoint that is based on (e.g., selected by reference to and / or fixed to) a location and / or object in a three-dimensional environment (e.g., a physical or virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, and as a result, the environment-locked virtual object will appear at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will appear centered in the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes left-leaning in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will appear left-leaning in the user's viewpoint. In other words, the location and / or position in which an environment-locked virtual object is displayed in the user's viewpoint depends on the location and / or object's position and / or orientation in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a fixed location in the physical environment and / or a coordinate system fixed to an object) to determine the position in which the environment-locked virtual object is displayed from the user's viewpoint. The environment-locked virtual object can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object) or to a moving part of the environment (e.g., a vehicle, animal, person, or a representation of a part of the user's body that moves independently of the user's viewpoint, such as the user's hands, wrists, arms, or feet), so that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0061] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits delayed tracking behavior, reducing or delaying its movement in response to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting delayed tracking behavior, the computer system detects movement of the reference point that the virtual object is following (e.g., a part of the environment, a viewpoint, or a point fixed to the viewpoint, such as a point between 5 and 300 cm from the viewpoint) and intentionally delays the movement of the virtual object. For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first velocity, the virtual object is moved by the device so as to remain locked to the reference point, but at a second velocity slower than the first velocity (e.g., the virtual object begins to catch up to the reference point until the reference point stops or slows down). In some embodiments, when a virtual object exhibits delayed tracking behavior, the device ignores small movements of the reference point (e.g., ignoring movements of the reference point that are below a threshold movement amount, such as a movement of 0 to 5 degrees or a movement of 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a "delayed tracking" threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point.In some embodiments, a virtual object that maintains a substantially fixed position with respect to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / behind the position of the reference point).

[0062] Hardware: There are many different types of electronic systems that enable a person to perceive and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (e.g., contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface. In some embodiments, the controller 110 is configured to manage and adjust the XR experience for the user.In some embodiments, the controller 110 includes a preferred combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to Figure 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server located outside the scene 105 (e.g., a cloud server, a central server, etc.). In some embodiments, the controller 110 is communicably coupled to a display generation component 120 (e.g., an HMD, display, projector, touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within a housing (e.g., a physical housing) of one or more of the display generation components 120 (e.g., a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0063] In some embodiments, the display generation component 120 is configured to provide the user with an XR experience (e.g., at least the visual components of the XR experience). In some embodiments, the display generation component 120 includes a preferred combination of software, firmware, and / or hardware. The display generation component 120 is described in more detail below with reference to Figure 3. In some embodiments, the functions of the controller 110 are provided by and / or combined with the display generation component 120.

[0064] According to some embodiments, the display generation component 120 provides the user with an XR experience while the user is virtually and / or physically present in the scene 105.

[0065] In some embodiments, the display generation component is mounted on a part of the user's body (e.g., their head or hand). Thus, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device, which has a display directed towards the user's field of view and a camera directed towards scene 105. In some embodiments, the handheld device is optionally placed in a housing mounted on the user's head. In some embodiments, the handheld device is optionally placed on a support in front of the user (e.g., a tripod). In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present XR content when the user is not mounting or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interaction with XR content triggered based on interaction occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the XR content response is displayed through the HMD. Similarly, a user interface showing interaction with XR content triggered based on the movement of a handheld or tripod-mounted device relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)) may be implemented similarly to an HMD where the movement is triggered by the movement of the HMD relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)).

[0066] Relevant features of the operating environment 100 are shown in Figure 1A, but those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more suitable embodiments of the exemplary embodiments disclosed herein.

[0067] Figures 1A to 1P show various examples of computer systems used to carry out the method and to provide audio, visual, and / or haptic feedback as part of the user interface described herein. In some embodiments, the computer system optionally includes one or more display generating components (e.g., first and second display assemblies 1-120a, 1-120b and / or first and second optical modules 11.1.1-104a and 11.1.1-104b) for displaying to the user of the computer system a representation of virtual elements and / or a physical environment generated based on detected events and / or user input detected by the computer system. The user interface generated by the computer system is optionally corrected by one or more corrective lenses 11.3.2-216 (optionally detachably attached to one or more of the optical modules) to make it easier for users who otherwise correct their vision using glasses or contact lenses to view the user interface. While many of the user interfaces shown herein show a single view of the user interface, the user interface in the HMD is optionally displayed using two optical modules (e.g., first and second display assemblies 1-120a, 1-120b and / or first and second optical modules 11.1.1-104a and 11.1.1-104b), one for the user's right eye and a different one for the user's left eye, with slightly different images presented to the two different eyes to create a three-dimensional depth illusion, and the single view of the user interface is typically either the right-eye or left-eye view, and the depth effect is described in text or using other schematic diagrams or views.In some embodiments, the computer system includes one or more external displays (e.g., display assembly 1-108) for displaying status information of the computer system to the user of the computer system (when the computer system is not installed) and / or to other people near the computer system, which is optionally generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more audio output components (e.g., electronic component 1-112) for generating audio feedback, which is optionally generated based on detected events and / or user input detected by the computer system. In some embodiments, the computer system includes one or more input devices for detecting inputs such as one or more sensors (e.g., sensor assembly 1-356 and / or one or more sensors in Figure 1I) for detecting information about the physical environment of a device that can be used (optionally in conjunction with one or more illuminators, such as the illuminator shown in Figure 1I) to generate a digital passthrough image, capture a visual medium (e.g., photograph and / or video) corresponding to a physical environment, or determine the orientation (e.g., position and / or orientation) of physical objects and / or surfaces in the physical environment, so that virtual objects can be positioned based on the detected orientation of physical objects and / or surfaces. In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors for detecting the position and / or movement of a hand (e.g., sensor assembly 1-356 and / or one or more sensors in Figure 1I), which may be used to determine when one or more air gestures were performed (optionally in conjunction with one or more illuminators, such as illuminator 6-124 shown in Figure 1I).In some embodiments, the computer system includes one or more input devices for detecting input, such as one or more sensors for detecting eye movement (e.g., eye-tracking and gaze-tracking sensors in Figure 1I), which may be used (optionally, in conjunction with one or more lights, such as lights 11.3.2-110 in Figure 1O) to determine attention or gaze position and / or gaze movement, which may be used to detect gaze-only input based on gaze movement and / or dwell time. Using the various combinations of sensors described above, it may be possible to determine the user's facial expressions and / or hand movements for use when generating the user's avatar or representation, such as a personified avatar or representation for use in a real-time communication session, the avatar having facial expressions, hand movements and / or body movements that are based on or similar to the detected facial expressions, hand movements and / or body movements of the user of the device. Gaze and / or attention information is optionally combined with hand tracking information to determine interactions between the user and one or more user interfaces based on direct and / or indirect inputs such as air gestures or inputs using one or more hardware input devices, including buttons (e.g., first buttons 1-128, buttons 11.1.1-114, second buttons 1-132, and / or dials or buttons 1-328), knobs (e.g., first buttons 1-128, buttons 11.1.1-114, and / or dials or buttons 1-328), digital crowns (e.g., pressable, twistable, or rotatable first buttons 1-128, buttons 11.1.1-114, and / or dials or buttons 1-328), trackpads, touchscreens, keyboards, mice, and / or other input devices.One or more buttons (for example, the first buttons 1-128, buttons 11.1.1-114, the second button 1-132, and / or the dial or button 1-328) are optionally used to perform system actions such as re-centering content in a three-dimensional environment visible to the device user, displaying a home user interface for launching an application, starting a real-time communication session, or starting to display a virtual three-dimensional background. A knob or digital crown (e.g., a first button 1-128, button 11.1.1-114, and / or dial or button 1-328, which is pressable and twistable or rotatable) is optionally rotatable to adjust parameters of the visual content, such as the level of immersion of the virtual three-dimensional environment (e.g., the extent to which the virtual content occupies the user's viewport into the three-dimensional environment), or other parameters associated with the virtual content displayed via the three-dimensional environment and optical modules (e.g., first and second display assemblies 1-120a, 1-120b, and / or first and second optical modules 11.1.1-104a and 11.1.1-104b).

[0068] Figure 1B shows front, top, and perspective views of an example of a head-mountable display (HMD) device 1-100, which is worn by a user and configured to provide a virtual and augmented / mixed reality (VR / AR) experience. The HMD 1-100 may include a display unit 1-102 or assembly, an electronic strap assembly 1-104 connected to and extending from the display unit 1-102, and a band assembly 1-106 fixed to the electronic strap assembly 1-104 at either end. The electronic strap assembly 1-104 and the band 1-106 may be part of a retaining assembly configured to wrap around the user's head to hold the display unit 1-102 against the user's face.

[0069] In at least one example, the band assembly 1-106 may include a first band 1-116 configured to wrap around the back of the user's head and a second band 1-117 configured to extend over the top of the user's head. The second strap may extend between the first electronic strap 1-105a and the second electronic strap 1-105b of the electronic strap assembly 1-104, as shown in the illustration. The strap assembly 1-104 and the band assembly 1-106 may be part of a fastening mechanism that extends rearward from the display unit 1-102 and is configured to hold the display unit 1-102 against the user's face.

[0070] In at least one example, the fastening mechanism includes a first electronic strap 1-105a, which includes a first proximal end 1-134 coupled to a housing 1-150 of the display unit 1-102, for example, and a first distal end 1-136 opposite the first proximal end 1-134. The fastening mechanism may also include a second electronic strap 1-105b, which includes a second proximal end 1-138 coupled to the housing 1-150 of the display unit 1-102, and a second distal end 1-140 opposite the second proximal end 1-138. The fastening mechanism may also include a first band 1-116 having a first end 1-142 coupled to a first distal end 1-136 and a second end 1-144 coupled to a second distal end 1-140, and a second band 1-117 extending between the first electronic strap 1-105a and the second electronic strap 1-105b. The straps 1-105a and 1-105b and the band 1-116 may be connected via a connecting mechanism or assembly 1-114. In at least one example, the second band 1-117 includes a first end 1-146 coupled to a first electron strap 1-105a between a first proximal end 1-134 and a first distal end 1-136, and a second end 1-148 coupled to a second electron strap 1-105b between a second proximal end 1-138 and a second distal end 1-140.

[0071] In at least one example, the first and second electronic straps 1-105a-b include plastic, metal, or other structural material that forms the shape of substantially rigid straps 1-105a-b. In at least one example, the first and second bands 1-116, 1-117 are formed from an elastic flexible material, including woven fabric, rubber, etc. The first and second bands 1-116, 1-117 may be flexible to conform to the shape of the user's head when the HMD 1-100 is worn.

[0072] In at least one example, one or more of the first and second electronic straps 1-105a to b may define an internal strap volume and include one or more electronic components disposed within that internal strap volume. In one example, as shown in Figure 1B, the first electronic strap 1-105a may include electronic component 1-112. In one example, electronic component 1-112 may include a speaker. In another example, electronic component 1-112 may include a computing component such as a processor.

[0073] In at least one example, the housing 1-150 defines a first forward-facing opening 1-152. The display assembly 1-108 is positioned to block the first opening 1-152 from view when the HMD 1-100 is assembled, so the forward-facing opening is labeled with a dotted line at 1-152 in Figure 1B. The housing 1-150 may also define a second rearward-facing opening 1-154. The housing 1-150 also defines an internal volume between the first opening 1-152 and the second opening 1-154. In at least one example, the HMD 1-100 includes a display assembly 1-108, which may include a front cover and a display screen (shown in other figures) disposed within or across the front opening 1-152 to block the front opening 1-152. In at least one example, the display screen of display assembly 1-108 has a curvature configured to follow the curvature of the user's face, as well as the display assembly 1-108 as a whole. The display screen of display assembly 1-108 can be curved to complement the features of the user's face and the overall curvature from one side of the face to the other, for example, from left to right and / or from top to bottom when the display unit 1-102 is pressed.

[0074] In at least one example, the housing 1-150 may define a first aperture 1-126 between a first opening 1-152 and a second opening 1-154, and a second aperture 1-130 between the first opening 1-152 and the second opening 1-154. The HMD 1-100 may also include a first button 1-128 disposed in the first aperture 1-126 and a second button 1-132 disposed in the second aperture 1-130. The first and second buttons 1-128 and 1-132 may be pressable through their respective apertures 1-126 and 1-130. In at least one example, the first button 1-126 and / or the second button 1-132 may be a twistable dial and a pressable button. In at least one example, the first buttons 1-128 are pressable and twistable dial buttons, and the second buttons 1-132 are pressable buttons.

[0075] Figure 1C shows a rear perspective view of HMD1-100. HMD1-100 may include an optical seal 1-110 extending rearward from the housing 1-150 of the display assembly 1-108 and around the outer periphery of the housing 1-150, as shown. The optical seal 1-110 may be configured to extend from the housing 1-150 to the user's face around the user's eyes to block external light from being visible. In one example, HMD1-100 may include first and second display assemblies 1-120a, 1-120b disposed in or within a rearward-facing second opening 1-154 defined by the housing 1-150 and / or disposed within the internal volume of the housing 1-150 and configured to project light through the second opening 1-154. In at least one example, each display assembly 1-120a-b may include respective display screens 1-122a, 1-122b configured to project light backward through a second opening 1-154 toward the user's eyes.

[0076] In at least one example, referring to both Figures 1B and 1C, the display assembly 1-108 may be a forward-facing display assembly including a display screen configured to project light in a first forward direction, and the rear-facing display screens 1-122a-b may be configured to project light in a second rear direction opposite to the first direction. As described above, the light seal 1-110 may be configured to prevent external light from the HMD 1-100, including light projected by the forward-facing display screen of the display assembly 1-108 shown in the front perspective view of Figure 1B, from reaching the user's eyes. In at least one example, the HMD 1-100 may also include a curtain 1-124 that closes a second opening 1-154 between the housing 1-150 and the rear-facing display assemblies 1-120a-b. In at least one example, the curtain 1-124 may be elastic or at least partially elastic.

[0077] Any of the features, components, and / or parts shown in Figures 1B and 1C, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1D to 1F and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1D to 1F, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figures 1B and 1C.

[0078] Figure 1D shows an exploded view of an example of HMD1-200, which includes various parts or components separated according to modularity and the selective coupling of their components. For example, HMD1-200 may include a band 1-216 that can be selectively coupled to first and second electronic straps 1-205a, 1-205b. The first fastening strap 1-205a may include a first electronic component 1-212a, and the second fastening strap 1-205b may include a second electronic component 1-212b. In at least one example, the first and second straps 1-205a and 1-205b may be detachably coupled to a display unit 1-202.

[0079] In addition, the HMD1-200 may include an optical seal 1-210 configured to be detachably coupled to a display unit 1-202. The HMD1-200 may also include a lens 1-218 that can be detachably coupled to the display unit 1-202, for example, on first and second display assemblies including a display screen. The lens 1-218 may include a customized prescription lens configured for vision correction. As stated, each component shown in the exploded view of Figure 1D and described above may be detachably coupled, mounted, remounted, and replaced in order to update or replace parts for different users. For example, bands such as band 1-216, optical seals such as optical seal 1-210, lenses such as lens 1-218, and electronic straps such as straps 1-205a~b may be replaced on a user-by-user basis so that these components are customized to fit and correspond to individual users of the HMD1-200.

[0080] Any of the features, components, and / or parts shown in Figure 1D, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1B, 1C, and 1E-1F and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1B, 1C, and 1E-1F, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1D.

[0081] Figure 1E shows an exploded view of an example of a display unit 1-306 of an HMD. Display unit 1-306 may include a front display assembly 1-308, a frame / housing assembly 1-350, and a curtain assembly 1-324. Display unit 1-306 may also include a sensor assembly 1-356, a logic board assembly 1-358, and a cooling assembly 1-360, disposed between the frame assembly 1-350 and the front display assembly 1-308. In at least one example, display unit 1-306 may also include a rear-facing display assembly 1-320, which includes first and second rear-facing display screens 1-322a, 1-322b, disposed between the frame 1-350 and the curtain assembly 1-324.

[0082] In at least one example, the display unit 1-306 may also include a motor assembly 1-362 configured as an adjustment mechanism for adjusting the position of the display screens 1-322a-b of the display assembly 1-320 relative to the frame 1-350. In at least one example, the display assembly 1-320 is mechanically coupled to a motor assembly 1-362 with at least one motor for each display screen 1-322a-b, so that the motors can translate the display screens 1-322a-b to match the interpupillary distance of the user's eyes.

[0083] In at least one example, the display unit 1-306 may include a dial or button 1-328 that is pressable relative to the frame 1-350 and accessible to the user outside the frame 1-350. The button 1-328 may be electronically connected to the motor assembly 1-362 via a controller so that the user can operate the button 1-328 to cause the motors of the motor assembly 1-362 to adjust the position of the display screens 1-322a-b.

[0084] Any of the features, components, and / or parts shown in Figure 1E, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1B, 1D, and 1F and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1B, 1D, and 1F, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1E.

[0085] Figure 1F shows an exploded view of another example of a display unit 1-406 of an HMD device similar to other HMD devices described herein. Display unit 1-406 may include a forward display assembly 1-402, a sensor assembly 1-456, a logic board assembly 1-458, a cooling assembly 1-460, a frame assembly 1-450, a rear-facing display assembly 1-421, and a curtain assembly 1-424. Display unit 1-406 may also include a motor assembly 1-462 for adjusting the positions of the first and second display subassemblies 1-420a, 1-420b of the rear-facing display assembly 1-421, which include the first and second display screens, respectively, for interpupillary adjustment, as described above.

[0086] Various components, systems, and assemblies shown in the exploded view of Figure 1F are described in more detail herein with reference to Figures 1B to 1E and subsequent figures referenced herein. Display units 1-406 shown in Figure 1F may be assembled and integrated with fastening mechanisms shown in Figures 1B to 1E, which include other components such as electronic straps, bands, and optical seals, and connecting assemblies.

[0087] Any of the features, components, and / or parts shown in Figure 1F, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1B to 1E and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1B to 1E, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1F.

[0088] Figure 1G shows a perspective exploded view of a front cover assembly 3-100 of an HMD device described herein, for example, front cover assembly 3-1 of the HMD 3-100 shown in Figure 1G, or any other HMD device illustrated and described herein. The front cover assembly 3-100 shown in Figure 1G may include a transparent or translucent cover 3-102, a shroud 3-104 (or "canopy"), an adhesive layer 3-106, a display assembly 3-108 including a lenticular lens panel or array 3-110, and a structural trim 3-112. The adhesive layer 3-106 can fasten the shroud 3-104 and / or the transparent cover 3-102 to the display assembly 3-108 and / or the trim 3-112. The trim 3-112 can fasten various components of the front cover assembly 3-100 to the frame or chassis of the HMD device.

[0089] In at least one example, as shown in Figure 1G, a display assembly 3-108 including a transparent cover 3-102, a shroud 3-104, and a lenticular lens array 3-110 can be curved to adapt to the curvature of the user's face. The transparent cover 3-102 and shroud 3-104 can be curved in two or three dimensions, for example, curving perpendicularly in the Z direction inside and outside the ZX plane, and curving horizontally in the X direction inside and outside the ZX plane. In at least one example, the display assembly 3-108 may include a display panel having a lenticular lens array 3-110, as well as pixels configured to project light through the shroud 3-104 and the transparent cover 3-102. The display assembly 3-108 can be curved in at least one direction, for example, horizontally, to adapt to the curvature of the user's face from one side (e.g., left side) to the other side (e.g., right side). In at least one example, as shown and described in more detail in subsequent figures, each layer or component of the display assembly 3-108, which may include a lenticular lens array 3-110 and a display layer, can be curved horizontally, similarly or concentrically, to adapt to the curvature of the user's face.

[0090] In at least one example, the shroud 3-104 may include a transparent or translucent material from which the display assembly 3-108 projects light. In one example, the shroud 3-104 may include one or more opaque portions, such as opaque ink-printed portions or other opaque film portions, on the rear surface of the shroud 3-104. The rear surface may be the surface of the shroud 3-104 that faces the user's eyes when the HMD device is worn. In at least one example, the opaque portions may be on the front surface of the shroud 3-104 opposite the rear surface. In at least one example, one or more opaque portions of the shroud 3-104 may include perimeter portions that visually conceal any components around the perimeter of the display screen of the display assembly 3-108. In this way, the opaque portions of the shroud conceal any other components, including electronic components, structural components, etc., of the HMD device that would otherwise be visible through the transparent or translucent cover 3-102 and / or the shroud 3-104.

[0091] In at least one example, the shroud 3-104 can define one or more aperture transparent portions 3-120 through which a sensor can send and receive signals. In one example, portion 3-120 is an aperture through which a sensor can extend or send and receive signals. In one example, portion 3-120 is a transparent portion, or a portion more transparent than the translucent or opaque portion around the shroud, through which the sensor can send and receive signals through the shroud and through the transparent cover 3-102. In one example, the sensor may include a camera, an IR sensor, a LUX sensor, or any other visual or non-visual environment sensor of the HMD device.

[0092] Any of the features, components, and / or parts shown in Figure 1G, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts described herein. Similarly, any of the features, components, and / or parts shown and described herein, including their arrangement and configuration, may be included, individually or in any combination, in the example of devices, features, components, and parts shown in Figure 1G.

[0093] Figure 1H shows an exploded view of an example of HMD device 6-100. HMD device 6-100 may include a sensor array or system 6-102 which includes one or more sensors, cameras, projectors, etc., attached to one or more components of HMD 6-100. In at least one example, the sensor system 6-102 may include a bracket 1-338 to which one or more sensors of the sensor system 6-102 can be fixed / attached.

[0094] Figure 1I shows a portion of the HMD device 6-100, including the front transparent cover 6-104 and the sensor system 6-102. The sensor system 6-102 may include multiple different sensors, emitters, and receivers, including a camera, IR sensor, and projector. The transparent cover 6-104 is shown in front of the sensor system 6-102 to show the relative positions of the various sensors and emitters and the orientation of each sensor / emitter in the system 6-102. As used herein, “lateral,” “side,” “lateral,” “horizontal,” and other similar terms refer to orientation or direction as indicated by the X-axis shown in Figure 1J. Terms such as “vertical,” “up,” “down,” and similar terms refer to orientation or direction as indicated by the Z-axis shown in Figure 1J. Terms such as “forward,” “backward,” “front,” “rear,” and similar terms refer to orientation or direction as indicated by the Y-axis shown in Figure 1J.

[0095] In at least one example, a transparent cover 6-104 can define the outer front surface of the HMD device 6-100, and a sensor system 6-102, including various sensors and their components, can be positioned behind the cover 6-104 in the Y-axis / direction. The cover 6-104 may be transparent or translucent to allow both the light detected by the sensor system 6-102 and the light emitted thereby to pass through the cover 6-104.

[0096] As described elsewhere in this specification, the HMD device 6-100 may include one or more controllers, including processors, for electrically coupling the various sensors and emitters of the sensor system 6-102 to one or more motherboards, processing units, and other electronic devices such as display screens. In addition, as will be shown in more detail below with reference to other figures, the various sensors, emitters, and other components of the sensor system 6-102 may be coupled to various structural frame members, brackets, etc. of the HMD device 6-100, which are not shown in Figure 1I. Figure 1I shows components of the sensor system 6-102 that are not attached to and electrically coupled from other components, for the sake of clarity as an example.

[0097] In at least one example, the device may include one or more controllers having processors configured to execute instructions stored on memory components electrically coupled to the processors. The instructions may include, or be executed by, one or more algorithms for self-correcting the angles and positions of various cameras described herein over time with use as the initial position, angle, or orientation of the cameras is impacted or deformed due to an unintended fall event or other event.

[0098] In at least one example, the sensor system 6-102 may include one or more scene cameras 6-106. System 6-102 may include two scene cameras 6-106 positioned on either side of the bridge or arch of the HMD device 6-100, such that each of the two cameras 6-102 roughly corresponds to the positions of the user's left and right eyes behind the cover 6-103. In at least one example, the scene cameras 6-106 are generally oriented forward in the Y direction to capture images in front of the user while the HMD 6-100 is in use. In at least one example, the scene cameras are color cameras and provide images and content for MR video passthrough to a display screen facing the user's eyes when the HMD device 6-100 is in use. The scene cameras 6-106 can also be used for environment and object reconstruction.

[0099] In at least one example, the sensor system 6-102 may include a first depth sensor 6-108 that is generally oriented forward in the Y direction. In at least one example, the first depth sensor 6-108 can be used for reconstructing the environment and objects, as well as tracking the user's hands and body. In at least one example, the sensor system 6-102 may include a second depth sensor 6-110 that is centrally positioned along the width of the HMD device 6-100 (for example, along the X axis). For example, the second depth sensor 6-110 can be positioned to align with the central bridge or feature above the user's nose when the HMD 6-100 is worn. In at least one example, the second depth sensor 6-110 can be used for reconstructing the environment and objects, as well as tracking the hands and body. In at least one example, the second depth sensor may include a LIDAR sensor.

[0100] In at least one example, the sensor system 6-102 may include a generally forward-facing depth projector 6-112 to project electromagnetic waves, for example, in the form of a predetermined pattern of light dots, into and within the field of view of the user and / or scene camera 6-106, or into and within the field of view including and beyond the field of view of the user and / or scene camera 6-106. In at least one example, the depth projector may project electromagnetic waves of light in the form of a dot light pattern that is reflected from objects and returned to the aforementioned depth sensors, including depth sensors 6-108, 6-110. In at least one example, the depth projector 6-112 may be used for environment and object reconstruction and hand and body tracking.

[0101] In at least one example, the sensor system 6-102 may include a downward-facing camera 6-114 having a field of view generally directed downward relative to the HMD device 6-100 in the Z-axis. In at least one example, the downward-facing camera 6-114 may be positioned on the left and right sides of the HMD device 6-100 as shown in the figure and may be used for hand and body tracking, headset tracking, and face avatar detection and creation in order to display a user avatar on the forward-facing display screen of the HMD device 6-100 as described elsewhere in this specification. The downward-facing camera 6-114 may be used to capture the facial expressions and movements of the user below the HMD device 6-100, including, for example, the cheeks, mouth, and chin.

[0102] In at least one example, the sensor system 6-102 may include a jaw camera 6-116. In at least one example, the jaw camera 6-116 may be positioned on the left and right sides of the HMD device 6-100 as shown in the figure and may be used for hand and body tracking, headset tracking, and face avatar detection and creation in order to display a user avatar on the forward-facing display screen of the HMD device 6-100 as described elsewhere in this specification. The jaw camera 6-116 may be used to capture the user's facial expressions and movements below the HMD device 6-100, including, for example, the user's jaw, cheeks, mouth, and chin. Regarding hand and body tracking, headset tracking, and face avatar,

[0103] In at least one example, the sensor system 6-102 may include a side camera 6-118. The side camera 6-118 may be oriented to capture left and right side views in the X-axis or direction relative to the HMD device 6-100. In at least one example, the side camera 6-118 may be used for hand and body tracking, headset tracking, and detection and reproduction of a facial avatar.

[0104] In at least one example, the sensor system 6-102 may include multiple eye-tracking and gaze-tracking sensors for determining the user's eye identification information, status, and gaze direction during and / or before use. In at least one example, the eye / gaze-tracking sensor may include nasal eye cameras 6-120 positioned on either side of the user's nose and adjacent to the user's nose when the HMD device 6-100 is worn. The eye / gaze sensor may also include lower eye cameras 6-122 positioned below each user's eye for capturing images of the eye for face avatar detection and creation, gaze tracking, and iris recognition functions.

[0105] In at least one example, the sensor system 6-102 includes an infrared illuminator 6-124 directed outward from the HMD device 6-100, which can illuminate the external environment and any objects within it with IR light for IR detection by one or more IR sensors of the sensor system 6-102. In at least one example, the sensor system 6-102 may include a flicker sensor 6-126 and an ambient light sensor 6-128. In at least one example, the flicker sensor 6-126 may detect the overhead light refresh rate to avoid display flicker. In one example, the infrared illuminator 6-124 may include a light-emitting diode and can be used in low-light environments, in particular, to illuminate the user's hand and other objects with low light for detection by the infrared sensors of the sensor system 6-102.

[0106] In at least one example, multiple sensors, including a scene camera 6-106, a downward-facing camera 6-114, a jaw camera 6-116, a side camera 6-118, a depth projector 6-112, and depth sensors 6-108 and 6-110, can be used in combination with an electrically coupled controller to combine depth data with camera data for hand tracking and sizing, for better hand tracking and object recognition and tracking capabilities of the HMD device 6-100. In at least one example, the downward-facing camera 6-114, jaw camera 6-116, and side camera 6-118 described above and shown in Figure 1I may be wide-angle cameras capable of operating in the visible and infrared spectra. In at least one example, these cameras 6-114, 6-116, and 6-118 may operate with monochrome light detection only to simplify image processing and increase sensitivity.

[0107] Any of the features, components, and / or parts shown in Figure 1I, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1J to 1L and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1J to 1L, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1I.

[0108] Figure 1J shows a downward perspective view of an example of the HMD6-200, including a cover or shroud 6-204 fixed to the frame 6-230. In at least one example, the sensor 6-203 of the sensor system 6-202 may be positioned around the periphery of the HDM6-200 such that the sensor 6-203 is positioned outward around the periphery of the display area or area 6-232 so as not to obstruct the view of the displayed light. In at least one example, the sensor may be positioned behind the shroud 6-204 and aligned with the transparent portion of the shroud to allow the sensor and projector to pass light back and forth through the shroud 6-204. In at least one example, opaque ink or other opaque material or film / layer can be placed on the shroud 6-204 around the display area 6-232 to conceal components of the HMD 6-200 outside the display area 6-232 other than the transparent portion defined by the opaque portion, through which sensors and projectors transmit and receive light and electromagnetic signals during operation. In at least one example, the shroud 6-204 allows light to pass through from the display (e.g., within the display area 6-232) but not radially outward from the display area around the periphery of the display and the shroud 6-204.

[0109] In some examples, the shroud 6-204 includes a transparent portion 6-205 and an opaque portion 6-207, as described above and elsewhere in this specification. In at least one example, the opaque portion 6-207 of the shroud 6-204 can define one or more transparent regions 6-209 from which sensors 6-203 of the sensor system 6-202 can send and receive signals. In the illustrated example, the sensor 6-203 of the sensor system 6-202, which transmits and receives signals through the shroud 6-204, or more specifically through the transparent area 6-209 of (or defined by) the opaque portion 6-207 of the shroud 6-204, may include the same or similar sensors as those shown in the example in Figure 1I, e.g., depth sensors 6-108 and 6-110, depth projector 6-112, first and second scene cameras 6-106, first and second downward-facing cameras 6-114, first and second side cameras 6-118, and first and second infrared illuminators 6-124. These sensors are also shown in the examples in Figures 1K and 1L. Other sensors, sensor types, number of sensors, and their relative positions may be included in one or more other examples of the HMD.

[0110] Any of the features, components, and / or parts shown in Figure 1J, including their arrangement and configuration, may be included, either individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1I and 1K-1L and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1I and 1K-1L, including their arrangement and configuration, may be included, either individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1J.

[0111] Figure 1K shows a partial front view of an example of an HMD device 6-300, including a display 6-334, brackets 6-336 and 6-338, and a frame or housing 6-330. The example shown in Figure 1K does not include a front cover or shroud to show brackets 6-336 and 6-338. For example, the shroud 6-204 shown in Figure 1J includes an opaque portion 6-207 that visually covers / obscures the view of anything outside (e.g., radially / circumferentially outward) of the display / display area 6-334, including sensors 6-303 and brackets 6-338.

[0112] In at least one example, various sensors of sensor system 6-302 are coupled to brackets 6-336, 6-338. In at least one example, scene cameras 6-306 have tight tolerances for angles relative to each other. For example, the tolerance for the mounting angle between two scene cameras 6-306 may be 0.5 degrees or less, e.g., 0.3 degrees or less. To achieve and maintain such tight tolerances, in one example, scene cameras 6-306 can be mounted to bracket 6-338 rather than to the shroud. The bracket may include a cantilever arm to which scene cameras 6-306 and other sensors of sensor system 6-302 can be mounted, such that their position and orientation remain undeformed in the event of a user-induced drop event resulting in any deformation of the other brackets 6-226, housing 6-330, and / or shroud.

[0113] Any of the features, components, and / or parts shown in Figure 1K, including their arrangement and configuration, may be included, either individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1I, 1J, and 1L and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1I-1J and 1L, including their arrangement and configuration, may be included, either individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1K.

[0114] Figure 1L shows a bottom view of an example of the HMD 6-400, including the front display / cover assembly 6-404 and the sensor system 6-402. The sensor system 6-402 may be similar to other sensor systems described above and elsewhere in this specification, including referring to Figures 1I to 1K. In at least one example, the jaw camera 6-416 may be oriented downward to capture an image of the user's lower facial features. In one example, the jaw camera 6-416 may be directly coupled to the frame or housing 6-430, or to one or more internal brackets directly coupled to the illustrated frame or housing 6-430. The frame or housing 6-430 may include one or more apertures / openings 6-415 from which the jaw camera 6-416 can send and receive signals.

[0115] Any of the features, components, and / or parts shown in Figure 1L, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in Figures 1I to 1K and described herein. Similarly, any of the features, components, and / or parts shown and described with reference to Figures 1I to 1K, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1L.

[0116] Figure 1M shows a rear perspective view of the interpupillary distance (IPD) adjustment system 11.1.1-102, which includes first and second optical modules 11.1.1-104a-b that are slidably engaged / coupled to the respective guide rods 11.1.1-108a-b and motors 11.1.1-110a-b of the left and right adjustment subsystems 11.1.1-106a-b. The IPD adjustment system 11.1.1-102 may include buttons 11.1.1-114 that are coupled to the bracket 11.1.1-112 and communicate electrically with the motors 11.1.1-110a-b. In at least one example, the buttons 11.1.1-114 can electrically communicate with the first and second motors 11.1.1-110a~b via a processor or other circuit component to activate the first and second motors 11.1.1-110a~b and change the positions of the first and second optical modules 11.1.1-104a~b relative to each other.

[0117] In at least one example, the first and second optical modules 11.1.1-104a~b may include respective display screens configured to project light toward the user's eyes when the HMD 11.1.1-100 is worn. In at least one example, the user can operate (e.g., press and / or rotate) the button 11.1.1-114 to activate the position adjustment of the optical modules 11.1.1-104a~b to match the interpupillary distance of the user's eyes. The optical modules 11.1.1-104a~b may also include one or more cameras or other sensors / sensor systems for imaging and measuring the user's IPD so that the optical modules 11.1.1-104a~b can be adjusted to match the IPD.

[0118] In one example, the user can operate buttons 11.1.1-114 to trigger automatic position adjustment of the first and second optical modules 11.1.1-104a~b. In another example, the user can operate buttons 11.1.1-114 to trigger manual adjustment, for example, by rotating buttons 11.1.1-114 in one or the other direction, so that the optical modules 11.1.1-104a~b move further away or closer until the user visually matches their IPD. In another example, the manual adjustment is communicated electronically via one or more circuits, and power for the movement of the optical modules 11.1.1-104a~b via motors 11.1.1-110a~b is provided by a power supply. In yet another example, the adjustment and movement of the optical modules 11.1.1-104a~b via the operation of buttons 11.1.1-114 is mechanically actuated via the movement of buttons 11.1.1-114.

[0119] Any of the features, components, and / or parts shown in Figure 1M, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts shown in any other figures shown and described herein. The same applies to any of the features, components, and / or parts shown in Figure 1M, including their arrangement and configuration, which may be shown and described, individually or in any combination, with reference to any other figures shown and described herein.

[0120] Figure 1N shows a partial front perspective view of the HMD 11.1.2-100, including an outer structural frame 11.1.2-102 and an inner or intermediate structural frame 11.1.2-104 that define the first and second apertures 11.1.2-106a and 11.1.2-106b. Views of apertures 11.1.2-106a-b may be obstructed by one or more other components of the HMD 11.1.2-100 coupled to the inner frame 11.1.2-104 and / or the outer frame 11.1.2-102, as shown in the figure; therefore, apertures 11.1.2-106a-b are shown as dotted lines in Figure 1N. In at least one example, the HMD 11.1.2-100 may include a first mounting bracket 11.1.2-108 coupled to the inner frame 11.1.2-104. In at least one example, the mounting bracket 11.1.2-108 is coupled to the inner frame 11.1.2-104 between the first and second apertures 11.1.2-106a and 11.1.2-106b.

[0121] The mounting bracket 11.1.2-108 may include an intermediate or central portion 11.1.2-109 coupled to the inner frame 11.1.2-104. In some examples, the intermediate or central portion 11.1.2-109 may not be the geometric middle or center of the bracket 11.1.2-108. Rather, the intermediate / central portion 11.1.2-109 may be positioned between a first cantilever extension arm and a second cantilever extension arm extending away from the intermediate portion 11.1.2-109. In at least one example, the mounting bracket 108 includes a first cantilever arm 11.1.2-112 and a second cantilever arm 11.1.2-114 that extend away from the intermediate portion 11.1.2-109 of the mounting bracket 11.1.2-108 coupled to the inner frame 11.1.2-104.

[0122] As shown in Figure 1N, the outer frame 11.1.2-102 may be defined with a curved shape on its underside to accommodate the user's nose when the user wears the HMD 11.1.2-100. The curved shape may be referred to as the nose bridge 11.1.2-111 and may be located in the center of the underside of the HMD 11.1.2-100 as shown. In at least one example, the mounting bracket 11.1.2-108 may be connected to the inner frame 11.1.2-102 between apertures 11.1.2-106a-b, such that the cantilever arms 11.1.2-112, 11.1.2-114 extend downward and laterally outward away from the intermediate portion 11.1.2-109 to complement the shape of the nose bridge 11.1.2-111 of the outer frame 11.1.2-104. In this way, the mounting bracket 11.1.2-108 is configured to accommodate the user's nose as described above. The shape of the nose bridge 11.1.2-111 accommodates the nose in such a way that the nose bridge 11.1.2-111 provides a curvature that curves above, over, and around the nose, along with the user's nose, for comfort and fit.

[0123] The first cantilever arm 11.1.2-112 may extend away from the intermediate portion 11.1.2-109 of the mounting bracket 11.1.2-108 in a first direction, and the second cantilever arm 11.1.2-114 may extend away from the intermediate portion 11.1.2-109 of the mounting bracket 11.1.2-10 in a second direction opposite to the first direction. The first and second cantilever arms 11.1.2-112 and 11.1.2-114 are referred to as "cantilevered" or "cantilevered" arms because each arm 11.1.2-112 and 11.1.2-114 includes a distal free end 11.1.2-116 and 11.1.2-118 that is not fixed to the inner and outer frames 11.1.2-102 and 11.1.2-104, respectively. In this way, arms 11.1.2-112 and 11.1.2-114 are cantilevered from an intermediate section 11.1.2-109 that can be connected to the inner frame 11.1.2-104, with their distal ends 11.1.2-102 and 11.1.2-104 not attached.

[0124] In at least one example, the HMD 11.1.2-100 may include one or more components coupled to the mounting bracket 11.1.2-108. In one example, the components include a plurality of sensors 11.1.2-110a~f. Each of the plurality of sensors 11.1.2-110a~f may include various types of sensors, such as cameras and IR sensors. In some examples, one or more of the sensors 11.1.2-110a~f may be used for object recognition in three-dimensional space, such that it is important to maintain the precise relative positions of two or more of the plurality of sensors 11.1.2-110a~f. The cantilevered nature of the mounting bracket 11.1.2-108 can protect the sensors 11.1.2-110a~f from damage and displacement in the event of an accidental drop by the user. Since sensors 11.1.2-110a~f are cantilevered on arms 11.1.2-112 and 11.1.2-114 of mounting bracket 11.1.2-108, stresses and deformations in the inner and / or outer frames 11.1.2-104 and 11.1.2-102 are not transmitted to the cantilever arms 11.1.2-112 and 11.1.2-114, and therefore do not affect the relative positioning of sensors 11.1.2-110a~f coupled to / mounted on mounting bracket 11.1.2-108.

[0125] Any of the features, components, and / or parts shown in Figure 1N, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, and components described herein. Similarly, any of the features, components, and / or parts shown and described herein, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and components shown in Figure 1N.

[0126] Figure 10 shows an example of an optical module 11.3.2-100 for use in electronic devices such as HMDs, including the HDM devices described herein. As shown in one or more other examples described herein, the optical module 11.3.2-100 may be one of two optical modules in an HMD, each optical module being positioned to project light toward the user's eye. In this way, the first optical module can project light toward the user's first eye via a display screen, and the second optical module of the same device can project light toward the user's second eye via another display screen.

[0127] In at least one example, the optical module 11.3.2-100 may include an optical frame or housing 11.3.2-102, which may also be referred to as a barrel or optical module barrel. The optical module 11.3.2-100 may also include a display 11.3.2-104, which includes one or more display screens, coupled to the housing 11.3.2-102. The display 11.3.2-104 may be coupled to the housing 11.3.2-102 such that the display 11.3.2-104 is configured to project light toward the user's eyes when the HMD, of which the display module 11.3.2-100 is part, is worn in use. In at least one example, the housing 11.3.2-102 may surround the display 11.3.2-104 and provide a coupling mechanism for coupling other components of the optical module described herein.

[0128] In one example, the optical module 11.3.2-100 may include one or more cameras 11.3.2-106 coupled to the housing 11.3.2-102. The cameras 11.3.2-106 may be positioned relative to the display 11.3.2-104 and the housing 11.3.2-102 so that the cameras 11.3.2-106 are configured to capture one or more images of the user's eyes while in use. In at least one example, the optical module 11.3.2-100 may also include a light strip 11.3.2-108 surrounding the display 11.3.2-104. In one example, the light strip 11.3.2-108 is positioned between the display 11.3.2-104 and the cameras 11.3.2-106. The light strip 11.3.2-108 may include multiple lights 11.3.2-110. Multiple lights may include one or more light-emitting diodes (LEDs) or other lights configured to project light toward the user's eyes when the HMD is worn. Individual lights 11.3.2-110 of the light strip 11.3.2-108 can be spaced apart around the strip 11.3.2-108 and thus can be spaced uniformly or unevenly around the display 11.3.2-104 at various locations on the strip 11.3.2-108 and around the display 11.3.2-104.

[0129] In at least one example, the housing 11.3.2-102 defines a viewing aperture 11.3.2-101 through which the user can see the display 11.3.2-104 when the HMD device is worn. In at least one example, LEDs are configured and positioned to emit light over the user's eyes through the viewing aperture 11.3.2-101. In one example, a camera 11.3.2-106 is configured to capture one or more images of the user's eyes through the viewing aperture 11.3.2-101.

[0130] As described above, each of the components and features of the optical module 11.3.2-100 shown in Figure 1O can be replicated in another (e.g., a second) optical module arranged with the HMD to interact with the user's other eye (e.g., project light and capture images).

[0131] Any feature, component, and / or part shown in Figure 1O, including their arrangement and configuration, alone or in any combination, may be included in any other example of devices, features, components, and parts shown in Figure 1P or otherwise described herein. Similarly, any feature, component, and / or part illustrated and described with reference to Figure 1P or otherwise described herein, including their arrangement and configuration, alone or in any combination, may be included in the examples of devices, features, components, and parts shown in Figure 1O.

[0132] Figure 1P shows a cross-sectional view of an example of an optical module 11.3.2-200, which includes a housing 11.3.2-202, a display assembly 11.3.2-204 coupled to the housing 11.3.2-202, and a lens 11.3.2-216 coupled to the housing 11.3.2-202. In at least one example, the housing 11.3.2-202 defines a first aperture or channel 11.3.2-212 and a second aperture or channel 11.3.2-214. Channels 11.3.2-212 and 11.3.2-214 may be configured to slidably engage with the respective rails or guide rods of the HMD device to allow the optical module 11.3.2-200 to adjust its position relative to the user's eyes to match the user's interpupillary distance (IPD). The housing 11.3.2-202 can slidably engage with the guide rod to fix the optical module 11.3.2-200 in place within the HMD.

[0133] In at least one example, the optical module 11.3.2-200 may also include a lens 11.3.2-216 coupled to the housing 11.3.2-202 and positioned between the display assembly 11.3.2-204 and the user's eyes when the HMD is worn. The lens 11.3.2-216 may be configured to direct light from the display assembly 11.3.2-204 to the user's eyes. In at least one example, the lens 11.3.2-216 may be part of a lens assembly that includes a corrective lens detachably attached to the optical module 11.3.2-200. In at least one example, lens 11.3.2-216 is positioned above light strip 11.3.2-208 and one or more eye-tracking cameras 11.3.2-206, so that the cameras 11.3.2-206 are configured to capture an image of the user's eye through lens 11.3.2-216, and light strip 11.3.2-208 includes a light configured to project light onto the user's eye through lens 11.3.2-216 during use.

[0134] Any of the features, components, and / or parts shown in Figure 1P, including their arrangement and configuration, may be included, individually or in any combination, in any other example of devices, features, components, and parts described herein. Similarly, any of the features, components, and / or parts shown and described herein, including their arrangement and configuration, may be included, individually or in any combination, in the examples of devices, features, components, and parts shown in Figure 1P.

[0135] Figure 2 is a block diagram of an example of the controller 110 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), central processing unit (CPU), processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global Mobile Communication System (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZiGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0136] In some embodiments, one or more communication buses 204 include circuits for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of the following: a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0137] Memory 220 includes high-speed random-access memory such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-temporary computer-readable storage medium. In some embodiments, memory 220, or the non-temporary computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 230 and XR experience module 240.

[0138] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for each group of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0139] In some embodiments, the data acquisition unit 241 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 of Figure 1A, and optionally from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0140] In some embodiments, the tracking unit 242 is configured to map scene 105 and track the position / location of at least the display generation component 120 relative to scene 105 in Figure 1A, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for this purpose, as well as heuristics and metadata for this purpose. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand, relative to the display generation component 120 and / or a coordinate system defined relative to the user's hand, relative to scene 105 in Figure 1A. The hand tracking unit 244 is described in more detail below with respect to Figure 4. In some embodiments, the eye-tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to the scene 105 (e.g., the physical environment and / or the user (e.g., the user's hands)) or to XR content displayed via the display generation component 120. The eye-tracking unit 243 is described in more detail below with reference to Figure 5.

[0141] In some embodiments, the adjustment unit 246 is configured to manage and adjust the XR experience presented to the user by the display generation component 120 and optionally by one or more of the output devices 155 and / or peripheral devices 195. For this purpose, in various embodiments, the adjustment unit 246 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0142] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data transmission unit 248 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0143] While the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 may be located in separate computing devices.

[0144] Furthermore, Figure 2 is intended to illustrate the functionality of various features that may be present in a particular embodiment, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 2 can be implemented within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0145] Figure 3 is a block diagram of an example of a display generation component 120 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. For that purpose, in some non-limiting examples, the display generation component 120 (e.g., HMD) may include one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, Bluetooth, ZiGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional in-facing and / or out-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0146] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include at least one of the following: an inertial measuring unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.).

[0147] In some embodiments, one or more XR displays 312 are configured to provide the user with an XR experience. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holographic. For example, a display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each of the user's eyes. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0148] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hands and optionally a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene that the user would view if a display generation component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), one or more infrared (IR) cameras, one or more event-based cameras, and / or similar.

[0149] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-temporary computer-readable storage medium. In some embodiments, memory 320, or the non-temporary computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 330 and XR presentation module 340.

[0150] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to the user via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0151] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 in Figure 1A. To this end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0152] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0153] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (for example, a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate extended reality) based on media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0154] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110 and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data transmission unit 348 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0155] Although the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 in Figure 1A), it should be understood that in other embodiments, any combination of the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 may reside in separate computing devices.

[0156] Furthermore, Figure 3 is intended to illustrate the functionality of various features that may be present in a particular implementation, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 3 can be realized within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0157] Figure 4 is a schematic diagram of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 (Figure 1A) is controlled by the hand tracking unit 244 (Figure 2) to track the position / location of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 in Figure 1A (e.g., relative to parts of the physical environment surrounding the user, relative to the display generation component 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined for the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0158] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures a hand image with sufficient resolution to allow for the distinction of fingers and their respective positions. The image sensor 404 can typically capture images of other parts of the user's body, or images of the entire body, and may have either a zoom function or a dedicated sensor with high magnification to capture an image of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures a 2D color video image of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor or a portion thereof is used to define an interaction space in which hand movements captured by the image sensor are processed as input to the controller 110.

[0159] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, thereby extracting high-level information from the map data. This high-level information is typically provided to an application running on the controller via an application programming interface (API), which drives the display generation components 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving their hand 406 to change the orientation of their hand.

[0160] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spot in the pattern. This approach is advantageous in that the user does not need to hold or wear any kind of beacon, sensor, or other marker. This gives the depth coordinates of points in the scene relative to a given reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x, y, and z axes such that the depth coordinates of points in the scene correspond to a z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods such as stereoscopic imaging or time-of-flight measurement based on one or more cameras or other types of sensors.

[0161] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the processor in the image sensor 404 and / or controller 110 processes the 3D map data to extract patch descriptors of the hand within these depth maps. Based on previous training, the software matches these descriptors against patch descriptors stored in the database 408 to estimate the hand pose in each frame. The pose typically includes the 3D location of the user's wrist and fingertips.

[0162] The software can also analyze the trajectory of the hand and / or fingers across multiple frames in a sequence to identify gestures. The posture estimation function described herein may be interleaved with the motion tracking function, so that patch-based posture estimation is performed only once every two (or more) frames, while tracking is used to detect changes in posture that occur over the remaining frames. Posture, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify the image presented on the display generation component 120, or perform other functions, depending on the posture and / or gesture information.

[0163] In some embodiments, the gesture includes an air gesture. An air gesture is a gesture detected by the user without (or independently of) touching an input element that is part of a device (e.g., a computer system 101, one or more input devices 125, and / or a hand tracking device 140), and is based on detected movement of a part of the user's body in the air (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs), including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture including movement of the hand in a predetermined posture by a predetermined amount and / or speed, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body).

[0164] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures, as in some embodiments, performed by moving one or more of the user's fingers relative to other fingers or parts of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, an air gesture is a gesture detected without the user touching (or independently of) an input element that is part of the device, and is based on detected movement of a part of the user's body in the air, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0165] In some embodiments where the input gesture is an air gesture (i.e., no physical contact with an input device that provides the computer system with information about which user interface element is the target of user input, such as contact with a user interface element displayed on a touchscreen or contact with a mouse or trackpad to move a cursor over a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of user input (e.g., in the case of direct input, as described below). Thus, in implementations involving air gestures, the input gesture is the detected attention (e.g., gaze) to the user interface element in combination (e.g., simultaneously) with the movement of the user's fingers (one or more) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0166] In some embodiments, input gestures directed towards a user interface object are performed directly or indirectly by reference to the user interface object. For example, user input is performed directly toward the user interface object in response to the user performing an input gesture with their hand at a position corresponding to the user interface object's position in a three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly toward the user interface object according to the user performing the input gesture while the user's hand position is not at a position corresponding to the user interface object's position in a three-dimensional environment, while detecting the user's attention (e.g., gaze) toward the user interface object. For example, in the case of a direct input gesture, the user can direct their input toward the user interface object by initiating the gesture at or near a position corresponding to the user interface object's display position (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm from the optional outer edge or optional central portion). In the case of indirect input gestures, the user can direct their input towards the user interface object by paying attention to the user interface object (for example, by gazing at the user interface object), and while paying attention to the options, the user initiates the input gesture (for example, at any position detectable by the computer system) (for example, at a position that does not correspond to the display position of the user interface object).

[0167] In some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch and tap inputs for interacting with virtual or mixed reality environments, as in some embodiments. For example, the pinch and tap inputs described later are performed as air gestures.

[0168] In some embodiments, a pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other, i.e., including an optional interruption (e.g., within 0 to 1 second) immediately after the touch. A long pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other for at least a threshold time amount (e.g., at least 1 second) before detecting an interruption of contact between them. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., if two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture includes two (e.g., or more) pinch inputs (e.g., performed with the same hand) that are detected directly and consecutively (e.g., within a predetermined period of time) to each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and then performs a second pinch input within a predetermined period (e.g., within 1 second or 2 seconds) after releasing the first pinch input.

[0169] In some embodiments, an air gesture, a pinch-and-drag gesture, includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in relation to (e.g., after) a drag input that changes the user's hand position from a first position (e.g., a drag initiation position) to a second position (e.g., a resistance termination position). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers) to terminate the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together and touches them to each other, and then moves the same hand to a second position in the air with a drag gesture). In some embodiments, the pinch input is performed by the user's first hand and the drag input is performed by the user's second hand (e.g., the user's second hand moves from the first position to the second position in the air while the user continues the pinch input with the user's first hand). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of the user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in relation to each other (e.g., simultaneously or within a predetermined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using the user's first hand, and a second pinch input performed using the other hand (e.g., a second hand of the user's hands) in relation to performing the pinch input using the first hand.

[0170] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes the movement of one or more of the user's fingers toward the user interface element, the movement of the user's hand toward the user interface element with the user's fingers (one or more) optionally extended toward the user interface element, a downward movement of the user's fingers (e.g., mimicking a mouse click or a tap on a touchscreen), or other default movements of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand that performs the tap gesture movement away from the user's viewpoint and / or toward the object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand that performs the tap gesture (e.g., away from the user's viewpoint and / or the end of the movement toward the object that is the target of the tap input, a reversal of the direction of the finger or hand movement, and / or a reversal of the direction of acceleration of the finger or hand movement).

[0171] In some embodiments, the user's attention is determined to be directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards that part of the three-dimensional environment (optionally, without requiring any other conditions). In some embodiments, for the device to determine that the user's attention is directed towards a part of the three-dimensional environment, the device determines that the user's attention is directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards a part of the three-dimensional environment, subject to one or more additional conditions such as the user's viewpoint being within a distance threshold from the part of the three-dimensional environment, at least for a threshold duration (e.g., dwell time), and / or the gaze being directed towards a part of the three-dimensional environment. If one of the additional conditions is not met, the device determines that the user's attention is not directed towards the part of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0172] In some embodiments, the detection of a ready state configuration of the user or a part of the user is detected by the computer system. The detection of a ready state configuration of the hand is used by the computer system as an indication that the user is likely to be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape where the thumb and one or more fingers are extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap shape where one or more fingers are extended and the palm is facing away from the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head, above the user's waist, or extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., moved towards the area in front of the user above the user's waist, below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of the user interface is responsive to attention (e.g., gaze) input.

[0173] In scenarios where the input is described in reference to an air gesture, similar gestures may also be detected using hardware input devices attached to or held by one or more of the user's hands, in which case the position of the hardware input device in space may be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units, and it should be understood that the position and / or movement of the hardware input device is used instead of the position and / or movement of one or more hands in the corresponding air gesture(s). User input can be detected using controls included in hardware input devices, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers capable of detecting the position or change in position of parts of the hands and / or fingers relative to each other, relative to the user's body, and / or the user's physical environment, and / or other hardware input device controls. User input using controls included in hardware input devices is used in place of hand and / or finger gestures such as air taps or air pinches in corresponding air gestures(single or multiple). For example, a selection input described as being performed by an air tap or air pinch input can alternatively be detected by a button press, a tap on a touch-sensitive surface, a press on a pressure-sensitive surface, or other hardware input.As another example, a movement input described as being performed by an air pinch-and-drag (e.g., an air drag gesture or an air swipe gesture) can alternatively be detected based on interaction with hardware input controls such as button press-and-hold, touch on a touch-sensitive surface, or press on a pressure-sensitive surface, or based on other hardware inputs that follow the movement of a hardware input device in space (e.g., accompanying the hand to which the hardware input device is associated). Similarly, two-handed inputs, including movements of both hands relative to each other, can also be performed using various combinations of inputs detected by air gestures and / or one or more of the aforementioned hardware input devices, using one air gesture and one hardware input device held in the hand not performing the air gesture, two hardware input devices held in separate hands, or two air gestures performed by separate hands.

[0174] In some embodiments, the software may be downloaded electronically to the controller 110, for example, over a network, or instead, it may be provided on a tangible non-temporary medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively or additionally, some or all of the computer's described functions may be implemented in dedicated hardware such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in Figure 4, for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit configuration within the housing of the image sensor 404 (e.g., a hand-tracking device), or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or by any other suitable computerized device such as a game console or media player. The sensing function of the image sensor 404 can also be integrated into a computer or other computerized device controlled by the sensor output.

[0175] Figure 4 further includes schematic diagrams of depth maps 410 captured by image sensor 404 according to several embodiments. The depth map includes a matrix of pixels, each having a depth value, as described above. Pixels 412 corresponding to the hand 406 are segmented in this map from the background and the wrist. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from image sensor 404, with the gradation becoming richer as the depth increases. Controller 110 processes these depth values ​​to identify and segment image components (i.e., groups of adjacent pixels) that have the characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame movement of the depth map sequence.

[0176] Figure 4 also schematically shows the hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to several embodiments. In Figure 4, the hand skeleton 414 is superimposed on the hand background 416, which has been segmented from the original depth map. In some embodiments, the hand (e.g., knuckles, fingertips, center of the palm, end of the hand connected to the wrist), and optionally major feature points on the wrist or arm connected to the hand, are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these major feature points across multiple image frames are used by the controller 110 to determine, according to several embodiments, a hand gesture performed by the hand or the current state of the hand.

[0177] Figure 5 shows an exemplary embodiment of the eye-tracking device 130 (Figure 1A). In some embodiments, the eye-tracking device 130 is controlled by an eye-tracking unit 243 (Figure 2) to track the position and movement of the user's gaze to the scene 105 or to the XR content displayed via the display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device positioned in a wearable frame, the head-mounted device includes both a component for generating XR content for user viewing and a component for tracking the user's gaze to the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or XR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used with a display generation component that is mounted on the head or a display generation component that is not mounted on the head. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally part of a non-head-mounted display generation component.

[0178] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames containing left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include, or be coupled to, one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display on which the user can directly view the physical environment and display virtual objects on a transparent or translucent display. In some embodiments, the display generation component projects virtual objects onto the physical environment. The virtual objects are projected, for example, onto a physical surface or as holograms, so that the individual can use the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0179] As shown in Figure 5, in some embodiments, the eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) camera or a near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, it may be directed toward a "hot" mirror positioned between the user's eye and a display panel that reflects IR or NIR light from the eye to the eye-tracking camera while allowing visible light to pass through. The eye-tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by their respective eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked by a separate eye-tracking camera and light source.

[0180] In some embodiments, the eye-tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye-tracking device for a specific operating environment 100, e.g., the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at the factory or another facility before delivery of the AR / VR device to the end user. The device-specific calibration process may be an automated calibration process or a manual calibration process. A user-specific calibration process may include estimating the eye parameters of a particular user, e.g., pupil location, central visual location, optical axis, visual axis, interpupillary distance. According to some embodiments, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, the images captured by the eye-tracking camera can be processed using a Glint-assisted method to determine the user's current visual axis and gaze point relative to the display.

[0181] As shown in Figure 5, the eye-tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and a gaze tracking system which includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes(s) 592. The eye-tracking camera 540 is positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display or projector of a handheld device) and may be directed towards a mirror 550 that transmits visible light while reflecting IR or NIR light from the eye(s) 592 (e.g., as shown at the top of Figure 5), or may be directed towards the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown at the bottom of Figure 5).

[0182] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. For various purposes, for example, when processing the frames 562 for display, the controller 110 uses gaze tracking input 542 from the eye-tracking camera 540. The controller 110 optionally uses a glint-assisted method or other appropriate method to estimate the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye-tracking camera 540. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0183] The following describes, but is not intended to be limiting, several possible use cases of the user's current gaze direction. As an exemplary use case, the controller 110 may render virtual content differently based on the determined user gaze direction. For example, the controller 110 may generate virtual content at a higher resolution in the central visual region determined from the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content within the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may capture the physical environment of the XR experience and orient an external camera to focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface in the environment that the user is currently viewing on the display 510. In another exemplary use case, the eyepiece 520 may be a focusing lens, and gaze tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object currently being viewed by the user has appropriate binocular coordination to match the convergence of the user's eye 592. The controller 110 can utilize the gaze tracking information to orient and adjust the focus of the eyepiece 520 so that the nearby object being viewed by the user appears at the correct distance.

[0184] In some embodiments, the eye-tracking device is part of a head-mounted device, which is housed within a wearable housing and includes a display (e.g., display 510), two eyepieces (e.g., one or more eyepieces 520), an eye-tracking camera (e.g., one or more eye-tracking cameras 540), and a light source (e.g., an illumination source 530 (e.g., IR or NIR LEDs)). The light source emits light (e.g., IR or NIR light) towards the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circular pattern around each lens, as shown in Figure 5. In some embodiments, as an example, eight illumination sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer illumination sources 530 may be used, and other arrangements and locations of the illumination sources 530 may be used.

[0185] In some embodiments, the display 510 emits light within the visible light range and does not emit light within the IR or NIR range, thus not introducing noise into the gaze tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is positioned on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0186] Embodiments of gaze tracking systems, such as those shown in Figure 5, can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0187] Figure 6 shows glint-assisted gaze tracking pipelines according to several embodiments. In some embodiments, the gaze tracking pipeline is implemented by a glint-assisted gaze tracking system (e.g., an eye-tracking device 130 as shown in Figures 1A and 5). The glint-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the glint-assisted gaze tracking system tracks the pupil contour and glint in the current frame by using prior information from previous frames when analyzing the current frame. When not in tracking state, the glint-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues in tracking state for the next frame.

[0188] As shown in Figure 6, the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then fed into the gaze tracking pipeline for processing, which begins at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images may be fed into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0189] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as shown in 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. If they are not successfully detected, the method returns to element 610 and processes the next image of the user's eyes.

[0190] At 640, if proceeding from element 610, the current frame is analyzed to track the pupil and glint based in part on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the detected pupil and glint in the current frame. The results of processing at element 640 are checked to confirm that the tracking or detection results are reliable. For example, the results may be checked to determine whether a sufficient number of glints for pupil and gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are unreliable, the tracking state is set to no at element 660, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0191] Figure 6 is intended to serve as an example of an eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies that currently exist or may be developed in the future may be used in computer system 101 to provide users with XR experiences in various embodiments, either in place of or in combination with the Glint-assisted eye-tracking technology described herein.

[0192] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, for example, a mixed reality environment in which one or more virtual objects are superimposed on a representation of the real-world environment 602.

[0193] Accordingly, this description describes several embodiments of three-dimensional environments (e.g., XR environments) that include representations of real-world objects and virtual objects. For example, a three-dimensional environment optionally includes a representation of a table existing in a physical environment, which is captured and displayed within the three-dimensional environment (e.g., actively via a computer system's camera and display, or passively via a computer system's transparent or translucent display). As described above, a three-dimensional environment optionally is a mixed reality system based on a physical environment, in which the three-dimensional environment is captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system may optionally selectively display parts and / or objects of the physical environment so that each part and / or object of the physical environment appears to exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system may optionally display virtual objects in a three-dimensional environment so that the virtual objects appear to exist in the real world (e.g., the physical environment) by placing virtual objects in each location within the three-dimensional environment that have corresponding locations in the real world. For example, a computer system may optionally display a vase in such a way that it appears as if a real vase were placed on a table in a physical environment. In some embodiments, individual locations in a three-dimensional environment have corresponding locations in the physical environment.Thus, when a computer system is described as displaying a virtual object at an individual location with respect to a physical object (e.g., the location of a user's hand, or near it, or on a physical table, or near it, etc.), the computer system displays the virtual object at a specific location within the three-dimensional environment such that the virtual object appears as if it were at or near the physical object within the physical world. (For example, if the virtual object were a real object at that specific location, the virtual object is displayed at a location within the three-dimensional environment that corresponds to the location within the physical environment where the virtual object would be displayed if it were a real object at that specific location.)

[0194] In some embodiments, real-world objects that exist within the physical environment that is displayed within the three-dimensional environment (e.g., and / or real-world objects that are visible via a display generation component) can interact with virtual objects that exist only within the three-dimensional environment. For example, the three-dimensional environment can include a table and a vase placed on the table, where the table is a view (or representation) of the physical table within the physical environment, and the vase is a virtual object.

[0195] In a three-dimensional environment (for example, a real environment, a virtual environment, or an environment including a mixture of real and virtual objects), an object may be said to have depth or simulated depth, or an object may be said to be visible, displayed, or positioned at a different depth. In this context, depth refers to dimensions other than height or width. In some embodiments, depth is defined relative to a fixed set of coordinates (for example, a room or object has height, depth, and width defined relative to a fixed set of coordinates). In some embodiments, depth is defined relative to the user's location or viewpoint, in which case the depth dimension varies based on the user's location and / or the location and angle of the user's viewpoint. In some embodiments where depth is defined relative to the user's location positioned with respect to the surface of the environment (e.g., the floor or ground surface of the environment), objects that are away from the user along a line extending parallel to the surface are considered to have a greater depth in the environment, and / or the depth of an object is measured along an axis that extends outward from the user's location and is parallel to the surface of the environment (e.g., depth is defined in a coordinate system of a cylinder or substantially a cylinder, with the user's position at the center of a cylinder extending from the user's head to the user's feet). In some embodiments, depth is defined relative to the user's viewpoint (e.g., a direction relative to a point in space that determines which parts of the environment are visible through a head-mounted device or other display). Objects that are farther away from the user's viewpoint along a line extending parallel to the user's viewpoint are considered to have greater depth in the environment, and / or the depth of an object is measured along an axis extending outward from a line that extends from the user's viewpoint and is parallel to the user's viewpoint (e.g., depth is defined in a spherical or substantially spherical coordinate system with the origin of the viewpoint at the center of a sphere extending outward from the user's head).In some embodiments, depth is defined relative to a user interface container (e.g., a window or application on which application and / or system content is displayed), where the user interface container has height and / or width, and depth is a dimension orthogonal to the height and / or width of the user interface container. In some embodiments, where depth is defined relative to a user interface container, the height and / or width of the container is typically orthogonal or substantially orthogonal to a line extending from a user-based location (e.g., the user's viewpoint or the user's location) to the user interface container (e.g., the center of the user interface container, or another feature point of the user interface container) when the container is placed in a three-dimensional environment or is first displayed (e.g., consequently, the depth dimension of the container extends outward away from the user or the user's viewpoint). In some embodiments, where depth is defined relative to a user interface container, the depth of an object relative to the user interface container refers to the position of the object along the depth dimension of the user interface container. In some embodiments, multiple different containers may have different depth dimensions (e.g., different depth dimensions extending in different directions from the user or the user's viewpoint and / or away from different starting points). In some embodiments, when depth is defined relative to a user interface container, the direction of the depth dimension remains constant relative to the user interface container when the location of the user interface container, the user, and / or the user's viewpoint changes (e.g., when multiple different viewers are viewing the same container in a three-dimensional environment, such as during a face-to-face collaboration session, and / or when multiple participants are in a real-time communication session with shared virtual content containing the container). In some embodiments, for curved containers (e.g., including containers with curved surfaces or curved content areas), the depth dimension optionally extends within the surface of the curved container.In some situations, z-separation (e.g., separation of two objects in the depth dimension), z-height (e.g., distance of one object from another object in the depth dimension), z-position (e.g., position of one object in the depth dimension), z-depth (e.g., position of one object in the depth dimension), or simulated z-dimension (e.g., depth used as the dimension of an object, dimension of an environment, direction in space, and / or direction in a simulated space) is used to refer to the concept of depth as described above.

[0196] In some embodiments, the user can optionally interact with virtual objects in a three-dimensional environment using one or more hands as if the virtual objects were actual objects within a physical environment. For example, as described above, one or more sensors of the computer system can optionally capture one or more of the user's hands and display a representation of the user's hand in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment above), or, in some embodiments, due to the transparency / translucency of a part of the display generation component that displays a user interface, a projection of the user interface onto a transparent / translucent surface, or a projection of the user interface onto the user's eye or the user's field of view, the user's hand is visible through the display generation component by the ability to see the physical environment through the user interface. Thus, in some embodiments, the user's hands are displayed at individual locations in the three-dimensional environment and are treated as if they were objects in the three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if they were actual physical objects within the physical environment. In some embodiments, the computer system can update the display of the representation of the user's hand in the three-dimensional environment in conjunction with the movement of the user's hand in the physical environment.

[0197] In some of the embodiments described below, for example, to determine whether a physical object is directly interacting with a virtual object (e.g., whether a hand is touching, grasping, or holding a virtual object, or whether it is within a threshold distance from the virtual object), the computer system may optionally determine the "effective" distance between the physical object in the physical world and the virtual object in the three-dimensional environment. For example, a hand directly interacting with a virtual object may optionally include one or more of the fingers of a hand pressing a virtual button, a user's hand grasping a virtual vase, two fingers of a user's hand pinching / holding an application's user interface together, and other types of interactions described herein. For example, when determining whether a user is interacting with a virtual object and / or how a user is interacting with a virtual object, the computer system may optionally determine the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the location of the hand in the three-dimensional environment and the location of the virtual object of interest in the three-dimensional environment. For example, one or more of the user's hands are located in a specific position in the physical world, which the computer system optionally captures and displays at a specific corresponding position in a three-dimensional environment (e.g., the position in the three-dimensional environment where the hands are displayed, if the hands are virtual hands rather than physical hands). The position of the hands in the three-dimensional environment is optionally compared to the position of a target virtual object in the three-dimensional environment to determine the distance between the one or more of the user's hands and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the position in the physical world (as opposed to comparing the position in the three-dimensional environment).For example, when determining the distance between one or more of the user's hands and a virtual object, the computer system optionally determines the corresponding location of the virtual object in the physical world (for example, the position in the physical world where the virtual object would be located if it were a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and one or more of the user's hands. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, when determining whether a physical object is in contact with a virtual object, or whether a physical object is within a threshold distance of a virtual object, as described herein, the computer system optionally performs one of the techniques described above to map the location of the physical object to a three-dimensional environment and / or to map the location of the virtual object to a physical environment.

[0198] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed, and / or where and what the physical stylus held by the user is directed. For example, if the user's gaze is directed to a particular position in the physical environment, the computer system optionally determines the corresponding position in the three-dimensional environment (e.g., the virtual position of the gaze), and if a virtual object is located at that corresponding virtual position, the computer system optionally determines that the user's gaze is directed to that virtual object. Similarly, the computer system optionally determines, based on the orientation of the physical stylus, where in the physical environment the stylus is pointing. In some embodiments, based on this determination, the computer system optionally determines the corresponding virtual position in the three-dimensional environment corresponding to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual position in the three-dimensional environment.

[0199] Similarly, embodiments described herein may refer to the location of a user (e.g., a user of a computer system) and / or the location of a computer system in a three-dimensional environment. In some embodiments, the user of a computer system is holding, wearing, or otherwise positioned near the computer system. Thus, in some embodiments, the location of the computer system is used as a proxy for the user's location. In some embodiments, the location of the computer system and / or the user in the physical environment corresponds to individual locations in the three-dimensional environment. For example, if a user stands at a location facing an individual part of the physical environment that is visible through a display-generating component, the location of the computer system is the location in the physical environment (and its corresponding location in the three-dimensional environment) where the user will see objects in the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other) as the objects are visible through the display-generating component of the computer system in the three-dimensional environment. Similarly, if a virtual object displayed in a three-dimensional environment is a physical object in a physical environment (for example, the physical object is located in the same physical environment location as the one in the three-dimensional environment and has the same size and orientation as the one in the three-dimensional environment), then the computer system and / or user's location is the position from which the user will view the virtual object in the physical environment in the same position, orientation, and / or size (for example, absolutely, and / or relative to each other, and in relation to real-world objects) as it was displayed by the computer system's display generation components in the three-dimensional environment.

[0200] This disclosure describes various input methods for interaction with computer systems. Where one example is provided using one input device or method, and another example is provided using a different input device or method, each example may be compatible with the input device or method described in the other example, and their use should be considered optional. Similarly, various output methods for interaction with computer systems are described. Where one example is provided using one output device or method, and another example is provided using a different output device or method, each example may be compatible with the output device or method described in the other example, and their use should be considered optional. Similarly, various methods for interaction with virtual or mixed reality environments via computer systems are described. Where one example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, each example may be compatible with the method described in the other example, and their use should be considered optional. Therefore, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0201] User interface and related processes Here, we focus on embodiments of a user interface ("UI") and related processes that may be performed in a computer system such as a portable multifunction device or head-mounted device, which comprises display generation components, one or more input devices, and (optionally) one or more cameras.

[0202] Figures 7A to 7D show examples of computer systems in which the level of detail displayed for individual environments changes based on the number of application user interfaces displayed simultaneously with the individual environments, according to several embodiments.

[0203] Figure 7A shows a computer system (e.g., an electronic device) 101 that displays a three-dimensional environment 704 from the user's perspective (e.g., facing the back wall of the physical environment in which the computer system 101 is located) via a display generation component (e.g., the display generation component 120 in Figure 1). In some embodiments, the computer system 101 includes a display generation component (e.g., a touchscreen) and a plurality of image sensors (e.g., the image sensor 314 in Figure 3). The image sensors optionally include one or more of the following: a visible light camera, an infrared camera, a depth sensor, or any other sensors that the computer system 101 may use to capture one or more images of the user or a part of the user (e.g., one or more of the user's hands) while the user interacts with the computer system 101. In some embodiments, the user interfaces illustrated and described below may also be implemented on a head-mounted display, which includes a display generating component that displays the user interface or three-dimensional environment to the user, and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user), and / or the user's attention (e.g., including gaze) (e.g., internal sensors facing inward towards the user's face).

[0204] In some embodiments, the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., the operating environment 100), including one or more objects in the physical environment surrounding the computer system 101. In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment, or a portion of the physical environment is visible via the display generation component 120 of the computer system 101. In some embodiments, individual environments are optionally displayed in a simulated three-dimensional environment (e.g., a virtual environment) simultaneously with, or optionally in place of, a representation of the physical environment.

[0205] As shown in Figure 7A, the computer system 101 displays individual environments 706 within the three-dimensional environment 704 instead of representing the physical environment 702 (e.g., full immersion). In Figure 7A, the computer system 101 displays an immersion level indicator 716. In some embodiments, the immersion level indicator 716 indicates the current immersion level (e.g., out of a maximum number of immersion levels) that the computer system 101 is displaying in the three-dimensional environment 704. In some embodiments, the immersion level includes the amount of view of the physical environment obscured (e.g., replaced) by the individual environments 706. In Figure 7A, the immersion level indicator 716 indicates full immersion, and therefore the physical environment is completely replaced by the individual environments 706. In some embodiments, the computer system does not display an immersion level indicator within the three-dimensional environment.

[0206] As shown in Figure 7A, the individual environment 706 is background 1. Some examples of background 1 include a desert background, a mountain background, a beach background, a sporting event background, etc. In some embodiments, the individual environment 706 is based on a physical location. In some embodiments, the individual environment 706 is a location designed by an artist or a simulated physical space. Thus, displaying the individual environment 706 within the three-dimensional environment 704 provides the user with a virtual experience as if they were physically located within the individual environment 706. In Figure 7A, the individual environment 706 corresponding to background 1 includes ambient elements 738, 740, 742, and 744 such as a virtual sky, virtual clouds, virtual animals, and virtual trees.

[0207] In some embodiments, the three-dimensional environment 704 includes virtual content such as application user interfaces. For example, the virtual content optionally includes user interfaces for messaging applications, content browsing applications, media playback applications, etc., as described with reference to Method 800. As shown in Figure 7A, the computer system displays application user interfaces 726a and 726b simultaneously with the individual environment 706. In some embodiments, the computer system 101 displays the individual environment 706 at a lower level of detail when at least one application user interface is displayed simultaneously with the individual environment 706 compared to when there are no application user interfaces displayed simultaneously with the individual environment 706. In some embodiments, the computer system 101 increases or decreases the level of detail at which the individual environment 706 is displayed based on an increase or decrease in the number of application user interfaces displayed simultaneously with the individual environment 706. As described with reference to Method 800, the level of detail at which the individual environment 706 is displayed corresponds to the number of animations of the virtual content displayed in the individual environment, the type of animations displayed in the individual environment, the resolution associated with the animations displayed in the individual environment, and / or the frame rate associated with the animations displayed in the individual environment.

[0208] As shown in Figure 7A, ambient elements 738, 740, 742, and 744 displayed in the individual environment 706 are animated as indicated by curved arrows. The animation of ambient elements 740, 742, and 744 optionally depends on whether the number of application user interfaces displayed simultaneously with the individual environment 706 exceeds the threshold number of application user interfaces (e.g., 1, 2, 3, 4, 5, 10, 20, 50, or 100 application user interfaces). In Figure 7A, the number of application user interfaces displayed simultaneously with the individual environment 706 (e.g., 2 application user interfaces) does not exceed the threshold number of application user interfaces, as indicated by the threshold 722 of the application user interface indicator 720. However, the number of application user interfaces displayed (e.g., as indicated by X applications in Figure 7A) optionally exceeds the threshold number of application user interfaces. Additionally, or alternatively, the computer system 101 may optionally change the level of detail at which it displays the individual environment 706 based on whether the application user interface displayed concurrently with the individual environment 706 is active (e.g., currently in use by the user and / or actively consuming resources). In Figure 7A, application user interface 726a is currently active, but application user interface 726b is not (as indicated by the shading). In some embodiments, the computer system maintains the characteristics of application user interface 726a (e.g., frame rate or pixel density) at a higher level than the characteristics of the individual environment 706 (e.g., frame rate or pixel density) because application user interface 726a is active, as described with reference to Method 800.Conversely, in some embodiments, the computer system maintains the characteristics of the application user interface 726b (e.g., frame rate or pixel density) at a lower level than the characteristics of the individual environment 706 (e.g., frame rate or pixel density) because the application user interface 726b is inactive. In some embodiments, the computer system 101 maintains the characteristics of the application user interface 726a (e.g., frame rate or pixel density) at a higher level than the characteristics of the application user interface 726b (e.g., frame rate or pixel density) because the application user interface 726a is active but the application user interface 726b is inactive.

[0209] In some embodiments, the computer system 101 receives user input corresponding to a request to stop displaying all application user interfaces (e.g., the two application user interfaces in Figure 7A). In some embodiments, the computer system stops displaying the application user interfaces if it does not detect user attention directed towards the displayed application user interfaces (e.g., the two application user interfaces in Figure 7A) for a longer period than a threshold time (e.g., 10 minutes, 30 minutes, 1 hour, 5 hours, or 24 hours). Thus, in Figure 7B, the computer system 101 displays zero application user interfaces simultaneously with the individual environment 706. Since the number of displayed application user interfaces has decreased from Figure 7A to Figure 7B, the computer system 101 increases the level of detail of the individual environment 706 from Figure 7A to Figure 7B. In some embodiments, increasing the level of detail of the individual environment 706 includes increasing the frame rate of the individual environment 706, the pixel density of the individual environment 706, the number of animations displayed in the individual environment 706, the frame rate of the animations displayed in the individual environment 706, the number of ambient elements displayed in the individual environment 706 (e.g., ambient elements 738, 740, 742, and 744 corresponding to a virtual sky, virtual clouds, virtual animals, and virtual trees), and / or the pixel density of ambient elements displayed in the individual environment 706, as described in detail with reference to Method 800. In some embodiments, if an application user interface is not displayed simultaneously with the individual environment 706, the computer system 101 displays the individual environment 706 at the highest level of detail. Therefore, the computer system 101 displays the individual environment 706 at the highest level of detail in Figure 7B by displaying ambient elements 738, 740, 742, and 744 from Figure 7A (corresponding to the virtual sky, virtual clouds, virtual animals, and virtual trees), as well as additional ambient elements 746, 748, and 750 not shown in Figure 7A.As shown in Figure 7B, the additional ambient elements 746 and 748 are virtual shadows (e.g., virtual cloud shadows 746 and virtual tree shadows 748). As an example, the additional ambient element 750 corresponds to virtual water. In some embodiments, as described with reference to Method 800, distortion effects (e.g., ripple effects) are applied to ambient element 750 (e.g., virtual water) based on changes in ambient elements and / or other content displayed in the individual environment 706. For example, changing the animation of simulated wind in the individual environment optionally changes the distortion (e.g., ripples or other texture movement effects) of the virtual water displayed in the individual environment (e.g., an increase in the ripple effect in the virtual water with an increase in simulated wind, or a decrease in the ripple effect in the virtual water with a decrease in simulated wind). In addition to the increase in the number of ambient elements shown from Figure 7A to Figure 7B, each of ambient elements 738, 740, 742, 744, 746, 748, and 750 is animated in Figure 7B (as indicated by the curved arrows).

[0210] Figure 7A1 shows the same and / or same concepts as those shown in Figure 7A (having many of the same reference numerals). Unless otherwise noted below, it is understood that elements shown in Figure 7A1 that have the same reference numerals as elements shown in Figures 7A-7D have one or more or all of the same characteristics. Figure 7A1 includes a computer system 101 which includes (or is the same as) a display generation component 120. In some embodiments, the computer system 101 and the display generation component 120 each have one or more characteristics of the computer system 101 shown in Figures 7A and 7A-7D, and the display generation component 120 shown in Figures 1 and 3, and in some embodiments, the computer system 101 and the display generation component 120 shown in Figures 7A-7D have one or more characteristics of the computer system 101 and the display generation component 120 shown in Figure 7A1.

[0211] In Figure 7A1, the display generation component 120 includes one or more internal image sensors 314a (e.g., an eye-tracking camera 540 as described with reference to Figure 5) oriented toward the user's face. In some embodiments, the internal image sensors 314a are used for eye tracking (e.g., detecting the user's gaze). The internal image sensors 314a are optionally positioned on the left and right portions of the display generation component 120 to enable eye tracking of the user's left and right eyes. The display generation component 120 also includes external image sensors 314b and 314c facing outward from the user to detect and / or capture the physical environment and / or the user's hand movements. In some embodiments, the image sensors 314a, 314b, and 314c have one or more of the characteristics of the image sensor 314 as described with reference to Figures 7A to 7D.

[0212] In Figure 7A1, the display generation component 120 is shown to display content that optionally corresponds to the content described as being displayed and / or visible via the display generation component 120 with reference to Figures 7A to 7D. In some embodiments, the content is displayed by a single display included in the display generation component 120 (e.g., display 510 in Figure 5). In some embodiments, the display generation component 120 includes two or more displays (e.g., left and right display panels for the user's left and right eyes, respectively, as described with reference to Figure 5) having displayed outputs that are merged (e.g., by the user's brain) to create a view of the content shown in Figure 7A1.

[0213] The display generation component 120 has a field of view corresponding to the content shown in Figure 7A1 (for example, a field of view captured by external image sensors 314b and 314c and / or visible to the user via the display generation component 120, indicated by a dashed line in the overhead view). Since the display generation component 120 is optionally a head-mounted device, the field of view of the display generation component 120 is optionally the same as or similar to the user's field of view.

[0214] In FIG. 7A1, the user is depicted as performing an air pinch gesture to provide input to computer system 101 and to provide user input directed to the content displayed by computer system 101. Such depiction is intended to be illustrative and not limiting, and the user may optionally use different air gestures and / or other forms of input to provide user input as described with reference to FIGS. 7A-7D.

[0215] In some embodiments, computer system 101 responds to user input as described with reference to FIGS. 7A-7D.

[0216] In the example of FIG. 7A1, the user's hand is visible within the three-dimensional environment because it is within the field of view of display generation component 120. That is, the user can optionally see any part of their body that is within the field of view of display generation component 120 within the three-dimensional environment. One or more or all aspects of the present disclosure shown in FIGS. 7A-7D, or described with reference to FIGS. 7A-7D, and / or described with reference to the corresponding method(s) are optionally implemented on computer system 101 and display generation unit 120 in a manner similar or analogous to that shown in FIG. 7A1.

[0217] In some embodiments, changing the level of detail of an individual environment 706 (for example, increasing the level of detail in Figure 7B) includes changing the level of detail of a simulated sky using a flow map, as described in detail with respect to Method 800. The flow map optionally includes at least two layers of virtual content corresponding to a simulated sky. Figure 7B shows a side view 745 of the simulated sky to show different layers of the simulated sky and their relative locations. As shown in the side view 745, the flow map of the simulated sky includes a layer of virtual sky (represented by ambient element 738), a layer of virtual clouds (represented by ambient element 740), and a layer of virtual cloud shadows (represented by ambient element 746). In some embodiments, when the computer system 101 changes the level of detail of an individual environment 706, it changes one or more or each of the layers corresponding to the simulated sky having the same level of detail. In some embodiments, when the computer system changes the level of detail of an individual environment 706, it changes each of the layers corresponding to the simulated sky having different levels of detail, as described in detail with respect to Method 800. For example, the computer system 101 optionally controls, separately and independently, one or more of the layers corresponding to the simulated sky, or the corresponding movement, content, and / or level of detail of each layer.

[0218] In Figures 7B to 7C, the computer system 101 optionally receives user input corresponding to a request to display an application user interface. Therefore, in Figure 7C, the computer system 101 displays the application user interface 726a simultaneously with the individual environment 706. Although the computer system 101 is displaying the application user interface 726a, the number of application user interfaces displayed simultaneously with the individual environment 706 does not exceed the threshold number of application user interfaces, as indicated by the threshold 722 of the application user interface indicator 720. However, since the number of displayed application user interfaces has increased from Figure 7B to Figure 7C, the computer system 101 reduces the level of detail of the individual environment 706 from Figure 7B to Figure 7C. In some embodiments, reducing the level of detail of the individual environment 706 includes reducing the frame rate of the individual environment 706, the pixel density of the individual environment 706, the number of animations displayed in the individual environment 706, the frame rate of the animations displayed in the individual environment 706, the number of ambient elements displayed in the individual environment 706 (e.g., ambient elements 738, 740, 742, and 744 corresponding to a virtual sky, virtual clouds, virtual animals, and virtual trees), and / or the pixel density of ambient elements displayed in the individual environment 706, as described in detail with reference to Method 800. In some embodiments, if at least one application user interface, such as application user interface 726a, is displayed simultaneously with the individual environment 706, the individual environment 706 is displayed with less detail than when the application user interface is not displayed. Therefore, the computer system 101 displays individual environments 706 with a reduced level of detail in Figure 7C by either collectively stopping the display of some or all ambient elements and their respective animations, or by stopping the animation of some or all ambient elements displayed within individual environments 706, as described with respect to method 800.As shown in Figure 7C, ambient elements 748 and 750, corresponding to virtual tree shadows and virtual water, respectively, are not animated. In Figure 7C, while the computer system 101 maintains the display of some ambient elements based on simulated light, such as ambient element 748 (e.g., virtual tree shadows), the computer system 101 discontinues the display of some ambient elements and their respective animations based on simulated light, such as ambient element 746 (e.g., virtual cloud shadows from Figure 7B). However, the computer system 101 maintains the animation of ambient elements 738, 740, 742, 744, and 748, corresponding to the virtual sky, virtual clouds, virtual animals, virtual trees, and virtual tree shadows, respectively, from Figure 7B to Figure 7C. Furthermore, since the application user interface 726a is active, the computer system 101 maintains the characteristics of the application user interface 726a (e.g., frame rate or pixel density) at a higher level than the characteristics of the individual environment 706 (e.g., frame rate or pixel density).

[0219] In Figure 7D, the computer system 101 optionally maintains the display of the application user interface from Figure 7C and receives user input corresponding to a request to display an additional application user interface. In Figure 7D, the computer system 101 optionally discontinues the display of the application user interface from Figure 7B and instead receives user input corresponding to a request to display a new application user interface. In Figure 7D, the computer system 101 displays five application user interfaces 726c simultaneously with the individual environment 706. As indicated by the threshold 722 of the application user interface indicator 720, the number of application user interfaces displayed simultaneously with the individual environment 706 exceeds the threshold number of application user interfaces (e.g., four application user interfaces). In some embodiments, if the number of displayed application user interfaces exceeds the threshold number of application user interfaces, the computer system 101 discontinues displaying some or all of the ambient elements (optionally maintaining the display of some ambient elements). As shown in Figure 7D, the computer system 101 reduces the level of detail of the individual environment 706 in Figure 7C by ceasing to display ambient elements 740, 742, 746, 748, and 750 in Figure 7C. Additionally or alternatively, the computer system 101 ceasing to display the animation of some or all ambient elements if the number of displayed application user interfaces exceeds a threshold number of application user interfaces. As shown in Figure 7D, the computer system ceasing the animation of all ambient elements, such as ambient elements 738, 740, and 744, displayed in the individual environment 706. In some embodiments, the computer system 101 resumes the animation of some or all ambient elements when the number of displayed application user interfaces is reduced to within a threshold number of application user interfaces, as described in detail with reference to Method 800.

[0220] Figures 7E to 7J show examples of displaying simulated clouds and / or background elements within an environment, such as the environment described with reference to Figures 7A to 7D.

[0221] In Figure 7E, the three-dimensional environment 706 is visible via the display generation component 120 of the computer system 101. The environment 706 optionally has one or more of the characteristics of the environments in Figures 7A to 7D. The environments 706 in Figures 7E to 7J optionally include user interface elements 726d corresponding to one or more of the user interfaces 726a to c. The environment 706 in Figure 7E includes simulated clouds 740a and 740b of a simulated sky or a real sky (e.g., corresponding to ambient elements 738, 740, 742, and / or 744), simulated water (e.g., of a simulated ocean) corresponding to the illustrated parts 760a, 706b, and 760c, and / or simulated sand (e.g., of a simulated beach) corresponding to the part of the environment 706 below / before part 760a. In some embodiments, the user interface element 726d is displayed in front of or overlapping with one or more parts of the environment 706, as shown in Figure 7E.

[0222] In some embodiments, in order to reduce the computing resources required to display the environment 706, the computer system 101 displays one or more portions of the environment 706 in the manner described with reference to Figures 7E to 7J. For example, in the case of simulated water, the computer system 101 optionally displays a portion 760a of the simulated water with a relatively high level or quality of animation detail (e.g., the portion of the simulated water closest to the user's viewpoint), a portion 760b of the simulated water with a moderate level or quality of animation detail (e.g., a portion of the simulated water that is further away than portion 760a but closer to the user's viewpoint than portion 760c), and a portion 760c with a relatively low level or quality of animation detail or without any animation at all (e.g., the portion of the simulated water furthest from the user's viewpoint). For example, the animations described above optionally correspond to the animation of ripples on the surface of the simulated water. Relatively high-level animation detail optionally includes using relatively high-resolution elements for section 760a, a relatively large number of elements animated within section 760a, and / or relatively high-frequency animation of elements within section 760a. Similarly, relatively medium-level animation detail optionally includes using relatively medium-resolution elements for section 760a, a relatively medium number of elements animated within section 760a, and / or relatively medium-frequency animation of elements within section 760a.

[0223] In some embodiments, the computer system 101 displays simulated shadows within the environment 706, as shown in Figure 7F. In Figure 7F, the environment 706 includes a simulated shadow 746a projected by a simulated cloud 740a, a simulated shadow 746b projected by a simulated cloud 740b, and a simulated shadow 748a projected by a simulated tree 744a. In some embodiments, as indicated by the dashed areas of the simulated shadows, the texture(s) displayed by the computer system 101 as the visual appearance of the simulated shadows are optionally variable across a given shadow. For example, the central region of the simulated shadow 746a optionally has higher or lower visual saturation (e.g., opacity, diffuseness, color, and / or brightness) than the outer regions of the simulated shadow 746a. In some embodiments, the visual saturation of a given simulated shadow changes (optionally smoothly) from the center point of the shadow to the edge of the shadow. In some embodiments, the visual appearance of shadows 746b and 748a also has one or more of the above-described characteristics.

[0224] The texture used to display a given simulated shadow (e.g., color, brightness, contour, reflectivity, and / or opacity) is optionally different depending on the portion of the environment 706 in which the simulated shadow is displayed. For example, the simulated shadow 746a in Figure 7F is optionally displayed on water at a relatively large simulated depth and is therefore displayed with a texture having a first visual appearance, which is optionally different from the visual appearance of the texture of the simulated shadow 746b, which is optionally displayed on a portion of simulated water at a relatively small simulated depth. The simulated shadow 748a optionally has a texture having a different appearance from the simulated shadows 746a and / or simulated shadows 746b, because the simulated shadow 748a is displayed on simulated sand in the environment 706. Further details regarding the visual appearance of the textures of the simulated shadows are provided by referring to Method 2100.

[0225] In Figure 7F, the computer system 101 also displays simulated reflections or simulated lighting effects, such as glints 752, on various surfaces within the environment 706. For example, the computer system 101 displays simulated reflections 752a, 752b, and 752c on simulated water and simulated sand surfaces. The simulated reflection 752a optionally has a different visual appearance from the simulated reflection 752b, and the simulated reflection 752b optionally has a different visual appearance from the simulated reflection 752c. The visual appearance of the simulated reflections 752a, 752b, and 752c optionally depends on the characteristics of the light source, which is the simulated light source of the reflection, and / or the part of the environment 706 on which the simulated reflection is displayed. Additional details regarding the visual appearance of the simulated reflections are provided by referring to Method 2100.

[0226] From Figure 7F to Figure 7G, the simulated clouds 740a and 740b are moving relative to the environment 706. Furthermore, the simulated cloud 740b changes size and / or shape. As a result, in Figure 7G, the computer system 101 displays the simulated shadow 746a moving to the right within the environment 706. By moving to the right, the simulated shadow 746a ceases to display the simulated reflection 752b that was displayed in Figure 7F in Figure 7G (for example, because the simulated shadow 746a occupies the area where the simulated reflection 752b was displayed in Figure 7F), and causes the simulated reflection 752b that was not displayed in Figure 7F to be displayed in Figure 7G (for example, because the simulated shadow 746a no longer occupies the area where the simulated reflection 752b is displayed in Figure 7G).

[0227] In Figure 7G, the size and / or shape of the simulated cloud 740b has changed, and therefore the simulated shadow 746b corresponding to the simulated cloud 740b has also changed in size and / or shape in Figure 7G accordingly. Furthermore, the simulated cloud 740b has moved to the right from Figure 7F to Figure 7G, toward the user's viewpoint, and therefore the simulated shadow 746b has also moved to the right from Figure 7F to Figure 7G, toward the user's viewpoint. Since the simulated shadow 746b is displayed here on simulated sand rather than simulated water, the computer system 101 optionally changes the visual appearance of the texture used to display the simulated shadow 746, as previously described. Furthermore, by moving from Figure 7F to Figure 7G, the simulated shadow 746b ceases to display the simulated reflection 752c displayed in Figure 7F in Figure 7G (for example, because the simulated shadow 746b occupies the area where the simulated reflection 752c was displayed in Figure 7F).

[0228] The movement of the simulated shadow 746b from Figure 7F to Figure 7G also causes the simulated shadows 746b and 748a to overlap at least partially, as represented by region 746x. In some embodiments, when two (or more) simulated shadows overlap at least partially, the computer system 101 selects a texture to use for the overlapping region 746x based on which simulated shadow has higher visual splendor in that region. For example, in Figure 7G, since the simulated shadow 748a has higher visual splendor than the simulated shadow 746b, the computer system 101 displays the overlapping region 746x with the texture of the simulated shadow 748a and stops displaying the texture of the simulated shadow 746b within the overlapping region 746x. The computer system 101 does this instead of adding or combining the textures of the two simulated shadows in some other way to produce a more realistic appearance of the overlapping region 746x in a power-efficient manner. The portions of the simulated shadows 746b and / or 748a outside the overlapping region 746x are optionally continued to be displayed with their respective simulated shadow textures. Additional details regarding the visual appearance of the simulated shadows are provided in Method 2100.

[0229] From Figure 7G to Figure 7H, the user's viewpoint changes as shown in the schematic overhead view. For example, when the user turns their head to the left, the computer system 101 updates the display of environment 706 in Figure 7H to reveal a portion of environment 706 that is further to the left than the portion of environment 706 that was visible in Figure 7G. In response to the change in viewpoint, the computer system 101 optionally changes the number and / or position of the simulated reflections 752a, 752b, and / or 752c, optionally displays simulated reflections that were not previously displayed (for example, even if the portion of environment 706 on which the simulated reflections are displayed is also visible from the viewpoint shown in Figure 7G), and / or optionally discontinues the display of previously displayed simulated reflections (for example, even if the portion of environment 706 on which the simulated reflections are displayed remains visible from the viewpoint shown in Figure 7H). Additional details regarding the changes in the visual appearance of environment 706 based on the change in viewpoint are provided by referring to Method 2100.

[0230] Figures 7I and 7J illustrate an example of displaying background elements within an environment (e.g., environment 706) composed of multiple layers of virtual elements whose visual appearance can be independently controlled. The environment, visible via the display generation component 120 in Figure 7I, includes simulated water (as described with reference to Figures 7E-7H, for example) and a simulated sky, which is optionally a background element within the environment. The simulated sky optionally consists of three layers of virtual elements. In the first or bottom layer (e.g., closest to the user's viewpoint), the simulated sky optionally includes simulated clouds 740a and 740b. In the second or middle layer (e.g., further from the user's viewpoint than the first or bottom layer), the simulated sky optionally includes a simulated moon 762a. In the third or top layer (e.g., further from the user's viewpoint than the second or middle layer), the simulated sky optionally includes simulated stars 760a-e. The background element is optionally displayed with its center optionally on the surface of a spherical volume which is the user's viewpoint, as indicated by the curved appearance of the cross-section of the background element in the upper right region of Figure 7I. The cross-section of the background element optionally reflects the relative placement and / or movement of the virtual elements in the three layers of the background element in a simulated empty area indicated by a dashed box.

[0231] The computer system 101 in Figure 7I also displays simulated lighting effects 764 (e.g., simulated rays) corresponding to one or more simulated light sources, whether or not they are within background elements. The computer system 101 also displays various simulated reflections 752a-c on the surface of the simulated water, as described with reference to Figures 7E-7H. As shown in Figure 7I, the simulated lighting effects 764 are displayed to appear to emanate from beneath the simulated cloud 740b, for example, when they are produced by simulated light from a simulated moon 762a passing through the simulated cloud 740b, resulting in simulated lighting effects 764 extending from beneath the simulated cloud 740b onto the surface of the simulated water. In some embodiments, as shown in Figure 7I, the computer system 101 displays one or more simulated reflections 752a-c on the surface of the simulated water where the simulated lighting effects 764 intersect with the surface of the simulated water.

[0232] From Figure 7I to Figure 7J, the simulated star 760a in the uppermost layer of the background elements remains stationary, the simulated moon 762a in the middle layer of the background elements moves to the left, and the simulated clouds 740a and 740b in the lowermost layer of the background elements move to the right. As a result, the computer system 101 selectively updates the visual appearance of the environment, as shown in Figure 7J. For example, the simulated lighting effects 764 are updated to have different orientations relative to the environment (e.g., the simulated moon 762a, to maintain the alignment of the simulated lighting effects 764 with the simulated light source of their effects), and the computer system 101 changes the display of one or more of the simulated reflections 752a-c, as shown in Figures 7I to Figure 7J. Additional details regarding the changes in the visual appearance of the background elements are provided by referring to Method 2200.

[0233] Figures 8A to 8F are flowcharts illustrating exemplary methods 800 that facilitate depth contention mitigation for one or more virtual objects in a three-dimensional environment by modifying the visual properties of one or more virtual objects, according to several embodiments. In some embodiments, method 800 is performed on a computer system (e.g., computer system 101 in Figure 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 in Figures 1, 3, and 4) (e.g., a head-up display, display, touchscreen, and / or projector) and one or more cameras (e.g., a camera pointing downward from the user's hand (e.g., a color sensor, infrared sensor, and other depth-sensing camera), or a camera pointing forward from the user's head). In some embodiments, method 800 is stored in a non-temporary computer-readable storage medium and managed by instructions executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 in Figure 1A). Some operations of method 800 are combined in an optional manner, and / or the order of some operations is changed in an optional manner.

[0234] In some embodiments, Method 800 is performed on a computer system, such as the computer system 101 in Figure 1, which communicates with a display generation component and one or more input devices. For example, a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer or other computer system. In some embodiments, the display generation component is a display integrated with the computer system (optionally a touchscreen display), an external display such as a monitor, projector, or television, or a hardware component (optionally integrated or external) for projecting a user interface or making a user interface visible to one or more users. In some embodiments, one or more input devices include a computer system or component capable of receiving user input (e.g., capturing user input and / or detecting user input) and transmitting information associated with the user input to the computer system. Examples of input devices include touchscreens, mice (e.g., external), trackpads (optionally integrated or external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the computer system), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand movement sensors). In some embodiments, the computer system communicates with the hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (touchscreen, trackpad)). In some embodiments, the hand-tracking device is a wearable device such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device such as a remote control or stylus.

[0235] In some embodiments, while displaying individual environments such as the individual environments 704 in Figures 7A and 7A1 via the display generation component, the computer system detects a change in the number of application user interfaces (e.g., media applications (e.g., television or photography), messaging applications, health applications, and / or web browsing application user interfaces) 726a and 726b in Figures 7A and 7A1 that are displayed simultaneously with the individual environments (802a). In some embodiments, the individual environments include a three-dimensional environment. In some embodiments, the three-dimensional environment includes an environment corresponding to the physical environment surrounding the display generation component. In some embodiments, the three-dimensional environment has one or more of the characteristics of the (three-dimensional) environment of methods 1000, 1200, 1400, 1600, 1800 and / or 2000. In some embodiments, the three-dimensional environment is generated, displayed, or otherwise made viewable by a computer system (e.g., an Extended Reality (XR) environment such as a Virtual Reality (VR) environment, a Mixed Reality (MR) environment, and / or an Augmented Reality (AR) environment). In some embodiments, the physical environment is visible through the transparency of the display generation component (e.g., true or actual passthrough). In some embodiments, a representation of the physical environment is displayed within the three-dimensional environment via a display generation component (e.g., virtual or video passthrough). In some embodiments, a separate virtual environment (e.g., a simulated three-dimensional environment) is displayed via a display generation component, optionally in place of a representation of the physical environment (e.g., fully immersive), or optionally simultaneously with a representation of the physical environment (e.g., partially immersive), as described with respect to step (singular or plural) 804. In some embodiments, the separate virtual environment represents a simulated physical space. Some examples of virtual environments include a lake environment, a mountain environment, a sunset scene, a sunrise scene, a night environment, a grassland environment, and / or a concert scene. In some embodiments, the virtual environment is based on an actual physical location, such as a museum and / or aquarium. In some embodiments, the virtual environment is a location designed by an artist.Therefore, displaying a virtual environment optionally provides the user with a virtual experience as if they were physically located within the virtual environment. In some embodiments, individual environments have one or more characteristics of the environments described with reference to Methods 1000, 1200, 1400, 1600, 1800 and / or 2000. In some embodiments, in response to receiving user input to initiate the startup or shutdown of an application user interface(s), the computer system displays or discontinues the display of an application user interface(s), thereby detecting a change in the number of application user interfaces(s).

[0236] In some embodiments, in response to detecting a change in the number of application user interfaces displayed concurrently with the individual environment, the computer system changes the level of detail to which the individual environment is displayed, such as the level of detail in Figure 7B (802b). In some embodiments, the number of application user interfaces corresponds to the number of different applications running concurrently while the individual environment is displayed. In some embodiments, the number of application user interfaces corresponds to the number of windows of the same and / or different applications running concurrently while the individual environment is displayed. In some embodiments, changing the level of detail (e.g., increasing or decreasing it), as described in detail below, is based on the number of application user interfaces displayed concurrently with the individual environment. In some embodiments, the level of detail to which the individual environment is displayed is decreased based on an increase in the number of application user interfaces displayed concurrently with the individual environment. In some embodiments, if at least one application user interface is displayed concurrently with the individual environment, the individual environment is displayed at a lower level of detail. In some embodiments, the level of detail to which the individual environment is displayed is increased based on a decrease in the number of application user interfaces displayed concurrently with the individual environment. In some embodiments, if no application user interfaces are displayed concurrently with the individual environment, the individual environment is displayed at a higher level of detail. In some embodiments, the level of detail used to display individual environments corresponds to the number of animations displayed in the individual environment, the type of animations displayed in the individual environment, the resolution associated with the animations displayed in the individual environment, and / or the frame rate associated with the animations displayed in the individual environment. In some embodiments, the frame rate includes how often frames of an image or video (e.g., animation) are displayed in the individual environment.In some embodiments, the level of detail is selected based on the resource usage associated with displaying a number of application user interfaces along with their individual environments, and / or whether the number of application user interfaces includes any active application user interfaces (e.g., any application user interfaces currently in use by the user and / or actively consuming resources). Changing the level of detail at which individual environments are displayed according to the number of application user interfaces displayed simultaneously with the individual environments ensures efficient consumption of computing resources by the computer system without requiring user input to do so (e.g., reducing the level of detail when more application user interfaces are displayed to reduce computing resource consumption), thereby improving user-device interaction.

[0237] In some embodiments, displaying a separate environment at a separate level of detail (804a) includes displaying a separate environment at a first level of detail, such as the level of detail in Figure 7C, while simultaneously displaying a first set of one or more application user interfaces (804b), according to the determination that a first set of one or more application user interfaces, such as the application user interface 726a in Figure 7C, is to be displayed simultaneously with the separate environment. In some embodiments, the separate environment is displayed at a first level of detail based on resource usage associated with displaying the first set of one or more application user interfaces together with the separate environment, and / or whether each application user interface of the first set of one or more application user interfaces is active. In some embodiments, after determining the first level of detail to display the separate environment, the computer system displays the separate environment at a transition level of detail for a threshold time amount (e.g., 0.1, 1, 2, 5, or 10 seconds) before displaying the separate environment at the first level of detail. For example, before determining the first level of detail, the separate environment is optionally displayed at a second level of detail, as described below. Thus, the transition level of detail optionally includes the characteristics of both the second and first levels of detail.

[0238] In some embodiments, displaying a separate environment at a separate level of detail (804a) includes displaying the separate environment at a second level of detail, such as the level of detail in Figure 7D, simultaneously with a second set of application user interfaces, such as application user interface 726c in Figure 7D, which is different from a first set of application user interfaces, according to the determination that such a second set of application user interfaces is to be displayed simultaneously with the separate environment, the second level of detail being different from the first level of detail (804c). In some embodiments, the second level of detail is higher than the first level of detail if the second set of application user interfaces includes fewer application user interfaces and / or corresponds to a smaller amount of resource usage. For example, if the second level of detail is higher than the first level of detail, the separate environment includes an increased number of active animations of ambient elements (e.g., virtual sky, water, rain, fog, grass, plants, and / or animals), an increased resolution (e.g., pixel density) of the ambient elements, and / or an increased resolution (e.g., pixel density) of the application user interfaces. In some embodiments, if the application user interface is not displayed simultaneously with the individual environment, the individual environment is displayed at a higher level of detail. In some embodiments, if a second set of one or more application user interfaces includes more application user interfaces and / or corresponds to higher resource usage, the second level of detail is lower than the first level of detail. For example, if the second level of detail is lower than the first level of detail, the individual environment includes a reduction in the number of active animations of ambient elements (e.g., virtual sky, water, rain, fog, grass, plants, and / or animals), a reduction in the resolution (e.g., pixel density) of the ambient elements, and / or a reduction in the resolution (e.g., pixel density) of the application user interface.In some embodiments, the computer system automatically displays (e.g., without user input) a separate environment having a level of detail (e.g., a first level of detail or a second level of detail) based on a set of application user interfaces (e.g., a first set of one or more application user interfaces or a second set of one or more application user interfaces). In some embodiments, if at least one application user interface is displayed simultaneously with the separate environment, the separate environment is displayed at a lower level of detail. In some embodiments, if the set of application user interfaces displayed simultaneously with the separate environment changes, the computer system automatically changes the level of detail of the separate environment (e.g., without user input). In some embodiments, the computer system detects user input that triggers a change in the set of application user interfaces. In some embodiments, the computer system is configured to dynamically switch between a first level of detail and a second level of detail when displaying the separate environment and application user interfaces based on resource usage associated with displaying application user interfaces and / or ambient elements. In some embodiments, after determining the second level of detail for displaying the separate environment, the computer system displays the separate environment at a transitional level of detail for a threshold time amount (e.g., 0.1, 1, 2, 5, or 10 seconds) before displaying the separate environment at the second level of detail. Displaying individual environments at a level of detail according to the type, number, and / or other characteristics of application user interfaces displayed simultaneously ensures efficient consumption of computing resources by the computer system without requiring user input to do so (for example, reducing the level of detail when more application user interfaces are displayed to reduce computing resource consumption), thereby improving user-device interaction.

[0239] In some embodiments, displaying individual environments via a display generation component includes displaying a three-dimensional virtual environment, such as the three-dimensional virtual environment 702 in Figures 7A and 7A1, via a display generation component (806). In some embodiments, individual environments are simulated three-dimensional environments and / or three-dimensional environments displayed within a three-dimensional environment, optionally in place of a representation of a physical environment (e.g., fully immersive), or optionally simultaneously with a representation of a physical environment (e.g., partially immersive). Some examples of three-dimensional virtual environments include lake environments, mountain environments, sunset scenes, sunrise scenes, nighttime environments, grassland environments, and / or concert scenes. In some embodiments, the three-dimensional virtual environment is based on an actual physical location, such as a museum and / or aquarium. In some embodiments, the three-dimensional virtual environment is a location designed by an artist. Thus, displaying a virtual environment within a three-dimensional environment and / or as a three-dimensional environment optionally provides the user with a virtual experience as if they were physically located within the virtual environment. In some embodiments, the three-dimensional virtual environment has one or more characteristics of the virtual environment described with reference to Methods 1000, 1200, 1400, 1600, 1800, and / or 2000. By displaying individual environments as three-dimensional virtual environments and displaying virtual elements and applications as well as application user interfaces simultaneously with the three-dimensional virtual environment, flexibility in using the computer system is increased and user-device interaction is improved.

[0240] In some embodiments, upon detecting a change in the number of application user interfaces displayed simultaneously with an individual environment, the computer system increases the level of detail at which the individual environment is displayed, such as the level of detail at which the individual environment is displayed in Figure 7C, in accordance with the determination that the number of application user interfaces displayed simultaneously with the individual environment has decreased, such as the decrease in the number of application user interfaces in Figure 7C (808) (for example, increasing the frame rate of the individual environment, the pixel density of the individual environment, the number of animations displayed in the individual environment, the frame rate of the animations displayed in the individual environment, the number of ambient elements displayed in the individual environment, and / or the pixel density of ambient elements displayed in the individual environment). In some embodiments, a smaller number of application user interfaces displayed simultaneously with an individual environment corresponds to a smaller amount of resources used by the computer system to display those application user interfaces. Therefore, a reduction in resource usage may allow the computer system to selectively increase the level of detail at which the individual environment is displayed. Increasing the level of detail at which the individual environment is displayed in response to a reduction in the number of application user interfaces displayed with the individual environment ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0241] In some embodiments, in response to detecting a change in the number of application user interfaces displayed simultaneously with the individual environment, the number of application user interfaces displayed simultaneously with the individual environment is reduced to zero, such as the zero application user interfaces shown in Figure 7B (810). In some embodiments, if no application user interfaces are displayed simultaneously with the individual environment, the individual environment is displayed at the highest level of detail. Increasing the level of detail at which the individual environment is displayed in response to the absence of application user interfaces displayed with the individual environment ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0242] In some embodiments, upon detecting a change in the number of application user interfaces displayed simultaneously with an individual environment, the computer system reduces the level of detail of the individual environment, such as the reduced level of detail of the individual environment in Figure 7C (812) (for example, reducing the frame rate of the individual environment, the pixel density of the individual environment, the number of animations displayed in the individual environment, the frame rate of the animations displayed in the individual environment, the number of ambient elements displayed in the individual environment, and / or the pixel density of ambient elements displayed in the individual environment). In some embodiments, a larger number of application user interfaces displayed simultaneously with an individual environment corresponds to a larger amount of resources used by the computer system to display those application user interfaces. Therefore, an increase in resource usage may cause the computer system to selectively reduce the level of detail of the individual environment. Reducing the level of detail of the individual environment in response to an increase in the number of application user interfaces displayed with the individual environment ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0243] In some embodiments, the number of application user interfaces displayed simultaneously with an individual environment increases to one (814), such as displaying application user interface 726a in Figure 7C, in response to the detection of a change in the number of application user interfaces displayed simultaneously with an individual environment. In some embodiments, the application user interface is not displayed simultaneously with the individual environment before detecting a change in the number of application user interfaces (e.g., an increase to one). In some embodiments, when at least one application user interface is displayed simultaneously with an individual environment, the individual environment is displayed with less detail than when no application user interface is displayed. Reducing the level of detail displayed in the individual environment in response to the display of at least one application user interface with the individual environment ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0244] In some embodiments, changing the level of detail at which an individual environment is displayed includes changing the frame rate of each of one or more animations, such as changing the frame rate of the animation from Figure 7B to Figure 7C, as indicated by the curved arrows of the individual environment (e.g., virtual elements such as a virtual car, one or more virtual elements such as a virtual car, one or more ambient elements such as a step, 804, or any virtual object) (816). In some embodiments, decreasing the level of detail at which an individual environment is displayed includes decreasing the frame rate of each of one or more animations. In some embodiments, increasing the level of detail at which an individual environment is displayed includes increasing the frame rate of each of one or more animations. Changing the frame rate of each of one or more animations to increase or decrease the level of detail at which an individual environment is displayed ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0245] In some embodiments, changing the level of detail at which individual environments are displayed includes maintaining the frame rate of at least one active application user interface, such as the properties of application user interface 726a in Figure 7C, at a higher level than the frame rate of the individual environments, according to the determination that at least one active application user interface is displayed simultaneously with the individual environments (818). In some embodiments, the frame rate of an individual environment includes the frame rate of the entire individual environment. In some embodiments, the frame rate of an individual environment includes the frame rate of one or more components within the individual environment (e.g., application user interfaces (one or more), virtual elements (one or more), such as virtual cars, and / or ambient elements (one or more), such as virtual clouds or virtual animals). In some embodiments, the active application user interface corresponds to an application for media content playback, a navigation application, or a health application. In some embodiments, the computer system determines that an application user interface is active based on receiving input from a user interacting with the application user interface (e.g., the application user interface that is the most recent target of user input is the active application user interface). In some embodiments, the computer system determines that an application user interface is active based on the detection of user attention directed towards that application user interface (for example, the application user interface that is the most recent target of the user's attention is the active application user interface). In some embodiments, the frame rate of the active application user interface is maintained at the same frame rate as the individual environment. In some embodiments, multiple active application user interfaces are displayed simultaneously with individual environments.Therefore, each of the multiple active application user interfaces is optionally maintained at a higher frame rate than its individual environment. In some embodiments, if an inactive application user interface is displayed simultaneously with its individual environment, the frame rate of each inactive application user interface is modified (e.g., reduced) compared to the frame rate of its individual environment. Maintaining active application user interfaces at a higher frame rate than its individual environment helps ensure desired interaction with the active application user interface, reduces errors in interaction with the application user interface, and thus improves user-device interaction.

[0246] In some embodiments, changing the level of detail at which an individual environment is displayed includes changing one or more properties (e.g., number, type, duration, pixel density, and / or frame rate) of one or more animations of one or more ambient elements displayed within the individual environment, such as changing the properties of the background in Figure 7C (820). In some embodiments, decreasing the level of detail at which an individual environment is displayed includes reducing the number of ambient elements displayed within the individual environment (e.g., virtual sky or clouds, virtual water, virtual fog, virtual grass, virtual plants, or virtual animals), reducing the number of animations of ambient elements displayed within the individual environment (e.g., movement of virtual clouds, virtual rainfall, movement of virtual animals, or virtual sunrise or sunset), reducing the pixel density of each ambient element displayed within the individual environment, and / or reducing the frame rate of each animation of ambient elements displayed within the individual environment. In some embodiments, increasing the level of detail displayed in an individual environment includes increasing the number of ambient elements displayed within the individual environment, increasing the number of animations of ambient elements displayed within the individual environment, increasing the pixel density of each ambient element displayed within the individual environment, and / or increasing the frame rate of each animation of ambient elements displayed within the individual environment. Changing the animation of each ambient element within an individual environment to increase or decrease the level of detail displayed in the individual environment ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0247] In some embodiments, changing one or more properties (e.g., number, type, duration, pixel density, and / or frame rate) of one or more animations for one or more ambient elements displayed in a separate environment includes changing (and / or applying) distortion effects applied to one or more planes in the separate environment, such as distortion effects applied to virtual water represented by ambient element 750 in Figure 7C (822). In some embodiments, ambient elements having planes, such as simulated water, are distorted based on changes in ambient elements or other content displayed in the separate environment. For example, changing the animation of simulated wind in a separate environment optionally changes the distortion (e.g., ripple or other texture movement effects) of simulated materials displayed in the separate environment (e.g., water, sand, snow, fog, grass, leaves, etc.) (e.g., increased ripple effect in simulated water with increased simulated wind, or decreased ripple effect in simulated water with decreased simulated wind, movement effect of grass or leaves moving in the wind, movement of particles such as snow or sand with the wind, and / or movement of fog or clouds). In some embodiments, the changes in distortion effects applied to the animation of ambient elements are based on the respective positions of the application displayed within the individual environment (e.g., whether the application is displayed at a viewing location within the individual environment). In some embodiments, simulated light projected by content such as media content within the individual environment is virtually reflected from a plane corresponding to simulated water. Therefore, changing the distortion effect optionally involves changing the reflection of simulated light projected onto a plane corresponding to a simulated material (e.g., water, sand, snow, fog, grass, leaves, etc.). In some embodiments, the changes in distortion effects are not applied to the plane unless the media content is displayed within the individual environment. Therefore, changing the distortion effect optionally requires a reduction in processing power for the distortion effect when the content corresponding to the plane is displayed.In some embodiments, the distortion effect does not apply to a plane unless the content corresponding to the plane is displayed at a specific location within the individual environment, such as a position within the individual environment where simulated light is reflected from the plane (e.g., a location within the individual environment that is a fixed or otherwise designated location within the individual environment to which media content can be docked for viewing). In some embodiments, if the content is displayed within the individual environment at different locations within the individual environment, the distortion effect is not optionally applied. In some embodiments, the location of the distortion effect on the plane changes based on the relative location of the content with respect to the plane being distorted (e.g., different locations of the distortion effect for different relative locations of media content with respect to the plane). In some embodiments, the distortion effect applied to the animation of ambient elements is independent of the number of applications or application user interfaces displayed within the individual environment. Applying distortion effects to the animation of ambient elements prevents unintended copying of copyrighted content when displaying copyrighted content in an individual environment, reduces the computational power required to display reflections from surfaces, and thus ensures efficient consumption of computational resources by the computer system.

[0248] In some embodiments, modifying the distortion effect applied to one or more planes includes modifying the animation of simulated water ripple effects, such as the virtual water ripple animation represented by the ambient element 750 in Figure 7C, in a separate environment (824). In some embodiments, the distortion effect applied to the animation of simulated water includes one or more ripple effects that appear to decompose the reflections of the simulated water, and therefore the reflections of content from the surface of the simulated water. In some embodiments, the distortion effect includes removing raindrop ripples from the simulated water so that the simulated water appears less realistic (e.g., optionally while maintaining the underwater ripple effect caused by simulated waves or simulated wind). Modifying the animation of ripple effects on simulated water prevents unintended copying of copyrighted content when displaying copyrighted content in a separate environment, reduces the computational power required to display reflections from surfaces, and therefore ensures efficient consumption of computational resources by the computer system.

[0249] In some embodiments, changing one or more properties of one or more animations includes stopping the animation of one or more ambient elements, such as stopping the animation of ambient elements 740 and 744 in Figure 7D, upon determination that the number of application user interfaces displayed simultaneously with the individual environment exceeds a threshold number of application user interfaces (e.g., 1, 2, 3, 4, 5, 10, 20, 50, or 100 application user interfaces), such as threshold 722 in Figure 7D (826). In some embodiments, the computer system stops animating some or all ambient elements if the number of displayed application user interfaces exceeds a threshold number of application user interfaces. In some embodiments, the computer system stops animating some ambient elements if the number of displayed application user interfaces exceeds a threshold number of application user interfaces. In some embodiments, if the number of displayed application user interfaces exceeds a threshold number of application user interfaces, the computer system reduces the frame rate and / or pixel density of one or more application user interfaces, reduces the number and / or pixel density of ambient elements, reduces the frame rate of ambient element animations, and / or reduces the frame rate and / or pixel density of individual environments. In some embodiments, ceasing to animate ambient elements includes freezing the animation of ambient elements while continuing to display them as static versions. In some embodiments, ceasing to animate ambient elements includes ceasing the display of some or all of the ambient elements and their respective animations together.Stopping animations within individual environments when the number of displayed application user interfaces exceeds a threshold ensures efficient use of computing resources by the computer system (e.g., by reducing animations to lower computing resource consumption) without requiring user input to do so, thereby improving user-device interaction.

[0250] In some embodiments, after ceasing to animate one or more ambient elements (828a), the computer system detects that the number of application user interfaces displayed simultaneously with the individual environments has decreased to a threshold number of application user interfaces (e.g., 1, 3, 5, 10, 20, 50, or 100 application user interfaces), such as the number of displayed application user interfaces within threshold 722 in Figure 7C (828b).

[0251] In some embodiments, upon detecting that the number of application user interfaces displayed simultaneously with individual environments has decreased to a threshold number of application user interfaces, the computer system resumes the animation of one or more ambient elements, such as resuming the animation of ambient elements 738, 742, 744, 748, and 750 from Figure 7D to Figure 7C (828c). In some embodiments, the computer system resumes the animation of some or all ambient elements when the number of displayed application user interfaces has decreased to a threshold number of application user interfaces. In some embodiments, the computer system resumes the animation of some ambient elements (e.g., those that were aborted) when the number of displayed application user interfaces has decreased to a threshold number of application user interfaces. In some embodiments, the computer system increases the frame rate and / or pixel density of one or more application user interfaces, increases the number and / or pixel density of ambient elements, increases the frame rate of the animation of ambient elements, and / or increases the frame rate and / or pixel density of individual environments when the number of displayed application user interfaces has decreased to a threshold number of application user interfaces. In some embodiments, resuming the animation of an ambient element includes unfreezing (e.g., resuming) the animation of the ambient element if the ambient element was previously frozen as described with respect to step 826. In some embodiments, resuming the animation of an ambient element includes redisplaying some or all of the ambient element and animating some or all of the ambient element if the display of some or all of the ambient element and their respective animations was all stopped together as described with respect to step 826.Resuming animation in individual environments when the number of application user interfaces is reduced to a threshold number ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0252] In some embodiments, changing one or more properties of one or more animations includes discontinuing the animation of at least one ambient element (e.g., virtual sky or virtual water) among one or more ambient elements, such as discontinuing the animation of the virtual sky represented by ambient element 738 in Figure 7D, in response to detecting that the number of application user interfaces displayed simultaneously with individual environments exceeds a threshold number of application user interfaces (e.g., 1, 3, 5, 10, 20, 50, or 100 application user interfaces), while maintaining the animation of at least one other ambient element (e.g., virtual grass, virtual plants, or virtual animals) among one or more ambient elements, such as maintaining the animation of the virtual mountain when displayed in Figure 7D (830). In some embodiments, discontinuing the animation of at least one ambient element includes freezing the animation of at least one ambient element, or discontinuing the display of at least one ambient element and individual animations together, as described with respect to step 826. Stopping certain animations within individual environments when the number of displayed application user interfaces exceeds a threshold ensures efficient use of computing resources by the computer system (e.g., by reducing unnecessary animations to lower computing resource consumption) without requiring user input to do so, thereby improving user-device interaction.

[0253] In some embodiments, maintaining the animation of at least one of the other ambient elements includes maintaining the animation of simulated water, such as the virtual water represented by ambient element 750 in Figure 7C (e.g., water reflection animation), the animation of simulated sky, such as the virtual sky represented by ambient element 738 in Figure 7C (e.g., rain, snow, wind, or fog animation), or both (832). Maintaining several animations within individual environments, even when the number of application user interfaces exceeds a threshold number, does not require user input to do so and ensures efficient consumption of computing resources by the computer system while maintaining consistency in the presentation of the environment, thereby improving user-device interaction.

[0254] In some embodiments, changing the level of detail displayed for individual environments includes changing the level of detail corresponding to a simulated sky within an individual environment having a flow map, such as the simulated sky shown via side view 745 in Figure 7B (834). In some embodiments, the flow map includes one or more layers corresponding to the simulated sky, each layer representing a subset of the simulated sky or a visualization of common features, which together form the entire simulated sky. In some embodiments, the flow map shows the movement of ambient elements and their respective animations associated with the change in the level of detail for each layer corresponding to the simulated sky. In some embodiments, changing the level of detail corresponding to the simulated sky includes changing the pixel density of the simulated sky, the number of ambient elements displayed (e.g., virtual clouds), and the animation of ambient elements (e.g., virtual cloud shadows). Flow maps help to precisely change the level of detail when generating a simulated sky in individual environments, and thus improve user-device interaction.

[0255] In some embodiments, the flow map includes two or more layers corresponding to the simulated sky (e.g., a virtual sky layer, one or more virtual cloud layers, and one or more corresponding virtual cloud shadow layers), such as the layers corresponding to ambient elements 738, 740, and 746 in Figure 7B, and changing the level of detail corresponding to the simulated sky includes changing the level of detail corresponding to one or more layers of the flow map (836). In some embodiments, the level of detail of each layer corresponding to the simulated sky is changed simultaneously. In some embodiments, the level of detail of each layer corresponding to the simulated sky is changed one at a time. In some embodiments, changing the level of detail of the virtual sky layer includes changing the pixel density of the virtual sky. In some embodiments, changing the level of detail of the virtual cloud layer includes changing the number of virtual clouds displayed (e.g., animated virtual clouds), the pixel density, and / or the frame rate. In some embodiments, changing the level of detail of the virtual cloud shadow layer includes changing the number of virtual cloud shadows displayed (e.g., animated virtual cloud shadows), the pixel density, and / or the frame rate. In some embodiments, each layer corresponding to the simulated sky is changed by the same level of detail. In some embodiments, each layer corresponding to the simulated sky is modified to a different level of detail. In some embodiments, a set of layers corresponding to the simulated sky is modified by the same level of detail, while another set of layers corresponding to the simulated sky is modified by a different level of detail. In some embodiments, a set of layers corresponding to the simulated sky is modified by the same or different levels of detail, while another set of layers corresponding to the simulated sky remains unchanged. Modifying the level of detail of the simulated sky by modifying the level of detail of each layer of the simulated sky generates an accurate simulated sky with a higher level of detail, is less resource-intensive, but provides greater flexibility for adjusting the sky to be simulated, and thus improves user-device interaction.

[0256] In some embodiments, changing the level of detail at which individual environments are displayed includes changing the pixel density of an individual environment, such as changing the pixel density of an individual environment 704 in Figures 7A and 7A1 (838). In some embodiments, the pixel density of an individual environment includes the pixel density of the entire individual environment. In some embodiments, the pixel density of an individual environment includes the pixel density of one or more components within the individual environment (e.g., one or more application user interfaces, one or more virtual elements such as virtual cars, and / or one or more ambient elements such as virtual clouds or virtual animals). In some embodiments, changing the level of detail includes changing the pixel density of each virtual element such as a virtual car, one or more ambient elements, or one or more virtual objects displayed within an individual environment, and / or changing the pixel density of each application user interface displayed simultaneously with each environment. In some embodiments, changing the level of detail includes reducing the level of detail at which an individual environment is displayed by reducing the pixel density of each virtual element(s), such as individual environments, ambient elements(s), and / or application user interfaces(s). In some embodiments, the pixel density of an individual environment is reduced from 40 pixels / degree (ppd) to 20 pixels / degree (ppd). In some embodiments, changing the level of detail includes increasing the level of detail at which an individual environment is displayed by increasing the pixel density of each virtual element(s), such as individual environments, ambient elements(s), and / or application user interfaces(s). In some embodiments, the pixel density of an individual environment is increased from 20 pixels / degree (ppd) to 40 pixels / degree (ppd).Changing the pixel density of individual environments to increase or decrease the level of detail displayed in those environments ensures efficient consumption of computing resources by the computer system without requiring user input to do so, thereby improving user-device interaction.

[0257] In some embodiments, changing the level of detail at which individual environments are displayed includes maintaining the pixel density of at least one active application user interface at a higher level than the pixel density of the individual environment, such as the properties of application user interface 726a being higher than the properties of application user interface 726b in Figures 7A and 7A1, according to the determination that at least one active application user interface is displayed simultaneously with the individual environment (840). In some embodiments, the pixel density of the individual environment includes the pixel density of the entire individual environment. In some embodiments, the pixel density of the individual environment includes the pixel density of one or more components within the individual environment (e.g., one or more application user interfaces, one or more virtual elements such as virtual cars, and / or one or more ambient elements such as virtual clouds or virtual animals). In some embodiments, the pixel density of the active application user interface is maintained at the same level of pixel density as the individual environment. In some embodiments, multiple active application user interfaces are displayed simultaneously with individual environments. Therefore, each of the multiple active application user interfaces is optionally maintained at a higher pixel density than the individual environment. In some embodiments, when an inactive application user interface is displayed simultaneously with a separate environment, the pixel density of each inactive application user interface is modified (e.g., reduced) compared to the pixel density of the separate environment. Maintaining an active application user interface at a higher pixel density than the separate environment helps ensure desired interaction with the active application user interface, reduces errors in interaction with the application user interface, and therefore improves user-device interaction.

[0258] In some embodiments, changing (e.g., reducing) the level of detail in which an individual environment is displayed includes discontinuing the display of one or more ambient elements based on simulated light, such as ambient elements 746 and 748 in Figure 7B (e.g., virtual shadows such as virtual cloud shadows or virtual tree shadows) in the individual environment (842). In some embodiments, reducing the level of detail includes reducing the number and / or pixel density of ambient elements based on simulated light (e.g., within the individual environment). In some embodiments, the simulated light corresponds to natural light and / or artificial light (e.g., lamp light from the physical environment) (e.g., sunrise, afternoon, or sunset from the physical environment). Thus, virtual shadows based on natural light and / or artificial light (e.g., afternoon shadows or shadows based on lamp light) are optionally displayed in the individual environment. In some embodiments, reducing the level of detail includes reducing the number, pixel density, and / or frame rate of animations of ambient elements based on simulated light. Changing the level of detail that displays individual environments by discontinuing the display of ambient elements based on simulated lighting ensures efficient consumption of computing resources by the computer system (e.g., to reduce computing resource consumption) without requiring user input to do so, thereby improving user-device interaction.

[0259] In some embodiments, changing the level of detail in which an individual environment is displayed includes discontinuing the display of at least one ambient element based on simulated light (e.g., virtual cloud shadows), such as ambient element 746 from Figure 7B, which is not displayed in Figure 7C, while maintaining the display of at least another ambient element based on simulated light (e.g., virtual tree shadows), such as ambient element 748 from Figure 7C (844). In some embodiments, changing the level of detail includes decreasing the number and / or pixel density of virtual cloud shadows while maintaining the number and / or pixel density of virtual tree shadows. In some embodiments, changing the level of detail includes decreasing the number, pixel density, and / or frame rate of virtual cloud shadows, while maintaining the level of detail includes decreasing the number, pixel density, and / or frame rate of virtual tree shadows. By discontinuing the display of some ambient elements based on simulated light while maintaining the display of other ambient elements based on simulated light, the level of detail displayed for individual environments can be altered, ensuring efficient use of computing resources by the computer system without requiring user input to do so (for example, reducing computing resource consumption by discontinuing the display of unnecessary ambient elements based on simulated light), thereby improving user-device interaction.

[0260] In some embodiments, changing the level of detail at which individual environments are displayed includes changing the level of detail based on the amount of processing power required by the number of application user interfaces displayed simultaneously with the individual environment, such as the processing power required by the application user interfaces 726a and 726b displayed simultaneously with the individual environment 704 in Figures 7A and 7A1 (846). In some embodiments, the level of detail is changed based on the amount of processing power required by the individual environment, including virtual elements (single or multiple), such as ambient elements (single or multiple), animations of virtual elements (single or multiple), and / or application user interfaces (single or multiple) displayed simultaneously with the individual environment. In some embodiments, if the amount of power required by the application user interfaces displayed simultaneously with the individual environment is greater than a threshold (for example, if the application user interfaces consume more than 30%, 50%, 70%, or 90% of the electronic device's battery), the level of detail at which the individual environment is displayed is reduced. For example, a computer system can reduce the level of detail by optionally reducing the total number of application user interfaces (one or more) displayed simultaneously with individual environments, and / or by reducing the number of power-intensive application user interfaces (one or more) displayed simultaneously with individual environments, thereby reducing the processing power required by the application user interfaces (one or more) to a threshold. In some embodiments, a computer system can reduce the level of detail by reducing the total number of active application user interfaces (one or more) displayed simultaneously with individual environments and consuming processing power, thereby reducing the processing power required by the application user interfaces (one or more) to a threshold.In some embodiments, a computer system can reduce th...

Claims

1. It is a method, In a computer system that communicates with a display generation component and one or more input devices, The display generation component detects changes in the number of application user interfaces displayed simultaneously with the individual environment while the individual environment is being displayed, A method comprising: detecting a change in the number of application user interfaces displayed simultaneously with the individual environments, and changing the level of detail at which the individual environments are displayed.

2. Displaying the aforementioned individual environments at a detailed level is possible. In accordance with the determination that one or more application user interfaces are to be displayed simultaneously with the individual environments, the individual environments are to be displayed at a first level of detail while simultaneously displaying the first set of application user interfaces. The method according to claim 1, comprising displaying the individual environment at a second level of detail simultaneously with the second set of application user interfaces, which is different from the first set of application user interfaces, according to a determination that the second set of application user interfaces is to be displayed simultaneously with the individual environment, wherein the second level of detail is different from the first level of detail.

3. The method according to claim 1 or 2, wherein displaying the individual environments via the display generation component includes displaying a three-dimensional virtual environment via the display generation component.

4. The method according to any one of claims 1 to 3, further comprising detecting a change in the number of application user interfaces displayed simultaneously with the individual environment, and, in accordance with a determination that the number of application user interfaces displayed simultaneously with the individual environment has decreased, increasing the level of detail at which the individual environment is displayed.

5. The method according to claim 4, wherein the number of application user interfaces displayed simultaneously with the individual environments is reduced to zero in response to the detection of a change in the number of application user interfaces displayed simultaneously with the individual environments.

6. The method according to any one of claims 1 to 5, further comprising detecting a change in the number of application user interfaces displayed simultaneously with the individual environment, and, in accordance with a determination that the number of application user interfaces displayed simultaneously with the individual environment has increased, reducing the level of detail at which the individual environment is displayed.

7. The method according to claim 6, wherein the number of application user interfaces displayed simultaneously with the individual environments is increased to 1 in response to the detection of the change in the number of application user interfaces displayed simultaneously with the individual environments.

8. The method according to any one of claims 1 to 7, wherein changing the level of detail at which the individual environment is displayed includes changing the frame rate of each of the one or more animations displayed in the individual environment.

9. Changing the level of detail at which the aforementioned individual environments are displayed The method according to claim 8, comprising maintaining the frame rate of the at least one active application user interface at a higher level than the frame rate of the individual environments, in accordance with the determination that at least one active application user interface is being displayed simultaneously with the individual environments.

10. The method according to any one of claims 1 to 9, wherein changing the level of detail at which the individual environment is displayed includes changing one or more properties of one or more animations of one or more ambient elements displayed within the individual environment.

11. The method according to claim 10, wherein modifying one or more properties of one or more animations for one or more ambient elements displayed within the individual environment includes modifying a distortion effect applied to one or more planes within the individual environment.

12. The method according to claim 11, wherein modifying the distortion effect applied to one or more planes includes modifying the simulated water ripple effect animation in the individual environments.

13. Changing one or more characteristics of the one or more animations The method according to any one of claims 10 to 12, comprising stopping the animation of one or more ambient elements in accordance with the determination that the number of application user interfaces displayed simultaneously with the individual environments exceeds a threshold number of application user interfaces.

14. After stopping the animation of one or more ambient elements, To detect that the number of application user interfaces displayed simultaneously with the individual environments has been reduced to within the threshold number of application user interfaces, The method according to claim 13, further comprising: resuming the animation of one or more ambient elements in response to detecting that the number of application user interfaces displayed simultaneously with the individual environments has been reduced to within the threshold number of application user interfaces.

15. Changing one or more characteristics of the one or more animations The method according to any one of claims 10 to 14, comprising detecting that the number of application user interfaces displayed simultaneously with the individual environments exceeds a threshold number of application user interfaces, discontinuing the animation of at least one of the one or more ambient elements while maintaining the animation of at least one of the other ambient elements.

16. The method according to claim 15, wherein maintaining the at least one animation for the other ambient elements includes maintaining the animation of simulated water, the animation of simulated sky, or both.

17. The method according to any one of claims 10 to 16, wherein changing the level of detail at which the individual environments are displayed includes changing the level of detail at which the simulated emptiness in the individual environments is displayed using a flow map.

18. The method according to claim 17, wherein the flow map includes two or more layers corresponding to the simulated void, and changing the level of detail corresponding to the simulated void includes changing the level of detail corresponding to one or more layers of the flow map.

19. The method according to any one of claims 1 to 18, wherein changing the level of detail at which the individual environment is displayed includes changing the pixel density of the individual environment.

20. Changing the level of detail at which the aforementioned individual environments are displayed The method according to claim 19, comprising maintaining the pixel density of the at least one active application user interface at a higher level than the pixel density of the individual environment, in accordance with the determination that at least one active application user interface is displayed simultaneously with the individual environment.

21. The method according to any one of claims 1 to 20, wherein changing the level of detail at which the individual environment is displayed includes discontinuing the display of one or more ambient elements based on simulated light in the individual environment.

22. The method according to claim 21, wherein changing the level of detail at which the individual environment is displayed includes discontinuing the display of at least one ambient element based on simulated light in the individual environment, while maintaining the display of at least another ambient element based on simulated light in the individual environment.

23. The method according to any one of claims 1 to 22, wherein changing the level of detail at which the individual environments are displayed includes changing the level of detail based on the amount of processing power required by the number of application user interfaces displayed simultaneously with the individual environments.

24. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, The system is configured to include, and one or more programs are stored in the memory and executed by one or more processors, Through the display generation component, while an individual environment is being displayed, changes in the number of application user interfaces displayed simultaneously with the individual environment are detected. A computer system that includes instructions to change the level of detail at which an individual environment is displayed in response to detecting a change in the number of application user interfaces that are displayed simultaneously with the individual environment.

25. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system The display generation component detects changes in the number of application user interfaces displayed simultaneously with the individual environment while the individual environment is being displayed, A non-temporary computer-readable storage medium that causes a method to be performed, which includes detecting a change in the number of application user interfaces displayed simultaneously with the individual environments, and changing the level of detail at which the individual environments are displayed.

26. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and Means for detecting changes in the number of application user interfaces displayed simultaneously with the individual environment while an individual environment is being displayed via the display generation component, A computer system comprising means for changing the level of detail at which an individual environment is displayed in response to detecting a change in the number of application user interfaces displayed simultaneously with the individual environment.

27. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 1 to 23 and 183 to 187.

28. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 1 to 23 and 183 to 187.

29. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for carrying out the method described in any one of claims 1 to 23 and 183 to 187.

30. It is a method, In a computer system that communicates with a display generation component and one or more input devices, While at least a portion of the physical environment of the user of the computer system is visible through the display generation component, the system receives a first input via one or more input devices corresponding to a request to apply a first visual effect to the representation of the physical environment. A method comprising, in response to receiving the first input, displaying the representation of the physical environment via the display generation component, wherein the display is In accordance with the determination that at least a portion of the physical environment has a first visual appearance, the first visual adjustment is applied to generate the visible representation of the physical environment via the display generation component. A method comprising: determining that at least a portion of the physical environment has a second visual appearance different from the first visual appearance, and applying a second visual adjustment different from the first visual adjustment to generate a representation of the physical environment that is visible via the display generating component.

31. The method according to claim 30, wherein applying the first visual adjustment, the second visual adjustment, or both, includes applying color neutralization to produce the representation of the physical environment.

32. The method according to claim 30 or 31, wherein the first input corresponding to the request to apply the first visual effect to the representation of the physical environment includes an input corresponding to the request to apply a first color filter to the representation of the physical environment.

33. Receiving a second user input via one or more input devices corresponding to a request to apply a second visual effect, which includes a second color filter different from the first color filter, to the representation of the physical environment, In response to receiving the second user input, the display generation component further includes displaying the representation of the physical environment, In accordance with the determination that at least a portion of the physical environment has a third visual appearance, a third visual adjustment is applied to generate the visible representation of the physical environment through the display generation component to which the second color filter is applied. The method of claim 32, comprising: applying a fourth visual adjustment different from the third visual adjustment to produce a representation of the physical environment that is visible through the display generating component to which the second color filter is applied, in accordance with the determination that at least a portion of the physical environment has a fourth visual appearance different from the third visual appearance.

34. The method according to any one of claims 30 to 33, wherein the first visual effect includes at least a portion of the virtual environment.

35. In accordance with the determination that at least a portion of the physical environment has the first visual appearance, while applying the first visual adjustment to generate the representation of the physical environment, the detection of a change in the appearance of at least a portion of the physical environment from the first visual appearance to the second visual appearance, wherein the change in the appearance of at least a portion of the physical environment from the first visual appearance to the second visual appearance includes a change in ambient light within the physical environment, The method according to any one of claims 30 to 34, further comprising: detecting the change in the appearance of at least a portion of the physical environment from the first visual appearance to the second visual appearance, and applying a second visual adjustment different from the first visual adjustment to generate a visible representation of the physical environment via the display generating component.

36. The method according to claim 35, wherein the change in ambient light in the physical environment includes a change in natural light in the physical environment.

37. The method according to claim 35, wherein the change in ambient light includes a change in artificial light in the physical environment.

38. The first visual effect includes at least a part of the virtual environment, and the method is The method according to any one of claims 30 to 37, further comprising replacing at least a portion of the representation of the physical environment with at least a portion of the virtual environment in response to receiving the first input.

39. The method according to any one of claims 30 to 38, wherein applying the first visual adjustment, the second visual adjustment, or both thereof, includes applying an increased automatic white balance adjustment to produce the representation of the physical environment.

40. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, The system is configured to include, and one or more programs are stored in the memory and executed by one or more processors, While at least a portion of the physical environment of the user of the computer system is visible through the display generation component, a first input is received via one or more input devices corresponding to a request to apply a first visual effect to the representation of the physical environment. A computer system that, upon receiving the first input, includes an instruction to display the representation of the physical environment via the display generation component, wherein the display is In accordance with the determination that at least a portion of the physical environment has a first visual appearance, the first visual adjustment is applied to generate the visible representation of the physical environment via the display generation component. A computer system comprising: applying a second visual adjustment different from the first visual adjustment to generate a visible representation of the physical environment via the display generating component, in accordance with a determination that at least a portion of the physical environment has a second visual appearance different from the first visual appearance.

41. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system While at least a portion of the physical environment of the user of the computer system is visible through the display generation component, the system receives a first input via one or more input devices corresponding to a request to apply a first visual effect to the representation of the physical environment. A non-temporary computer-readable storage medium that causes a method to be performed, which includes, upon receiving the first input, displaying the representation of the physical environment via the display generation component, In accordance with the determination that at least a portion of the physical environment has a first visual appearance, the first visual adjustment is applied to generate the visible representation of the physical environment via the display generation component. A non-temporary computer-readable storage medium, comprising: applying a second visual adjustment different from the first visual adjustment to generate a representation of the physical environment that is visible via the display generating component, in accordance with the determination that at least a portion of the physical environment has a second visual appearance different from the first visual appearance.

42. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and Means for receiving a first input via one or more input devices that corresponds to a request to apply a first visual effect to a representation of the physical environment, while at least a portion of the physical environment of the user of the computer system is visible through the display generation component, A computer system comprising means for displaying the representation of the physical environment via the display generation component in response to receiving the first input, wherein the display is In accordance with the determination that at least a portion of the physical environment has a first visual appearance, the first visual adjustment is applied to generate the visible representation of the physical environment via the display generation component. A computer system comprising: applying a second visual adjustment different from the first visual adjustment to generate a visible representation of the physical environment via the display generating component, in accordance with a determination that at least a portion of the physical environment has a second visual appearance different from the first visual appearance.

43. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 30 to 39.

44. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 30 to 39.

45. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for carrying out the method described in any one of claims 30 to 39.

46. It is a method, In a computer system that communicates with a display generation component and one or more input devices, The display generation component enables the display of a first user interface object in the environment, wherein the first user interface object is selectable for displaying first content, and the first user interface object has a first visual appearance. While the first user interface object having the first visual appearance is displayed, the attention of the computer system user directed towards the first user interface object is detected via one or more input devices. In response to detecting the user's attention directed towards the first user interface object, A method comprising displaying the first user interface object in the environment having a second visual appearance different from the first visual appearance via the display generation component, wherein displaying the first user interface object having the second visual appearance includes displaying the first user interface object having a three-dimensional stereoscopic effect corresponding to a plurality of different views of the first content corresponding to the first user interface object, and the first visual appearance of the first user interface object displayed before the user's attention is directed to the first user interface object does not include displaying the first user interface object having a three-dimensional stereoscopic effect.

47. The method according to claim 46, wherein the first content includes a three-dimensional virtual environment.

48. The method according to claim 46 or 47, wherein the first content includes a stereoscopic image.

49. The method according to any one of claims 46 to 48, wherein the first content includes an application.

50. While the first user interface object is displayed in the environment having the first visual appearance, the display generation component enables the display of a second user interface object in the environment, wherein the second user interface object is selectable to display second content, and the second user interface object displays the second user interface object having the first visual appearance. While the first user interface object and the second user interface object having the first visual appearance are being displayed, the user's attention directed towards the second user interface object is detected via one or more input devices. The method according to any one of claims 46 to 49, further comprising detecting the user's attention directed toward the second user interface object, and via the display generation component, displaying the second user interface object in the environment having a second visual appearance while maintaining the display of the first user interface object having a first visual appearance.

51. While the first user interface object is displayed in the environment having the second visual appearance, the display generation component enables the display of a second user interface object in the environment, wherein the second user interface object is selectable to display second content corresponding to the second user interface object, and the second user interface object displays the second user interface object having the first visual appearance. While the first user interface object having the second visual appearance and the second user interface object having the first visual appearance are being displayed, it is detected via one or more input devices that the user's attention moves away from the first user interface object to the second user interface object, The method according to any one of claims 46 to 49, further comprising: detecting that the user's attention has moved away from the first user interface object to the second user interface object, and displaying the second user interface object in the environment having the second visual appearance via the display generating component.

52. The method according to claim 51, further comprising gradually modifying the visual appearance of the first user interface object from the second visual appearance to the first visual appearance in response to the detection that the user's attention moves away from the first user interface object to the second user interface object.

53. The method according to any one of claims 50 to 52, wherein displaying the first user interface object having the second visual appearance includes displaying the first user interface object at an expanded size compared to the first user interface object displayed with the first visual appearance.

54. The method according to claim 53, wherein the size of the first user interface object expands in response to the detection of the user's gaze directed toward the first user interface object.

55. While displaying the first user interface object having the first visual appearance, displaying a plurality of user interface objects within the environment having the first visual appearance, wherein the plurality of user interface objects are located in a first location within the environment and are displayed in a first spatial arrangement relative to the first user interface object; The method according to claim 53 or 54, further comprising detecting the user's attention directed toward the first user interface object, moving the plurality of user interface objects from the first location in the environment to the second location in the environment, wherein the plurality of user interface objects are displayed in a second spatial arrangement relative to the first user interface object in the second location, and the second spatial arrangement relative to the first user interface object occupies a larger area of ​​the three-dimensional environment than the first spatial arrangement relative to the first user interface object.

56. The method according to any one of claims 53 to 55, wherein the transition from displaying the first user interface object having the first visual appearance to displaying the first user interface object having the second visual appearance is a greater extension of the first user interface object in a first dimension than extending the first user interface object in a second dimension different from the first dimension.

57. The method according to any one of claims 53 to 56, wherein displaying the first user interface object having the first visual appearance includes displaying a first portion of the first content, and displaying the first user interface having a second visual appearance includes displaying a second portion of the first content that is larger than the first portion of the first content.

58. The display generation component includes a first display and a second display, Displaying the first user interface object having the first visual appearance is Displaying a first representation of the first content via the first display, This includes displaying a second representation of the first content via the second display, Displaying the first user interface object having the second visual appearance is To display a third representation of the first content via the first display, wherein the third representation of the first content is different from the first representation of the first content in a first manner, The method according to any one of claims 46 to 57, comprising displaying a fourth representation of the first content via the second display, wherein the fourth representation of the first content is different from the second representation of the first content in a second form different from the first form.

59. The method according to claim 58, wherein transitioning from displaying the first user interface object having the first visual appearance to displaying the first user interface object having the second visual appearance includes modifying the third representation of the first content on the first display and the fourth representation of the first content on the second display to become progressively different.

60. The first representation of the first content on the first display and the second representation of the first content on the second display include a first image corresponding to a first perspective of the first content, and display the first user interface object having a second visual appearance, On the first display, the display of the first image is transitioned to a second image corresponding to a second perspective of the first content that is different from the first perspective of the first content, The method according to claim 59, further comprising, on the second display, transitioning the display of the first image to a third image corresponding to a third perspective of the first content that is different from the first perspective of the first content and the second perspective of the first content.

61. The method according to any one of claims 58 to 60, wherein the transition from the first representation of the first content to the third representation of the first content on the first display, and the transition from the second representation of the first content to the fourth representation of the first content on the second display is progressive.

62. While the first user interface object having the second visual appearance is displayed, In accordance with the determination that the user's current viewpoint orientation relative to the first user interface object is a first orientation, the first user interface object having a first amplitude of the three-dimensional stereoscopic effect is displayed, The method according to any one of claims 46 to 61, further comprising displaying the first user interface object having a second amplitude of the three-dimensional stereo effect, based on the determination that the orientation of the user's current viewpoint to the first user interface object is a second orientation different from the first orientation, wherein the second amplitude is different from the first amplitude.

63. The method according to claim 62, wherein the first orientation of the user's current viewpoint to the first user interface object includes a more direct field of view than the second orientation of the user's current viewpoint, and the first amplitude of the three-dimensional stereo effect is greater than the second amplitude of the three-dimensional stereo effect.

64. The method according to claim 62 or 63, wherein displaying the first user interface object having the first amplitude of the three-dimensional stereoscopic effect includes displaying a crossfade between a first representation of the first content and a second representation of the first content that is different from the first representation of the first content.

65. The display generation component includes a first display and a second display, Displaying the first user interface object having the first amplitude of the three-dimensional stereoscopic effect, Displaying, via the first display, a first viewing angle to the first content, wherein the first viewing angle is separated from the first reference viewing angle to the first content by a first amount in a first direction by a first amount, and the first portion of the first representation of the first content is displayed. The second display includes displaying a first portion of a second representation of the first content corresponding to a second viewing angle of the first content, wherein the second viewing angle is separated from a second reference viewing angle of the first content by a first amount in a second direction different from the first direction, in a second direction, via the second display, Displaying the first user interface object having the second amplitude of the three-dimensional stereoscopic effect is Displaying a second portion of the first representation of the first content corresponding to a third viewing angle, which is different from the first viewing angle, via the first display, wherein the third viewing angle separates the first content by a second amount different from the first reference viewing angle in the first direction, The method according to claim 62 or 63, comprising displaying within the first content a second portion of the second representation of the first content corresponding to a fourth viewing angle different from the second viewing angle via the second display, wherein the fourth viewing angle is separated from the second reference viewing angle by a second amount in the second direction.

66. A second user interface object having the second visual appearance, wherein the second user interface object displays a second user interface object that is selectable for displaying second content, While displaying the second user interface object having the second visual appearance, In accordance with the determination that the user's current viewpoint orientation relative to the second user interface object is the first orientation, Displaying a first portion of the first representation of the second content corresponding to a fifth viewing angle of the second content, wherein the fifth viewing angle is separated from the first reference viewing angle of the second content by a third amount different from the first amount in the first direction, via the first display, Displaying, via the second display, a sixth viewing angle to the second content, wherein the sixth viewing angle is separated from the second reference viewing angle to the second content by the third amount in the second direction, and the first portion of the second representation of the second content is displayed. In accordance with the determination that the orientation of the user's current viewpoint relative to the second user interface object is the second orientation, To display on the second content a second portion of the first representation of the second content that corresponds to a seventh viewing angle different from the fifth viewing angle, wherein the seventh viewing angle is separated from the first reference viewing angle by a fourth amount different from the second amount in the first direction, via the first display, The method according to claim 65, further comprising displaying a second portion of the second representation of the second content on the second content via the second display, wherein the eighth viewing angle is separated from the second reference viewing angle by the fourth amount in the second direction.

67. Detecting a change in the user's current viewpoint from a first viewpoint to a second viewpoint, including changing the orientation of the current viewpoint to the first user interface object away from the first orientation while the user's current viewpoint is a first viewpoint where the orientation of the current viewpoint to the first user interface object is a first orientation, and while displaying the first user interface object having a first amplitude of the three-dimensional stereoscopic effect, The method according to claim 66, further comprising: detecting the change in the user's current viewpoint from the first viewpoint to the second viewpoint; and displaying the first user interface object having a third amplitude of the three-dimensional stereo effect different from the first amplitude of the three-dimensional stereo effect.

68. The method according to claim 67, further comprising detecting the change in the user's current viewpoint from the first viewpoint to the second viewpoint, and gradually transitioning from displaying the first user interface object having the first amplitude of the three-dimensional stereo effect to displaying the first user interface object having the third amplitude of the three-dimensional stereo effect.

69. The method according to claim 67 or 68, wherein changing the orientation of the current viewpoint with respect to the first user interface object away from the first orientation includes changing the orientation of the first user interface object with respect to an axis parallel to the first plane.

70. The method according to any one of claims 67 to 69, wherein changing the orientation of the current viewpoint with respect to the first user interface object away from the first orientation includes changing the orientation of the first user interface object with respect to an axis perpendicular to the first plane.

71. The method according to any one of claims 46 to 70, wherein, in response to detecting the user's attention directed toward the first user interface object, transitioning from displaying the first user interface object having a first visual appearance to displaying the first user interface object having a second visual appearance includes displaying the first user interface object having a three-dimensional stereoscopic effect and extending the size of the first user interface object relative to the environment.

72. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, Through the display generation component, a first user interface object is generated in the environment, wherein the first user interface object is selectable to display first content, and the first user interface object displays a first user interface object having a first visual appearance. While displaying the first user interface object having the first visual appearance, the user's attention directed towards the first user interface object is detected via one or more input devices. In response to detecting the user's attention directed towards the first user interface object, A computer system comprising a command to display the first user interface object in the environment having a second visual appearance different from the first visual appearance via the display generation component, wherein displaying the first user interface object having the second visual appearance includes displaying the first user interface object having a three-dimensional stereoscopic effect corresponding to a plurality of different views of the first content corresponding to the first user interface object, and the first visual appearance of the first user interface object displayed before the user's attention is directed to the first user interface object does not include displaying the first user interface object having a three-dimensional stereoscopic effect.

73. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system The display generation component enables the display of a first user interface object in the environment, wherein the first user interface object is selectable for displaying first content, and the first user interface object has a first visual appearance. While displaying the first user interface object having the first visual appearance, the user's attention directed towards the first user interface object is detected via one or more input devices. In response to detecting the user's attention directed towards the first user interface object, A non-temporary computer-readable storage medium that causes a method to be performed, which includes displaying the first user interface object in the environment having a second visual appearance different from the first visual appearance via the display generation component, wherein the display of the first user interface object having a second visual appearance includes displaying the first user interface object having a three-dimensional stereoscopic effect corresponding to a plurality of different views of the first content corresponding to the first user interface object, and the first visual appearance of the first user interface object displayed before the user's attention is directed to the first user interface object does not include displaying the first user interface object having a three-dimensional stereoscopic effect.

74. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and Means for displaying a first user interface object, wherein the environment contains a first user interface object which is selectable for displaying first content and which has a first visual appearance, via the display generation component, Means for detecting the user's attention directed towards the first user interface object via one or more input devices while the first user interface object having the first visual appearance is displayed, In response to detecting the user's attention directed towards the first user interface object, A computer system comprising means for displaying the first user interface object in the environment having a second visual appearance different from the first visual appearance via the display generation component, wherein displaying the first user interface object having the second visual appearance includes displaying the first user interface object having a three-dimensional stereoscopic effect corresponding to a plurality of different views of the first content corresponding to the first user interface object, and the first visual appearance of the first user interface object displayed before the user's attention is directed to the first user interface object does not include displaying the first user interface object having a three-dimensional stereoscopic effect.

75. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 46 to 71.

76. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 46 to 71.

77. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for carrying out the method described in any one of claims 46 to 71.

78. It is a method, In a computer system that communicates with a display generation component and one or more input devices, A method comprising displaying a three-dimensional environment including objects via the aforementioned display generation component, wherein the display is In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the user's physical environment of the computer system is visible, and a second region containing one or more virtual objects, the object is... One or more visual characteristics of at least a part of the representation of the physical environment, A method comprising displaying in a three-dimensional environment using virtual lighting effects based on one or more visual characteristics of at least a portion of the one or more virtual objects.

79. In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the detection of one or more changes in one or more visual characteristics of at least a portion of the representation of the physical environment via one or more input devices while displaying the objects in the three-dimensional environment using the virtual lighting effect, In response to detecting one or more changes in at least a portion of the one or more visual characteristics of the representation of the physical environment, the display generation component generates the display of the object in the three-dimensional environment, The one or more changes in at least a part of the one or more visual characteristics of the representation of the physical environment, The method of claim 78, further comprising updating using a second virtual lighting effect based on the one or more visual characteristics of at least a portion of the one or more virtual objects.

80. In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the detection of one or more changes in one or more visual characteristics of at least a portion of the one or more virtual objects via one or more input devices while displaying the objects in the three-dimensional environment using the virtual lighting effect, In response to detecting one or more changes in at least some of the one or more visual characteristics of the one or more virtual objects, the display generation component generates the display of the object in the three-dimensional environment. The one or more visual characteristics of at least a part of the representation of the physical environment, The method according to claim 78 or 79, further comprising updating using a second virtual lighting effect based on one or more changes in one or more of the visual properties of at least a portion of the one or more virtual objects.

81. The three-dimensional environment including the object is displayed from the user's first viewpoint, and the method is In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the movement of the user's viewpoint from the first viewpoint to a second viewpoint different from the first viewpoint relative to the object in the three-dimensional environment is detected via one or more input devices while the object is displayed in the three-dimensional environment from the user's first viewpoint using the virtual lighting effect. In response to detecting the movement of the user's viewpoint, and in accordance with the determination that the movement of the viewpoint from the first viewpoint to the second viewpoint changes one or more visual properties of at least a portion of the representation of the physical environment and / or one or more visual properties of at least a portion of the virtual objects with respect to the second viewpoint, The display generation component generates the display of the object in the three-dimensional environment. One or more changes in one or more visual characteristics of at least a part of the representation of the physical environment, and / or The method according to any one of claims 78 to 80, further comprising updating using a second virtual lighting effect based on one or more changes in one or more of the visual properties of at least a portion of the one or more virtual objects.

82. Displaying the object in the three-dimensional environment using the aforementioned virtual lighting effect, The method according to any one of claims 78 to 81, comprising simultaneously displaying a first portion of an object in a three-dimensional environment using a first lighting effect based on one or more visual characteristics of the representation of at least a portion of the physical environment via the display generation component, and displaying a second portion of an object in the three-dimensional environment, different from the first portion, using a second lighting effect based on one or more visual characteristics of at least a portion of one or more virtual objects.

83. The object is displayed in a first spatial arrangement within the three-dimensional environment with respect to the first and second regions within the three-dimensional environment, and the method is In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the first portion of the object is displayed in the three-dimensional environment using a first lighting effect, and at the same time, the second portion of the object is displayed using a second lighting effect, while simultaneously detecting inputs via one or more input devices corresponding to the movement of the object from a first spatial arrangement in the three-dimensional environment to a second spatial arrangement different from the first spatial arrangement in the three-dimensional environment, to the first region and the second region in the three-dimensional environment. In response to detecting the aforementioned input, The display generation component moves the object from the first spatial arrangement to the second spatial arrangement in the first and second regions of the three-dimensional environment according to the input, The method further includes displaying the object in the three-dimensional environment using a second virtual lighting effect, and the display is Displaying a third portion of an object in a three-dimensional environment using a third lighting effect based on one or more visual characteristics of the representation of at least a portion of the physical environment, The method according to claim 82, further comprising displaying a fourth portion of the object in the three-dimensional environment using a fourth lighting effect based on one or more visual characteristics of at least a portion of the one or more virtual objects.

84. The method according to any one of claims 78 to 83, wherein the object is a first virtual object separate from the one or more virtual objects.

85. The method according to any one of claims 78 to 84, wherein the object is a first physical object that is visible in the three-dimensional environment.

86. The method includes a virtual environment in which one or more virtual objects are displayed within the three-dimensional environment at a first level of immersion, and the method In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the system detects inputs via one or more input devices that correspond to requests to change the level of immersion of the virtual environment while the objects are being displayed in the three-dimensional environment using the virtual lighting effect. In response to detecting the aforementioned input, The virtual environment is displayed in the three-dimensional environment at a second level of immersion, different from the first level of immersion, according to the input, via the display generation component. In accordance with the determination that displaying the virtual environment at the second level of immersion changes the visual prominence of at least some of the visual characteristics of one or more of the one or more virtual objects, The method according to any one of claims 78 to 85, further comprising updating the display of the objects in the three-dimensional environment using a second virtual lighting effect based on the modified visual prominence of at least some of the visual properties of the one or more virtual objects.

87. The input corresponds to a request to increase the immersion level of the virtual environment, In response to detecting the aforementioned input, The aforementioned second level of immersion is greater than the aforementioned first level of immersion. In accordance with the determination that displaying the virtual environment at the second level of immersion increases the visual prominence of at least some of the visual characteristics of one or more of the one or more virtual objects, The method according to claim 86, wherein the second virtual lighting effect is based on the increased visual prominence of one or more visual properties of at least a portion of the one or more virtual objects.

88. The input corresponds to a request to reduce the immersion level of the virtual environment, In response to detecting the aforementioned input, The second level of immersion is lower than the first level of immersion. In accordance with the determination that displaying the virtual environment at the second level of immersion reduces the visual prominence of at least some of the visual characteristics of one or more of the one or more virtual objects, The method according to claim 87, wherein the second virtual lighting effect is based on the reduced visual prominence of at least some of the one or more visual properties of the one or more virtual objects.

89. Displaying the object in the three-dimensional environment using the aforementioned virtual lighting effect, In accordance with the determination that a portion of the virtual lighting effect based on one or more visual characteristics of at least a part of the representation of the physical environment overlaps at least partially with a portion of the virtual lighting effect based on one or more visual characteristics of at least a part of the one or more virtual objects in a first part of the object, The method according to any one of claims 78 to 88, comprising displaying the first portion of the object in the three-dimensional environment using a visual effect based on a combination of one or more visual characteristics of at least a portion of the representation of the physical environment and one or more visual characteristics of at least a portion of the one or more virtual objects, via the display generation component.

90. In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the physical environment is visible, and a second region containing one or more virtual objects, the system receives input via one or more input devices corresponding to a request to apply individual visual effects to at least a portion of the representation of the physical environment while displaying the objects in the three-dimensional environment using the virtual lighting effect, In response to receiving the aforementioned input, To generate one or more second visual characteristics of at least a portion of the representation of the physical environment, the application of individual visual adjustments to at least a portion of the representation of the physical environment further includes, The method according to any one of claims 78 to 89, comprising updating the display of the object in the three-dimensional environment using a second virtual lighting effect based on one or more generated second visual characteristics of at least a portion of the representation of the physical environment.

91. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, A computer system that includes instructions for displaying a three-dimensional environment, including objects, via the aforementioned display generation component, wherein the display is In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the user's physical environment of the computer system is visible, and a second region containing one or more virtual objects, the object is... One or more visual characteristics of at least a part of the representation of the physical environment, A computer system that includes displaying one or more of the visual properties of at least some of the aforementioned virtual objects in a three-dimensional environment using virtual lighting effects.

92. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system The method of displaying a three-dimensional environment, including objects, via the aforementioned display generation component, is performed and displayed. In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the user's physical environment of the computer system is visible, and a second region containing one or more virtual objects, the object is... One or more visual characteristics of at least a part of the representation of the physical environment, A non-temporary computer-readable storage medium that includes displaying in the three-dimensional environment using virtual lighting effects based on one or more visual characteristics of at least some of the one or more virtual objects.

93. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for displaying a three-dimensional environment including objects via the aforementioned display generation component, wherein the display is In accordance with the determination that the three-dimensional environment includes a first region in which at least a portion of the representation of the user's physical environment of the computer system is visible, and a second region containing one or more virtual objects, the object is... One or more visual characteristics of at least a part of the representation of the physical environment, A computer system that includes displaying one or more of the visual properties of at least some of the aforementioned virtual objects in a three-dimensional environment using virtual lighting effects.

94. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 78 to 90.

95. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 78 to 90.

96. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for carrying out the method described in any one of claims 78 to 90.

97. It is a method, In a computer system that communicates with a display generation component and one or more input devices, The first environment is displayed via the aforementioned display generation component, While the first environment is being displayed, a request to display a second environment different from the first environment is detected via one or more input devices, In response to detecting the request to display the second environment, In accordance with the determination that one or more first criteria are met, including the criteria that are met when the first environment is an environment of a first type, the transition during the transition is from displaying the first environment using a first visual effect to displaying the second environment, A method comprising: transitioning from displaying the first environment to displaying the second environment using a second visual effect in the transition that is different from the first visual effect, in accordance with a determination that one or more second criteria are met, including criteria that are met when the first environment is a second type of environment different from the first type of environment.

98. The method according to claim 97, wherein one or more of the first criteria include a second criterion that is satisfied when the second environment is a third type of environment, and one or more of the second criteria include a third criterion that is satisfied when the second environment is a fourth type of environment.

99. The first type of environment is a virtual environment type, the third type of environment is a virtual environment type, and the first visual effect is The method according to claim 98, comprising gradually reducing the visual splendor of the first environment, and gradually increasing the visual splendor of the second environment after gradually reducing the visual splendor of the first environment.

100. The first type of environment is a physical environment type, the third type of environment is a virtual environment type, and the first visual effect is The method according to claim 98, comprising gradually replacing the display of the increasing portion of the first environment with the display of the corresponding increasing portion of the second environment.

101. The first type of environment is a virtual environment type, the third type of environment is an atmospheric environment type, and the first visual effect is To clarify the representation of the physical environment through the aforementioned display generation component, the visual prominence of the first environment is gradually reduced, The method of claim 98, comprising gradually increasing the visual prominence of an atmospheric effect associated with a second environment after at least partially revealing the representation of the physical environment via the display generating component, wherein the atmospheric effect is applied to the representation of the physical environment.

102. The method according to claim 101, wherein gradually reducing the visual prominence of the first environment is performed at least partially simultaneously with gradually increasing the visual prominence of the atmosphere effect.

103. The first type of environment is a virtual environment type, the third type of environment is a physical environment type, and the first visual effect is The method of claim 98, comprising gradually replacing the representation of an increasing portion of the first environment with the representation of the physical environment via the representation generation component until the representation of the physical environment replaces all representations of the first environment.

104. The third type of environment is a virtual environment type, and the first visual effect is The method according to any one of claims 98 to 100, comprising displaying the second environment at a default immersion level.

105. While the second environment is displayed at the default immersion level, the system detects user input via one or more input devices that corresponds to a request to increase the immersion level of the second environment from the default immersion level to the second immersion level. The method according to claim 104, further comprising displaying the second environment at the second level of immersion in response to the detection of the user input.

106. While the second environment is displayed at the default immersion level, the system detects user input via one or more input devices that corresponds to a request to reduce the immersion level from the default immersion level to the second immersion level. The method according to claim 104, further comprising displaying the second environment at the second level of immersion in response to the detection of the user input.

107. The method according to claim 98, wherein the first type of environment is a virtual environment type, the third type of environment is the virtual environment type, the first environment is displayed at a first level of immersion, and the second environment is displayed at the first level of immersion after the first visual effect.

108. The first type of environment is an atmospheric environment type, the second type of environment is a virtual environment type, and the method is The method according to claim 98, further comprising displaying a virtual environment at a first level of immersion before displaying the first environment, wherein the second environment is displayed at the first level of immersion after the first visual effect.

109. The method according to claim 98, wherein the first type of environment is an atmospheric environment type, the second type of environment is a virtual environment type, and the second environment is displayed at a default immersion level after the first visual effect.

110. The second type of environment is a mixed virtual and atmospheric environment type, which includes a first virtual environment having one or more animated elements, and the first visual effect is The visual prominence of the atmospheric effect corresponding to the second environment is to be gradually increased, The method according to claim 98, comprising gradually increasing the visual prominence of the atmospheric effect corresponding to the second environment to a final visual prominence, and then displaying the first virtual environment.

111. The first type of environment is an atmospheric environment type, the second type of environment is the same atmospheric environment type, and the first visual effect is A first atmospheric effect associated with the first environment, wherein the first atmospheric effect is applied to the representation of the physical environment, gradually reducing the visual prominence of the first atmospheric effect. The method of claim 98, comprising gradually increasing the visual prominence of a second atmospheric effect associated with the second environment, while simultaneously decreasing the visual prominence of the first atmospheric effect, wherein the second atmospheric effect is applied to the representation of the physical environment.

112. The first type of environment is an atmospheric environment type, and the second type of environment is a mixed virtual and atmospheric environment type that includes a first virtual environment having one or more virtual animated elements, and the first visual effect is A first atmospheric effect associated with the first environment, wherein the first atmospheric effect is applied to the representation of the physical environment, gradually reducing the visual prominence of the first atmospheric effect. To reduce the visual prominence of the first atmospheric effect, and at the same time to gradually increase the visual prominence of a second atmospheric effect associated with the second environment, wherein the second atmospheric effect is applied to the representation of the physical environment. The method according to claim 98, comprising increasing the visual prominence of the second atmospheric effect to its final visual prominence, and then displaying the first virtual environment.

113. The first type of environment is a mixed virtual and atmospheric environment type that includes a first virtual environment having one or more virtual animated elements, and the first visual effect is To stop displaying the first virtual environment, After discontinuing the display of the first virtual environment, Gradually reducing the visual prominence of the first atmospheric effect associated with the first environment, The method of claim 98, comprising gradually increasing the visual prominence of a second atmospheric effect associated with a second environment, while simultaneously decreasing the visual prominence of the first atmospheric effect, wherein the second atmospheric effect is applied to a representation of a physical environment.

114. Displaying the first environment includes displaying media content in the first environment, and the first visual effect is The method according to any one of claims 97 to 113, comprising reducing the visual prominence of the visual portion of the media content before displaying the second environment.

115. The method according to claim 114, wherein the first visual effect includes pausing the media content.

116. The first visual effect described above is The method according to claim 114 or 115, comprising continuing to play the audio portion of the media content while reducing the visual prominence of the visual portion of the media content.

117. The method according to any one of claims 114 to 116, wherein when the request to display the second environment is received, the media content is displayed in a first spatial arrangement with respect to the user's viewpoint of the computer system, the second environment is of a virtual environment type, and displaying the second environment includes displaying the media content in a second spatial arrangement different from the first spatial arrangement with respect to the user's viewpoint.

118. Displaying the first environment includes displaying media content in the first environment, and the first visual effect is The method according to any one of claims 97 to 117, comprising discontinuing the display of the media content and, after discontinuing the display of the media content, reducing the visual prominence of the first environment.

119. Displaying the media content in the first environment includes displaying a first simulated lighting effect associated with the media content outside of the media content, wherein the light associated with the media content is virtually projected by the media content onto one or more virtual or physical object representations. The method according to claim 118, wherein discontinuing the display of the media content includes discontinuing the display of the first simulated lighting effect associated with the media content.

120. The method according to claim 118 or 119, further comprising increasing the visual splendor of the second environment, which includes reducing the visual splendor of the first environment and then increasing the visual splendor of the media content displayed in the second environment.

121. The method according to claim 120, wherein displaying the media content in the second environment includes displaying a second simulated lighting effect associated with the media content outside of the media content.

122. The virtual content is displayed along with its individual environment, and the display of the said virtual content is In accordance with the determination that the individual environment is the first environment, the virtual content is displayed with the first value of the individual visual parameter. The method according to any one of claims 97 to 121, wherein, in accordance with the determination that the individual environment is a second environment different from the first environment, the virtual content is displayed with a second value of the individual visual parameter, and the second value of the individual virtual parameter is different from the second value of the individual virtual parameter.

123. Before detecting the request to display the second environment, virtual content is displayed within the first environment using a first value of a specific visual parameter associated with the first environment, The method according to any one of claims 97 to 122, further comprising detecting the request to display the second environment, and then displaying the virtual content within the second environment using a second value of the individual visual parameter associated with the second environment, wherein the second value of the individual visual parameter is different from the first value of the individual visual parameter.

124. The method according to claim 123, wherein the first value of the individual visual parameter includes a first luminance associated with the first environment, and the second value of the individual visual parameter includes a second luminance associated with the second environment, which is different from the first luminance.

125. The method according to any one of claim 123 or 124, wherein transitioning from displaying the first environment to displaying the second environment includes changing the brightness of the virtual content from the first brightness to the second brightness.

126. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, The first environment is displayed via the aforementioned display generation component. While the first environment is being displayed, a request to display a second environment different from the first environment is detected via one or more input devices. In response to detecting the request to display the second environment, In accordance with the determination that one or more first criteria are met, including the criteria that are met when the first environment is an environment of a first type, the transition occurs from displaying the first environment using a first visual effect to displaying the second environment during the transition. A computer system including instructions to transition from displaying the first environment to displaying the second environment using a second visual effect in the transition that is different from the first visual effect, in accordance with a determination that one or more second criteria are met, including criteria that are met when the first environment is a second type of environment different from the first type of environment.

127. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system The first environment is displayed via the aforementioned display generation component, While the first environment is being displayed, a request to display a second environment different from the first environment is detected via one or more input devices, In response to detecting the request to display the second environment, In accordance with the determination that one or more first criteria are met, including the criteria that are met when the first environment is an environment of a first type, the transition during the transition is from displaying the first environment using a first visual effect to displaying the second environment, A non-temporary computer-readable storage medium that causes a method to be performed that includes transitioning from displaying the first environment to displaying the second environment using a second visual effect in the transition that is different from the first visual effect, in accordance with a determination that one or more second criteria are met, including criteria that are met when the first environment is a second type of environment different from the first type of environment.

128. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A means for displaying the first environment via the aforementioned display generation component, Means for detecting a request to display a second environment different from the first environment via one or more input devices while the first environment is being displayed, In response to detecting the request to display the second environment, In accordance with the determination that one or more first criteria are met, including the criteria that are met when the first environment is an environment of a first type, the transition occurs from displaying the first environment using a first visual effect to displaying the second environment during the transition. A computer system comprising: means for transitioning from displaying the first environment to displaying the second environment using a second visual effect during the transition that is different from the first visual effect, according to a determination that one or more second criteria are met, including criteria that are met when the first environment is a second type of environment different from the first type of environment.

129. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 97 to 125.

130. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 97 to 125.

131. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for performing the method described in any one of claims 97 to 125.

132. It is a method, In a computer system that communicates with a display generation component and one or more input devices, While displaying a representation of a three-dimensional environment that includes at least a portion of a portal to a virtual environment, the system detects, via one or more input devices, the movement of the user's viewpoint relative to the three-dimensional environment. In response to detecting the movement of the user's viewpoint into the three-dimensional environment, The portal to the virtual environment opens in a first direction relative to the three-dimensional environment, and the display of at least a portion of the portal to the virtual environment within the representation of the three-dimensional environment is maintained in accordance with the determination that the movement is greater than a movement threshold. A method comprising: discontinuing the display of at least the portion of the portal to the virtual environment within the representation of the three-dimensional environment, while at least the portion of the representation of the three-dimensional environment that included at least the portion of the portal to the virtual environment remains visible from the user's viewpoint, in accordance with the determination that the portal opens in a second direction relative to the three-dimensional environment, the second direction differs from the first direction, and the movement is greater than the movement threshold.

133. The method according to claim 132, wherein the first direction lies within a first angular threshold of the gravity vector or the normal of the floor plane associated with the representation of the three-dimensional environment, and the second direction lies within a second angular threshold of the normal of the horizon plane associated with the three-dimensional environment.

134. The method according to claim 132 or 133, further comprising detecting the movement of the user's viewpoint to the three-dimensional environment, and maintaining the display of at least a portion of the portal to the virtual environment within the representation of the three-dimensional environment, regardless of whether the portal opens to the three-dimensional environment in a first direction or a second direction, according to a determination that the movement is less than the movement threshold.

135. The method according to any one of claims 132 to 134, wherein displaying the representation of the three-dimensional environment, which includes at least a portion of the portal to the virtual environment, includes displaying atmospheric effects associated with the virtual environment within the representation of the three-dimensional environment.

136. The virtual environment includes a simulated physical space and displays the atmospheric effects associated with the virtual environment. The computer system displays a first atmospheric effect associated with the virtual environment, in accordance with the determination that the computer system is operating in a first mode, and that the first mode is associated with a first simulated time in the simulated physical space. The method according to claim 135, comprising displaying a second ambient effect associated with the virtual environment, in accordance with the determination that the computer system is operating in a second mode different from the first mode, and the second mode is associated with a second simulated time in the simulated physical space different from the first simulated time, wherein the second ambient effect is different from the first ambient effect.

137. While the representation of the three-dimensional environment, which includes at least a portion of the portal to the virtual environment, is not displayed, The method according to claim 136, further comprising displaying a second representation of the three-dimensional environment that does not include a portal to any virtual environment, wherein the second representation of the three-dimensional environment is displayed with a third atmospheric effect, regardless of whether the computer system is operating in the first mode or the second mode.

138. Displaying the aforementioned representation of the three-dimensional environment In accordance with the determination that the portal to the virtual environment is open in the first direction relative to the three-dimensional environment, the representation of the three-dimensional environment is displayed using the atmosphere effects associated with the virtual environment. The method according to any one of claims 132 to 137, comprising displaying the representation of the three-dimensional environment without atmospheric effects associated with the virtual environment, in accordance with the determination that the portal to the virtual environment is open in the second direction with respect to the three-dimensional environment.

139. The method according to any one of claims 132 to 138, wherein the virtual environment includes animated virtual content that is visible through at least a portion of the portal.

140. Before displaying the representation of the three-dimensional environment, which includes at least a portion of the portal to the virtual environment, While the user's viewpoint is directed in a specific direction relative to the three-dimensional environment, the system detects input from the user corresponding to a request to display the virtual environment, The method according to any one of claims 132 to 139, further comprising: detecting the aforementioned input and determining that the virtual environment is a first virtual environment, displaying the representation of the three-dimensional environment, including at least a portion of the portal to the virtual environment, with the portal open to the three-dimensional environment in a first direction, wherein the first direction is independent of the individual directions.

141. In response to detecting the aforementioned input, and in accordance with the determination that the virtual environment is a second virtual environment different from the first virtual environment, In accordance with the determination that the individual directions are third directions, the representation of the three-dimensional environment including at least a portion of the portal to the virtual environment is displayed in a state where the portal is open in the second direction relative to the three-dimensional environment, and the second direction corresponds to the third direction. The method according to claim 140, further comprising displaying the representation of the three-dimensional environment including at least a portion of the portal to the virtual environment, with the portal open in a fifth direction to the three-dimensional environment, in accordance with the determination that the individual direction is a fourth direction different from the third direction, wherein the fifth direction corresponds to the fourth direction.

142. In response to the detection of the movement of the user's viewpoint relative to the three-dimensional environment, according to the determination that the portal to the virtual environment opens in the first direction relative to the three-dimensional environment, The method according to any one of claims 132 to 141, further comprising shifting the boundary of at least a portion of the portal with respect to the three-dimensional environment in accordance with the movement of the user's viewpoint.

143. The method according to claim 142, wherein the movement of the user's viewpoint is a first direction of movement relative to the three-dimensional environment, and shifting the boundary of at least a portion of the portal extends the first portion of the portal in the first direction of movement in accordance with the movement of the user's viewpoint.

144. The method according to claim 142 or 143, wherein the movement of the user's viewpoint is a first direction of movement relative to the three-dimensional environment, and shifting the boundary of at least a portion of the portal includes shrinking a second portion of the portal in the first direction of movement in accordance with the movement of the user's viewpoint.

145. Displaying the representation of the three-dimensional environment, which includes at least a portion of the portal to the virtual environment, The method according to any one of claims 132 to 144, comprising reducing the visual prominence of the first portion of the portal, including the first portion of the virtual environment, to the three-dimensional environment, based on the determination that a first physical object in the user's physical environment, which is visible in the three-dimensional environment, has a spatial collision with the first portion of the portal, including the first portion of the virtual environment from the user's viewpoint.

146. Detecting the movement of the user's viewpoint includes detecting that the user's viewpoint has moved from a first viewpoint to a second viewpoint, and the method is Before detecting the movement of the user's viewpoint, and while the first physical object in the user's physical environment, which is visible in the three-dimensional environment, has a spatial collision with the first portion of the portal, which includes the second portion of the virtual environment, from the user's viewpoint, the second portion of the portal, which includes the second portion of the virtual environment, is displayed with a first visual splendor greater than the visual splendor of the first portion of the portal, which includes the first portion of the virtual environment, relative to the three-dimensional environment. The method of claim 145, further comprising: detecting the movement of the user's viewpoint relative to the three-dimensional environment, and in accordance with the determination that a second physical object in the user's physical environment is visible in the three-dimensional environment and has a spatial collision with the second portion of the portal including the second portion of the virtual environment from the user's viewpoint, and that the physical object does not have a spatial collision with the first portion of the portal, the method of displaying the first portion of the portal including the first portion of the virtual environment having a second visual visibility relative to the three-dimensional environment, wherein the second visual visibility is greater than the visual visibility of the second portion of the portal including the second portion of the virtual environment relative to the three-dimensional environment.

147. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, While displaying a representation of a three-dimensional environment, including at least a portion of a portal to a virtual environment, the computer system detects a shift in the user's viewpoint relative to the three-dimensional environment via one or more input devices. In response to detecting the movement of the user's viewpoint into the three-dimensional environment, The portal to the virtual environment opens in a first direction relative to the three-dimensional environment, and in accordance with the determination that the movement is greater than the movement threshold, the display of at least a portion of the portal to the virtual environment within the representation of the three-dimensional environment is maintained. A computer system including a command to cease displaying the at least portion of the portal to the virtual environment within the representation of the three-dimensional environment, while at least the portion of the representation of the three-dimensional environment that included the at least portion of the portal to the virtual environment remains visible from the user's viewpoint, in accordance with the determination that the portal opens in a second direction relative to the three-dimensional environment, the second direction differs from the first direction, and the movement is greater than the movement threshold.

148. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system While displaying a representation of a three-dimensional environment that includes at least a portion of a portal to a virtual environment, the system detects, via one or more input devices, the movement of the user's viewpoint relative to the three-dimensional environment. In response to detecting the movement of the user's viewpoint into the three-dimensional environment, The portal to the virtual environment opens in a first direction relative to the three-dimensional environment, and the display of at least a portion of the portal to the virtual environment within the representation of the three-dimensional environment is maintained in accordance with the determination that the movement is greater than a movement threshold. A non-temporary computer-readable storage medium that causes a method to be performed, which includes: the portal opening in a second direction relative to the three-dimensional environment, and the second direction being different from the first direction, and the movement being greater than the movement threshold, and while at least a portion of the representation of the three-dimensional environment that included at least a portion of the portal to the virtual environment remains visible from the user's viewpoint.

149. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and Means for detecting a shift in the user's viewpoint relative to the three-dimensional environment via one or more input devices while displaying a representation of a three-dimensional environment that includes at least a portion of a portal to a virtual environment, In response to detecting the movement of the user's viewpoint into the three-dimensional environment, The portal to the virtual environment opens in a first direction relative to the three-dimensional environment, and in accordance with the determination that the movement is greater than the movement threshold, the display of at least a portion of the portal to the virtual environment within the representation of the three-dimensional environment is maintained. A computer system comprising: means for ceasing to display the at least portion of the portal to the virtual environment within the representation of the three-dimensional environment, while at least the portion of the representation of the three-dimensional environment that included the at least portion of the portal to the virtual environment remains visible from the user's viewpoint, in accordance with the determination that the portal opens in a second direction relative to the three-dimensional environment and the second direction is different from the first direction and the movement is greater than the movement threshold.

150. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 132 to 146.

151. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 132 to 146.

152. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for performing the method described in any one of claims 132 to 146.

153. It is a method, In a computer system that communicates with a display generation component and one or more input devices, The system receives a first user input corresponding to a request to display an individual virtual three-dimensional environment via one or more input devices. A method comprising, in response to receiving the first user input, displaying the individual virtual three-dimensional environment via the display generation component, wherein the display is In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect is output when the display of the first virtual three-dimensional environment is started. A method comprising: outputting a second sound effect different from the first sound effect when starting to display the second virtual three-dimensional environment, based on the determination that the individual virtual three-dimensional environment is a second virtual three-dimensional environment different from the first virtual three-dimensional environment.

154. The method according to claim 153, wherein outputting the first sound effect includes, when displaying the first virtual three-dimensional environment, outputting one or more first sound effects based on one or more first ambient sound effects corresponding to the first virtual three-dimensional environment, and outputting the second sound effect includes, when displaying the second virtual three-dimensional environment, outputting one or more second sound effects different from the one or more first ambient sound effects and based on one or more second ambient sound effects corresponding to the second virtual three-dimensional environment.

155. The method according to claim 154, wherein outputting the first sound effect includes outputting the first sound effect having spatialized audio having a simulated position relative to the user's viewpoint which moves in conjunction with changes in the appearance of the first virtual three-dimensional environment when displaying the first virtual three-dimensional environment begins, and outputting the second sound effect includes outputting the second sound effect having spatialized audio having a simulated position relative to the user's viewpoint which moves in conjunction with changes in the appearance of the second virtual three-dimensional environment.

156. While the individual virtual three-dimensional environment is being displayed at a first level of immersion via the display generation component, a second user input corresponding to a request to change the level of immersion of the individual virtual three-dimensional environment is received via one or more input devices. Upon receiving the second user input, In accordance with the second input, the individual virtual three-dimensional environment is displayed at a second immersion level different from the first immersion level, The method according to any one of claims 153 to 155, further comprising outputting individual sound effects when changing the immersion level of the individual virtual three-dimensional environment.

157. When changing the immersion level of the individual virtual three-dimensional environment, the individual sound effects are output. When the immersion level of the first virtual three-dimensional environment is changed according to the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, an individual sound effect corresponding to the change in the immersion level is output. The method according to claim 156, further comprising: outputting the individual sound effect corresponding to the change in the immersion level when changing the immersion level of the second virtual three-dimensional environment in accordance with the determination that the individual virtual three-dimensional environment is the second virtual three-dimensional environment.

158. When changing the immersion level of the individual virtual three-dimensional environment, the individual sound effects are output. In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect corresponding to the change in the immersion level of the first virtual three-dimensional environment is output. The method according to claim 156, further comprising: determining that the individual virtual three-dimensional environment is the second virtual three-dimensional environment, outputting a second sound effect corresponding to a change in the immersion level of the second virtual three-dimensional environment, which is different from the first sound effect corresponding to a change in the immersion level of the first virtual three-dimensional environment.

159. When changing the immersion level of the individual virtual three-dimensional environment, the individual sound effects are output. In accordance with the determination that the second input corresponds to a request to increase the immersion level of the individual virtual three-dimensional environment, an individual sound effect corresponding to increasing the immersion level of the individual virtual three-dimensional environment is output. The method according to any one of claims 156 to 158, comprising: outputting a separate sound effect corresponding to reducing the immersion level of the individual virtual three-dimensional environment, which is different from the separate sound effect corresponding to reducing the immersion level of the individual virtual three-dimensional environment, in accordance with the determination that the second input corresponds to a request to reduce the immersion level of the individual virtual three-dimensional environment.

160. When changing the immersion level of the individual virtual three-dimensional environment, the individual sound effects are output. In accordance with the determination that the second input corresponds to a request to display the individual virtual three-dimensional environment at the maximum immersion level of the individual virtual three-dimensional environment, the system outputs an individual sound effect corresponding to the maximum immersion level of the individual virtual three-dimensional environment. The method according to any one of claims 156 to 159, comprising: outputting a separate sound effect corresponding to the minimum immersion level of the separate virtual three-dimensional environment, in accordance with the determination that the second input corresponds to a request to display the separate virtual three-dimensional environment at the minimum immersion level of the separate virtual three-dimensional environment.

161. The method according to claim 160, wherein the individual sound effect corresponding to the maximum immersion level of the individual virtual three-dimensional environment is different from the individual sound effect corresponding to the minimum immersion level of the individual virtual three-dimensional environment.

162. Each individual virtual three-dimensional environment has a distinct visual appearance corresponding to a first time in the physical space simulated by the individual virtual three-dimensional environment, and the method While displaying the individual virtual three-dimensional environment having the individual visual appearance corresponding to the first time, receiving a second user input via one or more input devices that corresponds to a request to change the individual visual appearance in the physical space simulated by the individual virtual three-dimensional environment from one corresponding to the first time to one corresponding to a second time different from the first time, Upon receiving the second user input, Displaying the individual virtual three-dimensional environments having the individual visual appearance corresponding to the second time, The method according to any one of claims 153 to 161, further comprising outputting a separate sound effect when changing the separate visual appearance from one corresponding to the first time to one corresponding to the second time in the physical space simulated by the separate virtual three-dimensional environment.

163. In the physical space simulated by the individual virtual three-dimensional environment, when the individual visual appearance is changed from one corresponding to the first time to one corresponding to the second time, the individual sound effect is output. In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, an individual sound effect is output that corresponds to changing the individual visual appearance from one corresponding to the first time to one corresponding to the second time in the physical space simulated by the first virtual three-dimensional environment. The method according to claim 162, comprising: determining that the individual virtual three-dimensional environment is the second virtual three-dimensional environment, outputting the individual sound effect in the physical space simulated by the second virtual three-dimensional environment, which corresponds to changing the individual visual appearance from one corresponding to the first time to one corresponding to the second time.

164. In the physical space simulated by the individual virtual three-dimensional environment, when the individual visual appearance is changed from one corresponding to the first time to one corresponding to the second time, the individual sound effect is output. In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect is output that corresponds to changing the individual visual appearance from one corresponding to the first time to one corresponding to the second time in the physical space simulated by the first virtual three-dimensional environment. The method according to claim 162, comprising: determining that the individual virtual three-dimensional environment is the second virtual three-dimensional environment, outputting a second sound effect that is different from the first

165. While the individual virtual three-dimensional environment is being displayed, a second user input is received via one or more input devices in response to a request to stop displaying within the individual virtual three-dimensional environment. Upon receiving the second user input, The display of the aforementioned individual virtual three-dimensional environments will be discontinued, The method according to any one of claims 153 to 164, further comprising outputting an individual sound effect when discontinuing the display of the individual virtual three-dimensional environment.

166. When the display of the individual virtual three-dimensional environment is stopped, the individual sound effect is output. In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, an individual sound effect corresponding to ceasing the display of the individual virtual three-dimensional environment is output. The method according to claim 165, further comprising: outputting the individual sound effect corresponding to ceasing the display of the individual virtual three-dimensional environment in accordance with the determination that the individual virtual three-dimensional environment is the second virtual three-dimensional environment.

167. When the display of the individual virtual three-dimensional environment is stopped, the individual sound effect is output. In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect corresponding to ceasing the display of the first virtual three-dimensional environment is output. The method according to claim 165, further comprising: determining that the individual virtual three-dimensional environment is the second virtual three-dimensional environment, outputting a second sound effect corresponding to discontinuing the display of the second virtual three-dimensional environment, which is different from the first sound effect corresponding to discontinuing the display of the first virtual three-dimensional environment.

168. The method according to any one of claims 165 to 167, wherein outputting the individual sound effect when discontinuing the display of the individual virtual three-dimensional environment includes outputting the individual sound effect having spatialized audio having a simulated position relative to the user's viewpoint that moves in conjunction with the change in appearance of the individual virtual three-dimensional environment when the display of the individual virtual three-dimensional environment is gradually discontinued.

169. While displaying the first virtual three-dimensional environment via the display generation component, a second user input corresponding to a request to start displaying the second virtual three-dimensional environment is received via one or more input devices. Upon receiving the second user input, The display of the first virtual three-dimensional environment is discontinued, The second virtual three-dimensional environment is displayed via the aforementioned display generation component, The method according to any one of claims 153 to 168, further comprising outputting the second sound effect when initiating the display of the second virtual three-dimensional environment.

170. The method according to claim 169, further comprising refraining from outputting a sound effect when discontinuing the display of the first virtual three-dimensional environment in response to receiving the second user input.

171. The method according to any one of claims 153 to 170, wherein outputting the first sound effect or the second sound effect includes outputting the first sound effect or the second sound effect for 1 to 3 seconds.

172. The method according to any one of claims 153 to 171, wherein outputting the first sound effect or the second sound effect includes outputting an individual sound effect which includes the sound of sand being blown by the wind.

173. The method according to any one of claims 153 to 172, wherein outputting the first sound effect or the second sound effect includes outputting an individual sound effect that includes a simulated sound of a cricket chirping.

174. While the first virtual three-dimensional environment is displayed and after the first sound effect is output, one or more third sound effects corresponding to the first virtual three-dimensional environment are output. The system further includes, while displaying the second virtual three-dimensional environment, outputting the second sound effect, and then outputting one or more fourth sound effects, which are different from the one or more third sound effects, that correspond to the second virtual three-dimensional environment. Outputting the first sound effect includes outputting one or more amplified portions of the one or more third sound effects, wherein the one or more third sound effects are output after the one or more amplified portions of the one or more third sound effects have been output. The method according to any one of claims 153 to 173, wherein outputting the second sound effect includes outputting one or more amplified portions of the one or more fourth sound effects, and the one or more fourth sound effects are output after the one or more amplified portions of the one or more fourth sound effects have been output.

175. The computer system receives a second user input via one or more input devices that corresponds to a request to display an atmospheric effect applied to a representation of the user's physical environment. Upon receiving the second user input, The display generation component displays the representation of the user's physical environment to which the atmosphere effect has been applied, The method according to any one of claims 153 to 174, further comprising outputting a separate sound effect when initiating the display of the atmospheric effect applied to the representation of the user's physical environment of the computer system.

176. The display generation component provides for one or more application user interfaces, wherein the first user input is received while the one or more application user interfaces are being displayed, and the display of one or more application user interfaces is provided via the display generation component. The method according to any one of claims 153 to 175, further comprising: maintaining the display of one or more application user interfaces while displaying the individual virtual three-dimensional environments in response to receiving the first user input.

177. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, The system receives a first user input via one or more input devices that corresponds to a request to display an individual virtual three-dimensional environment. A computer system that includes an instruction to display the individual virtual three-dimensional environment via the display generation component in response to receiving the first user input, wherein the display is In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect is output when the display of the first virtual three-dimensional environment is started. A computer system that, upon determining that the individual virtual three-dimensional environment is a second virtual three-dimensional environment different from the first virtual three-dimensional environment, outputs a second sound effect different from the first sound effect when starting to display the second virtual three-dimensional environment.

178. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system The system receives a first user input corresponding to a request to display an individual virtual three-dimensional environment via one or more input devices. A non-temporary computer-readable storage medium that causes a method to be executed, which includes displaying the individual virtual three-dimensional environment via the display generation component in response to receiving the first user input, and which displays In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect is output when the display of the first virtual three-dimensional environment is started. A non-temporary computer-readable storage medium, which includes outputting a second sound effect different from the first sound effect when starting to display the second virtual three-dimensional environment, based on the determination that the individual virtual three-dimensional environment is a second virtual three-dimensional environment different from the first virtual three-dimensional environment.

179. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and Means for receiving first user input corresponding to a request to display an individual virtual three-dimensional environment via one or more input devices, A computer system comprising means for displaying the individual virtual three-dimensional environment via the display generation component in response to receiving the first user input, wherein the display is In accordance with the determination that the individual virtual three-dimensional environment is the first virtual three-dimensional environment, a first sound effect is output when the display of the first virtual three-dimensional environment is started. A computer system that, upon determining that the individual virtual three-dimensional environment is a second virtual three-dimensional environment different from the first virtual three-dimensional environment, outputs a second sound effect different from the first sound effect when starting to display the second virtual three-dimensional environment.

180. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions to perform the method according to any one of claims 153 to 176.

181. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 153 to 176.

182. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for performing the method described in any one of claims 153 to 176.

183. The method according to any one of claims 1 to 23, wherein changing the level of detail at which the individual environment is displayed includes changing the resolution of one or more virtual elements within the individual environment.

184. While displaying the individual environments at a first level of detail higher than the second level of detail that can display the individual environments, Displaying a first part of the individual environment using a texture that includes a first level of animation, The method further includes displaying a second portion of the individual environment using a texture that includes a second level of animation lower than the first level of animation, The method according to any one of claims 1 to 23 and 183, wherein the first portion of the individual environment is closer to the user's perspective within the individual environment than the second portion.

185. The method according to claim 184, wherein the individual environment includes a third portion, the second portion of the individual environment is closer to the user's viewpoint within the individual environment than the third portion of the individual environment, and the third portion of the individual environment is displayed with a texture that does not include animation.

186. The method according to claim 184 or 185, wherein the texture including the first level animation and the texture including the second level animation correspond to a simulated water surface in the individual environment.

187. The method according to any one of claims 1 to 23 and 183 to 186, wherein changing the level of detail on which the individual environment is displayed from a first level of detail to a second level of detail includes changing the level of detail such that the display of the individual environment requires the maximum amount of individual power corresponding to the second level of detail, and the amount of individual power corresponding to the second level of detail is the same regardless of whether the individual environment is a first environment or a second environment different from the first environment.

188. It is a method, In a computer system that communicates with a display generation component and one or more input devices, A method comprising displaying a first simulated shadow corresponding to a first virtual object in the environment via a display generation component when the environment is visible, wherein the first simulated shadow has a size and shape based on the size and shape of the virtual object onto which the first simulated shadow is projected and on the individual portion of the environment on which the first simulated shadow appears, and has an individual visual appearance based on the shadow texture of the first simulated shadow, and the display of the first simulated shadow is In accordance with the determination that the individual part of the environment is the first part of the environment, the first simulated shadow is displayed on the individual virtual element using the shadow texture having a first visual appearance, A method comprising: displaying the first simulated shadow on the individual virtual element using the shadow texture having a second visual appearance different from the first visual appearance, in accordance with the determination that the individual part of the environment is a second part of the environment different from the first part of the environment.

189. The method according to claim 188, wherein the first portion of the environment is virtual water having a first simulated depth, and the second portion of the environment is virtual water having a second simulated depth different from the first simulated depth.

190. The method according to claim 189, wherein the shadow texture having the first visual appearance includes a first color, and the shadow texture having the second visual appearance includes a second color different from the first color.

191. The method according to claim 189 or 190, wherein the shadow texture having the first visual appearance has a first visual splendor with respect to the environment, and the shadow texture having the second visual appearance has a second visual splendor with respect to the environment that is different from the first visual splendor.

192. The method according to any one of claims 188 to 191, wherein the first portion of the environment is virtual water and the second portion of the environment is virtual land.

193. The method according to any one of claims 188 to 192, wherein the first portion of the environment is associated with a second simulated shadow, and the second portion of the environment is not associated with a simulated shadow different from the first simulated shadow.

194. The individual parts of the environment are the first parts of the environment, and the method is Before displaying the first simulated shadow on the first portion of the environment, display the second simulated shadow on the first portion of the environment, wherein the shadow texture of the second simulated shadow has a third visual appearance. The method according to claim 193, further comprising: changing the visual appearance of the shadow texture of the second simulated shadow to deviate from the third visual appearance, while the first portion of the environment is also associated with the second simulated shadow in response to the display of the first simulated shadow on the first portion of the environment.

195. Displaying the second simulated shadow on the individual part of the environment before displaying the first simulated shadow on the individual part of the environment, The method according to any one of claim 193 or 194, further comprising discontinuing the display of at least a portion of the second simulated shadow on the individual portion of the environment, in accordance with the determination that one or more criteria are met while the first simulated shadow on the individual portion of the environment is also associated with the second simulated shadow, in response to the display of the first simulated shadow on the individual portion of the environment.

196. The method according to any one of claims 188 to 195, wherein the shadow texture of the first simulated shadow has a third visual appearance in the central region of the first simulated shadow, and the shadow texture of the first simulated shadow has a fourth visual appearance different from the third visual appearance in the outer region of the first simulated shadow surrounding the central region of the first simulated shadow.

197. The method according to any one of claims 193 to 196, wherein the shadow texture of the second simulated shadow has a third visual appearance in the central region of the second simulated shadow, and the shadow texture of the second simulated shadow has a fourth visual appearance different from the third visual appearance in the outer region of the second simulated shadow surrounding the central region of the second simulated shadow.

198. The method according to any one of claims 188 to 197, further comprising displaying the first simulated shadow moving away from the individual part of the environment in which the first simulated shadow appears to a second individual part of the environment.

199. The method according to any one of claims 188 to 198, further comprising displaying the first simulated shadow which changes from having a first size and / or shape to having a second size and / or shape different from the first size and / or shape.

200. The method according to any one of claims 188 to 199, further comprising displaying a second simulated shadow on a second separate part of the environment via the display generation component while displaying the first simulated shadow corresponding to the first virtual object on the separate part of the environment.

201. The method further includes displaying one or more simulated reflections corresponding to one or more light sources on the individual part of the environment before displaying the first simulated shadow on the individual part of the environment, The method according to any one of claims 188 to 200, wherein displaying the first simulated shadow on the individual portion of the environment is equivalent to refraining from displaying the one or more simulated reflections corresponding to the one or more light sources on the individual portion of the environment.

202. The method according to claim 201, wherein the individual part of the environment includes simulated sand, and the one or more simulated reflections correspond to one or more simulated reflections from the surface of the simulated sand.

203. The method according to claim 201 or 202, wherein the individual part of the environment includes simulated water, and the one or more simulated reflections correspond to one or more simulated reflections from the surface of the simulated water.

204. To detect an event corresponding to a change in the user's viewpoint from the first viewpoint to the second viewpoint, while the individual part of the environment, which is visible from the user's second viewpoint, displays one or more simulated reflections corresponding to one or more light sources on the individual part of the environment, from the first viewpoint of the user of the computer system; The method according to any one of claims 201 to 203, further comprising: changing the display of the one or more simulated reflections corresponding to the one or more light sources on the individual portion of the environment in response to the detection of the event.

205. The method according to claim 204, wherein changing the display of the one or more simulated reflections corresponding to the one or more light sources on the individual parts of the environment includes gradually changing the display of the one or more simulated reflections after the user's viewpoint has changed to the second viewpoint.

206. The method according to claim 204 or 205, wherein changing the display of the one or more simulated reflections corresponding to the one or more light sources on the individual portion of the environment includes stopping the display of a first one or more simulated reflections on the individual portion of the environment and starting the display of a second one or more simulated reflections on the individual portion of the environment.

207. The method according to any one of claims 188 to 206, further comprising displaying media content and the first simulated shadow simultaneously within the environment.

208. The method according to any one of claims 188 to 207, further comprising displaying the first simulated shadow and one or more virtual elements corresponding to the communication session between the user of the computer system and one or more other participants in the communication session simultaneously within the environment.

209. While the first simulated shadow is displayed on the individual portion of the environment, the environment is visible from a first viewpoint of the user of the computer system, the environment has a first environmental appearance, and the method The further includes detecting an event corresponding to changing the user's viewpoint from the first viewpoint to the second viewpoint while the environment has the first environmental appearance, The method according to any one of claims 188 to 208, wherein, while the environment is visible from the second viewpoint, the environment has a second environmental appearance different from the first environmental appearance.

210. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, A computer system that, when the environment is visible, includes an instruction to display a first simulated shadow corresponding to a first virtual object in the environment via the display generation component, wherein the first simulated shadow has a size and shape based on the size and shape of the virtual object onto which the first simulated shadow is projected and on the individual parts of the environment on which the first simulated shadow appears, and has an individual visual appearance based on the shadow texture of the first simulated shadow, and the display of the first simulated shadow is In accordance with the determination that the individual part of the environment is the first part of the environment, the first simulated shadow is displayed on the individual virtual element using the shadow texture having a first visual appearance, A computer system comprising: displaying the first simulated shadow on the individual virtual element using the shadow texture having a second visual appearance different from the first visual appearance, in accordance with the determination that the individual part of the environment is a second part of the environment, different from the first part of the environment.

211. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system A non-temporary computer-readable storage medium that causes a method to be performed, which includes displaying a first simulated shadow corresponding to a first virtual object in the environment via the display generation component when the environment is visible, wherein the first simulated shadow has a size and shape based on the size and shape of the virtual object onto which the first simulated shadow is projected and on the individual parts of the environment on which the first simulated shadow appears, and has an individual visual appearance based on the shadow texture of the first simulated shadow, and the display of the first simulated shadow is performed. In accordance with the determination that the individual part of the environment is the first part of the environment, the first simulated shadow is displayed on the individual virtual element using the shadow texture having a first visual appearance, A non-temporary computer-readable storage medium, comprising: displaying the first simulated shadow on the individual virtual element using the shadow texture having a second visual appearance different from the first visual appearance, in accordance with the determination that the individual part of the environment is a second part of the environment, different from the first part of the environment.

212. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising: when the environment is visible, means for displaying a first simulated shadow corresponding to a first virtual object in the environment via the display generation component, wherein the first simulated shadow has a size and shape based on the size and shape of the virtual object onto which the first simulated shadow is projected and on the individual parts of the environment on which the first simulated shadow appears, and has an individual visual appearance based on the shadow texture of the first simulated shadow, and the display of the first simulated shadow is In accordance with the determination that the individual part of the environment is the first part of the environment, the first simulated shadow is displayed on the individual virtual element using the shadow texture having a first visual appearance, A computer system comprising: displaying the first simulated shadow on the individual virtual element using the shadow texture having a second visual appearance different from the first visual appearance, in accordance with the determination that the individual part of the environment is a second part of the environment, different from the first part of the environment.

213. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 188 to 209.

214. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 188 to 209.

215. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for carrying out the method described in any one of claims 188 to 209.

216. It is a method, In a computer system that communicates with a display generation component and one or more input devices, A method that, when the environment is visible, displays a background element via the display generation component, the background element comprising one or more first virtual elements in a first layer and one or more second virtual elements in a second layer, wherein the visual appearance of the background element from the current viewpoint of the user of the computer system is based on a combination of the visual appearance of the one or more first virtual elements in the first layer and the visual appearance of the one or more second virtual elements in the second layer, wherein the display of the background element is A method comprising changing the visual appearance of one or more first virtual elements over time in a first manner relative to the visual appearance of one or more second virtual elements.

217. The method according to claim 216, wherein the one or more first virtual elements are one or more simulated astronomical light sources.

218. The method according to claim 216 or 217, wherein the one or more first virtual elements are simulated clouds, and changing the visual appearance of the one or more simulated clouds includes moving the one or more simulated clouds relative to the environment.

219. The method according to any one of claims 216 to 218, wherein the one or more first virtual elements are simulated clouds, and changing the visual appearance of the one or more simulated clouds includes changing the size and / or shape of the one or more simulated clouds.

220. Displaying the aforementioned background elements The method according to any one of claims 216 to 219, comprising changing the visual appearance of one or more second virtual elements over time in a second manner different from the first manner.

221. Displaying the aforementioned background elements The method according to any one of claims 216 to 220, comprising displaying one or more of the first virtual elements over time with one or more animations.

222. The method according to claim 221, wherein the one or more first virtual elements include a simulated sun, and the one or more animations include an animation of the simulated sun.

223. The method according to claim 221 or 222, wherein the one or more first virtual elements include one or more simulated stars, and the one or more animations include one or more animations of the one or more simulated stars.

224. The method according to any one of claims 221 to 223, wherein the one or more first virtual elements include a simulated light source, and the one or more animations include an animation of one or more simulated lighting effects in the environment based on the simulated light source.

225. The method according to any one of claims 221 to 224, wherein the one or more first virtual elements include a simulated moon, and the one or more animations include an animation of the simulated moon.

226. The method according to any one of claims 221 to 225, wherein the one or more animations include changing one or more of the color, location, brightness, and / or movement speed of the one or more virtual elements over time.

227. The method according to any one of claims 216 to 226, further comprising displaying one or more simulated lighting effects on a simulated ground element in the environment based on one or more simulated light sources while the background element is being displayed in the environment.

228. The method according to claim 227, wherein the simulated ground element includes simulated sand, and the one or more simulated lighting effects include simulated lighting effects corresponding to the reflection of simulated light from a simulated light source from the simulated sand.

229. The method according to claim 227 or 228, wherein the simulated ground element includes simulated snow, and the one or more simulated lighting effects include simulated lighting effects corresponding to the reflection of simulated light from a simulated light source from the simulated snow.

230. The method according to any one of claims 227 to 229, wherein the simulated ground element includes simulated water, and the one or more simulated lighting effects include simulated lighting effects corresponding to the reflection of simulated light from a simulated light source from the simulated water.

231. While the background elements are displayed in the environment from the user's first viewpoint, one or more first simulated lighting effects are displayed on the simulated ground elements in the environment based on one or more simulated light sources. Within the environment, based on the one or more simulated light sources, while the simulated ground elements within the environment are displaying the one or more first simulated lighting effects on the simulated ground elements which are visible from the user's second viewpoint, an event corresponding to changing the user's viewpoint from a first viewpoint to a second viewpoint is detected. The method according to any one of claims 227 to 230, further comprising, in response to detecting the event, displaying one or more second simulated lighting effects, different from the one or more first simulated lighting effects, on the simulated ground elements in the environment based on the one or more simulated light sources.

232. The method according to any one of claims 216 to 231, further comprising displaying the background elements and media content simultaneously within the environment.

233. The method according to any one of claims 216 to 232, further comprising displaying the background elements and one or more virtual elements corresponding to the communication session between the user of the computer system and one or more other participants in the communication session, simultaneously within the environment.

234. While the background element is displayed in the environment from the user's first viewpoint, the background element having a first visual appearance is displayed in the environment. In the environment described above, while the background element having the first visual appearance is displayed, an event corresponding to changing the user's viewpoint from the first viewpoint to the second viewpoint is detected. The method according to any one of claims 216 to 233, further comprising displaying the background element having a second visual appearance different from the first visual appearance within the environment in response to the detection of the event.

235. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and One or more programs, An electronic device comprising, wherein one or more programs are stored in the memory and executed by one or more processors, When the environment is visible, a computer system includes, via the display generation component, a background element comprising one or more first virtual elements in a first layer and one or more second virtual elements in a second layer, wherein the visual appearance of the background element from the current viewpoint of the user of the computer system is based on a combination of the visual appearance of the one or more first virtual elements in the first layer and the visual appearance of the one or more second virtual elements in the second layer, and the display of the background element is, A computer system comprising changing the visual appearance of one or more first virtual elements over time in a first manner relative to the visual appearance of one or more second virtual elements.

236. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, the computer system A non-temporary computer-readable storage medium that, when the environment is visible, causes to perform a method that includes displaying a background element comprising one or more first virtual elements in a first layer and one or more second virtual elements in a second layer, wherein the visual appearance of the background element from the current viewpoint of the user of the computer system is based on a combination of the visual appearance of the one or more first virtual elements in the first layer and the visual appearance of the one or more second virtual elements in the second layer, and the display of the background element is performed via the display generation component, A non-temporary computer-readable storage medium comprising changing the visual appearance of one or more first virtual elements over time in a first manner relative to the visual appearance of one or more second virtual elements.

237. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising, when the environment is visible, means for displaying a background element comprising one or more first virtual elements in a first layer and one or more second virtual elements in a second layer, wherein the visual appearance of the background element from the current viewpoint of the user of the computer system is based on a combination of the visual appearance of the one or more first virtual elements in the first layer and the visual appearance of the one or more second virtual elements in the second layer, wherein the display of the background element is, A computer system comprising changing the visual appearance of one or more first virtual elements over time in a first manner relative to the visual appearance of one or more second virtual elements.

238. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing the method according to any one of claims 216 to 234.

239. A non-temporary computer-readable storage medium for storing one or more programs, wherein the one or more programs, when executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, include instructions causing the computer system to perform the method according to any one of claims 216 to 234.

240. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, Memory and A computer system comprising means for performing the method described in any one of claims 216 to 234.

Images