Methods for displaying and repositioning objects within an environment

Abstract

In some embodiments, the computer system selectively recenters virtual content to a user's viewpoint in the presence of physical or virtual obstacles, and / or automatically recenters one or more virtual objects in response to a display generation component changing state, selectively recenters content associated with a communication session between multiple users in response to detected user input, alters visual salience of content included in a virtual object based on a viewpoint and / or based on detected user attention of a user, alters visual salience of one or more virtual objects to resolve apparent obscuration of one or more virtual objects, alters visual salience based on a user viewpoint relative to a virtual object, and simultaneously alters visual salience based on various types of user interactions and / or alters the amount of visual impact of an environmental effect in response to detected user input.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 301,020, filed on January 19, 2022; U.S. Patent Application No. 63 / 377,002, filed on September 23, 2022; and U.S. Provisional Patent Application No. 63 / 480,494, filed on January 18, 2023, the contents of which are hereby incorporated by reference in their entirety for all purposes. (Technical Field)

[0002] This relates generally, but without limitation, to computer systems that provide computer - generated experiences, including electronic devices that provide virtual and mixed reality experiences via a display.

Background Art

[0003] The development of computer systems for augmented reality has advanced significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices such as cameras, controllers, joysticks, touch - sensing surfaces, and touch - screen displays for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements such as buttons and other graphics.

Summary of the Invention

[0004] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and restrictive. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in augmented reality environments, and systems where manipulating virtual objects is complex and error-prone impose a significant cognitive burden on the user and detract from the virtual / augmented reality experience. In addition, these methods are unnecessarily time-consuming, thereby wasting the energy of the computer system. This latter consideration is particularly important in battery-powered devices.

[0005] Therefore, there is a need for computer systems with improved methods and interfaces to provide users with computer-generated experiences that make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can optionally complement or replace conventional methods of providing users with extended reality experiences. Such methods and interfaces reduce the number, extent, and / or types of user input by helping the user understand the connection between the inputs provided and the device response to those inputs, thereby generating a more efficient human-machine interface.

[0006] The above-mentioned drawbacks and other problems associated with the user interface of a computer system are mitigated or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, tablet computer, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a “touchscreen” or “touchscreen display”). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to display-generating components, the output devices include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in memory for performing multiple functions. In some embodiments, the user interacts with the GUI (and / or computer system) through stylus and / or finger touch and gestures on a touch-sensitive surface, the movement of the user's eyes and hands in space relative to the user's body, and / or voice input captured by one or more audio input devices.In some embodiments, the functions performed through interaction optionally include image editing, drawing, presentation, word processing, spreadsheet creation, gameplay, making phone calls, video conferencing, sending emails, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note-taking, and / or digital video playback. The executable instructions for performing those functions optionally reside in a primary computer-readable storage medium and / or a non-primary computer-readable storage medium, or in other computer program products configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can complement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the number, degree, and / or type of user input, resulting in a more efficient human-machine interface. In the case of battery-operated computing devices, such methods and interfaces conserve power and extend the interval between battery charges.

[0008] In some embodiments, the computer system selectively recenters virtual content to the user's viewpoint. In some embodiments, the computer system recenters one or more virtual objects in the presence of physical or virtual obstacles. In some embodiments, the computer system selectively and automatically recenters one or more virtual objects in response to a change in state of a display generation component. In some embodiments, the computer system selectively recenters content associated with a communication session between multiple users in response to input detected in the computer system. In some embodiments, the computer system changes the visual splendor of content contained in a virtual object based on the viewpoint. In some embodiments, the computer system changes the visual splendor of one or more virtual objects based on detected user attention. In some embodiments, the computer system changes the visual splendor of one or more virtual objects to resolve obvious obscuration of one or more virtual objects. In some embodiments, the computer system gradually changes the visual splendor of one or more virtual objects in accordance with the determination that the user's viewpoint corresponds to different regions of a three-dimensional environment. In some embodiments, the computer system changes the visual splendor of one or more parts of a virtual object when the user's viewpoint is close to the virtual object. In some embodiments, the computer system changes the visual splendor of a virtual object when one or more simultaneous types of user interaction are detected. In some embodiments, the computer system changes the amount of visual impact of environmental effects on the three-dimensional environment in which virtual content is displayed, in response to detecting input (one or more) shifting to different elements within the three-dimensional environment (e.g., user attention).

[0009] It should be noted that the various embodiments described herein can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, in particular, in light of the drawings, specification and claims. Furthermore, it should be noted that the language used herein has been selected solely for readability and explanatory purposes and not to define or limit the subject matter of the invention. [Brief explanation of the drawing]

[0010] To better understand the various embodiments described, the following “Modes for Carrying Out the Invention” should be referenced in conjunction with the following drawings, and similar reference numbers throughout the following drawings refer to the corresponding parts.

[0011] [Figure 1] This block diagram shows the operating environment of a computer system for providing an XR experience, according to several embodiments.

[0012] [Figure 2] Block diagram showing a controller for a computer system configured to manage and adjust the user's XR experience, according to several embodiments.

[0013] [Figure 3] This block diagram shows display generation components of a computer system configured to provide users with visual components of an XR experience, according to several embodiments.

[0014] [Figure 4] This is a block diagram showing a hand tracking unit for a computer system configured to capture user gesture input, according to several embodiments.

[0015] [Figure 5]A block diagram showing an eye-tracking unit of a computer system configured to capture a user's gaze input according to some embodiments.

[0016] [Figure 6] A flowchart showing a grint-assisted gaze tracking pipeline according to some embodiments.

[0017] [Figure 7A] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown. [Figure 7B] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown. [Figure 7C] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown. [Figure 7D] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown. [Figure 7E] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown. [Figure 7F] An example of a computer system that selectively recenters virtual content at a user's viewpoint according to some embodiments is shown.

[0018] [Figure 8A] A flowchart showing an exemplary method of selectively recentering virtual content at a user's viewpoint according to some embodiments. [Figure 8B] A flowchart showing an exemplary method of selectively recentering virtual content at a user's viewpoint according to some embodiments. [Figure 8C] A flowchart showing an exemplary method of selectively recentering virtual content at a user's viewpoint according to some embodiments. [Figure 8D] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments. [Figure 8E] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments. [Figure 8F] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments. [Figure 8G] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments. [Figure 8H] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments. [Figure 8I] A flowchart illustrating an exemplary method for selectively recentering virtual content to a user's perspective, according to some embodiments.

[0019] [Figure 9A] Examples of computer systems for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to some embodiments, are shown. [Figure 9B] Examples of computer systems for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to some embodiments, are shown. [Figure 9C] Examples of computer systems for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to some embodiments, are shown.

[0020] [Figure 10A] A flowchart illustrating a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to some embodiments. [Figure 10B]This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. [Figure 10C] This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. [Figure 10D] This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. [Figure 10E] This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. [Figure 10F] This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. [Figure 10G] This flowchart illustrates a method for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments.

[0021] [Figure 11A] This document describes an embodiment of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in the state of a display generation component, according to several embodiments. [Figure 11B] This document describes an embodiment of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in the state of a display generation component, according to several embodiments. [Figure 11C] This document describes an embodiment of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in the state of a display generation component, according to several embodiments. [Figure 11D] This document describes an embodiment of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in the state of a display generation component, according to several embodiments. [Figure 11E] This document describes an embodiment of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in the state of a display generation component, according to several embodiments.

[0022] [Figure 12A] This flowchart shows a method, according to several embodiments, for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component. [Figure 12B] This flowchart shows a method, according to several embodiments, for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component. [Figure 12C] This flowchart shows a method, according to several embodiments, for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component. [Figure 12D] This flowchart shows a method, according to several embodiments, for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component. [Figure 12E] This flowchart shows a method, according to several embodiments, for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component.

[0023] [Figure 13A] This document presents an example of a computer system that selectively recenters content associated with communication sessions between multiple users in response to input detected in the computer system, according to several embodiments. [Figure 13B] This document presents an example of a computer system that selectively recenters content associated with communication sessions between multiple users in response to input detected in the computer system, according to several embodiments. [Figure 13C]This document presents an example of a computer system that selectively recenters content associated with communication sessions between multiple users in response to input detected in the computer system, according to several embodiments.

[0024] [Figure 14A] This flowchart shows a method, according to several embodiments, for selectively recentering content associated with communication sessions between multiple users in response to detected input in a computer system. [Figure 14B] This flowchart shows a method, according to several embodiments, for selectively recentering content associated with communication sessions between multiple users in response to detected input in a computer system. [Figure 14C] This flowchart shows a method, according to several embodiments, for selectively recentering content associated with communication sessions between multiple users in response to detected input in a computer system. [Figure 14D] This flowchart shows a method, according to several embodiments, for selectively recentering content associated with communication sessions between multiple users in response to detected input in a computer system. [Figure 14E] This flowchart shows a method, according to several embodiments, for selectively recentering content associated with communication sessions between multiple users in response to detected input in a computer system.

[0025] [Figure 15A] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15B] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15C]This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15D] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15E] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15F] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15G] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15H] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15I] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments. [Figure 15J] This document presents an example of a computer system that modifies the visual prominence of content contained in a virtual object based on viewpoint, according to several embodiments.

[0026] [Figure 16A] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16B] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16C]This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16D] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16E] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16F] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16G] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16H] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16I] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16J] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16K] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16L] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16M] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16N] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16O] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint. [Figure 16P] This flowchart shows a method, according to several embodiments, for changing the visual saturation of content contained in a virtual object based on viewpoint.

[0027] [Figure 17A] This document describes an embodiment of a computer system that modifies the visual prominence of content contained in a virtual object based on the attention of a user of the computer system, according to several embodiments. [Figure 17B] This document describes an embodiment of a computer system that modifies the visual prominence of content contained in a virtual object based on the attention of a user of the computer system, according to several embodiments. [Figure 17C] This document describes an embodiment of a computer system that modifies the visual prominence of content contained in a virtual object based on the attention of a user of the computer system, according to several embodiments. [Figure 17D] This document describes an embodiment of a computer system that modifies the visual prominence of content contained in a virtual object based on the attention of a user of the computer system, according to several embodiments. [Figure 17E] This document describes an embodiment of a computer system that modifies the visual prominence of content contained in a virtual object based on the attention of a user of the computer system, according to several embodiments.

[0028] [Figure 18A] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18B]This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18C] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18D] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18E] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18F] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18G] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18H] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18I] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18J] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments. [Figure 18K] This flowchart shows a method for changing the visual prominence of a virtual object based on user attention, according to several embodiments.

[0029] [Figure 19A] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 19B] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 19C] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 19D] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 19E] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments.

[0030] [Figure 20A] This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 20B] This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 20C] This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 20D]This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 20E] This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments. [Figure 20F] This flowchart shows a method for changing the visual prominence of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content, according to several embodiments.

[0031] [Figure 21A] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21B] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21C] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21D] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21E] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21F] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21G]This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21H] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21I] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21J] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21K] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments. [Figure 21L] This document presents an example of a computer system that gradually changes the visual prominence of each virtual object in accordance with a change in the user's viewpoint, according to several embodiments.

[0032] [Figure 22A] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22B] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22C] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22D] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22E] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22F] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22G] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22H] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22I] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint. [Figure 22J] This flowchart shows a method, according to several embodiments, for gradually changing the visual prominence of each virtual object in accordance with a change in the user's viewpoint.

[0033] [Figure 23A] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object based on the user's proximity to that virtual object, according to several embodiments. [Figure 23B] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object based on the user's proximity to that virtual object, according to several embodiments. [Figure 23C] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object based on the user's proximity to that virtual object, according to several embodiments. [Figure 23D]This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object based on the user's proximity to that virtual object, according to several embodiments. [Figure 23E] This document describes an embodiment of a computer system that modifies the visual prominence of each virtual object based on the user's proximity to that virtual object, according to several embodiments.

[0034] [Figure 24A] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object. [Figure 24B] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object. [Figure 24C] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object. [Figure 24D] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object. [Figure 24E] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object. [Figure 24F] This flowchart shows a method, according to several embodiments, for changing the visual prominence of each virtual object based on the user's proximity to that virtual object.

[0035] [Figure 25A]This document presents an example of a computer system that modifies the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments. [Figure 25B] This document presents an example of a computer system that modifies the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments. [Figure 25C] This document presents an example of a computer system that modifies the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments.

[0036] [Figure 26A] This flowchart shows a method for changing the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments. [Figure 26B] This flowchart shows a method for changing the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments. [Figure 26C] This flowchart shows a method for changing the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments. [Figure 26D] This flowchart shows a method for changing the visual prominence of each virtual object based on one or more simultaneous user interactions, according to several embodiments.

[0037] [Figure 27A] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27B]This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27C] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27D] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27E] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27F] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27G] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27H]This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27I] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment. [Figure 27J] This document describes an embodiment of a computer system that simultaneously displays virtual content and environmental effects with varying amounts of visual impact on the three-dimensional environment, in response to the computer system detecting input (e.g., user attention) shifting to different elements within a three-dimensional environment.

[0038] [Figure 28A] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28B] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28C] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28D]This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28E] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28F] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28G] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28H] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Figure 28I] This flowchart shows a method for dynamically displaying environmental effects, with varying amounts of visual impact on the appearance of a three-dimensional environment, in response to detecting inputs (e.g., user attention) shifting to different elements within the three-dimensional environment, according to several embodiments. [Modes for carrying out the invention]

[0039] This disclosure relates to user interfaces that provide users with Extended Reality (XR) experiences, in several embodiments.

[0040] The systems, methods, and GUIs described herein facilitate interaction between electronic devices and objects in a three-dimensional environment and provide improved methods for manipulating objects.

[0041] In some embodiments, the computer system displays virtual objects in the environment. In some embodiments, in response to input to recenter virtual objects to the user's viewpoint, the computer system recenters virtual objects that meet certain criteria and does not recenter virtual objects that do not meet such criteria. In some embodiments, virtual objects that are snapped to parts of the physical environment are not recentered. In some embodiments, virtual objects that were last placed or moved into the environment from the user's current viewpoint are not recentered.

[0042] In some embodiments, the computer system displays virtual objects in the environment. In some embodiments, in response to input that recenters the virtual objects to the user's viewpoint, the computer system avoids physical objects when recentering those virtual objects. In some embodiments, the computer system avoids virtual objects when recentering other virtual objects.

[0043] In some embodiments, the computer system displays virtual objects in the environment from a first viewpoint. In some embodiments, when the state of the computer system changes (for example, from on to off, and then back on), the computer system automatically recenters the virtual objects to the new viewpoint depending on one or more characteristics of the new viewpoint. In some embodiments, the computer system does not automatically recenter the virtual objects to the new viewpoint.

[0044] In some embodiments, the computer system displays virtual objects in an environment where the virtual objects are accessible to multiple computer systems. In some embodiments, in response to input that recenters the virtual objects to the user's viewpoint, the computer system does not change the spatial arrangement of virtual objects accessible to multiple computer systems relative to the viewpoint associated with those multiple computer systems. In some embodiments, the computer system changes the spatial arrangement of virtual objects that are not accessible to other computer systems relative to the viewpoint associated with this computer system.

[0045] In some embodiments, the computer system displays virtual objects containing content within an environment. In some embodiments, the computer system displays content with different visual splendor depending on the angle from which the content is visible from the user's current viewpoint. In some embodiments, visual splendor is greater when the angle is closer to the front and less when the angle is further from the front. In some embodiments, the computer system modifies the visual splendor of one or more virtual objects based on detected user attention. In some embodiments, the computer system modifies the visual splendor of one or more virtual objects to resolve obvious obscuration of one or more virtual objects.

[0046] Figures 1 to 6 provide a description of exemplary computer systems for providing an XR experience to a user (as described below with respect to methods 800, 1000, 1200, 1400, 1600, 1800, and / or 2000). Figures 7A to 7F show examples of computer systems that selectively recenter virtual content to the user's viewpoint, according to several embodiments. Figures 8A to 8I are flowcharts illustrating exemplary methods of selectively recentering virtual content to the user's viewpoint, according to several embodiments. The user interfaces in Figures 7A to 7F are used to illustrate the processes in Figures 8A to 8I. Figures 9A to 9C show examples of computer systems that recenter one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. Figures 10A to 10G are flowcharts illustrating methods of recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. The user interfaces in Figures 9A to 9C are used to illustrate the processes in Figures 10A to 10G. Figures 11A to 11E show embodiments of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in state of a display generation component. Figures 12A to 12E are flowcharts showing methods for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component, according to several embodiments. The user interfaces in Figures 11A to 11E are used to illustrate the processes in Figures 12A to 12E. Figures 13A to 13C show embodiments of a computer system that selectively recenters content associated with a communication session between multiple users in response to input detected in the computer system. Figures 14A to 14E are flowcharts showing methods for selectively recentering content associated with a communication session between multiple users in response to input detected in the computer system, according to several embodiments. The user interfaces in Figures 13A to 13C are used to illustrate the processes in Figures 14A to 14E.Figures 15A to 15J show embodiments of a computer system that changes the visual saturation of content contained in a virtual object based on viewpoint, according to several embodiments. Figures 16A to 16P are flowcharts of methods for changing the visual saturation of content contained in a virtual object based on viewpoint, according to several embodiments. The user interfaces in Figures 15A to 15J are used to illustrate the processes in Figures 16A to 16P. Figures 17A to 17E show embodiments of a computer system that changes the visual saturation of content contained in a virtual object based on the attention of the computer system user, according to several embodiments. Figures 18A to 18K are flowcharts of methods for changing the visual saturation of a virtual object based on the attention of the user, according to several embodiments. The user interfaces in Figures 17A to 17E are used to illustrate the processes in Figures 18A to 18K. Figures 19A to 19E show embodiments of a computer system that changes the visual saturation of each virtual object in order to change the obvious obscuration of each virtual object by the virtual content, according to several embodiments. Figures 20A to 20F are flowcharts illustrating methods, according to several embodiments, for changing the visual splendor of each virtual object in order to alter the apparent obscuration of each virtual object by virtual content. The user interfaces in Figures 19A to 19E are used to illustrate the processes in Figures 20A to 20F. Figures 21A to 21L show examples of computer systems, according to several embodiments, that gradually change the visual splendor of each virtual object in accordance with a change in the user's viewpoint. Figures 22A to 22J are flowcharts illustrating methods, according to several embodiments, for gradually changing the visual splendor of each virtual object in accordance with a change in the user's viewpoint. The user interfaces in Figures 21A to 21L are used to illustrate the processes in Figures 22A to 22J. Figures 23A to 23E show examples of computer systems, according to several embodiments, that change the visual splendor of each virtual object based on the user's proximity to each virtual object.Figures 24A to 24F are flowcharts illustrating, in several embodiments, methods for changing the visual splendor of each virtual object based on the user's proximity to each virtual object. The user interfaces in Figures 23A to 23E are used to illustrate the processes in Figures 24A to 24F. Figures 25A to 25C show examples of a computer system that changes the visual splendor of each virtual object based on one or more simultaneous user interactions, in several embodiments. Figures 26A to 26D are flowcharts illustrating, in several embodiments, methods for changing the visual splendor of each virtual object based on one or more simultaneous user interactions. The user interfaces in Figures 25A to 25C are used to illustrate the processes in Figures 26A to 26D. Figures 27A to 27J show examples of a computer system that changes the amount of visual impact of environmental effects on the appearance of a three-dimensional environment in which a first virtual content is displayed, in response to the detection of inputs such as the user's attention shifted away from the first virtual content, and / or other inputs different from the user's attention directed to elements different from the first virtual content. Figures 28A to 28I are flowcharts illustrating how, in several embodiments, environmental effects are dynamically displayed, involving varying amounts of visual impact on the appearance of a three-dimensional environment, in response to the detection of inputs (e.g., user attention) shifting to different elements within the three-dimensional environment. The user interfaces in Figures 27A to 27J are used to illustrate the processes in Figures 28A to 28I.

[0047] The processes described below enhance the usability of the device and streamline the user-device interface by providing improved visual feedback to the user, reducing the number of inputs required to perform actions, offering additional control options without cluttering the user interface with additional controls displayed, performing actions without requiring further user input when a set of conditions is met, improving privacy and / or security, and / or other technologies. These technologies also reduce power consumption and improve the battery life of the device by enabling users to use the device more quickly and efficiently.

[0048] Furthermore, in any method described herein that is conditional on one or more conditions being met in one or more steps, it should be understood that the method described can be repeated in multiple iterations such that all the conditions that the steps of the method are conditional on are met in different iterations of the method. For example, if a method requires that a first step be performed if a condition is met, and a second step be performed if the condition is not met, a person skilled in the art will understand that the steps described in the claim are repeated in a specific order until the conditions are met and then not met. Thus, a method described in one or more steps that depends on one or more conditions being met can be rewritten as a method that is repeated until each of the conditions described in the method is met. However, this is not required for a claim of a system or computer-readable medium that includes instructions for performing a conditional operation based on the satisfaction of the corresponding one or more conditions, and thus can determine whether a contingency has been met without explicitly repeating the steps of the method until all the conditions that the steps of the method are conditional on are met. Those skilled in the art will also understand that, as with a method having conditional steps, a system or computer-readable storage medium may repeat the steps of the method as many times as necessary to ensure that all of the conditional steps have been performed.

[0049] In some embodiments, as shown in Figure 1, the XR experience is provided to the user via an operating environment 100 which includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, or a touchscreen), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, and other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, or a velocity sensor), and optionally one or more peripheral devices 195 (e.g., a consumer electronics product or a wearable device). In some embodiments, one or more of the input device 125, output device 155, sensor 190, and peripheral device 195 are integrated with the display generation component 120 (for example, within a head-mounted device or handheld device).

[0050] When describing an XR experience, various terms are used to refer individually to several related but distinct environments that the user perceives and / or interacts with (for example, using inputs detected by the computer system 101, which causes the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101 that generates the XR experience). The following is a subset of these terms.

[0051] Physical Environment: The physical environment refers to the physical world that people can perceive and / or interact with without the help of electronic systems. Examples of physical environments, such as a physical park, include physical objects such as physical trees, physical buildings, and physical people. People can directly perceive and / or interact with the physical environment through their senses of sight, touch, hearing, taste, and smell.

[0052] Extended reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people perceive and / or interact with through an electronic system. In XR, a subset of a person's bodily movements or their representations are tracked, and accordingly, one or more properties of one or more virtual objects simulated within the XR environment are adjusted to behave according to at least one law of physics. For example, an XR system can detect a person's head rotation and, accordingly, adjust the graphic content and sound field presented to the person in a similar way to how such views and sounds would change in a physical environment. Depending on the circumstances (e.g., for reasons of accessibility), adjustments to the properties(s) of virtual objects(s) in the XR environment may be made in response to representations of bodily movements (e.g., voice commands). A person may perceive and / or interact with XR objects using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person can perceive and / or interact with audio objects that create a 3D or spatial audio environment, providing the perception of point audio sources in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person may perceive and / or interact with only audio objects.

[0053] Examples of XR include virtual reality and mixed reality.

[0054] Virtual reality: A virtual reality (VR) environment refers to a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can perceive and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can perceive and / or interact with virtual objects in a VR environment through a simulation of their presence within the computer-generated environment and / or through a simulation of a subset of their physical movement within the computer-generated environment.

[0055] Mixed Reality: A mixed reality (MR) environment is a simulated environment designed to incorporate sensory input or its representation from a physical environment, in addition to including computer-generated sensory input (e.g., virtual objects), in contrast to a virtual reality (VR) environment designed to rely entirely on computer-generated sensory input. On a virtual continuum, a mixed reality environment is any location between, but not including, the complete physical environment at one end and the virtual reality environment at the other end. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Also, some electronic systems for presenting an MR environment may track location and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical articles or their representations from the physical environment). For example, the system may take movement into account so that a virtual tree appears stationary relative to the physical ground.

[0056] Examples of mixed reality include augmented reality and augmented virtual reality.

[0057] Extended reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on or onto a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display that allows a person to directly view the physical environment. The system may also be configured to present virtual objects on the transparent or translucent display, thereby allowing a person to use the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture an image or video of the physical environment, which is a representation of the physical environment. The system composites the image or video with the virtual objects and presents the composite on the opaque display. A person uses this system to perceive the virtual objects superimposed on the physical environment by indirectly viewing the physical environment through the image or video of the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example, as holograms, into or onto the physical environment, so that a person can use the system to perceive the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which the representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may transform one or more sensor images to plane a selected perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion of it, so that the modified portion is a non-photorealistic altered version of the original captured image. As yet another example, the representation of the physical environment may be transformed by graphically removing or obscuring a portion of it.

[0058] Augmented Virtuality (AV) refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. These sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park might have virtual trees and buildings, but people with faces might be realistically reproduced from images of real people. Another example is that a virtual object might adopt the shape or color of a physical article captured by one or more imaging sensors. A further example is that a virtual object might adopt shadows that correspond to the position of the sun in the physical environment.

[0059] Viewpoint-locked virtual objects: A virtual object is viewpoint-locked when the computer system displays the virtual object in the same location and / or position within the user's view, even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked in the forward direction of the user's head (e.g., the user's viewpoint is at least a portion of the user's field of view when the user is looking straight ahead). Thus, the user's viewpoint remains fixed even if the user's gaze moves, without moving the user's head. In embodiments where the computer system has a display generation component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper-left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) will continue to be displayed in the upper-left corner of the user's viewpoint even if the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the location and / or position in which a viewpoint-locked virtual object is displayed from the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so that the virtual object is also referred to as a "head-locked virtual object."

[0060] Environment-Locked Virtual Objects: A virtual object is environment-locked (or "world-locked") when a computer system displays it at a location and / or position in the user's viewpoint that is based on (e.g., selected by reference to and / or fixed to) a location and / or object in a three-dimensional environment (e.g., a physical or virtual environment). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, and as a result, the environment-locked virtual object will appear at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree directly in front of the user will appear at the center of the user's viewpoint. If the user's viewpoint shifts to the right (e.g., the user's head is turned to the right) and the tree becomes left-leaning in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree will appear left-leaning in the user's viewpoint. In other words, the location and / or position in which an environment-locked virtual object is displayed in the user's viewpoint depends on the location and / or object's position and / or orientation in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a fixed location in the physical environment and / or a coordinate system fixed to an object) to determine the position in which the environment-locked virtual object is displayed from the user's viewpoint. The environment-locked virtual object can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object) or to a moving part of the environment (e.g., a vehicle, animal, person, or a representation of a part of the user's body that moves independently of the user's viewpoint, such as the user's hands, wrists, arms, or feet), so that the virtual object moves as the viewpoint or the part of the environment moves in order to maintain a fixed relationship between the virtual object and the part of the environment.

[0061] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits delayed tracking behavior, reducing or delaying its movement in response to the movement of a reference point that the virtual object is following. In some embodiments, when exhibiting delayed tracking behavior, the computer system detects movement of the reference point that the virtual object is following (e.g., a part of the environment, a viewpoint, or a point fixed to the viewpoint, such as a point between 5 and 300 cm from the viewpoint) and intentionally delays the movement of the virtual object. For example, when the reference point (e.g., a part of the environment or the viewpoint) moves at a first velocity, the virtual object is moved by the device so as to remain locked to the reference point, but at a second velocity slower than the first velocity (e.g., the virtual object begins to catch up to the reference point until the reference point stops or slows down). In some embodiments, when a virtual object exhibits delayed tracking behavior, the device ignores small movements of the reference point (e.g., ignoring movements of the reference point that are below a threshold movement amount, such as a movement of 0 to 5 degrees or a movement of 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint from which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object first increases (e.g., because the virtual object is displayed to maintain a fixed or substantially fixed position relative to a different viewpoint or part of the environment from which the virtual object is locked), and then decreases as the amount of movement of the reference point increases beyond a threshold (e.g., a "delayed tracking" threshold) as the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point.In some embodiments, a virtual object that maintains a substantially fixed position with respect to a reference point includes the virtual object being displayed within a threshold distance (e.g., 1, 2, 3, 5, 15, 20, 50 cm) of the reference point in one or more dimensions (e.g., above / below, left / right, and / or forward / behind the position of the reference point).

[0062] Hardware: There are many different types of electronic systems that enable a person to perceive and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be positioned over a person's eyes (e.g., contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or videos of the physical environment and / or one or more microphones for capturing sounds of the physical environment. A head-mounted system may have a transparent or translucent display instead of an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical coupler, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. The projection-based system may employ retinal projection technology to project a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or onto a physical surface. In some embodiments, the controller 110 is configured to manage and adjust the XR experience for the user.In some embodiments, the controller 110 includes a preferred combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to Figure 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server located outside the scene 105 (e.g., a cloud server or a central server). In some embodiments, the controller 110 is communicably coupled to a display generation component 120 (e.g., an HMD, display, projector, or touchscreen) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, or IEEE 802.3x). In another example, the controller 110 is contained within a housing (e.g., a physical housing) of one or more of the display generation components 120 (e.g., an HMD, or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0063] In some embodiments, the display generation component 120 is configured to provide the user with an XR experience (e.g., at least the visual components of the XR experience). In some embodiments, the display generation component 120 includes a preferred combination of software, firmware, and / or hardware. The display generation component 120 is described in more detail below with reference to Figure 3. In some embodiments, the functions of the controller 110 are provided by and / or combined with the display generation component 120.

[0064] According to some embodiments, the display generation component 120 provides the user with an XR experience while the user is virtually and / or physically present in the scene 105.

[0065] In some embodiments, the display generation component is mounted on a part of the user's body (e.g., their head or hand). Thus, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device, which has a display directed towards the user's field of view and a camera directed towards scene 105. In some embodiments, the handheld device is optionally placed in a housing mounted on the user's head. In some embodiments, the handheld device is optionally placed on a support in front of the user (e.g., a tripod). In some embodiments, the display generation component 120 is an XR chamber, housing, or room configured to present XR content when the user is not wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interaction with XR content triggered based on interaction occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the XR content response is displayed through the HMD. Similarly, a user interface showing interaction with XR content triggered based on the movement of a handheld or tripod-mounted device relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)) may be implemented similarly to an HMD where the movement is triggered by the movement of the HMD relative to a physical environment (e.g., Scene 105 or a part of the user's body (e.g., the user's eyes, head, or hands)).

[0066] While relevant features of the operating environment 100 are shown in Figure 1, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more appropriate embodiments of the exemplary embodiments disclosed herein.

[0067] Figure 2 is a block diagram of an example of the controller 110 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), central processing unit (CPU), processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global Mobile Communication System (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZiGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0068] In some embodiments, one or more communication buses 204 include a circuit configuration for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices 206 include at least one of the following: a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0069] Memory 220 includes high-speed random-access memory such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-temporary computer-readable storage medium. In some embodiments, memory 220, or the non-temporary computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 230 and XR experience module 240.

[0070] The operating system 230 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate one or more XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for each group of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0071] In some embodiments, the data acquisition unit 241 is configured to acquire data (e.g., presentation data, interaction data, sensor data, or location data) from at least the display generation component 120 of Figure 1, and optionally from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0072] In some embodiments, the tracking unit 242 is configured to map scene 105 and track the position / location of at least the display generation component 120 relative to scene 105 in Figure 1, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for this purpose, as well as heuristics and metadata for this purpose. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand, and / or the movement of one or more parts of the user's hand relative to scene 105 in Figure 1, relative to the display generation component 120, and / or relative to a coordinate system defined for the user's hand. The hand tracking unit 244 is described in more detail below with respect to Figure 4. In some embodiments, the eye-tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to the scene 105 (e.g., the physical environment and / or the user (e.g., the user's hands)) or to XR content displayed via the display generation component 120. The eye-tracking unit 243 is described in more detail below with reference to Figure 5.

[0073] In some embodiments, the adjustment unit 246 is configured to manage and adjust the XR experience presented to the user by the display generation component 120 and optionally by one or more of the output devices 155 and / or peripheral devices 195. For this purpose, in various embodiments, the adjustment unit 246 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0074] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data or location data) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0075] While the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (including, for example, eye-tracking unit 243 and hand-tracking unit 244), adjustment unit 246, and data transmission unit 248 may be located in separate computing devices.

[0076] Furthermore, Figure 2 is intended to illustrate the functionality of various features that may be present in a particular embodiment, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 2 can be implemented within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0077] Figure 3 is a block diagram of an example of a display generation component 120 according to several embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features have been omitted for brevity so as not to obscure more suitable embodiments of the embodiments disclosed herein. For that purpose, in some non-limiting examples, the display generation component 120 (e.g., HMD) may include one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, Bluetooth, ZiGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional in-facing and / or out-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0078] In some embodiments, one or more communication buses 304 include a circuit configuration for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include at least one of the following: an inertial measuring unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, or a blood glucose sensor), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.).

[0079] In some embodiments, one or more XR displays 312 are configured to provide the user with an XR experience. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, and / or holographic waveguide displays. For example, a display generation component 120 (e.g., HMD) includes a single XR display. In another embodiment, the display generation component 120 includes an XR display for each of the user's eyes. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0080] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hands and optionally a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene that the user would view if a display generation component 120 (e.g., an HMD) were not present (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., complementary metal-oxide-semiconductor (CMOS) image sensors or charge-coupled device (CCD) image sensors), one or more infrared (IR) cameras, one or more event-based cameras, and / or similar.

[0081] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-temporary computer-readable storage medium. In some embodiments, memory 320, or the non-temporary computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or subsets thereof, including an optional operating system 330 and XR presentation module 340.

[0082] The operating system 330 includes instructions for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the XR presentation module 340 is configured to present XR content to the user via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0083] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, or location data) from at least the controller 110 in Figure 1. For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0084] In some embodiments, the XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation unit 344 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0085] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (for example, a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate extended reality) based on media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0086] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data or location data) to at least the controller 110 and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. For this purpose, in various embodiments, the data transmission unit 348 includes instructions and / or logic for that purpose, as well as heuristics and metadata for that purpose.

[0087] While the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 are shown as existing on a single device (e.g., the display generation component 120 in Figure 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, XR presentation unit 344, XR map generation unit 346, and data transmission unit 348 may be located in separate computing devices.

[0088] Furthermore, Figure 3 is intended to illustrate the functionality of various features that may be present in a particular implementation, in contrast to the structural schematics of the embodiments described herein. As will be recognized by those skilled in the art, the separately shown items can be combined, and some items can be separated. For example, several functional modules shown separately in Figure 3 can be realized within a single module, and the various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of certain functions and how functions are assigned between them, will vary depending on the implementation and, in some embodiments, will partially depend on a particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0089] Figure 4 is a schematic diagram of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 (Figure 1) is controlled by the hand tracking unit 244 (Figure 2) to track the position / location of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of Figure 1 (e.g., relative to a part of the physical environment surrounding the user, relative to the display generation component 120, or relative to a part of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined for the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0090] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures a hand image with sufficient resolution to allow for the distinction of fingers and their respective positions. The image sensor 404 can typically capture images of other parts of the user's body, or images of the entire body, and may have either a zoom function or a dedicated sensor with high magnification to capture an image of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures a 2D color video image of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor or a portion thereof is used to define an interaction space in which hand movements captured by the image sensor are processed as input to the controller 110.

[0091] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, thereby extracting high-level information from the map data. This high-level information is typically provided to an application running on the controller via an application programming interface (API), which drives the display generation components 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving their hand 406 to change the orientation of their hand.

[0092] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spot in the pattern. This approach is advantageous in that the user does not need to hold or wear any kind of beacon, sensor, or other marker. This gives the depth coordinates of points in the scene relative to a given reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines a set of orthogonal x, y, and z axes such that the depth coordinates of points in the scene correspond to a z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods such as stereoscopic imaging or time-of-flight measurement based on one or more cameras or other types of sensors.

[0093] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the processor in the image sensor 404 and / or controller 110 processes the 3D map data to extract patch descriptors of the hand within these depth maps. Based on previous training, the software matches these descriptors against patch descriptors stored in the database 408 to estimate the hand pose in each frame. The pose typically includes the 3D location of the user's wrist and fingertips.

[0094] The software can also analyze the trajectory of the hand and / or fingers across multiple frames in a sequence to identify gestures. The posture estimation function described herein may be interleaved with the motion tracking function, so that patch-based posture estimation is performed only once every two (or more) frames, while tracking is used to detect changes in posture that occur over the remaining frames. Posture, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify the image presented on the display generation component 120, or perform other functions, depending on the posture and / or gesture information.

[0095] In some embodiments, the gesture includes an air gesture. An air gesture is a gesture detected by the user without (or independently of) touching an input element that is part of a device (e.g., a computer system 101, one or more input devices 125, and / or a hand tracking device 140), and is based on detected movement of a part of the user's body in the air (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs), including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture including movement of the hand in a predetermined posture by a predetermined amount and / or speed, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body).

[0096] In some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures, as in some embodiments, performed by moving one or more of the user's fingers relative to other fingers or parts of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, an air gesture is a gesture detected without the user touching (or independently of) an input element that is part of the device, and is based on detected movement of a part of the user's body in the air, including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground, or the distance of the user's hand relative to the ground), movement of the user's body relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of the user's other hand relative to one hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture involving movement of the hand in a predetermined pose by a predetermined amount and / or speed, or a shake gesture involving rotation of a part of the user's body by a predetermined speed or amount).

[0097] In some embodiments where the input gesture is an air gesture (i.e., without physical contact with an input device that provides the computer system with information about which user interface element is the target of user input, such as contact with a user interface element displayed on a touchscreen or contact with a mouse or trackpad to move a cursor over a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of user input (e.g., in the case of direct input, as described below). Thus, in implementations involving air gestures, the input gesture is the detected attention (e.g., gaze) to the user interface element in combination (e.g., simultaneously) with the movement of the user's fingers (one or more) and / or hand to perform pinch and / or tap input, as described in more detail below.

[0098] In some embodiments, input gestures directed towards a user interface object are performed directly or indirectly by reference to the user interface object. For example, user input is performed directly towards the user interface object in response to the user performing an input gesture with their hand at a position corresponding to the user interface object's position in a three-dimensional environment (e.g., determined based on the user's current viewpoint). In some embodiments, the input gesture is performed indirectly towards the user interface object according to the user performing the input gesture while the user's hand position is not at a position corresponding to the user interface object's position in a three-dimensional environment, while detecting the user's attention (e.g., gaze) to the user interface object. For example, in the case of a direct input gesture, the user can direct their input towards the user interface object by initiating the gesture at or near a position corresponding to the user interface object's display position (e.g., within a distance of 0.5 cm, 1 cm, 5 cm, or 0-5 cm from the optional outer edge or optional central portion). In the case of indirect input gestures, the user can direct their input towards the user interface object by paying attention to the user interface object (for example, by gazing at the user interface object), and while paying attention to the options, the user initiates the input gesture (for example, at any position detectable by the computer system) (for example, at a position that does not correspond to the display position of the user interface object).

[0099] In some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch and tap inputs for interacting with virtual or mixed reality environments, as in some embodiments. For example, the pinch and tap inputs described later are performed as air gestures.

[0100] In some embodiments, a pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch-and-drag gesture, or a double pinch gesture. For example, a pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other, i.e., including an optional interruption (e.g., within 0 to 1 second) immediately after the touch. A long pinch gesture that is an air gesture involves moving two or more fingers of a hand to touch each other for at least a threshold time amount (e.g., at least 1 second) before detecting an interruption of contact between them. For example, a long pinch gesture includes the user holding a pinch gesture (e.g., if two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture that is an air gesture includes two (e.g., or more) pinch inputs (e.g., performed with the same hand) that are detected directly and consecutively (e.g., within a predetermined period of time) to each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and then performs a second pinch input within a predetermined period (e.g., within 1 second or 2 seconds) after releasing the first pinch input.

[0101] In some embodiments, an air gesture, a pinch-and-drag gesture, includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in relation to (e.g., after) a drag input that changes the user's hand position from a first position (e.g., a drag initiation position) to a second position (e.g., a resistance termination position). In some embodiments, the user maintains the pinch gesture while performing the drag input and releases the pinch gesture (e.g., spreading two or more fingers) to terminate the drag gesture (e.g., at the second position). In some embodiments, the pinch input and drag input are performed by the same hand (e.g., the user pinches two or more fingers together and touches them to each other, and then moves the same hand to a second position in the air with a drag gesture). In some embodiments, the pinch input is performed by the user's first hand and the drag input is performed by the user's second hand (e.g., the user's second hand moves from the first position to the second position in the air while the user continues the pinch input with the user's first hand). In some embodiments, an input gesture that is an air gesture includes an input (e.g., a pinch input and / or a tap input) performed using both of the user's hands. For example, an input gesture includes two (e.g., or more) pinch inputs performed in relation to each other (e.g., simultaneously or within a predetermined period of time). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) performed using the user's first hand, and a second pinch input performed using the other hand (e.g., a second hand of the user's hands) in relation to performing a pinch input using the first hand. In some embodiments, movement between the user's hands (e.g., to increase and / or decrease the distance or relative orientation between the user's hands).

[0102] In some embodiments, a tap input performed as an air gesture (e.g., directed towards a user interface element) includes the movement of one or more of the user's fingers toward the user interface element, the movement of the user's hand toward the user interface element with the user's fingers (one or more) optionally extended toward the user interface element, a downward movement of the user's fingers (e.g., mimicking a mouse click or a tap on a touchscreen), or other default movements of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand that performs the tap gesture movement away from the user's viewpoint and / or toward the object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand that performs the tap gesture (e.g., away from the user's viewpoint and / or the end of the movement toward the object that is the target of the tap input, a reversal of the direction of the finger or hand movement, and / or a reversal of the direction of acceleration of the finger or hand movement).

[0103] In some embodiments, the user's attention is determined to be directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards that part of the three-dimensional environment (optionally, without requiring any other conditions). In some embodiments, for the device to determine that the user's attention is directed towards a part of the three-dimensional environment, the device determines that the user's attention is directed towards a part of the three-dimensional environment based on the detection of a gaze directed towards a part of the three-dimensional environment, with one or more additional conditions such as the gaze being directed towards the part of the three-dimensional environment for at least a threshold duration (e.g., dwell time) while the user's viewpoint is within a distance threshold from the part of the three-dimensional environment, and / or the gaze being directed towards a part of the three-dimensional environment. If one of the additional conditions is not met, the device determines that the user's attention is not directed towards the part of the three-dimensional environment to which the gaze is directed (e.g., until one or more additional conditions are met).

[0104] In some embodiments, the detection of a ready state configuration of the user or a part of the user is detected by the computer system. The detection of a ready state configuration of the hand is used by the computer system as an indication that the user is likely to be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of a hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape where the thumb and one or more fingers are extended and spaced apart, ready to perform a pinch or grab gesture, or a pre-tap shape where one or more fingers are extended and the palm is facing away from the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head, above the user's waist, or extended at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular way (e.g., moved towards the area in front of the user above the user's waist, below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of the user interface is responsive to attention (e.g., gaze) input.

[0105] In some embodiments, the software may be downloaded electronically to the controller 110, for example, over a network, or instead, it may be provided on a tangible non-temporary medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively or additionally, some or all of the computer's described functions may be implemented in dedicated hardware such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in Figure 4, for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit configuration within the housing of the image sensor 404 (e.g., a hand-tracking device), or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or by any other suitable computerized device such as a game console or media player. The sensing function of the image sensor 404 can also be integrated into a computer or other computerized device controlled by the sensor output.

[0106] Figure 4 further includes schematic diagrams of depth maps 410 captured by image sensor 404 according to several embodiments. The depth map includes a matrix of pixels, each having a depth value, as described above. Pixels 412 corresponding to the hand 406 are segmented in this map from the background and the wrist. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from image sensor 404, with the gradation becoming denser as the depth increases. Controller 110 processes these depth values ​​to identify and segment image components (i.e., adjacent pixel groups) that have the characteristics of a human hand. These characteristics may include, for example, the overall size, shape, and frame-to-frame movement of the depth map sequence.

[0107] Figure 4 also schematically shows the hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to several embodiments. In Figure 4, the hand skeleton 414 is superimposed on the hand background 416, which has been segmented from the original depth map. In some embodiments, the hand (e.g., knuckles, fingertips, center of the palm, or end of the hand connected to the wrist), and optionally major feature points on the wrist or arm connected to the hand, are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these major feature points across multiple image frames are used by the controller 110 to determine, according to several embodiments, a hand gesture performed by the hand or the current state of the hand.

[0108] Figure 5 shows an exemplary embodiment of the eye-tracking device 130 (Figure 1). In some embodiments, the eye-tracking device 130 is controlled by an eye-tracking unit 243 (Figure 2) to track the position and movement of the user's gaze toward the scene 105 or toward the XR content displayed via the display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, if the display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device positioned in a wearable frame, the head-mounted device includes both a component for generating XR content for user viewing and a component for tracking the user's gaze toward the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, if the display generation component is a handheld device or an XR chamber, the eye-tracking device 130 is optionally a separate device from the handheld device or XR chamber. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used with a display generation component that is mounted on the head or a display generation component that is not mounted on the head. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye-tracking device 130 is not a head-mounted device, but is optionally part of a non-head-mounted display generation component.

[0109] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames containing left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include, or be coupled to, one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display on which the user can directly view the physical environment and display virtual objects on a transparent or translucent display. In some embodiments, the display generation component projects virtual objects onto the physical environment. The virtual objects are projected, for example, onto a physical surface or as holograms, so that the individual can use the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0110] As shown in Figure 5, in some embodiments, the eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) camera or a near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, it may be directed toward a "hot" mirror positioned between the user's eye and a display panel that reflects IR or NIR light from the eye to the eye-tracking camera while allowing visible light to pass through. The eye-tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60–120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, both of the user's eyes are tracked separately by their respective eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked by a separate eye-tracking camera and light source.

[0111] In some embodiments, the eye-tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye-tracking device for a specific operating environment 100, e.g., the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at the factory or another facility before delivery of the AR / VR device to the end user. The device-specific calibration process may be an automated calibration process or a manual calibration process. User-specific calibration processes may include estimating the eye parameters of a specific user, e.g., pupil location, foveal location, optical axis, visual axis, and / or interpupillary distance. Once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, in some embodiments, images captured by the eye-tracking camera can be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0112] As shown in Figure 5, the eye-tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye-tracking system which includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes(s) 592. The eye-tracking camera 540 is positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, or a projector) and may be directed towards a mirror 550 that transmits visible light while reflecting IR or NIR light from the eye(s) 592 (e.g., as shown at the top of Figure 5), or may be directed towards the user's eye(s) 592 to receive reflected IR or NIR light from the eye(s) 592 (e.g., as shown at the bottom of Figure 5).

[0113] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames of left and right display panels) and provides the frames 562 to the display 510. For various purposes, for example, when processing the frames 562 for display, the controller 110 uses gaze tracking input 542 from the eye-tracking camera 540. The controller 110 optionally uses a glint-assisted method or other appropriate method to estimate the user's viewpoint on the display 510 based on the gaze tracking input 542 obtained from the eye-tracking camera 540. The viewpoint estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0114] The following describes, but is not intended to be limiting, several possible use cases of the user's current gaze direction. As an exemplary use case, the controller 110 may render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content at a higher resolution in the central visual region determined from the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least partially on the user's current gaze direction. As yet another example, the controller may display specific virtual content within the view based at least partially on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may capture the physical environment of the XR experience and orient an external camera to focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface in the environment that the user is currently viewing on the display 510. In another exemplary use case, the eyepiece 520 may be a focusing lens, and the controller uses eye-tracking information to adjust the focus of the eyepiece 520 so that the virtual object currently being viewed by the user has appropriate binocular coordination to match the convergence of the user's eye 592. The controller 110 can use the eye-tracking information to orient and adjust the focus of the eyepiece 520 so that the nearby object being viewed by the user appears at the correct distance.

[0115] In some embodiments, the eye-tracking device is part of a head-mounted device mounted on a wearable housing, which includes a display (e.g., display 510), two eyepieces (e.g., one or more eyepieces 520), an eye-tracking camera (e.g., one or more eye-tracking cameras 540), and a light source (e.g., a light source 530 (e.g., an IR LED or NIR LED)). The light source emits light (e.g., IR light or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in Figure 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0116] In some embodiments, the display 510 emits light within the visible light range and does not emit light within the IR or NIR range, thus not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is positioned on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0117] Embodiments of eye-tracking systems, such as those shown in Figure 5, can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0118] Figure 6 shows glint-assisted eye-tracking pipelines according to several embodiments. In some embodiments, the eye-tracking pipeline is implemented by a glint-assisted eye-tracking system (e.g., an eye-tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye-tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the glint-assisted eye-tracking system tracks the pupil contour and glint in the current frame by using prior information from previous frames when analyzing the current frame. When not in tracking state, the glint-assisted eye-tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues in tracking state for the next frame.

[0119] As shown in Figure 6, the eye-tracking camera can capture left and right images of the user's left and right eyes. The captured images are then fed into the eye-tracking pipeline for processing, which begins at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images may be fed into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0120] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as shown in 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. If they are not successfully detected, the method returns to element 610 and processes the next image of the user's eyes.

[0121] At 640, if the process proceeds from element 610, the current frame is analyzed to track the pupil and glint based in part on previous information from the previous frame. At 640, if the process proceeds from element 630, the tracking state is initialized based on the detected pupil and glint in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results may be checked to determine whether a sufficient number of glints for pupil and gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are unreliable, the tracking state is set to no at element 660, and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze.

[0122] Figure 6 is intended to serve as an example of an eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies that currently exist or may be developed in the future may be used in computer system 101 to provide users with XR experiences in various embodiments, either in place of or in combination with the Glint-assisted eye-tracking technology described herein.

[0123] In some embodiments, the captured portion of the real-world environment 602 is used to provide the user with an XR experience, for example, a mixed reality environment in which one or more virtual objects are superimposed on a representation of the real-world environment 602.

[0124] Accordingly, this description describes several embodiments of three-dimensional environments (e.g., XR environments) that include representations of real-world objects and virtual objects. For example, a three-dimensional environment optionally includes a representation of a table existing in a physical environment, which is captured and displayed within the three-dimensional environment (e.g., actively via a computer system's camera and display, or passively via a computer system's transparent or translucent display). As described above, a three-dimensional environment optionally is a mixed reality system based on a physical environment, in which the three-dimensional environment is captured by one or more sensors of a computer system and displayed via a display generation component. As a mixed reality system, the computer system may optionally selectively display parts and / or objects of the physical environment so that each part and / or object of the physical environment appears to exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system may optionally display virtual objects in a three-dimensional environment so that the virtual objects appear to exist in the real world (e.g., the physical environment) by placing virtual objects in each location within the three-dimensional environment that have corresponding locations in the real world. For example, a computer system may optionally display a vase in such a way that it appears as if a real vase were placed on a table in a physical environment. In some embodiments, individual locations in a three-dimensional environment have corresponding locations in the physical environment.Therefore, when a computer system is described as displaying virtual objects in separate locations relative to physical objects (for example, at or near the location of the user's hand, or on or near a physical table), the computer system displays the virtual objects in specific locations within a three-dimensional environment so that they appear to be at or near physical objects in the physical world (for example, if the virtual object is a real object at that specific location, the virtual object will be displayed in the location within the three-dimensional environment that corresponds to the location within the physical environment where the virtual object would have been displayed).

[0125] In some embodiments, real-world objects existing in a physical environment displayed within a three-dimensional environment (e.g., real-world objects visible via and / or display-generating components) can interact with virtual objects existing only within the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on the table, where the table is a view (or representation) of a physical table in the physical environment, and the vase is a virtual object.

[0126] Similarly, just as virtual objects are real objects in a physical environment, the user can optionally interact with virtual objects in a three-dimensional environment using one or more hands. For example, as described above, one or more sensors in the computer system can optionally capture one or more of the user's hands and display a representation of the user's hands in a three-dimensional environment (in a similar manner to, for example, displaying real-world objects in a three-dimensional environment as described above), or, in some embodiments, the user's hands are visible through the display-generating components by the ability to see the physical environment through the user interface, due to the transparency / transparency of some of the display-generating components displaying the user interface, or the projection of the user interface onto a transparent / translucent surface, or the projection of the user interface onto the user's eyes or the user's field of view. Thus, in some embodiments, the user's hands are displayed at separate locations in the three-dimensional environment and are treated as if they were objects in a three-dimensional environment that can interact with virtual objects in the three-dimensional environment as if they were actual physical objects in the physical environment. In some embodiments, the computer system can update the display of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0127] In some of the embodiments described below, for example, to determine whether a physical object is directly interacting with a virtual object (e.g., whether a hand is touching, grasping, or holding a virtual object, or whether it is within a threshold distance of the virtual object), the computer system may optionally determine the "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment. For example, a hand directly interacting with a virtual object may optionally include one or more of the fingers of a hand pressing a virtual button, a user's hand grasping a virtual vase, two fingers of a user's hand pinching / holding an application's user interface together, and other types of interactions described herein. For example, when determining whether a user is interacting with a virtual object and / or how a user is interacting with a virtual object, the computer system may optionally determine the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the location of the hand in the three-dimensional environment and the location of the virtual object of interest in the three-dimensional environment. For example, one or more of the user's hands are located in a specific position in the physical world, which the computer system optionally captures and displays at a specific corresponding position in a three-dimensional environment (e.g., the position in the three-dimensional environment where the hands are displayed, if the hands are virtual hands rather than physical hands). The position of the hands in the three-dimensional environment is optionally compared to the position of a target virtual object in the three-dimensional environment to determine the distance between the one or more of the user's hands and the virtual object. In some embodiments, the computer system optionally determines the distance between the physical object and the virtual object by comparing the position in the physical world (as opposed to comparing the position in the three-dimensional environment).For example, when determining the distance between one or more of the user's hands and a virtual object, the computer system optionally determines the corresponding location of the virtual object in the physical world (e.g., the position in the physical world where the virtual object would be located if it were a physical object rather than a virtual object), and then determines the distance between the corresponding physical position and one or more of the user's hands. In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, when determining whether a physical object is in contact with a virtual object, or whether a physical object is within a threshold distance of a virtual object, as described herein, the computer system optionally performs one of the techniques described above to map the location of the physical object to a three-dimensional environment and / or to map the location of the virtual object to a physical environment.

[0128] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed, and / or where and what the physical stylus held by the user is directed. For example, if the user's gaze is directed to a particular position in the physical environment, the computer system optionally determines the corresponding position in the three-dimensional environment (e.g., the virtual position of the gaze), and if a virtual object is located at that corresponding virtual position, the computer system optionally determines that the user's gaze is directed to that virtual object. Similarly, the computer system optionally determines, based on the orientation of the physical stylus, where in the physical environment the stylus is pointing. In some embodiments, based on this determination, the computer system optionally determines the corresponding virtual position in the three-dimensional environment corresponding to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual position in the three-dimensional environment.

[0129] Similarly, embodiments described herein may refer to the location of a user (e.g., a user of a computer system) and / or the location of a computer system in a three-dimensional environment. In some embodiments, the user of a computer system is holding, wearing, or otherwise positioned near the computer system. Thus, in some embodiments, the location of the computer system is used as a proxy for the user's location. In some embodiments, the location of the computer system and / or the user in the physical environment corresponds to individual locations in the three-dimensional environment. For example, if a user stands at a location facing an individual part of the physical environment that is visible through a display-generating component, the location of the computer system is the location in the physical environment (and its corresponding location in the three-dimensional environment) where the user will see objects in the physical environment in the same position, orientation, and / or size (e.g., absolutely and / or relative to each other) as the objects are visible through the display-generating component of the computer system in the three-dimensional environment. Similarly, if a virtual object displayed in a three-dimensional environment is a physical object in a physical environment (for example, the virtual object is located in the same physical environment as it is in the three-dimensional environment, and has the same size and orientation as it does in the three-dimensional environment), then the computer system and / or user's location is the position from which the user views the virtual object in the physical environment in the same position, orientation, and / or size (for example, absolutely, and / or relative to each other, and in relation to real-world objects) as it was displayed by the computer system's display generation components in the three-dimensional environment.

[0130] This disclosure describes various input methods for interaction with computer systems. Where one example is provided using one input device or method, and another example is provided using a different input device or method, each example may be compatible with the input device or method described in the other example, and their use should be considered optional. Similarly, various output methods for interaction with computer systems are described. Where one example is provided using one output device or method, and another example is provided using a different output device or method, each example may be compatible with the output device or method described in the other example, and their use should be considered optional. Similarly, various methods for interaction with virtual or mixed reality environments via computer systems are described. Where one example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, each example may be compatible with the method described in the other example, and their use should be considered optional. Therefore, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User interface and related processes

[0131] Here, we focus on embodiments of a user interface ("UI") and related processes that may be performed in a computer system such as a portable multifunction device or head-mounted device, which includes display generation components, one or more input devices, and (optionally) one or more cameras.

[0132] Figures 7A to 7F show examples of computer systems that selectively recenter virtual content to the user's viewpoint, according to several embodiments.

[0133] Figure 7A shows a three-dimensional environment 702 visible through a display generation component of the computer system 101 (e.g., display generation component 120 in Figure 1), and the three-dimensional environment 702 is visible from a user's viewpoint 726a shown in an overhead view (e.g., facing the back wall of the physical environment where the computer system 101 is located, and near the lower left corner of the physical environment). As described above with reference to Figures 1 to 6, the computer system 101 optionally includes a display generation component (e.g., a touchscreen) and a plurality of image sensors (e.g., image sensor 314 in Figure 3). The image sensors optionally include one or more of the following: a visible light camera, an infrared camera, a depth sensor, or any other sensors that the computer system 101 may use to capture one or more images of the user or a part of the user (e.g., one or more of the user's hands) while the user interacts with the computer system 101. In some embodiments, the user interfaces illustrated and described below may also be implemented on a head-mounted display, which includes a display generating component that displays the user interface or three-dimensional environment to the user, and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0134] As shown in Figure 7A, the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., the operating environment 100), including one or more objects in the physical environment surrounding the computer system 101. In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 702, and / or the physical environment is visible in the three-dimensional environment 702 via the display generation component 120. For example, the three-dimensional environment 702 visible via the display generation component 120 includes a representation of the physical floor, as well as the back and side walls of the room in which the computer system 101 is located. The three-dimensional environment 702 also includes a sofa 724b (shown in the overhead view) which is invisible via the display generation component 120 from the user's viewpoint 726a in Figure 7A.

[0135] In Figure 7A, the three-dimensional environment 702 also includes virtual objects 712a (corresponding to object 712b in the overhead view) and 714a (corresponding to object 714b in the overhead view) that are visible from viewpoint 726a. The three-dimensional environment 702 also includes virtual object 710b (shown in the overhead view) that is invisible from the user's viewpoint 726a in Figure 7A via the display generation component 120. In Figure 7A, objects 712a, 714a, and 710b are two-dimensional objects. It is understood that the examples of this disclosure apply equally to three-dimensional objects at will. The virtual objects 712a, 714a, and 710b are, at will, one or more of the following: an application user interface (e.g., a messaging user interface, or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other elements displayed by the computer system 101 that are not included in the physical environment of the computer system 101.

[0136] In some embodiments, virtual objects that were last placed or repositioned from a particular previous viewpoint (or a set of previous viewpoints) of the user may be recentered to the user's new current viewpoint, as will be described in more detail below. For example, in Figure 7A, virtual objects 712a, 714a, and 710a are placed and / or positioned in their current locations and / or orientations within the three-dimensional environment 702 so as reflected in the overhead view from the user's viewpoint 726a. Furthermore, virtual object 712a is snapped or fixed to the back wall of the physical environment, as shown in Figure 7A. As will be described in more detail with reference to Method 800, virtual objects may optionally be snapped or fixed to a physical object in response to user input, depending on whether they have been moved to a location within a threshold distance (e.g., 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, or 100 cm) of the physical object within the three-dimensional environment 702. Furthermore, in some embodiments, the computer system 101 displays a visual indication in the three-dimensional environment 702 that a virtual object is snapped to or fixed to a physical object. For example, in Figure 7A, the computer system 101 displays a virtual drop shadow 713 on the back wall of the room in the physical environment as if it were generated by the virtual object 712a (e.g., a virtual object snapped to or fixed to a physical object). In some embodiments, the computer system 101 does not display such a visual indication for the virtual object 714a, because it is optionally not snapped to or fixed to a physical object.

[0137] In Figure 7B, the user's viewpoint 726a within the three-dimensional environment 702 has changed to move further away from the back and left walls of the room in the physical environment and closer to the center of the room, as shown in the overhead view. The viewpoint 726b in the overhead view corresponds to the user's previous viewpoint shown in Figure 7A. The user's viewpoint 726a can optionally change in the manner described with reference to Method 800, including the user moving towards the center of the room in the physical environment. The user's viewpoint 726a in Figure 7B is still directed towards the back wall of the room.

[0138] From viewpoint 726a shown in Figure 7B, virtual objects 710a, 712a, and 714a (last placed or positioned in the three-dimensional environment 702 from viewpoint 726b, as explained with reference to Figure 7A) are displayed from a greater distance from viewpoint 726a, at the same location and / or orientation within the three-dimensional environment 702. Furthermore, the user places or positions virtual objects 706a (corresponding to 706b in the overhead view) and 708a (corresponding to 708b in the overhead view) within the three-dimensional environment 702 from viewpoint 726a in Figure 7B.

[0139] In Figure 7B, the computer system 101 detects an input (e.g., selection of a physical button on the computer system 101) that recenters one or more virtual objects to the user's viewpoint 726a, as will be described in more detail with reference to Method 800. In some embodiments, virtual objects 706a and 708a do not move in the three-dimensional environment 702 in response to the input because they were last placed or repositioned in the three-dimensional environment from the user's current viewpoint 726a. However, one or more virtual objects that were last placed or repositioned in the three-dimensional environment 702 from the user's previous viewpoint (one or more) (e.g., viewpoint 726b) are optionally recentered to viewpoint 726a, as will be described below and in more detail with reference to Method 800.

[0140] For example, Figure 7C shows an exemplary result of the input shown in Figure 7B. In Figure 7C, objects 706a and 708a remain in their locations and / or orientations within the three-dimensional environment 702 in response to the recentering input. Object 712a also remains in its location and / or orientation within the three-dimensional environment 702 in response to the recentering input, even though it was last placed or repositioned within the three-dimensional environment 702 from the previous viewpoint 726b, because object 712a is snapped or fixed to the back wall of the physical environment of the computer system 101.

[0141] In contrast, objects 710b and 714a are recentered to the user's viewpoint 726a. In some embodiments, the relative location and / or orientation of objects 710b and 714a with respect to viewpoint 726a is the same as the relative location and / or orientation of objects 710b and 714a with respect to viewpoint 726a. For example, object 714a is optionally displayed in the same location as in Figure 7A with respect to viewpoint 726a in Figure 7C. Furthermore, object 710b is optionally invisible from viewpoint 726a in Figure 7C, as it was in Figure 7A. Furthermore, the spatial arrangement of objects 710b and 714a relative to each other is also optionally maintained before and after the recentering input. Further details regarding the movement of objects 710b and 714a in response to the recentering input are described with reference to Method 800. In this way, virtual objects associated with the user's previous viewpoint can be easily moved to the user's current viewpoint to facilitate interaction with and / or visibility of those virtual objects.

[0142] In some embodiments, the simulated environment can also be recentered to the user's new current viewpoint in a similar manner to how virtual objects are recentered to such viewpoints. For example, in Figure 7D, the user's viewpoint 726a is as shown in the overhead view. The user provides input to position or reposition virtual objects 706a and 708a to their current positions and / or orientations within the three-dimensional environment 702 from viewpoint 726a, as shown in Figure 7D. Furthermore, the user provides input to display the simulated environment 703 from viewpoint 726a to the computer system. The simulated environment 703 optionally consumes a portion of the three-dimensional environment 702, as shown in the overhead view. Further details about the simulated environment 703 are described with reference to Method 800.

[0143] In Figure 7E, the viewpoint 726a has changed to what is shown in the overhead view (for example, in the physical environment, it has moved downwards and is facing the left wall instead of the back wall). The viewpoint 726a may optionally move as previously described and / or as described with reference to Method 800. The virtual objects 706a and 708a are no longer visible via the display generation component 120. Furthermore, in some embodiments, the computer system 101 removes the simulated environment 703 from the three-dimensional environment 702 in response to the movement of the user's viewpoint 726a, as shown in the overhead view. In some embodiments, the computer system 101 maintains the simulated environment 703 within the three-dimensional environment 702 in response to the movement of the user's viewpoint 726a, but the simulated environment 703 is no longer within the field of view of the three-dimensional environment 702 from the user's current viewpoint 726a. In Figure 7E, virtual objects 706b and 708b are also not within the field of view of the three-dimensional environment 702 from the user's current viewpoint 726a.

[0144] In Figure 7E, the computer system 101 can detect at least two different inputs: 1) a recentering input (e.g., as described above), or 2) an input to increase the level of immersion in which the three-dimensional environment 702 is displayed. Immersion and level of immersion are described in more detail with reference to Method 800. The recentering input is optionally a press-down of an input element (e.g., a press-down dial that is also rotatable, as described below). The input to increase the level of immersion is optionally a rotation of the input element in a particular direction. Additional details regarding the above inputs are provided with reference to Method 800. The computer system 101 optionally responds differently to the above two inputs, as described below.

[0145] Figure 7F shows a continuation of the recentering input described earlier with reference to Figure 7E. In Figure 7F, objects 706a and 708a are recentered to the user's viewpoint 726a. In some embodiments, the relative location and / or orientation of objects 706a and 708a with respect to viewpoint 726a in Figure 7F is the same as the relative location and / or orientation of objects 706a and 708a with respect to viewpoint 726a in Figure 7D. For example, object 706a is optionally displayed in the same location with respect to viewpoint 726a in Figure 7F as it is in Figure 7D. Furthermore, the relative spatial arrangement of objects 706a and 708a with respect to each other is also optionally maintained before and after the recentering input. Further details regarding the movement of objects 706a and 708a in response to the recentering input are described with reference to Method 800.

[0146] In addition to recentering objects 706a and 708a to viewpoint 726a in Figure 7F in response to the recentering input, the computer system 101 also redisplays the simulated environment 703 in the three-dimensional environment 702. As shown in Figure 7F, the computer system 101 places the simulated environment 703 in a different position within the three-dimensional environment 702 than in Figure 7D (for example, occupying a different portion of the three-dimensional environment 702). In some embodiments, the position and / or orientation of the simulated environment 703 is based on the location and / or orientation of viewpoint 726a in Figure 7F. For example, the simulated environment 703 is optionally placed at the same distance from viewpoint 726a in Figure 7F as it is from viewpoint 726a in Figure 7D. Additionally or alternatively, the simulated environment 703 may optionally be centered on and / or oriented towards viewpoint 726a in Figure 7F (for example, the orientation of viewpoint 726a may be oriented towards the center of the simulated environment 703, and / or the orientation of the simulated environment 703 may be oriented towards viewpoint 726a). Further details regarding the display of the simulated environment 703 in response to the recentering input are provided by referring to Method 800.

[0147] In contrast to recentering inputs, if the computer system 101 in Figure 7E detects an input that increases the level of immersion to which the computer system was displaying the three-dimensional environment 702, the computer system 101 optionally redisplays the simulated environment 703 in the manner described above, but the virtual objects 706a and 708a are optionally not recentered to the viewpoint 726a in Figure 7F. For example, objects 706a and 708a optionally remain in their positions and / or orientations in the three-dimensional environment shown in Figure 7E. Further details of the computer system 101's response to the detection of such inputs to increase the level of immersion of the three-dimensional environment 702 are provided by referring to Method 800.

[0148] Figures 8A to 8I are flowcharts illustrating exemplary methods for selectively recentering virtual content to the user's viewpoint, according to several embodiments. In some embodiments, Method 800 is performed on a computer system (e.g., computer system 101 in Figure 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 in Figures 1, 3, and 4) (e.g., a head-up display, display, touchscreen, or projector) and one or more cameras (e.g., a camera pointing downwards from the user's hands (e.g., a color sensor, infrared sensor, and other depth-sensing camera) or a camera pointing forward from the user's head). In some embodiments, Method 800 is stored in a non-temporary computer-readable storage medium and executed by instructions performed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 in Figure 1A). Some operations of Method 800 are optionally combined, and / or the order of some operations is optionally changed.

[0149] In some embodiments, Method 800 is performed in a computer system (e.g., 101) that communicates with a display generating component and one or more input devices, such as a mobile device (e.g., a tablet, smartphone, media player, or wearable device), or a computer or other electronic device. In some embodiments, the display generating component is an integrated display with the electronic device (optionally a touchscreen display), an external display such as a monitor, projector, television, or hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, one or more input devices include electronic devices or components that can receive user input (e.g., capture user input or detect user input) and transmit information associated with the user input to the computer system. Examples of input devices include touchscreens, mice (e.g., external), trackpads (optionally integrated or external), touchpads (optionally integrated or external), remote control devices (e.g., external), another mobile device (e.g., separate from the computer system), handheld devices (e.g., external), controllers (e.g., external), cameras, depth sensors, eye-tracking devices, and / or motion sensors (e.g., hand-tracking devices, hand movement sensors). In some embodiments, the computer system communicates with the hand-tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (touchscreen, trackpad)). In some embodiments, the hand-tracking device is a wearable device such as a smart glove. In some embodiments, the hand-tracking device is a handheld input device such as a remote control or stylus.

[0150] In some embodiments, the three-dimensional environment (e.g., 702) is viewable through display generation components (e.g., the three-dimensional environment is generated, displayed, or otherwise made visible by a computer system (e.g., a computer-generated reality (CGR) environment such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment), but the three-dimensional environment includes first virtual objects having a first spatial arrangement with respect to the user's first viewpoint of the three-dimensional environment, which is the user's current viewpoint of the computer system, such as objects 706a-714a with respect to viewpoint 726a in Figure 7B (e.g., the first virtual objects are at a certain distance from the user's current viewpoint and have a certain orientation with respect to the user's current viewpoint (e.g., above and to the right of the user's current viewpoint). In some embodiments, the first virtual objects are placed in their current location within the three-dimensional environment by the user of the computer system, regardless of whether the user's viewpoint was the user's current viewpoint or the user's previous viewpoint. In some embodiments, the user's first viewpoint is the user, the computer system, and Corresponding to the user's current location and / or orientation in the physical environment of the display generation component, the computer system displays at least some portion of the three-dimensional environment from a viewpoint corresponding to the user's current location and / or orientation in the physical environment. In some embodiments, the first virtual object is the user interface of an application, a representation of content (e.g., images, videos, audio, or music), a three-dimensional rendering of an object (e.g., a tent, building, or car), or any other object not present in the user's physical environment, and the computer system (e.g., 101) receives a first input via one or more input devices, such as an input detected in Figure 7B (e.g., a “recentering” input as described in more detail below, and / or methods 1000 and / or 1400), in response to a request to update the spatial arrangement of one or more virtual objects relative to the user's first viewpoint to satisfy a first set of one or more criteria defining the range of distances or orientations of one or more virtual objects relative to the user's first viewpoint (802a).In some embodiments, the three-dimensional environment includes one or more virtual objects (e.g., a first virtual object) such as application windows, operating system elements, representations of other users, and / or content items. In some embodiments, the three-dimensional environment includes representations of physical objects in the physical environment of the computer system. In some embodiments, representations of physical objects are displayed in the three-dimensional environment via a display-generating component (e.g., a virtual passthrough or video passthrough). In some embodiments, the representation of physical objects is a view of the physical object in the physical environment of the computer system that is visible through the transparency portion of the display-generating component (e.g., a true passthrough or actual passthrough). In some embodiments, the computer system displays the three-dimensional environment from the user's viewpoint at locations in the three-dimensional environment corresponding to the physical locations of the computer system, the user, and / or the display-generating component in the physical environment of the computer system. In some embodiments, an input that responds to a request to update the spatial arrangement of objects relative to the user's viewpoint to satisfy one or more first criteria is an input directed to a hardware button or switch (e.g., built into the computer system) communicating with the computer system. In some embodiments, the first input is an input directed to selectable options displayed via a display-generating component. In some embodiments, one or more first criteria are met when the interactive portion of a virtual object is oriented toward the user's viewpoint, the virtual object does not obstruct the view of other virtual objects from the user's viewpoint, the virtual object is within a threshold distance of the user's viewpoint (e.g., 10, 20, 30, 40, 50, 100, 200, 300, 400, 500, 1000, or 2000 centimeters), and / or the virtual objects are within a threshold distance of each other (e.g., 1, 5, 10, 20, 30, 40, 50, 100, 200, 300, 400, 500, 1000, or 2000 centimeters).In some embodiments, the first input is different from an input that requests updating the position of one or more objects in a three-dimensional environment (for example, relative to the user's viewpoint), such as an input for manually moving an object in the three-dimensional environment.

[0151] In some embodiments, in response to the reception of a first input (802b), a determination is made that a first virtual object, such as objects 714a and 710a in Figure 7B, satisfies a second set of one or more criteria (for example, as will be described in more detail below, the second set of one or more criteria being when the first virtual object was last placed or moved in the three-dimensional environment while the user's viewpoint was a different viewpoint from the first viewpoint, and / or when the first virtual object was last placed or moved in the three-dimensional environment from a previous viewpoint). 802c) is the first virtual object (which is optionally satisfied when the point is greater than a threshold distance (e.g., 1, 3, 5, 10, 20, 30, 50, 100, 200, 500, or 1000 cm from the first viewpoint)). The computer system (e.g., 101) displays a first virtual object in a three-dimensional environment having a second spatial arrangement different from a first spatial arrangement with respect to the user's first viewpoint, wherein the second spatial arrangement of the first virtual object satisfies a first set of one or more criteria, such as objects 714a and 710a in Figure 7C. In some embodiments, displaying the first virtual object in the second spatial arrangement involves updating the location (e.g., orientation and / or pose) of the first virtual object while maintaining the user's first viewpoint at a fixed location in the three-dimensional environment. In some embodiments, in response to a first input, the computer system updates the position of the first virtual object from a location not necessarily oriented around the user's first viewpoint to a location oriented around the user's first viewpoint.

[0152] In some embodiments, upon receiving a first input (802b), and in accordance with the determination that a first virtual object, such as objects 706a and 708a in Figure 7B, does not satisfy a second set of one or more criteria, the computer system (e.g., 101) maintains the first spatial arrangement of the first virtual object in the three-dimensional environment relative to the user's first viewpoint (802d) (e.g., does not change the location of the first virtual object in the three-dimensional environment), as shown in objects 706a and 708a in Figure 7C. In some embodiments, the first virtual object is visible from the user's current viewpoint via the display generation component. In some embodiments, the first virtual object is invisible from the user's current viewpoint via the display generation component. In some embodiments, the computer system, in response to the first input, similarly changes (or does not change) the locations of other virtual objects in the three-dimensional environment. In some embodiments, the input described with reference to Method 800 is or includes an air gesture input. Changing the location of some, but not all, objects in a three-dimensional environment in response to a first input reduces the number of inputs required to properly position the objects relative to the user's viewpoint in response to the first input.

[0153] In some embodiments, when a first input is detected, the three-dimensional environment includes a first virtual object and a second virtual object, such as objects 714a and 706a in Figure 7B (for example, having one or more properties of the first virtual object), the second virtual object having a third spatial arrangement with respect to the user's first viewpoint (for example, the second virtual object is at a certain distance from the user's current viewpoint and is oriented in a certain direction with respect to the user's current viewpoint) (804a).

[0154] In some embodiments, upon receiving a first input, the first virtual object has a second spatial position relative to the user's first viewpoint, as shown in objects 714a and 706a of Figure 7C, and the second virtual object has a third spatial position relative to the user (804b). In some embodiments, the first virtual object is recentered in response to the first input as described above, but the second virtual object is not recentered in response to the first input (e.g., remains at its current location and / or orientation relative to the user's first viewpoint). In some embodiments, the second virtual object is not recentered because its current location and / or orientation already satisfies a first set of one or more criteria. In some embodiments, the second virtual object is not recentered because it was last placed or positioned in the three-dimensional environment from the user's first viewpoint, or is fixed to a physical object, both of which are described in more detail below. Changing the location of some, but not all, objects in the three-dimensional environment in response to the first input reduces the number of inputs required to properly position the objects relative to the user's viewpoint in response to the first input.

[0155] In some embodiments, a second set of one or more criteria includes criteria that are not met when a first virtual object, such as objects 706a and 708a, which were last placed or moved into the environment 702 from viewpoint 726a in Figure 7B (corresponding, for example, the user's current physical position or orientation in the user's physical environment), is last placed or moved into the three-dimensional environment from a viewpoint that satisfies a third set of one or more criteria for the user's first viewpoint (806). In some embodiments, the user's current viewpoint (e.g., the location and / or orientation of the current viewpoint) corresponds to the user's current location and / or orientation in the user's physical environment (e.g., the user's head or torso). In some embodiments, a virtual object last placed or positioned in the three-dimensional environment from the user's first viewpoint (e.g., within a threshold distance and / or threshold orientation of the user's current viewpoint, as described in more detail below) is not recentered in response to a first input, while a virtual object last placed or positioned in the three-dimensional environment from a viewpoint different from the user's first viewpoint (or whose location and / or orientation are sufficiently different from the user's first viewpoint) is recentered in response to a first input. Changing the location of an object last placed or positioned in the three-dimensional environment from the user's previous viewpoint in response to a first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to a first input.

[0156] In some embodiments, a third set of one or more criteria includes a criterion that is met when the viewpoint is within a threshold distance (e.g., 3, 5, 50, 100, 1000, 5000, or 10000 cm) of a first viewpoint (808), such as when objects 706a and 708a were last placed or moved into the environment 702 from a viewpoint within a threshold distance of viewpoint 726a in Figure 7B. Thus, in some embodiments, the criterion is not met if the viewpoint from which the first virtual object was last placed or positioned in the three-dimensional environment is farther than the threshold distance from the user's current viewpoint, and the criterion is met if the viewpoint from which the first virtual object was last placed or positioned in the three-dimensional environment is closer than the threshold distance from the user's current viewpoint. Changing the location of the object that was last placed or positioned from the user's previous viewpoint, which is relatively far from the current viewpoint, in response to a first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to a first input.

[0157] In some embodiments, a third set of one or more criteria includes a criterion that is satisfied when the viewpoint has an orientation in the three-dimensional environment that is within the threshold orientation of the orientation of a first viewpoint (810) in the three-dimensional environment (e.g., within 1, 3, 5, 10, 20, 30, 45, or 90 degrees), such as when objects 706a and 708a were last placed or moved into the environment 702 from a viewpoint within the threshold orientation of viewpoint 726a in Figure 7B. Thus, in some embodiments, the criterion is not satisfied if the orientation of the viewpoint to which the first virtual object was last placed or positioned in the three-dimensional environment is greater than the threshold orientation away from the orientation of the user's current viewpoint, and the criterion is satisfied if the orientation of the viewpoint to which the first virtual object was last placed or positioned in the three-dimensional environment is less than the threshold orientation away from the orientation of the user's current viewpoint. Changing the location of an object that was last placed or positioned from the user's previous viewpoint, which is off-angle relative to the current viewpoint in the three-dimensional environment, in response to a first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to a first input.

[0158] In some embodiments, a second set of one or more criteria includes criteria that are not met when the first virtual object is fixed to a part of the user's physical environment, such as when object 712a is fixed to the back wall of the room in Figure 7B (e.g., fixed to the surface of a physical object in the user's physical environment, such as a wall or table surface) (812). In some embodiments, the first virtual object is fixed to a part of a physical object (e.g., a surface) in response to a computer system detecting an input to move the first virtual object within a threshold distance (e.g., 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 30, or 50 cm) of a part of the physical object, which optionally causes the first virtual object to snap to a location and / or orientation of a part of the physical object. An object thus fixed to a physical object is optionally not recentered in response to the first input. In some embodiments, the criteria are met when the first virtual object is not fixed to a physical object. Changing the location of an object that is not fixed to a physical object in a three-dimensional environment in response to a first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to the first input.

[0159] In some embodiments, while displaying a first virtual object in a three-dimensional environment (814a), according to a determination that the first virtual object is fixed to a part of the user's physical environment, the computer system (e.g., 101) displays a visual indication in the three-dimensional environment that the first virtual object is fixed to a part of the physical environment, such as the virtual drop shadow 713 in Figures 7A-7B (e.g., a virtual drop shadow of the first virtual object displayed on a part of the physical environment as if a drop shadow had been cast onto the part of the physical environment by the first virtual object, and / or an icon displayed in association with the first virtual object indicating that the first virtual object is fixed or pinned to a part of the physical environment (e.g., a pin icon)) (814b).

[0160] In some embodiments, while displaying a first virtual object in a three-dimensional environment (814a), if it is determined that the first virtual object is not anchored to any part of the user's physical environment, the computer system (e.g., 101) displays the first virtual object in the three-dimensional environment without displaying any visual indications, such as displaying object 714a without the virtual drop shadow shown in Figures 7A and 7B (814c) (e.g., the drop shadow and / or icon are not displayed unless or until the first virtual object is anchored to any part of the physical environment). Indicating the anchoring status of the first virtual object provides feedback regarding the state of the first virtual object.

[0161] In some embodiments, the first virtual object is part of a set of multiple virtual objects in a three-dimensional environment that satisfy a second set of one or more criteria, such as the set of objects 710a and 714a in Figure 7B (for example, the multiple virtual objects were last placed or positioned in the three-dimensional environment from the same previous viewpoint of the user) (816a).

[0162] In some embodiments, the set has a first individual spatial arrangement with respect to a first viewpoint when a first input is received, such as the spatial arrangement of the set of objects 710a and 714a with respect to viewpoint 726a in Figure 7B (816b).

[0163] In some embodiments, upon receiving a first input, the set is displayed in a second distinct spatial arrangement different from the first distinct spatial arrangement to the first viewpoint, such as the spatial arrangement of the set of objects 710a and 714a to viewpoint 726a in Figure 7C (e.g., the set of multiple virtual objects is recentered as a group in response to the first input (e.g., moved and / or reoriented)), and the spatial arrangement of the multiple virtual objects in the set to the first viewpoint after the first input is received satisfies a first set of one or more criteria (e.g., the virtual objects in the set are recentered to positions and / or orientations that satisfy a first set of one or more criteria) (816c). In some embodiments, virtual objects are recentered within a group or based on a group in response to a recentering input. A group of virtual objects that were last placed or positioned in the three-dimensional environment from the same previous viewpoint of the user are optionally recentered together as a group to the first viewpoint (e.g., the virtual objects are moved together to their updated locations and / or orientations). In some embodiments, the three-dimensional environment includes multiple distinct sets of virtual objects that were last placed or positioned within the three-dimensional environment from different shared previous viewpoints of the user and simultaneously recentered as a group of virtual objects in response to a first input. In some embodiments, the three-dimensional environment includes a set of virtual objects that were last placed or positioned within the three-dimensional environment from the user's first viewpoint and are therefore not recentered as a group in response to a first input. Recentering virtual objects as a group of objects reduces the number of inputs required to properly position the objects relative to the user's viewpoint in response to a first input.

[0164] In some embodiments, before receiving a first input, while the set has a first distinct spatial arrangement with respect to a first viewpoint, multiple virtual objects within the set have distinct positional arrangements with respect to each other, such as the positional arrangement between objects 710a and 714a in Figure 7B (for example, the virtual objects within the set have specific positions with respect to each other, such as four virtual objects positioned at the vertices of a square arrangement) (818a).

[0165] In some embodiments, after receiving a first input, while the set has a second distinct spatial arrangement with respect to a first viewpoint, multiple virtual objects within the set have distinct positional arrangements with respect to each other, such as the positional arrangement between objects 710a and 714a in Figure 7C (818b). For example, the relative positions of virtual objects within a set of virtual objects are maintained in response to the first input, even if the set of virtual objects is rearranged and / or reoriented in the three-dimensional environment in response to the first input (for example, a square arrangement may have different positions and / or orientations in the three-dimensional environment, but the four virtual objects remain positioned at the vertices of the same square arrangement in response to the first input). Maintaining the positional arrangement of virtual objects within a set reduces the number of inputs required to properly position the objects with respect to the user's viewpoint in response to the first input.

[0166] In some embodiments, before receiving a first input, while the set has a first distinct spatial arrangement with respect to a first viewpoint, multiple virtual objects within the set have distinct orientations relative to each other, such as the orientation between objects 710a and 714a in Figure 7B (for example, the virtual objects within the set have specific orientations relative to each other, such as four virtual objects oriented so that the virtual objects are parallel to each other) (820a).

[0167] In some embodiments, after receiving a first input, while the set has a second distinct spatial arrangement with respect to a first viewpoint, the multiple virtual objects within the set have distinct orientations relative to each other, such as the orientation between objects 710a and 714a in Figure 7C (820b). For example, even if the set of virtual objects is rearranged and / or reoriented in the three-dimensional environment in response to the first input, the relative orientation of the virtual objects within the set of virtual objects is maintained in response to the first input (for example, the virtual objects have new positions and / or orientations in the three-dimensional environment, but the four virtual objects remain parallel to each other). Maintaining the orientation of virtual objects within the set reduces the number of inputs required to properly position the objects with respect to the user's viewpoint in response to the first input.

[0168] In some embodiments, multiple virtual objects in a set are last positioned or moved into the three-dimensional environment from a second viewpoint of the user that is different from the user's first viewpoint, such as viewpoint 726a in Figure 7A or viewpoint 726b in Figure 7B, before the first input is received (for example, the second viewpoint is sufficiently different from the first viewpoint to result in a set of virtual objects that are recentered in response to the first input, as described above) (822a).

[0169] In some embodiments, the average orientation of a set of virtual objects to a second viewpoint while the set has a first individual spatial arrangement to a first viewpoint is an individual orientation (822b), such as the average orientation of objects 714a and 710a to viewpoint 726a in Figure 7A. For example, a set of virtual objects includes three virtual objects, each having its own orientation to the user's second viewpoint (e.g., the first of the objects is directly in front of and / or in the center of the second viewpoint, the second of the objects is about 45 degrees to the right of the center of the second viewpoint, and the third of the objects is about 60 degrees to the right of the center of the second viewpoint). The relative orientation of each virtual object is optionally with respect to and / or corresponding to the orientation of the user's shoulders, head, and / or chest when the user last placed or positioned each virtual object from the second viewpoint. In some embodiments, the average of the above orientations is the average of the orientations of the three virtual objects described above.

[0170] In some embodiments, the set has a second distinct spatial arrangement relative to a first viewpoint in response to receiving a first input, while the set has a distinct orientation relative to the user's first viewpoint, such as the average orientation of objects 714a and 710a relative to viewpoint 726a in Figure 7C (822c). For example, a group or set of virtual objects recentered to a first viewpoint in response to a first input is positioned in the three-dimensional environment with an orientation relative to the first viewpoint that corresponds to the average relative orientation of the virtual objects in the set of virtual objects relative to the second viewpoint (e.g., when those objects were last placed or positioned in the three-dimensional environment). Thus, if the average orientation of the virtual objects relative to the second viewpoint is 30 degrees to the right of the centerline of the second viewpoint, the set of virtual objects is optionally oriented / positioned at 30 degrees to the right of the centerline of the first viewpoint (e.g., the relative positions and / or orientations of the virtual objects in the set remain unchanged). Positioning the set of virtual objects with an average orientation relative to the first viewpoint reduces the number of inputs required to properly position the objects relative to the user's viewpoint in response to a first input.

[0171] In some embodiments, the first virtual object was last placed or moved into the three-dimensional environment from a second viewpoint of the user that is different from the user's first viewpoint, such as viewpoint 726a in Figure 7A or viewpoint 726b in Figure 7B, before the first input is received (for example, the second viewpoint is sufficiently different from the first viewpoint to result in a set of virtual objects that are recentered in response to the first input, as described above) (824a).

[0172] In some embodiments, the first virtual object has a first spatial arrangement with respect to the user's first viewpoint, but the first virtual object is a first distance from a second viewpoint (e.g., a different distance from the first viewpoint), such as the distance of object 714a from viewpoint 726a in Figure 7A (824b).

[0173] In some embodiments, the first virtual object has a second spatial position relative to the user's first viewpoint, but the first virtual object is a first distance from the first viewpoint (e.g., a different distance from the second viewpoint), such as the distance of object 714a from viewpoint 726a in Figure 7C (824c). Thus, in some embodiments, when virtual objects are recentered, their distance(s) from the user's current viewpoint are based on (e.g., the same as) their distance(s) from the user's previous viewpoint where those virtual objects were last placed or positioned in the three-dimensional environment. Positioning recentered virtual objects at a distance from a viewpoint corresponding to a previous distance from the user's previous viewpoint reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to a first input.

[0174] In some embodiments, before the first input is received, the first virtual object is located at a first location in the three-dimensional environment, such as the location of object 714a in Figure 7B, and the first virtual object remains at the first location in the three-dimensional environment until an input is received to reposition the first virtual object in the three-dimensional environment (826). In some embodiments, the first virtual object remains at its location in the three-dimensional environment (e.g., not recentered) until an input for recentering is received or until an input for moving the first virtual object in the three-dimensional environment is received (e.g., separately from the recentering input). In some embodiments, other inputs, such as an input for changing the user's viewpoint, do not cause the first virtual object to change its location in the three-dimensional environment. If no recentering input is received, maintaining the position and / or orientation of the first virtual object in the three-dimensional environment reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to the first input.

[0175] In some embodiments, before receiving a first input, a first virtual object, such as object 708a positioned from viewpoint 726a in Figure 7D, was last positioned or moved into the three-dimensional environment from a second viewpoint of the user, which is different from the user's first viewpoint (828a).

[0176] In some embodiments, before receiving a first input (828b), while the three-dimensional environment is visible from the user's second viewpoint via a display-generating component, the computer system (e.g., 101) displays a simulated environment, such as the simulated environment 703 in Figure 7D, and the first virtual object via the display-generating component (828c). For example, while the user's viewpoint is a second viewpoint, the user provides input to the computer system to display the simulated environment in the three-dimensional environment that is visible from the user's second viewpoint. In some embodiments, the simulated environment occupies a portion of the three-dimensional environment that is visible via the display-generating component.

[0177] In some embodiments, before receiving a first input (828b), while displaying a simulated environment in a three-dimensional environment, the computer system (e.g., 101) detects a shift in the user's viewpoint from a second viewpoint to a first viewpoint, such as from Figure 7D to Figure 7E (e.g., a change in the user's movement and / or orientation in the user's physical environment corresponding to the shift in the user's viewpoint from the second viewpoint to the first viewpoint) (828d).

[0178] In some embodiments, before receiving a first input (828b), in response to detecting a shift in the user's viewpoint from a second viewpoint to a first viewpoint, the computer system (e.g., 101) maintains the first virtual object in the three-dimensional environment (e.g., maintains the location and / or orientation of the first virtual object in the three-dimensional environment), as shown in the overhead view of Figure 7E, and stops including at least part (or all) of the simulated environment in the three-dimensional environment (828e) (e.g., the simulated environment no longer exists in the three-dimensional environment), as shown in the overhead view of Figure 7E where the simulated environment 703 does not exist. In some embodiments, the change in viewpoint from a second viewpoint to a first viewpoint must be large enough (e.g., as described above with respect to a third set of one or more criteria) for the computer system to stop including the simulated environment in the three-dimensional environment in response to the change in the user's viewpoint. In some embodiments, the simulated environment remains in the three-dimensional environment in response to the change in the user's viewpoint, but is no longer visible via the display generation components (e.g., because the simulated environment is outside the user's field of view). Disabling the inclusion of simulated environments automatically reduces resource usage and clutter in the computer system within the three-dimensional environment.

[0179] In some embodiments, in response to receiving a first input while the user's viewpoint is a first viewpoint, as shown in Figure 7E, the computer system (e.g., 101) displays the simulated environment from the first viewpoint in the three-dimensional environment (830) as shown in Figure 7F (e.g., optionally, without changing the level of immersion in the three-dimensional environment, as described below). In some embodiments, the simulated environment is redisplayed and / or recentered to the first viewpoint in the three-dimensional environment (e.g., the new location and / or orientation in which the simulated environment is displayed in the three-dimensional environment is different from the location and / or orientation in the three-dimensional environment in which the simulated environment was last displayed from the user's second viewpoint). For example, if the simulated environment was last displayed facing a first wall of the user's physical room and occupying a first portion of the three-dimensional environment, when the simulated environment is redisplayed from the first viewpoint, the simulated environment faces a second wall (different from the first wall) of the user's physical room and occupies a second portion (different from the first portion) of the three-dimensional environment. The simulated environment is optionally redisplayed to face and center on the user's first viewpoint. The simulated environment is optionally redisplayed and / or recentered along with the recentering of the first virtual object, as described above. Redisplaying the simulated environment in response to the first input reduces the number of inputs required to view the simulated environment in a three-dimensional environment.

[0180] In some embodiments, while the user's viewpoint is the first viewpoint and before receiving the first input, the computer system (e.g., 101) detects a second input (832a) that corresponds to a request to increase the level of immersion of the three-dimensional environment, such as receiving an input that increases the level of immersion in Figure 7E via one or more input devices. In some embodiments, the second input includes the rotation of a rotatable mechanical input element that is integrated with and / or communicates with the computer system. In some embodiments, rotating the rotatable mechanical input element in the first direction is an input to increase the level of immersion of the three-dimensional environment that is visible through the display-generating components. In some embodiments, rotating the rotatable mechanical input element in the opposite direction is an input to decrease the level of immersion of the three-dimensional environment that is visible through the display-generating components.

[0181] In some embodiments, the immersion level includes the degree to which content visible by a computer system (e.g., a simulated environment or virtual object, also called “virtual content”) obscures background content (e.g., non-virtual content) surrounding / behind the virtual content, and optionally includes the number of visible background content items, the visual characteristics (e.g., color, contrast, opacity) on which the background content is displayed, and / or the angular range of content visible through the display-generating components (e.g., 60 degrees for low-immersion content, 120 degrees for medium-immersion content, and 180 degrees for high-immersion content), and / or the percentage of the field of view displayed through the display-generating components occupied by the virtual content (e.g., 33% of the field of view occupied by the virtual content at low-immersion, 66% of the field of view occupied by the virtual content at medium-immersion, and 100% of the field of view occupied by the virtual content at high-immersion). In some embodiments, the background content is included in the background on which the virtual content is displayed. In some embodiments, background content includes user interfaces (e.g., user interfaces generated by a computer system corresponding to the application), virtual objects (e.g., files or representations of other users generated by a computer system), and / or real objects (e.g., pass-through objects corresponding to real objects in the physical environment around the user's viewpoint, which are visible through display-generating components and / or visible through transparent or translucent display-generating components, so that the computer system does not obscure / hinder their visibility through the display-generating components). In some embodiments, at a first (e.g., low) level of immersion, the background, virtual, and / or real objects are visible without obscuration. For example, a simulated environment at a low level of immersion is optionally visible simultaneously with the background content, and the background content is optionally visible with full brightness, color, and / or translucency.In some embodiments, at a second (e.g., high) level of immersion, background, virtual, and / or real objects are visible but obscured (e.g., dimmed, blurred, or removed from display). For example, a separate simulated environment with a high level of immersion may be displayed without background content (e.g., in full-screen or fully immersive mode). As another example, a simulated environment displayed at an intermediate level of immersion may be visible while background content is dimmed, blurred, or otherwise de-highlighted. In some embodiments, the visual properties of background objects differ among them. For example, at a particular level of immersion, one or more first background objects may be visually de-highlighted (e.g., dimmed, blurred, and made more transparent to be visible) than one or more second background objects, while one or more third background objects may become invisible.

[0182] In some embodiments, upon receiving a second input, the computer system (e.g., 101) displays the simulated environment from a first viewpoint in a three-dimensional environment, as shown in Figure 7F (e.g., optionally, displaying the three-dimensional environment at a higher level of immersion than before the second input was received) (832b). Thus, in some embodiments, upon receiving a second input, the simulated environment is redisplayed and / or recentered to the user's first viewpoint in the same or similar manner as described above with respect to redisplaying and / or recentering the simulated environment in response to the first input. Redisplaying the simulated environment in response to a second input reduces the number of inputs required to view the simulated environment in a three-dimensional environment.

[0183] In some embodiments, upon receiving a second input, the electronic device maintains the first spatial arrangement of the first virtual object in the three-dimensional environment relative to the user's first viewpoint (834), such as when objects 706a and 708a in Figure 7F remained in their locations in the environment 702 in Figure 7E instead (for example, the first virtual object is not moved or reoriented in the three-dimensional environment in response to the second input). Not recentering the first virtual object in response to the second input reduces the number of inputs required to properly position the virtual element in the three-dimensional environment.

[0184] In some embodiments, the three-dimensional environment includes a first set of one or more virtual objects whose spatial arrangement relative to a first viewpoint is changed in response to the reception of a first input (for example, these virtual objects were last placed or positioned in the three-dimensional environment from a user's previous viewpoint that is sufficiently different from the user's first viewpoint, as described with reference to a third set of one or more criteria) and a second set of one or more virtual objects whose spatial arrangement relative to a first viewpoint is not changed in response to the reception of a first input (for example, these virtual objects were last placed or positioned in the three-dimensional environment from a user's previous viewpoint that is not sufficiently different from the first viewpoint, as described with reference to a third set of one or more criteria) (836a).

[0185] In some embodiments, after receiving a first input (for example, after recentering a first set of virtual objects in the manner described above, but not a second set of virtual objects, while the first and second sets of virtual objects are in the locations and / or orientations resulting from those resulting from the first input), the computer system (e.g., 101) detects a shift in the user's viewpoint in the three-dimensional environment from a first viewpoint to a second viewpoint different from the first viewpoint (e.g., the second viewpoint is optionally sufficiently different from the user's first viewpoint to allow recentering) (e.g., corresponding to a change in the user's orientation and / or position in the user's physical environment) (836b), and in response to detecting the shift in the user's viewpoint, the three-dimensional environment is visible from the user's second viewpoint via the display generation components, and the positions or orientations of one or more first and second sets of virtual objects in the three-dimensional environment are not changed, such as the shift of viewpoint 726a away from the location in Figure 7C after the computer system 101 displays the environment 702 as in Figure 7C.

[0186] In some embodiments, while the three-dimensional environment is visible from the user's second viewpoint via a display generation component, a computer system (e.g., 101) receives a second input, similar to or the same as the input in Figure 7B (e.g., a recentering input following the aforementioned recentering input) via one or more input devices, in response to a request to update the spatial arrangement of one or more virtual objects relative to the user's second viewpoint and satisfy a first set of one or more criteria specifying a range of distances or orientations of one or more virtual objects relative to the user's second viewpoint (836c).

[0187] In some embodiments, in response to receiving a second input, the positions and orientations of the first and second sets of one or more virtual objects are changed (836d) within a three-dimensional environment such that the updated positions and orientations of the first and second sets of one or more virtual objects satisfy a first set of one or more criteria relative to a second perspective of the user, such as re-centering 706a, 708a, 710a, and 714a in response to the second input (e.g., re-centering both the first and second sets of virtual objects in response to a subsequent re-centering input in one or more of the methods described above). Thus, two different groups or sets of virtual objects (e.g., as described above) are optionally processed separately in response to a first re-centering input (e.g., one set is re-centered while the second set is not), but in response to the first re-centering input, the two sets are optionally combined and processed as a single set proceeding (e.g., according to the set rules described above). Thus, in response to a subsequent re-centering input, the virtual objects within the combined set of virtual objects are optionally re-centered together according to various conditions for re-centering described above. Re-centering groups of virtual objects together in response to further re-centering inputs reduces the number of inputs required to properly position virtual elements within a three-dimensional environment.

[0188] It should be understood that the particular order in which the operations in method 800 are described is illustrative only and does not indicate that the described order is the only order in which the operations may be performed. One of ordinary skill in the art will recognize various ways to reorder the operations described herein.

[0189] 9A-9C illustrate examples of computer systems for re-centering one or more virtual objects in the presence of physical or virtual obstacles, according to some embodiments.

[0190] FIG. 9A shows a three-dimensional environment 902 visible via a display generation component of computer system 101 (e.g., display generation component 120 of FIG. 1), and the three-dimensional environment 902 is visible from a user's viewpoint 926a shown in a top view (e.g., facing the left wall of the physical environment in which computer system 101 is located). As described above with reference to FIGS. 1-6, computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., image sensors 314 of FIG. 3). The image sensors optionally include a visible light camera, an infrared camera, a depth sensor, or one or more of any other sensors that computer system 101 could use to capture one or more images of the user or a part of the user (e.g., one or more hands of the user) while the user interacts with computer system 101. In some embodiments, the user interface illustrated and described below may also be implemented on a head-mounted display that includes a display generation component for displaying the user interface or the three-dimensional environment to the user and sensors for detecting movement of the physical environment and / or the user's hand (e.g., external sensors facing outward from the user), and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0191] As shown in Figure 9A, the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., the operating environment 100), including one or more objects in the physical environment surrounding the computer system 101. In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 902, and / or the physical environment is visible in the three-dimensional environment 902 via the display generation component 120. For example, the three-dimensional environment 902 visible via the display generation component 120 includes a representation of the physical floor, as well as the back and side walls of the room in which the computer system 101 is located. The three-dimensional environment 902 also includes a table 922a (corresponding to 922b in the overhead view) visible via the display generation component 120 from viewpoint 926a in Figure 9A, and a sofa 924b (shown in the overhead view) invisible via the display generation component 120 from the user's viewpoint 926a in Figure 9A.

[0192] In Figure 9A, the three-dimensional environment 902 also includes virtual objects 906a (corresponding to object 906b in the overhead view), 908a (corresponding to object 908b in the overhead view), and 910a (corresponding to object 910b in the overhead view) that are visible from viewpoint 926a. The three-dimensional environment 902 also includes virtual objects 912b, 914b, 916b, 918b, and 920b (shown in the overhead view) that are invisible from the user's viewpoint 926a in Figure 9A via the display generation component 120. Virtual objects 912b, 914b, 916b, 918b, and 920b are, optionally, the virtual objects that were last placed or positioned within the three-dimensional environment 902 from viewpoint 926b (e.g., the user's previous viewpoint), as described with reference to Figures 7A-7F and / or Method 800. In Figure 9A, objects 906a, 908a, 910a, 912b, 914b, 916b, 918b, and 920b are two-dimensional objects, but the examples of this disclosure apply equally to three-dimensional objects at will. The virtual objects 906a, 908a, 910a, 912b, 914b, 916b, 918b, and 920b are, at will, one or more of the following: an application user interface (e.g., a messaging user interface, or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by computer system 101 that is not included in the physical environment of computer system 101.

[0193] As illustrated with reference to Figures 7A-7F and / or Method 800, in some embodiments, virtual objects that were last placed or repositioned from a particular previous viewpoint (or a set of previous viewpoints) of the user may be recentered in the user's new current viewpoint. However, in some situations, the locations where those virtual objects would otherwise be recentered in the current viewpoint may already be occupied by other objects (virtual or physical) in the current viewpoint. Therefore, the computer system 101 may need to adjust or shift the locations where the aforementioned virtual objects are recentered, as will be described in more detail below with reference to Method 1000.

[0194] For example, in Figure 9A, the computer system 101 detects a recentering input (as described in more detail, for example, with reference to Method 1000). In some embodiments, in response to such a recentering input, the computer system 101 displays an animation of the recentered virtual objects moving to their initial target locations for recentering, and then shifting from those initial target locations to the final target locations if those initial target locations are already occupied by the objects, as shown in Figures 9B-9C. In some embodiments, the computer system 101 instead simply displays the animation of the recentered virtual objects moving to their final target locations (as shown, for example, as shown in Figure 9C) without displaying the virtual objects moving to their initial target locations (as shown, for example, as shown in Figure 9B).

[0195] Referring to Figure 9B, in some embodiments, the computer system 101 moves and displays the recentered virtual objects to their initial target locations in response to the recentering input in Figure 9A. For example, virtual objects 912a, 914a, 916a, 918a, and 920a are shown in Figure 9B at their initial locations (e.g., locations where the objects are recentered if they are not yet occupied by another virtual or physical object) and / or final target locations for recentering. For example, virtual object 912a is optionally animated to move from its location in Figure 9A to its location in Figure 9B in response to the recentering input in Figure 9A. The location and / or orientation of virtual object 912a shown in Figure 9B is optionally determined by the computer system 101 in one or more of the methods described with reference to Method 800. The location of virtual object 912a in Figure 9B is optionally its final target location because that location is not occupied by another object, whether virtual or physical.

[0196] The virtual object 920a was optionally animated to move from the location in Figure 9A to the location in Figure 9B in response to the recentering input in Figure 9A. The location and / or orientation of the virtual object 920a shown in Figure 9B is optionally determined by the computer system 101 in one or more of the methods described with reference to Method 800. The location of the virtual object 920a in Figure 9B is optionally its final target location, because that location is not occupied by another object, whether virtual or physical.

[0197] Virtual objects 914a, 916a, and 918a are optionally animated to move from their locations in Figure 9A to their locations in Figure 9B in response to a recentering input in Figure 9A. The locations and / or orientations of the virtual objects 914a, 916a, and 918a shown in Figure 9B are optionally determined by the computer system 101 in one or more of the methods described with reference to Method 800. The locations of the virtual objects 914a, 916a, and 918a in Figure 9B are optionally their initial target locations and not their final target locations, because locations are occupied by other objects, whether virtual or physical. For example, virtual object 914a is optionally recentered to a location inside and / or behind the left wall of the physical environment of the computer system 101 and / or occupied by the left wall, according to one or more features of Method 800. The virtual object 916a is optionally recentered to a location within and / or occupied by table 922a, according to one or more features of method 800. Finally, the virtual object 918a is optionally recentered to a location within and / or occupied by virtual object 910a, according to one or more features of method 800.

[0198] Furthermore, in some embodiments, virtual objects that were last placed or repositioned within the three-dimensional environment 902 from the current viewpoint 926a and do not overlap and / or collide with other virtual objects are not moved within the three-dimensional environment 902 in response to the recentering input, such as being reflected by virtual object 910a which does not move in response to the recentering input. However, in some embodiments, virtual objects that were last placed or repositioned within the three-dimensional environment 902 from the current viewpoint 926a and overlap and / or collide with other virtual objects are moved within the three-dimensional environment 902 in response to the recentering input, such as being reflected by virtual objects 906a and 908a. For example, in Figure 9A, virtual object 908a obscured virtual object 906a from viewpoint 926a. Therefore, in response to the recentering input, the computer system 101 moves virtual objects 906a and 908b apart to reduce and / or eliminate the interference of virtual object 906a by virtual object 908a. Further details on how the computer system 101 shifts such overlapping or conflicting virtual objects are provided by referring to Method 800.

[0199] In some embodiments, in response to the reception of a recentering input and / or during the movement of a virtual object in response to the recentering input, the computer system 101 modifies the display of the virtual objects to indicate that recentering is taking place, is taking place, and / or has taken place, as reflected by the crosshatch pattern of one or more virtual objects displayed by the computer system 101 in Figure 9B. For example, the computer system 101 optionally reduces the opacity, brightness, saturation, blur, and / or otherwise reduces the visual prominence of one or more virtual objects displayed by the computer system 101. In some embodiments, the computer system 101 applies the above-described visual modifications to all virtual objects displayed by the computer system 101, regardless of whether those virtual objects have been moved in response to the recentering input. In some embodiments, the computer system 101 applies the above-described visual modifications to virtual objects that have been moved in response to the recentering input, regardless of whether those virtual objects were last placed or positioned in the three-dimensional environment 902 from the current viewpoint 926a or a previous viewpoint 926b, but not to virtual objects that have not been moved in response to the recentering input. In some embodiments, the computer system 101 applies the above-described visual changes to virtual objects that were last placed or positioned in the three-dimensional environment 902 from the previous viewpoint 926b (e.g., virtual objects recentered to viewpoint 926a), but does not apply them to virtual objects that were last placed or positioned in the three-dimensional environment 902 from the current viewpoint 926a (e.g., virtual objects 906a and / or 908a), even if such virtual objects have moved in response to the recentering input.

[0200] In some embodiments, as described above, the computer system 101 shifts virtual objects that have been recentered to an initial target location containing another object to a final target location, as described in more detail with reference to Method 1000, to avoid collisions (one or more) between those recentered virtual objects and / or objects occupying their initial target locations. The computer system 101 optionally shifts the recentered virtual objects differently depending on the type of object they are colliding with. For example, the initial target location of the virtual object 914a shown in Figure 9B is occupied by a physical wall in the physical environment of the computer system 101. Therefore, the computer system 101 optionally moves the virtual object 914a to a final target location away from the physical wall, toward viewpoint 926a (and / or toward viewpoint 926a, rather than up, down, left, and / or right), as shown in Figure 9C.

[0201] In contrast, the initial target location of the virtual object 916a shown in Figure 9B is occupied by the physical table 922a. Therefore, the computer system 101 optionally moves the virtual object 916a to a final target location away from the physical table 922a, in the direction of viewpoint 926a (not optionally toward viewpoint 926a), up, down, left, and / or right, as shown in Figure 9C. In some embodiments, the computer system 101 moves the virtual object in one or more of the above directions such that the amount of movement of the virtual object is minimized in order to avoid collision objects. For example, from Figure 9B to Figure 9C, the computer system 101 moves the virtual object 916a to a final target location where the virtual object 916a is no longer in collision with the physical table 922a.

[0202] As a final example, the initial target location of virtual object 918a shown in Figure 9B is occupied by virtual object 910a. Therefore, the computer system 101 optionally moves virtual object 918a to a final target location away from virtual object 910a, as shown in Figure 9C, by moving up, down, left, and / or right relative to viewpoint 926a, and / or toward or away from it. In some embodiments, the computer system 101 moves the virtual object in one or more of the above directions such that the amount of movement of the virtual object is minimized in order to avoid collision objects. For example, from Figure 9B to Figure 9C, the computer system 101 moves virtual object 918a to the left to a final target location where virtual object 918a is no longer in collision with virtual object 910a.

[0203] As described above, in Figures 9B to 9C, virtual objects other than virtual objects 914a, 916a, and 918a are optionally not moved by the computer system 101. The computer system 101 optionally reverses, at least partially or completely, the visual changes of a given virtual object as described with reference to Figure 9B, depending on whether the virtual object has reached its final target location. In some embodiments, the computer system 101 reverses, at least partially or completely, the visual changes of each virtual object as described with reference to Figure 9B, depending on whether each virtual object has reached its final target location. The partial or complete reversal of the visual changes of the virtual objects as described with reference to Figure 9B is optionally reflected in Figure 9C by the absence of a crosshatch pattern in the displayed virtual object.

[0204] Figures 10A to 10G are flowcharts illustrating methods for recentering one or more virtual objects in the presence of physical or virtual obstacles, according to several embodiments. In some embodiments, Method 1000 is performed on a computer system (e.g., computer system 101 in Figure 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generating component (e.g., display generating component 120 in Figures 1, 3, and 4) (e.g., a head-up display, display, touchscreen, or projector) and one or more cameras (e.g., a camera pointing downwards from the user's hand (e.g., a color sensor, infrared sensor, and other depth-sensing camera) or a camera pointing forward from the user's head). In some embodiments, Method 1000 is performed by instructions stored in a non-temporary computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., controller 110 in Figure 1A). Some operations of method 1000 are combined in an optional manner, and / or the order of some operations is changed in an optional manner.

[0205] In some embodiments, Method 1000 is performed in a computer system (e.g., 101) that communicates with a display generation component and one or more input devices. In some embodiments, the computer system has one or more characteristics of the computer system of Method 800. In some embodiments, the display generation component has one or more characteristics of the display generation component of Method 800. In some embodiments, one or more input devices have one or more characteristics of the one or more input devices of Method 800.

[0206] In some embodiments, a three-dimensional environment (e.g., 902) (for example, the three-dimensional environment optionally has one or more characteristics of the three-dimensional environment of Method 800) is visible from a first user viewpoint (e.g., as described with reference to Method 800), such as viewpoint 926a in Figure 9A, via a display generation component, but the three-dimensional environment includes a first virtual object in a first location within the three-dimensional environment (e.g., object 916a in Figure 9A) (for example, the first virtual object optionally has one or more characteristics of the first virtual object in Method 800). In some embodiments, the first virtual object is visible from the user If, while the viewpoint was the viewpoint prior to the first viewpoint, the computer system (e.g., 101) has been placed in a first location in a three-dimensional environment by the user of the computer system, and has been last reoriented or last moved, the computer system (e.g., 101) receives a first input via one or more input devices in response to a request to update the spatial positioning of the first virtual object relative to the user's first viewpoint, such as the input in Figure 9A (e.g., as described with reference to Method 800) (1002a). The first input optionally has one or more properties of the first input (e.g., a recentering input) as described with reference to Method 800 and / or 1400.

[0207] In some embodiments, in response to receiving a first input (1002b), the spatial arrangement of the second location with respect to the user's first viewpoint satisfies a first set of one or more criteria, as described with reference to Method 800, according to the determination that a second location in a three-dimensional environment that satisfies a first set of one or more criteria (e.g., a location to which the computer system moves a first virtual object if no object already exists at the second location, such as according to one or more aspects of Method 800) is not occupied by an object, such as the location shown in Figure 9B where object 912a is (e.g., does not contain a distinct object whether virtual or physical, does not contain a distinct type of virtual or physical object, or does not contain a virtual or physical object), and so on. In some embodiments, the spatial arrangement of the second location relative to the first viewpoint corresponds to the spatial arrangement of the first location relative to the user's previous viewpoint from which the first virtual object was last placed or moved (e.g., the same), and the computer system (e.g., 101) displays the first virtual object at the second location in a three-dimensional environment, such as the location shown in Figure 9C (e.g., moves the first virtual object) (1002c). In some embodiments, the orientation of the first virtual object at the second location relative to the first viewpoint corresponds to the orientation of the first virtual object at the first location when the first input was received relative to the user's previous viewpoint from which the first virtual object was last placed or moved (e.g., the same).

[0208] In some embodiments, upon receiving a first input (1002b), and in accordance with the determination that a second location in the three-dimensional environment that satisfies a first set of one or more criteria, such as the location where object 916a is shown in Figure 9B, is occupied (e.g., containing at least one distinct object, whether physical or virtual, or containing one or more virtual or physical objects of distinct types), the computer system (e.g., 101) displays the first virtual object in a third location in the three-dimensional environment that satisfies a first set of one or more criteria, and is spaced apart from the second location in the three-dimensional environment, such as the location where object 916a is shown in Figure 9C (e.g., moves the first virtual object) (1002d). In some embodiments, the orientation of the first virtual object in the third location relative to the first viewpoint corresponds to the orientation of the first virtual object in the first location when the first input was received relative to the user's previous viewpoint where the first virtual object was last placed or moved (e.g., they are the same). In some embodiments, the orientation of the first virtual object in the third location relative to the first viewpoint is different from the orientation of the first virtual object in the first location when the first input is received relative to the user's previous viewpoint to which the first virtual object was last placed or moved. In some embodiments, the spatial arrangement of the third location relative to the first viewpoint is different from the spatial arrangement of the first location relative to the user's previous viewpoint to which the first virtual object was last placed or moved. In some embodiments, the distance and / or orientation of the third location relative to the user's first viewpoint satisfies one or more first criteria, as described with reference to Method 800. In some embodiments, the computer system selects the third location so that it is sufficiently far from the second location that the first virtual object in the third location does not occupy any volume of the three-dimensional environment that is also occupied by individual objects in the second location, as will be described in more detail below.In some embodiments, the input described with reference to Method 1000 is an air gesture input or includes an air gesture input. By shifting the location where virtual objects are recentered, the computer system automatically avoids collisions between objects in a three-dimensional environment.

[0209] In some embodiments, the second location is determined to be occupied when object 918a includes a virtual object such as the location shown in Figure 9B and is occupied by object 910a (e.g., other virtual objects described herein and / or virtual objects having one or more of the characteristics of Methods 800, 1200, 1400, and / or 1600) (1004). In some embodiments, the second location is determined to be occupied if the first virtual object appears in the second location and collides with the virtual object (any part thereof). In some embodiments, the second location is determined to be occupied if the first virtual object appears in the second location and obscures or is (at least partially) obscured by the virtual object, regardless of whether the first virtual object collides with the virtual object. Thus, in some embodiments, the recentered virtual object will be shifted to avoid collision with existing virtual objects in the second location. By shifting the location where virtual objects are recentered, computer systems automatically avoid collisions between virtual objects in a three-dimensional environment.

[0210] In some embodiments, the second location is determined to be occupied if it corresponds to the location of a physical object in the user's physical environment, such as the location where object 916a is shown in Figure 9B, and is occupied by a table 922a (e.g., a wall or a table) (1006). In some embodiments, the second location is determined to be occupied if the first virtual object is displayed in the second location and collides with any physical object. The physical object is optionally visible in the second location via a display-generating component, and / or a representation of the physical object is displayed in the second location via a display-generating component. In some embodiments, the second location is determined to be occupied if the first virtual object is displayed in the second location and obscures or (at least partially) obscures any physical object, regardless of whether the first virtual object collides with a virtual object. Thus, in some embodiments, the recentered virtual object will be shifted to avoid collision with existing physical objects in the second location. By shifting the location where virtual objects are recentered, computer systems automatically avoid collisions between virtual and physical objects in a three-dimensional environment.

[0211] In some embodiments, the third location is closer to the user's first viewpoint than the second location, according to a determination that the second location corresponds to a location inside or behind a physical wall in the user's physical environment, such as the location where object 914a is shown in FIG. 9B (e.g., when the first virtual object is displayed at the second location, the surface of the wall facing the user's viewpoint is closer to the user's viewpoint than the second location, such that the first virtual object is displayed inside or behind the physical wall in the three-dimensional environment). In some embodiments, the third location is in front of a physical wall with respect to the user's first viewpoint, such as the location where object 914a is shown in FIG. 9C (1008). In some embodiments, when a recentered virtual object collides with and / or is behind a physical wall in a three-dimensional environment, the computer system avoids the collision by shifting the location of the recentered virtual object closer to the user's viewpoint (e.g., without shifting the location of the recentered virtual object laterally with respect to the user's viewpoint). The computer system optionally additionally performs the above in the case of other physical objects that are wall-like objects but not walls (e.g., objects that are relatively perpendicular to the user's viewpoint and have a size or area that exceeds a threshold size or area, such as 0.2, 0.5, 1, 3, 5, or 10 meters vertically and / or horizontally, and / or 0.04, 0.25, 1, 9, 25, or 100 square meters). Shifting the location where the virtual object is recentered towards the user's viewpoint in the case of a wall reduces the number of inputs required to ensure the visibility and / or interactability of the virtual object in the three-dimensional environment, since a lateral shift of the location of the virtual object is likely not to resolve a collision between the virtual object and the wall.

[0212] In some embodiments, in response to a first input, a third location is determined to be at the same distance from the user's first viewpoint as the second location, and is laterally separated from the second location relative to the first viewpoint, such as the location shown in Figure 9C, where object 916a is located (1010). In some embodiments, if the recentered virtual object collides with a physical object other than a wall in the three-dimensional environment, the computer system avoids the collision by shifting the location of the recentered virtual object laterally relative to the user's viewpoint (e.g., up, down, left, and / or right) (e.g., by not shifting the location of the recentered virtual object toward or away from the user's viewpoint). For non-wall objects, shifting the location where the virtual object is recentered laterally relative to the user's viewpoint reduces the number of inputs required to ensure the visibility and / or interactability of the virtual object within the three-dimensional environment.

[0213] In some embodiments, upon receiving a first input, the three-dimensional environment further includes a second virtual object that overlaps with the first virtual object, such as objects 906a and 908a in Figure 9A (for example, the first and second virtual objects collide with each other at least partially, and / or the first virtual object at least partially obscures the second virtual object from the first viewpoint, or the second virtual object at least partially obscures the first virtual object from the first viewpoint) (1012a).

[0214] In some embodiments, upon receiving a first input, the computer system (e.g., 101) separates the first and second virtual objects from each other (e.g., laterally with respect to the first viewpoint and / or toward or away from the first viewpoint) (1012b) in order to reduce or eliminate overlap between the first and second virtual objects, as shown in Figure 9C with respect to objects 906a and 908a. In some embodiments, both virtual objects are moved to achieve the above separation. In some embodiments, only one of the virtual objects is moved to achieve the above separation. In some embodiments, both the first and second virtual objects are recentered in response to the first input and, in processing, separated from each other to achieve the above separation. Separating overlapping virtual objects reduces the number of inputs required to ensure the visibility and / or interactability of the virtual objects in the three-dimensional environment.

[0215] In some embodiments, in response to a first input and a determination that the second location is occupied by a separate object (e.g., a physical object such as a wall or a non-wall object, or a virtual object) (1014a), the third location is separated from the second location in a first direction, such as shifting object 916a upward rather than downward from the location shown in Figure 9B, according to a determination that the amount of separation from the second location in a first direction required for the first virtual object to avoid the separate object at the second location is less than the amount of separation from the second location in a second direction different from the first direction required for the first virtual object to avoid the separate object at the second location (1014b). For example, if shifting the location of a first virtual object in a first direction (e.g., right, left, up, down, away from or towards the viewpoint, or any combination of these directions) to avoid collisions or overlaps between the first virtual object and individual objects requires a smaller shift than shifting the location of the first virtual object in a second direction (e.g., right, left, up, down, away from or towards the viewpoint, or any combination of these directions) to avoid collisions or overlaps between the first virtual object and individual objects, the computer system will optionally shift the location of the first virtual object in the first direction (e.g., by a smaller amount).

[0216] In some embodiments, in response to a first input and a determination that the second location is occupied by a separate object (e.g., a physical object such as a wall or a non-wall object, or a virtual object) (1014a), the third location is separated from the second location in a second direction by shifting object 916a downward rather than upward from the location shown in Figure 9B, such as when the movement of object 916a is small and table 922a is avoided, in response to a first input and a determination that the second location is occupied by a separate object (e.g., a physical object such as a wall or a non-wall object, or a virtual object) (1014a), in response to a determination that the amount of separation from the second location in a second direction required for the first virtual object to avoid the separate object at the second location is less than the amount of separation from the second location in a first direction required for the first virtual object to avoid the separate object at the second location (1014c). For example, if shifting the location of a first virtual object in a second direction to avoid collision or overlap with an individual object requires a smaller shift than shifting the location of the first virtual object in a first direction to avoid collision or overlap with an individual object, the computer system may optionally shift the location of the first virtual object in the second direction (by a smaller amount). Thus, in some embodiments, the computer system shifts the location of the first virtual object in a direction that requires a smaller (e.g., minimum) shift of the location of the first virtual object to avoid collision or overlap with an individual object. Shifting the first virtual object in a direction that requires a smaller shift automatically causes the computer system to appropriately position the first virtual object to avoid collisions while keeping it closer (e.g., as close as possible) to its initial target location.

[0217] In some embodiments, displaying the first virtual object at a third location includes, via a display generation component, displaying an animation of the representation of the first virtual object moving to a second location, such as an animation of the object 916a moving to the location shown in Figure 9B, then an animation of the object 916a moving to the location shown in Figure 9C, and subsequently an animation of the representation of the first virtual object moving from the second location to the third location (1016). In some embodiments, the computer system displays an animation of the first virtual object moving to a second location in a three-dimensional environment in response to a first input (e.g., a faded, visually de-emphasized, darker, blurred, desaturated, and / or more translucent representation of the first virtual object), and then displays an animation of the first virtual object moving from the second location to the third location in a three-dimensional environment (e.g., a faded, visually de-emphasized, darker, blurred, desaturated, and / or more translucent representation of the first virtual object). In some embodiments, the first and second animations occur after the first input (e.g., in response to the first input) without any further input being detected. In some embodiments, when the first virtual object reaches the third location, the computer system displays the first virtual object as brighter, less blurred, more saturated, and / or less translucent (e.g., the visual appearance the first virtual object had when the first input was received), no longer faded, no longer visually suppressed. Feedback regarding the original recentered location of the first virtual object is provided by displaying an animation in which the first virtual object first moves to the second location and then to the third location.

[0218] In some embodiments, the third location is separated from the second location by one or more of the following: distance from the user's first viewpoint, horizontal distance to the user's first viewpoint, or vertical distance to the user's first viewpoint (1018). For example, the computer system may optionally shift the location of the first virtual object from the second location in any direction, for example, toward the user's viewpoint, away from the user's viewpoint, horizontally to the viewpoint or user, vertically to the user's viewpoint, or any combination thereof. Shifting the location of the first virtual object in the above directions reduces the number of inputs required to properly position the first virtual object in the three-dimensional environment.

[0219] In some embodiments, the first virtual object is last placed or positioned at a first location in the three-dimensional environment from a second viewpoint of the user that is different from the user's first viewpoint (for example, from a viewpoint that is sufficiently different from the user's current viewpoint, as will be described in more detail with reference to Method 800) (1020a).

[0220] In some embodiments, the second location is a first individual location, such as object 920a in Figure 9C (1020b), according to the determination that the spatial arrangement of the first location with respect to the second viewpoint is the first spatial arrangement. For example, if the location and / or orientation of the first virtual object with respect to the second viewpoint (e.g., the viewpoint from which the first virtual object was last placed or positioned in the three-dimensional environment) is such that the first virtual object is to the right and upward with respect to the second viewpoint, the computer system selects the second location such that the location and / or orientation of the first virtual object in the second location with respect to the first viewpoint is also to the right and upward with respect to the first viewpoint (e.g., the same relative location and / or orientation). In some embodiments, the magnitude of the relative location and / or orientation of the second location with respect to the first viewpoint is also maintained with respect to the relative location and / or orientation of the first location with respect to the second viewpoint.

[0221] In some embodiments, the second location is a second distinct location distinct from the first distinct location (1020c), based on the determination that the spatial arrangement of the first location with respect to the second viewpoint is a second spatial arrangement different from the first spatial arrangement. For example, if object 920a has a different spatial arrangement with respect to viewpoint 926b in Figure 9A, object 920a will optionally have that different spatial arrangement with respect to viewpoint 926a in Figure 9C. For example, if the location and / or orientation of the first virtual object with respect to the second viewpoint is such that the first virtual object is to the left and below the second viewpoint, the computer system selects the second location such that the location and / or orientation of the first virtual object in the second location with respect to the first viewpoint is also to the left and below the first viewpoint (e.g., the same relative location and / or orientation). In some embodiments, the magnitude of the location and / or orientation of the second location with respect to the first viewpoint is also maintained with respect to the relative location and / or orientation of the first location with respect to the second viewpoint. Setting a target location for a recentered virtual object, based on the virtual object's location relative to the user's previous viewpoint when the virtual object was last positioned in the three-dimensional environment, allows the computer system to automatically position the virtual object at the relative location previously provided for the virtual object.

[0222] In some embodiments, the first virtual object is last placed or positioned in the three-dimensional environment from a second viewpoint of the user that is different from the first viewpoint (e.g., from a viewpoint that is sufficiently different from the user's current viewpoint, as described in more detail with reference to Method 800), and when the first input is received, the three-dimensional environment further includes the second and third virtual objects that were last placed or positioned in the three-dimensional environment from the user's first viewpoint (or a viewpoint of the user that is not sufficiently different from the user's current viewpoint, as described in more detail with reference to Method 800), and the second and third virtual objects have a first separate spatial arrangement with respect to the first viewpoint, such as objects 906a, 908a, and / or 910a in Figure 9A and their spatial arrangement with respect to viewpoint 926a in Figure 9A (1022a).

[0223] In some embodiments, upon receiving a first input (1022b), and in accordance with a determination that the second and third virtual objects are overlapping, such as objects 906a and 908a in Figure 9A overlapping (e.g., colliding at least partially with each other in a three-dimensional environment and / or obscuring each other at least partially from the user's first viewpoint), the computer system (e.g., 101) updates the spatial arrangement of the second and third virtual objects to the second separate spatial arrangement with respect to the first viewpoint (1022c), thereby reducing or eliminating the overlap between the second and third virtual objects, as shown in objects 906a and 908a in Figures 9B and 9C (e.g., moving and / or reorienting both the first, second, or first and second virtual objects to increase the (e.g., horizontal, vertical, and / or depth) distance between the objects with respect to the first viewpoint to reduce or eliminate collisions between the two objects and / or obscuration of the two objects).

[0224] In some embodiments, upon receiving a first input (1022b), if the objects 906a and 908a in Figure 9A do not overlap (e.g., they do not collide with each other at least partially in the three-dimensional environment and / or do not obscure each other at least partially from the user's first viewpoint), the computer system (e.g., 101) maintains the second and third virtual objects having a first distinct spatial arrangement with respect to the first viewpoint, such as by not moving objects 906a and / or 908a in response to the input of Figure 9A (e.g., not moving or changing the orientation of the first and second virtual objects in the three-dimensional environment) (1022d). Thus, in some embodiments, the virtual objects last placed or positioned in the three-dimensional environment from the user's current viewpoint do not respond to the first input unless they overlap in the three-dimensional environment. Shifting the first and / or second virtual objects only when they overlap reduces the number of inputs required to properly position the first and second virtual objects within the three-dimensional environment.

[0225] In some embodiments, upon receiving a first input, a computer system (e.g., 101) displays a visual indication (1024) via a display generation component that a first input has been received, such as a change in the visual appearance of objects 906a, 908a, and / or 910a from Figure 9A to Figure 9B. In some embodiments, the visual indication is displayed for a predetermined amount of time (e.g., 0.3, 0.5, 1, 2, 3, 5, or 10 seconds) after the first input has been received. In some embodiments, the visual indication is displayed for the duration of the movement of one or more virtual objects in the three-dimensional environment in response to the first input, and the display stops upon the end of that movement. In some embodiments, the visual indication is or includes a change in the visual appearance of one or more elements that were in the three-dimensional environment when the first input was received (e.g., a change in the visual appearance of one or more virtual objects that were in the three-dimensional environment when the first input was received, as will be described in more detail below). In some embodiments, the visual indication is or includes a display of an element (e.g., a notification) that was not displayed or included in the three-dimensional environment when the first input was received. Displaying an indication of the first input provides feedback on the current status of the computer system when recentering one or more virtual objects in the three-dimensional environment.

[0226] In some embodiments, when a first input is received, the first virtual object has (for example, is displayed with) visual properties having first values ​​(e.g., having first luminance, first opacity, first blur, and / or first color), and a visual indication that a first input has been received includes temporarily updating (and / or displaying) the first virtual object to have visual properties having second values ​​different from the first values ​​(e.g., a second luminance lower than the first luminance, a second opacity lower than the first opacity, a second blur higher than the first blur, and / or a second color lower than the first color), such as the visual appearance of object 916a in Figure 9B, and then displaying the first virtual object with visual properties having first values ​​(e.g., returning the first virtual object to have its initial visual appearance), such as the visual appearance of object 916a in Figure 9C (1026). In some embodiments, in response to a first input, a first virtual object is temporarily visually de-emphasized in the three-dimensional environment (for example, with respect to the rest of the three-dimensional environment and / or with respect to parts of the three-dimensional environment that have not changed position and / or orientation in response to the first input). In some embodiments, the aforementioned change in the visual appearance of the first virtual object is maintained for the duration of the movement of the virtual object(s) in the three-dimensional environment in response to the first input and is restored upon completion of that movement. In some embodiments, the above change in visual appearance is additionally or alternatively applied to other virtual objects that are moved / reoriented in the three-dimensional environment in response to the first input. In some embodiments, the above change in visual appearance is additionally or alternatively applied to virtual objects that are not moved / reoriented in the three-dimensional environment in response to the first input. In some embodiments, the virtual object whose visual appearance is altered is partially or completely faded in the three-dimensional environment in response to the first input until the fading disappears as described above. Adjusting the visual appearance of one or more virtual objects in response to a first input provides feedback on the current status of the computer system when recentering one or more virtual objects in a three-dimensional environment.

[0227] In some embodiments, upon receiving a first input, the three-dimensional environment further includes a second virtual object at a fourth location within the three-dimensional environment (for example, the second virtual object is an object that is recentered within the three-dimensional environment together with the first virtual object in response to the first input), and the first and second virtual objects have a first separate spatial arrangement with respect to each other, such as objects 912a and 920a in Figure 9A (1028a).

[0228] In some embodiments, upon receiving a first input (1028b), and in accordance with the determination that a second location is not occupied by an object, the computer system (e.g., 101) displays the first virtual object in the second location and the second virtual object in a fifth location different from the fourth location, which satisfies a first set of one or more criteria (e.g., moving and / or reorienting both the first and second virtual objects in a three-dimensional environment in response to the first input, as described above and / or with reference to Method 800), where the first and second virtual objects in the second and fifth locations have a first distinct spatial arrangement relative to each other, such as objects 912a and 920a having the same spatial arrangement relative to each other in Figure 9C as in Figure 9A (1028c) (e.g., the relative orientation and / or position of the first and second virtual objects are maintained in accordance with the recentering of these virtual objects, as described in more detail with reference to Method 800).

[0229] In some embodiments, upon receiving a first input (1028b), and in accordance with the determination that a second location is occupied with respect to an object 918a in Figure 9B, the computer system (e.g., 101) displays the first virtual object in a third location and the second virtual object in a sixth location different from the fourth location (e.g., optionally the same as or different from the fifth location), which satisfies a first set of one or more criteria (as described above, the target location(s) of those objects(s) Except that the orientation is shifted by the computer system, both the first and second virtual objects are moved and / or reoriented in the three-dimensional environment in response to the first input. The first and second virtual objects at the third and sixth locations have a second distinct spatial arrangement relative to each other, different from the first distinct spatial arrangement, such as objects 918a and 920a in Figure 9C, which have different spatial arrangements relative to each other than those in Figure 9A (1028d) (for example, if the target location(s) of the virtual object(s) is occupied when the first input is received, the virtual objects will optionally not maintain their relative orientation and / or position in response to re-inputting those virtual objects). Maintaining the relative spatial arrangement of recentered virtual objects, where possible, reduces the number of inputs required to properly position the objects relative to the user's viewpoint in response to the first input.

[0230] In some embodiments, when a first input is received, the three-dimensional environment includes a first distinct virtual object (e.g., the first virtual object or a different virtual object) located at a first distinct location within the three-dimensional environment, and a second distinct virtual object located at a second distinct location within the three-dimensional environment (e.g., the first distinct virtual object is recentered in response to the first input, and the second distinct virtual object is optionally recentered in response to the first input or optionally not recentered in response to the first input) (1030a).

[0231] In some embodiments, upon receiving a first input (1030b), the computer system (e.g., 101) displays a second separate virtual object at a third separate location in the three-dimensional environment (e.g., a location different from the second separate location if the second separate virtual object is recentered in response to the first input, or the same as the second separate location if the second separate virtual object is not recentered in response to the first input) (1030c).

[0232] In some embodiments, upon receiving a first input (1030b), the fourth individual location is further from the user's first viewpoint than the third individual location and satisfies a first set of one or more criteria (for example, the fourth individual location is the initial target location of the first individual virtual object in response to the first input, referring to the method and / or method 800 described above), according to the determination that the difference in distance between the fourth individual location and the third individual location from the user's first viewpoint is greater than a threshold distance (e.g., a difference in distance of 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, 100, 500, 1000, or 5000 cm from the user's first viewpoint), and the computer system (e.g., 10 1) displays the first virtual object in a fourth separate location, and the second separate virtual object in the third separate location is at least partially obscured from the user's first viewpoint by object 910a, because object 918a is recentered to the location behind object 910a and remains there, and is obscured by object 910a in Figure 9B, and the location behind object 910a is at least a threshold distance away from object 910a (optionally, the first and second virtual objects, respectively, do not collide in the three-dimensional environment when displayed in the fourth separate location and the third separate location) (1030d). For example, the computer system recenters the first separate virtual object in the fourth separate location even if the second separate virtual object at least partially obscures the first separate virtual object from the user's first viewpoint. Therefore, in some embodiments, the computer system shifts the target location of a virtual object in response to a first input if the virtual object collides with another virtual object, but does not shift the target location of the virtual object in response to the first input based on the virtual object that obscures (but does not collide with) the other virtual object from the user's viewpoint (or vice versa) if the two objects are sufficiently far apart in depth relative to the user's viewpoint.

[0233] In some embodiments, upon receiving a first input (1030b), and in accordance with the determination that the difference in distance between a fourth individual location and a third individual location from the user's first viewpoint is less than a threshold distance (e.g., a difference in distance of 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, 100, 500, 1000, or 5000 cm from the user's first viewpoint), the computer system (e.g., 101) displays the first individual virtual object at a fifth individual location different from the fourth individual location, where the fifth individual location is further from the user's first viewpoint than the third individual location and satisfies a first set of one or more criteria (e.g., the fifth individual location is a shift of the first individual virtual object in response to the first input, referring to the methods described above and / or method 800). If object 918a is recentered to a location behind object 910a in Figure 9B (which is the target location), but that location behind object 910a is not separated from object 910a by at least a threshold distance, and therefore the computer system 101 changes the location of object 918a so that it is not obscured by object 910a, then each of the second virtual objects in the third separate location will not at least partially obscure the first separate virtual object in the fifth separate location from the user's first viewpoint (1030e) (and optionally, the first and second separate virtual objects will not collide in the three-dimensional environment when displayed in the fifth separate location and the third separate location, respectively). For example, the computer system recenters the first separate virtual object to a fifth separate location selected by the computer system such that the second separate virtual object does not even partially obscure the first separate virtual object from the user's viewpoint.Therefore, in some embodiments, the computer system shifts the target location of a virtual object in response to a first input if the virtual object collides with another virtual object and / or if the two objects obscure another virtual object from the user's viewpoint when they are not sufficiently separated from each other in the depth direction relative to the user's viewpoint (or vice versa). Shifting the recentering location based on collision or line-of-sight obstruction in response to the separation (in depth) of the virtual objects in response to recentering reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to the first input.

[0234] It should be understood that the specific order in which the operations in Method 1000 are described is illustrative only and does not indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein.

[0235] Figures 11A to 11E illustrate embodiments of a computer system that selectively and automatically recenters one or more virtual objects in response to a change in state of a display generation component.

[0236] Figure 11A shows a three-dimensional environment 1102 that is visible through a display generation component of the computer system 101 (e.g., display generation component 120 in Figure 1), and the three-dimensional environment 1102 is invisible from the user's viewpoint 1126 shown in an overhead view (e.g., facing the left wall of the first room 1103a in the physical environment where the computer system 101 is located). As described above with reference to Figures 1 to 6, the computer system 101 optionally includes a display generation component (e.g., a touchscreen) and a plurality of image sensors (e.g., the image sensor 314 in Figure 3). The image sensors optionally include one or more of the following: a visible light camera, an infrared camera, a depth sensor, or any other sensors that the computer system 101 may use to capture one or more images of the user or a part of the user (e.g., one or more of the user's hands) while the user interacts with the computer system 101. In some embodiments, the user interfaces illustrated and described below may also be implemented on a head-mounted display, which includes a display generating component that displays the user interface or three-dimensional environment to the user, and sensors for detecting the physical environment and / or the movement of the user's hands (e.g., external sensors facing outward from the user) and / or the user's line of sight (e.g., internal sensors facing inward toward the user's face).

[0237] As shown in Figure 11A, the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., the operating environment 100), including one or more objects in the physical environment surrounding the computer system 101. In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 1102, and / or the physical environment is visible in the three-dimensional environment 1102 via the display generation component 120. For example, the three-dimensional environment 1102 visible via the display generation component 120 includes representations of the physical floor, as well as the back and side walls of the room 1103a in which the computer system 101 is located. The three-dimensional environment 1102 also includes a table 1122a (corresponding to 1122b in the overhead view) which is visible from viewpoint 1126 in Figure 11A via the display generation component 120, and a sofa 1124b (shown in the overhead view) in a second room 1103b within the physical environment which is invisible from the user's viewpoint 1126 in Figure 11A via the display generation component 120.

[0238] In Figure 11A, the three-dimensional environment 1102 also includes virtual objects 1106a (corresponding to object 1106b in the overhead view), 1108a (corresponding to object 1108b in the overhead view), and 1110a (corresponding to object 1110b in the overhead view) that are visible from viewpoint 1126. Virtual objects 1106a, 1108a, and 1110a are, optionally, virtual objects that were last placed or positioned within the three-dimensional environment 1102 from viewpoint 1126 in Figure 11A, as described with reference to Figures 7A–7F and / or Method 800. In Figure 11A, objects 1106a, 1108a, and 1110a are two-dimensional objects, but the examples of this disclosure apply equally to three-dimensional objects, optionally. The virtual objects 1106a, 1108a, and 1110a are optionally one or more of the following: an application user interface (e.g., a messaging user interface, or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by the computer system 101 that is not included in the physical environment of the computer system 101.

[0239] As described with reference to Figures 7A-7F and / or Method 800, in some embodiments, virtual objects that were last placed or repositioned from a particular previous viewpoint (or a set of previous viewpoints) of the user may be recentered to the user's new current viewpoint. Thus, in some embodiments, if the user's viewpoint changes from that shown in Figure 11A and the computer system 101 detects a recentering input, the computer system 101 recenters the virtual objects 1106a, 1108a, and 1110a to its changed viewpoint, as described with reference to Figures 7A-7F and / or Method 800. However, in some embodiments, the computer system 101 automatically recenters the virtual objects 1106a, 1108a, and 1110a to the user's changed viewpoint if the user's changed viewpoint is sufficiently different from the user's previous viewpoint (in location and / or orientation, for example, as described in more detail with reference to Method 1200). Furthermore, in some embodiments, the computer system 101 performs (or does not perform) such automatic recentering in response to the display generation component 120 transitioning from a second state (e.g., a power-off or off state in which the three-dimensional environment 1102 is invisible via the display generation component 120) to a first state (e.g., a power-on or on state in which the three-dimensional environment 1102 is visible via the display generation component 120), as will be described in more detail below with reference to Method 1200. In the case of a wearable device (e.g., a head-mounted device), the display generation component is optionally in a first state while the device is mounted on the user's head, optionally transitions to a second state in response to detection that the device has been removed from the user's head, and optionally returns to (and optionally remains in) the first state in response to detection that the device is positioned and mounted on the user's head.

[0240] For example, from Figure 11A to Figure 11B, the display generation component 120 transitions from a first state to a second state, and the user moves to a new location in the user's physical environment (e.g., a new location and / or a new orientation) compared to Figure 11A. For example, in Figure 11B, the user moves to a new location corresponding to a new viewpoint 1126 in the first room 1103a in the physical environment, facing the left rear wall of that room 1103a. The three-dimensional environment 1102 is invisible and cannot be displayed via the computer system 101 because the computer system 101 is optionally turned off and / or not mounted on the user's head. Therefore, no virtual objects are shown in Figure 11B.

[0241] In Figures 11B and 11C, the display generation component 120 transitions from a second state to a first state while the user is in a location within the physical environment shown in Figure 11B (and Figure 11C). As shown in Figure 11C, the three-dimensional environment 1102 is again visible through the display generation component 120 of the computer system 101. Furthermore, the user's viewpoint in the three-dimensional environment 1102 corresponds to the user's updated location and / or orientation in the physical environment. In Figure 11C, the user's updated location and / or orientation in the physical environment, and / or the user's viewpoint in the three-dimensional environment 1102 in Figures 11B and 11C, are not, optionally, sufficiently different from those in Figure 11A, and therefore the computer system 101 does not automatically recenter the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint in response to the display generation component 120 transitioning from a second state to a first state. For example, since the user remains in the same room 1103a within the physical environment as shown in Figure 11A, the computer system 101 does not, optionally, automatically recenter the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint. Additional or alternative criteria for automatically recentering the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint are described with reference to Method 1200. As a result, in Figure 11C, the three-dimensional environment 1102 is visible only from a viewpoint different from Figure 11A, rather than being optionally recentered to a different viewpoint in Figure 11C.

[0242] In contrast to Figures 11B and 11C, in Figure 11D, the display generation component 120 has transitioned from a first state to a second state, and the user has moved to a new location (e.g., a new location and / or a new orientation) in the user's physical environment compared to Figure 11A or Figure 11C. For example, in Figure 11D, the user has moved to a new location corresponding to a new viewpoint 1126 in a second room 1103b in the physical environment, facing the back wall of that room 1103b. The three-dimensional environment 1102 is invisible and cannot be displayed via the computer system 101 because the computer system 101 is optionally turned off and / or not mounted on the user's head. Therefore, no virtual objects are shown in Figure 11D.

[0243] In Figures 11D to 11E, the display generation component 120 transitions from a second state to a first state while the user is at a location in the physical environment shown in Figure 11D (and Figure 11E). As shown in Figure 11E, the three-dimensional environment 1102 is again visible via the display generation component 120 of the computer system 101. Furthermore, the user's viewpoint in the three-dimensional environment 1102 corresponds to the user's updated location and / or orientation in the physical environment. In Figure 11E, the user's updated location and / or orientation in the physical environment, and / or the user's viewpoint in the three-dimensional environment 1102 in Figures 11D and 11E, are not, optionally, sufficiently different from those in Figure 11A (and / or Figure 11C), and therefore the computer system 101 does not automatically recenter the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint in response to the display generation component 120 transitioning from a second state to a first state. For example, since the user has moved to a second room 1103b in the physical environment, the computer system 101 optionally automatically recenters the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint, as shown in Figure 11E. Details on how the virtual objects 1106a, 1108a, and 1110a are recentered to the user's updated viewpoint are provided with reference to Methods 800 and / or 1000. Additional or alternative criteria for automatically recentering the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint are described with reference to Method 1200. As a result, in Figure 11E, the three-dimensional environment 1102 is optionally recentered to a different viewpoint than that in Figure 11A (and / or Figure 11C), and is visible from there.

[0244] In some embodiments, the computer system 101 does not automatically recenter the virtual objects and / or the three-dimensional environment to the user's updated viewpoint while the user is in an updated location and / or viewpoint (optionally, after transitioning from a first state to a second state), unless the display generation component transitions from the second state to the first state. For example, if the user moves from the location and / or viewpoint shown in Figure 11A to the location and / or viewpoint shown in Figure 11E while the display generation component 120 remains in the first state, the computer system 101 does not optionally automatically recenter the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint. Instead, while the virtual objects 1106a, 1108a, and 1110a remain in their respective locations within the three-dimensional environment 1102 shown in Figure 11A, the three-dimensional environment 1102 is optionally only visible from the user's updated viewpoint. Therefore, in some embodiments, the condition required for the three-dimensional environment and / or virtual objects to be automatically recentered to the user's updated viewpoint is that the display generation components transition from a second state to a first state while the user is in a location and / or viewpoint that satisfies the automatic recentering criteria (sufficiently different from the user's previous viewpoint, as described in more detail with reference to Method 1200, for example).

[0245] Figures 12A to 12E are flowcharts illustrating methods for selectively and automatically recentering one or more virtual objects in response to a change in state of a display generation component, according to several embodiments. In some embodiments, Method 1200 is performed on a computer system (e.g., computer system 101 in Figure 1, such as a tablet, smartphone, wearable computer, or head-mounted device) that includes a display generation component (e.g., display generation component 120 in Figures 1, 3, and 4) (e.g., a head-up display, display, touchscreen, or projector) and one or more cameras (e.g., a camera pointing downwards from the user's hand (e.g., a color sensor, infrared sensor, and other depth-sensing camera) or a camera pointing forward from the user's head). In some embodiments, Method 1200 is performed by instructions stored in a non-temporary computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., controller 110 in Figure 1A). Some operations of method 1200 are combined in an optional manner, and / or the order of some operations is changed in an optional manner.

[0246] In some embodiments, method 1200 is performed in a computer system (e.g., 101) that communicates with a display generation component and one or more input devices. In some embodiments, the computer system has one or more characteristics of the computer system of method 800 and / or 1000. In some embodiments, the display generation component has one or more characteristics of the display generation component of method 800 and / or 1000. In some embodiments, one or more input devices have one or more characteristics of the one or more input devices of method 800 and / or 1000.

[0247] In some embodiments, while the display generation component is operating in a first state (e.g., the display generation component is active and / or on), a three-dimensional environment (e.g., 1102) (e.g., the three-dimensional environment optionally has one or more characteristics of the three-dimensional environment of Method 800 and / or 1000, and optionally includes at least a portion of the physical environment of the user of the computer system. In some embodiments, the physical environment (or portion of it) is displayed in the three-dimensional environment via the display generation component (e.g., virtual passthrough or video passthrough). In some embodiments, the physical environment (or portion of it) is a view of the physical environment (or portion of it) of the computer system that is visible through the transparent portion of the display generation component (e.g., true or actual passthrough)) is visible from the user's first viewpoint (see, for example, Method 800 and / or 1000). As such, the user's first viewpoint is associated with the user's first individual spatial arrangement with respect to the three-dimensional environment, as shown in Figure 11A (for example, the viewpoint from which the three-dimensional environment is displayed and / or visible corresponds to the user's location and / or orientation in the user's three-dimensional environment and / or physical environment, and as the user rotates their head and / or torso and / or moves in the three-dimensional environment and / or physical environment, corresponding different parts of the three-dimensional environment become displayed and / or visible via the display generation component), and the computer system (e.g., 101) displays in the three-dimensional environment, via the display generation component, a first virtual object having a first spatial arrangement with respect to the user's first viewpoint and a second spatial arrangement with respect to the three-dimensional environment, such as objects 1106a, 1108a and / or 1110a in Figure 11A (1202a). The first virtual object optionally has one or more characteristics of the first virtual object in method 800 and / or 1000. The first spatial arrangement optionally corresponds to the relative location and / or relative orientation of the first virtual object with respect to the user's first viewpoint in a three-dimensional environment (optionally including the orientation of the first virtual object itself) (e.g., 10 feet from the first viewpoint, 30 degrees to the right of the centerline of the first viewpoint).The second spatial arrangement optionally corresponds to the relative location and / or relative orientation (optionally including the orientation of the first virtual object itself) of the first object with respect to a reference point in the user's three-dimensional environment and / or physical environment (e.g., the user's location in the physical environment, the orientation of the user's head and / or torso in the three-dimensional environment, the center of the room where the user is located, or the location of the user's viewpoint in the three-dimensional environment) (e.g., 10 feet from the center of the room, 30 degrees to the right of the line from the center of the room to the back wall of the room, and perpendicular to the back wall of the room). Thus, in some embodiments, the first virtual object having a relative location in the three-dimensional environment with respect to the user's viewpoint also optionally has a relative location with respect to the physical environment that is visible through the display generation components. In some embodiments, the first spatial arrangement satisfies one or more criteria of Method 800 and / or 1000 that specify a range of distances or orientations of the virtual object with respect to the user's viewpoint. In some embodiments, the first spatial arrangement does not satisfy one or more criteria of Method 800 and / or 1000.

[0248] In some embodiments, while displaying a first virtual object in a first spatial arrangement relative to the user's first viewpoint and a second spatial arrangement relative to the three-dimensional environment, a computer system (e.g., 101) detects a first event corresponding to a change in the state of the display generation component to a second state different from the first state (e.g., a state in which the display generation component is inactive or off), and while the display generation component is in the second state, the three-dimensional environment is invisible through the display generation component, such as turning off the computer system 101 as shown from Figure 11A to Figure 11B (1202b). For example, the second state is optionally activated in response to an input detected by the computer system (e.g., the first event) to stop and / or terminate the display of the three-dimensional environment (e.g., selection of a displayed selectable option or selection of a hardware button included in the computer system). In some embodiments, the display generation component is contained within a head-mounted device worn on the user's head, and when worn on the user's head, the display generation component is in the first state, and the user can see the three-dimensional environment which is visible through the display generation component. In some embodiments, for example, upon detecting a first event, such as the head-mounted device being removed from the user's head (e.g., no longer being worn by the user), the computer system transitions the display generation component to a second state.

[0249] In some embodiments, after a state change of the display generation component from a first state to a second state (e.g., after a state change of the display generation component from a first state to a second state, after the user changes orientation and / or moves to a different location in the physical environment) (1202c), the computer system (e.g., 101) detects a second event corresponding to the state change of the display generation component from the second state to the first state in which the three-dimensional environment is visible via the display generation component, and while the display generation component is in the first state after the detection of the second event, the three-dimensional environment is visible via the display generation component from a second viewpoint different from the user's first viewpoint (e.g., corresponding to the user's changed orientation and / or location in the physical environment), and the second viewpoint is associated with the user's second separate spatial positioning relative to the three-dimensional environment, such as viewpoint 1126 in Figure 11C (1202d). For example, the second event is optionally an input detected by the computer system to redisplay and / or input the three-dimensional environment (e.g., selection of a displayed selectable option, or selection of a hardware button included in the computer system). In some embodiments, a second event is the detection that the head-mounted device has been placed on the user's head (e.g., put on again by the user). For example, a computer system then displays the three-dimensional environment from the user's updated viewpoint (e.g., having an updated location and / or orientation in the three-dimensional environment that corresponds to the user's new location and / or orientation in the user's physical environment).

[0250] In some embodiments, after a change in the state of the display generation component from a first state to a second state (for example, after the user turns around and / or moves to a different location in the physical environment) (1202c), in response to detecting a second event, while the three-dimensional environment is visible from a second viewpoint (for example, the computer system transitions the display generation component back to the first state in response to detecting a second event), the computer system displays the first virtual object in the three-dimensional environment via the display generation component. (1202e), this includes (1202f) displaying the first virtual object in the three-dimensional environment in a first spatial arrangement relative to the user's second viewpoint and a third spatial arrangement different from the second spatial arrangement relative to the three-dimensional environment, such as objects 1106a, 1108a and / or 1110a in Figure 11E, according to the determination that one or more criteria are met (for example, one or more criteria for recentering the three-dimensional environment to the user's updated viewpoint, such as methods 800 and / or 1000, which include the first virtual object; the one or more criteria are described in more detail below). For example, because one or more criteria are met, the computer system displays the first virtual object in the same relative location and / or orientation relative to the second viewpoint as when the first virtual object was displayed relative to the first viewpoint from which the three-dimensional environment was last viewed (for example, in a different location and / or orientation in the three-dimensional environment than before). In some embodiments, the user is now in a different orientation and / or location in the three-dimensional environment and / or physical environment (for example, the second viewpoint corresponds to a different orientation and / or location), and the first virtual object is displayed in the same first spatial arrangement relative to the second viewpoint, so the first virtual object is now displayed in a different spatial arrangement relative to the previous three-dimensional environment and / or physical environment (for example, the first virtual object is no longer displayed on a physical table in the physical environment, but is now displayed on a physical sofa in the physical environment).

[0251] In some embodiments, after a change in the state of the display generation component from a first state to a second state (for example, after the user turns around and / or moves to a different location in the physical environment) (1202c), in response to detecting a second event, while the three-dimensional environment is visible from a second viewpoint (for example, the computer system transitions the display generation component to the first state in response to detecting a second event), the computer system displays a first virtual object in the three-dimensional environment via the display generation component (1202e), which includes displaying the first virtual object in the three-dimensional environment in a fourth spatial arrangement different from the first spatial arrangement to the user's second viewpoint and in a second spatial arrangement to the three-dimensional environment, such as object 1110a in Figure 11C, in accordance with the determination that one or more criteria are not met (1202g). For example, because one or more criteria are not met, the computer system displays the first virtual object in a location and / or orientation relative to a second viewpoint that is different from when the first virtual object was displayed relative to a first viewpoint in which the three-dimensional environment was last viewed (e.g., in the same location and / or orientation within the three-dimensional environment as before). Thus, the first virtual object is not repositioned within the three-dimensional environment, and the first virtual object is optionally displayed in the same second spatial arrangement relative to the three-dimensional environment and / or the physical environment (e.g., the first virtual object is still displayed on a physical table in the physical environment). In some embodiments, the input described with reference to Method 1200 is an air gesture input or includes an air gesture input. Selectively recentering objects based on the user's updated viewpoint reduces the number of inputs required for the user to make objects accessible when displaying the three-dimensional environment begins.

[0252] In some embodiments, one or more criteria are met when the duration between a first event and a second event exceeds a time threshold (e.g., 5 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, 12 hours, 24 hours, 48 ​​hours, 96 hours, or 192 hours), such as between Figures 11A and 11D / E, and are not met when the duration between a first event and a second event is less than the time threshold (1204). For example, the computer system may optionally not automatically recenter the 3D environment to the user's new viewpoint in response to the detection of a second event if the time since the detection of the first event is less than the time threshold, and optionally automatically recenter the 3D environment to the user's new viewpoint in response to the detection of a second event if the time since the detection of the first event is greater than the time threshold. Selectively recentering objects to the user's updated viewpoint based on time allows recentering to be performed at the appropriate time without displaying additional controls.

[0253] In some embodiments, one or more criteria are met when the user's second viewpoint is greater than a threshold distance from the user's first viewpoint in the three-dimensional environment (e.g., 0.1, 0.5, 1, 3, 5, 10, 20, 50, 100, or 300 meters), such as between Figure 11A and Figure 11D / E, and are not met when the user's second viewpoint is less than the threshold distance from the user's first viewpoint in the three-dimensional environment, such as between Figure 11A and Figure 11B / C (1206). For example, if a second event is detected when the user has moved more than a threshold distance away from the location in the user's physical environment where the first event was detected, the computer system optionally recenters the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. On the other hand, if a second event is detected when the user has not moved more than a threshold distance away from the location in the user's physical environment where the first event was detected, the computer system optionally does not recenter the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. Selectively recentering objects to the user's updated viewpoint based on distance allows recentering to be performed at the appropriate time without displaying additional controls.

[0254] In some embodiments, one or more criteria are met when the difference in orientation between the user's first and second viewpoints in the three-dimensional environment is greater than a threshold (e.g., the orientation of the second viewpoint is rotated by more than 5, 10, 20, 30, 45, 90, 120, or 150 degrees relative to the orientation of the first viewpoint), such as between Figure 11A and Figure 11D / E, and are not met when the difference in orientation between the user's first and second viewpoints in the three-dimensional environment is less than a threshold (1208). For example, if a second event is detected when the user moves and / or reorients their head, body, shoulders, and / or torso beyond a threshold orientation from the orientation of the user (e.g., the user's head, body, shoulders, and / or torso) in the user's physical environment where the first event was detected, the computer system optionally automatically recenters the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. On the other hand, if a second event is detected when the user has not moved and / or reoriented their head, body, shoulders, and / or torso beyond a threshold orientation from the orientation of the user (e.g., the user's head, body, shoulders, and / or torso) within the user's physical environment where the first event was detected, the computer system will optionally not automatically recenter the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. Selectively recentering objects to the user's updated viewpoint based on orientation allows recentering to be performed at the appropriate time without displaying additional controls.

[0255] In some embodiments, one or more criteria are met when the user's first viewpoint corresponds to a location in a first room in a three-dimensional environment, such as between Figure 11A and Figure 11D / E, and the user's second viewpoint corresponds to a location in a second room different from the first room in the three-dimensional environment (for example, if the user's viewpoint is the first viewpoint, the user is located in the first room of the user's physical environment, and if the user's viewpoint is the second viewpoint, the user is located in the second room of the user's physical environment), but are not met when the user's first viewpoint and the user's second viewpoint correspond to locations in the same room in a three-dimensional environment, such as between Figure 11A and Figure 11B / C (1210). In some embodiments, one or more criteria are met additionally or alternatively when the user's location corresponding to the first viewpoint is separated from the user's location corresponding to the second viewpoint by at least one wall in the user's physical environment. For example, if a second event is detected when a user moves to a different room than the one containing the location in the user's physical environment where the first event was detected, the computer system will optionally automatically recenter the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. On the other hand, if a second event is detected when the user has not moved to a different room than the one containing the location in the user's physical environment where the first event was detected, the computer system will optionally not automatically recenter the three-dimensional environment to the user's new viewpoint in response to the detection of the second event. Selectively recentering objects to the user's updated viewpoint based on the user's movement to a different room allows recentering to be performed at the appropriate time without displaying additional controls.

[0256] In some embodiments, while the three-dimensional environment is visible from the user's second viewpoint, as shown in Figure 11C, a computer system (e.g., 101) detects via one or more input devices an input corresponding to a request to update the spatial position of the first virtual object to the user's second viewpoint so as to satisfy a first set of one or more criteria specifying a range of distances or orientations of the virtual object to the user's second viewpoint, such as the input detected in Figure 11C (as described with reference to Method 800) (1012a). The input optionally has one or more characteristics of a first input (e.g., a recentering input) as described with reference to Methods 800, 1000, and / or 1400.

[0257] In some embodiments, upon detecting input, the computer system (e.g., 101) displays the first virtual objects in the three-dimensional environment in a first spatial arrangement relative to the user's second viewpoint and a third spatial arrangement relative to the three-dimensional environment, such as when objects 1106a, 1108a, and / or 1110a are displayed in Figure 11C in the spatial arrangement relative to viewpoint 1126 in Figure 11C that they had relative to viewpoint 1126 in Figure 11A (1012b). Thus, in some embodiments, even if the computer system does not automatically recenter the three-dimensional environment to the user's second viewpoint upon detecting a second event, the user can then manually recenter the three-dimensional environment by providing input to do so. In some embodiments, the results of recentering in response to a second event and recentering in response to user input are the same. Providing manual recentering provides an efficient method for positioning virtual objects in the appropriate locations within the three-dimensional environment.

[0258] In some embodiments, the input corresponding to a request to update the spatial positioning of a first virtual object relative to the user's second viewpoint to satisfy a first set of one or more criteria includes the selection of a physical button on the computer system, such as the input described with reference to Figure 7B (1014). In some embodiments, the display generating component is included in a device (e.g., a physical device) that includes a physically pressable button. In some embodiments, the button is also rotatable (for example, to increase or decrease the level of immersion in which the computer system is displaying a three-dimensional environment, as described with reference to Method 800). In some embodiments, the device is a head-mounted device, such as a virtual or augmented reality headset. In some embodiments, the input is a button press or includes a button press (but does not include button rotation). By providing manual recentering via the activation of a physical button, an efficient method is provided for positioning virtual objects in the correct position within a three-dimensional environment.

[0259] In some embodiments, the display generating component is contained within a wearable device that can be worn by a user (e.g., a head-mounted device such as a virtual or augmented reality headset or glasses), and detecting a first event includes detecting that the user is no longer wearing the wearable device (e.g., detecting that the user has removed the head-mounted device from their head, and / or detecting that the head-mounted device is no longer on the user's head) (1016). Other wearable devices such as smartwatches are also envisioned. In some embodiments, detecting a second event includes detecting that the user has placed the head-mounted device on their head, and / or detecting that the head-mounted device is being worn again by the user. Transitioning the display generating component to a second state based on whether the user is wearing the device reduces the number of inputs required to transition to the second state.

[0260] In some embodiments, detecting a first event includes detecting an input corresponding to a request to discontinue visibility of a three-dimensional environment via a display generation component (1018). For example, the input is for closing a virtual or augmented reality experience presented by a computer system. In some embodiments, the virtual or augmented reality experience is provided by an application running on the computer system, and the input is for closing that application. In some embodiments, the input is for exiting full-screen mode of the virtual or augmented reality experience. In some embodiments, the input is for reducing the level of immersion in which the computer system is displaying the three-dimensional environment (for example, by rotating the aforementioned physical button in a first direction), as described with reference to Method 800. In some embodiments, a second event is an input for opening or starting a virtual or augmented reality experience. In some embodiments, a second event is an input for opening or launching an application that provides the virtual or augmented reality experience. In some embodiments, the second event is an input that increases the level of immersion (e.g., above a threshold immersion level) in which the computer system is displaying the three-dimensional environment (e.g., by rotating the aforementioned physical button in a second direction different from the first direction), as described with reference to Method 800. Transitioning the display generation component to a second state based on user input provides an efficient method for transitioning to the second state.

[0261] In some embodiments, detecting a first event includes detecting an input corresponding to a request to put the display generating component into a low-power state (1020). For example, in some embodiments, the display generating component is contained within a device (e.g., a head-mounted device), and the input is an input to turn off power to the device or put the device into sleep or low-power mode. In some embodiments, a second event is an input to turn on power to the device or put the device into normal power mode (e.g., to exit sleep or low-power mode). Transitioning the display generating component to a second state based on whether a user is wearing the device reduces the number of inputs required to transition to the second state.

[0262] It should be understood that the specific order in which the operations in Method 1200 are described is illustrative only and does not indicate that the described order is the only order in which the operations can be performed. Those skilled in the art will recognize various methods for rearranging the operations described herein.

[0263] Figures 13A to 13C illustrate examples of computer systems that selectively recenter content associated with communication sessions between multiple users in response to input detected in the computer system, according to several embodiments.

[0264] Figure 13A shows two three-dimensional environments 1302a and 1302b that are visible through the respective display generation components 120a and 120b of computer systems 101a and 101b (for example, the display generation component 120 in Figure 1). Computer system 101a is optionally placed in a first physical environment, and the three-dimensional environment 1302a is optionally visible through its display generation component 120a. Computer system 101b is optionally placed in a second physical environment, and the three-dimensional environment 1302b is optionally visible through its display generation component 120b. The three-dimensional environment 1302a is visible from the user's viewpoint 1328c shown in the overhead view (for example, facing the wall of the room where computer system 101a is located). The three-dimensional environment 1302b is visible from the user's viewpoint 1330c shown in the overhead view (for example, facing the wall of the room where the computer syst...

Claims

1. It is a method, In a computer system that communicates with a display generation component and one or more input devices, The first virtual object is displayed in the three-dimensional environment at a first visual splendor level to the three-dimensional environment while the first virtual object is displayed in the three-dimensional environment, at least partially overlapping individual parts of the second virtual object in the three-dimensional environment, and while the attention of the user of the computer system is directed towards the first virtual object. The computer system detects when the user's attention moves away from the first virtual object while the first virtual object having the first visual splendor level is being displayed via the display generation component, In response to the detection that the user's attention to the computer system has moved away from the first virtual object, While the second virtual object is currently displayed at a second visual saturation level relative to the three-dimensional environment, which is lower than the first visual saturation level, according to the determination that the user's attention is directed towards the second virtual object, The method involves displaying the second virtual object at a third visual splendor level higher than the second visual splendor level via the display generation component while the first virtual object is displayed at a fourth visual splendor level lower than the first visual splendor level, wherein displaying the second virtual object at the third visual splendor level and the first virtual object at the fourth visual splendor level increases the degree of visibility of the individual parts of the second virtual object that overlap with the first virtual object. Methods that include...

2. The method according to claim 1, further comprising displaying the first virtual object at the second visual splendor level via the display generation component, in accordance with the determination that the user's attention is directed to the second virtual object while the second virtual object is currently displayed to the three-dimensional environment at the second visual splendor level.

3. The method according to claim 1, wherein the first visual splendor level corresponds to a first level of translucency, and the second visual splendor level corresponds to a second level of translucency that is higher than the first level of translucency.

4. Displaying the first virtual object at the first visual saturation level includes, via the display generation component, displaying a first portion of the first virtual object having a third level of translucency, and via the display generation component, displaying a second portion of the first virtual object having a fourth level of translucency different from the third level of translucency. The method according to claim 3, wherein, in response to the detection that the attention of the user of the computer system has moved away from the first virtual object, displaying the second virtual object having a third visual splendor level includes changing the translucency of a first portion of the second virtual object by a first amount and changing the translucency of a second portion of the second virtual object by a second amount different from the first amount.

5. The method according to claim 1, wherein the detection of the user's attention moving away from the first virtual object includes detecting the user's line of sight directed to a specific position in the three-dimensional environment for a threshold amount of time.

6. The detection that the user's attention moves away from the first virtual object means that The method according to claim 1, comprising detecting the user's gaze directed to a specific position in the three-dimensional environment, and detecting a gesture performed by a specific part of the user in the computer system while the user's gaze is directed to the specific position in the three-dimensional environment.

7. The detection that the user's attention moves away from the first virtual object means that Detecting the user's gaze directed towards individual positions within the three-dimensional environment, While the user's gaze is directed towards the individual position in the three-dimensional environment, and while the second virtual object is displayed to the three-dimensional environment at the second visual prominence level, Without detecting selection input from individual parts of the user, the second virtual object is displayed via the display generation component at a third visual splendor level higher than the second visual splendor level, in accordance with the determination that one or more first criteria are met, including criteria that are met when the user's gaze is directed towards the second virtual object for a threshold amount of time, The method according to claim 1, comprising displaying the second virtual object via the display generating component at a third visual splendor level higher than the second visual splendor level, in accordance with a determination that one or more second criteria are met, including criteria that are met when the selection input from the individual part of the user is detected before the user's gaze is directed towards the second virtual object for a threshold time amount.

8. The method according to claim 7, wherein, in accordance with the determination that one or more of the second criteria are met, individual content contained in the second virtual object is not selected in response to the detection of selection input from the individual portion of the user.

9. While the first virtual object having the first visual splendor level is displayed via the display generation component, and while the user's attention of the computer system is directed towards the first virtual object, the user's attention moving within the first virtual object is detected. The method according to claim 1, further comprising: detecting the attention of the user moving within the first virtual object; and maintaining the display of the first virtual object at a first visual prominence level.

10. The method according to claim 1, further comprising, in response to the detection that the user's attention of the computer system moves away from the first virtual object, maintaining the display of the first virtual object with respect to the three-dimensional environment at a first visual splendor level via the display generation component, according to the determination that the user's attention is directed to a position in the three-dimensional environment that does not correspond to an individual virtual object.

11. The method according to claim 1, further comprising, in response to the detection that the attention of the user of the computer system has moved away from the first virtual object, maintaining the display of the first virtual object to the three-dimensional environment at a first level of visual prominence via the display generation component, in accordance with the determination that the user's attention is directed towards a non-interactive virtual object.

12. The first virtual object and the third virtual object are associated with a group of virtual objects, and the method is as follows: The method according to claim 1, further comprising maintaining the display of the first virtual object at a first visual splendor level with respect to the three-dimensional environment via the display generation component, in accordance with the determination that the user's attention is directed to the third virtual object, in response to the detection that the user's attention of the computer system has moved away from the first virtual object.

13. The method according to claim 1, further comprising, in response to the detection that the user's attention of the computer system has moved away from the first virtual object, maintaining the display of the first virtual object to the three-dimensional environment at a first visual splendor level via the display generation component, in accordance with the determination that the user is currently interacting with the first virtual object.

14. The current interaction with the first virtual object includes moving the first virtual object, and the method is The method according to claim 13, further comprising displaying a visual indication associated with the movement of the first virtual object via the display generation component while the first virtual object is being moved as part of the current interaction with the first virtual object.

15. The method according to claim 13, wherein the current interaction with the first virtual object includes selecting first content and moving it from the first virtual object to a separate virtual object other than the first virtual object.

16. The method according to claim 13, wherein the current interaction with the first virtual object includes moving the first content within the first virtual object.

17. The second visual splendor level corresponds to a second level of translucency that is higher than the first level of translucency corresponding to the first visual splendor level. The method according to claim 1, wherein the third level of visual splendor corresponds to a third level of translucency that is lower than the second level of translucency.

18. The second visual splendor level corresponds to a second blur level that is greater than the first blur level corresponding to the first visual splendor level. The method according to claim 1, wherein the third visual saturation level corresponds to a third blur level lower than the second blur level.

19. The method according to claim 1, wherein the first virtual object is displayed in front of the user's view of the user's physical environment.

20. The method according to claim 1, wherein the first virtual object is displayed in front of the virtual environment from the user's viewpoint.

21. The method according to claim 1, wherein the first virtual object is associated with a first application, and the second virtual object is associated with a second application different from the first application.

22. The method according to claim 1, wherein the second virtual object is a control user interface associated with the operating system of the computer system.

23. While the second virtual object is displayed to the three-dimensional environment at the second visual splendor level via the display generation component, individual selectable elements associated with the movement of the second virtual object are displayed at a fifth visual splendor level via the display generation component. While displaying the second virtual object at the second visual saturation level and the individual selectable elements at the fifth visual saturation level, the system receives a first input directed to the individual selectable elements via one or more input devices. The method according to claim 1, further comprising detecting the first input and moving the second virtual object in the three-dimensional environment according to the first input.

24. While the first virtual object is displayed at the fourth visual splendor level via the display generation component, individual selectable elements associated with the movement of the first virtual object are displayed via the display generation component. While displaying the first virtual object and the individual selectable elements associated with moving the first virtual object at the fourth visual splendor level via the display generation component, input directed to the individual elements associated with the first virtual object is detected. In response to detecting the input directed to the individual elements associated with the first virtual object, In accordance with the determination that the individual element is the individual selectable element, The process of moving the first virtual object in response to the input is initiated, The method according to claim 1, further comprising: displaying the first virtual object at a fifth visual splendor level higher than the fourth visual splendor level via the display generation component, without performing any actions associated with the individual content in accordance with the input, based on the determination that the individual elements correspond to individual content contained within the first virtual object.

25. The method includes the first virtual object currently playing media content, The method according to claim 1, further comprising maintaining playback of the media content contained within the first virtual object, which is displayed at a fourth visual splendor level, in response to the display generation component displaying the second virtual object at a third visual splendor level to the three-dimensional environment.

26. The first virtual object is an object of a first type, and the method is The computer system detects when the user's attention moves away from the third virtual object while displaying the third virtual object, with the third virtual object being a second type of object distinct from the first type of object, and the user's attention is directed towards the third virtual object, via the display generation component. The method according to claim 1, further comprising: detecting that the user's attention in the computer system has moved away from the third virtual object, and determining that the user's attention is now directed towards a fourth virtual object in the three-dimensional environment, maintaining the display of the third virtual object to the three-dimensional environment at a fifth level of visual prominence.

27. The method according to claim 26, wherein the second type of object is a representation of an individual user associated with the computer system.

28. The method according to claim 26, wherein the second type of object is a user interface for a media playback application.

29. The method according to claim 26, wherein the second type of the object is the status user interface of the computer system.

30. The method according to claim 27, wherein the second type of object is a user interface for a communication application.

31. A non-temporary computer-readable storage medium for storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component and one or more input devices, wherein the one or more programs include instructions for executing any one of the methods according to claims 1 to 30.

32. A computer system that communicates with a display generation component and one or more input devices, wherein the computer system is One or more processors, A computer system comprising: a memory for storing one or more programs configured to be executed by one or more processors, wherein the one or more programs include instructions for executing any one of the methods according to claims 1 to 30.

Images