Method for displaying and relocating objects in environment

Abstract

Methods for displaying and relocating objects in an environment are disclosed. In some embodiments, the computer system selectively recenters virtual content to a user's viewpoint in the presence of a physical or virtual obstacle, and / or automatically recenters one or more virtual objects in response to a display generating component changing state; selectively re-centering content associated with the communication session between the plurality of users in response to the detected user input; changing the visual saliency of the content included in the virtual object based on the viewpoint and / or based on the detected user attention of the user; modifying the visual saliency of the one or more virtual objects to address significant occlusion of the one or more virtual objects; modifying the visual saliency relative to the virtual object based on the user viewpoint; concurrently modifying visual saliency based on various types of user interactions; and / or changing a visual impact amount of the environmental effect in response to the detected user input.

Description

[0001] This application is a divisional application based on the Chinese invention patent application with application date of January 19, 2023, application number 2023800287723, and invention name “Method for displaying and repositioning objects in an environment”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 63 / 301,020, filed on January 19, 2022, U.S. Provisional Application No. 63 / 377,002, filed on September 23, 2022, and U.S. Provisional Application No. 63 / 480,494, filed on January 18, 2023, the contents of which are incorporated herein by reference in their entirety for all purposes. Technical Field

[0004] The present invention generally relates to computer systems that provide computer-generated experiences, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via displays. Background Art

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices for computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with virtual / augmented reality environments. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the Invention

[0006] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve desired results in an augmented reality environment, and systems where virtual object manipulation is complex, cumbersome, and error-prone can place a significant cognitive burden on users and detract from the experience of the virtual / augmented reality environment. Furthermore, these methods take longer than necessary, wasting the computer system's energy. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, there is a need for computer systems with improved methods and interfaces for providing computer-generated experiences to users, thereby making user interactions with computer systems more efficient and intuitive for the users. Such methods and interfaces optionally supplement or replace conventional methods for providing extended reality experiences to users. Such methods and interfaces reduce the amount, extent, and / or nature of inputs from users by helping users understand the connection between the inputs provided and the device's responses to those inputs, thereby forming a more effective human-computer interface.

[0008] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to the display generation component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contacts and gestures of a stylus and / or fingers on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, test support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with content in a three-dimensional environment. Such methods and interfaces can supplement or replace conventional methods for interacting with content in a three-dimensional environment. Such methods and interfaces reduce the amount, extent, and / or nature of input from a user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces conserve power and increase the time between battery charges.

[0010] In some embodiments, the computer system selectively recenters virtual content to the user's viewpoint. In some embodiments, the computer system recenters one or more virtual objects in the presence of physical or virtual obstructions. In some embodiments, the computer system selectively and automatically recenters one or more virtual objects in response to a display generation component changing state. In some embodiments, the computer system selectively recenters content associated with a communication session between multiple users in response to input detected at the computer system. In some embodiments, the computer system changes the visual salience of content included in a virtual object based on the viewpoint. In some embodiments, the computer system modifies the visual salience of one or more virtual objects based on detected user attention. In some embodiments, the computer system modifies the visual salience of one or more virtual objects to address apparent occlusion of the one or more virtual objects. In some embodiments, the computer system gradually modifies the visual salience of one or more virtual objects based on determining that the user's viewpoint corresponds to different areas of the three-dimensional environment. In some embodiments, the computer system modifies the visual salience of one or more portions of a virtual object as the user's viewpoint approaches the virtual object. In some embodiments, the computer system modifies the visual salience of a virtual object upon detecting one or more concurrent types of user interaction. In some embodiments, the computer system changes the amount of visual impact of environmental effects on the three-dimensional environment displaying virtual content in response to detecting a shift in input (e.g., user attention) to a different element in the three-dimensional environment.

[0011] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described in this specification are not comprehensive. In particular, many additional features and advantages will be apparent to those skilled in the art from the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification has been selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0013] Figure 1 is a block diagram illustrating an operating environment for a computer system for providing an XR experience according to some embodiments.

[0014] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0015] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user, according to some embodiments.

[0016] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0017] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input from a user according to some embodiments.

[0018] Figure 6 is a flowchart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0019] 7A to 7F An example of a computer system selectively re-centering virtual content to a user's viewpoint is illustrated according to some embodiments.

[0020] Figures 8A to 8I is a flowchart illustrating an exemplary method of selectively re-centering virtual content to a user's viewpoint according to some embodiments.

[0021] Figures 9A to 9C An example is illustrated of a computer system according to some embodiments re-centering one or more virtual objects in the presence of physical or virtual obstacles.

[0022] Figures 10A to 10G is a flowchart illustrating a method for re-centering one or more virtual objects in the presence of physical or virtual obstacles according to some embodiments.

[0023] Figures 11A to 11E An example is illustrated of a computer system selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments.

[0024] 12A to 12E is a flow chart illustrating a method for selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments.

[0025] 13A to 13C Illustrated is an example of a computer system according to some embodiments selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system.

[0026] 14A to 14E is a flowchart illustrating a method of selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system according to some embodiments.

[0027] Figures 15A to 15J An example is illustrated of a computer system according to some embodiments changing the visual saliency of content included in a virtual object based on viewpoint.

[0028] Figures 16A to 16P is a flowchart illustrating a method for changing the visual saliency of content included in a virtual object based on viewpoint according to some embodiments.

[0029] 17A to 17E An example is illustrated in which a computer system according to some embodiments changes the visual salience of content included in a virtual object based on the attention of a user of the computer system.

[0030] Figures 18A to 18K is a flowchart illustrating a method for modifying the visual saliency of a virtual object based on a user's attention according to some embodiments.

[0031] Figures 19A to 19E An example is illustrated in which a computer system according to some embodiments modifies the visual saliency of a corresponding virtual object to modify apparent occlusion of the corresponding virtual object by virtual content.

[0032] 20A to 20F is a flowchart illustrating a method of modifying the visual saliency of a corresponding virtual object to modify apparent occlusion of the corresponding virtual object by virtual content according to some embodiments.

[0033] Figures 21A to 21L An example is illustrated in which a computer system according to some embodiments gradually modifies the visual saliency of a corresponding virtual object based on changes in a user's viewpoint.

[0034] Figures 22A to 22J is a flowchart illustrating a method of gradually modifying the visual saliency of a corresponding virtual object according to a change in a user's viewpoint according to some embodiments.

[0035] Figures 23A to 23E An example is illustrated in which a computer system according to some embodiments modifies the visual salience of a corresponding virtual object based on a user's proximity to the corresponding virtual object.

[0036] 24A to 24F is a flowchart illustrating a method for modifying the visual saliency of a corresponding virtual object based on a user's proximity to the corresponding virtual object according to some embodiments.

[0037] Figures 25A to 25C An example is illustrated in which a computer system according to some embodiments modifies the visual saliency of a corresponding virtual object based on one or more concurrent types of user interactions.

[0038] 26A to 26D is a flowchart illustrating a method for modifying the visual saliency of corresponding virtual objects based on one or more concurrent types of user interactions according to some embodiments.

[0039] Figures 27A to 27J Illustrated are examples of a computer system according to some embodiments concurrently displaying virtual content and environmental effects having different amounts of visual impact on a three-dimensional environment in response to the computer system detecting input (e.g., user attention) shifting to different elements in the three-dimensional environment.

[0040] 28A to 28I is a flowchart illustrating a method for dynamically displaying environmental effects having varying amounts of visual impact on the appearance of a three-dimensional environment displaying virtual content in response to detecting input (e.g., user attention) shifting to different elements in the three-dimensional environment according to some embodiments. DETAILED DESCRIPTION

[0041] According to some embodiments, the present disclosure relates to a user interface for providing an extended reality (XR) experience to a user.

[0042] The systems, methods, and GUIs described herein provide improved ways for electronic devices to facilitate interaction with and manipulation of objects in a three-dimensional environment.

[0043] In some embodiments, a computer system displays virtual objects in an environment. In some embodiments, in response to input to re-center virtual objects to a user's viewpoint, the computer system re-centers those virtual objects that meet certain criteria and does not re-center those virtual objects that do not meet such criteria. In some embodiments, virtual objects that align with portions of the physical environment are not re-centered. In some embodiments, virtual objects that were last placed or moved into the environment from the user's current viewpoint are not re-centered.

[0044] In some embodiments, a computer system displays virtual objects in an environment. In some embodiments, in response to input to re-center virtual objects to a user's viewpoint, the computer system avoids physical objects when re-centering those virtual objects. In some embodiments, the computer system avoids other virtual objects when re-centering virtual objects.

[0045] In some embodiments, the computer system displays the virtual object in the environment from a first viewpoint. In some embodiments, when the state of the computer system changes (e.g., from on to off and then back on), the computer system automatically re-centers the virtual object to the new viewpoint based on one or more characteristics of the new viewpoint. In some embodiments, the computer system does not automatically re-center the virtual object to the new viewpoint.

[0046] In some embodiments, a computer system displays virtual objects in an environment accessible to multiple computer systems. In some embodiments, in response to input to re-center a virtual object to a user's viewpoint, the computer system does not change the spatial arrangement of virtual objects accessible to multiple computer systems relative to the viewpoints associated with those multiple computer systems. In some embodiments, the computer system changes the spatial arrangement of virtual objects not accessible to other computer systems relative to the viewpoint associated with the computer system.

[0047] In some embodiments, a computer system displays virtual objects including content in an environment. In some embodiments, the computer system displays content with different visual saliences depending on the angle from which the content is visible from the user's current viewpoint. In some embodiments, the closer the angle is to the front, the greater the visual salience, and the further the angle is from the front, the less visual salience. In some embodiments, the computer system modifies the visual salience of one or more virtual objects based on detected user attention. In some embodiments, the computer system modifies the visual salience of one or more virtual objects to address apparent occlusion of the one or more virtual objects.

[0048] Figures 1 to 6 Descriptions of exemplary computer systems for providing XR experiences to users are provided (such as described below with respect to methods 800 , 1000 , 1200 , 1400 , 1600 , 1800 , and / or 2000 ). 7A to 7F An example of a computer system selectively re-centering virtual content to a user's viewpoint is illustrated according to some embodiments. Figures 8A to 8I is a flowchart illustrating an exemplary method of selectively re-centering virtual content to a user's viewpoint according to some embodiments. 7A to 7F The user interface in Figures 8A to 8I in the process. 9A to 9C An example is illustrated of a computer system according to some embodiments re-centering one or more virtual objects in the presence of physical or virtual obstacles. Figures 10A to 10G is a flowchart illustrating a method for re-centering one or more virtual objects in the presence of physical or virtual obstacles according to some embodiments. 9A to 9C The user interface in Figures 10A to 10G in the process. Figures 11A to 11E An example is illustrated of a computer system selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments. 12A to 12E is a flow chart illustrating a method for selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments. Figures 11A to 11E The user interface in 12A to 12E in the process. 13A to 13C Illustrated is an example of a computer system according to some embodiments selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system. 14A to 14E is a flowchart illustrating a method of selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system according to some embodiments. 13A to 13C The user interface in 14A to 14E in the process. Figures 15A to 15J An example is illustrated of a computer system according to some embodiments changing the visual saliency of content included in a virtual object based on viewpoint. Figures 16A to 16P is a flowchart illustrating a method for changing the visual saliency of content included in a virtual object based on viewpoint according to some embodiments. Figures 15A to 15J The user interface in Figures 16A to 16P in the process. 17A to 17E An example is illustrated in which a computer system according to some embodiments changes the visual salience of content included in a virtual object based on the attention of a user of the computer system. Figures 18A to 18K is a flowchart illustrating a method for modifying the visual saliency of a virtual object based on a user's attention according to some embodiments. Figures 17A to 17E The user interface in Figures 18A to 18K in the process. Figures 19A to 19E An example is illustrated in which a computer system according to some embodiments modifies the visual saliency of a corresponding virtual object to modify apparent occlusion of the corresponding virtual object by virtual content. 20A to 20F is a flowchart illustrating a method of modifying the visual saliency of a corresponding virtual object to modify apparent occlusion of the corresponding virtual object by virtual content according to some embodiments. Figures 19A to 19E The user interface in 20A to 20F in the process. Figures 21A to 21L An example is illustrated in which a computer system according to some embodiments gradually modifies the visual saliency of a corresponding virtual object based on changes in a user's viewpoint. Figures 22A to 22J is a flowchart illustrating a method of gradually modifying the visual saliency of a corresponding virtual object according to a change in a user's viewpoint according to some embodiments. Figures 21A to 21L The user interface in Figures 22A to 22J in the process. Figures 23A to 23E An example is illustrated in which a computer system according to some embodiments modifies the visual salience of a corresponding virtual object based on a user's proximity to the corresponding virtual object. 24A to 24F is a flowchart illustrating a method for modifying the visual saliency of a corresponding virtual object based on a user's proximity to the corresponding virtual object according to some embodiments. Figures 23A to 23E The user interface in 24A to 24F in the process. Figures 25A to 25C An example is illustrated in which a computer system according to some embodiments modifies the visual saliency of a corresponding virtual object based on one or more concurrent types of user interactions. 26A to 26D is a flowchart illustrating a method for modifying the visual saliency of corresponding virtual objects based on one or more concurrent types of user interactions according to some embodiments. Figures 25A to 25C The user interface in 26A to 26D in the process. Figures 27A to 27J Illustrated are examples of a computer system according to some embodiments changing the amount of visual impact of an environmental effect on the appearance of a three-dimensional environment displaying first virtual content in response to detecting that input (such as user attention) has moved away from first virtual content and / or other input other than user attention directed to an element different from the first virtual content. 28A to 28I is a flowchart illustrating a method for dynamically displaying environmental effects having varying amounts of visual impact on the appearance of a three-dimensional environment displaying virtual content in response to detecting input (e.g., user attention) shifting to different elements in the three-dimensional environment according to some embodiments. Figures 27A to 27J The user interface in 28A to 28I in the process.

[0049] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0050] In addition, in the method described herein where one or more steps depend on having met one or more conditions, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions of the steps in the method of determining the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then those of ordinary skill will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on having met one or more conditions can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require a system or computer-readable medium to declare that the system or computer-readable medium includes instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether a possible situation has been met without explicitly repeating the steps of the method until all conditions of the steps in the method of determining the method have been met. Those of ordinary skill in the art will also understand that, similar to the method with a contingent step, a system or computer-readable storage medium can repeat the steps of the method as needed multiple times to ensure that all contingent steps have been performed.

[0051] In some embodiments, as Figure 1As shown in , an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, or a touch screen), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, or a speed sensor), and optionally one or more peripheral devices 195 (e.g., a household appliance or a wearable device). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted device or a handheld device).

[0052] When describing an XR experience, various terms are used to distinctly refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or haptic feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0053] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

[0054] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a subset of a person's physical movements, or representations thereof, is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that conforms to at least one law of physics. For example, an XR system can detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to features of virtual objects in an XR environment can be made in response to representations of physical movement (e.g., voice commands). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, people can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, audio objects can enable audio transparency, which selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, people can sense and / or interact only with audio objects.

[0055] Examples of XR include virtual reality and mixed reality.

[0056] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing human avatars are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through the simulation of the person's presence within the computer-generated environment and / or through the simulation of a subset of the person's physical movement within the computer-generated environment.

[0057] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0058] Examples of mixed reality include augmented reality and augmented virtuality.

[0059] Augmented Reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system can be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with the virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects superimposed on the physical environment. As used herein, a video of the physical environment displayed on an opaque display is referred to as "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on the opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as holograms or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0060] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input can be a representation of one or more characteristics of the physical environment. For example, an AV park can have virtual trees and virtual buildings, but people's faces are realistically reproduced from images taken of physical people. In another example, a virtual object can adopt the shape or color of a physical object imaged by one or more imaging sensors. In another example, a virtual object can adopt a shadow that conforms to the position of the sun in the physical environment.

[0061] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, even if the user's viewpoint shifts (e.g., changes), the virtual object is viewpoint-locked. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0062] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays it at a location and / or position in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or position in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation at which an environment-locked virtual object is displayed in the user's viewpoint depends on the position and / or orientation of the object in the environment to which the virtual object is locked. In some embodiments, the computer system uses a stationary reference frame (e.g., a coordinate system anchored to fixed locations and / or objects in the physical environment) to determine the positioning at which an environment-locked virtual object is displayed in the user's viewpoint. An environment-locked virtual object can be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object), or can be locked to a movable portion of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot of the user) so that the virtual object moves as the viewpoint or that portion of the environment moves to maintain a fixed relationship between the virtual object and that portion of the environment.

[0063] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of a reference point that the virtual object follows. In some embodiments, when exhibiting inertial following behavior, the computer system intentionally delays the movement of the virtual object when movement of a reference point that the virtual object is following (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) is detected. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as movement of 0 to 5 degrees or movement of 0 to 50 cm). For example, when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0064] Hardware: There are many different types of electronic systems that enable people to sense various XR environments and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display can be configured to become selectively opaque. The projection-based system can employ retinal projection technology that projects graphic images onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, for example as holograms or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. Reference is made below to Figure 2Controller 110 is described in more detail. In some embodiments, controller 110 is a computing device that is located locally or remotely relative to scene 105 (e.g., a physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server or a central server) located outside of scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, display, projector, or touchscreen) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, or IEEE 802.3x). For example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more input devices among the input devices 125, one or more output devices among the output devices 155, one or more sensors among the sensors 190, and / or one or more peripheral devices among the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0065] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0066] According to some embodiments, display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within scene 105.

[0067] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head or on his / her hand). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, housing, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) can be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface that shows interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device can similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and the responses to the XR content are displayed via the HMD. Similarly, a user interface that shows interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) can similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0068] Despite Figure 1 Relevant features of the operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

[0069] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), code division multiple access (CDMA), time division multiple access (TDMA), global positioning system (GPS), infrared (IR), Bluetooth, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0070] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0071] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located away from one or more processing units 202. Memory 220 includes non-transitory computer-readable storage media. In some embodiments, memory 220 or a non-transitory computer-readable storage medium of memory 220 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and an XR experience module 240.

[0072] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences for one or more users (e.g., a single XR experience for one or more users, or multiple XR experiences for corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0073] In some embodiments, the data acquisition unit 241 is configured to obtain data from at least Figure 1 1 and / or display generation component 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, or position data) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data acquisition unit 241 includes instructions and / or logic for instructions, as well as heuristics and metadata for the heuristics.

[0074] In some embodiments, the tracking unit 242 is configured to map the scene 105 and track at least the display generation component 120 relative to the scene 105. Figure 1 105, and optionally relative to the position / location of one or more of the tracked input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand and / or the position / location of one or more parts of the user's hand relative to the user's Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to the coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more broadly, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0075] In some embodiments, the coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by the display generation component 120, and optionally by one or more of the output device 155 and / or peripheral devices 195. To this end, in various embodiments, the coordination unit 246 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0076] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data or position data) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transmission unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0077] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are shown as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0078] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, rather than as a structural diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how features are distributed among them will vary depending on the specific implementation and, in some embodiments, will depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0079] Figure 3is a block diagram of an example of a display generation component 120 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0080] In some embodiments, the one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, the one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, or a blood glucose sensor), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), and the like.

[0081] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field effect transistor (OLET), organic light-emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction waveguide displays, reflective waveguide displays, polarization waveguide displays and / or holographic waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. For another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 are capable of presenting MR and VR content. In some embodiments, one or more XR displays 312 are capable of presenting MR or VR content.

[0082] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand-tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see in the absence of the display generation component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, among others.

[0083] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located away from one or more processing units 302. Memory 320 includes non-transitory computer-readable storage media. In some embodiments, memory 320 or a non-transitory computer-readable storage medium of memory 320 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and an XR rendering module 340.

[0084] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to the user via one or more XR displays 312. To this end, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0085] In some embodiments, the data acquisition unit 342 is configured to obtain data from at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, or location data). For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0086] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. For such purposes, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0087] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate an extended reality) based on the media content data. For this purpose, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0088] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data or position data) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0089] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0090] also, Figure 3 It serves more as a functional description of various features that may be present in a particular embodiment, rather than as a structural schematic diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how features are distributed among them will vary depending on the specific implementation and, in some embodiments, will depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0091] Figure 4 is a schematic illustration of an exemplary embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 244 ( Figure 2 ) to track the position / location of one or more parts of the user's hand and / or one or more parts of the user's hand relative to Figure 1The hand tracking device 140 is configured to monitor movement of the scene 105 relative to the user's surroundings (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hands). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0092] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information, including at least a human user's hand 406. The image sensor 404 captures images of the hand at a sufficient resolution to enable the fingers and their corresponding positions to be distinguished. The image sensor 404 typically captures images of other parts of the user's body, or may also capture images of all parts of the body, and may have zoom capabilities or specialized sensors with increased magnification to capture images of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or serves as an image sensor for capturing the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in such a way that the field of view of the image sensor 404, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are treated as input to the controller 110.

[0093] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and possibly color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application program interface (API) to an application running on the controller, which in turn drives the display generation component 120. For example, a user can interact with the software running on the controller 110 by moving his hand 406 and changing his hand posture.

[0094] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offsets of the spots in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of a point in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of a point in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0095] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand from these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's hand joints and fingertips.

[0096] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0097] In some embodiments, gestures include air gestures. An air gesture is a gesture that is detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of the device) and is based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture in which a part of the user's body is rotated at a predetermined speed or amount)).

[0098] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and is based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture in which a part of the user's body is rotated at a predetermined speed or amount)).

[0099] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving in-air gestures, for example, the input gesture is combined with (e.g., concurrently with) movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described below.

[0100] In some embodiments, an input gesture directed to a user interface object is performed directly or indirectly with reference to the user interface object. For example, user input is performed directly on the user interface object based on performing input with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, upon detecting the user's attention (e.g., gaze) to the user interface object, an input gesture is performed indirectly on the user interface object based on the user's hand being located not at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can direct the user's input to the user interface object by initiating a gesture at or near a location corresponding to the displayed location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can direct the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the displayed location of the user interface object).

[0101] In some embodiments, according to some embodiments, input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0102] In some embodiments, a pinch input is part of an air gesture that includes one or more of: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to make contact with each other, i.e., optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which the two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., more) pinch inputs (e.g., performed by the same hand) that are detected consecutively with each other immediately (e.g., within a predefined time period). For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.

[0103] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., spreads their two or more fingers apart) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to make contact with each other and moves the same hand to the second position in the air using the drag gesture). In some embodiments, a pinch input is performed by a first hand of a user, and a drag input is performed by a second hand of the user (e.g., the user's second hand moves in the air from a first position to a second position while the user continues the pinch input with the user's first hand. In some embodiments, the input gesture as an air gesture includes an input performed using both hands of the user (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., more) pinch inputs performed in conjunction with each other (e.g., concurrently or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using the first hand of the user, and in conjunction with the pinch input performed using the first hand, a second pinch input is performed using another hand (e.g., the second of the user's two hands). In some embodiments, movement between the two hands of the user (e.g., increasing and / or decreasing the distance or relative orientation between the two hands of the user)

[0104] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of the user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), a downward motion of the user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in movement characteristics of the finger or hand performing the tap gesture (e.g., an end of movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0105] In some embodiments, the user's attention is determined to be directed toward a portion of the three-dimensional environment based on detection of a gaze directed toward the portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, the user's attention is determined to be directed toward a portion of the three-dimensional environment based on detection of a gaze directed toward the portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed toward the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed toward the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed toward the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed toward the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0106] In some embodiments, the detection of a ready state configuration of a user or a portion of a user is detected by a computer system. The detection of the ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs performed by the hand (e.g., a pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein). For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart in preparation for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0107] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on tangible, non-transitory media such as optical, magnetic, or electronic memory media. In some embodiments, the database 408 is also stored in memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4, but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or integrated with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0108] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where shades of gray become darker with increasing depth. The controller 110 processes these depth values ​​in order to identify and segment components of the image (i.e., a group of adjacent pixels) that have characteristics of a human hand. These characteristics may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0109] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 , a hand skeleton 414 is superimposed on a hand background 416 that has been segmented from the original depth map. In some embodiments, key feature points of the hand and, optionally, on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, the center of the palm, or the end of the hand connected to the wrist) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine a gesture performed by the hand or the current state of the hand according to some embodiments.

[0110] Figure 5 An eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for the user to view and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of the head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally used in conjunction with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device and is optionally part of a non-head-mounted display generation component.

[0111] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display and display virtual objects on the transparent or translucent display, through which the user can directly view the physical environment. In some embodiments, the display generation component projects the virtual objects into the physical environment. The virtual objects may, for example, be projected onto a physical surface or projected as a hologram, so that the individual using the system observes the virtual objects superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0112] like Figure 5As shown in , in some embodiments, the eye tracking device 130 (e.g., a gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive IR or NIR light that the light source reflects directly from the eyes, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect IR or NIR light from the eyes toward the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes these images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, both eyes of the user are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by corresponding eye tracking camera and illumination source.

[0113] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometry and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. The user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, fovea position, optical axis, visual axis, and / or eye spacing. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0114] like Figure 5As shown in FIG, an eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system that includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 may be directed toward a reflector 550 (these reflectors reflect the IR or NIR light from the eye 592 while allowing visible light to pass) positioned between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, a display of a handheld device, or a projector). Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 (as shown in the bottom portion of the ).

[0115] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0116] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal region determined based on the user's current gaze direction than in the peripheral region. As another example, the controller may position or move virtual content within the view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content within the view based at least in part on the user's current gaze direction. As another example use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of an XR experience to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment on the display 510 that the user is currently looking at. As another example use case, the eye lens 520 may be a focusable lens, and the controller may use gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object the user is currently looking at has the appropriate vergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0117] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)). The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 can be used, and other arrangements and positions of the light sources 530 can be used.

[0118] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0119] like Figure 5 The embodiments of the gaze tracking system shown in can, for example, be used in computer-generated reality, virtual reality, and / or mixed reality applications to provide a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience to a user.

[0120] Figure 6 FIGURE 1 illustrates a flash-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and glint in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0121] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each set of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0122] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0123] At 640, if proceeding from element 610, the current frame is analyzed to track the pupil and glint based in part on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupil and glint detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result can be trusted. For example, the result can be checked to determine whether the pupil and a sufficient number of glints were successfully tracked or detected in the current frame to perform gaze estimation. At 650, if the result is not likely to be trusted, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is trustworthy, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0124] Figure 6 This is intended to be used as an example of an eye tracking technology that may be used for a particular implementation. As one of ordinary skill in the art will appreciate, according to various embodiments, other eye tracking technologies currently existing or developed in the future may be used in place of or in combination with the flash-assisted eye tracking technology described herein in the computer system 101 for providing an XR experience to a user.

[0125] In some embodiments, the captured portion of the real-world environment 602 is used to provide an XR experience to the user, such as a mixed reality environment in which one or more virtual objects are superimposed on top of the representation of the real-world environment 602.

[0126] Thus, the description herein describes some embodiments of a three-dimensional environment (e.g., an XR environment) that includes representations of real-world objects and representations of virtual objects. For example, the three-dimensional environment optionally includes a representation of a table present in a physical environment, which is captured and displayed in the three-dimensional environment (e.g., actively displayed via a camera and display of a computer system or passively displayed via a transparent or translucent display of the computer system). As previously described, the three-dimensional environment is optionally a mixed reality system, wherein the three-dimensional environment is based on a physical environment captured by one or more sensors of the computer system and displayed via a display generation component. As a mixed reality system, the computer system is optionally capable of selectively displaying portions and / or objects of the physical environment so that the corresponding portions and / or objects of the physical environment appear as if they exist in the three-dimensional environment displayed by the computer system. Similarly, the computer system is optionally capable of displaying virtual objects in the three-dimensional environment so that it appears as if the virtual objects exist in the real world (e.g., a physical environment) by placing the virtual objects at corresponding locations in the three-dimensional environment that have corresponding locations in the real world. For example, the computer system optionally displays a vase so that the vase appears as if a real vase is placed on top of a table in the physical environment. In some embodiments, a corresponding location in the three-dimensional environment has a corresponding location in the physical environment. Thus, when a computer system is described as displaying a virtual object at a corresponding location relative to a physical object (e.g., such as at a location at or near a user's hand or at a location at or near a physical table), the computer system displays the virtual object at a particular location in the three-dimensional environment such that it appears as if the virtual object is at or near the physical object in the physical environment (e.g., the virtual object is displayed at a location in the three-dimensional environment that corresponds to the location in the physical environment where the virtual object would be displayed if it were the real object at that particular location).

[0127] In some embodiments, real-world objects present in the physical environment that are displayed in the three-dimensional environment (e.g., and / or visible via display generation components) can interact with virtual objects that exist only in the three-dimensional environment. For example, the three-dimensional environment may include a table and a vase placed on top of the table, where the table is a view (or representation) of a physical table in the physical environment and the vase is a virtual object.

[0128] Similarly, the user can optionally use one or both hands to interact with virtual objects in the three-dimensional environment as if the virtual objects were real objects in the physical environment. For example, as described above, one or more sensors of the computer system can optionally capture one or more of the user's hands and display representations of the user's hands in the three-dimensional environment (e.g., in a manner similar to displaying real-world objects in the three-dimensional environment described above), or in some embodiments, the user's hands can be visible via the display generation component due to the transparency / translucency of a portion of the user interface being displayed by the display generation component, or due to the projection of the user interface onto a transparent / translucent surface or onto the user's eyes or into the field of view of the user's eyes, via the ability to see the physical environment through the user interface. Thus, in some embodiments, the user's hands are displayed at corresponding locations in the three-dimensional environment and are viewed as if they were objects in the three-dimensional environment, and these objects can interact with virtual objects in the three-dimensional environment as if these virtual objects were physical objects in the physical environment. In some embodiments, the computer system can update the display of the representations of the user's hands in the three-dimensional environment in conjunction with the movement of the user's hands in the physical environment.

[0129] In some embodiments described below, the computer system is optionally capable of determining an "effective" distance between a physical object in the physical world and a virtual object in a three-dimensional environment, for example, to determine whether the physical object is directly interacting with the virtual object (e.g., whether the hand is touching, grabbing, or holding the virtual object or is within a threshold distance of the virtual object). For example, a hand directly interacting with a virtual object optionally includes one or more of the following: a finger of a hand pressing a virtual button, a user's hand grabbing a virtual vase, two fingers of a user's hand coming together and pinching / holding the user interface of an application, and performing any other type of interaction described herein. For example, when determining whether a user is interacting with a virtual object and / or how the user is interacting with the virtual object, the computer system optionally determines the distance between the user's hand and the virtual object. In some embodiments, the computer system determines the distance between the user's hand and the virtual object by determining the distance between the position of the hand in the three-dimensional environment and the position of the virtual object of interest in the three-dimensional environment. For example, the user's hand(s) are positioned at a specific location in the physical world, and the computer system optionally captures the hand(s) and displays the hand(s) at a specific corresponding location in the three-dimensional environment (e.g., the location at which the hand(s) would be displayed in the three-dimensional environment if the hand(s) were virtual hands rather than physical hands). The location of the hand(s) in the three-dimensional environment is optionally compared to the location of the virtual object(s) of interest in the three-dimensional environment to determine the distance between the user's hand(s) and the virtual object(s). In some embodiments, the computer system optionally determines the distance between the physical object(s) and the virtual object(s) by comparing the locations in the physical world (e.g., rather than comparing the locations in the three-dimensional environment). For example, when determining the distance between the user's hand(s) and the virtual object(s), the computer system optionally determines the corresponding location of the virtual object(s) in the physical world (e.g., the location at which the virtual object(s) would be located in the physical world if the virtual object(s) were physical objects rather than virtual objects), and then determines the distance between the corresponding physical location and the user's hand(s). In some embodiments, the same technique is optionally used to determine the distance between any physical object and any virtual object. Thus, as described herein, when determining whether a physical object is in contact with a virtual object or whether a physical object is within a threshold distance of a virtual object, the computer system optionally executes any of the techniques described above to map the position of the physical object to a three-dimensional environment and / or map the position of the virtual object to the physical environment.

[0130] In some embodiments, the same or similar techniques are used to determine where and what the user's gaze is directed to, and / or where and what the physical stylus held by the user is pointed to. For example, if the user's gaze is directed to a particular location in the physical environment, the computer system optionally determines a corresponding location in the three-dimensional environment (e.g., a virtual location of the gaze), and if a virtual object is located at the corresponding virtual location, the computer system optionally determines that the user's gaze is directed to the virtual object. Similarly, the computer system is optionally able to determine the direction in which the stylus is pointing in the physical environment based on the orientation of the physical stylus. In some embodiments, based on this determination, the computer system determines a corresponding virtual location in the three-dimensional environment that corresponds to the location in the physical environment that the stylus is pointing to, and optionally determines that the stylus is pointing to the corresponding virtual location in the three-dimensional environment.

[0131] Similarly, the embodiments described herein may refer to the position of a user (e.g., a user of a computer system) in a three-dimensional environment and / or the position of a computer system in a three-dimensional environment. In some embodiments, the user of the computer system is holding, wearing, or otherwise located at or near the computer system. Thus, in some embodiments, the position of the computer system is used as a proxy for the position of the user. In some embodiments, the position of the computer system and / or the user in the physical environment corresponds to a corresponding position in the three-dimensional environment. For example, the position of the computer system will be a position in the physical environment (and its corresponding position in the three-dimensional environment) that, if the user were standing at that position, facing the corresponding portion of the physical environment visible via the display generation component, would be visible to the user from that position in the physical environment in the same position, orientation, and / or size (e.g., in absolute terms and / or relative to each other) as the objects displayed in the three-dimensional environment by the display generation component of the computer system or visible in the three-dimensional environment via the display generation component. Similarly, if the virtual objects displayed in the three-dimensional environment are physical objects in the physical environment (e.g., physical objects placed at the same location in the physical environment as the virtual objects are located in the three-dimensional environment, and physical objects that have the same size and orientation in the physical environment as they do in the three-dimensional environment), then the position of the computer system and / or user is the position from which the user would see the virtual objects in the physical environment at the same location, orientation, and / or size (e.g., in an absolute sense and / or relative to each other and real-world objects) as the virtual objects displayed in the three-dimensional environment by the display generation components of the computer system.

[0132] In this disclosure, various input methods are described with respect to interaction with a computer system. When one input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with and optionally utilizes the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. When one output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with and optionally utilizes the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment through a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, the present disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0133] User interface and associated processes

[0134] Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system (such as a portable multifunction device or a head-mounted device) having display generating components, one or more input devices, and (optionally) one or more cameras.

[0135] 7A to 7F An example of a computer system selectively re-centering virtual content to a user's viewpoint is illustrated according to some embodiments.

[0136] Figure 7A The display generation component (eg, Figure 1 101 ) is visible from a user's viewpoint 726a, illustrated in a top view (e.g., facing the back wall of the physical environment in which the computer system 101 is located and near the back left corner of the physical environment). Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3The image sensor 314 optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more hands of the user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays the user interface or three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0137] like Figure 7A As shown in , the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., operating environment 100), including one or more objects in the physical environment surrounding the computer system 101. In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 702, and / or the physical environment is visible in the three-dimensional environment 702 via the display generation component 120. For example, the three-dimensional environment 702 visible via the display generation component 120 includes a representation of the physical floor and the back and side walls of the room in which the computer system 101 is located. The three-dimensional environment 702 also includes images of the physical environment surrounding the computer system 101, including one or more objects in the physical environment surrounding the computer system 101. Figure 7A The sofa 724b (shown in a top view) is not visible from the user's viewpoint 726a.

[0138] exist Figure 7A , the three-dimensional environment 702 also includes virtual objects 712a (corresponding to the object 712b in the top view) and 714a (corresponding to the object 714b in the top view) visible from the viewpoint 726a. Figure 7A The virtual object 710b is not visible from the user's viewpoint 726a (shown in top view). Figure 7A , objects 712a, 714a, and 710b are two-dimensional objects. It should be understood that the examples of the present disclosure are optionally also applicable to three-dimensional objects. Virtual objects 712a, 714a, and 710b are optionally one or more of a user interface of an application (e.g., a messaging user interface or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by computer system 101 that is not included in the physical environment of computer system 101.

[0139] In some embodiments, a virtual object that was last placed or repositioned from a user's particular previous viewpoint (or multiple previous viewpoints) may be recentered to the user's new current viewpoint, as will be described in more detail below. Figure 7A , virtual objects 712a, 714a, and 710a are placed and / or positioned at their current locations and / or orientations in the three-dimensional environment 702 from the user's viewpoint 726a (as reflected in the top view). In addition, virtual object 712a has been aligned or anchored to the back wall of the physical environment, as shown in FIG. Figure 7A As described in more detail with reference to method 800, in response to the virtual object being moved to a position within a threshold distance (e.g., 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, or 100 cm) of a physical object in the three-dimensional environment 702, the virtual object optionally becomes aligned or anchored to the physical object in response to user input to the position. In addition, in some embodiments, the computer system 101 displays a visual indication in the three-dimensional environment 702 indicating that the virtual object is aligned or anchored to the physical object. For example, in Figure 7A , computer system 101 is displaying a virtual projected shadow 713 on the back wall of a room in a physical environment as if it were generated by a virtual object 712a (e.g., a virtual object aligned or anchored to a physical object). In some embodiments, computer system 101 does not display such a visual indication of virtual object 714a because the virtual object is optionally not aligned or anchored to a physical object.

[0140] exist Figure 7B , the user's viewpoint 726a in the three-dimensional environment 702 has changed to be further away from the back and left walls of the room of the physical environment and closer to the center of the room, as shown in the top view. Viewpoint 726b in the top view corresponds to Figure 7A The user's previous viewpoint is shown in . The user's viewpoint 726a is optionally changed in the manner described with reference to method 800, including movement of the user in the user's physical environment toward the center of a room in the physical environment. Figure 7B The user's viewpoint 726a in is still oriented toward the back wall of the room.

[0141] from Figure 7B From the viewpoint 726a shown in FIG, virtual objects 710a, 712a, and 714a (which are last placed or positioned in the three-dimensional environment 702 from the viewpoint 726b, as shown in FIG. Figure 7A ) are displayed in the three-dimensional environment 702 at the same position and / or orientation, but at a greater distance from the viewpoint 726a. Figure 7BThe viewpoint 726a in FIG. 7 places or positions virtual objects 706a (corresponding to 706b in the top view) and 708a (corresponding to 708b in the top view) in the three-dimensional environment 702 .

[0142] exist Figure 7B In some embodiments, the computer system 101 detects an input (e.g., selection of a physical button of the computer system 101) for re-centering one or more virtual objects to the user's viewpoint 726a, such as described in more detail with reference to method 800. In some embodiments, the virtual objects 706a and 708a do not move in the three-dimensional environment 702 in response to the input because those virtual objects were last placed or re-positioned in the three-dimensional environment from the user's current viewpoint 726a. However, one or more virtual objects that were last placed or re-positioned in the three-dimensional environment 702 from the user's previous viewpoint (e.g., viewpoint 726b) are optionally re-centered to viewpoint 726a, as will be described below and as described in more detail with reference to method 800.

[0143] For example, Figure 7C Illustrated Figure 7B Example results for the illustrated input. Figure 7C , in response to the re-centering input, objects 706 a and 708 a maintain their positions and / or orientations in the three-dimensional environment 702. Although object 712 a was last placed or re-positioned in the three-dimensional environment 702 from the previous viewpoint 726 b, in response to the re-centering input, the object also maintains its position and / or orientation in the three-dimensional environment 702 because object 712 a is aligned or anchored to the back wall of the physical environment of the computer system 101.

[0144] In contrast, objects 710b and 714a have been recentered to the user's viewpoint 726a. In some embodiments, the relative positions and / or orientations of objects 710b and 714a relative to viewpoint 726a are the same as the relative positions and / or orientations of objects 710b and 714a relative to viewpoint 726b. For example, object 714a is optionally Figure 7C In the display Figure 7A In addition, object 710b is optionally located at the same position relative to viewpoint 726a as in Figure 7A Same as in Figure 7C726a. Furthermore, the spatial arrangement of objects 710b and 714a relative to one another is optionally maintained before and after the re-centering input. Additional details regarding the movement of objects 710b and 714a in response to the re-centering input are described with reference to method 800. In this manner, virtual objects associated with a user's previous viewpoint can be easily moved to the user's current viewpoint to facilitate interaction with and / or visibility of those virtual objects.

[0145] In some embodiments, the simulated environment can also be recentered to this viewpoint in a manner similar to how virtual objects are recentered to the user's new current viewpoint. Figure 7D , the user's viewpoint 726a is shown in a top-down view. The user has provided input from viewpoint 726a to place or reposition virtual objects 706a and 708a at their current positions and / or orientations in the three-dimensional environment 702, as shown. Figure 7D In addition, the user has provided input that causes the computer system to display the simulated environment 703 from viewpoint 726a. The simulated environment 703 optionally occupies a portion of the three-dimensional environment 702, as shown in the top view. Additional details regarding the simulated environment 703 are described with reference to method 800.

[0146] exist Figure 7E , the viewpoint 726a has changed to the viewpoint illustrated in the top view (e.g., moved downward and oriented toward the left wall in the physical environment rather than the back wall). The viewpoint 726a is optionally moved in the manner previously described and / or as described with reference to method 800. The virtual objects 706a and 708a are no longer visible via the display generation component 120. Additionally, in some embodiments, the computer system 101 removes the simulated environment 703 from the three-dimensional environment 702 in response to the movement of the user's viewpoint 726a, as shown in the top view. In some embodiments, the computer system 101 maintains the simulated environment 703 in the three-dimensional environment 702 in response to the movement of the user's viewpoint 726a, even though the simulated environment 703 is no longer in the field of view of the three-dimensional environment 702 from the user's current viewpoint 726a. Figure 7E , virtual objects 706b and 708b are also not in the field of view of the three-dimensional environment 702 from the user's current viewpoint 726a.

[0147] exist Figure 7E, computer system 101 is capable of detecting at least two different inputs: 1) a re-centering input (e.g., as previously described); or 2) an input that increases the immersion level of displayed three-dimensional environment 702. Immersion and immersion level are described in more detail with reference to method 800. The re-centering input is optionally a press of an input element (e.g., a depressible dial that is also rotatable, as described below). The input that increases the immersion level is optionally a rotation of the input element in a particular direction. Additional details regarding the above inputs are provided with reference to method 800. Computer system 101 optionally responds differently to the above two inputs, as described below.

[0148] Figure 7F Reference Figure 7E Example results of the re-centering input described. Figure 7F In FIG. 7 , objects 706a and 708a have been recentered to the user's viewpoint 726a. In some embodiments, objects 706a and 708a are positioned relative to Figure 7F The relative position and / or orientation of viewpoint 726a in FIG. 1 and objects 706a and 708a relative to FIG. Figure 7D The relative position and / or orientation of the viewpoint 726a in the same. For example, the object 706a is optionally Figure 7F In the display Figure 7D 726a in the same position relative to viewpoint 726a. In addition, the relative spatial arrangement of objects 706a and 708a relative to each other is optionally maintained before and after the re-centering input. Additional details regarding the movement of objects 706a and 708a in response to the re-centering input are described with reference to method 800.

[0149] Except that objects 706a and 708a become recentered to Figure 7F The computer system 101 redisplays the simulated environment 703 in the three-dimensional environment 702. Figure 7F As shown in FIG, the computer system 101 places the simulation environment 703 in the three-dimensional environment 702. Figure 7D In some embodiments, the location and / or orientation of the simulated environment 703 is based on the location and / or orientation of the simulated environment 703. Figure 7F For example, the simulated environment 703 is optionally located at Figure 7F is placed at a distance from viewpoint 726a of Figure 7D In addition or alternatively, the simulated environment 703 is optionally Figure 7F The viewpoint 726a in the image is centered and / or oriented toward Figure 7F726a in the orientation of the simulated environment 703 (eg, the orientation of the viewpoint 726a is toward the center of the simulated environment 703 and / or the orientation of the simulated environment 703 is toward the viewpoint 726a). Additional details regarding the display of the simulated environment 703 in response to the re-centering input are provided with reference to the method 800.

[0150] In contrast to re-centering the input, if Figure 7E If the computer system 101 in FIG. 1 has detected an input that increases the immersion level of the three-dimensional environment 702 that the computer system is displaying, the computer system 101 will optionally redisplay the simulated environment 703 in the manner described above, however, the virtual objects 706a and 708a will optionally not be recentered to Figure 7F For example, objects 706a and 708a will optionally maintain their Figure 7E Additional details of the computer system 101's response to detecting such input that increases the immersion level of the three-dimensional environment 702 are provided with reference to the method 800.

[0151] Figures 8A to 8I is a flow chart illustrating an exemplary method of selectively re-centering virtual content to a user's viewpoint according to some embodiments. In some embodiments, method 800 is executed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4 In some embodiments, the method 800 is performed by a display generating component 120 in the computer system (e.g., a heads-up display, a display, a touch screen, or a projector) and one or more cameras (e.g., a camera pointing downward toward the user's hand (e.g., a color sensor, an infrared sensor, or other depth sensing camera) or a camera pointing forward from the user's head). In some embodiments, the method 800 is performed by a computer system stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of the computer system 101 (e.g., Figure 1 Some operations in method 800 may be optionally combined, and / or the order of some operations may be optionally changed.

[0152] In some embodiments, method 800 is performed at a computer system (e.g., 101) that communicates with a display generation component and one or more input devices. For example, a mobile device (e.g., a tablet, a smart phone, a media player, or a wearable device) or a computer or other electronic device. In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touch screen display), an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or making the user interface visible to one or more users. In some embodiments, the one or more input devices include a device capable of receiving user input (e.g., capturing user input or detecting user input) and transmitting information associated with the user input to the computer system. Examples of input devices include a touch screen, a mouse (e.g., external), a trackpad (optionally integrated or external), a touchpad (optionally integrated or external), a remote control device (e.g., external), another mobile device (e.g., separate from the computer system), a handheld device (e.g., external), a controller (e.g., external), a camera, a depth sensor, an eye tracking device, and / or a motion sensor (e.g., a hand tracking device, a hand motion sensor). In some embodiments, the computer system communicates with a hand tracking device (e.g., one or more cameras, depth sensors, proximity sensors, touch sensors (e.g., touch screen, trackpad)). In some embodiments, the hand tracking device is a wearable device, such as a smart glove. In some embodiments, the hand tracking device is a handheld input device, such as a remote control or a stylus.

[0153] In some embodiments, when a three-dimensional environment (e.g., 702) is visible via a display generation component (e.g., the three-dimensional environment is generated, displayed, or otherwise enabled for viewing by a computer system (e.g., a computer-generated reality (CGR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment)), wherein the three-dimensional environment includes a first virtual object having a first spatial arrangement relative to a first viewpoint of a user of the three-dimensional environment, the first viewpoint being a current viewpoint of the computer system, such as relative to Figure 7BIn some embodiments, the first virtual object is placed by a user of the computer system at its current position in the three-dimensional environment, regardless of whether the user's viewpoint is the user's current viewpoint or a previous viewpoint of the user. In some embodiments, the user's first viewpoint corresponds to the user's current position and / or orientation in the physical environment of the user, the computer system, and / or the display generation component, and the computer system and / or the display generation component are computed. The computer system displays at least some portion of the three-dimensional environment from a viewpoint corresponding to the user's current position and / or orientation in the physical environment. In some embodiments, the first virtual object is a user interface of an application, a representation of content (e.g., images, video, audio, or music), a three-dimensional rendering of an object (e.g., a tent, a building, or a car), or any other object not present in the user's physical environment), and the computer system (e.g., 101) receives (802a) via the one or more input devices a first input corresponding to a request to update the spatial arrangement of the one or more virtual objects relative to the user's first viewpoint to satisfy a first set of one or more criteria, such as in Figure 7BIn some embodiments, the three-dimensional environment includes one or more virtual objects (e.g., a first virtual object), such as an application window, an operating system element, a representation of another user, and / or a content item. In some embodiments, the three-dimensional environment includes representations of physical objects in the physical environment of the computer system. In some embodiments, the representations of the physical objects are displayed in the three-dimensional environment via a display generation component (e.g., a virtual or video pass-through). In some embodiments, the representations of the physical objects are views of the physical objects in the physical environment of the computer system that are visible through a transparent portion of the display generation component (e.g., a real or true pass-through). In some embodiments, the computer system displays the three-dimensional environment from the user's viewpoint at a location in the three-dimensional environment that corresponds to the physical location of the computer system, the user, and / or the display generation component in the physical environment of the computer system. In some embodiments, the input corresponding to the request to update the spatial arrangement of objects relative to the user's viewpoint to satisfy the first one or more criteria is an input to a hardware button or switch in communication with (e.g., incorporated with) the computer system. In some embodiments, the first input is an input to a selectable option displayed via the display generation component. In some embodiments, the first one or more criteria include criteria that are satisfied when an interactive portion of the virtual object is oriented toward the user's viewpoint, the virtual object does not obstruct the view of other virtual objects from the user's viewpoint, the virtual object is within a threshold distance (e.g., 10 cm, 20 cm, 30 cm, 40 cm, 50 cm, 100 cm, 200 cm, 300 cm, 400 cm, 500 cm, 1000 cm, or 2000 cm) of the user's viewpoint, and / or the virtual objects are within a threshold distance (e.g., 1 cm, 5 cm, 10 cm, 20 cm, 30 cm, 40 cm, 50 cm, 100 cm, 200 cm, 300 cm, 400 cm, 500 cm, 1000 cm, or 2000 cm) of each other, etc. In some embodiments, the first input is different from an input requesting an update to the positioning (e.g., relative to the user's viewpoint) of one or more objects in the three-dimensional environment, such as an input to manually move an object in the three-dimensional environment.

[0154] In some embodiments, in response to receiving the first input (802b), the first virtual object is determined to satisfy a second set of one or more criteria, such as Figure 7B20, 500, or 1000 cm) from a previous viewpoint from which the first virtual object was last placed or moved into the three-dimensional environment. Figure 7C In some embodiments, displaying the first virtual object in the second spatial arrangement includes updating a position (e.g., and / or pose) of the first virtual object while maintaining the user's second viewpoint at a constant position in the three-dimensional environment. In some embodiments, in response to the first input, the computer system updates the positioning of the first virtual object from a position that is not necessarily oriented around the user's first viewpoint to a position that is oriented around the user's first viewpoint.

[0155] In some embodiments, in response to receiving the first input (802b), the first virtual object is determined not to satisfy a second set of one or more criteria, such as Figure 7B The computer system (e.g., 101) maintains (802d) a first spatial arrangement of the first virtual objects in the three-dimensional environment relative to the user's first viewpoint, such as in Figure 7C 800. In some embodiments, the first virtual object is visible from the user's current viewpoint via the display generation component. In some embodiments, the first virtual object is not visible from the user's current viewpoint via the display generation component. In some embodiments, the computer system similarly changes (or does not change) the positions of other virtual objects in the three-dimensional environment in response to the first input. In some embodiments, the input described with reference to method 800 is or includes an air gesture input. Changing the position of some, but not all, objects in the three-dimensional environment in response to the first input reduces the number of inputs required to appropriately place the objects relative to the user's viewpoint in response to the first input.

[0156] In some embodiments, when the first input is detected, the three-dimensional environment includes a first virtual object and a second virtual object, such as Figure 7BIn one embodiment, the present invention relates to objects 714a and 706a in the user's first viewpoint (e.g., having one or more characteristics of the first virtual object), and the second virtual object has a third spatial arrangement relative to the user's first viewpoint (e.g., the second virtual object is a specific distance from the user's current viewpoint and has a specific orientation relative to the user's current viewpoint) (804a).

[0157] In some embodiments, in response to receiving the first input, the first virtual object has a second spatial arrangement relative to the user's first viewpoint, and the second virtual object has a third spatial arrangement relative to the user (804b), such as in Figure 7C , shown by objects 714a and 706a in . In some embodiments, the first virtual object is recentered in response to the first input as described above, but the second virtual object is not recentered in response to the first input (e.g., maintaining its current position and / or orientation relative to the user's first viewpoint). In some embodiments, the second virtual object is not recentered because its current position and / or orientation already satisfies the first set of one or more criteria. In some embodiments, the second virtual object is not recentered because it was last placed or positioned in the three-dimensional environment from the user's first viewpoint, or is anchored to a physical object, both of which are described in more detail below. Changing the position of some, but not all, objects in the three-dimensional environment in response to the first input reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0158] In some embodiments, the second set of one or more criteria includes criteria that are not satisfied when the first virtual object is last placed or moved in the three-dimensional environment from a viewpoint that satisfies the third set of one or more criteria relative to the user's first viewpoint, such as when objects 706a and 708a are moved from a viewpoint that satisfies the third set of one or more criteria. Figure 7BIn some embodiments, the user's current viewpoint 726a (e.g., corresponding to the user's current physical position or orientation in the user's physical environment) is last placed or moved into the environment 702 (806). In some embodiments, the user's current viewpoint (e.g., the position and / or orientation of the current viewpoint) corresponds to the user's (e.g., the user's head or torso's) current position and / or orientation in the user's physical environment. In some embodiments, a virtual object that was last placed or positioned in the three-dimensional environment from the user's first viewpoint (e.g., within a threshold distance from the user's current viewpoint and / or within a threshold orientation of the user's current viewpoint, as described in more detail below) is not recentered in response to the first input, while a virtual object that was last placed or positioned in the three-dimensional environment from a viewpoint different from the user's first viewpoint (or sufficiently different in position and / or orientation from the user's first viewpoint) is recentered in response to the first input. Changing the position of an object that was last placed or positioned in the three-dimensional environment from the user's previous viewpoint in response to the first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to the first input.

[0159] In some embodiments, the third set of one or more criteria includes criteria that are satisfied when the viewpoint is within a threshold distance (e.g., 3, 5, 50, 100, 1000, 5000, or 10000 cm) of the first viewpoint (808), such as if objects 706a and 708a are within Figure 7B 726a in the three-dimensional environment was last placed or moved into the environment 702. Thus, in some embodiments, if the first virtual object is farther than the threshold distance from the viewpoint from which it was last placed or positioned in the three-dimensional environment than the user's current viewpoint, the criterion is not satisfied, and if the first virtual object is closer than the threshold distance from the viewpoint from which it was last placed or positioned in the three-dimensional environment than the user's current viewpoint, the criterion is satisfied. Changing the position of an object that was last placed or positioned from a previous viewpoint of the user that is relatively far from the current viewpoint in the three-dimensional environment in response to the first input reduces the number of inputs required to properly place the object relative to the user's viewpoint in response to the first input.

[0160] In some embodiments, the third set of one or more criteria includes a criterion (810) that is satisfied when the viewpoint has an orientation in the three-dimensional environment that is within a threshold orientation (e.g., within 1 degree, 3 degrees, 5 degrees, 10 degrees, 20 degrees, 30 degrees, 45 degrees, or 90 degrees) of the orientation of the first viewpoint in the three-dimensional environment, such as if objects 706a and 708a are viewed from the left side of the viewpoint. Figure 7B726a in the environment 702. Thus, in some embodiments, if the orientation of the first virtual object from the viewpoint at which it was last placed or positioned in the three-dimensional environment is greater than the threshold orientation from the orientation of the user's current viewpoint, the criterion is not satisfied, and if the orientation of the first virtual object from the viewpoint at which it was last placed or positioned in the three-dimensional environment is less than the threshold orientation from the orientation of the user's current viewpoint, the criterion is satisfied. Changing the position of an object that was last placed or positioned in the three-dimensional environment from a previous viewpoint of the user that is relatively offset from the current viewpoint in response to the first input reduces the number of inputs required to properly position the object relative to the user's viewpoint in response to the first input.

[0161] In some embodiments, the second set of one or more criteria includes criteria that are not satisfied when the first virtual object is anchored to a portion of the user's physical environment, such as object 712a being anchored to Figure 7B In some embodiments, the computer system detects an input to move the first virtual object to within a threshold distance (e.g., 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 30, or 50 cm) of a portion of the physical object, the first virtual object becomes anchored to the portion (e.g., surface) of the physical object, optionally causing the first virtual object to align with the position and / or orientation of the portion of the physical object. Objects anchored to the physical object are thus optionally not recentered in response to the first input. In some embodiments, the criterion is met if the first virtual object is not anchored to the physical object. Changing the position of an object that is not anchored to the physical object in the three-dimensional environment in response to the first input reduces the number of inputs required to appropriately place the object relative to the user's viewpoint in response to the first input.

[0162] In some embodiments, when displaying the first virtual object in the three-dimensional environment (814a), based on determining that the first virtual object is anchored to the portion of the user's physical environment, the computer system (e.g., 101) displays (814b) in the three-dimensional environment a visual indication that the first virtual object is anchored to the portion of the physical environment, such as 7A to 7B , a virtual projected shadow 713 in the physical environment (e.g., a virtual projected shadow of the first virtual object, the virtual projected shadow displayed on the portion of the physical environment as if the projected shadow was cast by the first virtual object onto the portion of the physical environment; and / or an icon (e.g., a pin icon) displayed in association with the first virtual object, the icon indicating that the first virtual object is anchored or pinned to the portion of the physical environment); and

[0163] In some embodiments, when displaying the first virtual object in the three-dimensional environment (814a), based on determining that the first virtual object is not anchored to the portion of the user's physical environment, the computer system (e.g., 101) displays (814c) the first virtual object in the three-dimensional environment without displaying a visual indication, such as in 7A to 7B In the embodiment of the present invention, the object 714a is displayed without a virtual projected shadow (e.g., the projected shadow and / or icon is not displayed unless or until the first virtual object is anchored to a portion of the physical environment). Indicating the anchored state of the first virtual object provides feedback about the state of the first virtual object.

[0164] In some embodiments, the first virtual object is part of a set of multiple virtual objects in the three-dimensional environment that meet a second set of one or more criteria, such as Figure 7B The set of objects 710a and 714a in (eg, the plurality of virtual objects that were last placed or positioned in the three-dimensional environment from the same previous viewpoint of the user) is (816a).

[0165] In some embodiments, when the first input is received, the set has a first corresponding spatial arrangement relative to the first viewpoint (816b), such as Figure 7B The set of objects 710a and 714a is spatially arranged relative to viewpoint 726a.

[0166] In some embodiments, in response to receiving the first input, the collection is displayed in a second corresponding spatial arrangement relative to the first viewpoint that is different from the first corresponding spatial arrangement, such as Figure 7CThe spatial arrangement of the set of virtual objects 710a and 714a relative to the viewpoint 726a is performed (e.g., re-centering (e.g., moving and / or reorienting) the set of multiple virtual objects as a group in response to the first input), wherein the spatial arrangement of the multiple virtual objects in the set relative to the first viewpoint satisfies a first set of one or more criteria after receiving the first input (e.g., re-centering the virtual objects in the set to positions and / or orientations that satisfy the first set of one or more criteria) (816c). In some embodiments, in response to the re-centering input, the virtual objects are re-centered in groups or on a group basis. The group of virtual objects that were last placed or positioned in the three-dimensional environment from the same previous viewpoint of the user are optionally re-centered as a group to the first viewpoint (e.g., the virtual objects move together to their updated positions and / or orientations). In some embodiments, the three-dimensional environment includes multiple different sets of virtual objects that were last placed or positioned in the three-dimensional environment from different shared previous viewpoints of the user and are concurrently re-centered as a group of virtual objects in response to the first input. In some embodiments, the three-dimensional environment includes a collection of virtual objects that were last placed or positioned in the three-dimensional environment from a first viewpoint of the user and are therefore not recentered as a group in response to the first input. Recentering the virtual objects as a group of objects reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0167] In some embodiments, before receiving the first input and when the set has a first corresponding spatial arrangement relative to the first viewpoint, the plurality of virtual objects within the set have a corresponding positional arrangement relative to each other, such as Figure 7B A positioning arrangement is performed between objects 710a and 714a in the set (eg, the virtual objects in the set have a particular positioning relative to each other, such as four virtual objects positioned at the vertices of a square arrangement) (818a).

[0168] In some embodiments, after receiving the first input and when the set has a second corresponding spatial arrangement relative to the first viewpoint, the plurality of virtual objects within the set have a corresponding positional arrangement relative to each other (818b), such as Figure 7C, and / or the like. For example, even if the set of virtual objects is repositioned and / or reoriented in the three-dimensional environment in response to the first input, the relative positioning of the virtual objects in the set of virtual objects is maintained in response to the first input (e.g., the four virtual objects remain positioned at vertices of the same square arrangement in response to the first input, even though the square arrangement has different positions and / or orientations in the three-dimensional environment). Maintaining the positional arrangement of the virtual objects in the set reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0169] In some embodiments, before receiving the first input and when the set has a first corresponding spatial arrangement relative to the first viewpoint, the plurality of virtual objects within the set have a corresponding orientation arrangement relative to each other, such as Figure 7B An orientation arrangement between objects 710a and 714a in the set is performed (eg, the virtual objects in the set have a particular orientation relative to each other, such as four virtual objects oriented such that the virtual objects are parallel to each other) (820a).

[0170] In some embodiments, after receiving the first input and when the set has a second corresponding spatial arrangement relative to the first viewpoint, the plurality of virtual objects within the set have a corresponding orientation arrangement relative to each other (820b), such as Figure 7C , the orientation arrangement between objects 710a and 714a in the three-dimensional environment. For example, even if the set of virtual objects is repositioned and / or reoriented in the three-dimensional environment in response to the first input, the relative orientations of the virtual objects in the set of virtual objects are maintained in response to the first input (e.g., the four virtual objects remain parallel to each other despite the new positions and / or orientations of the virtual objects in the three-dimensional environment). Maintaining the orientation arrangement of the virtual objects in the set reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0171] In some embodiments, before receiving the first input, the plurality of virtual objects in the set are viewed from a second viewpoint of the user that is different from the first viewpoint of the user (such as from Figure 7A Viewpoint 726a or Figure 7B 726b) is finally placed or moved into the three-dimensional environment (e.g., as previously described, the second viewpoint is sufficiently different from the first viewpoint to cause the set of virtual objects to be re-centered in response to the first input) (822a).

[0172] In some embodiments, when the set has a first corresponding spatial arrangement relative to the first viewpoint, the average orientation of the plurality of virtual objects relative to the second viewpoint is a corresponding orientation (822b), such as Figure 7AThe average orientation of objects 714a and 710a relative to viewpoint 726a in the figure. For example, the set of virtual objects includes three virtual objects, and the three virtual objects have their own respective orientations relative to the user's second viewpoint (e.g., a first of the objects is relatively facing the center of the second viewpoint and / or located in the center of the second viewpoint, a second of the objects is about 45 degrees to the right relative to the center of the second viewpoint, and a third of the objects is about 60 degrees to the right relative to the center of the second viewpoint). When the user finally places or positions the respective virtual objects from the second viewpoint, the relative orientations of the respective virtual objects are optionally relative to and / or correspond to the orientation of the user's shoulders, head and / or chest. In some embodiments, the average of the above-mentioned orientations is the average of the orientations of the above-mentioned three virtual objects.

[0173] In some embodiments, when the set has a second corresponding spatial arrangement relative to the first viewpoint in response to receiving the first input, the set has a corresponding orientation relative to the user's first viewpoint (822c), such as Figure 7C 726a. For example, a group or set of virtual objects that are recentered to the first viewpoint in response to the first input are placed in the three-dimensional environment at an orientation relative to the first viewpoint that corresponds to an average of the relative orientations of the virtual objects in the set of virtual objects relative to the second viewpoint (e.g., when the objects were last placed or positioned in the three-dimensional environment). Thus, if the average orientation of the virtual objects relative to the second viewpoint is 30 degrees to the right relative to the centerline of the second viewpoint, the set of virtual objects is optionally oriented / positioned 30 degrees to the right relative to the centerline of the first viewpoint (e.g., while the relative positioning and / or orientation of the virtual objects in the set remains unchanged). Placing the set of virtual objects at the average orientation relative to the first viewpoint reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0174] In some embodiments, before receiving the first input, the first virtual object is viewed from a second viewpoint of the user that is different from the first viewpoint of the user (such as from Figure 7A Viewpoint 726a or Figure 7B 726b) is finally placed or moved into the three-dimensional environment (e.g., as previously described, the second viewpoint is sufficiently different from the first viewpoint to cause the set of virtual objects to be re-centered in response to the first input) (824a).

[0175] In some embodiments, when the first virtual object has a first spatial arrangement relative to the user's first viewpoint, the first virtual object is at a first distance from the second viewpoint (e.g., and at a different distance from the first viewpoint) (824b), such as Figure 7AThe distance between the object 714a and the viewpoint 726a.

[0176] In some embodiments, when the first virtual object has a second spatial arrangement relative to the user's first viewpoint, the first virtual object is a first distance from the first viewpoint (e.g., and a different distance from the second viewpoint) (824c), such as Figure 7C 726a. Thus, in some embodiments, when virtual objects are recentered, the distances of the virtual objects from the user's current viewpoint are based on (e.g., are the same as) the distances of the virtual objects from the user's previous viewpoint from which they were last placed or positioned in the three-dimensional environment. Placing the recentered virtual objects at a distance from the viewpoint corresponding to their previous distance from the user's previous viewpoint reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0177] In some embodiments, prior to receiving the first input, the first virtual object is located at a first position in the three-dimensional environment, such as Figure 7B In some embodiments, the first virtual object remains at the first position in the three-dimensional environment until an input is received for repositioning the first virtual object in the three-dimensional environment (826). In some embodiments, the first virtual object remains at its position in the three-dimensional environment (e.g., not re-centered) until an input is received for re-centering or an input is received for moving the first virtual object in the three-dimensional environment (e.g., separately from the re-centering input). In some embodiments, other inputs (such as input for changing the user's viewpoint) do not cause the first virtual object to change its position in the three-dimensional environment. If no re-centering input is received, maintaining the position and / or orientation of the first virtual object in the three-dimensional environment reduces the number of inputs required to properly place the object relative to the user's viewpoint in response to the first input.

[0178] In some embodiments, before receiving the first input, the first virtual object is last placed or moved into the three-dimensional environment from a second viewpoint of the user that is different from the first viewpoint of the user (828a), such as Figure 7D Object 708a is placed from viewpoint 726a.

[0179] In some embodiments, before receiving the first input (828b), when the three-dimensional environment is visible from the user's second viewpoint via the display generation component, the computer system (e.g., 101) displays (828c) the simulated environment and the first virtual object via the display generation component, such as Figure 7D703. For example, when the user's viewpoint is the second viewpoint, the user provides input to the computer system to display the simulated environment in a three-dimensional environment visible from the user's second viewpoint. In some embodiments, the simulated environment occupies a portion of the three-dimensional environment visible via the display generation component.

[0180] In some embodiments, before receiving the first input (828b), while displaying the simulated environment in the three-dimensional environment, the computer system (e.g., 101) detects (828d) a movement of the user's viewpoint from the second viewpoint to the first viewpoint, such as from Figures 7D to 7E movement of the user in the user's physical environment (e.g., movement of the user in the user's physical environment and / or a change in the user's orientation in the user's physical environment corresponds to movement of the user's viewpoint from the second viewpoint to the first viewpoint).

[0181] In some embodiments, before receiving the first input (828b), in response to detecting movement of the user's viewpoint from the second viewpoint to the first viewpoint, the computer system (e.g., 101) maintains (828e) the first virtual object in the three-dimensional environment, such as Figure 7E (e.g., maintaining the position and / or orientation of the first virtual object in the three-dimensional environment), and ceasing to include at least a portion (or all) of the simulated environment in the three-dimensional environment, such as in Figure 7E 703 is shown in the top view of the 3D environment as being absent (e.g., the simulated environment ceases to be present in the 3D environment). In some embodiments, the change in viewpoint from the second viewpoint to the first viewpoint must be sufficiently large (e.g., as previously described with respect to the third set of one or more criteria) to cause the computer system to cease including the simulated environment in the 3D environment in response to the change in viewpoint of the user. In some embodiments, the simulated environment remains in the 3D environment in response to the change in viewpoint of the user, but is no longer visible via the display generation component (e.g., because the simulated environment is outside the user's field of view). Ceasing to include the simulated environment causes the computer system to automatically reduce resource usage and clutter in the 3D environment.

[0182] In some embodiments, in response to receiving a first input when the user's viewpoint is a first viewpoint, such as in Figure 7E In the example, a computer system (e.g., 101) displays (830) a simulated environment in a three-dimensional environment from a first viewpoint, such as in Figure 7FIn some embodiments, the simulated environment is redisplayed and / or recentered to the first viewpoint in the three-dimensional environment (e.g., the new position and / or orientation at which the simulated environment is displayed in the three-dimensional environment is different from the position and / or orientation at which the simulated environment was last displayed from the user's second viewpoint in the three-dimensional environment). For example, if the simulated environment was last displayed facing a first wall of the user's physical room and occupying a first portion of the three-dimensional environment, then when the simulated environment is redisplayed from the first viewpoint, the simulated environment faces a second wall (different from the first wall) of the user's physical room and occupies a second portion (different from the first portion) of the three-dimensional environment. The simulated environment is optionally redisplayed so that it faces the user's first viewpoint and is centered about the user's first viewpoint. The simulated environment is optionally redisplayed and / or recentered along with the recentering of the first virtual object, as previously described. Redisplaying the simulated environment in response to the first input reduces the number of inputs required to view the simulated environment in the three-dimensional environment.

[0183] In some embodiments, when the user's viewpoint is the first viewpoint and before receiving the first input, the computer system (e.g., 101) detects (832a) via the one or more input devices a second input corresponding to a request to increase the immersion level of the three-dimensional environment, such as in Figure 7E In some embodiments, the second input comprises a rotation of a rotatable mechanical input element integrated with and / or in communication with the computer system. In some embodiments, rotating the rotatable mechanical input element in a first direction is an input that increases the immersion level of the three-dimensional environment visible via the display generation component. In some embodiments, rotating the rotatable mechanical input element in an opposite direction is an input that decreases the immersion level of the three-dimensional environment visible via the display generation component.

[0184] In some embodiments, the immersion level includes an associated degree to which background content (e.g., content other than the virtual content) surrounding / behind the virtual content is obscured by content displayed by the computer system (e.g., a simulated environment or virtual objects, otherwise referred to as "virtual content"), optionally including the number of items of background content that are visible and the visual characteristics of the background content that are visible (e.g., color, contrast, opacity), and / or the angular range of content displayed via the display generation component (e.g., 60 degrees of content are displayed in low immersion, 120 degrees of content are displayed in medium immersion, and 180 degrees of content are displayed in high immersion), and / or the proportion of the field of view visible via the display generation component that is occupied by the virtual content (e.g., 33% of the field of view in low immersion, 66% of the field of view in medium immersion, and 100% of the field of view in high immersion). In some embodiments, the background content is included in the background on which the virtual content is displayed. In some embodiments, the background content includes a user interface (e.g., a user interface generated by a computer system corresponding to an application), virtual objects (e.g., files or other user representations generated by a computer system), and / or real objects (e.g., transmittal objects corresponding to real objects in the physical environment surrounding the user's viewpoint, which are visible via display generation components and / or visible via transparent or translucent display generation components because the computer system does not block / impede their visibility through the display generation components). In some embodiments, at a first (e.g., low) immersion level, the background, virtual, and / or real objects are visible in an unobstructed manner. For example, a simulated environment with a low immersion level is optionally visible concurrently with the background content, which is optionally visible in a manner with full brightness, color, and / or translucency. In some embodiments, at a second (e.g., higher) immersion level, the background, virtual, and / or real objects are visible in an obstructed manner (e.g., dimmed, blurred, or removed from the display). For example, a corresponding simulated environment with a high immersion level is displayed, while the background content is not visible concurrently (e.g., in full screen or fully immersive mode). As another example, a simulated environment displayed at a medium immersion level is concurrently visible with background content that is dimmed, blurred, or otherwise de-emphasized. In some embodiments, the visual characteristics of the background objects vary between the background objects. For example, at a particular immersion level, one or more first background objects are more visually de-emphasized (e.g., dimmed, blurred, visible with increased transparency) than one or more second background objects, and one or more third background objects cease to be visible.

[0185] In some embodiments, in response to receiving the second input, the computer system (e.g., 101) displays (832b) the simulated environment from a first viewpoint in the three-dimensional environment, such as Figure 7F, and optionally displaying the three-dimensional environment at a higher immersion level than before receiving the second input. Thus, in some embodiments, in response to the second input, the simulated environment is redisplayed and / or recentered to the user's first viewpoint in the same or similar manner as described above with respect to redisplaying and / or recentering the simulated environment in response to the first input. Redisplaying the simulated environment in response to the second input reduces the number of inputs required to view the simulated environment in a three-dimensional environment.

[0186] In some embodiments, in response to receiving the second input, the electronic device maintains the first spatial arrangement of the first virtual object relative to the user's first viewpoint in the three-dimensional environment, such as if Figure 7F Objects 706a and 708a in the Figure 7E Recentering the first virtual object at a location in the environment 702 in the three-dimensional environment (e.g., the first virtual object is not moved or reoriented in the three-dimensional environment in response to the second input) (834). Recentering the first virtual object without responding to the second input reduces the number of inputs required to properly position the virtual element in the three-dimensional environment.

[0187] In some embodiments, the three-dimensional environment includes: a first set of one or more virtual objects, the spatial arrangement of the first set of one or more virtual objects relative to the first viewpoint changes in response to receiving a first input, such as Figure 7B the first viewpoint (e.g., because the virtual objects were last placed or positioned in the three-dimensional environment from a previous viewpoint of the user that is sufficiently different from the user's first viewpoint, such as described with reference to the third set of one or more criteria); and a second set of one or more virtual objects whose spatial arrangement relative to the first viewpoint does not change in response to receiving the first input, such as Figure 7B objects 706a and 708a in the three-dimensional environment (e.g., because these virtual objects were last placed or positioned in the three-dimensional environment from the first viewpoint or from a previous viewpoint of the user that was not sufficiently different from the user's first viewpoint, such as described with reference to the third set of one or more criteria) (836a).

[0188] In some embodiments, after receiving the first input (e.g., after re-centering the first group of virtual objects without re-centering the second group of virtual objects in the manner described above, and when the first group of virtual objects and the second group of virtual objects are in their resulting positions and / or orientations resulting from the first input), the computer system (e.g., 101) detects (836b) movement of the user's viewpoint in the three-dimensional environment from a first viewpoint to a second viewpoint that is different from the first viewpoint (e.g., the second viewpoint is optionally sufficiently different from the user's first viewpoint to allow re-centering) (e.g., corresponding to a change in the orientation and / or position of the user in the user's physical environment), wherein in response to detecting the movement of the user's viewpoint, the three-dimensional environment is visible from the user's second viewpoint via the display generation component, and the positions or orientations of the first group of one or more virtual objects and the second group of one or more virtual objects in the three-dimensional environment do not change, such as when the computer system 101 detects (e.g., Figure 7C After the environment 702 is displayed as in FIG, the viewpoint 726a moves away from its Figure 7C Movement of position in .

[0189] In some embodiments, when the three-dimensional environment is visible from a second viewpoint of the user via the display generation component, the computer system (e.g., 101) receives (836c) via the one or more input devices a second input corresponding to a request to update the spatial arrangement of the one or more virtual objects relative to the second viewpoint of the user to satisfy the first set of one or more criteria, such as Figure 7B The first set of one or more criteria specifies a distance range or orientation range of the one or more virtual objects relative to the user's second viewpoint, based on an input similar to or identical to the input in (e.g., a re-centering input subsequent to the previously described re-centering input).

[0190] In some embodiments, in response to receiving the second input, the position or orientation of the first group of one or more virtual objects and the second group of one or more virtual objects in the three-dimensional environment is changed so that the updated position and orientation of the first group of one or more virtual objects and the second group of one or more virtual objects meet the first set of one or more criteria relative to the user's second viewpoint, such as re-centering 706a, 708a, 710a, and 714a in response to the second input (e.g., re-centering both the first group of virtual objects and the second group of virtual objects in response to the subsequent re-centering input in one or more of the previously described manners) (836d). Thus, although two different virtual object groups and virtual object sets are optionally processed differently in response to the first re-centering input (e.g., as previously described) (e.g., one set is re-centered and the second set is not re-centered), in response to the first re-centering input, the two sets may optionally be combined and processed as a single set in a next step (e.g., according to the previously described set rules). Thus, in response to the subsequent re-centering input, the virtual objects in the combined virtual object set are optionally re-centered together, subject to the various conditions for re-centering described previously. Re-centering the group of virtual objects together in response to further re-centering input reduces the number of inputs required to properly position the virtual elements in the three-dimensional environment.

[0191] It should be understood that the specific order in which the operations in method 800 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to rearrange the operations described herein.

[0192] 9A to 9C An example is illustrated of a computer system according to some embodiments re-centering one or more virtual objects in the presence of physical or virtual obstacles.

[0193] Figure 9A The display generation component (eg, Figure 1 The three-dimensional environment 902 is visible from the user's viewpoint 926a (eg, facing the left wall of the physical environment in which the computer system 101 is located) as illustrated in the top view. Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3The image sensor 314 optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more hands of the user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays the user interface or three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0194] like Figure 9A As shown, the computer system 101 captures one or more images of the physical environment surrounding the computer system 101 (e.g., operating environment 100) (including one or more objects in the physical environment surrounding the computer system 101). In some embodiments, the computer system 101 displays a representation of the physical environment in the three-dimensional environment 902, and / or the physical environment is visible in the three-dimensional environment 902 via the display generation component 120. For example, the three-dimensional environment 902 visible via the display generation component 120 includes a representation of the physical floor and the back and side walls of the room in which the computer system 101 is located. The three-dimensional environment 902 also includes images of the physical environment surrounding the computer system 101, including one or more objects in the physical environment surrounding the computer system 101. Figure 9A Table 922a visible from viewpoint 926a in FIG (corresponding to 922b in the top view) and the table 922a visible from viewpoint 926a in FIG via display generation component 120 Figure 9A The sofa 924b (shown in a top view) is not visible from the user's viewpoint 926a.

[0195] exist Figure 9A In FIG. 1 , the three-dimensional environment 902 further includes virtual objects 906a (corresponding to the object 906b in the top view), 908a (corresponding to the object 908b in the top view), and 910a (corresponding to the object 910b in the top view) visible from the viewpoint 926a. Figure 9A Virtual objects 912b, 914b, 916b, 918b, and 920b (shown in top view) are not visible from the user's viewpoint 926a. 7A to 7F As described in and / or method 800, virtual objects 912b, 914b, 916b, 918b, and 920b are, optionally, virtual objects that were last placed or positioned in three-dimensional environment 902 from viewpoint 926b (e.g., the user's previous viewpoint). Figure 9AIn the embodiment of the present disclosure, objects 906a, 908a, 910a, 912b, 914b, 916b, 918b, and 920b are two-dimensional objects, but the examples of the present disclosure are optionally also applicable to three-dimensional objects. Virtual objects 906a, 908a, 910a, 912b, 914b, 916b, 918b, and 920b are optionally one or more of a user interface of an application (e.g., a messaging user interface or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by computer system 101 that is not included in the physical environment of computer system 101.

[0196] As reference 7A to 7F As described in and / or method 800, in some embodiments, virtual objects that were last placed or repositioned from a user's particular previous viewpoint (or multiple previous viewpoints) can be recentered to the user's new current viewpoint. However, in some cases, the positions to which those virtual objects would have been recentered in the current viewpoint may already be occupied by other (virtual or physical) objects in the current viewpoint. Therefore, the computer system 101 may need to adjust and shift the positions to which the virtual objects mentioned above would have been recentered, as discussed in more detail below and with reference to method 1000.

[0197] For example, in Figure 9A In some embodiments, in response to such re-centering input, the computer system 101 detects a re-centering input (e.g., as described in more detail with reference to method 1000). In some embodiments, in response to such re-centering input, the computer system 101 displays the following animation: the virtual object being re-centered moves to its initial target positions for re-centering, and then, if those initial target positions are already occupied by objects, moves away from those initial target positions to the final target positions, e.g., Figures 9B to 9C In some embodiments, the computer system 101 instead displays only the virtual object being recentered moving to its final target location (e.g., Figure 9C ) instead of showing the virtual object moving to its initial target position (e.g., as Figure 9B exemplified in ).

[0198] refer to Figure 9B In some embodiments, the computer system 101 responds to Figure 9A The virtual objects being recentered are shown moving to their initial target locations by the re-centering input in the image. For example, virtual objects 912a, 914a, 916a, 918a, and 920a are shown in the image. Figure 9Bis illustrated as being at its initial target position for re-centering (e.g., the position to which the object will be re-centered if not already occupied by a virtual or physical object) and / or its final target position. Figure 9A The virtual object 912a is optionally animated from its original position in the Figure 9A Move the position in Figure 9B The position in. Figure 9B The position and / or orientation of the virtual object 912a shown in is, optionally, determined by the computer system 101 in one or more of the manners described with reference to method 800 . Figure 9B The position of virtual object 912a in is optionally its final target position because that position is not occupied by another object (whether virtual or physical).

[0199] In response to Figure 9A The virtual object 920a is optionally animated from its original position in the Figure 9A Move the position in Figure 9B The position in. Figure 9B The position and / or orientation of the virtual object 920a shown in is, optionally, determined by the computer system 101 in one or more of the manners described with reference to method 800 . Figure 9B The position of virtual object 920a in is optionally its final target position because that position is not occupied by another object (whether virtual or physical).

[0200] In response to Figure 9A The virtual objects 914a, 916a and 918a are optionally animated to move from their original positions in the Figure 9A Move the position in Figure 9B The position in. Figure 9B The positions and / or orientations of the virtual objects 914a , 916a , and 918a shown in are, optionally, determined by the computer system 101 in one or more of the manners described with reference to the method 800 . Figure 9BThe positions of virtual objects 914a, 916a, and 918a in are optionally their initial target positions, rather than their final target positions, because these positions are occupied by other objects (whether virtual or physical). For example, virtual object 914a has optionally been recentered to a position within and / or behind and / or occupied by a left wall of the physical environment of computer system 101 in accordance with one or more features of method 800. Virtual object 916a has optionally been recentered to a position within and / or occupied by table 922a in accordance with one or more features of method 800. Finally, virtual object 918a has optionally been recentered to a position within and / or occupied by virtual object 910a in accordance with one or more features of method 800.

[0201] Furthermore, in some embodiments, virtual objects that were last placed or repositioned in the three-dimensional environment 902 from the current viewpoint 926a that do not overlap and / or conflict with other of those virtual objects do not move in the three-dimensional environment 902 in response to the re-centering input, such as reflected by virtual object 910a not moving in response to the re-centering input. However, in some embodiments, virtual objects that were last placed or repositioned in the three-dimensional environment 902 from the current viewpoint 926a that overlap and / or conflict with other of those virtual objects do move in the three-dimensional environment 902 in response to the re-centering input, such as reflected by virtual objects 906a and 908a. For example, in Figure 9A , virtual object 908a is obstructing virtual object 906a from viewpoint 926a. Therefore, in response to the re-centering input, computer system 101 moves virtual objects 906a and 908b separately to reduce and / or eliminate the obstruction of virtual object 908a from virtual object 906a. Additional details on how computer system 101 shifts such overlapping or conflicting virtual objects are provided with reference to method 800.

[0202] In some embodiments, in response to receiving the re-centering input and / or during movement of the virtual object in response to the re-centering input, the computer system 101 modifies the display of the virtual object to indicate that re-centering will occur, is occurring, and / or has occurred, such as Figure 9B926a or previous viewpoint 926b), but not to virtual objects that are not moving in response to the re-centering input. In some embodiments, the computer system 101 applies the above-mentioned visual modifications to virtual objects last placed or positioned in the three-dimensional environment 902 from the previous viewpoint 926b (e.g., virtual objects that are recentered to viewpoint 926a), but not to virtual objects last placed or positioned in the three-dimensional environment 902 from the current viewpoint 926a, even if such virtual objects are moving in response to the recentering input (e.g., virtual objects 906a and / or 908a).

[0203] In some embodiments, as previously mentioned, the computer system 101 shifts those virtual objects that have been recentered to an initial target position that includes another object to a final target position to reduce and / or eliminate conflicts of those recentered virtual objects with objects occupying their initial target positions, as described in more detail with reference to method 1000. The computer system 101 optionally shifts the recentered virtual objects in different ways depending on the type of object that conflicts with the recentered virtual object. For example, Figure 9B The initial target position of the virtual object 914a shown in FIG is occupied by a physical wall in the physical environment of the computer system 101. Therefore, the computer system 101 optionally moves the virtual object 914a toward the viewpoint 926a (optionally without moving it up, down, left, and / or right relative to the viewpoint 926a) to a final target position away from the physical wall, as shown in FIG. Figure 9C As shown in .

[0204] on the contrary, Figure 9B The initial target position of the virtual object 916a shown in FIG is occupied by the physical table 922a. Therefore, the computer system 101 optionally moves the virtual object 916a up, down, left, and / or right relative to the viewpoint 926a (optionally not toward the viewpoint 926a) to a final target position away from the physical table 922a, as shown in FIG. Figure 9C In some embodiments, the computer system 101 moves the virtual object in one or more of the above directions that require the least amount of movement of the virtual object to clear the conflicting object. For example, Figures 9B to 9C , the computer system 101 has moved the virtual object 916a upward to a final target position where the virtual object 916a no longer conflicts with the physical table 922a.

[0205] As a final example, Figure 9B The initial target position of virtual object 918a shown in FIG is occupied by virtual object 910a. Therefore, computer system 101 optionally moves virtual object 918a up, down, left, and / or right relative to viewpoint 926a and / or toward or away from the viewpoint to a final target position away from virtual object 910a, as shown in FIG. Figure 9C In some embodiments, the computer system 101 moves the virtual object in one or more of the above directions that require the least amount of movement of the virtual object to clear the conflicting object. For example, Figures 9B to 9C , the computer system 101 has moved the virtual object 918a to the left to a final target position where the virtual object 918a no longer conflicts with the virtual object 910a.

[0206] As mentioned above, virtual objects other than virtual objects 914a, 916a, and 918a are optionally not displayed by computer system 101. Figure 9B Move to Figure 9C The computer system 101 optionally at least partially or completely reverses the reference position in response to the virtual object reaching its final target position. Figure 9B In some embodiments, the computer system 101 at least partially or completely reverses the reference position in response to each virtual object reaching its final target position. Figure 9B Visual modification of the virtual object. Figure 9B Partial or complete reversal of the visual modification of the virtual object is optionally performed in Figure 9C This is reflected by the absence of a cross-hatching pattern in the displayed virtual objects.

[0207] Figures 10A to 10G is a flow chart illustrating a method for re-centering one or more virtual objects in the presence of physical or virtual obstacles according to some embodiments. In some embodiments, method 1000 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4In some embodiments, method 1000 is performed by a computer system, such as one or more processors 202 of computer system 101 (e.g., a computer system 101, a computer system 102, a computer system 103, a computer system 104, a computer system 105, a computer system 106, a computer system 107, a computer system 108, a computer system 109, a computer system 110, a computer system 111, a computer system 112, a computer system 113, a computer system 114, a computer system 115, a computer system 116, a computer system 117, a computer system 118, a computer system 119, a computer system 120, a computer system 121, a computer system 122, a computer system 123, a computer system 124, a computer system 125, a computer system 126, a computer system 12 Figure 1 Some operations in method 1000 may be optionally combined, and / or the order of some operations may be optionally changed.

[0208] In some embodiments, method 1000 is performed at a computer system (e.g., 101) in communication with a display generation component and one or more input devices. In some embodiments, the computer system has one or more characteristics of the computer system in method 800. In some embodiments, the display generation component has one or more characteristics of the display generation component of method 800. In some embodiments, the one or more input devices have one or more of the characteristics of the one or more input devices of method 800.

[0209] In some embodiments, when a three-dimensional environment (e.g., 902) (e.g., the three-dimensional environment optionally has one or more characteristics of the three-dimensional environment of method 800) is displayed via a display generation component from a user's first viewpoint (such as Figure 9A When the viewpoint 926a in the three-dimensional environment is visible (e.g., such as described with reference to method 800), the three-dimensional environment includes a first virtual object at a first position in the three-dimensional environment, such as Figure 9A In some embodiments, the computer system (e.g., 101) receives (1002a) via the one or more input devices a first input corresponding to a request to update the spatial arrangement of the first virtual object relative to the user's first viewpoint to satisfy a first set of one or more criteria, such as Figure 9A The first input may include one or more criteria that specify a distance range or orientation range of a virtual object relative to a first viewpoint of a user (e.g., such as described in reference method 800. The first input may optionally have one or more characteristics of the first input (e.g., a re-centering input) described in reference methods 800 and / or 1400).

[0210] In some embodiments, in response to receiving the first input (1002b), the computer system moves the first virtual object to a second location in the three-dimensional environment that satisfies the first set of one or more criteria (e.g., if an object does not already exist at that location, the computer system will move the first virtual object to the second location, such as in accordance with one or more aspects of method 800) that is not occupied by an object, such as at Figure 9B , wherein the position of object 912a is shown in the three-dimensional environment (e.g., excluding the corresponding object, virtual or physical, excluding any virtual or physical object of the corresponding type, or excluding any virtual or physical object), wherein the spatial arrangement of the second position relative to the user's first viewpoint satisfies the first set of one or more criteria (e.g., the distance and / or orientation of the second position relative to the user's first viewpoint satisfies the first one or more criteria, such as described with reference to method 800. In some embodiments, the spatial arrangement of the second position relative to the first viewpoint corresponds to (e.g., is the same as) the spatial arrangement of the first position relative to the user's previous viewpoint from which the first virtual object was last placed or moved), the computer system (e.g., 101) displays (1002c) the first virtual object at the second position in the three-dimensional environment (e.g., moves the first virtual object to the second position), such as in Figure 9C 912 a is shown in FIG. In some embodiments, the orientation of the first virtual object at the second position relative to the first viewpoint corresponds to (e.g., is the same as) the orientation of the first virtual object at the first position relative to the previous viewpoint of the user from which the first virtual object was last placed or moved when the first input was received.

[0211] In some embodiments, in response to receiving the first input (1002b), a second location in the three-dimensional environment that satisfies a first set of one or more criteria is determined to be occupied, such as in Figure 9B , wherein the computer system (e.g., 101) displays (1002d) the first virtual object at a third location in the three-dimensional environment that satisfies the first set of one or more criteria (e.g., moves the first virtual object to the third location), wherein the third location is spaced apart from the second location in the three-dimensional environment, such as at Figure 9C. In some embodiments, the orientation of the first virtual object at the third position relative to the first viewpoint corresponds to (e.g., is the same as) the orientation of the first virtual object at the first position relative to the previous viewpoint of the user from which the first virtual object was last placed or moved when the first input was received. In some embodiments, the orientation of the first virtual object at the third position relative to the first viewpoint is different from the orientation of the first virtual object at the first position relative to the previous viewpoint of the user from which the first virtual object was last placed or moved when the first input was received. In some embodiments, the spatial arrangement of the third position relative to the first viewpoint is different from the spatial arrangement of the first position relative to the previous viewpoint of the user from which the first virtual object was last placed or moved. In some embodiments, the distance and / or orientation of the third position relative to the user's first viewpoint satisfies the first one or more criteria, such as described with reference to method 800. In some embodiments, the computer system selects the third position to be sufficiently far from the second position so that the first virtual object at the third position does not occupy any volume of the three-dimensional environment that is also occupied by the corresponding object at the second position, as described in more detail below. In some embodiments, the input described with reference to method 1000 is or includes an air gesture input. Shifting the location to which the virtual object is recentered enables the computer system to automatically avoid collisions between objects in the three-dimensional environment.

[0212] In some embodiments, when the second location includes a virtual object (such as in Figure 9B 800, 1200, 1400, and / or 1600 described herein), the second position is determined to be occupied (1004). In some embodiments, the second position is determined to be occupied if the first virtual object (if displayed at the second position) would conflict with (any portion of) the virtual object. In some embodiments, the second position is determined to be occupied if the first virtual object (if displayed at the second position) would obscure (any portion of) the virtual object or would be (at least partially) obscured by the virtual object, regardless of whether the first virtual object would conflict with the virtual object. Therefore, in some embodiments, the recentered virtual object will be shifted to avoid conflict with the existing virtual object at the second position. Shifting the position to which the virtual object is recentered enables the computer system to automatically avoid conflicts between virtual objects in the three-dimensional environment.

[0213] In some embodiments, when the second location corresponds to the location of a physical object in the user's physical environment (such as in Figure 9BIn some embodiments, the second position is determined to be occupied if the first virtual object (if displayed at the second position) would conflict with (any portion of) the physical object. The physical object is optionally visible at the second position via the display generation component, and / or a representation of the physical object is displayed at the second position via the display generation component. In some embodiments, the second position is determined to be occupied if the first virtual object (if displayed at the second position) would obscure (any portion of) the physical object or would be (at least partially) obscured by the physical object, regardless of whether the first virtual object would conflict with the physical object. Therefore, in some embodiments, the recentered virtual object will be shifted to avoid conflict with the existing physical object at the second position. Shifting the position to which the virtual object is recentered enables the computer system to automatically avoid conflicts between virtual objects and physical objects in the three-dimensional environment.

[0214] In some embodiments, based on determining that the second location corresponds to a location within or behind a physical wall in the user's physical environment, such as Figure 9B , the first virtual object is closer to the user's viewpoint than the second position, such that if the first virtual object is displayed at the second position, the first virtual object will be displayed inside or behind the physical wall in the three-dimensional environment), the third position is closer to the user's first viewpoint than the second position, and the third position is in front of the physical wall relative to the user's first viewpoint (1008), such as in Figure 9C 914a is shown in FIG. 915 . In some embodiments, if the recentered virtual object collides with and / or is behind a physical wall in the three-dimensional environment, the computer system avoids the collision by shifting the position of the recentered virtual object closer to the user's viewpoint (e.g., without laterally shifting the position of the recentered virtual object relative to the user's viewpoint). In the case of other physical objects that are wall-like objects other than walls (e.g., objects that are relatively vertical relative to the user's viewpoint and have a size or area greater than a threshold size or area, such as 0.2 meters, 0.5 meters, 1 meter, 3 meters, 5 meters, or 10 meters vertically and / or horizontally, or 0.04 square meters, 0.25 square meters, 1 square meter, 9 square meters, 25 square meters, or 100 square meters), the computer system optionally additionally performs the above operations. Shifting the position to which the virtual object is recentered toward the user's viewpoint in the case of a wall reduces the number of inputs required to ensure visibility of and / or interactivity with the virtual object in the three-dimensional environment, because a lateral shift of the virtual object's position will be less likely to resolve the collision between the virtual object and the wall.

[0215] In some embodiments, in response to the first input, the second location is determined to correspond to a corresponding physical object other than a physical wall (such as in Figure 9B 922a) (e.g., the first virtual object at the second location conflicts with a table, desk, chair, or other physical object other than a wall or wall-like physical object), the third location is the same distance from the user's first viewpoint as the second location is from the user's first viewpoint, and the third location is laterally separated from the second location relative to the first viewpoint (1010), such as at Figure 9C 916a is shown in FIG. 916b. In some embodiments, if the recentered virtual object collides with a physical object other than a wall in the three-dimensional environment, the computer system avoids the collision by shifting the position of the recentered virtual object laterally (e.g., up, down, left, and / or right) relative to the user's viewpoint (e.g., without shifting the position of the recentered virtual object toward or away from the user's viewpoint). In the case of non-wall objects, shifting the position to which the virtual object is recentered laterally relative to the user's viewpoint reduces the number of inputs required to ensure visibility of and / or interactivity with the virtual object in the three-dimensional environment.

[0216] In some embodiments, when the first input is received, the three-dimensional environment also includes a second virtual object that overlaps with the first virtual object, such as Figure 9A objects 906a and 908a in the image (e.g., the first virtual object and the second virtual object at least partially collide with each other and / or the first virtual object at least partially obscures the second virtual object from the first viewpoint, or the second virtual object at least partially obscures the first virtual object from the first viewpoint) (1012a).

[0217] In some embodiments, in response to receiving the first input, the computer system (e.g., 101) separates the first virtual object and the second virtual object from each other (e.g., laterally relative to the first viewpoint and / or toward or away from the first viewpoint) (1012b) to reduce or eliminate overlap between the first virtual object and the second virtual object, such as in Figure 9C908a relative to objects 906a and 908a. In some embodiments, both virtual objects are moved to achieve the separation. In some embodiments, only one of the virtual objects is moved to achieve the separation. In some embodiments, both the first virtual object and the second virtual object are recentered in response to the first input, and in doing so, the first virtual object and the second virtual object are separated relative to each other to achieve the separation. Separating overlapping virtual objects reduces the number of inputs required to ensure visibility of and / or interactivity with the virtual objects in the three-dimensional environment.

[0218] In some embodiments, in response to the first input, and based on determining that the second position is occupied by a corresponding object (e.g., a physical object (such as a wall or non-wall object) or a virtual object) (1014a), based on determining that the amount of separation from the second position in the first direction required for the first virtual object to avoid the corresponding object at the second position is less than the amount of separation from the second position in a second direction different from the first direction required for the first virtual object to avoid the corresponding object at the second position, the third position is separated from the second position in the first direction (1014b), such as by moving object 916a from the second position in the first direction. Figure 9B 916a is shown as being shifted upward instead of downward. For example, if the amount of shift required to shift the position of the first virtual object in a first direction (e.g., to the right, left, up, down, away from the viewpoint, or toward the viewpoint, or any combination of these directions) to avoid collision or overlap of the first virtual object with the corresponding object is less than the amount of shift required to shift the position of the first virtual object in a second direction (e.g., to the right, left, up, down, away from the viewpoint, or toward the viewpoint, or any combination of these directions) to avoid collision or overlap of the first virtual object with the corresponding object, the computer system optionally shifts the position of the first virtual object in the first direction (e.g., by the smaller amount).

[0219] In some embodiments, in response to the first input, and based on determining that the second position is occupied by a corresponding object (e.g., a physical object (such as a wall or non-wall object) or a virtual object) (1014a), based on determining that the amount of separation from the second position in the second direction required for the first virtual object to avoid the corresponding object at the second position is less than the amount of separation from the second position in the first direction required for the first virtual object to avoid the corresponding object at the second position, the third position is separated from the second position in the second direction (1014c), such as if object 916a is moved from the second position to the third position. Figure 9B916a is shifted downward instead of upward, the table 922a will be avoided with less movement of object 916a. For example, if the amount of shift required to shift the position of the first virtual object in the second direction to avoid collision or overlap of the first virtual object with the corresponding object is less than the amount of shift required to shift the position of the first virtual object in the first direction to avoid collision or overlap of the first virtual object with the corresponding object, the computer system optionally shifts the position of the first virtual object in the second direction (e.g., by a smaller amount). Therefore, in some embodiments, the computer system shifts the position of the first virtual object in a direction that requires less (e.g., the least) amount of shift of the position of the first virtual object to avoid collision or overlap of the first virtual object with the corresponding object. Shifting the first virtual object in a direction that requires less shift automatically causes the computer system to appropriately place the first virtual object to avoid collision while maintaining the first virtual object closer (e.g., as close as possible) to its initial target position.

[0220] In some embodiments, displaying the first virtual object at the third position includes: displaying, via the display generation component, an animation of the representation of the first virtual object moving to the second position, and then displaying an animation (1016) of the representation of the first virtual object moving from the second position to the third position, such as object 916a moving to Figure 9B , and then object 916a moves to the position shown in Figure 9C In some embodiments, the computer system displays an animation of the first virtual object initially moving to a second position in the three-dimensional environment (e.g., the representation of the first virtual object fades out, is visually de-emphasized, dimmed, blurred, desaturated, and / or changes translucency) in response to the first input, and then displays an animation of the first virtual object moving from the second position in the three-dimensional environment to a third position (e.g., the representation of the first virtual object fades out, is visually de-emphasized, dimmed, blurred, desaturated, and / or changes translucency). In some embodiments, the first animation and the second animation occur after the first input (e.g., in response to the first input) without detecting further input. In some embodiments, when the first virtual object reaches the third position, the computer system displays the first virtual object as not faded out, no longer visually de-emphasized, brightened, less blurred, more saturated, and / or less translucent (e.g., the visual appearance that the first virtual object had when the first input was received). Displaying the animation of the first virtual object first moving to the second position and then to the third position provides feedback about the original re-centered position of the first virtual object.

[0221] In some embodiments, the third position is separated from the second position by one or more of the following distances: a distance from the user's first viewpoint, a horizontal distance relative to the user's first viewpoint, or a vertical distance relative to the user's first viewpoint (1018). For example, the computer system optionally shifts the position of the first virtual object in any direction from the second position (such as toward or away from the user's viewpoint, horizontally relative to the user's viewpoint, vertically relative to the user's viewpoint, or any combination thereof). Shifting the position of the first virtual object in the above directions reduces the number of inputs required to properly place the first virtual object in the three-dimensional environment.

[0222] In some embodiments, the first virtual object is ultimately placed or positioned at the first location (1020a) in the three-dimensional environment from a second viewpoint of the user that is different from the user's first viewpoint (e.g., such as from a viewpoint sufficiently different from the user's current viewpoint, as described in more detail with reference to method 800).

[0223] In some embodiments, based on determining that the spatial arrangement of the first position relative to the second viewpoint is a first spatial arrangement, the second position is a first corresponding position (1020b), such as Figure 9C 920a in the object. For example, if the position and / or orientation of the first virtual object relative to the second viewpoint (e.g., the viewpoint from which the first virtual object was last placed or positioned in the three-dimensional environment) is such that the first virtual object is to the right and above relative to the second viewpoint, the computer system selects the second position such that the position and / or orientation of the first virtual object at the second position relative to the first viewpoint is also to the right and above relative to the first viewpoint (e.g., the same relative position and / or orientation). In some embodiments, the magnitude of the relative position and / or orientation of the second position relative to the first viewpoint is also maintained relative to the relative position and / or orientation of the first position relative to the second viewpoint.

[0224] In some embodiments, based on determining that the spatial arrangement of the first position relative to the second viewpoint is a second spatial arrangement different from the first spatial arrangement, the second position is a second corresponding position different from the first corresponding position (1020c), such as if the object 920a has a relative Figure 9A If the viewpoint 926b in FIG. 1 is different from the viewpoint 926b in FIG. 1 , the object 920a will optionally have a relative Figure 9C926a in the different spatial arrangement of the viewpoint 926a in. For example, if the position and / or orientation of the first virtual object relative to the second viewpoint is such that the first virtual object is to the left and below relative to the second viewpoint, the computer system selects the second position so that the position and / or orientation of the first virtual object at the second position relative to the first viewpoint is also to the left and below relative to the first viewpoint (e.g., the same relative position and / or orientation). In some embodiments, the magnitude of the position and / or orientation of the second position relative to the first viewpoint is also maintained relative to the relative position and / or orientation of the first position relative to the second viewpoint. When the virtual object is last positioned in the three-dimensional environment, a target position is set for the re-centered virtual object based on the position of the virtual object relative to the user's previous viewpoint, so that the computer system automatically places the virtual object at the previously provided relative position for the virtual object.

[0225] In some embodiments, the first virtual object is last placed or positioned in the three-dimensional environment from a second viewpoint of the user that is different from the first viewpoint (e.g., such as from a viewpoint sufficiently different from the user's current viewpoint, as described in more detail with reference to method 800), and when the first input is received, the three-dimensional environment also includes a second virtual object and a third virtual object that are last placed or positioned in the three-dimensional environment from the user's first viewpoint (or from a viewpoint that is not sufficiently different from the user's current viewpoint, as described in more detail with reference to method 800), the second virtual object and the third virtual object having a first respective spatial arrangement (1022a) relative to the first viewpoint, such as Figure 9A Objects 906a, 908a and / or 910a in and their relative Figure 9A The spatial arrangement of viewpoint 926a in .

[0226] In some embodiments, in response to receiving the first input (1022b), based on determining that the second virtual object and the third virtual object overlap, such as in Figure 9A If objects 906a and 908a overlap (e.g., at least partially collide with each other in the three-dimensional environment and / or at least partially obscure each other from the user's first viewpoint), the computer system (e.g., 101) updates (1022c) the spatial arrangement of the second virtual object and the third virtual object to a second corresponding spatial arrangement relative to the first viewpoint to reduce or eliminate the overlap between the second virtual object and the third virtual object, such as in Figure 9B and 9C 906a and 908a (e.g., moving and / or changing the orientation of the first virtual object, the second virtual object, or both the first virtual object and the second virtual object such that a distance (e.g., horizontal, vertical, and / or depth) between the objects relative to the first viewpoint increases to reduce or eliminate conflict between the two objects and / or occlusion of the two objects).

[0227] In some embodiments, in response to receiving the first input (1022b), the second virtual object and the third virtual object are determined not to overlap, such as if Figure 9A In the example embodiment, in which objects 906a and 908a do not overlap (e.g., at least partially conflict with each other in the three-dimensional environment and / or at least partially obscure each other from the user's first viewpoint), the computer system (e.g., 101) maintains (1022d) the second virtual object and the third virtual object having a first respective spatial arrangement relative to the first viewpoint, such as in response to Figure 9A The user may also move objects 906a and / or 908a in response to the first input (e.g., without moving or changing the orientation of the first and second virtual objects in the three-dimensional environment). Thus, in some embodiments, the virtual object that was last placed or positioned in the three-dimensional environment from the user's current viewpoint does not respond to the first input unless the virtual objects overlap in the three-dimensional environment. Shifting the first and / or second virtual objects only if they overlap reduces the number of inputs required to properly place the first and second virtual objects in the three-dimensional environment.

[0228] In some embodiments, in response to receiving the first input, the computer system (e.g., 101) displays (1024) via the display generation component a visual indication indicating that the first input was received, such as from Figures 9A to 9B In some embodiments, the visual indication is or includes a modification of the visual appearance of objects 906a, 908a, and / or 910a of the computer system when the first input is received. In some embodiments, after receiving the first input, the visual indication is displayed for a predetermined amount of time (e.g., 0.3 seconds, 0.5 seconds, 1 second, 2 seconds, 3 seconds, 5 seconds, or 10 seconds). In some embodiments, the visual indication is displayed for the duration of the movement of the virtual object in the three-dimensional environment in response to the first input, and the visual indication is stopped from being displayed in response to the end of the movement. In some embodiments, the visual indication is or includes a modification of the visual appearance of one or more elements included in the three-dimensional environment when the first input is received (e.g., a modification of the visual appearance of one or more virtual objects included in the three-dimensional environment when the first input is received, as will be described in more detail below). In some embodiments, the visual indication is or includes the display of an element (e.g., a notification) that was not displayed or was not included in the three-dimensional environment when the first input was received. Displaying an indication of the first input provides feedback on the current state of the computer system when re-centering one or more virtual objects in the three-dimensional environment.

[0229] In some embodiments, when the first input is received, the first virtual object has a visual characteristic having a first value (e.g., having a first brightness, having a first opacity, having a first blurriness, and / or having a first color saturation) (e.g., is displayed with the visual characteristic), and the visual indication indicating that the first input has been received includes temporarily updating (and / or displaying) the first virtual object to have a visual characteristic having a second value different from the first value, such as Figure 9B The first virtual object 916a is then displayed with a visual characteristic having a first value, such as Figure 9C In some embodiments, the first virtual object is temporarily de-emphasized in the three-dimensional environment in response to the first input (e.g., relative to the rest of the three-dimensional environment and / or relative to portions of the three-dimensional environment that do not change position and / or orientation in response to the first input). In some embodiments, the change in the visual appearance of the first virtual object is maintained for the duration of the movement of the virtual object in the three-dimensional environment in response to the first input and is restored in response to the end of the movement. In some embodiments, the change in visual appearance is additionally or alternatively applied to other virtual objects that move / reorient in the three-dimensional environment in response to the first input. In some embodiments, the change in visual appearance is additionally or alternatively applied to virtual objects that do not move / reorient in the three-dimensional environment in response to the first input. In some embodiments, the virtual objects whose visual appearance changes partially or completely fade out in the three-dimensional environment in response to the first input until they no longer fade out, as described above. Adjusting the visual appearance of the virtual objects in response to the first input provides feedback regarding the current state of the computer system when re-centering one or more virtual objects in the three-dimensional environment.

[0230] In some embodiments, when the first input is received, the three-dimensional environment also includes a second virtual object at a fourth location in the three-dimensional environment (e.g., the second virtual object is an object that will be recentered in the three-dimensional environment along with the first virtual object in response to the first input), and the first virtual object and the second virtual object have a first corresponding spatial arrangement (1028a) relative to each other, such as Figure 9A Objects 912a and 920a in have a spatial arrangement relative to each other.

[0231] In some embodiments, in response to receiving the first input (1028b), based on determining that the second position is not occupied by an object, the computer system (e.g., 101) displays (1028c) the first virtual object at the second position and displays the second virtual object at a fifth position that is different from the fourth position and satisfies the first set of one or more criteria (e.g., moving and / or reorienting both the first virtual object and the second virtual object in the three-dimensional environment in response to the first input, as previously described and / or as described with reference to method 800), wherein the first virtual object at the second position and the second virtual object at the fifth position have a first corresponding spatial arrangement relative to each other, such as objects 912a and 920a at Figure 9C relative to each other and in Figure 9A The same spatial arrangement in the image (e.g., maintaining the relative orientation and / or positioning of the first virtual object and the second virtual object in response to re-centering those virtual objects, as described in more detail with reference to method 800).

[0232] In some embodiments, in response to receiving the first input (1028b), the second position is determined to be occupied, such as relative to Figure 9B 800, except that the computer system has shifted the positions of the first virtual object and, optionally, the second virtual object because the target positions of the first virtual object and, optionally, the second virtual object are occupied by other objects, as previously described), wherein the first virtual object at the third position and the second virtual object at the sixth position each have a second corresponding spatial arrangement relative to each other that is different from the first corresponding spatial arrangement, such as objects 918a and 920a at the sixth position that satisfies the first set of one or more criteria. Figure 9C relative to each other and in Figure 9A In one embodiment, the present invention provides a method for maintaining a different spatial arrangement of virtual objects in a different spatial arrangement in response to the first input (e.g., if the target location of the virtual object is occupied when the first input is received, the virtual objects optionally do not maintain their relative orientation and / or position in response to re-centering the virtual objects). Maintaining the relative spatial arrangement of the re-centered virtual objects, if possible, reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0233] In some embodiments, when the first input is received, the three-dimensional environment includes a first corresponding virtual object (e.g., the first virtual object or a different virtual object) at a first corresponding position in the three-dimensional environment and a second corresponding virtual object at a second corresponding position in the three-dimensional environment (e.g., the first corresponding virtual object is being recentered in response to the first input, and optionally the second corresponding virtual object is being recentered in response to the first input, or optionally the second corresponding virtual object is not being recentered in response to the first input) (1030a).

[0234] In some embodiments, in response to receiving the first input (1030b), the computer system (e.g., 101) displays (1030c) the second corresponding virtual object at a third corresponding position in the three-dimensional environment (e.g., if the second corresponding virtual object is recentered in response to the first input, the third corresponding position is different from the second corresponding position, or if the second corresponding virtual object is not recentered in response to the first input, the third corresponding position is the same as the second corresponding position).

[0235] In some embodiments, in response to receiving the first input (1030b), based on determining that the distance difference between the distance between the fourth corresponding position and the user's first viewpoint and the distance between the third corresponding position and the user's first viewpoint is greater than a threshold distance (e.g., the difference in distance from the user's first viewpoint: 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, 100, 500, 1000, or 5000 cm), wherein the fourth corresponding position is farther from the user's first viewpoint than the third corresponding position and satisfies one or more of the first set of Based on a criterion (e.g., the fourth corresponding position is an initial target position of the first corresponding virtual object generated in response to the first input in the manner described above and / or with reference to method 800), the computer system (e.g., 101) displays (1030d) the first corresponding virtual object at the fourth corresponding position, wherein the second corresponding virtual object at the third corresponding position at least partially obscures the first corresponding virtual object at the fourth corresponding position from the user's first viewpoint, such as if object 918a is recentered to and remains behind object 910a and in Figure 9B0a in the three-dimensional environment because the position behind object 910a is separated from object 910a by at least the threshold distance (and optionally, the first and second corresponding virtual objects do not conflict in the three-dimensional environment when displayed at the fourth and third corresponding positions, respectively). For example, even if the second corresponding virtual object at least partially obscures the first corresponding virtual object from the user's first viewpoint, the computer system re-centers the first corresponding virtual object to the fourth corresponding position. Thus, in some embodiments, if virtual objects would conflict with other virtual objects, the computer system will shift the target positions of those virtual objects in response to the first input, but if the two objects are sufficiently separated from each other in depth relative to the user's viewpoint, the computer system will not move the target positions of those virtual objects in response to the first input based on the virtual objects obscuring other virtual objects (but not conflicting with the other virtual objects) (or vice versa) from the user's viewpoint.

[0236] In some embodiments, in response to receiving the first input (1030b), based on determining that the difference in distance between the fourth corresponding position and the distance between the third corresponding position and the user's first viewpoint is less than a threshold distance (e.g., the difference in distance from the user's first viewpoint is 0.1, 0.3, 0.5, 1, 3, 5, 10, 20, 50, 100, 500, 1000, or 5000 cm), the computer system (e.g., 101) displays (1030e) the first corresponding virtual object at a fifth corresponding position that is different from the fourth corresponding position, wherein the fifth corresponding position is farther away from the user's first viewpoint than the third corresponding position and satisfies a first set of one or more criteria (e.g., the fifth corresponding position is a shifted target position of the first corresponding virtual object generated in response to the first input in the manner described above and / or with reference to method 800), and from the user's first viewpoint, the second corresponding virtual object at the third corresponding position does not at least partially obscure the first corresponding virtual object at the fifth corresponding position, such as if object 918a is recentered to Figure 9B, the first corresponding virtual object is moved to a position behind object 910a, but the position behind object 910a is not separated from object 910a by at least the threshold distance, so the computer system 101 changes the position of object 918a so that it is not obscured by object 910a (and optionally, the first corresponding virtual object and the second corresponding virtual object do not conflict in the three-dimensional environment when displayed at the fifth corresponding position and the third corresponding position, respectively). For example, the computer system re-centers the first corresponding virtual object to the fifth corresponding position, which is selected by the computer system so that from the user's point of view, the second corresponding virtual object does not even partially obscure the first corresponding virtual object. Therefore, in some embodiments, if a virtual object would conflict with other virtual objects and / or would obscure other virtual objects (or vice versa) if the two objects are not sufficiently separated from each other in depth relative to the user's point of view, the computer system will shift the target positions of those virtual objects in response to the first input. Shifting the recenter position based on collision or line of sight obstruction according to the separation (in depth) of the virtual objects in response to recentering reduces the number of inputs required to properly place the objects relative to the user's viewpoint in response to the first input.

[0237] It should be understood that the specific order in which the operations in method 1000 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to rearrange the operations described herein.

[0238] Figures 11A to 11E An example is illustrated of a computer system selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments.

[0239] Figure 11A The display generation component (eg, Figure 1 The three-dimensional environment 1102 is visible from the user's viewpoint 1126 (e.g., facing the left wall of the first room 1103a in the physical environment in which the computer system 101 is located) as illustrated in the top view. Figures 1 to 6 As described above, the computer system 101 optionally includes a display generation component (e.g., a touch screen) and a plurality of image sensors (e.g., Figure 3The image sensor 314 optionally includes one or more of the following: a visible light camera; an infrared camera; a depth sensor; or any other sensor that the computer system 101 can use to capture one or more images of the user or a portion of the user (e.g., one or more hands of the user) as the user interacts with the computer system 101. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays the user interface or three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0240] like Figure 11A As shown, the computer system 101 captures one or more images of the physical environment (e.g., operating environment 100) surrounding the computer system 101 (including one or more objects in the physical environment surrounding the computer system 101). In some embodiments, the computer system 101 displays a representation of the physical environment in a three-dimensional environment 1102, and / or the physical environment is visible in the three-dimensional environment 1102 via the display generation component 120. For example, the three-dimensional environment 1102 visible via the display generation component 120 includes a representation of the physical floor and the back and side walls of the room 1103a in which the computer system 101 is located. The three-dimensional environment 1102 also includes a representation of the physical environment viewed from the display generation component 120. Figure 11A Table 1122a (corresponding to 1122b in the top view) visible from viewpoint 1126 in FIG. 1 and table 1122b visible from viewpoint 1126 in FIG. 1 via display generation component 120 Figure 11A A sofa 1124b (shown in a top view) in a second room 1103b in the physical environment is not visible from the user's viewpoint 1126.

[0241] exist Figure 11A , the three-dimensional environment 1102 also includes virtual objects 1106a (corresponding to the object 1106b in the top view), 1108a (corresponding to the object 1108b in the top view), and 1110a (corresponding to the object 1110b in the top view) visible from the viewpoint 1126. 7A to 7F and / or method 800, virtual objects 1106a, 1108a, and 1110a are optionally derived from Figure 11A The viewpoint 1126 in is finally placed or positioned in the three-dimensional environment 1102. Figure 11AIn the example of FIG100 , objects 1106 a, 1108 a, and 1110 a are two-dimensional objects, but the examples of the present disclosure are optionally also applicable to three-dimensional objects. Virtual objects 1106 a, 1108 a, and 1110 a are optionally one or more of a user interface of an application (e.g., a messaging user interface or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by computer system 101 that is not included in the physical environment of computer system 101.

[0242] As reference 7A to 7F As described in and / or method 800, in some embodiments, a virtual object that was last placed or repositioned from a particular previous viewpoint (or multiple previous viewpoints) of the user can be recentered to the user's new current viewpoint. Thus, in some embodiments, if the user's viewpoint changes from Figure 11A If the viewpoint illustrated in FIG. 1 changes and the computer system 101 detects a re-centering input, the computer system 101 re-centers the virtual objects 1106a, 1108a, and 1110a to the changed viewpoint, as shown in FIG. 7A to 7F and / or method 800. However, in some embodiments, if the user's changed viewpoint is sufficiently different from the user's previous viewpoint (e.g., in position and / or orientation, such as described in more detail with reference to method 1200), the computer system 101 automatically re-centers the virtual objects 1106a, 1108a, and 1110a to the user's changed viewpoint. Furthermore, in some embodiments, the computer system 101 performs (or does not perform) such automatic re-centering in response to the display generation component 120 transitioning from the second state (e.g., a powered-off or turned-off state in which the three-dimensional environment 1102 is not visible via the display generation component 120) to the first state (e.g., a powered-on or turned-on state in which the three-dimensional environment 1102 is visible via the display generation component 120), as will be discussed in more detail below and with reference to method 1200. In the case of a wearable device (e.g., a head-mounted device), the display generation component is optionally in a first state when the device is being worn on a user's head, the display generation component optionally transitions to a second state in response to detecting that the device has been removed from the user's head, and the display generation component optionally transitions back to the first state in response to detecting that the device has been placed on the user's head and is being worn on the user's head (and optionally remains in the first state upon detecting that the device has been placed on the user's head and is being worn on the user's head).

[0243] For example, from Figures 11A to 11B , the display generation component 120 has changed from the first state to the second state, and as Figure 11AThe user has moved to a new location (e.g., a new position and / or a new orientation) in the user's physical environment compared to Figure 11B , the user has moved to a new location in the first room 1103a in the physical environment, corresponding to a new viewpoint 1126, and is facing the left rear wall of the room 1103a. The three-dimensional environment 1102 is not visible or displayed via the computer system 101 because the computer system 101 is optionally turned off and / or not worn on the user's head. Figure 11B No virtual objects are instantiated in .

[0244] from Figures 11B to 11C , when the user is Figure 11B (and Figure 11C ) in the physical environment, the display generation component 120 has transitioned from the second state to the first state. Figure 11C As shown in , the three-dimensional environment 1102 is again visible via the display generation component 120 of the computer system 101. In addition, the user's viewpoint in the three-dimensional environment 1102 corresponds to the user's updated position and / or orientation in the physical environment. Figure 11C The updated position and / or orientation of the user in the physical environment and / or in Figure 11B and Figure 11C The user's viewpoint in the three-dimensional environment 1102 is optionally not the same as that in Figure 11A , so that the computer system 101 does not automatically re-center the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint in response to the display generation component 120 transitioning from the second state to the first state. For example, because the user remains in the physical environment and in Figure 11A 103a in the same room 1103a, so the computer system 101 optionally does not automatically re-center the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint. Additional or alternative criteria for automatically re-centering the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint are described with reference to method 1200. Figure 11C In the three-dimensional environment 1102, optionally only from Figure 11A The different viewpoints in the Figure 11C Different viewpoints in .

[0245] and Figure 11B and Figure 11C On the contrary, Figure 11D In the display generation component 120 has changed from the first state to the second state, and as with Figure 11A or Figure 11CThe user has moved to a new location (e.g., a new position and / or a new orientation) in the user's physical environment compared to Figure 11D , the user has moved to a new location in the second room 1103b in the physical environment, corresponding to a new viewpoint 1126, and is facing the back wall of the room 1103b. The three-dimensional environment 1102 is not visible or displayed via the computer system 101 because the computer system 101 is optionally turned off and / or not worn on the user's head. Figure 11D No virtual objects are instantiated in .

[0246] from Figures 11D to 11E , when the user is Figure 11D (and Figure 11E ) in the physical environment, the display generation component 120 has transitioned from the second state to the first state. Figure 11E As shown in , the three-dimensional environment 1102 is again visible via the display generation component 120 of the computer system 101. In addition, the user's viewpoint in the three-dimensional environment 1102 corresponds to the user's updated position and / or orientation in the physical environment. Figure 11E The updated position and / or orientation of the user in the physical environment and / or in Figure 11D and Figure 11E The user's viewpoint in the three-dimensional environment 1102 is optionally aligned with the Figure 11A (and / or Figure 11C ) are sufficiently different in the second state, so the computer system 101 automatically re-centers the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint in response to the display generation component 120 transitioning from the second state to the first state. For example, because the user has moved to the second room 1103b in the physical environment, the computer system 101 optionally has automatically re-centered the virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint, as shown in FIG. Figure 11E Details regarding how virtual objects 1106a, 1108a, and 1110a are re-centered to the user's updated viewpoint are provided with reference to methods 800 and / or 1000. Additional or alternative criteria for automatically re-centering virtual objects 1106a, 1108a, and 1110a to the user's updated viewpoint are described with reference to method 1200. Figure 11E , the three-dimensional environment 1102 is optionally recentered to coincide with the Figure 11A (and / or Figure 11C ) in a different viewpoint and visible from that viewpoint.

[0247] In some embodiments, the computer system 101 does not automatically re-center the virtual object and / or three-dimensional environment to the user's updated viewpoint unless the display generation component transitions from the second state to the first state while the user is in the updated position and / or viewpoint (optionally after having transitioned from the first state to the second state). For example, if the user has transitioned from Figure 11A The position and / or viewpoint shown in the example is moved to Figure 11E 106a, 1108a, and 1110a to the user's updated viewpoint, and instead, the three-dimensional environment 1102 will optionally be visible only from the user's updated viewpoint, while the virtual objects 1106a, 1108a, and 1110a remain in their original positions. Figure 11A 1102. Thus, in some embodiments, the required condition for automatically re-centering the three-dimensional environment and / or virtual objects to the user's updated viewpoint is that the display generation component transitions from the second state to the first state when the user is at a position and / or viewpoint that satisfies the automatic re-centering criteria (e.g., sufficiently different from the user's previous viewpoint, as described in more detail with reference to method 1200).

[0248] 12A to 12E is a flowchart illustrating a method for selectively and automatically re-centering one or more virtual objects in response to a display generation component changing state according to some embodiments. In some embodiments, method 1200 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4 In some embodiments, the method 1200 is performed by a display generating component 120 in the computer system, such as a head-up display, a display, a touch screen, or a projector, and one or more cameras, such as a camera pointing downward toward the user's hand (e.g., a color sensor, an infrared sensor, or other depth sensing camera) or a camera pointing forward from the user's head. In some embodiments, the method 1200 is performed by a computer system stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., Figure 1 Some operations in method 1200 may be optionally combined, and / or the order of some operations may be optionally changed.

[0249] In some embodiments, method 1200 is performed at a computer system (e.g., 101) in communication with a display generation component and one or more input devices. In some embodiments, the computer system has one or more characteristics of the computer system of methods 800 and / or 1000. In some embodiments, the display generation component has one or more characteristics of the display generation component of methods 800 and / or 1000. In some embodiments, the one or more input devices have one or more of the characteristics of the one or more input devices of methods 800 and / or 1000.

[0250] In some embodiments, when a display generation component operates (e.g., such as described with reference to methods 800 and / or 1000) in a first state (e.g., a state in which the display generation component is active and / or turned on) in which a three-dimensional environment (e.g., 1102) (e.g., the three-dimensional environment optionally has one or more characteristics of the three-dimensional environment of methods 800 and / or 1000, and optionally includes at least a portion of the physical environment of a user of the computer system. In some embodiments, (the portion of) the physical environment is displayed in the three-dimensional environment (e.g., virtual or video pass-through) via the display generation component. In some embodiments, (the portion of) the physical environment is a view of (the portion of) the physical environment of the computer system visible through a transparent portion of the display generation component (e.g., real or real pass-through)) is visible from a user's viewpoint and the user's first viewpoint is associated with a first corresponding spatial arrangement of the three-dimensional environment relative to the user, such as in Figure 11A In one embodiment, the computer system (e.g., 101) displays (1202a) in the three-dimensional environment via the display generation component a first virtual object having a first spatial arrangement relative to the first viewpoint of the user and a second spatial arrangement relative to the three-dimensional environment, such as a first virtual object having a first spatial arrangement relative to the first viewpoint of the user and a second spatial arrangement relative to the three-dimensional environment, in the three-dimensional environment (e.g., the viewpoint from which the three-dimensional environment is displayed and / or visible corresponds to the position and / or orientation of the user in the three-dimensional environment and / or the user's physical environment, such that if the user were to rotate their head and / or torso and / or move in the three-dimensional environment and / or its physical environment, corresponding different portions of the three-dimensional environment would be displayed and / or visible via the display generation component). Figure 11A106a, 1108a, and / or 1110a in the three-dimensional environment. The first virtual object optionally has one or more characteristics of the first virtual object of method 800 and / or 1000. The first spatial arrangement optionally corresponds to a relative position and / or relative orientation (optionally including the orientation of the first virtual object itself) of the first virtual object relative to a first viewpoint of the user in the three-dimensional environment (e.g., 10 feet from the first viewpoint and 30 degrees to the right relative to a centerline of the first viewpoint). The second spatial arrangement optionally corresponds to a relative position and / or relative orientation (optionally including the orientation of the first virtual object itself) of the first object relative to a reference point in the three-dimensional environment and / or the user's physical environment (e.g., the user's position in the physical environment, the orientation of the user's head and / or torso in the three-dimensional environment, the center of the room in which the user is located, or the position of the user's viewpoint in the three-dimensional environment) (e.g., 10 feet from the center of the room and 30 degrees to the right relative to a line from the center of the room to the back wall of the room, and perpendicular to the back wall of the room). Thus, in some embodiments, a first virtual object having a relative position relative to the user's viewpoint in the three-dimensional environment also has a relative position relative to the physical environment, optionally visible via the display generation component. In some embodiments, the first spatial arrangement satisfies the one or more criteria of methods 800 and / or 1000, which specify a range of distances or orientations of the virtual object relative to the user's viewpoint. In some embodiments, the first spatial arrangement does not satisfy those one or more criteria of methods 800 and / or 1000.

[0251] In some embodiments, when displaying a first virtual object in a first spatial arrangement relative to a first viewpoint of a user and in a second spatial arrangement relative to a three-dimensional environment, a computer system (e.g., 101) detects (1202b) a first event corresponding to a change in a state of a display generation component to a second state different from the first state (e.g., a state in which the display generation component is inactive or turned off), wherein when the display generation component is in the second state, the three-dimensional environment is not visible via the display generation component, such as from Figures 11A to 11BTurn off the computer system 101. For example, the second state is optionally activated in response to an input (e.g., a first event) detected by the computer system (e.g., a selection of a displayed selectable option, or a selection of a hardware button included on the computer system) to stop displaying and / or exit the three-dimensional environment. In some embodiments, the display generation component is included in a head-mounted device worn on the user's head, and when worn on the user's head, the display generation component is in the first state and the user is able to view the three-dimensional environment visible via the display generation component. In some embodiments, in response to detecting that the head-mounted device has been removed from the user's head (e.g., no longer worn by the user) (e.g., the first event), the computer system transitions the display generation component to the second state.

[0252] In some embodiments, after a change in the state of the display generation component from a first state to a second state (e.g., after the user has changed orientation and / or moved to a different location in the physical environment after the change in the state of the display generation component from the first state to the second state) (1202c), the computer system (e.g., 101) detects (1202d) a second event corresponding to the change in the state of the display generation component from the second state to the first state in which the three-dimensional environment is visible via the display generation component, wherein when the display generation component is in the first state after detecting the second event, the three-dimensional environment is visible via the display generation component from a second viewpoint of the user that is different from the first viewpoint (e.g., corresponding to a changed orientation and / or position of the user in the physical environment), wherein the second viewpoint is associated with a second corresponding spatial arrangement of the user relative to the three-dimensional environment, such as Figure 11C 1126 in the viewpoint. For example, the second event is optionally an input detected by the computer system to redisplay and / or enter the three-dimensional environment (e.g., a selection of a displayed selectable option, or a selection of a hardware button included on the computer system). In some embodiments, the second event is a detection that the head mounted device has been placed on the user's head (e.g., worn by the user again). For example, the computer system now displays the three-dimensional environment from the user's updated viewpoint (e.g., with an updated position and / or orientation in the three-dimensional environment corresponding to the user's new position and / or orientation in the user's physical environment).

[0253] In some embodiments, after a change in the state of a display generation component from a first state to a second state (e.g., after the user has changed orientation and / or moved to a different location in the physical environment after the change in the state of the display generation component from the first state to the second state) (1202c), in response to detecting a second event and when the three-dimensional environment is visible from a second viewpoint (e.g., the computer system transitions the display generation component to the first state in response to detecting the second event), the computer system displays a first virtual object in the three-dimensional environment via the display generation component (1202e), including displaying (1202f) the first virtual object in the three-dimensional environment in a first spatial arrangement relative to the user's second viewpoint and in a third spatial arrangement relative to the three-dimensional environment that is different from the second spatial arrangement, such as relative to the user, based on determining that one or more criteria are met (e.g., one or more criteria for re-centering the three-dimensional environment (including the first virtual object) to the user's updated viewpoint (such as described in references 800 and / or 1000). The one or more criteria will be described in more detail below.). Figure 11E For example, because the one or more criteria are met, the computer system displays the first virtual object as being in a relative position and / or orientation relative to the second viewpoint (e.g., a different position and / or orientation in the three-dimensional environment than before), which is the same position and / or orientation as the first virtual object was displayed relative to the first viewpoint from which the three-dimensional environment was last displayed. In some embodiments, because the user is now in a different orientation and / or position in the three-dimensional environment and / or physical environment (e.g., the second viewpoint corresponds to a different orientation and / or position), and because the first virtual object was displayed in the same first spatial arrangement relative to the second viewpoint as before, the first virtual object is now displayed in a different spatial arrangement relative to the three-dimensional environment and / or physical environment than before (e.g., the first virtual object is no longer displayed above a physical table in the physical environment, but is now displayed above a physical sofa in the physical environment).

[0254] In some embodiments, after a change in the state of a display generation component from a first state to a second state (e.g., after the user has changed orientation and / or moved to a different location in the physical environment after the change in the state of the display generation component from the first state to the second state) (1202c), in response to detecting a second event and when the three-dimensional environment is visible from a second viewpoint (e.g., the computer system transitions the display generation component to the first state in response to detecting the second event), the computer system displays a first virtual object in the three-dimensional environment via the display generation component (1202e), including: based on determining that the one or more criteria are not met, displaying (1202g) the first virtual object in the three-dimensional environment in a fourth spatial arrangement different from the first spatial arrangement relative to the user's second viewpoint and in the second spatial arrangement relative to the three-dimensional environment, such as relative to Figure 11C For example, because the one or more criteria are not met, the computer system displays the first virtual object as being in a relative position and / or orientation relative to the second viewpoint (e.g., the same position and / or orientation in the three-dimensional environment as before) that is different from when the first virtual object was displayed relative to the first viewpoint from which the three-dimensional environment was last displayed. Therefore, because the first virtual object is not repositioned in the three-dimensional environment, the first virtual object is optionally displayed in the same second spatial arrangement relative to the three-dimensional environment and / or physical environment as before (e.g., the first virtual object is still displayed above the physical table in the physical environment). In some embodiments, the input described with reference to method 1200 is or includes an air gesture input. Selectively re-centering the object based on the user's updated viewpoint reduces the number of inputs required to enable the user to access the object when initiating display of the three-dimensional environment.

[0255] In some embodiments, the one or more criteria are met when the duration between the first event and the second event is greater than a time threshold (e.g., 5 minutes, 30 minutes, 1 hour, 3 hours, 6 hours, 12 hours, 24 hours, 48 ​​hours, 96 hours, or 192 hours), such as in Figure 11A and Figure 11D / E, and the one or more criteria are not met when the duration between the first event and the second event is less than the time threshold (1204), such as when Figure 11A and Figure 11B / C. For example, if the time since the first event was detected is less than the time threshold, the computer system optionally does not automatically re-center the three-dimensional environment to the user's new viewpoint in response to detecting the second event, and if the time since the first event was detected is greater than the time threshold, the computer system optionally automatically re-centers the three-dimensional environment to the user's new viewpoint in response to detecting the second event. Selectively re-centering objects to the user's updated viewpoint based on time enables re-centering to be performed when appropriate without displaying additional controls.

[0256] In some embodiments, the one or more criteria are met when the distance between the user's second viewpoint and the user's first viewpoint in the three-dimensional environment is greater than a threshold distance (e.g., 0.1 meters, 0.5 meters, 1 meter, 3 meters, 5 meters, 10 meters, 20 meters, 50 meters, 100 meters, or 300 meters), such as in Figure 11A and Figure 11D / E, and the one or more criteria are not satisfied when the distance between the user's second viewpoint and the user's first viewpoint in the three-dimensional environment is less than a threshold distance (1206), such as when Figure 11A and Figure 11B / C. For example, when a second event is detected while the user has moved more than a threshold distance from the location where the first event was detected in the user's physical environment, the computer system optionally automatically re-centers the three-dimensional environment to the user's new viewpoint in response to detecting the second event. On the other hand, when the second event is detected while the user has not yet moved more than a threshold distance from the location where the first event was detected in the user's physical environment, the computer system optionally does not automatically re-center the three-dimensional environment to the user's new viewpoint in response to detecting the second event. Selectively re-centering objects to the user's updated viewpoint based on distance enables re-centering to be performed when appropriate without displaying additional controls.

[0257] In some embodiments, the one or more criteria are met when the orientation difference between the orientation of the user's first viewpoint and the orientation of the user's second viewpoint in the three-dimensional environment is greater than a threshold (e.g., the orientation of the second viewpoint is rotated more than 5 degrees, 10 degrees, 20 degrees, 30 degrees, 45 degrees, 90 degrees, 120 degrees, or 150 degrees relative to the orientation of the first viewpoint). Figure 11A and Figure 11D / E, and the one or more criteria (1208) are not satisfied when the orientation difference between the orientation of the user's first viewpoint and the orientation of the user's second viewpoint in the three-dimensional environment is less than a threshold, such as in Figure 11A and Figure 11B / C. For example, when a second event is detected where the user's orientation (e.g., the user's head, body, shoulders, and / or torso) has moved and / or reoriented beyond a threshold orientation from the user's physical environment when the first event was detected, the computer system optionally automatically re-centers the three-dimensional environment to the user's new viewpoint in response to detecting the second event. On the other hand, when the second event is detected where the user's orientation (e.g., the user's head, body, shoulders, and / or torso) has not moved and / or re-oriented beyond a threshold orientation from the user's physical environment when the first event was detected, the computer system optionally does not automatically re-center the three-dimensional environment to the user's new viewpoint in response to detecting the second event. Selectively re-centering objects to the user's updated viewpoint based on orientation enables re-centering to be performed when appropriate without displaying additional controls.

[0258] In some embodiments, the one or more criteria are satisfied when the user's first viewpoint corresponds to a location within a first room in the three-dimensional environment and the user's second viewpoint corresponds to a location within a second room in the three-dimensional environment that is different from the first room (e.g., when the user's viewpoint is the first viewpoint, the user is located in a first room of the user's physical environment, and when the user's viewpoint is the second viewpoint, the user is located in a second room of the user's physical environment), such as in Figure 11A At Figure 11D / E, and the one or more criteria (1210) are not satisfied when the user's first viewpoint and the user's second viewpoint correspond to locations within the same room in the three-dimensional environment, such as in Figure 11A and Figure 11B / C. In some embodiments, the one or more criteria are additionally or alternatively satisfied when the user's position corresponding to the first viewpoint and the user's position corresponding to the second viewpoint are separated by at least one wall in the user's physical environment. For example, when the second event is detected when the user has moved to a room different from the room that includes the location in the user's physical environment where the first event was detected, the computer system optionally automatically re-centers the three-dimensional environment to the user's new viewpoint in response to detecting the second event. On the other hand, when the second event is detected when the user has not yet moved to a room different from the room that includes the location in the user's physical environment where the first event was detected, the computer system optionally does not automatically re-center the three-dimensional environment to the user's new viewpoint in response to detecting the second event. Selectively re-centering objects to the user's updated viewpoint based on the user's movement to different rooms enables re-centering to be performed when appropriate without displaying additional controls.

[0259] In some embodiments, when the three-dimensional environment is visible from the user's second viewpoint, and when the first virtual object is displayed via the display generation component in a fourth spatial arrangement relative to the user's second viewpoint and in the second spatial arrangement relative to the three-dimensional environment based on a determination that the one or more criteria are not satisfied (e.g., the three-dimensional environment is not automatically recentered to the user's second viewpoint in response to detecting the second event), such as in Figure 11C In the embodiment of the present invention, a computer system (e.g., 101) detects (1012a) via the one or more input devices an input corresponding to a request to update the spatial arrangement of the first virtual object relative to the second viewpoint of the user to satisfy the first set of one or more criteria, such as in Figure 11C Such input is detected in the method (e.g., such as described with reference to method 800), the first set of one or more criteria specifying a distance range or orientation range of the virtual object relative to the user's second viewpoint. The input optionally has one or more of the characteristics of the first input (e.g., re-centering input) described with reference to methods 800, 1000, and / or 1400).

[0260] In some embodiments, in response to detecting the input, the computer system (e.g., 101) displays (1012b) the first virtual object in the three-dimensional environment in a first spatial arrangement relative to the user's second viewpoint and in a third spatial arrangement relative to the three-dimensional environment, such as if Figure 11C China is relative to Figure 11C The spatial arrangement of viewpoints 1126 in Figure 11A 1126 in the 3D environment). Thus, in some embodiments, even if the computer system does not automatically re-center the 3D environment to the user's second viewpoint in response to detecting the second event, the user can subsequently manually re-center the 3D environment by providing input. In some embodiments, the results of re-centering in response to the second event and re-centering in response to the user input are the same. Providing manual re-centering provides an efficient way to place virtual objects at appropriate locations in the 3D environment.

[0261] In some embodiments, the input corresponding to the request to update the spatial arrangement of the first virtual object relative to the user's second viewpoint to satisfy the first set of one or more criteria includes selection of a physical button of the computer system (1014), such as a reference button. Figure 7BIn some embodiments, the display generation component is included in a device (e.g., a physical device) that includes a physical depressible button. In some embodiments, the button is also rotatable (e.g., to increase or decrease the immersion level of the computer system displaying the three-dimensional environment, as described with reference to method 800). In some embodiments, the device is a head-mounted device, such as a virtual or augmented reality headset. In some embodiments, the input is or includes pressing a button (and does not include rotating the button). Providing manual re-centering via activation of a physical button provides an efficient way to place a virtual object at an appropriate location in a three-dimensional environment.

[0262] In some embodiments, the display generation component is included in a wearable device that can be worn by the user (e.g., a head-mounted device, such as a virtual or augmented reality headset or glasses), and detecting the first event includes detecting that the user is no longer wearing the wearable device (e.g., detecting that the user has removed the head-mounted device from their head and / or detecting that the head-mounted device is no longer on the user's head) (1016). Other wearable devices, such as smart watches, are also contemplated. In some embodiments, detecting the second event includes detecting that the user has placed the head-mounted device on their head and / or detecting that the head-mounted device is again being worn by the user. Transitioning to the second state of the display generation component based on whether the user is wearing the device reduces the number of inputs required to transition to the second state.

[0263] In some embodiments, detecting the first event includes detecting an input corresponding to a request to stop the three-dimensional environment from being visible via the display generation component (1018). For example, the input is an input to close a virtual or augmented reality experience presented by the computer system. In some embodiments, the virtual or augmented reality experience is being provided by an application running on the computer system, and the input is an input to close the application. In some embodiments, the input is an input to exit full-screen mode of the virtual or augmented reality experience. In some embodiments, the input is an input to reduce the immersion level of the three-dimensional environment displayed by the computer system (e.g., by rotating the previously described physical button in a first direction), such as described with reference to method 800. In some embodiments, the second event is an input to open or initiate the virtual or augmented reality experience. In some embodiments, the second event is an input to open or launch an application that provides the virtual or augmented reality experience. In some embodiments, the second event is an input to increase the immersion level of the three-dimensional environment displayed by the computer system (e.g., above or reaching a threshold immersion level) (e.g., by rotating the previously described physical button in a second direction different from the first direction), such as described with reference to method 800. Transitioning to the second state of the display generation component based on user input provides an efficient way to transition to the second state.

[0264] In some embodiments, detecting the first event includes detecting an input corresponding to a request to place the display generation component in a lower power state (1020). For example, in some embodiments, the display generation component is included in a device (e.g., a head-mounted device) and the input is an input to turn off power to the device or put the device into a sleep or low power mode. In some embodiments, the second event is an input to turn on power to the device or put the device into a normal power mode (e.g., exit a sleep or lower power mode). Transitioning to the second state of the display generation component based on whether the user is wearing the device reduces the number of inputs required to transition to the second state.

[0265] It should be understood that the particular order in which the operations in method 1200 are described is merely exemplary and is not intended to indicate that the described order is the only order in which the operations may be performed. A person of ordinary skill in the art will recognize many ways to rearrange the operations described herein.

[0266] 13A to 13C Illustrated is an example of a computer system according to some embodiments selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system.

[0267] Figure 13A The display generation components 120a and 120b (eg, Figure 1101a). Computer system 101a is optionally located in a first physical environment, with three-dimensional environment 1302a optionally visible via its display generation component 120a, and computer system 101b is optionally located in a second physical environment, with three-dimensional environment 1302b optionally visible via its display generation component 120b. Three-dimensional environment 1302a is visible from a user's viewpoint 1328c, illustrated in a top-down view (e.g., facing a wall of the room in which computer system 101a is located). Three-dimensional environment 1302b is visible from a user's viewpoint 1330c, illustrated in a top-down view (e.g., facing a wall of the room in which computer system 101b is located). This top-down view optionally corresponds to the layout of representations of various virtual objects and / or users relative to one another in three-dimensional environment 1302a, visible via computer system 101a. Both of these virtual objects and / or user representations will be described in greater detail later. The top view of the three-dimensional environment 1302b visible via computer system 101b will optionally include corresponding elements and / or will reflect the corresponding relative layout. Computer systems 101a and 101b optionally participate in a communication session such that the relative positions of the user's representations and shared virtual objects with respect to each other in the respective three-dimensional environments displayed by computer systems 101a and 101b are consistent and / or identical, as will be described in more detail below and with reference to method 1400.

[0268] As mentioned above Figures 1 to 6 As described above, computer systems 101a and 101b optionally include a display generation component (e.g., a touch screen) and multiple image sensors 314a and 314b (e.g., Figure 3 101b). The image sensor optionally includes one or more of: a visible light camera; an infrared camera; a depth sensor; or any other sensor that computer systems 101a and 101b can use to capture one or more images of a user or a portion of a user (e.g., one or more hands of a user) as the user interacts with computer system 101 or 101b. In some embodiments, the user interface shown and described below may also be implemented on a head-mounted display that includes a display generation component that displays a user interface or a three-dimensional environment to the user, as well as sensors that detect movement of the physical environment and / or the user's hands (e.g., external sensors facing outward from the user) and / or sensors that detect the user's gaze (e.g., internal sensors facing inward toward the user's face).

[0269] like Figure 13AAs shown, the computer system 101a captures one or more images of the physical environment surrounding the computer system 101a (e.g., operating environment 100), including one or more objects in the physical environment surrounding the computer system 101a. In some embodiments, the computer system 101a displays a representation of the physical environment in a three-dimensional environment 1302a, and / or the physical environment is visible in the three-dimensional environment 1302a via the display generation component 120a. For example, the three-dimensional environment 1302a visible via the display generation component 120a includes a representation of the physical floor and the back and side walls of the room in which the computer system 101a is located. The three-dimensional environment 1302a also includes images of the physical environment surrounding the computer system 101a, including one or more objects in the physical environment surrounding the computer system 101a. Figure 13A Table 1322a is visible from viewpoint 1328c.

[0270] The computer system 101b optionally similarly captures one or more images of the physical environment surrounding the computer system 101b (e.g., operating environment 100), including one or more objects in the physical environment surrounding the computer system 101b. In some embodiments, the computer system 101b displays a representation of the physical environment in a three-dimensional environment 1302b, and / or the physical environment is visible in the three-dimensional environment 1302b via the display generation component 120b. For example, the three-dimensional environment 1302b visible via the display generation component 120b includes a representation of the physical floor and back and side walls of the room in which the computer system 101b is located. The three-dimensional environment 1302b also includes images of the physical environment surrounding the computer system 101b, including one or more objects in the physical environment surrounding the computer system 101b. Figure 13A Sofa 1324a is visible from viewpoint 1330c.

[0271] exist Figure 13A In FIG, the three-dimensional environment 1302a also includes virtual objects 1306a (corresponding to the object 1306c in the top view), 1308a (corresponding to the object 1308c in the top view), and 1310a (corresponding to the object 1310c in the top view) visible from the viewpoint 1328c. Figure 13A In the example, objects 1306a, 1308a, and 1310a are two-dimensional objects, but the examples of the present disclosure are optionally also applicable to three-dimensional objects. The three-dimensional environment 1302a also includes, optionally, the current Figure 13A101a is a user of computer system 101a. A virtual object 1312c is not visible in three-dimensional environment 1302a from the viewpoint 1328c of the user of computer system 101a in the communication session. Virtual objects 1306a, 1308a, 1310a, and 1312c are optionally one or more of an application's user interface (e.g., a messaging user interface or a content browsing user interface), a three-dimensional object (e.g., a virtual clock, a virtual ball, or a virtual car), or any other element displayed by computer system 101a that is not included in the physical environment of computer system 101a. Three-dimensional environment 1302a also includes a representation 1330a of a user of computer system 101b, as well as a representation 1332a of a user of another computer system that has also joined the communication session. The representations of users described herein are optionally avatars or other visual representations of their corresponding users. Additional or alternative details regarding such representations of users are described with reference to method 1400.

[0272] Three-dimensional environment 1302b visible via computer system 101b also includes virtual object 1308b (corresponding to virtual objects 1308a and 1308c), virtual object 1310b (corresponding to virtual objects 1310a and 1310c), and a representation 1332b (corresponding to representation 1332a) of a user of another computer system (in addition to computer systems 101a and 101a) also participating in the communication session. However, virtual objects 1308b and 1310b and representation 1332b are visible from a different perspective than via computer system 101a, corresponding to a different viewpoint 1330c of the user of computer system 101b, as shown in the top view. Three-dimensional environment 1302b visible via computer system 101b also includes a representation 1328b of the user of computer system 101a, visible from viewpoint 1330c of the user of computer system 101b.

[0273] Returning to the three-dimensional environment 1302a, virtual objects 1308a and 1310a are optionally shared virtual objects (e.g., 13A to 13C ). Shared virtual objects are optionally accessible and / or visible to users and / or computer systems in their respective three-dimensional environments sharing those shared virtual objects. For example, three-dimensional environment 1302b includes those shared virtual objects 1308b and 1310b, as shown in FIG. Figure 13A , because virtual objects 1308a and 1310a are optionally shared with computer system 101b. In contrast, virtual object 1306a is optionally private to computer system 101a (e.g., 13A to 13C101b). Virtual objects 1312c are also optionally private to computer system 101a. Private virtual objects are optionally accessible and / or visible to the user and / or computer system that owns them, and inaccessible and / or invisible to the user and / or computer system that does not own them. For example, three-dimensional environment 1302b does not include a representation of virtual object 1306a because virtual object 1306a is optionally private to computer system 101a, not computer system 101b. Additional or alternative details regarding shared and private virtual objects are described with reference to method 1400.

[0274] In some embodiments, because shared virtual objects and / or representations of users are accessible and / or visible to multiple users and / or computer systems participating in a communication session, input that moves such shared virtual objects and / or representations of users relative to the viewpoint of a given user in the communication session optionally preferably avoids moving those shared virtual objects relative to the viewpoints of other users in the communication session. Additionally, private virtual objects are optionally shifted to avoid conflict with shared virtual objects and / or representations of users (e.g., as described with reference to methods 1000 and / or 1400). Examples of the foregoing will now be described.

[0275] exist Figure 13A In the embodiment of the present invention, the computer system 101b detects input from a hand 1303b of a user of the computer system 101b to move a shared virtual object 1308b in a three-dimensional environment 1302b (e.g., an air gesture input as described with reference to method 1400). In response, the computer system 101b moves the virtual object 1308b away from the user's viewpoint 1330c in the three-dimensional environment 1302b based on the input from the hand 1303b. Figure 13B Therefore, the virtual object 1308a (corresponding to the virtual object 1308b) in the three-dimensional environment 1302a is correspondingly moved to the left in the three-dimensional environment 1302a by the computer system 101a, as shown in FIG. Figure 13B (included in the top view).

[0276] exist Figure 13B, computer system 101a detects input to reposition and / or reorient shared virtual objects 1308a and 1310a and / or representations 1330a and 1332a relative to viewpoint 1328c. For example, the input is optionally a re-centering input (e.g., as described with reference to methods 800, 1000, 1200, and / or 1400) detected at computer system 101a that updates the relative position and / or orientation of virtual objects 1306a, 1308a, 1310a, and / or 1312a and / or representations 1330 and / or 1332a relative to viewpoint 1328c to satisfy one or more sets of criteria (e.g., as described with reference to methods 800, 1000, 1200, and / or 1400).

[0277] In response, the computer system 101a updates the relative positions and / or orientations of the shared virtual objects and the user's representations relative to the viewpoint 1328c, as shown in FIG. Figure 13C For example, because virtual objects 1308a and 1310a are shared among multiple users in a communication session, computer system 101a optionally does not change the relative positions and / or orientations of virtual objects 1308a and 1310a relative to the viewpoints of users other than the user of computer system 101a (e.g., viewpoints 1330c and 1332c). Instead, computer system 101a moves viewpoint 1328c such that virtual objects 1308a and 1310a move relative to viewpoint 1328c (e.g., move closer to viewpoint 1328c), as shown in FIG. Figure 13C . The movement of viewpoint 1328 c also optionally occurs in the same manner relative to viewpoints 1330 c and 1332 c and representations 1330 a (now outside the user's field of view from viewpoint 1328 c) and 1332 a. Thus, from viewpoint 1328 c, virtual objects 1308 a and 1310 a and representations 1330 a and 1332 a have moved within three-dimensional environment 1302 a, but virtual objects 1308 b and 1310 b and representation 1332 b have not yet moved within three-dimensional environment 1302 b. Figure 13C The relative movement of the viewpoint 1328c in the 3D environment 1302b relative to the virtual objects 1308a and 1310a and relative to the viewpoints 1330c and 1332c also causes the representation 1328b in the 3D environment 1302b to move accordingly, as shown in FIG. Figure 13C As shown in .

[0278] Furthermore, because computer system 101a optionally does not change the relative positioning of virtual objects 1308a and 1310a with respect to viewpoints 1330c and 1332c (e.g., because they are shared virtual objects), virtual objects 1308a and 1310a remain in their original positions. Figure 13C, even if they conflict with physical objects (e.g., table 1322a) in the three-dimensional environment 1302a. Figure 13C In response to Figure 13B 1308a and 1310a, the virtual object 1310a collides with (e.g., intersects) the table 1322a at its target location. However, the computer system 101a optionally does not perform operations with respect to the position and / or orientation of the virtual object 1310a to avoid the collision with the table 1322a (e.g., the viewpoint 1328c moves with respect to the virtual objects 1308a and 1310a independently of and / or without regard to physical objects in the three-dimensional environment 1302a).

[0279] In contrast to sharing virtual objects, the computer system 101a optionally responds to Figure 13B The operation of changing the position and / or orientation of the private virtual object to avoid collision with other virtual objects or physical objects is performed based on the input detected in the communication session, because changing the position and / or orientation of the private virtual object does not affect the three-dimensional environment displayed by other computer systems participating in the communication session (e.g., because those private virtual objects are not accessible to those other computer systems). Figure 13C In the computer system 101a, the computer system 101a has responded to Figure 13B The virtual object 1306a is moved from its Figure 13B position in the (e.g., to the right) to avoid Figure 13B In addition, regarding virtual object 1312a, it responds to the input detected in Figure 13B The position of the input in the top view will optionally be as indicated by 1312c' in the top view, however, at this position, it will optionally collide with the table 1322a. Therefore, the computer system 101a has shifted the virtual object 1312a (e.g., away from the viewpoint 1328c) to avoid the collision with the table 1322a. The shifting of the object to avoid the collision is optionally performed according to one or more aspects of the method 1000 described previously.

[0280] 14A to 14E is a flow chart illustrating a method for selectively re-centering content associated with a communication session between multiple users in response to input detected at the computer system according to some embodiments. In some embodiments, method 1400 is performed on a computer system (e.g., Figure 1 The computer system 101 in the embodiment of the present invention is executed at a computer system 101 in the embodiment of the present invention, such as a tablet computer, smart phone, wearable computer or head mounted device, which includes a display generating component (e.g., Figure 1 、 Figure 3 and Figure 4In some embodiments, method 1400 is performed by a display generating component 120 in a computer system, such as a head-up display, a display, a touch screen, or a projector, and one or more cameras, such as a camera pointing downward toward a user's hand (e.g., a color sensor, an infrared sensor, or other depth sensing camera) or a camera pointing forward from a user's head. In some embodiments, method 1400 is performed by a computer system, such as one or more processors 202 of computer system 101 (e.g., a processor 203 of computer system 101). Figure 1 Some operations in method 1400 are optionally combined, and / or the order of some operations is optionally changed.

[0281] In some embodiments, method 1400 is performed at a first computer system (e.g., 101a) in communication with a display generation component (e.g., 120a) and one or more input devices. In some embodiments, the computer system has one or more characteristics of the computer system of methods 800, 1000, and / or 1200. In some embodiments, the display generation component has one or more characteristics of the display generation component of methods 800, 1000, and / or 1200. In some embodiments, the one or more input devices have one or more of the characteristics of the one or more input devices of methods 800, 1000, and / or 1200.

[0282] In some embodiments, while a communication session is ongoing between a first user of a first computer system and a second user of a second computer system (such as with respect to Figure 13A 101a and 101b in the computer system) and a three-dimensional environment (e.g., a three-dimensional environment optionally having one or more characteristics of the three-dimensional environment of methods 800, 1000 and / or 1200) (such as Figure 13A When the three-dimensional environment 1302a in the computer system is visible from a first viewpoint of a first user via display generation components (e.g., such as described with reference to methods 800, 1000, and / or 1200), the computer system displays (1402a) via the display generation components a plurality of virtual objects in the three-dimensional environment, including a first virtual object and a second virtual object, such as Figure 13A In some embodiments, a first virtual object in the plurality of virtual objects can be accessed by a first computer system and a second computer system (1402b), such as Figure 13AIn some embodiments, the first computer system and the second computer system are located in the same physical environment (e.g., in different locations in the same room). In some embodiments, the first computer system and the second computer system are located in different physical environments (e.g., in different cities, different rooms, different states, and / or different countries). In some embodiments, the first computer system and the second computer system communicate with each other so that the display of the objects and / or the three-dimensional environment by the two computer systems is coordinated (e.g., changes to the objects and / or the three-dimensional environment in response to input from the first user of the first computer system are reflected in the display of the objects and / or the three-dimensional environment by the second computer system).

[0283] In some embodiments, the three-dimensional environment includes (1402c) a representation (1402d) of a second user of a second computer system at a first location in the three-dimensional environment, such as Figure 13A In some embodiments, a first virtual object accessible by the first computer system (e.g., the first virtual object optionally having one or more characteristics of the virtual object of methods 800, 1000, 1200, and / or 1600) is displayed at a second location in the three-dimensional environment, the first virtual object being accessible by the second computer system (e.g., 1402e), such as an avatar corresponding to a user of the second computer system and / or a cartoon or realistic (three-dimensional) model of the user of the second computer system; in some embodiments, the first location corresponds to a location of a viewpoint from which the second computer system displays the three-dimensional environment, which optionally corresponds to a physical location in the physical environment of the user of the second computer system. Figure 13A In some embodiments, a second virtual object accessible by the first computer system (e.g., the second virtual object optionally having one or more characteristics of the virtual object of methods 800, 1000, 1200, and / or 1600) is displayed at a third location in the three-dimensional environment, the second virtual object not accessible by the second computer system (1402f), such as Figure 13A1306a in. In some embodiments, the first virtual object is a shared virtual object (e.g., shared by a user of the first computer system and a user of the second computer system, or vice versa). Shared virtual objects are optionally displayed in a three-dimensional environment displayed by the computer systems sharing these shared virtual objects. Thus, the first virtual object is optionally displayed by both the first computer system and the second computer system at a second position in their respective three-dimensional environments. In addition, the users of the computer systems sharing the shared virtual object are optionally able to interact with the shared virtual object (e.g., provide input to the shared virtual object or move the shared virtual object in the three-dimensional environment). In some embodiments, the second virtual object is a private virtual object (e.g., private to the user of the first computer system). Private virtual objects are optionally displayed in a three-dimensional environment only by those computer systems that own the private virtual object. Thus, the second virtual object is optionally displayed by the first computer system at a third position in the three-dimensional environment, but not by the second computer system. In some embodiments, the second computer system displays an outline or other indication of the second virtual object at a third location in the three-dimensional environment displayed by the second computer system without displaying the contents of the second virtual object in the three-dimensional environment, while the first computer system displays the contents of the second virtual object in the three-dimensional environment displayed by the first computer system. Additionally, in some embodiments, only the user of the computer system to which the private virtual object is private can interact with the private virtual object (e.g., provide input to the private virtual object, move the private virtual object in the three-dimensional environment).

[0284] In some embodiments, the representation of the second user has a first spatial arrangement (1402g) relative to the first virtual object, such as Figure 13A 1332a relative to the spatial arrangement of the object 1308a. For example, the orientation of the second user's representation relative to the orientation of the first virtual object is a specific relative orientation, the distance between the second user's representation and the first virtual object is a specific distance, the position of the second user's representation relative to the position of the first virtual object in the three-dimensional environment is a specific relative position, and / or the relative height of the second user's representation and the first virtual object in the three-dimensional environment is a specific relative height. In some embodiments, the second virtual object has a second spatial arrangement (1402h) relative to the first virtual object and the second user's representation, such as Figure 13A13. The spatial arrangement of object 1306a relative to object 1308a and representation 1332a in the three-dimensional environment. For example, the orientation of the second virtual object relative to the first virtual object and / or the representation of the second user is a specific relative orientation, the distance between the second virtual object and the representation of the first virtual object and / or the second user is a specific distance, the position of the second virtual object relative to the position of the representation of the first virtual object and / or the second user in the three-dimensional environment is a specific relative position, and / or the height of the second virtual object relative to the representation of the first virtual object and / or the second user in the three-dimensional environment is a specific relative height.

[0285] In some embodiments, when multiple virtual objects are displayed in a three-dimensional environment, the computer system receives (1402i) via the one or more input devices a first input corresponding to a request to update the spatial arrangement of the one or more virtual objects relative to the current viewpoint of the first user, such as Figure 13B In some embodiments, in response to receiving the first input (1402j), the computer system moves the representation of the second user and content associated with the communication session between the first user and the second user (e.g., representations of users in the communication session or virtual objects shared in the communication session, such as the first virtual object) relative to the three-dimensional environment. Figure 13C In some embodiments, based on determining that at least a portion of the content associated with the communication session between the first user and the second user has been moved to a position within a threshold distance (e.g., 0.1 cm, 0.3 cm, 0.5 cm, 1 cm, 3 cm, 5 cm, 10 cm, 20 cm, 30 cm, 50 cm, 100 cm, 250 cm, or 500 cm) of the second virtual object in the three-dimensional environment, the computer system moves (14021) the second...

Claims

1. A method comprising: At a computer system in communication with a display generating component and one or more input devices: When displaying a first virtual object in a three-dimensional environment via the display generation component, and when the user's attention is directed toward the first visual object, displaying the first virtual object at a first visual prominence level relative to the three-dimensional environment; detecting a movement of the attention of the user of the computer system away from the first virtual object while displaying the first virtual object at the first visual saliency level via the display generation component; as well as In response to detecting that the attention of the user of the computer system moves away from the first virtual object: Based on determining that the attention of the user is directed toward the second virtual object when the second virtual object is currently displayed at a second visual significance level relative to the three-dimensional environment that is lower than the first visual significance level: While displaying the first virtual object at a fourth visual saliency level lower than the first visual saliency level, the second virtual object is displayed at a third visual saliency level higher than the second visual saliency level via the display generation component.

2. The method according to claim 1, further comprising: Based on the determination that the user's attention is directed to the second virtual object when the second virtual object is currently displayed at the second visual saliency level relative to the three-dimensional environment, the first virtual object is displayed at the second visual saliency level via the display generation component. 3 . The method of claim 1 , wherein the first visual significance level corresponds to a first translucency level and the second visual significance level corresponds to a second translucency level greater than the first translucency level.

4. The method according to claim 3, wherein: displaying the first virtual object at the first visual saliency level includes displaying, via the display generation component, a first portion of the first virtual object at a third translucency level, and displaying, via the display generation component, a second portion of the first virtual object at a fourth translucency level different from the third translucency level, and Displaying the second virtual object at the third visual saliency level in response to detecting that the attention of the user of the computer system moves away from the first virtual object includes changing the translucency of a first portion of the second virtual object by a first amount, and changing the translucency of a second portion of the second virtual object by a second amount that is different from the first amount. 5 . The method of claim 1 , wherein the detecting that the user's attention moves away from the first virtual object comprises detecting that the user's gaze is directed toward a corresponding location within the three-dimensional environment for a threshold amount of time.

6. The method according to claim 1, wherein detecting that the user's attention moves away from the first virtual object comprises: detecting a gaze of the user directed toward a corresponding location within the three-dimensional environment; And detecting a gesture performed by a corresponding portion of the user of the computer system when the gaze of the user is directed to the corresponding location within the three-dimensional environment.

7. The method according to claim 1, wherein detecting that the user's attention moves away from the first virtual object comprises: detecting a gaze of the user directed toward a corresponding location within the three-dimensional environment; as well as When the gaze of the user is directed toward the corresponding location within the three-dimensional environment, and when the second virtual object is displayed at the second visual saliency level relative to the three-dimensional environment: displaying, via the display generation component, the second virtual object at the third visual prominence level higher than the second visual prominence level based on determining that one or more first criteria are satisfied, including a criterion satisfied when the gaze of the user is directed at the second virtual object for a threshold amount of time without detecting a selection input from a corresponding portion of the user; as well as based on determining that one or more second criteria are satisfied, including a criterion that is satisfied when the selection input from the corresponding portion of the user is detected before the gaze of the user is directed toward the second virtual object for the threshold amount of time, The second virtual object is displayed via the display generation component at the third visual significance level higher than the second visual significance level.

8. The method of claim 7, wherein, based on the determining that the one or more second criteria are satisfied, the corresponding content included in the second virtual object is not selected in response to detecting the selection input from the corresponding portion of the user.

9. The method according to claim 1, further comprising: When the first virtual object is displayed at the first visual saliency level via the display generation component, and when the attention of the user of the computer system is directed to the first virtual object, detecting that the attention of the user moves within the first virtual object; as well as In response to detecting that the attention of the user moves within the first virtual object, maintaining display of the first virtual object at the first visual saliency level.

10. The method according to claim 1, further comprising: In response to detecting that the attention of the user of the computer system moves away from the first virtual object, based on determining that the user's attention is directed to a location within the three-dimensional environment that does not correspond to a corresponding virtual object, maintaining the display of the first virtual object at the first visual saliency level relative to the three-dimensional environment via the display generation component.

11. The method according to claim 1, further comprising: In response to detecting that the attention of the user of the computer system moves away from the first virtual object, based on determining that the user's attention is directed to a non-interactive virtual object, maintaining the display of the first virtual object at the first visual saliency level relative to the three-dimensional environment via the display generation component.

12. The method of claim 1, wherein the first virtual object and the third virtual object are associated with a group of virtual objects, and the method further comprises: In response to detecting that the attention of the user of the computer system moves away from the first virtual object, based on determining that the user's attention is directed toward the third virtual object, maintaining the display of the first virtual object at the first visual saliency level relative to the three-dimensional environment via the display generation component.

13. The method according to claim 1, further comprising: In response to detecting that the attention of the user of the computer system moves away from the first virtual object, based on determining that the user is currently interacting with the first virtual object, maintaining the display of the first virtual object at the first visual saliency level relative to the three-dimensional environment via the display generation component.

14. The method of claim 13, wherein the current interaction with the first virtual object comprises moving the first virtual object, the method further comprising: When the first virtual object is moved as part of the current interaction with the first virtual object, a visual indication associated with moving the first virtual object is displayed via the display generation component. 15 . The method of claim 13 , wherein the current interaction with the first virtual object comprises selecting first content and moving the first content from the first virtual object to a corresponding virtual object other than the first virtual object.

16. The method of claim 13, wherein the current interaction with the first virtual object comprises moving the first content within the first virtual object.

17. The method of claim 1, wherein: The second visual significance level corresponds to a second translucency level that is greater than a first translucency level corresponding to the first visual significance level, and The third visual significance level corresponds to a third translucency level that is lower than the second translucency level.

18. The method of claim 1, wherein: The second visual significance level corresponds to a second blur level that is greater than a first blur level corresponding to the first visual significance level, and the third visual significance level corresponds to a third blur level that is lower than the second blur level.

19. The method of claim 1, wherein the first virtual object is displayed in front of a visual representation of the user's physical environment relative to the user's viewpoint.

20. The method of claim 1, wherein the first virtual object is displayed in front of the virtual environment relative to the user's viewpoint.

21. The method of claim 1, wherein the first virtual object is associated with a first application and the second virtual object is associated with a second application different from the first application.

22. The method of claim 1, wherein the second virtual object is a control user interface associated with an operating system of the computer system.

23. The method of claim 1, further comprising: when displaying, via the display generation component, the second virtual object at the second visual saliency level relative to the three-dimensional environment, displaying, via the display generation component, a corresponding selectable element associated with moving the second virtual object at a fifth visual saliency level; as well as receiving, via the one or more input devices, a first input directed to the corresponding selectable element while the second virtual object is displayed at the second visual saliency level and the corresponding selectable element is displayed at the fifth visual saliency level; as well as In response to detecting the first input, the second virtual object is moved in the three-dimensional environment according to the first input.

24. The method of claim 1, further comprising: When displaying the first virtual object at the fourth visual saliency level via the display generation component, displaying via the display generation component a corresponding selectable element associated with moving the first virtual object; while displaying the first virtual object at the fourth visual saliency level via the display generation component and displaying the corresponding selectable element associated with moving the first virtual object via the display generation component, detecting an input directed to a corresponding element associated with the first virtual object; and In response to detecting the input directed to the corresponding element associated with the first virtual object: According to determining that the corresponding element is the corresponding element that can be selected: initiating a process of moving the first virtual object according to the input; and Based on determining that the corresponding element corresponds to corresponding content included in the first virtual object, the first virtual object is displayed at a fifth visual significance level greater than the fourth visual significance level via the display generation component, without performing an operation associated with the corresponding content based on the input.

25. The method according to claim 1, wherein the first virtual object comprises currently playing media content, the method further comprising: In response to displaying the second virtual object at the third visual significance level relative to the three-dimensional environment via the display generation component, playback of the media content included within the first virtual object displayed at the fourth visual significance level is maintained.

26. The method of claim 1, wherein the first virtual object is an object of a first type, the method further comprising: when displaying a third virtual object in the three-dimensional environment at a fifth visual saliency level relative to the three-dimensional environment via the display generation component, wherein the third virtual object is an object of a second type different from objects of the first type and the attention of the user is directed toward the third virtual object, detecting a movement of the attention of the user of the computer system away from the third virtual object; as well as In response to detecting that the attention of the user of the computer system moves away from the third virtual object, and based on determining that the user's attention is directed toward a fourth virtual object in the three-dimensional environment, maintaining display of the third virtual object at the fifth visual significance level relative to the three-dimensional environment.

27. The method of claim 26, wherein the second type of object is a representation of a respective user associated with the computer system.

28. The method of claim 26, wherein the second type of object is a user interface of a media playback application.

29. The method of claim 26, wherein the second type of object is a status user interface of the computer system.

30. The method of any one of claims 27, wherein the second type of object is a user interface of a communication application.

31. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system communicating with a display generating component and one or more input devices, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 30.

32. A computer system in communication with a display generation component and one or more input devices, the computer system comprising: one or more processors; and A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 30.

Images