Device, method and graphical user interface for interacting with a three-dimensional environment

Abstract

It simplifies the user's use of a computer system when interacting with a three-dimensional environment. The system displays a first user interface object having a first appearance at a first position within a first view of a three-dimensional environment at least partially shared between a first user and a second user. While displaying the first user interface object, the computer system detects a first user input directed to the first user. In response to detecting the first user input, the system performs a first action in accordance with a determination that the second user is not currently interacting with the first user interface object, and displays a visual indication that the first user interface object is unavailable for interaction and cancels performance of the first action in accordance with a determination that the second user is not currently interacting with the first user interface object.

Description

[Technical Field]

[0001] (Related Applications) This application is a continuation of U.S. Patent Application No. 17 / 483,730, filed September 23, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 083,816, filed September 25, 2020, each of which is incorporated by reference herein in its entirety.

[0002] (Technical field) The present disclosure generally relates to computer systems having a display generation component, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display, and one or more input devices that provide computer-generated reality (CGR) experiences. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include digital images, video, text, icons, and virtual objects such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulating virtual objects is complex and error-prone create a significant cognitive burden for users and detract from the experience of the virtual / augmented reality environment. In addition, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Therefore, there is a need for a computer system with improved methods and interfaces for providing users with computer-generated experiences that make interacting with the computer system more efficient and intuitive for the user. The above-mentioned deficiencies and other problems associated with user interfaces for computer systems having a display generation component and one or more input devices are reduced or eliminated by the disclosed systems, methods, and user interfaces. Such systems, methods, and interfaces optionally complement or replace conventional systems, methods, and user interfaces that provide users with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from the user by helping the user understand the connection between the input provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] According to some embodiments, a method is performed in a computer system in communication with a first display generation component and one or more first input devices, the method including: displaying a first user interface object in a first view of a three-dimensional environment that is at least partially shared between a first user and a second user, the first user interface object being displayed at a first position within the first view of the three-dimensional environment with a first set of appearance characteristics; and detecting a first user input provided by the first user while displaying the first user interface object in the first view of the three-dimensional environment with the first set of appearance characteristics, the first user input being directed at the first user interface object. The method further includes, in response to detecting a first user input directed at the first user interface object, performing a first action on the first user interface object in accordance with the first user input in accordance with a determination that a second user is not currently interacting with the first user interface object; and, in accordance with a determination that the second user is currently interacting with the first user interface object, displaying a visual indication that the first user interface object is unavailable for interaction with the first user, where displaying the visual indication includes changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment; and canceling performing the first action on the first user interface object in accordance with the first user input.

[0007] According to some embodiments, a method is executed on a computer system in communication with a first display generation component and one or more first input devices, and includes: displaying a first view of a three-dimensional environment including first user interface objects representing a first object in a second physical environment different from the first physical environment while a first user is at a first location in the first physical environment, the first user interface objects corresponding to a first viewpoint associated with the first location in the first physical environment, wherein individual positions of the first user interface objects in the three-dimensional environment correspond to individual locations of the first object in the second physical environment in a first manner; detecting at least one of movement of the first user in the first physical environment and movement of the first object in the second physical environment; displaying a second view of the three-dimensional environment corresponding to the second viewpoint in response to detecting at least one of movement of the first user in the first physical environment and movement of the first object in the second physical environment; and displaying the first user interface objects in the second view of the three-dimensional environment.Displaying the first user interface object in the second view of the three-dimensional environment includes: displaying the first user interface object at a first display position in the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object in the three-dimensional environment, which corresponds to the individual location of the first object in the second physical environment in a first manner, is more than a threshold distance from the individual position in the three-dimensional environment corresponding to a second viewpoint associated with the second view of the three-dimensional environment; and displaying the first user interface object at a second display position in the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object in the three-dimensional environment, which corresponds to the individual location of the first object in the second physical environment in the first manner, is less than a threshold distance from the individual position in the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment, which second display position is offset from the individual position of the first user interface object in the three-dimensional environment.

[0008] According to some embodiments, a method is executed on a computer system in communication with a first display generation component and one or more first input devices, and includes displaying a first computer-generated experience at a first immersion level; receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level; in response to receiving the biometric data corresponding to the first user, in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, displaying the first computer-generated experience at a second immersion level, wherein the first computer-generated experience displayed at the second immersion level occupies a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level; and in accordance with a determination that the biometric data corresponding to the first user does not satisfy the first criterion, continuing to display the first computer-generated experience at the first immersion level.

[0009] According to some embodiments, a method is executed in a computer system in communication with a first display generation component and one or more first input devices, the method comprising: displaying a first view of the physical environment including a first representation of a first portion of the physical environment; detecting, while displaying the first view of the physical environment, a first user input corresponding to a request to activate a first type of computer-generated sensory adjustment of two or more types of computer-generated sensory adjustment; and, in response to detecting the first user input, displaying a second view of the physical environment, the second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted relative to the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment. displaying a second view of the physical environment; detecting, while displaying the second view of the physical environment, a second user input corresponding to a request to activate a second type of computer-generated sensory adjustment among the two or more types of computer-generated sensory adjustments, the second type of computer-generated sensory adjustment being different from the first type of computer-generated sensory adjustment; and in response to detecting the second user input, displaying a third view of the physical environment, the third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment and second display characteristics adjusted for the second representation of the physical environment in accordance with the second type of computer-generated sensory adjustment.

[0010] According to some embodiments, a method is executed on a computer system in communication with a first display generation component and one or more first input devices, the method comprising: displaying a first view of a three-dimensional environment including a first representation of a first portion of the physical environment; detecting, while displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment, movement of a first user from a first location to a second location in the physical environment; and, in response to detecting the movement of the first user from the first location to the second location, and in response to determining that the movement to the second location satisfies a first criterion, the first criterion including a first requirement that the second location correspond to a location associated with a first type of exercise, displaying a second view of the three-dimensional environment, the second view of the three-dimensional environment including a first virtual content set corresponding to the first type of exercise. displaying a second view of the three-dimensional environment, in which the first set of virtual content replaces at least a portion of the second representation of a second portion of the physical environment including the second location; and in accordance with a determination that travel to the second location satisfies a second criterion that is different from the first criterion, whereby the second criterion is satisfied, the second criterion including a second requirement that the second location correspond to a location associated with a second type of exercise, the second type of exercise being different from the first type of exercise, displaying a third view of the three-dimensional environment, the third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, the second set of virtual content being different from the first set of virtual content, and the second set of virtual content replacing at least a portion of the third representation of a third portion of the physical environment including the second location.

[0011] According to some embodiments, a computer system includes or is in communication with a display generation component (e.g., a display, projector, head-mounted display, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, one or more processors, and a memory that stores one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface on a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein. According to some embodiments, a computer system includes a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein.According to some embodiments, an information processing apparatus for use in a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, includes means for performing or causing to be performed the operations of any of the methods described herein.

[0012] Thus, computer systems having display generation components are provided with improved methods and interfaces for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments, thereby increasing the effectiveness, efficiency, and safety and satisfaction of such computer systems. Such methods and interfaces can complement or replace conventional methods for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments.

[0013] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0014] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0015] [Figure 1]FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0016] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0017] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a user with visual components of a CGR experience, according to some embodiments.

[0018] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0019] [Figure 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0020] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0021] [Figure 7A] FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments. [Figure 7B] FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments. [Figure 7C]FIG. 1 is a block diagram illustrating interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments.

[0022] [Figure 7D] FIG. 1 is a block diagram illustrating methods for displaying representations of physical objects in various ways relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movement within a first physical environment and the representations of the physical objects move according to the movement of the physical objects in a second physical environment different from the first physical environment, and a change in the way the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint meeting a preset criterion. [Figure 7E] FIG. 1 is a block diagram illustrating methods for displaying representations of physical objects in various ways relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movement within a first physical environment and the representations of the physical objects move according to the movement of the physical objects in a second physical environment different from the first physical environment, and a change in the way the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint meeting a preset criterion. [Figure 7F] FIG. 1 is a block diagram illustrating methods for displaying representations of physical objects in various ways relative to a viewpoint of a currently displayed view of a three-dimensional environment, where in some embodiments the viewpoint moves according to a user's movement within a first physical environment and the representations of the physical objects move according to the movement of the physical objects in a second physical environment different from the first physical environment, and a change in the way the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint meeting a preset criterion.

[0023] [Figure 7G]FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. [Figure 7H] FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. [Figure 7I] FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments. [Figure 7J] FIG. 1 is a block diagram illustrating changes in the immersion level displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments.

[0024] [Figure 7K] FIG. 1 is a block diagram illustrating aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments. [Figure 7L] FIG. 1 is a block diagram illustrating aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments. [Figure 7M] FIG. 1 is a block diagram illustrating aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

[0025] [Figure 7N]FIG. 10 is a block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, in accordance with a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise. [Figure 7O] FIG. 10 is a block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, in accordance with a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise. [Figure 7P] FIG. 10 is a block diagram illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, in accordance with a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to a particular type of exercise.

[0026] [Figure 8] 1 is a flowchart of a method for supporting interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments.

[0027] [Figure 9A] 1 is a flowchart of a method for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, wherein in some embodiments the viewpoint moves in accordance with a user's movement within a first physical environment and the representations of the physical objects move in accordance with the movement of the physical objects in a second physical environment different from the first physical environment, and wherein a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a predetermined criterion. [Figure 9B]1 is a flowchart of a method for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, wherein in some embodiments the viewpoint moves in accordance with a user's movement within a first physical environment and the representations of the physical objects move in accordance with the movement of the physical objects in a second physical environment different from the first physical environment, and wherein a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a predetermined criterion.

[0028] [Figure 10] 1 is a flowchart of a method for changing the immersion level of displaying an environment of a computer-generated experience according to changes in biometric data of a user received by a computer system, according to some embodiments.

[0029] [Figure 11] 1 is a flowchart of a method for aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

[0030] [Figure 12] 1 is a flowchart of a method for selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, in accordance with a determination that a portion of a physical environment within the view of the three-dimensional environment corresponds to the particular type of exercise. DETAILED DESCRIPTION OF THE INVENTION

[0031] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0032] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0033] In some embodiments, a computer system allows multiple users to have the right to access a first user interface object displayed in a three-dimensional environment, but prevents a user from accessing the first user interface object while another user is interacting with the first user interface object. When displaying a view of the three-dimensional environment including the first user interface object via a first display generation component used by the first user, the computer system detects a first user input directed at the first user interface object. In response to detecting the first user input, the computer system either performs a first action on the first user interface object corresponding to the first user input, or displays a visual indication that the first user interface object is unavailable for interaction with the first user and cancels performance of the first action, depending on whether the first user interface object is currently available for interaction with the first user. The computer system provides a visual indication and aborts execution of the first action pursuant to a determination that another user currently controls the first user interface object (e.g., another user is interacting with the first user interface object, interacting with the first user interface object in a manner that precludes simultaneous interaction of the first user, and / or locking the first user interface object for the type of action the first user intends to perform, etc.). In some embodiments, displaying the visual indication includes moving the first user interface object within the view of the three-dimensional environment shown to the first user to maintain a preset distance between the first user interface object and a close representation of the first user's hand. In some embodiments, displaying the visual indication includes changing a visual appearance of the first user interface object within the view of the three-dimensional environment shown to the first user.In some embodiments, when the first user interface object is released by the controlling user onto the first user (e.g., via a throw gesture, a toss gesture, etc.), the computer system rotates the first user interface object so that the first user interface object is displayed at a preset orientation relative to the viewpoint of the currently displayed view of the three-dimensional environment shown to the first user. In some embodiments, the computer system controls access to the first user interface object by displaying a representation of the first user interface object at or near a position of a representation of some part of the first user (e.g., a representation of the first user's hand, within arm's reach of a virtual position of the user's face, etc.). Displaying a visual indication in a view of the three-dimensional environment displayed via a display generation component used by the first user in response to the first user attempting to interact with the first user interface object, indicating that the first user interface object is unavailable for interaction with the first user, provides intuitive and timely feedback when an interaction is attempted and reduces unnecessary visual clutter in the view of the three-dimensional environment. Additionally, there is no need to display the same visual indication to other users sharing the environment with the first user, reducing user confusion and improving the efficiency of the man-machine interface.

[0034] In some embodiments, a computer system displays a view of a three-dimensional environment including representations of physical objects (e.g., a second user, an animal, a moving drone, etc.) located in a physical environment different from the physical environment of a first user (and a first display generation component used by the first user to view the three-dimensional environment). The computer system optionally moves a viewpoint corresponding to the currently displayed view of the three-dimensional environment according to movement of the first user (and / or the first display generation component) within the physical environment. The computer system determines positions and movement paths of representations of the physical objects within the three-dimensional environment based on the locations and movement paths of the physical objects within the physical environment. The computer system utilizes a first type of correspondence relationship (e.g., mapping relationships and transformation relationships, optionally various mapping relationships and transformation relationships for the viewpoint, the physical objects, the first user, etc.) between positions within the three-dimensional environment and locations within respective physical environments (e.g., the physical environments of the first user and first display generation component, the physical environments of the physical objects, etc.). In some circumstances (e.g., due to movement of the first user and / or movement of the physical object, etc.), the position of the representation of the physical object would be within a threshold distance (e.g., arm's length, 3 feet, a user-specified distance, etc.) of the position shown via the first display generation component if the position(s) were determined using a first type of correspondence between positions in the three-dimensional environment and locations in the physical environment. Under such conditions, the computer system displays the representation of the physical object at an adjusted position that is offset from the position determined based on the first type of correspondence. In some embodiments, the adjusted position is determined based on a second type of correspondence that is different from the first type of correspondence, ensuring that the adjusted position remains beyond the threshold distance from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generation component.The computer system continues to use the second type of correspondence to determine the adjusted position of the representation of the physical object until the unadjusted position calculated based on the first type of correspondence is more than a threshold distance away from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generation component. By monitoring the relative distance between the position of the representation of the physical object and the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generation component, the computer can timely adjust the display position of the representation of the physical object, thereby avoiding visual collisions between the viewpoint and the representation of the physical object. This improves the user's visual experience and reduces user confusion and errors when the user interacts with the three-dimensional environment.

[0035] In some embodiments, the computer system alters the immersion level at which a computer-generated experience (e.g., a visual experience, an audiovisual experience, a virtual reality experience, an augmented reality experience, etc.) is presented to the user according to biometric data corresponding to the user. For example, after the computer-generated experience is initiated, the computer system may detect changes in the biometric data (e.g., heart rate, blood pressure, respiratory rate, etc.) corresponding to the user, e.g., when the user adjusts his / her physical and emotional state, actively or under the influence of the computer-generated content. According to the changes in the biometric data with respect to respective sets of preset criteria associated with different immersion levels, the computer system increases or decreases the immersion level at which the computer-generated experience is provided to the user by altering the visual prominence (including, e.g., spatial extent, visual depth, saturation, visual contrast, etc.) of the virtual content relative to the visual prominence of the representation of the physical environment (e.g., by increasing the complexity, spatial extent, and / or visual characteristics of the virtual content and / or reducing the visual clarity, blur radius, opacity, saturation, etc. of the representation of the physical environment). Adjusting the level of immersion at which a computer-generated experience is provided to a user based on changes in biometric data corresponding to the user helps the computer system provide smoother transitions between low and high immersion level experiences that better correspond to the user's perceptual state with respect to the computer-generated experience, thereby reducing user confusion and improving the effectiveness of the computer-generated experience.

[0036] In some embodiments, the computer system provides multiple types of sensory modulation features that enhance a user's ability to perceive various aspects of the physical environment that may not be readily perceptible without the assistance of special equipment or the computer system. Instead of only allowing a user to use a single type of sensory modulation feature when viewing a portion of the physical environment at a time, the computer system aggregates the effects of two or more types of sensory enhancement features on a representation of a portion of the physical environment such that features and characteristics present in a portion of the physical environment that were previously hidden within the view of the physical environment provided by the computer system can be revealed. By allowing the effects of multiple types of sensory modulation features to be aggregated on a representation of the same portion of the physical environment and presented in a view of a three-dimensional environment that includes the representation of the portion of the physical environment, the user is able to better perceive and understand the physical environment and improves the usability of the computer-generated view of the physical environment.

[0037] In some embodiments, the computer system displays virtual content (e.g., virtual scenery, visual and functional enhancements to exercise equipment, user interfaces, health and leaderboards, etc.) corresponding to the individual type of exercise pursuant to determining that the physical location represented in the view of the three-dimensional environment is associated with the individual type of exercise. For example, as the user and the display generation component move through the real-world location, the virtual content shown in the view of the three-dimensional environment is adjusted to correspond to the type of exercise associated with the current location of the user and the display generation component. In some embodiments, if the location is associated with multiple types of exercise, the computer system selects a type of exercise from the multiple types of exercise associated with the location based on other contextual information (e.g., the user's movement, the user's engagement with objects at the location, etc.) and displays visual content corresponding to the selected type of exercise. Automatically selecting and / or modifying virtual content based on the individual type of exercise associated with the user and the location of the display generation component creates a more efficient human-machine interface by narrowing the number, range, and / or nature of inputs from the user to achieve a desired result (e.g., selecting virtual content appropriate for the type of exercise, initiating a particular mode of exercise, etc.).

[0038] Figures 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. Figures 7A-7C are block diagrams illustrating interactions with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments. Figures 7D-7F are block diagrams illustrating a method of displaying representations of physical objects in various ways relative to the viewpoint of a currently displayed view of the three-dimensional environment, where, according to some embodiments, the viewpoint moves according to the user's movement in a first physical environment, the representations of the physical objects move according to the movement of the physical objects in a second physical environment different from the first physical environment, and a change in the way the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint meeting a preset criterion. Figures 7G-7J are block diagrams illustrating a change in the immersion level for displaying an environment of a computer-generated experience according to changes in user biometric data received by the computer system, according to some embodiments. Figures 7K-7M are block diagrams illustrating aggregating the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment including a representation of a physical environment, according to some embodiments. 7N-7P are block diagrams illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment pursuant to a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to the particular type of exercise, according to some embodiments. The user interfaces of FIGS. 7A-7P are used to illustrate the processes of FIGS. 8-12, respectively.

[0039] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a consumer electronics device, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., within a head-mounted or handheld device).

[0040] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or with which the user can interact (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following is a subset of these terms:

[0041] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses, such as sight, touch, hearing, taste, and smell.

[0042] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and one or more properties of one or more virtual objects simulated within the CGR environment are adjusted accordingly to behave according to at least one law of physics. For example, a CGR system may detect a person's head rotation and adjust the graphical content and sound field presented to the person accordingly, in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to the property(ies) of a virtual object(s) in a CGR environment may be made in response to a representation of a body movement (e.g., a voice command). A person may sense and / or interact with a CGR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatially expansive audio environment, providing the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0043] Examples of CGR include virtual reality and mixed reality.

[0044] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0045] Mixed Reality: A mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects), as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between, but not including, a complete physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track location and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0046] Examples of mixed reality include augmented reality and augmented virtuality.

[0047] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person using the system perceives the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects overlaid on the physical environment. As used herein, videos of a physical environment shown on an opaque display are referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) different from the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, thereby rendering the altered portion a non-photorealistic, altered version of the originally captured image. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0048] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0049] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed toward a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects graphical images onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., physical setting / environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, a display, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of, or shares the same physical housing or support structure as, one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195.

[0050] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of the CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0051] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0052] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on their head, their hand, etc.). Thus, display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, display generation component 120 surrounds the user's field of view. In some embodiments, display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward scene 105. In some embodiments, the handheld device is optionally located within a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with CGR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interactions occur in the space in front of the HMD and the CGR content responses are displayed via the HMD. Similarly, a user interface showing interactions with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hands)) may be implemented similarly to an HMD where the interactions are triggered by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hands)).

[0053] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0054] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0055] In some embodiments, one or more communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0056] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0057] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0058] 1 , and optionally one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0059] In some embodiments, tracking unit 244 is configured to map scene 105 and at least display generation component 120 relative to scene 105 of FIG. 1 , and optionally track the position / location of one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, tracking unit 244 includes hand tracking unit 243 and / or eye tracking unit 245. In some embodiments, hand tracking unit 243 is configured to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 , relative to display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 243 is described in more detail below with respect to FIG. 4 . In some embodiments, eye tracking unit 245 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)) or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with respect to FIG. 5.

[0060] In some embodiments, adjustment unit 246 is configured to manage and adjust the CGR experience presented to the user by display generation component 120, and optionally by one or more of output devices 155 and / or peripheral devices 195. To that end, in various embodiments, adjustment unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0061] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0063] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functionality and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0064] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0065] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communication between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0066] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content.

[0067] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as scene cameras). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0068] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0069] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0070] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0071] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0072] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0073] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0075] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular implementations, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in Figure 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0076] 4 is a schematic diagram of an example embodiment of hand tracking device 140. In some embodiments, hand tracking device 140 (FIG. 1) is controlled by hand tracking unit 243 (FIG. 2) to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 (e.g., relative to parts of the physical environment surrounding the user, relative to display generation component 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand. In some embodiments, hand tracking device 140 is part of display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, hand tracking device 140 is separate from display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0077] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0078] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives display generation component 120 accordingly. For example, a user can interact with software running on controller 110 by moving hand 408 and changing hand posture.

[0079] In some embodiments, the image sensor 404 projects a spot pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the pattern's spots. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define a set of orthogonal x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0080] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors with patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

[0081] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0082] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or alternatively may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively, or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). While the controller 110 is shown in FIG. 4 as, by way of example, a separate unit from the image sensor 440, some or all of the processing functionality of the controller may be implemented by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or otherwise associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device) or using any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0083] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing depth resulting in darker shades. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0084] 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In FIG. 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and placed on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0085] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 245 (FIG. 2) to track the position and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device, such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if the display generation component is a handheld device or a CGR chamber, eye tracking device 130 is optionally a device separate from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally used in conjunction with head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally part of non-head-mounted display generation components.

[0086] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide a 3D virtual view to the user. For example, a head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0087] As shown in FIG. 5 , in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye-tracking camera and lighting source.

[0088] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil location, central visual location, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

[0089] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be pointed at the eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0090] In some embodiments, controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye tracking camera 540 for various purposes, such as in processing frames 562 for display. Controller 110 optionally estimates the user's viewpoint on display 510 based on gaze tracking input 542 obtained from eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from gaze tracking input 542 is optionally used to determine the user's current looking direction.

[0091] Some possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0092] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520, as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0093] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR range, thereby not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0094] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and / or augmented virtuality) experience.

[0095] Figure 6 shows a glint-assisted eye tracking pipeline, according to some embodiments. In some embodiments, the eye tracking pipeline is implemented by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0096] As shown in FIG. 6, an eye-tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0097] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0098] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0099] 6 is intended to serve as an example of eye-tracking technology that may be used in particular implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0100] In this disclosure, various input methods are described with respect to interaction with a computer system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example may be compatible with, and optionally utilize, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example may be compatible with, and optionally utilize, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. Where one example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with, and optionally utilize, the method described with respect to the other example. Thus, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User Interface and Related Processes

[0101] We now turn our attention to embodiments of user interfaces (“UIs”) and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, equipped with a display generation component, one or more input devices, and (optionally) one or more cameras.

[0102] 7A-7P illustrate a three-dimensional environment displayed via display generation components (e.g., display generation component 7100, display generation component 7200, display generation component 120, etc.) and interactions occurring in the three-dimensional environment caused by user input directed at the three-dimensional environment and / or input received from other computer systems and / or sensors. In some embodiments, input is directed at a virtual object in three dimensions by a user's gaze detected at the position of the virtual object in the three-dimensional environment, by a hand gesture performed at a location in the physical environment corresponding to the position of the virtual object, by a hand gesture performed at a location in the physical environment unrelated to the position of the virtual object while the virtual object has input focus (e.g., selected by simultaneously and / or previously detected gaze input, selected by simultaneously or previously detected pointer input, selected by simultaneously and / or previously detected gesture input, etc.), by an input device placing a focus selector object (e.g., a pointer object, a selector object, etc.) at the position of the virtual object, etc. In some embodiments, input is directed toward a physical object or a representation of a virtual object corresponding to a physical object by a user's hand movements (e.g., moving the entire hand, moving the entire hand in a discrete pose, moving one part of the hand relative to another part of the hand, relative movement between two hands, etc.) and / or manipulations of the physical object (e.g., touching, swiping, tapping, opening, moving toward, moving relatively, etc.). In some embodiments, the computer system alters the display of the three-dimensional environment (e.g., displaying additional virtual content or ceasing to display existing virtual content, transitioning between different immersion levels displaying visual content, etc.) according to input from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, proximity sensors, etc.) and contextual conditions (e.g., location, time, the presence of different objects in the environment, etc.).In some embodiments, the computer system modifies the display of the three-dimensional environment (e.g., displaying additional virtual content or ceasing to display existing virtual content, transitioning between different immersion levels displaying visual content, etc.) according to input from other computers used by other users who are sharing the computer-generated environment with the user of the computer system (e.g., in a shared computer-generated experience, a shared virtual environment, a shared virtual or augmented reality environment in a communication session, etc.).

[0103] In some embodiments, the three-dimensional environment displayed via the display generation component is a virtual three-dimensional environment that includes virtual objects and content at various virtual positions within the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at various virtual positions within the three-dimensional environment constrained by one or more physical aspects of the physical environment (e.g., the position and orientation of walls, floors, surfaces, direction of gravity, time of day, etc.). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. The representation of the physical environment includes respective representations of physical objects and surfaces at different positions within the three-dimensional environment, such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. When the virtual objects are positioned relative to the positions of the representations of the physical objects and surfaces in the three-dimensional environment, the virtual objects appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment. In some embodiments, the computer system transitions between displaying different types of environments (e.g., transitioning between presenting computer-generated environments or experiences at different levels of immersion, adjusting the relative prominence of audio / visual sensory inputs from the virtual content and from the representation of the physical environment, etc.) based on user input and / or contextual conditions.

[0104] In some embodiments, the display generation component includes a pass-through portion in which a representation of the physical environment is displayed. In some embodiments, the pass-through portion is a transparent or translucent (e.g., see-through) portion of the display generation component that surrounds the user's field of view and reveals at least a portion of the physical environment in the user's field of view. For example, the pass-through portion is a portion of a head-mounted display that is translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world surrounding the user through the pass-through portion without removing the head-mounted display or moving away from the head-up display. In some embodiments, the pass-through portion gradually transitions from translucent or transparent to fully opaque when displaying a virtual or mixed reality environment. In some embodiments, the pass-through portion of the display generation component displays images or a live video feed of at least a portion of the physical environment captured by one or more cameras (e.g., rear-facing camera(s) of a mobile device or associated with the head-mounted display, or other cameras that provide image data to an electronic device). In some embodiments, one or more cameras are aimed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generation component). In some embodiments, one or more cameras are aimed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user).

[0105] In some embodiments, when displaying virtual objects in positions corresponding to the locations of one or more physical objects in a physical environment (e.g., a virtual reality environment, a mixed reality environment, an augmented reality environment, etc.), at least some of the virtual objects are displayed in place of (e.g., replace the display of) a portion of a camera's live view (e.g., a portion of the physical environment captured in the live view). In some embodiments, at least some of the virtual objects and content are projected onto physical surfaces or open space in the physical environment and are visible through a pass-through portion of the display generation component (e.g., as part of a camera view of the physical environment or visible through a transparent or semi-transparent portion of the display generation component, etc.). In some embodiments, at least some of the virtual objects and content are displayed to overlay a portion of the display and block the view of at least a portion of the physical environment that is visible through the transparent or semi-transparent portion of the display generation component.

[0106] In some embodiments, the display generation component displays different views of the three-dimensional environment according to user input or movement that changes the virtual position of the viewpoint of the currently displayed view of the three-dimensional environment relative to the three-dimensional environment. In some embodiments, when the three-dimensional environment is a virtual environment, the viewpoint moves according to a navigation or movement request (e.g., an air hand gesture, a gesture made by moving one part of a hand relative to another part of a hand, etc.) without requiring movement of the user's head, torso, and / or the display generation component within the physical environment. In some embodiments, movement of the user's head and / or torso relative to the physical environment (e.g., by the user holding a display generation component or wearing an HMD, etc.) and / or movement of the display generation component or other location-sensing element of the computer system, etc., causes a corresponding movement of the viewpoint relative to the three-dimensional environment (e.g., with a corresponding change in direction of movement, distance of movement, speed of movement, and / or orientation, etc.), resulting in a corresponding change in the currently displayed view of the three-dimensional environment. In some embodiments, when a virtual object has a preset spatial relationship with respect to the viewpoint, movement of the viewpoint relative to the three-dimensional environment causes the virtual object to move relative to the three-dimensional environment while the position of the virtual object within the field of view is maintained (e.g., the virtual object is said to be head-locked). In some embodiments, the virtual object moves relative to the three-dimensional environment when the user is body-locked and moves as a whole within the physical environment (e.g., carrying or wearing the display generating components and / or other location sensing components of the computer system), but does not move within the three-dimensional environment in response to movement of the user's head (e.g., the display generating components and / or other location sensing components of the computer system rotate around the user's fixed location within the physical environment).

[0107] In some embodiments, the views of the three-dimensional environment shown in FIGS. 7A-7P include representation(s) of the user's hand(s), arm(s), and / or wrist(s). In some embodiments, the representation(s) are part of a representation of the physical environment provided via a display generation component. In some embodiments, the representation(s) are not part of the representation of the physical environment but are displayed within the three-dimensional environment independently of the view of the three-dimensional environment (e.g., by one or more cameras pointed at the user's hand(s), arm(s), and wrist(s)). In some embodiments, the representation(s) include camera images captured by one or more cameras of the computer system(s) or stylized versions of the arm(s), wrist(s), and / or hand(s) based on information captured by various sensors. In some embodiments, the representation(s) replace the display of, overlap with, or block the view of a portion of the representation of the physical environment. In some embodiments, when the display generation component does not provide a view of the physical environment but provides a complete virtual environment (e.g., no camera view or transparent pass-through portions), a real-time visual representation (e.g., a stylized representation or segmented camera image) of one or both arms, wrists, and / or hands of the user may still be displayed within the virtual environment.

[0108] 7A-7C are block diagrams illustrating interaction with user interface objects within a computer-generated three-dimensional environment shared between two or more users, according to some embodiments.

[0109] In some embodiments, the computer system allows multiple users (e.g., a first user 7102, a second user 7002, another user, etc.) to have the right to access a first user interface object (e.g., a first user interface object 7016, another user interface object, a control panel, a virtual menu, a media object, etc.) displayed in a three-dimensional environment (e.g., a three-dimensional environment 7015, another virtual environment or an augmented reality environment, etc.), but prevents a user (e.g., the first user 7102, or another user different from the first user 7102, etc.) from accessing the first user interface object while another user (e.g., the second user 7002, another user different from the second user 7002, etc.) is interacting with the first user interface object. When displaying a view of a three-dimensional environment including a first user interface object via a first display generation component (e.g., display generation component 7200, a different type of display generation component such as an HMD, etc.) used by a first user (e.g., first user 7102), the computer system detects a first user input (e.g., gaze input, hand movement, a combination of gaze input and user hand movement, etc.) directed at the first user interface object. In response to detecting the first user input, the computer system either performs a first action on the first user interface object corresponding to the first user input (e.g., moving the first user interface object or a representation thereof toward the representation 7202′ of the first user's hand 7202, performing a function associated with the first user interface object that changes the three-dimensional environment (e.g., causing virtual content within the three-dimensional environment to be displayed or dismissed, changing other virtual content within the three-dimensional environment, etc.), depending on whether the first user interface object is currently available for interaction with the first user, or displays a visual indication that the first user interface object is not available for interaction with the first user and cancels performance of the first action.The computer system provides a visual indication and aborts execution of the first action pursuant to a determination that another user (e.g., second user 7002) currently controls the first user interface object (e.g., another user is interacting with the first user interface object, interacting with the first user interface object in a manner that precludes simultaneous interaction by the first user, and / or locking the first user interface object for the type of action the first user intends to perform, etc.). In some embodiments, displaying the visual indication includes moving the first user interface object within a view of the three-dimensional environment shown to the first user to maintain a preset distance between the first user interface object and a close representation of the first user's hand. In some embodiments, displaying the visual indication includes changing the visual appearance of the first user interface object within a view of the three-dimensional environment shown to the first user (e.g., a view from user 7102's side as shown in FIG. 7C ). In some embodiments, when the first user interface object is released by the controlling user to the first user (e.g., via a throw gesture, a toss gesture, etc.), the computer system rotates the first user interface object so that the first user interface object is displayed in a preset orientation (e.g., with the content or control side facing the first user 7102) relative to the viewpoint of the currently displayed view of the three-dimensional environment shown to the first user. In some embodiments, the computer system controls the first user's access to the first user interface object by displaying a representation of the first user interface object at or near a representation of some part of the first user (e.g., a representation of the first user's 7102's hand 7202, within arm's reach of the virtual position of the user's face, etc.).

[0110] 7A-7C , according to some embodiments, a three-dimensional environment 7015 is shared between a first user 7102 and a second user 7002 in response to a request initiated by one of users 7102 and 7002 using a computer system controlled by the first user and accepted by the other of users 7102 and 7002 using a computer system controlled by the other user. In some embodiments, both users receive and accept a request to share the three-dimensional environment from a computer system used by a third user using their respective computer systems. In some embodiments, both users send a request to share the three-dimensional environment to a server using their respective computer systems, and the user requests are accepted by the server. When sharing a computer-generated three-dimensional environment, the location and orientation of the users, as well as the location and orientation of their respective heads, eyes, hands, arms, and / or wrists, are captured in real time or periodically by sensors (e.g., cameras, motion sensors, etc.), and the location and orientation data is provided to one or both of the computer systems controlled by the users and / or a server in communication with the computer systems. The location data is used by the computer system and / or server to determine the respective positions and orientations of the users within the computer-generated three-dimensional environment, as well as the respective positions and orientations of the users' respective heads, eyes, hands, arms, and / or wrists, and correspondingly, the respective positions of the users' representations including their respective heads, arms, hands, and / or wrists within views of the three-dimensional environment provided via different display generation components associated with the users, as well as the fields of view and viewpoints of the views of the three-dimensional environment provided via different display generation components associated with the users.

[0111] In some embodiments, when two or more users share a computer-generated environment (e.g., a virtual conference call, a chat session, a multiplayer game, a shared computer-generated experience (e.g., a group meditation, exercise, game, collaboration, etc.), etc.), they may desire to control and / or manipulate the same user interface object (e.g., a virtual ball, a virtual control panel, a document or media content, a virtual menu, a user interface, etc.) present within the computer-generated environment. This may make it difficult for the computer system to consistently prioritize different users' actions on the user interface object, and the resulting changes in the three-dimensional environment may be confusing to the users. As disclosed herein, the computer system provides visual feedback in response to a first user's 7102 attempting to interact with a first user interface object 7016 already under the control of a second user 7002 in the environment by altering a set of appearance characteristics of the first user interface object 7016 in the view 7015-1 of the environment presented to the first user 7102, thereby reducing conflicts between the users' actions and reducing user confusion when the users interact with the first user interface object 7016. In some embodiments, the first user interface object 7016 presented in the view 7015-2 of the three-dimensional environment shown to the second user 7002 controlling the first user interface object is not changed as a result of the first user's attempt to interact with the first user interface object, causing no confusion to the second user 7002 when the second user 7002 interacts with the first user interface object 7016.

[0112] 7A illustrates an exemplary physical environment (e.g., scene 105, another indoor or outdoor physical environment, etc.). In some embodiments, as illustrated in FIG. 7A , two or more users (e.g., user 7102, user 7002, etc.) are present in the same physical environment. A first user 7102 views a first view 7015-1 of a three-dimensional environment 7015 (e.g., an augmented reality environment, a virtual environment, etc.) through a first display generation component (e.g., display generation component 7200, another type of display generation component such as an HMD used by the first user, etc.). A second user 7002 views a second view 7015-2 of the same three-dimensional environment 7015 through a second display generation component (e.g., display generation component 7100, another type of display generation component such as an HMD used by the second user, etc.). In some embodiments, the three-dimensional environment 7015 (e.g., labeled 7015-1 when presented via a first display generation component 7200 and labeled 7015-2 when presented via a second display generation component 7100) is an environment for a shared computer-generated experience, a communication session, an application environment, a game, a movie, or the like.

[0113] In some embodiments, the first user 7102 and the second user 7002 are not necessarily located in the same physical environment at the same time, but may be located separately in two different physical environments. In some embodiments, the three-dimensional environment 7015 includes a representation of the physical environment of the first user but not the second user, and the first user and the second user have a shared experience in the three-dimensional environment based on the physical environment of the first user. In some embodiments, the three-dimensional environment 7015 includes a representation of the physical environment of the second user but not the first user, and the first user and the second user have a shared experience in the three-dimensional environment based on the physical environment of the second user. In some embodiments, the three-dimensional environment 7015 includes a representation of a third physical environment that is not the physical environment of the first user or the second user, and the first user and the second user have a shared experience in the three-dimensional environment based on the third physical environment (e.g., the physical environment of a third user participating in the shared experience, another physical environment not associated with the users or associated with a user not participating in the shared experience, etc.). In some embodiments, the three-dimensional environment 7015 includes a virtual three-dimensional environment, and the first user and the second user have a shared experience within the virtual three-dimensional environment. In some embodiments, the positions and movements of the first user and the second user within their respective physical environments (e.g., the same physical environment, different physical environments, etc.) are mapped to positions and movements within the same three-dimensional environment (e.g., using the same mapping relationship, different mapping relationships, etc.), but the appearance of the three-dimensional environment can be adjusted (e.g., with different wallpaper, color schemes, different virtual furniture, etc.) to accommodate the individual users within their views of the three-dimensional environment.

[0114] In some embodiments, the computer system determines that the three-dimensional environment is at least partially shared between the first user 7102 and the second user 7002 according to a determination that at least a spatial portion of the environment 7015 is shared (e.g., a spatial portion of the environment that corresponds to the living room but not the kitchen, a spatial portion of the environment that corresponds to a portion of the physical space in front of the first user but not a portion of the physical space behind the first user, etc.). In some embodiments, the computer system determines that the three-dimensional environment is at least partially shared between the first user and the second user according to a determination that at least a spatial portion of the environment 7015 is shared during at least a period of time (e.g., during a communication session between the first user and the second user, in the morning, during work hours, when both users are online, etc.). In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user according to a determination that the objects in the environment 7015 are fully or partially shared (e.g., visible and accessible at the same time, visible but not accessible at the same time, visible but not accessible when controlled by another person (e.g., the other person can see the object or cannot see it, etc.)). In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user according to a determination that at least a portion of the three-dimensional environment 7015 (e.g., the portion shown in the first view 7015-1 of the three-dimensional environment, another portion of the three-dimensional environment 7015, etc.) is displayed for viewing by both the first user and the second user simultaneously. In some embodiments, the computer system determines that the three-dimensional environment 7015 is at least partially shared between the first user and the second user according to a determination that some or all of the virtual objects in the three-dimensional environment are displayed in the three-dimensional environment to both the first user and the second user simultaneously.

[0115] 7B and 7C , a computer system displays, via a first display generation component 7200, a first view 7015-1 of a three-dimensional environment 7015 that is at least partially shared between a first user 7102 and a second user 7002, and substantially simultaneously (e.g., adjusted for network delays, processing time delays, etc.), the computer system, or another computer system in communication with the computer system, displays a second view 7015-2 of the three-dimensional environment 7105 via a second display generation component 7100. According to some embodiments, both the first view 7015-1 and the second view 7015-2 include at least a first portion of the three-dimensional environment (e.g., distinct portions corresponding to the same portion of a physical environment represented in the three-dimensional environment, distinct portions corresponding to the same portion of a virtual environment of the three-dimensional environment, etc.). In some embodiments, a first portion of the three-dimensional environment is optionally shown from different viewing angles in a first view 7015-1 and a second view 7015-2 of the three-dimensional environment 7105 (e.g., based on the respective spatial relationships between the first user and its physical environment, and / or the respective spatial relationships between the first user and its physical environment, etc.).

[0116] In some embodiments, the first view 7015-1 has a first viewpoint having a position corresponding to the first user's 7102's current location within the first user's 7102's physical environment, which position moves within the three-dimensional environment 7015 according to the first user's 7102's movement within the first user's 7102's physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, the second view 7015-2 has a second viewpoint having a position in the three-dimensional environment 7015 corresponding to the second user's 7002's current location within the second user's 7002's physical environment, which position moves within the three-dimensional environment 7015 according to the second user's 7002's movement within the second user's physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, the viewpoint of a currently displayed view of the three-dimensional environment 7015 shown via an individual display generating component (e.g., first display generating component 7200, second display generating component 7100, etc.) has a position within the three-dimensional environment 7015 that corresponds to the current location of the individual display generating component, and that position moves within the three-dimensional environment 7015 according to movement of the individual display generating component within its physical environment (e.g., scene 105, another physical environment, etc.). In some embodiments, the viewpoint of a currently displayed view of the three-dimensional environment 7015 shown via an individual display generating component (e.g., first display generating component 7200, second display generating component 7100, etc.) has a position within the three-dimensional environment that corresponds to the current location of one or more cameras associated with the individual display generating component, and that position moves within the three-dimensional environment 7015 according to movement of one or more cameras associated with the individual display generating component within its physical environment (e.g., scene 105, another physical environment, etc.).In the examples shown in Figures 7A-7C, the first view 7015-1 and the second view 7015-2 appear to have the same perspective, but it should be understood that the respective views and their corresponding perspectives shown via the first display generation component 7200 and the second display generation component 7100 are determined separately and independently based on the spatial relationships and movements that exist in the respective physical environments of the first display generation component (and first user) and the second display generation component (and second user), and need not be exactly the same at a given time.

[0117] In Figures 7B and 7C, the first view 7015-1 and the second view 7015-2 of the three-dimensional environment 7015 include one or more user interface objects (e.g., first user interface object 7016, second user interface object 7018, other user interface objects, virtual three-dimensional objects, etc.), and optionally one or more surfaces (e.g., a representation 7004' or 7004'' of wall 7004, a representation 7006' or 7006'', a representation 7008' or 7008'', a virtual surface such as a virtual wall, a virtual screen, a virtual window, a virtual landscape, etc.), and / or representations of one or more physical objects (e.g., a representation 7014' or 7014'' of physical object 7014 in physical environment 7015, representations of other physical objects in another physical environment represented in the three-dimensional environment 7014, etc.). In some embodiments, the first view 7015-1 and the second view 7015-2 do not include a representation of a physical environment, but rather include a virtual three-dimensional environment (e.g., a virtual conference room, a gaming environment, a virtual experience, a virtual stadium, etc.).

[0118] In some embodiments, the first user interface object 7016 is a representation of an application, and interaction with the first user interface object that meets predetermined criteria causes the computer system to launch the application in a three-dimensional environment or execute an application function of the application. In some embodiments, the first user interface object 7016 is a user interface that includes multiple user interface objects (e.g., selectable avatars, selectable menu items, selectable device controls, selectable content items, slider controls, buttons, etc.). In some embodiments, the first user interface object 7016 is a virtual three-dimensional object that can be manipulated (e.g., transformed, separated into parts, rotated, moved, etc.) within the three-dimensional environment according to the movement of a user's hand within the physical environment. In some embodiments, the first user interface object 7016 is a single control or control panel that includes multiple controls corresponding to different functions or actions. In some embodiments, the first user interface object 7016 is an information item, a notification, an alert, etc. In some embodiments, the first user interface object 7016 is a media item, a document, etc.

[0119] 7B and 7C, the first view 7015-1 includes a representation 7202′ of the hand 7202 of the first user 7102 and a representation 7028′ of the hand 7028 of the second user 7002, and the second view 7015-2 includes a representation 7202″ of the hand 7202 of the first user 7102 and a representation 7028″ of the hand 7028 of the second user 7002. In the scenario shown in FIGS. 7B and 7C, the second user 7002 controls the first user interface object 7016, excluding simultaneous interaction between the first user 7102 and the first user interface object 7016. For example, in some embodiments, when the first user interface object 7016 is under the control of the first user 7002, the first user interface object 7016 is displayed at a position within the three-dimensional environment 7015 that corresponds to the location of the hand 7028 of the second user 7002 within the physical environment of the second user 7002. In some embodiments, when the first user interface object 7016 is under the control of the second user 7002, a representation of the first user interface object 7016 is displayed at a position within the three-dimensional environment 7015 that corresponds to the location of the hand 7028 of the second user 7002 within the physical environment of the second user 7002, and the first user interface object 7016 is displayed at another position away from the position of the representation of the first user interface object 7016. In this example, a second user 7002 is controlling a first user interface object 7016, which is displayed at a position within the three-dimensional environment 7015 that corresponds to the location of the second user's hand 7028.In some embodiments, when the first user interface object 7016 is under the control of the second user 7002, the first user interface object 7016 is oriented within the three-dimensional environment 7015 so that a preset surface of the first user interface object 7016 (e.g., front surface A, content presentation surface, interactive surface, etc.) faces a viewpoint corresponding to a currently displayed second view 7015-2 of the three-dimensional environment 7105 (e.g., a view shown to the second user 7002 controlling the first user interface object 7016, a view displayed by the second display generation component 7100, etc.). In some embodiments, the first user interface object 7016 can be reoriented within the three-dimensional environment by the second user 7002 controlling the first user interface object 7016 so that a preset surface of the first user interface object 7016 faces toward a viewpoint corresponding to a currently displayed first view 7015-1 of the three-dimensional environment (e.g., a view shown to a first user 7102 not currently controlling the first user interface object 7016, a view displayed by the first display generation component 7200, etc.). In some embodiments, when the first user interface object 7016 is under the control of the second user 7002 and is not shared with the first user 7102 (e.g., even if the content display side of the first user interface object 7016 is within the first view 7015-1 of the three-dimensional environment presented to the first user 7102 by the first display generation component 7200), at least a portion of the content on the first user interface object 7016 is shown only in the second view 7015-2 of the three-dimensional environment and not in the first view 7015-1 of the three-dimensional environment.In some embodiments, the second user 7002 can make the hidden content of the first user interface object 7016 visible to the first user 7102 by reorienting the first user interface object 7016 so that the content-presenting side of the first user interface object 7016 faces away from the viewpoint of the second view 7015-2.

[0120] 7B and 7C , both the first view 7015-1 and the second view 7015-2 include a respective representation of the first user's hand 7202 (e.g., representation 7202′ or 7202″) and a respective representation of the second user's hand 7028 (e.g., representation 7028′ or 7028″). In some embodiments, the computer system displays the hand representations based on a camera view of the users' hands. In some embodiments, the computer system provides a view of the hand representations through a transparent portion of the display generation component(s). In some embodiments, the computer system generates a stylized representation of the users' hands based on sensor information received from one or more sensors located in the first user's and second user's physical environment(s). In some embodiments, the position and configuration of the representation of the user's hand(s) varies according to the location(s) and configuration(s) of the user's hand(s) in the user's physical environment(s). In some embodiments, the computer system displays hand representations based on camera views of the users' hands. In some embodiments, the computer system displays hand representations of one user but not the other user at a given time. For example, if a second user 7002 is controlling a first user interface object 7016, a representation of the second user's hand 7028 is optionally displayed only in the second view 7015-2 shown to the second user 7002 and not in the first view 7015-1 shown to the first user 7102. In another example, the user's hand representations may move in and out of the fields of view provided via the respective display generation components due to movement of the first user and / or second user (and / or their respective display generation components or cameras, etc.) within their respective physical environments.

[0121] 7B and 7C , within a second view 7015-2 of a three-dimensional environment displayed via a second display generation component 7100 used by a second user 7002 to control a first user interface object 7016, the first user interface object 7016 is displayed with a first set of appearance characteristics (e.g., the first user interface object's normal appearance as displayed to the second user by the second display generation component (e.g., a first shape, a first size, a first color, a first opacity, a first saturation, a first brightness, etc.)). The first user interface object 7016 maintains the first set of appearance characteristics under the control of the second user 7002 regardless of whether the first user 7102 attempts to access the first user interface object 7016 using a separate movement or input directed at the first user interface object. The first user interface object 7016 can change its appearance in a distinct manner according to an interaction between the second user 7002 and the first user interface object 7016 via a computer system used by the second user 7002. These changes in appearance caused by the interaction between the second user 7002 and the first user interface object 7016 are, optionally, shown in both the first view 7015-1 and the second view 7015-2 at any given time that the change occurs.

[0122] In FIG. 7B , while the first user interface object 7016 is under the control of the second user 7002, if the first user 7002 is not attempting to access or gain control of the first user interface object 7016 (e.g., via movement of a part of the user, such as the user's hand, via gaze input, via an air gesture, via a gesture involving movement of a part of the hand relative to another part of the hand, via input provided via a control object, etc.), the first user interface object 7016 is displayed in the first view 7015-1 with the same first set of appearance characteristics as in the second view 7015-2 of the three-dimensional environment (optionally, from a different perspective and / or with editing of hidden content, etc.). If both the first view and the second view capture a portion of the three-dimensional environment corresponding to a location in physical space including the first user's hand 7202, the movement of the first user's hand 7202 within the first user's 7102 physical environment can be represented in both the first view 7015-1 and the second view 7015-2.

[0123] 7C , the computer system detects a first user input provided by the first user 7102 directed toward the first user interface object 7016. For example, in some embodiments, the computer system detects movement of a part of the first user 7102 (e.g., the user's hand 7202, another hand of the first user, etc.) to a location in the first user's 7102's physical environment that corresponds to the position of the first user interface object 7016 in the three-dimensional environment 7015. In some embodiments, the computer system detects a gaze input directed toward the first user interface object 7016 and a control input (e.g., a finger movement gesture, an air gesture, an input provided by a controller, etc.) that is detected along with the gaze input. In the example shown in FIG. 7C , the first user input is a movement of the first user's hand 7202 to a location that corresponds to the position of the first user interface object 7016, optionally accompanied by a movement or posture to grasp the first user interface object 7016 in the three-dimensional environment. In some embodiments, a representation of the movement, position, and / or posture of the hand 7202 of the first user 7102 is shown in both the first view 7015-1 and the second view 7015-2. In some embodiments, a representation of the movement, position, and / or posture of the hand 7202 of the first user 7102 is shown only in the first view 7015-1 and not in the second view 7015-2. In some embodiments, by not showing the movement, position, and / or posture of the hand 7202 of the first user 7102 in the second view 7015-2, the computer system used by the second user 7002 reduces confusion for the second user 7002 when the second user 7002 interacts with the first user interface object 7016.

[0124] 7C , in response to detecting a first user input directed at the first user interface object 7016, and following a determination that the second user 7002 is currently interacting with the first user interface object (e.g., controlling the first user interface object 7016, controlling the first user interface object to the exclusion of a requested interaction by the first user 7102, etc.), the computer system displays a visual indication that the first user interface object 7016 is unavailable for interaction with the first user 7102. In some embodiments, displaying the visual indication includes altering at least one of an appearance of the first user interface object 7016 or a position of the first user interface object 7016 within the first view 7015-1 of the three-dimensional environment 7015.

[0125] In some embodiments, the computer system determines that the second user 7002 is currently interacting with the first user interface object 7016 according to a determination that the first user interface object 7016 has a predetermined spatial relationship with respect to the second user's 7002 virtual position within the three-dimensional environment (e.g., the first user interface object 7016 is within a representation of the second user's palm or hand 7028, the first user interface object 7016 is within the second user's private space within the first view 7015-1 of the three-dimensional environment, etc.). In some embodiments, the computer system determines that the second user 7002 is currently interacting with the first user interface object 7016 according to a determination that the second user 7002 is controlling, selecting, moving, modifying, and / or otherwise interacting with the first user interface object 7016 through the computer system displaying the second view 7015-2 of the three-dimensional environment via the second display generation component 7100.

[0126] In some embodiments, to display a visual indication in the first view 7015-1 of the three-dimensional environment 7015 that the first user interface object 7016 is unavailable for interaction with the first user 7102, the computer system displays the first user interface object 7016 with a second set of appearance characteristics (e.g., a second shape, a second size, a second color, a second opacity, a second saturation, a second brightness, etc.) that are different from the first set of appearance characteristics (e.g., the second set of appearance characteristics provide a visual indication that the first user interface object is now controlling the second user and is unavailable for interaction with the first user). For example, the first user interface object 7016 shown in the first view 7015-1 of FIG. 7C is semi-transparent compared to that shown in the second view 7015-2 of FIG. 7C. In some embodiments, to display a visual indication in the first view 7015-1 of the three-dimensional environment 7015 to indicate that the first user interface object 7016 is unavailable for interaction with the first user 7102, the computer system moves the first user interface object 7016 out of the way when the first user 7102 attempts to grasp it. In some embodiments, the first user interface object 7016 maintains its appearance and / or position in the second view 7015-2 displayed to the second user 7002, as the visual indication need only be displayed to the first user 7102.In some embodiments, if the first user input provided by the first user 7102 corresponds to a request to perform a first action with respect to the first user interface object 7016, the computer system does not perform the first action with respect to the first user interface object 7016 in accordance with a determination that the second user 7002 is currently interacting with the first user interface object 7016 (e.g., controlling the first user interface object 7016, controlling the first user interface object 7016 excluding the interaction requested by the first user 7102, etc.). For example, in some embodiments, the computer system does not show the first user interface object 7106 being grasped by the representation 7202′ of the first user's hand 7202. In some embodiments, the computer system does not show a ghost image or another representation of the first user interface object 7016 moving within the representation 7202′ of the first user's hand 7202.

[0127] In some embodiments, in response to detecting a first user input directed at the first user interface object 7106 and in accordance with a determination that the second user 7002 is not currently interacting with the first user interface object 7016, the computer system performs a first action on the first user interface object in accordance with the first user input. In some embodiments, performing the first action includes indicating that the first user interface object 7016 is being grasped or moved by the first user 7102 in accordance with the first user input (e.g., moved toward a virtual position of the first user 7102 in the three-dimensional environment, moved in accordance with the movement of the first user input, etc.). In some embodiments, performing the first action includes showing a ghost image or other representation of the first user interface object 7016 being grasped and / or moved within the representation 7202′ of the first user's hand 7202. In some embodiments, following a determination that the second user 7002 was not interacting with the first user interface object 7106 when the first user input from the first user 7102 was detected, the first user interface object 7016 continues to be displayed with the first set of appearance characteristics (e.g., in its original location, or with a representation of the first user's hand, etc.).

[0128] In some embodiments, when the first user 7102 attempts to grasp or otherwise interact with the first user interface object 7016 while the second user 7002 is interacting with the first user interface object, the computer system changes the appearance of the first user interface object, such as fading out the first user interface object in the first view 7015-1 displayed to the first user 7102 as the first user 7102 attempts to grasp the first user interface object 7016. For example, the computer system modifies at least one of the first set of appearance characteristics of the first user interface object 7016 (e.g., increasing the transparency level, decreasing saturation, decreasing opacity, blurring, darkening, decreasing resolution, reducing size, etc.) to, optionally, reduce the visual prominence of the first user interface object 7016 in the first view 7015-1 of the three-dimensional environment while maintaining the appearance of the surrounding environment of the first user interface object 7016 (e.g., not changing the appearance and / or visual prominence of the surrounding environment). In some embodiments, in response to detecting that the first user 7102 has stopped attempting to interact with the first user interface object 7016, the computer system restores at least one (e.g., some, all, etc.) of the first set of appearance characteristics of the first user interface object that were changed in response to the first user's attempt to grasp or interact with the first user interface object (e.g., to the level that existed immediately before detecting the first user input or before the change was made in response to detecting the first user input, etc.) in order to restore the visual prominence of the first user interface object.

[0129] In some embodiments, when the first user interface object 7016 is moved away from a position corresponding to the location of the first user's hand 7202 (e.g., moved away from the representation 7202′ of the hand 7202 in the three-dimensional environment 7015 due to an action of the second user 7002 and / or in accordance with another event that occurs within the three-dimensional environment (e.g., an event unrelated to an interaction attempt by the first user 7102), etc.), the computer system restores at least one (e.g., some, all, etc.) of the first set of appearance characteristics of the first user interface object that was changed in response to the first user's attempt to grasp or otherwise interact with the first user interface object (e.g., to a level that existed immediately before detecting the first user input or before the change was made in response to detecting the first user input), in order to restore the visual prominence of the first user interface object.

[0130] In some embodiments, after a visual indication that the first user interface object 7016 is unavailable for interaction with the first user 7102 is displayed in the first view 7015-1, the computer system continues to display the visual indication until the computer system detects that the second user 7002 is no longer interacting with the first user interface object and / or has relinquished control of the first user interface object, such that the first user interface object becomes available for interaction with the first user 7102. In some embodiments, after a visual indication that the first user interface object 7016 is unavailable for interaction with the first user 7102 is displayed in the first view 7015-1, the computer system continues to display the visual indication for a preset period of time (e.g., 10 seconds, 5 seconds, etc.) after the first user stops attempting to interact with the first user interface object 7106 via the first user input or another input.

[0131] In some embodiments, the first user interface object 7016 can be sent to a position corresponding to the location of the first user (e.g., a position corresponding to the hand 7202 of the first user 7102, a position corresponding to a private space surrounding the first user 7102, etc.) according to a gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by the second user 7002 controlling the first user interface object 7016. In some embodiments, the first user interface object 7016 rotates (e.g., reorients, changes its facing direction, etc.) while moving from a first position to a second position within the three-dimensional environment 7015 as a result of the gesture input provided by the second user 7002. In some embodiments, the first user interface object 7016 can also be sent to a position corresponding to the location of the second user 7002 according to gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by the first user 7102 after the first user 7102 gains control of the first user interface object 7016. In some embodiments, the first user interface object 7016 rotates (e.g., changes orientation, changes facing direction, etc.) while moving from the second position to the third position within the three-dimensional environment 7015 as a result of the gesture input provided by the first user 7102. In some embodiments, the first user interface object 7106 rotates so that its content-presenting or interactive side faces the side receiving the first user interface object.

[0132] In some embodiments, the first user interface object 7016 can be sent to a position within the three-dimensional environment where the first user interface object can be better viewed by both the first user and the second user (e.g., displayed at the center of the three-dimensional environment 7015, displayed at a position corresponding to a wall of the physical environment 105, displayed on a virtual surface within the three-dimensional environment 7015, etc.) in response to a gesture input (e.g., a toss gesture, a throw gesture, a push gesture, etc.) provided by a user controlling the first user interface object. In some embodiments, the first user interface object has an orientation that allows both the first user and the second user to view its content and / or interactive side when it reaches the position within the three-dimensional environment, and / or rotates (e.g., re-facing, changing facing direction, etc.) while moving to a position within the three-dimensional environment such that it has a predetermined spatial relationship (e.g., overlapping, parallel, diagonal, perpendicular, upright, etc.) with respect to a surface (e.g., a representation of a wall surface, a table surface, a virtual surface, a virtual screen, a virtual tabletop, etc.) at the position in the three-dimensional environment.

[0133] In some embodiments, the computer system alters the position of the first user interface object 7016 within the first view 7015-1 of the three-dimensional environment as a visual indication that the first user interface object 7016 is unavailable for interaction with the first user 7102. In some embodiments, altering the position of the first user interface object within the first view 7015-1 of the three-dimensional environment includes moving the first user interface object 7016 from its original position to maintain at least a predetermined distance between the first user interface object and the representation 7202′ of the hand 7202 of the first user 7102 that provided the first user input (e.g., the first user interface object appears to move in one or more directions to avoid the representation 7202′ of the hand 7202 of the first user 7102 attempting to grasp the first user interface object). In some embodiments, the movement of the first user interface object 7016 is accompanied by changes made to the appearance of the first user interface object (e.g., the first user interface object appears to fade or darken while being moved to prevent the representation 7202′ of the first user 7102's hand from getting too close to itself).

[0134] In some embodiments, when the first user interface object 7016 is not under the control of the second user 7002 and is available for interaction with the first user 7102, the computer system moves the first user interface object 7016 toward a representation 7202' of the first user's hand 7202 in the first view 7015-1 of the three-dimensional environment 7015, and optionally also in the second view 7015-2 of the three-dimensional environment.

[0135] In some embodiments, the first user input provided by the first user 7102 includes a predetermined selection gesture (e.g., is, includes, starts with, ends with, etc.) (e.g., the selection gesture is a pinch gesture involving touching down of the index finger on the thumb of the same hand (optionally followed by lifting the index finger from the thumb, or flicking the wrist connected to the hand, or translating the entire hand, etc.), a gesture involving pulling the index finger and thumb of the same hand away from each other from a touching position, a pinch gesture, a pinch-and-drag gesture, a pinch-and-flick gesture, etc.). In some embodiments, the computer system, in response to detecting a first user input while the second user 7002 is not interacting with the first user interface object 7016, selects the first user interface object 7016 as the target for a subsequent input received from the first user 7102 (e.g., a drag gesture while a pinch gesture is maintained, a flick gesture while a pinch gesture is maintained, a drag gesture after a predetermined selection gesture has ended, etc.). In some embodiments, in conjunction with selecting the first user interface object 7016 as the target for subsequent input received from the first user 7102, the computer system maintains the first user interface object 7016 in a first position within the first view 7015-1 of the three-dimensional environment while displaying a representation of the first user interface object 7106 (e.g., a replica of the first user interface object, a ghost image of the first user interface object, etc.) at a position corresponding to the location of the first user's 7102's hand 7202 (e.g., the first user interface object remains in its original location but can be "remotely" controlled by the first user 7102 according to interaction between the first user 7102 and the representation of the first user interface object).In some embodiments, the representation of the first user interface object is displayed near the representation 7202’ of the hand 7102 of the first user, but does not advance to a position corresponding to the location of the first user's hand until the computer system detects another selection input provided by the first user 7202. In some embodiments, the computer system changes the shape of the representation of the first user interface object according to a determination that the first user 7102 is providing an input that meets the requirements of the selection input, and the change in the shape of the representation of the first user interface object optionally provides visual guidance regarding the requirements for completing the selection input. In some embodiments, the user interaction with the representation of the first user interface object is converted into an interaction with the first user interface object, causing the computer system to perform an operation on the first user interface object according to the interaction between the first user 7102 and the representation of the first user interface object. In some embodiments, the representation of the first user interface object remains displayed at the position of the representation 7202’ of the hand 7202 of the first user, and optionally, the first user 7102 indicates that the first user controls the first user interface object while excluding the interaction of other users sharing the three-dimensional environment with the first user.

[0136] In some embodiments, some or all of the features described above regarding the behavior of the computer system, the first display generation component 7200, and the second display generation component 7100 in FIGS. 7A-7C are equally applicable to other scenarios where the roles of the first user 7102 and the second user 7002 with respect to the first user interface object 7016 are reversed. In such other scenarios, the operations of the computer system and the display generation components used by the first user and the second user may be appropriately reversed in a particular scenario. The features described above still hold, and thus are not repeated herein for the sake of brevity.

[0137] 7D-7F are block diagrams illustrating a method, in accordance with some embodiments, for displaying representations of physical objects in various manners relative to a viewpoint of a currently displayed view of a three-dimensional environment, where the viewpoint moves in accordance with the user's movement within a first physical environment, the representations of the physical objects move in accordance with the movement of the physical objects in a second physical environment different from the first physical environment, and a change in the manner in which the representations are displayed is triggered in response to the spatial relationship between the representations of the physical objects and the viewpoint satisfying a predetermined criterion.

[0138] In some embodiments, the computer system displays a view of the three-dimensional environment 7204 that includes representations of physical objects (e.g., second user 7102, animals, moving drones, etc.) located in a physical environment (e.g., scene 105-b, or another indoor or outdoor physical environment, etc.) different from the physical environment (e.g., scene 105-a, or another indoor or outdoor physical environment, etc.) of the first user 7002 (and first display generation component 7100 used by the first user 7002 to view the three-dimensional environment 7304). The computer system optionally moves a viewpoint corresponding to the currently displayed view of the three-dimensional environment 7304 according to the movement of the first user 7002 within the physical environment (e.g., scene 105-a, or another physical environment, etc.) of the first user 7002 (and / or first display generation component 7100). The computer system determines the position and path of movement of a representation of the physical object (e.g., representation 7102′-a of second user 7102, a representation of another physical object, etc.) in the three-dimensional environment 7204 based on the location and path of movement of the physical object in its physical environment (e.g., scene 105-b, or another physical environment, etc.). The computer system utilizes a first type of correspondence (e.g., mapping and transformation relationships, optionally different mapping and transformation relationships between viewpoints, physical objects, and first users, etc.) between positions in the three-dimensional environment 7304 and locations in individual physical environments (e.g., physical environment 105-a of first user 7002 and first display generation component 7100, the physical object's physical environment (e.g., physical environment 105-b of second user 7102, another physical environment of the physical object, etc.)).Under some conditions (e.g., due to movement of first user 7002 and / or movement of a physical object (e.g., the physical object represented by second user 7102 in this example), etc.), the position(s) of the representation of the physical object, when determined using a first type of correspondence between positions in three-dimensional environment 7304 and locations in the physical environment (e.g., scenes 105-a, 105-b, etc.), is within a threshold distance (e.g., an arm's length, three feet, a user-specified distance, etc.) of the position of the viewpoint of the currently displayed view (e.g., view 7304-a, 7304-a′, etc.) of three-dimensional environment 7304 shown via first display generation component 7100. Under such conditions, the computer system displays the representation of the physical object (e.g., representation 7102′-a in this example) at an adjusted position that is offset from the position determined based on the first type of correspondence (e.g., as shown in FIG. 7F ). In some embodiments, the adjusted position is determined based on a second type of correspondence different from the first type of correspondence to ensure that the adjusted position remains more than a threshold distance from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generation component (e.g., view 7304-a'', a subsequent view shown via the first display generation component 7100, etc.). The computer system continues to use the second type of correspondence to determine the adjusted position of the representation of the physical object (e.g., representation 7102'-a in this example) until the unadjusted position calculated based on the first type of correspondence is more than a threshold distance away from the position of the viewpoint of the currently displayed view of the three-dimensional environment shown via the first display generation component (e.g., view 7304-a'', a subsequent view shown via the first display generation component 7100, etc.).

[0139] In some embodiments, when the computer system provides a view of the three-dimensional environment 7304 to the first user 7002 and the viewpoint position corresponding to the currently displayed view of the three-dimensional environment 7304 is based on the location of the first user's head, body, or eyes within the first user's 7002 physical environment, the computer system sometimes displays representations of other physical objects (e.g., physical objects represented by the second user 7102 in this example, but which may be inanimate or animate objects that do not share the computer-generated environment 7304 with the first user 7002, etc.) at positions corresponding to the locations of the physical objects within their respective physical environments. In some situations, even if there is no risk or possibility of an actual physical collision or uncomfortable spatial proximity between the first user 7002 and other physical objects in the real world, the position of the representation of the physical objects may clash with or be too close (e.g., if not specifically adjusted, otherwise addressed, etc.) to the position of the viewpoint corresponding to the view shown to the first user, making the first user's visual experience within the three-dimensional environment uncomfortable or sometimes irritating to the first user.

[0140] As disclosed herein, a computer system determines a position of a representation of a physical object located in a physical environment other than that of a first user based on a first type of correspondence or mapping relationship between a position in the three-dimensional environment and a corresponding location in the physical environment of the physical object when the position of the representation of the physical object determined based on the first type of correspondence is not within a threshold range of a viewpoint corresponding to a currently displayed view of the three-dimensional environment shown to the first user. This means that when the representation of the physical object is at a certain distance from the virtual position of the viewpoint, movement of the representation of the physical object in the three-dimensional environment can correspond to movement of the physical object in a manner that mimics movement and spatial relationships in the real world, and the representation of the physical object does not violate the first user's sense of personal space. However, when the representation of the physical object is very close to the virtual position of the viewpoint, movement of the representation of the physical object that corresponds to movement of the physical object in the same way (e.g., according to the first type of correspondence or mapping relationship) can cause the representation of the physical object to be displayed at an unreasonable size, overlap with the viewpoint, and / or violate the first user's sense of personal space. Thus, following a determination based on the first type of correspondence or mapping relationship that the representation of the physical object is within a threshold distance from the viewpoint, the computer system uses the second type of correspondence or mapping relationship between the position in the three-dimensional environment and the corresponding location in the physical environment of the physical object to calculate an adjusted position of the representation of the physical object, whereby the representation of the physical object can be displayed in the adjusted position and / or moved to avoid being displayed at an unreasonable size, overlapping with the viewpoint, and / or violating the first user's sense of personal space.

[0141] 7D illustrates a scenario in which two users, e.g., a first user 7002 and a second user 7102, are sharing a computer-generated three-dimensional environment 7304, according to some embodiments. In some embodiments, the first user 7002 is located in a first physical environment 105-a, and the second user 7102 is located in a second physical environment 105-b. In some embodiments, the first physical environment and the second physical environment are part of the same physical environment that may overlap with each other. In some embodiments, the first physical environment and the second physical environment are separate physical environments that do not overlap with each other. In some embodiments, the first physical environment and the second physical environment are, optionally, indoor environments, outdoor environments, one indoor environment and one outdoor environment, a mixture of indoor and outdoor environments, etc. In this example, the first physical environment includes physical surfaces (e.g., walls 7004-a and 7006-a, a floor 7008-a, etc.) and physical objects (e.g., physical object 7010, other physical objects, etc.). The second physical environment includes physical surfaces (e.g., walls 7004-b and 7006-b, floor 7008-b, etc.) and physical objects (e.g., physical object 7014, other physical objects, etc.). A first user 7002 is a user of a first display generation component 7100 via which a first view 7304-a (and subsequent updated first views 7304-a′, 7304-a″, etc.) of a shared three-dimensional environment 7304 is provided. A second user 7102 is a user of a second display generation component 7200 via which a second view 7304-b (and subsequent updated first views 7304-b′, 7304-b″, etc.) of a shared three-dimensional environment 7304 is provided.For purposes of illustration, a first user 7002 moves forward along a straight line 7300 in a first physical environment 105-a, a second user 7102 moves forward along a straight line 7302 in a second physical environment 105-b, a representation 7300' of the straight line 7300 in a second view 7304-b of a three-dimensional environment 7304 passes through the viewpoint of the second view 7304-b, and a representation 7302' of the straight line 7302 in a first view 7304-a of a three-dimensional environment 7304 passes through the viewpoint of the first view 7304-a. In some embodiments, the paths of travel of the first user 7002 and the second user 7102 need not be straight lines; the paths may be any shape and / or have any spatial extent suitable within their physical environments. In some embodiments, both the first user and the second user need not move within separate physical environments. In some embodiments, the viewpoint of the currently displayed view of the three-dimensional environment provided via the individual display generation components may not be stationary and / or may move according to movement of the individual display generation components and / or movement of the individual users of the individual display generation components. In some embodiments, there is no requirement that the three-dimensional environment be a shared environment between the first user and the second user. For example, in some embodiments, from the perspective of the first display generation component 7100, the second user 7102 in this example is simply a representation of a physical object in the second physical environment (e.g., an animal, a drone, a person not using or providing input to the three-dimensional environment, etc.). Similarly, in some embodiments, from the perspective of the second display generation component, the first user 7002 in this example is simply a representation of a physical object in the first physical environment (e.g., an animal, a drone, a person not using or providing input to the three-dimensional environment, etc.). In some embodiments, only one of the display generation components (e.g., the first display generation component, the second display generation component, etc.) is used, and the other display generation components are not present or are not involved in the processes described herein.

[0142] 7D , according to some embodiments, three-dimensional environment 7304 is shared between user 7002 and user 7102 in response to a request initiated by one of users 7002 and 7102 using a computer system controlled by one user and accepted by another of users 7002 and 7102 using a computer system controlled by the other user. In some embodiments, both users receive and accept a request to share the three-dimensional environment from a computer system used by a third user. In some embodiments, both users send a request to share the three-dimensional environment to a server using their respective computer systems, and the users' requests are accepted by the server. When sharing a computer-generated three-dimensional environment, the users' location and orientation, as well as the location and orientation of their respective heads, eyes, hands, arms, and / or wrists, are captured in real time or periodically by sensors (e.g., cameras, motion sensors, etc.), and the location and orientation data is provided to one or both of the computer systems controlled by the users and / or a server in communication with the computer systems. The location data is used by the computer system and / or server to determine the location and orientation of each of the users within the computer-generated three-dimensional environment, as well as the location and orientation of each of the users' heads, eyes, hands, arms, and / or wrists, and correspondingly, the position of each of the users' representations including their respective heads, arms, hands, and / or wrists within views of the three-dimensional environment provided via different display generation components associated with the users, as well as the field of view and viewpoint of the views of the three-dimensional environment provided via different display generation components associated with the users. In some embodiments, the computer-generated environment shared by the users is an environment such as a virtual conference call, a chat session, a multiplayer game, a shared computer-generated experience (e.g., group meditation, exercise, gaming, collaboration, etc.). In some embodiments, the representation of the user is an avatar of the user.In some embodiments, the user's representation is optionally not attached to or supported by a surface within the three-dimensional environment.

[0143] In part (A) of Figure 7D, the computer system displays a first view 7304-a of a three-dimensional environment 7304 via the first display generation component 7100. In the first view 7304-a of the three-dimensional environment, a representation 7102'-a of a second user 7102 is displayed at a position corresponding to the second user's 7102's current location in the second physical environment 105-b. Other objects, such as a virtual path 7306-a and a virtual object 7308-a, are present within the first view 7304-a of the three-dimensional environment. The appearance and display positions of the representation 7102'-a of the second user 7102, the virtual object 7308-a, and the virtual path 7306-a within the first view 7304-a are based on their respective positions within the three-dimensional environment relative to the position of the viewpoint of the currently displayed first view 7304-a of the three-dimensional environment shown via the first display generation component 7100. In some embodiments, representation 7002'-a of first user 7002 is optionally visible in first view 7304-a of the three-dimensional environment at a position corresponding to the virtual position of first user 7002 and / or the viewpoint of first view 7304-a currently displayed in the three-dimensional environment. In this example, as shown in Figure 7D, part (A), the computer system displays movement of representation 7102'-a along representation 7302' of line 7302 toward the virtual position of the viewpoint of first view 7304-a. At the moment shown in Figure 7D, representation 7102'-a is displayed at a position calculated according to a first type of correspondence between a position in three-dimensional environment 7304 and a location in a second physical environment (e.g., scene 105-b, or another physical environment of second user 7102, etc.). Representation 7102'-a of second user 7102 is shown moving towards and approaching the viewpoint of first view 7304-a as second user 7102 moves forward along line 7302 within the second physical environment.

[0144] 7D , the computer system, or another computer system in communication with the computer system, optionally displays a second view 7304-b of the three-dimensional environment 7304 via the display generation component 7200. In the second view 7304-b of the three-dimensional environment, a representation 7002′-b of the first user 7002 is displayed at a position corresponding to the first user's 7002's current location within the first physical environment (e.g., scene 105-a, or another physical environment of the first user). The second view 7304-b of the three-dimensional environment includes other objects, such as a virtual path 7306-b (e.g., the same virtual path as virtual path 7306-a, but viewed from the perspective of the second view 7304-b), a virtual object 7308-b (e.g., the same virtual object as virtual object 7308-a, but viewed from the perspective of the second view 7304-b), etc. The appearance and display position of each of the representation 7002′-b of the first user 7002, the virtual object 7308-b, and the virtual path 7306-b in the second view 7304-b is based on their respective positions in the three-dimensional environment relative to the position of the viewpoint of the currently displayed second view 7304-b of the three-dimensional environment shown via the second display generation component 7200. In some embodiments, the representation 7102′-b of the second user 7102 is visible in the second view 7304-b of the three-dimensional environment at a position corresponding to the virtual position of the second user 7102 and / or the viewpoint of the currently displayed second view 7304-b. In this example, as shown in part (B) of FIG. 7D , the computer system displays movement of the representation 7002′-b along the representation 7300′ of the straight line 7300 toward the virtual position of the viewpoint of the second view 7304-b. At the moment shown in FIG. 7D, representation 7002'-b is displayed at a position calculated according to a first type of correspondence between a position in three-dimensional environment 7304 and a location in a first physical environment (e.g., scene 105-a, or another physical environment of the first user).Representation 7002'-b of first user 7002 is shown moving toward and approaching the virtual position of viewpoint of second view 7304-b as first user 7002 moves forward along line 7300 within the first physical environment.

[0145] 7E illustrates a point in time when either or both of first user 7002 and second user 7102 have moved within their respective physical environments such that the respective positions of the first user and second user within three dimensional environment 7304, calculated according to a first type of correspondence (e.g., a first type of correspondence between positions within three dimensional environment 7304 and locations in the first physical environment, a first type of correspondence between positions within three dimensional environment 7304 and locations in the second physical environment, etc.), are at a respective preset threshold distance from each other within three dimensional environment 7304. In some embodiments, at this point in time, as shown in portion (A) of FIG. 7E , the respective positions of representation 7102′-a of second user 7102 and the position of the viewpoint of updated first view 7304-a′, calculated according to the first type of correspondence between positions within three dimensional environment 7304 and locations in the second physical environment, are at a first threshold distance from each other within three dimensional environment 7304. 7E , the respective positions of representation 7002′-b of first user 7002 and the position of the viewpoint of updated second view 7304-b′, calculated according to a first type of correspondence between positions in three-dimensional environment 7304 and locations in the first physical environment, are at a second preset threshold distance from each other in the three-dimensional environment (e.g., the same as the first preset threshold distance, a different preset threshold distance, etc.). In some embodiments, the first threshold distance is different from the second threshold distance depending on the personal settings and other characteristics (e.g., size, shape, posture, activity, etc.) of the first user and the second user, respectively.

[0146] 7E , in response to detecting movement of the first user 7002 and the second user 7102 within their / her physical environment, the computer system displays an updated first view 7304-a′ of the three-dimensional environment with a viewpoint that is moved according to the movement of the first user 7002 within the first physical environment. In some embodiments, the viewpoint of the updated first view 7304-a′ is stationary within the three-dimensional environment if the first user 7002 and / or the first display generation component 7100 has not moved within the first physical environment. In some embodiments, in response to a determination that an individual position of the representation 7102'-a of the second user 7102 in the three-dimensional environment, calculated based on the second user's current location in the second physical environment according to the first type of correspondence, is greater than or equal to a first preset threshold distance from an individual position in the three-dimensional environment corresponding to a viewpoint associated with the updated first view 7304-a' of the three-dimensional environment, the computer system displays the representation 7102'-a at a first display position within the updated first view 7304-a' of the three-dimensional environment, the first display position being the individual position of the representation 7102'-a in the three-dimensional environment.

[0147] In some embodiments, the first preset threshold distance is an arm's length, a preset radius of a personal space for the first user 7002 within the three-dimensional environment 7304, and is defined by a preset bounding surface surrounding the virtual position of the first user 7002 within the three-dimensional environment (e.g., a virtual surface of a representation of the first user 7002, or a bounding box surrounding the virtual position of the first user 7002).

[0148] 7E , in response to detecting movement of the second user 7102 within the second user's 7102 physical environment, the computer system of the second user 7102 displays an updated second view 7304-b′ of the three-dimensional environment 7304 having a viewpoint that moves in accordance with the movement of the second user 7102 within the second physical environment. In some embodiments, the viewpoint of the updated second view 7304-b′ is stationary within the three-dimensional environment if the second user 7102 and / or the second display generation component 7200 has not moved within the second physical environment. In some embodiments, in response to a determination that an individual position of the representation 7002'-b of the first user 7002 in the three-dimensional environment, calculated based on the current location of the first user 7002 in the first physical environment according to the first type of correspondence, is greater than or equal to a second preset threshold distance from an individual position in the three-dimensional environment corresponding to a viewpoint associated with the updated second view 7304-b' of the three-dimensional environment, the computer system of the second user 7102 displays the representation 7002'-b at a second display position within the updated second view 7304-b' of the three-dimensional environment, the second display position being the individual position of the representation 7002'-b in the three-dimensional environment.

[0149] In some embodiments, the second preset threshold distance is an arm's length, a preset radius of the second user's 7102 personal space within the three-dimensional environment, and is defined by a preset bounding surface surrounding the second user's 7102 virtual position within the three-dimensional environment (e.g., a virtual surface of a representation of the second user 7102, a bounding box surrounding the second user's 7102 virtual position, etc.).

[0150] 7F , at a moment following that shown in FIG. 7E , movement of either or both of first user 7002 and second user 7102 continues within their respective physical environments such that the respective positions of the first user and second user within the three dimensional environment, as calculated according to a first type of correspondence (e.g., a first type of correspondence between positions within three dimensional environment 7304 and locations in the first physical environment, a first type of correspondence between positions in three dimensional environment 7304 and locations in the second physical environment, etc.), are within respective preset threshold distances of each other within the three dimensional environment. In some embodiments, at this point, the respective positions of the representation of second user 7102 and the position of the viewpoint of further updated first view 7304-a″, as calculated according to the first type of correspondence between positions in three dimensional environment 7304 and locations in the second physical environment, are less than the first preset threshold distance of each other within the three dimensional environment.

[0151] In portion (A) of Figure 7F, in response to detecting further movement of the first user 7002 and / or the second user 7102 within their respective physical environments, the computer system displays a further updated first view 7304-a'' of the three-dimensional environment with a viewpoint that moves in accordance with the further movement of the first user 7002 within the first physical environment. In some embodiments, the viewpoint of the further updated first view 7304-a'' remains stationary within the three-dimensional environment if the first user 7002 and / or the first display generation component 7100 did not move within the first physical environment. In some embodiments, following a determination that the individual position of the representation 7102'-a of the second user 7102 in the three-dimensional environment calculated based on the current location of the second user 7102 in the second physical environment according to the first type of correspondence is less than a first preset threshold distance from the individual position in the three-dimensional environment corresponding to the viewpoint associated with the further updated first view 7304-a'' of the three-dimensional environment, the computer system displays the representation 7102'-a at an adjusted display position in the further updated first view 7304-a'' of the three-dimensional environment, the adjusted display position being offset from the individual position of the representation 7102'-a in the three-dimensional environment at this point in time. For example, in portion (A) of FIG. 7F , instead of displaying representation 7002′-a in a position directly in front of representation 7102′-a, or overlapping with representation 7002′-a in the further updated first view 7304-a″, the adjusted display position of representation 7002′-a is offset to the side (e.g., to the right, or to another side or direction, etc.) of representation 7002′-a of first user 7002. Generally, instead of displaying representation 7102′-a in a position that is within a first preset threshold distance of the viewpoint of currently displayed first view 7304-a″, the computer system displays representation 7102′-a in an adjusted display position that is offset from the unadjusted position calculated according to the first type of correspondence.In some embodiments, the computer system continues to apply adjustments to the display position of representation 7102'-a while the first user 7002 and / or the second user 7102 are moving until the distance between the position of representation 7102'-a and the position of the viewpoint of the currently displayed first view 7304-a'' is no longer within a first preset threshold distance of each other.

[0152] In some embodiments, optionally, as shown in part (B) of FIG. 7F , the individual positions of representation 7002′-b of first user 7002 and the position of the viewpoint of further updated second view 7304-b″, calculated according to a first type of correspondence between positions in the three-dimensional environment 7304 and locations in the first physical environment, are less than a second preset threshold distance from each other in the three-dimensional environment (e.g., the same as the first preset threshold distance, a different preset threshold distance, etc.).

[0153] In portion (B) of Figure 7F, in response to detecting further movement of the first user 7002 and / or the second user 7102 within their respective physical environments, the computer system of the second user 7102 displays a further updated second view 7304-b'' of the three-dimensional environment with a viewpoint that moves in accordance with the second user's 7102 further movement within the second physical environment. In some embodiments, the viewpoint of the further updated second view 7304-a'' remains stationary within the three-dimensional environment if the second user 7102 and / or the second display generation component 7200 did not move within the second physical environment. In some embodiments, upon determining that the individual position of the representation 7002'-b of the first user 7002 in the three-dimensional environment calculated based on the current location of the first user 7002 in the first physical environment according to the first type of correspondence is less than a second predetermined threshold distance from the individual position in the three-dimensional environment corresponding to the viewpoint associated with the further updated second view 7304-b'' of the three-dimensional environment, the computer system of the second user 7102 displays the representation 7002'-b at an adjusted display position in the further updated second view 7304-b'' of the three-dimensional environment, the adjusted display position being offset from the individual position of the representation 7002'-b in the three-dimensional environment at this time. For example, in portion (B) of Figure 7F, instead of displaying representation 7002'-b in a position directly in front of, or even overlapping with, representation 7102'-b in updated second view 7304-b'', the adjusted display position of representation 7002'-b is offset to the side (e.g., to the right, to another side or direction, etc.) of representation 7102'-b of second user 7102. Generally, instead of displaying representation 7002'-b in a position that is within a second preset threshold distance of the viewpoint of currently displayed second view 7304-b'', the computer system of second user 7102 displays representation 7002'-b in an adjusted display position that is offset from the unadjusted position calculated according to the first type of correspondence.In some embodiments, the computer system of the second user 7102 continues to apply adjustments during movement of the first user and / or the second user until the distance between the position of representation 7002'-b and the position of the viewpoint of the currently displayed second view 7304-b'' is no longer within a preset second threshold distance of each other.

[0154] In some embodiments, in the above example, first user 7002 is moving and second user 7102 is stationary. As a result, unless adjusted as described above, the viewpoints of currently displayed views 7304-a, 7304-a', and 7304-a" will have different positions in the three-dimensional environment, and representation 7002'-b of first user 7002 will have different positions in the three-dimensional environment (e.g., in currently displayed first views 7304-a, 7304-a', and 7304-a" and currently displayed second views 7304-b, 7304-b', 7304-b" in FIGS. 7D-7F). Unless adjusted as described above, the viewpoints of the currently displayed views 7304-b, 7304-b', and 7304-b'' have the same position in the three-dimensional environment, and the representation 7102'-a of the second user 7102 has the same position in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', 7304-b'' in Figures 7D-7F).

[0155] In some embodiments, in the above example, first user 7002 is stationary and second user 7102 is moving within the second physical environment. As a result, unless adjusted as described above, the viewpoints of currently displayed views 7304-b, 7304-b', and 7304-b" have different positions in the three-dimensional environment, and representation 7102'-a of second user 7102 has different positions in the three-dimensional environment (e.g., in currently displayed first views 7304-a, 7304-a', and 7304-a" and currently displayed second views 7304-b, 7304-b', 7304-b" in FIGS. 7D-7F). Unless adjusted as described above, the viewpoints of the currently displayed views 7304-a, 7304-a', and 7304-a'' have the same position in the three-dimensional environment, and the representation 7002'-b of the first user 7002 has the same position in the three-dimensional environment (e.g., in the currently displayed first views 7304-a, 7304-a', and 7304-a'' and the currently displayed second views 7304-b, 7304-b', 7304-b'' in Figures 7D-7F).

[0156] In some embodiments, in the above example, both the first user 7002 and the second user 7102 are moving within their respective physical environments, such that the viewpoints of the currently displayed first views 7304-b, 7304-b', and 7304-b'' and the viewpoints of the currently displayed second views 7304-a, 7304-a', 7304-a'' all have different positions in the three-dimensional environment. Representation 7102'-a of second user 7102 has different positions within the three-dimensional environment in currently displayed first views 7304-a, 7304-a', and 7304-a'' and currently displayed second views 7304-b, 7304-b', and 7304-b'' of Figures 7D-7F, and representation 7002'-b of first user 7002 has different positions within the three-dimensional environment in currently displayed first views 7304-a, 7304-a', and 7304-a'' and currently displayed second views 7304-b, 7304-b', and 7304-b'' of Figures 7D-7F.

[0157] In some embodiments, representation 7002'-b of first user 7002 and / or representation 7102'-a of second user 7102 float in space within the first view and the second view. For example, in some embodiments, representation 7002'-b of first user 7002 is a floating avatar of first user 7002 that floats in second views 7034-b, 7034-b', and 7034-b'', etc. of the three-dimensional environment and automatically moves out of the way when the viewpoint of second view 7034-b'' comes within a second preset threshold distance of representation 7002'-b due to movement of the first user and / or movement of the second user. Similarly, in some embodiments, representation 7102'-a of second user 7102 is a floating avatar of second user 7102 floating in first views 7034-a, 7034-a', and 7034-a'', etc. of the three-dimensional environment, and automatically moves out of the way when the viewpoint of first view 7034-a'' comes within a first preset threshold distance of representation 7102'-a due to movement of the first user and / or movement of the second user. In some embodiments, the user's avatar in the three-dimensional environment is at a level of realism selected based on the realism level of the three-dimensional environment (e.g., a photorealism level, a cartoon realism level, etc.). In some embodiments, following a determination that the three-dimensional environment 7304 is to be displayed at a first level of reality, the user's representation is displayed with a first set of display characteristics corresponding to the first level of reality (e.g., a first resolution, a first number of dimensions, a first clarity, a first color palette, no lighting effects, etc.), and following a determination that the three-dimensional environment is to be displayed at a second level of reality different from the first level of reality (e.g., higher, lower, etc.), the user's representation is displayed with a second set of display characteristics corresponding to the second level of reality (e.g., a second resolution, a second number of dimensions, a second clarity, a second color palette, with lighting effects, etc.), and the second set of display characteristics different from the first set of display characteristics (e.g., higher, lower, adding, subtracting, etc.).

[0158] In some embodiments, when the display position of the individual user's representation is adjusted, the individual user's representation moves with a movement component that does not correspond to the individual user's movement in the physical environment in a normal manner (e.g., according to a first type of correspondence, without adjustment, etc.). In some embodiments, the amount of offset applied to the adjusted position of the individual user's individual representation is variable based on the spatial relationship between the individual representation and the virtual position of the viewpoint in the three-dimensional environment. In some embodiments, the adjustment to the display position of representation 7102′-a is optionally applied to a first view 7304-a″ displayed to the first user 7002 and not to a second view 7304-b″ displayed to the second user 7102. In some embodiments, the adjustment to the display position of representation 7002′-b is optionally applied to a second view 7304-b″ displayed to the second user 7102 rather than to a first view 7304-a″ displayed to the first user 7002.

[0159] In some embodiments, the three-dimensional environment 7304 comprises a virtual three-dimensional environment or an augmented reality environment, and the first user and the second user have a shared experience in the virtual three-dimensional environment. In some embodiments, the positions and movements of the first user and the second user within their respective physical environments (e.g., the same physical environment, different physical environments, etc.) are mapped to positions and movements within the same three-dimensional environment (e.g., using the same mapping relationship, different mapping relationships, etc.), but the appearance of the three-dimensional environment can be adjusted (e.g., with different wallpaper, color schemes, different virtual furniture, etc.) to accommodate the individual users within their views of the three-dimensional environment.

[0160] 7G-7J are block diagrams illustrating changes in the level of immersion displaying an environment of a computer-generated experience according to changes in a user's biometric data received by a computer system, according to some embodiments.

[0161] In some embodiments, the computer system alters the immersion level at which it presents a computer-generated experience (e.g., a visual experience, an audiovisual experience, a virtual reality experience, an augmented reality experience, etc.) to a user (e.g., user 7002) according to biometric data (e.g., biometric data represented by bar 7312, other biometric data, etc.) corresponding to the user. For example, after the computer-generated experience has begun, the computer system can detect changes in the biometric data (e.g., heart rate, blood pressure, respiratory rate, etc.) corresponding to the user as the user adjusts their physical and emotional state, e.g., actively and / or under the influence of the computer-generated content. In accordance with changes in the biometric data relative to respective sets of pre-established criteria (e.g., thresholds represented by indicator 7326 or other types of thresholds or criteria) associated with various levels of immersion, the computer system increases or decreases the level of immersion provided to the user in the computer-generated experience by altering the visual prominence (including, e.g., spatial extent, visual depth, saturation, visual contrast, etc.) of the virtual content relative to the visual prominence of the representation of the physical environment (e.g., by enhancing the complexity, spatial extent, and / or visual characteristics of the virtual content and / or reducing the visual clarity, blur radius, opacity, saturation, etc. of the representation of the physical environment).

[0162] 7G-7J , a computer system initially displays a view 7316 of a three-dimensional environment via a display generation component (e.g., display generation component 7100, or another type of display generation component such as an HMD). In some embodiments, the view 7316 of the three-dimensional environment is a pass-through view of the user's 7002 physical environment and includes no virtual content or a minimal amount of virtual content (e.g., system controls, indicators, etc.) within the peripheral portion of the field of view provided by the display generation component. The view 7316 corresponds to a low level of immersion, for example, providing a computer-generated experience to the user due to a minimal amount of virtual content being displayed relative to a representation of the user's physical environment. In this example, the view 7316 of the three-dimensional environment includes representations of physical surfaces (e.g., representations 7004' and 7006' of two adjacent walls 7004 and 7006 in the user's 7002's physical environment 105, a representation 7008' of a floor 7008, etc.), and representations of physical objects (e.g., a representation 7010' of a physical object 7010 in the user's 7002's physical environment 105, and representations of other physical objects, etc.).

[0163] 7G also shows that the computer system receives biometric data corresponding to user 7002 while displaying view 7316 of the three-dimensional environment at a low immersion level (e.g., displaying a pass-through view of the physical environment or displaying a representation of the physical environment with a minimal amount of virtual content). Following a determination that user 7002's biometric data does not meet the preset criteria corresponding to the next higher immersion level, the computer system maintains the display of first view 7316 of the three-dimensional environment without reducing the visual prominence of the representation of the physical environment within the currently displayed view of the three-dimensional environment. For example, as shown in FIG. 7G, the biometric data has a value or set of values ​​indicated by the length of bar 7312 relative to the range of values ​​of the biometric data, and a threshold corresponding to the preset criteria for transitioning to a different, higher immersion level is indicated by the position of indicator 7326 relative to the range of values ​​of the biometric data.

[0164] In some embodiments, the biometric data corresponding to the user 7002 includes one or more of the user's 7002 heart rate, respiration rate, body temperature, serum concentrations of certain chemicals, drugs, and / or hormones, etc., blood pressure, brain waves, concentration, pupil size, metabolic rate, blood glucose level, etc. In some embodiments, the biometric data corresponding to the user 7002 includes one or more types of biometric data (e.g., respiration rate, blood pressure, concentration, blood glucose level, etc.) that may change over time during the user's engagement with the computer-generated experience. In some embodiments, the biometric data corresponding to the user includes one or more types of biometric data that may vary through the user's physical actions (e.g., meditation, changes in breathing patterns, exercise, etc., as opposed to direct interaction with user interface elements or controls provided by the computer system during the user's engagement with the computer-generated experience). In some embodiments, the biometric data corresponding to the user includes one or more types of composite metrics of multiple types of biometric data corresponding to the user's mood, happiness, and / or stress level, etc. In some embodiments, the biometric data includes real-time data corresponding to the user's physiological state within a time or a preset amount of time prior to the display of the current view of the three-dimensional environment via the display generation component. In some embodiments, the biometric data is continuously and / or periodically collected through one or more biometric sensors (e.g., various suitable medical devices, vibration sensors, cameras, thermal sensors, chemical sensors, etc.) connected to or directed at the user and continuously and / or periodically transmitted to the computer system. In some embodiments, the biometric data does not include non-primary human characteristics (e.g., fingerprints, iris pattern and color, facial features, voiceprints, etc.) that typically do not change over the period of time that an average user is engaged in a computer-generated experience.

[0165] In some embodiments, the computer system determines that the biometric data does not satisfy a preset criterion for transitioning to displaying the computer-generated experience at a preset higher immersion level according to a determination that the heart rate is greater than a first threshold heart rate, the blood pressure is higher than a first threshold blood pressure, the user's movement is greater than a first threshold movement amount for a threshold amount of time, the user's body temperature is higher than a first threshold body temperature, the stress level metric exceeds a first threshold stress level, a metric corresponding to the user's mood indicates that the user is upset and unhappy, etc. In some embodiments, the computer system switches to displaying the three-dimensional environment at a preset higher immersion level (e.g., as shown in FIG. 7J ) when the preset criterion is satisfied, without undergoing a gradual transition based on changes in the biometric data before the preset criterion is satisfied. In some embodiments, optionally, the computer-generated experience includes visual and / or audio guidance (e.g., music, scenery, motivational messages, medication record instructions, visual, audio, or verbal breathing instructions, etc.) to assist the user in entering a state in which corresponding biometric data received from the user would satisfy the preset criterion.

[0166] 7H-7I illustrate that, in some embodiments, the computer system gradually adjusts the level of immersion at which the computer-generated experience is provided to the user according to the trend and / or magnitude of change in the biometric data corresponding to the user. For example, in some embodiments, when the biometric data exhibits changes that approach satisfaction of preset criteria for switching to a preset higher immersion level (e.g., an augmented reality view, an augmented virtuality view, a virtual reality view, etc.), the computer system increases the visual prominence and / or amount of the virtual content corresponding to the computer-generated experience and reduces the visual prominence and / or amount of the representation of the physical environment within the currently displayed view of the three-dimensional environment. In some embodiments, the computer system changes the visual balance between the virtual content corresponding to the computer-generated experience and the representation of the physical environment by an amount corresponding to the amount and / or nature of change in the biometric data corresponding to the user. Similarly, in some embodiments, when the biometric data exhibits a change away from meeting the preset criteria for switching to a preset higher immersion level, the computer system decreases the visual prominence and / or amount of the virtual content corresponding to the computer-generated experience and increases the visual prominence and / or amount of the representation of the physical environment within the currently displayed view of the three-dimensional environment.

[0167] In some embodiments, the computer system changes the visual balance between the virtual content and the representation of the physical environment by an amount corresponding to the amount and / or nature of the change in the biometric data corresponding to the user. As shown in Figure 7H, when the value of the biometric data changes to meet a preset criterion (e.g., as indicated by the increased length of bar 7312 approaching the position of indicator 7326), the amount of virtual content displayed in the view of the three-dimensional environment (e.g., view 7318 in Figure 7H) increases compared to the previous state (e.g., view 7316 in Figure 7G), and the visual prominence of the representation of the physical environment decreases. More specifically, in FIG. 7H , representations 7004′ and 7006′ of walls 7004 and 7006 are replaced or hidden by the display of virtual content 7320 and 7322 (e.g., visual effects, virtual surfaces, virtual objects, virtual scenery, etc. that visually obscure portions of the representation of the physical environment to which the visual effect is applied), and at least a portion of the surface of representation 7010′ is also replaced or hidden by the display of virtual content 7324 (e.g., visual effects, virtual surfaces, virtual objects, virtual scenery, etc. that visually obscure portions of the representation of the physical environment to which the visual effect is applied). 7I, which follows FIG. 7H, when the value of the biometric data changes away from meeting the preset criteria (e.g., as indicated by a decreased length of bar 7312 receding from the position of indicator 7326), the amount of virtual content displayed within the view of the three-dimensional environment (e.g., view 7328 in FIG. 7I) decreases compared to the previous state (e.g., view 7318 in FIG. 7H), and the visual prominence of the representation of the physical environment increases again (e.g., optionally still lower than the state shown in FIG. 7G). More specifically, in FIG. 7I, representation 7006' of wall 7006 reappears after virtual content 7332 has been removed, and representation 7004' of wall 7004 partially reappears when the visual prominence of the virtual content 7320 has been reduced (e.g., visual effects that visually obscure portions of the representation of the physical environment to which the visual effects are applied are reduced in size, virtual surfaces and objects are reduced in number, made more translucent, etc.).The visual prominence of portions of the surface of representation 7010′ that are replaced or obscured by the display of virtual content 7324 similarly increases with changes made to virtual content 7324 (e.g., made more translucent, less opaque, including a lesser amount of distortion relative to representation 7010′, etc.). In some embodiments, before a preset criterion for transitioning to a preset higher immersion level is met (e.g., before a threshold indicated by indicator 7326 is met by the biometric data corresponding to the user, or before other criteria are met by the biometric data, etc.), the computer system continuously or periodically adjusts the visual balance between the virtual content and the representation of the physical environment in the currently displayed view of the three-dimensional environment in accordance with the biometric data as the biometric data is updated based on the user's current state (e.g., increasing the visual prominence of the virtual content relative to the representation of the physical environment, decreasing the visual prominence of the virtual content relative to the representation of the physical environment, etc.).

[0168] In FIG. 7J, the computer system detects that updated biometric data corresponding to the user meets predetermined criteria for transitioning to a predetermined higher immersion level (e.g., an augmented reality environment, an augmented virtual reality environment, a virtual reality environment, etc.) having a higher immersion level compared to what was displayed before the predetermined criteria was met by the biometric data (e.g., views 7316, 7318, 7328, etc. of FIGS. 7G-7I), and the computer system transitions to a display of a three-dimensional environment at the predetermined higher immersion level (e.g., a display such as view 7334 of FIG. 7J, or another view of the three-dimensional environment at the predetermined higher immersion level). In this example, as shown in Figure 7J, the computer system increases the visual prominence of the virtual content and further decreases the visual prominence of the representations of the physical environment through the visual characteristics of the virtual content (e.g., virtual content 7322, 7320, 7330, and 7324 visually obscuring representations 7006', 7004', 7008', and 7010' of walls 7006, 7004, floor 7008, and physical object 7010 in view 7334 of Figure 7J), such that only hints of the physical environment (e.g., structural relationships between walls and floors, the presence of physical objects, etc.) remain visible within the three-dimensional environment. In some embodiments, the computer system further displays virtual objects at various positions within the three-dimensional environment. For example, virtual object 7332 may be displayed in a position corresponding to the location of physical object 7010 in the physical environment, virtual object 7326 may be displayed in a position corresponding to a location on floor 7008, and other virtual objects may be displayed in positions corresponding to free space in the physical environment or regardless of the state of the physical environment. In some embodiments, after preset criteria for transitioning to a preset higher immersion level are met, the computer system either suddenly increases the amount of virtual content in the currently displayed view of the three-dimensional environment or displays an entirely new environment (e.g., a new virtual world, a new scene, etc.) corresponding to the computer-generated experience.In some embodiments, after the preset criteria are met and the computer system displays the three-dimensional environment at a preset higher level of immersion, in accordance with a determination that the preset criteria are no longer met by the updated biometric data, the computer system gradually adjusts the level of immersion at which the three-dimensional environment is displayed based on the change in the biometric data, as shown in Figures 7H and 71. In some embodiments, after the preset criteria are met and the computer system displays the three-dimensional environment at a preset higher level of immersion, in accordance with a determination that the preset criteria are no longer met by the updated biometric data, the computer system abruptly switches back to displaying the three-dimensional environment at a lower level of immersion (e.g., as shown in Figure 7G).

[0169] In some embodiments, the preset criteria are met according to a determination that the heart rate is lower than a first threshold heart rate, the respiratory rate is lower than a first threshold respiratory rate, the blood pressure is lower than a first threshold blood pressure, the user's movement is less than a first threshold movement amount during a threshold time, the user's body temperature is lower than a first threshold body temperature, a stress level metric is lower than a first threshold stress level, and a metric corresponding to the user's mood indicates that the user is relaxed and happy.

[0170] In some embodiments, a view of a three-dimensional environment shown at a low immersion level (e.g., as shown in FIG. 7G or another view of the three-dimensional environment) is displayed when a display generation component of a computer system is first turned on or placed in front of a user's head or eyes, and no virtual elements or a minimal amount of virtual elements are displayed in the three-dimensional environment. This allows a user to start with a view of the three-dimensional environment that closely resembles a direct view of the real world, without the display generation component blocking the user's eyes. In some embodiments, a view of a three-dimensional environment corresponding to a low immersion level is a view of a user interface or environment (e.g., a two-dimensional environment, a three-dimensional environment, etc.) of an application or computer-generated experience that is displayed in a two-dimensional window or is constrained within a viewport displayed relative to a representation of the physical environment. In some embodiments, a view of a three-dimensional environment shown at a low immersion level (e.g., as shown in FIG. 7G or another view of the three-dimensional environment) is displayed when an application or computer-generated experience is first launched or initiated by a user, and the full spatial extent of the application or experience has not yet been displayed in the three-dimensional environment. This allows the user to start with a less immersive view of the three-dimensional environment that is seen in the context of a real-world view.

[0171] In some embodiments, virtual content (e.g., virtual wallpaper, virtual objects, virtual surfaces, virtual landscapes, virtual three-dimensional environments, etc.) displayed by the computer system at least partially occludes or obscures the view of the physical environment. In some embodiments, when displaying a view of the three-dimensional environment at a higher than preset immersion level, the computer system replaces or occludes the view of a first class of physical objects or surfaces (e.g., the front wall, front wall, and ceiling, etc.) with newly displayed virtual elements or newly displayed portions of existing virtual elements. In some embodiments, an animated transition is displayed in which the virtual elements gradually expand or become more opaque and fill in, thereby obscuring or occluding the view of the first class of physical objects or surfaces. In some embodiments, when displaying a view of the three-dimensional environment at a higher than preset immersion level, the computer system adds virtual elements to the three-dimensional environment without replacing any entire class of physical elements. In some embodiments, the added virtual elements optionally include user interface objects such as menus (e.g., application menus, documents, etc.), controls (e.g., display brightness controls, display focus controls, etc.), or other objects (e.g., virtual assistants, documents, media items, etc.) that can be manipulated by user input or that provide information or feedback to the three-dimensional environment. In some embodiments, the added virtual elements optionally include non-interactive objects or surfaces that cannot be manipulated by user input and that function to provide a look and feel of the three-dimensional environment that replaces the look and feel of the physical environment. In some embodiments, the virtual content displayed by the computer system includes visual effects that at least partially block or obscure the view of the physical environment (e.g., fading out, blurring, dimming a representation of the physical environment, etc.).

[0172] In some embodiments, the biometric data is updated, and pursuant to a determination that the updated biometric data meets preset criteria for transitioning to displaying the three-dimensional environment at an even higher level of immersion, the computer system increases the visual prominence of the virtual content corresponding to the computer-generated experience and reduces visual cues from the physical environment to another level corresponding to the even higher level of immersion. For example, in some embodiments, the computer system replaces, obscures, or occludes the additional class of physical object or surface (e.g., floor) with a newly displayed virtual element or a newly displayed portion of an existing virtual element. In some embodiments, an animated transition is displayed in which the virtual element gradually expands or becomes more opaque and fills in, thereby obscuring or occluding the v-view of the additional class of physical object and surface.

[0173] In some embodiments, the three-dimensional environment is an environment for a computer-generated mediated experience, and when the biometric data indicates that the user has achieved a level of concentration, relaxation, focus, etc. required to enter a state of deeper meditative experience, the computer system transforms the currently displayed view of the environment into a more immersive environment, for example, with expanded spatial extent (e.g., width, depth, angle, etc.) and visual prominence of virtual content corresponding to the meditative experience and reduced spatial extent and visual prominence of the representation of the physical environment.

[0174] In some embodiments, along with increasing the level of immersion in displaying the visual content of the computer-generated experience, the computer system also increases the level of sound suppression in the physical environment perceivable by the user through actions on the computer system's audio output devices and / or increases the level of immersion in the audio content of the computer-generated experience output by the audio output devices (e.g., increasing the volume, changing from a stereo audio output mode or a surround sound output mode to a spatial audio output mode, or changing from a stereo audio output mode to a surround sound output mode, etc.).

[0175] In some embodiments, a computing system is configured to display visual components of CGR content via a display generation component at two or more levels of immersion. In some embodiments, the computer system displays visual components of CGR content at at least a first immersion level, a second immersion level, and a third immersion level. In some embodiments, the computer system displays visual components of CGR content at at least two levels of immersion, providing less immersive and more immersive visual experiences, respectively. In some embodiments, the computing system transitions the visual content displayed via the display generation component between different levels of immersion in response to biometric data corresponding to a user satisfying different sets of criteria. In some embodiments, the first, second, and third levels of immersion correspond to increasing amounts of virtual content corresponding to the CGR experience present within the CGR environment and / or decreasing amounts of representations of the surrounding physical environment present within the CGR environment. In some embodiments, the first, second, and third immersion levels correspond to different modes of content presentation having increasing image fidelity (e.g., increasing pixel resolution, increasing color resolution, increasing color saturation, increasing luminance, increasing opacity, increasing image detail, etc.) and / or spatial extent (e.g., angular extent, spatial depth, etc.) of the computer-generated content, and / or decreasing image fidelity and / or spatial extent of the representation of the surrounding physical environment. In some embodiments, the first immersion level is a pass-through mode, in which the physical environment is fully visible to the user through the display generation components (e.g., as a camera view of the physical environment or through transparent or semi-transparent portions of the display generation components). In some embodiments, the visual CGR content presented in the pass-through mode includes a pass-through view of the physical environment with a minimal amount of virtual elements simultaneously visible as the view of the physical environment, or with only virtual elements peripheral to the user's view of the physical environment (e.g., indicators and controls displayed in a peripheral area of ​​the display).For example, the view of the physical environment occupies a central and majority area of ​​the field of view provided by the display generation component, with only a smaller number of controls (e.g., movie title, progress bar, playback controls (e.g., play button), etc.) displayed in a peripheral area of ​​the field of view provided by the display generation component. In some embodiments, the first level of immersion is a pass-through mode, where the physical environment is fully visible to the first user through the display generation component (e.g., as a camera view of the physical environment or through a transparent portion of the display generation component), and the visual CGR content is displayed in a virtual window or frame, such as overlaid on a representation of the physical environment, replacing a portion of the representation of the physical environment, or blocking the view of a portion of the representation of the physical environment. In some embodiments, the second level of immersion is a mixed reality mode, where the pass-through view of the physical environment is augmented with virtual elements generated by a computer system, where the virtual elements occupy a central and / or majority area of ​​the user's field of view (e.g., virtual content is integrated with the physical environment within the view of the computer-generated environment). In some embodiments, the second level of immersion is a mixed reality mode in which a pass-through view of the physical environment is augmented with a virtual window, viewport, or frame that is overlaid on, replaces the display of, or blocks the view of a portion of the representation of the physical environment, and has additional depth or spatial extent that is revealed when the display generating component is moved relative to the physical environment. In some embodiments, the third level of immersion is an augmented reality mode in which virtual content is displayed in the three-dimensional environment along with a representation of the physical environment, and virtual objects are distributed throughout the three-dimensional environment at positions that correspond to different locations in the physical environment. In some embodiments, the third level of immersion is a virtual reality mode in which virtual content is displayed in the three-dimensional environment without a representation of the physical environment. In some embodiments, the different levels of immersion described above represent increasing levels of immersion relative to each other.

[0176] In some embodiments, the computer system selects an audio output mode for outputting audio content of a computer-generated experience (e.g., an application, a communication session, a movie, a video, a game, etc.) according to an immersion level at which visual content of the computer-generated experience is being displayed by a display generation component. In some embodiments, as the immersion level at which the visual content is displayed increases (e.g., from a first immersion level to a second immersion level, from the first immersion level to a third immersion level, or from the second immersion level to the third immersion level, etc.), the computer system switches the audio output mode from an output mode with a lower immersion level to an output mode with a higher immersion level (e.g., the first audio output mode, the second audio output mode, and the third audio output mode correspond to audio output with increasing levels of immersion, such as from the first audio output mode to the second audio output mode, or from the first audio output mode to the third audio output mode, or from the second audio output mode to the third audio output mode). As described herein, the spatial audio output mode corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The spatial audio output mode corresponds to a higher level of immersion than the surround sound output mode. The surround sound output mode corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The stereo audio output mode corresponds to a higher level of immersion than the mono audio output mode. In some embodiments, the computer system selects an audio output mode from a plurality of available audio output modes, e.g., a mono audio output mode, a stereo audio output mode, a surround sound output mode, a spatial audio output mode, etc., based on the immersion level at which the visual content of the computer-generated experience is provided via the display generation component.

[0177] 7K-7M are block diagrams illustrating the aggregation of the effects of multiple types of sensory modulation provided by a computer system when displaying a view of an environment that includes a representation of a physical environment, according to some embodiments.

[0178] In some embodiments, the computer system provides multiple types of sensory modulation functionality that enhance a user's ability to perceive various aspects of the physical environment that may not be readily perceptible without the assistance of special equipment or the computer system. Instead of only allowing a user to use a single type of sensory modulation functionality when viewing a portion of the physical environment at a time, the computer system aggregates the effects of two or more types of sensory enhancement functionality on the representation of a portion of the physical environment such that features and characteristics present in a portion of the physical environment that were previously hidden within the view of the physical environment provided by the computer system can be revealed.

[0179] In some embodiments, when a computer system displays a three-dimensional environment including a representation of a physical environment via a display generation component (e.g., display generation component 7100, or another type of display generation component such as an HMD), the computer system optionally uses sensor input or information corresponding to a currently displayed portion of the physical environment to enhance and adjust the representation of the physical environment, thereby allowing a user to perceive a portion of the physical environment using sensory information that is not available to the user when the user views the portion of the physical environment without the assistance of the display generation component.

[0180] 7K, the computer system displays a view 7340 of a three-dimensional environment including a first representation of a first portion of the physical environment. In view 7340, the first representation of the first portion of the physical environment corresponds to the appearance of the first portion of the physical environment without sensory adjustments made by the computer system. In some embodiments, the first representation of the first portion of the physical environment corresponds to a view of the first portion of the physical environment captured by a color camera having a first level of imaging sensitivity corresponding to average color and intensity detection within the range of normal human perception. In some embodiments, the first representation of the first portion of the physical environment corresponds to a view of the first portion of the physical environment through a transparent portion of the display generation component and is not augmented or adjusted by the computer system.

[0181] In some embodiments, the computer system provides a plurality of affordances (e.g., hardware controls 7354, 7356, and 7358, user interface elements displayed in the three-dimensional environment, etc.) for activating each of a plurality of sensory adjustment functions provided by the computer system. In some embodiments, the computer system activates each of the plurality of sensory adjustment functions sequentially or in combination according to a user activation input (e.g., a button press input, a tap input, a gesture input, a touch input, a gaze input, a selection input, a combination thereof, etc.) directed at an affordance corresponding to each of the plurality of sensory adjustment functions. In some embodiments, each one of the plurality of sensory adjustment functions is optionally activated by a preset input (e.g., a gesture input, a touch input, a voice command, etc.) without requiring the presence of a corresponding hardware affordance associated with the computer system or a corresponding user interface control in the three-dimensional environment.

[0182] 7K, the first representation of the first portion of the physical environment includes a view from inside a room toward a window on a wall of the room. This example is non-limiting, and according to various embodiments, the first portion of the physical environment can be any indoor or outdoor environment. In this example, the first representation of the first portion of the physical environment includes a representation 7344' of a wall, a representation 7346' of a window, a representation 7348' of a hill outside the window at a first distance from the window, and a representation 7350' of a tree near the top of the hill a second distance from the window. Because the hills and trees are at a large distance from the display generation component and because hill representation 7348' and tree representation 7350' are far away from the viewpoint corresponding to the currently displayed view of the three-dimensional environment (e.g., the respective distances between the viewpoint and representations 7348' and 7350' correspond to the respective distances from the user's eyes (or display generation component) to the hills and trees), hill representation 7348' and tree representation 7350' occupy a small portion of the field of view provided by display generation component 7100.

[0183] In FIG. 7L , the computer system detects a user input that activates a first sense adjustment function among multiple sense adjustment functions provided by the computer system. For example, the computer system detects that a hardware affordance 7354 has been activated by a user input, that a user interface object corresponding to the first sense adjustment function has been activated or selected by a user input, that a user has provided a gesture input, a voice command, and / or a touch input, etc., that meets criteria for activating the first sense adjustment function, etc. In response, the computer system displays a second view 7361 of the three-dimensional environment that includes a second representation of a second portion of the physical environment, the second portion of the physical environment being contained within the first portion of the physical environment (e.g., all or part of the first portion of the physical environment shown in FIG. 7K or the portion of the physical environment that was shown prior to detection of the input that activated the first sense adjustment function, etc.). In the second view 7361 of the three-dimensional environment, as shown in the example of FIG. 7L , the display characteristics of a tree representation 7350″ are adjusted relative to the tree representation 7350′ shown in the first view 7340 of the three-dimensional environment in accordance with the operation of the first sense adjustment function. For example, if the first sensory adjustment function is a simulated telescope view that reduces the focal length of an object so that the object appears closer to the user, as shown in FIG. 7L, then the tree representation 7350″ will appear to be located much closer to the viewpoint than at a second distance, as shown in FIG. 7K (e.g., the adjusted distance is one-fifth the second distance, the adjusted distance is one-tenth the second distance, the adjusted distance is a distance selected based on a preset percentage of the second distance and / or the maximum magnification of the simulated telescope view, etc.). Similarly, the hill representation 7348″ will also appear to be located much closer to the user than at a first distance, as shown in FIG. 7K (e.g., the adjusted distance is one-fifth the first distance, the adjusted distance is one-tenth the first distance, the adjusted distance is a distance selected based on a preset percentage of the first distance and / or the maximum magnification of the simulated telescope function, etc.).In this example, the user's viewpoint or virtual position within view 7361 is moved, according to some embodiments, to the position of the window within view 7340. In this example, according to some embodiments, the user's viewpoint or virtual position within view 7361 is still based on the actual location of the user and / or display generating components within the physical environment.

[0184] In some embodiments, when applying the first sensory adjustment function, the computer selects a target portion of the physical environment based on the location of the user's gaze directed toward the current view of the three-dimensional environment. For example, as shown in FIG. 7K, the computer system detects that the user's gaze 7352 is directed toward a representation 7350′ of a tree in the first view 7340 of the three-dimensional environment and selects a portion of the physical environment that includes the tree from the first portion of the physical environment as the second portion of the physical environment to which the first sensory adjustment function is applied.

[0185] In some embodiments, simulated telescopic viewing is an illustrative example of a first type of sensory accommodation function provided by the computer system, and may be replaced by another type of sensory accommodation function provided by the computer system and selected by user input.

[0186] 7M , while the computer system displays a second view 7361 of the three-dimensional environment including a second representation of the physical environment adjusted according to the operation of the first sense adjustment feature activated by the user's input, the computer system detects a second user input that activates a second sense adjustment feature of the plurality of sense adjustment features, different from the first sense adjustment feature. For example, the computer system detects that a hardware affordance 7356 has been activated by the user's input, that a user interface object corresponding to the second sense adjustment feature has been activated or selected by the user's input, that a gesture input, a voice command, and / or a touch input, etc., that meets criteria for activating the second sense adjustment feature has been provided by the user, etc. In response, the computer system displays a third view 7364 of the three-dimensional environment including a third representation of a third portion of the physical environment, the third portion of the physical environment being contained within the second portion of the physical environment (e.g., all or part of the second portion of the physical environment shown in FIG. 7L or the portion of the physical environment shown prior to the detection of the input that activated the second sense adjustment feature). In third view 7364 of the three dimensional environment, as shown in the example of FIG. 7M , the display characteristics of tree representation 7350''' are further adjusted relative to tree representation 7350'' shown in second view 7361 of the three dimensional environment in accordance with operation of the second sensory adjustment function. For example, if the second sensory adjustment function is simulated heat vision that presents variations in color and / or intensity according to variations in temperature and / or thermal radiation, as shown in FIG. 7M , tree representation 7350''' will appear to have a different color and / or intensity relative to the background environment in third view 7364, and the display characteristics of portions 7366''' and 7368''' of representation 7350''' will be further adjusted based on the temperature of those portions of the tree relative to other portions of the tree in the physical environment (e.g., as detected by a thermal imaging sensor or other sensor in communication with the computer system, as indicated by thermal data transmitted to the computer system or retrieved by the computer system from another computer system, etc.).For example, the higher temperatures of these portions, represented by portions 7366''' and 7368''', are more likely to reveal small animals or objects that radiate more heat or have a higher temperature than the tree itself. Portions 7366''' and 7368''' in representation 7350''' have display characteristics that are generated based on the operation of both the first sensory modulation function and the second sensory modulation function on the original first representation 7350' of the tree, as shown in FIG. 7K.

[0187] In some embodiments, when applying the second sensory adjustment function, the computer system selects the target portion of the physical environment based on the location of the user's gaze directed toward the currently displayed view of the three-dimensional environment. For example, as shown in FIG. 7L , the computer system detects that the user's gaze 7360 is directed toward a representation 7350″ of a tree in the second view 7361 of the three-dimensional environment, and selects the portion of the physical environment that includes the tree from the second portion of the physical environment as the third portion of the physical environment to which both the first sensory adjustment function and the second sensory adjustment function are applied.

[0188] In some embodiments, simulated heat vision is an illustrative example of a second type of sensory modulation functionality provided by the computer system, and may be replaced by another type of sensory modulation functionality provided by the computer system and selected by user input.

[0189] In some embodiments, a first display characteristic (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) is adjusted relative to a baseline representation of a distinct portion of the physical environment (e.g., tree representation 7350' of FIG. 7K corresponding to portions 7366''', 7368''' of FIG. 7M, another portion of the physical environment, etc.) according to a first type of computer-generated sensory adjustment (e.g., binocular vision, telescopic vision, microscopic vision, night vision, thermal vision, etc.) to generate a first adjusted representation of the distinct portion of the physical environment (e.g., portion 7366''', 7368''', of tree representation 7350' of FIG. 7K corresponding to portions 7366''', 7368''', of another portion of the physical environment, etc.). 7L corresponding to portions 7366''', 7368''', or a further adjusted representation of another portion of the physical environment), and second display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) are adjusted for the second representation of the physical environment according to a second type of computer-generated sensory adjustment to obtain a third representation of the respective portion of the physical environment (e.g., portions 7366''', 7368''' of tree representation 7350''' of FIG. 7M, or a further adjusted representation of another portion of the physical environment). In some embodiments, the second display characteristics have the same value in the first representation and the second representation for some combinations of the first and second types of sensory adjustment functions, and the second display characteristics have different values ​​in the first representation and the second representation for some combinations of the first and second types of sensory adjustment functions.

[0190] In some embodiments, the computer system enables the representation of the physical environment to be further adjusted based on a third sensory adjustment feature (e.g., an affordance 7358 corresponding to the third sensory adjustment feature, a sensory adjustment feature that can be activated by interaction with a user interface object, a gestural input, a voice command, etc.). In some embodiments, while displaying a third view 7364 of the three-dimensional environment including the third representation of the physical environment, the computer system detects a third user input corresponding to a request to activate a third type of computer-generated sensory adjustment (e.g., binocular vision, microscopic vision, night vision, thermal vision, color filters, etc.) that is distinct from the first and second types of sensory adjustment features. In response, the computer system displays a fourth view of the three-dimensional environment including a fourth representation of a fourth portion of the physical environment (e.g., all or part of a third portion of the physical environment), the fourth representation of the physical environment having first display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the first representation of the fourth portion of the physical environment according to a first type of sensory adjustment function, second display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the second representation of the fourth portion of the physical environment according to a second type of sensory adjustment function, and third display characteristics (e.g., resolution, zoom level, magnification, color distribution, intensity distribution, focal length, etc.) adjusted for the third representation of the physical environment of the fourth portion of the physical environment according to a third type of sensory adjustment function.

[0191] In some embodiments, the first sensory adjustment function includes simulated telescopic viewing (e.g., binocular viewing, monocular viewing, telescopic viewing, etc.) for viewing distant physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user), and the second sensory adjustment function includes simulated microscopic viewing for magnifying nearby physical objects.

[0192] In some embodiments, the first sensory adjustment function includes simulated telescoping for viewing remote physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user), and the second sensory adjustment function includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.).

[0193] In some embodiments, the first sensory adjustment function includes simulated telescoping for viewing a distant physical object (e.g., reducing the focal length of the object so that the object appears closer to the user), and the second sensory adjustment function includes modifying the view of the physical object with filters (e.g., color filters, light frequency filters, intensity filters, motion filters, etc.).

[0194] In some embodiments, the first sensory adjustment function includes simulated telescoping for viewing distant physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user), and the second sensory adjustment function includes selective sound enhancement (e.g., boosting volume, selectively enhancing / suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object that is centered in the current field of view, etc.).

[0195] In some embodiments, concurrently with displaying the third representation of the physical environment, the computer system outputs sounds corresponding to portions of the physical environment that are visible in the third representation of the physical environment, and the sounds are selectively enhanced (e.g., by increasing the volume, modifying the amplitude of some selected frequencies, etc.) relative to sounds from sources outside the portion of the physical environment.

[0196] In some embodiments, concurrently with displaying the third representation of the physical environment, the computer system displays text output corresponding to speech emanating from a portion of the physical environment that is visible in both the second and third representations of the physical environment, the speech being selectively enhanced relative to sounds from sources outside the portion of the physical environment.

[0197] In some embodiments, the first sensory modulation function includes simulated microscopic vision for magnifying nearby physical objects, and the second sensory modulation function includes simulated thermal vision for viewing physical objects with different thermal emission profiles (e.g., increased sensitivity to temperature variations, presenting variations in color and / or intensity according to temperature and / or thermal emission variations, etc.).

[0198] In some embodiments, the first sensory adjustment function includes simulated night vision for viewing physical objects in low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.), and the second sensory adjustment function includes simulated telescopic vision for viewing remote physical objects (e.g., reducing the focal length of an object so that the object appears closer to the user).

[0199] In some embodiments, the first sensory modulation function includes simulated night vision for viewing physical objects in low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.), and the second sensory modulation function includes simulated microscopy for magnifying nearby physical objects.

[0200] In some embodiments, the first sensory adjustment function includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.), and the second sensory adjustment function includes simulated heat vision for viewing physical objects having different thermal radiation profiles (e.g., increased sensitivity to temperature variations, presenting variations in color and / or intensity in response to temperature and / or thermal radiation variations, etc.).

[0201] In some embodiments, the first sensory modulation function includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.), and the second sensory modulation function and second type of computer-generated sensory modulation includes selective sound enhancement (e.g., boosting volume, selectively enhancing / suppressing certain sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object centered in the current field of view, etc.).

[0202] In some embodiments, the first sensory adjustment function includes simulated thermal vision for viewing physical objects with different thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting color and / or intensity variations according to temperature and / or thermal radiation variations, etc.), and the second sensory adjustment function includes simulated telescopic vision for viewing remote physical objects (e.g., reducing the focal length of the object so that it appears closer to the user).

[0203] In some embodiments, the first sensory modulation function includes simulated thermal vision for viewing physical objects with varying thermal radiation profiles (e.g., high sensitivity to temperature variations, presenting variations in color and / or intensity according to temperature and / or thermal radiation variations, etc.), and the second sensory modulation function includes simulated microscopic vision for magnifying nearby physical objects.

[0204] In some embodiments, the first sensory adjustment operation includes simulated heat vision for viewing physical objects with varying thermal radiation profiles (e.g., increased sensitivity to temperature variations, presenting variations in color and / or intensity according to temperature and / or thermal radiation variations, etc.), and the second sensory adjustment operation includes simulated night vision for viewing physical objects under low light conditions (e.g., increased sensitivity in low light conditions, visual enhancement of object brightness, magnification of small variations in brightness, etc.).

[0205] In some embodiments, the first sensory modulation function includes simulated thermal vision for viewing physical objects with various thermal emission profiles (e.g., increased sensitivity to temperature variations, presenting variations in color and / or intensity according to temperature and / or thermal emission variations, etc.), and the second sensory modulation action includes selective sound enhancement (e.g., boosting volume, selectively enhancing / suppressing specific sound frequencies, etc.) for sounds corresponding to a subset of physical objects in the physical environment (e.g., a selected subset of all sound-producing physical objects, a physical object centered in the current field of view, etc.).

[0206] In some embodiments, the order in which multiple selected sensory adjustment features selected by the user are applied to the baseline representation of a portion of the physical environment is adjusted by the computer system based on one or more preset constraints, and optionally differs from the order in which these sensory adjustment features are activated by the user. For example, in some embodiments, adjustments corresponding to simulated telescopic viewing are performed before adjustments corresponding to other types of sensory adjustments in order to reduce the portions of the physical environment in which other types of sensory adjustments need to be performed in order to present the final result to the user. In some embodiments, the computer system observes the order in which different types of sensory adjustment features are activated by the user and presents intermediate results obtained in response to each additional sensory adjustment activated by the user.

[0207] 7N-7P are block diagrams illustrating selectively displaying virtual content corresponding to a particular type of exercise within a view of a three-dimensional environment in accordance with some embodiments, pursuant to a determination that a portion of the physical environment within the view of the three-dimensional environment corresponds to the particular type of exercise.

[0208] In some embodiments, the computer system displays virtual content (e.g., virtual open water body 7406, virtual hiking trail 7412, etc.) (e.g., virtual scenery, visual and functional enhancements to exercise equipment, user interfaces, health and leaderboards, etc.) corresponding to the particular type of exercise (e.g., rowing, hiking, etc.) pursuant to a determination that a physical location (e.g., location of physical object 7404, location of physical object 7402, etc.) represented in a view of the three-dimensional environment (e.g., view 7408, view 7410, etc.) is associated with the particular type of exercise (e.g., rowing, hiking, etc.). For example, as the user and display generation component (e.g., user 7002 and display generation component 7100, or another user with another type of display generation component such as an HMD, etc.) move between locations in the real world (e.g., within scene 105, or within another physical environment, etc.), the virtual content shown in the view of the three-dimensional environment is adjusted to correspond to the type of exercise associated with the current location of the user and display generation component. In some embodiments, if a location is associated with multiple types of exercise, the computer system selects a type of exercise from the multiple types of exercise associated with the location based on other contextual information (e.g., the user's movement, the user's engagement with objects at the location, etc.) and displays visual content corresponding to the selected type of exercise.

[0209] Part (A) of FIG. 7N shows a user 7002 located in a physical environment (e.g., scene 105, or another physical environment, etc.). The user 7002 may be located in a different physical environment, such as an outdoor environment or an indoor environment, or may move between an indoor environment and an outdoor environment. The user 7002 views the physical environment through a field of view provided via a first display generating component (e.g., display generating component 7100, another type of display generating component such as an HMD, etc.). The physical environment includes physical surfaces (e.g., walls 7004 and 7006, a floor 7008, other physical surfaces, etc.) and one or more physical objects (e.g., exercise equipment 7402, 7404, other physical objects, etc.). In some embodiments, the physical environment is a building that includes multiple separate rooms or sections that cannot be viewed simultaneously by the user. In some embodiments, the physical environment includes multiple separate areas, such as rooms within individual buildings, different parks, different geographic regions, etc. In some embodiments, the physical environment is an outdoor environment including outdoor physical objects and surfaces, such as roads, trees, sky, open water, rocks, mountains, vehicles, animals, people, etc. In some embodiments, the computer system stores information and / or implements rules and artificial intelligence to determine one or more types of exercise (e.g., indoor exercise, indoor sports, outdoor exercise, outdoor sports, physical activities that promote health and physical performance, restorative training and therapy, etc.) associated with individual locations within the user's physical environment (e.g., within the user's field of view through the display generation component, within a threshold vicinity of the user (e.g., within 5 meters, within a few steps, etc.)). In some embodiments, the computer system determines the type of exercise associated with the individual locations based on the type of physical objects present at the individual locations. In some embodiments, the computer system determines the type of exercise associated with the individual locations based on the type of environment or setting present at the individual locations.In some embodiments, the computer system determines the type of exercise associated with the location of the individual locations based on other types of markers and signals, or a combination of information present in the individual locations.

[0210] In part (B) of Figure 7N, the computer system displays a first view 7405 of a three-dimensional environment that includes a representation of the user's 7002's physical environment. In some embodiments, the first view 7405 of the three-dimensional environment is a real view with no virtual elements or minimal virtual elements, as shown in Figure 7N(B). In this example, the first view 7405 includes representations of physical surfaces (e.g., representations 7004' and 7006' of walls 7004 and 7006, representation 7008 of floor 7008, etc.) and representations of physical objects (e.g., representation 7402' of physical object 7402, representation 7404' of physical object 7404, etc.) without any virtual content. In some embodiments, the first view of the three-dimensional environment is a real view (e.g., view 7405 shown in Figure 7N(B)) with user interface objects for controlling basic functions of the computer system (e.g., application icons for launching various computer-generated experiences, display settings, audio controls, etc.). In some embodiments, the first view of the three-dimensional environment is an augmented reality view displayed at a low level of immersion (e.g., displaying user interface objects (e.g., application launch pad, welcome user interface, settings user interface) that are not part of a particular application experience (e.g., a wellness application, a meditation application, a workout application, a gaming application, etc.), that as a whole occupy only a small percentage (e.g., less than 10%, less than 20%) of the user's field of view, or that are displayed in a limited floating window, etc.). In some embodiments, the representation of the physical environment included in the first view of the three-dimensional environment 7405 is a camera view of a portion of the physical environment. In some embodiments, the portion of the physical environment shown in the first view of the three-dimensional environment 7405 changes as the user moves around the physical environment (e.g., when the user is wearing a display generating component on their head or holding a display generating component in their hand, etc.). In some embodiments, the portion of the physical environment shown in the first view of the three-dimensional environment 7405 changes as the display generating component moves around the physical environment.In some embodiments, the representation of the physical environment included in first view of three-dimensional environment 7405 is a view of the physical environment through a transparent portion of a display generation component. In this example, physical object 7402 is positioned at a first location within the first portion of the physical environment shown in first view of three-dimensional environment 7405, and physical object 7404 is positioned at a second location within the first portion of the physical environment shown in first view of three-dimensional environment 7405. In some embodiments, the first location and the second location are not necessarily within the same view of the three-dimensional environment, but may be located at two separate locations within the same physical environment or in different physical environments completely separate from each other. In this example, physical object 7402 corresponds to equipment or a setting corresponding to a first type of exercise (e.g., running, walking, etc.), and physical object 7402 corresponds to equipment or a setting corresponding to a second type of exercise (e.g., rowing, boating, water skiing, etc.).

[0211] In Figures 7O and 7P, a computer system detects movement of a user 7002 within a physical environment while displaying a first view 7450 of the three-dimensional environment. In some embodiments, the portions of the physical environment that are visible in the first view of the three-dimensional environment change as the user moves around in the physical environment. Figure 7O illustrates a first scenario in which the user 7002 has moved to a first location that includes physical objects 7404 or settings that correspond to a first type of exercise. Figure 7O illustrates a second scenario in which the user 7002 has moved to a second location that includes physical objects 7402 or settings that correspond to a second type of exercise.

[0212] In some embodiments, user movement includes movement of the entire user to a discrete location (e.g., a first location including the first physical object 7404, a second location including the second physical object 7402, etc.) (e.g., while the user is holding or wearing the display generating component, while a spatial relationship between the display generating component and the user is maintained such that the user can continue to view the physical environment through the display generating component, etc.). In some embodiments, user movement includes movement of the user to orient the display generating component or a camera associated with the display generating component to capture a view of a discrete location (e.g., a first location including the first physical object 7404, a second location including the second physical object 7402, etc.) (e.g., while the user is holding or wearing the display generating component, while a spatial relationship between the display generating component and the user is maintained such that the user can continue to view the physical environment through the display generating component, etc.). In some embodiments, user movement further includes movement corresponding to manipulation of a physical object(s) at a discrete location (e.g., turning on exercise equipment at a discrete location, picking up exercise equipment at a discrete location, beginning use of exercise equipment at a discrete location, etc.).

[0213] 7O, a user moves to a first location in a physical environment that includes a physical object 7404 corresponding to a first type of exercise. In this example, the user also moves to a position relative to the physical object 7404 that allows the user to begin using the physical object 7404 for the first type of exercise (e.g., sitting on the equipment, standing on the equipment, holding one or more parts of the equipment, etc.). In some embodiments, the computer system optionally detects that the user has begun one or more repetitions of an exercise corresponding to the first type of exercise (e.g., rowing, pulling a shift lever, assuming a starting position, etc.). In response to detecting movement of the first user to a first location including a physical object 7404 corresponding to the first type of exercise, and optionally in accordance with a determination that the first location corresponds to the first type of exercise, and that the user's movement satisfies a first set of criteria (e.g., criteria corresponding to the first location, criteria corresponding to the first type of exercise, etc.), the computer system displays a second view 7408 of the three-dimensional environment, the second view 7408 including first virtual content corresponding to the first type of exercise, the view of the first virtual content replacing at least a portion of the view of the physical environment including the first location (e.g., a location including physical object 7404 but not physical object 7402, a location not corresponding to the second type of exercise, etc.). In some embodiments, the first virtual content completely replaces the view of the physical environment in the second view 7408 of the three-dimensional environment. In some embodiments, the virtual content is displayed overlaying, occluding, or replacing the display of a representation of the physical environment in the second view 7408 of the three-dimensional environment.

[0214] In some embodiments, the computer system determines that the first location corresponds to a first type of exercise according to a determination that the first location has a first type of exercise equipment (e.g., a rowing machine, a boat, etc.) corresponding to the first type of exercise. In some embodiments, the computer system determines that the first location corresponds to a first type of exercise according to a determination that the first location is a location designed (e.g., has a suitable floor surface, structure, etc.) for the first type of exercise (e.g., rowing, meditation, etc.).

[0215] As shown in part (B) of Figure 7O, the computer system displays a second view 7408 of the three-dimensional environment when the user 7002 moves to a first location corresponding to the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view with more virtual elements corresponding to the first location and the first computer-generated experience corresponding to the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view showing a preview or start of the first computer-generated experience corresponding to the first location and the first type of exercise. In some embodiments, the second view 7408 is an augmented reality view displayed at a higher level of immersion (e.g., displaying user interface objects that are part of a first particular application experience corresponding to a first type of exercise (e.g., virtual hiking trails, virtual scenery, leaderboards, exercise statistics, controls to modify exercise parameters, etc.) and that occupy a significant percentage (e.g., more than 60%, more than 90%, etc.) of the user's field of view as a whole, or are displayed in the three-dimensional virtual or augmented reality environment. In this example, the virtual content displayed in the second view of the three-dimensional environment includes representations 7004′, 7006′, and / or a virtual open body of water 7406 that replaced the view in 7008′ of various portions of the physical environment potentially within the field of view provided by the display generation component of the first location. In some embodiments, all portions of the physical environment within the potential field of view provided by the display generation component are replaced or occluded by the display of the virtual content. In some embodiments, portions of the physical environment, such as parts of the user's body, at least a portion of the exercise equipment, etc., remain visible in the second view 7408 of the three-dimensional environment.

[0216] 7P, the user has moved to a second location in the physical environment that includes a physical object 7402 corresponding to a second type of exercise. In this example, the user has also moved to a position relative to the physical object 7402 that allows the user to begin using the physical object 7402 for the second type of exercise (e.g., sitting on the equipment, standing on the equipment, holding one or more parts of the equipment, etc.). In some embodiments, the computer system optionally detects that the user has begun one or more repetitions of a movement corresponding to the second type of exercise (e.g., taking a step, starting to pedal, starting to walk, etc.). In response to detecting movement of the user to a second location including a physical object 7402 corresponding to the second type of exercise, and in accordance with determining that the second location corresponds to the second type of exercise, and optionally determining that the user's movement satisfies a second set of criteria (e.g., criteria corresponding to the second location, criteria corresponding to the second type of exercise, etc.), the computer system displays a third view 7412 of the three-dimensional environment, the third view 7412 including second virtual content corresponding to the second type of exercise, wherein the view of the second virtual content replaces at least a portion of the view of the physical environment including the second location (e.g., a location corresponding to the second type of exercise but not the first type of exercise, a location that does not include the physical object 7404, etc.). In some embodiments, the first virtual content completely replaces the view of the physical environment in the third view 7410 of the three-dimensional environment. In some embodiments, the virtual content is displayed overlaying, occluding, or replacing the display of at least a portion of the representation of the physical environment.

[0217] In some embodiments, the computer system determines that the second location corresponds to the second type of exercise according to a determination that the second location has a second type of exercise equipment (e.g., stairs, a stepper, a treadmill, etc.) corresponding to the second type of exercise. In some embodiments, the computer system determines that the second location corresponds to the second type of exercise according to a determination that the second location is a location designed (e.g., has a suitable floor surface, structure, etc.) for the second type of exercise (e.g., hiking, running, etc.).

[0218] As shown in portion (B) of FIG. 7P , the computer system displays a third view 7410 of the three-dimensional environment when the user 7002 travels to a second location corresponding to a second type of exercise. In some embodiments, the third view 7410 is an augmented reality view with more virtual elements corresponding to the second location and a second computer-generated experience corresponding to the second type of exercise. In some embodiments, the third view 7410 is an augmented reality view that shows a preview or start of the second computer-generated experience corresponding to the second location and the second type of exercise. In some embodiments, the third view 7410 is an augmented reality view displayed at a higher level of immersion (e.g., displaying user interface objects that are part of a second specific application experience corresponding to the second type of exercise (e.g., virtual hiking trails, virtual scenery, leaderboards, exercise statistics, controls for changing exercise parameters, etc.) that collectively occupy a significant percentage (e.g., more than 60%, more than 90%, etc.) of the user's field of view or are displayed in the three-dimensional virtual or augmented reality environment). In this example, the virtual content displayed in the third view 7410 of the three-dimensional environment includes a virtual hiking trail 7412 that replaces the view of representations 7004', 7006', and / or 7008' of various portions of the physical environment potentially within the field of view provided by the display generation component at the second location. In some embodiments, all portions of the physical environment within the potential field of view provided by the display generation component are replaced or occluded by the display of the virtual content. In some embodiments, portions of the physical environment, such as portions of the user's body, at least a portion of exercise equipment, etc., remain visible in the third view 7410 of the three-dimensional environment.

[0219] In some embodiments, the computer system determines that the current location corresponds to a particular type of exercise according to detecting a particular type of exercise equipment corresponding to the particular type of exercise at the current location, hi some embodiments, detecting the particular type of exercise equipment based on detecting an RFID signal corresponding to the particular type of exercise equipment, detecting an image of the particular type of exercise equipment in a camera feed capturing the current location, detecting that the current location matches a registered location of the particular type of exercise equipment, etc.

[0220] In some embodiments, following a determination that the user's current location corresponds to a location associated with the respective type of exercise, the computer system displays a view of the three-dimensional environment corresponding to the respective type of exercise, which includes gradually reducing the visual prominence of a representation of the physical environment in the currently displayed view of the three-dimensional environment while increasing the visual prominence of virtual content corresponding to the respective type of exercise associated with the current location in the view of the three-dimensional environment. In some embodiments, reducing the visual prominence of the representation of the physical environment includes ceasing to display more and more portions of the representation of the physical environment, fading out the representation of the physical environment, etc. In some embodiments, gradually increasing the visual prominence of the virtual content corresponding to the respective type of exercise includes beginning to display the virtual content in the region of the view of the three-dimensional environment where the representation of the physical environment is gradually reduced, increasing the visibility of the virtual content, increasing the proportion of the user's field of view occupied by the virtual content, increasing the opacity or brightness of the virtual content, etc.

[0221] In some embodiments, the individual locations may correspond to multiple types of exercises, and the computer system requires the user to make some movement corresponding to a respective one of the multiple types of exercises to clarify which type of exercise the user wants to perform, and selects corresponding virtual content for display in the view of the three-dimensional environment at the individual location. For example, in some embodiments, the computer system detects a movement corresponding to a respective one of the multiple types of exercises associated with the individual location (e.g., the start of a characteristic motion (e.g., starting to walk on a treadmill, stepping on a stair stepper, moving legs back and forth on an elliptical, or starting rowing on a rowing machine, etc.), stepping / sitting on exercise equipment corresponding to the individual type of exercise (e.g., sitting on a rowing machine or weight training machine, etc.), or a readiness position corresponding to the individual type of exercise (e.g., standing in a readiness position to hit a virtual tennis ball, sitting on the floor to begin meditation or yoga, etc.), and the computer system displays a view of the three-dimensional environment including virtual content corresponding to the individual type of exercise.

[0222] In some embodiments, the computer system gradually changes the virtual content displayed in the view of the three-dimensional environment according to the progress of the individual types of exercise performed by the user at the individual locations. For example, in some embodiments, the view of the real world gradually disappears and / or ceases to be displayed and is gradually replaced by virtual content corresponding to the individual types of exercise. In some embodiments, the computer system gradually increases the amount of virtual content displayed in the first user's field of view until the individual virtual environment corresponding to the individual types of exercise is fully displayed via the first display generation component (e.g., a second view of the three-dimensional environment includes the virtual environment corresponding to the first type of exercise, a third view of the three-dimensional environment includes the virtual environment corresponding to the second type of exercise, etc.). For example, in some embodiments, when an open gym is a location associated with both yoga and dance, after the first user arrives at the open gym, if the first user sits in namaste pose, the computer system displays a virtual ocean view accompanied by ocean sounds for the user to practice yoga on a virtual beach, and if the first user stands in dancer's pose, the computer system displays a virtual stage accompanied by dance music for the user to practice dance.

[0223] In some embodiments, when the computer system detects that the user has left the individual location, the computer system stops displaying virtual content corresponding to the type of exercise associated with the individual location. For example, in FIG. 7O , if the computer system detects that the user 7002 has left the first location including the physical object 7404, after displaying the view 7408, the computer system stops displaying the view 7408 corresponding to the first type of exercise. In some embodiments, the computer system redisplays the view 7405, which does not include virtual content corresponding to either the first type of exercise or the second type of exercise. In some embodiments, when the computer system detects that the user has moved from the first location to the second location, the computer system displays virtual content 7410 corresponding to the second type of exercise.

[0224] In some embodiments, when the computer system displays a view of a three-dimensional environment including virtual content corresponding to a particular type of exercise, it displays status information corresponding to the particular type of exercise (e.g., progress during the current session, duration, speed, force, height, pace, stride length, performance level, score, number of repetitions completed, etc., historical statistics, average statistics for the first user and / or across multiple users, status of other users performing the same type of exercise, etc.).

[0225] In some embodiments, the computer system displays corresponding health information to the user (e.g., real-time biometric data (e.g., heart rate, blood pressure, respiratory rate, body temperature, blood glucose level, etc.), weight, BMI, etc.) when displaying a view of a three-dimensional environment including virtual content corresponding to a particular type of exercise.

[0226] In some embodiments, the computer system visually presents progress information (e.g., real-time score, completed laps, remaining laps, duration, number of steps, distance traveled, completed poses, etc.) of a particular type of exercise performed by a user when displaying a view of a three-dimensional environment including virtual content corresponding to the particular type of exercise.

[0227] In some embodiments, the three-dimensional environment including the virtual content corresponding to a particular type of exercise is an immersive environment and includes a spatial extent that is greater than the spatial extent included in the currently displayed view of the three-dimensional environment, for example, as the user turns their head or otherwise changes the viewpoint corresponding to the currently displayed view of the three-dimensional environment, different portions of the virtual content are displayed in the currently displayed view of the three-dimensional environment.

[0228] In some embodiments, the second and / or third views of the three-dimensional environment include a virtual representation of the user shown competing with the user to perform a particular type of exercise (e.g., based on the first user's previous best performance, based on the first user's preset configuration for the first type of exercise, etc.).

[0229] In some embodiments, the second and / or third views of the three-dimensional environment include a virtual representation of at least another user different from the user who is shown competing with the user to perform a particular type of exercise.

[0230] As disclosed herein, in some embodiments, the three-dimensional environment displayed via the display generation component is a virtual three-dimensional environment that includes virtual objects and content at various virtual positions within the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at various virtual positions within the three-dimensional environment constrained by one or more physical aspects of the physical environment (e.g., the position and orientation of walls, floors, surfaces, direction of gravity, time of day, etc.). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. The representation of the physical environment includes representations of physical objects and surfaces at different positions within the three-dimensional environment, such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. When virtual objects are placed relative to the positions of the representations of the physical objects and surfaces in the three-dimensional environment, the virtual objects appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment. In some embodiments, the display generation component includes a pass-through portion in which the representation of the physical environment is displayed. In some embodiments, the pass-through portion is a transparent or translucent (e.g., see-through) portion of the display generation component that surrounds the user's field of view and reveals at least a portion of the physical environment within the user's field of view. For example, the pass-through portion is a portion of a head-mounted display that is translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world surrounding the user through the pass-through portion without removing the head-mounted display or moving away from the head-up display. In some embodiments, the pass-through portion gradually transitions from translucent or transparent to fully opaque when displaying a virtual or mixed reality environment.In some embodiments, the pass-through portion of the display generation component displays a live feed of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., rear-facing camera(s) associated with a mobile device or a head-mounted display, or other cameras that provide image data to an electronic device). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generation component). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user). In some embodiments, when displaying virtual objects or content in positions that correspond to the locations of one or more physical objects in the physical environment, at least some of the virtual objects are displayed in place of (e.g., replace the display of) a portion of the camera's live view (e.g., a portion of the physical environment captured in the live view). In some embodiments, at least some of the virtual objects and content are projected onto physical surfaces or open space in the physical environment and are visible through the pass-through portion of the display generation component (e.g., as part of a camera view of the physical environment, or visible through a transparent or semi-transparent portion of the display generation component, etc.). In some embodiments, at least some of the virtual objects and content are displayed to overlay a portion of the display, blocking the view of at least some, but not all, of the physical environment that is visible through a transparent or semi-transparent portion of the display generation component. In some embodiments, at least some of the virtual objects are projected directly onto the user's retina in a position relative to an image of a representation of the physical environment (e.g., as seen through a camera view of the physical environment or through a transparent portion of the display generation component).

[0231] In some embodiments, input gestures used in various examples and embodiments described herein (e.g., with respect to Figures 7A-7P and 8-12) optionally include discrete, small-movement gestures performed by moving a user's finger(s) relative to other finger(s) or part(s) of the user's hand, optionally without requiring the user's entire hand or arm to move significantly away from their natural location(s) and posture(s) to perform an action just before or during the gesture, in order to interact with a virtual or mixed reality environment, according to some embodiments.

[0232] In some embodiments, the input gesture is detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1 , image sensor 314 of FIG. 3 ). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of, or provide data to, a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD)) that includes a display generation component (e.g., display generation component 120 of FIGS. 1 , 3, and 4 (e.g., a touchscreen display that functions as both a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the device's display. In some embodiments, the input gesture is detected by a sensor system of a head-mounted system (e.g., a VR headset that includes a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head-mounted system are mounted on the front and / or bottom of the head-mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head-mounted system is used (e.g., arrayed around the head-mounted system at various locations in a room) such that the imaging sensors capture images of the head-mounted system and / or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (e.g., a head-up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, the one or more imaging sensors are mounted on an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array).For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and / or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and / or sound emitters (e.g., ultrasonic emitters). For example, light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200) while images of the hand under the light illumination are captured by one or more cameras, and the captured images are analyzed to determine the position and / or configuration of the hand. By determining input gestures using signals from image sensors directed at the hand, as opposed to using signals of a touch-sensitive surface or other direct-contact or proximity-based mechanism, a user can freely choose to perform large movements or remain relatively still when providing input gestures with their hand without experiencing constraints imposed by a particular input device or input area.

[0233] In some embodiments, the tap input optionally indicates a thumb tap input on the index finger of the user's hand (e.g., on the side of the index finger adjacent to the thumb). In some embodiments, the tap input is detected without having to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected according to a determination that a downward movement of the thumb is followed by an upward movement of the thumb and the thumb is in contact with the side of the index finger for less than a threshold time. In some embodiments, the tap hold input is detected according to a determination that the thumb moves from an up position to a touch down position and remains in the touch down position for at least a first threshold time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some embodiments, the computer system requires that the entire hand remain substantially stationary in a location for at least a first threshold time to detect a thumb tap hold input with the thumb on the index finger. In some embodiments, the touch hold input is detected without requiring the hand to remain substantially stationary (e.g., the entire hand can move while the thumb is resting on the side of the index finger). In some embodiments, a taphole drag input is detected when the thumb touches the side of the index finger and the whole hand moves while the thumb remains stationary on the side of the index finger.

[0234] In some embodiments, the flick gesture optionally indicates a push or flick input of the thumb moving across the index finger (e.g., from the palm side of the index finger to the back side). In some embodiments, the extension movement of the thumb is accompanied by an upward movement away from the side of the index finger, e.g., as in an upward flick input by the thumb. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves forward and upward. In some embodiments, a reverse flick input is performed by the thumb moving from an extended position to a retracted position. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves backward and downward.

[0235] In some embodiments, the swipe gesture is a swipe input, optionally by movement of the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or along the side of the palm). In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or a bent state. In some embodiments, the index finger moves between an extended state and a bent state during movement of the thumb in the swipe input gesture.

[0236] In some embodiments, different phalanges of various fingers correspond to different inputs. Thumb tap inputs across various phalanges of various fingers (e.g., index, middle, ring, and optionally pinky) are optionally mapped to different actions. Similarly, in some embodiments, different push or click inputs can be performed by the thumb across different fingers and / or different portions of the fingers to trigger different actions in distinct user interface contacts. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and / or in different directions (e.g., toward the distal or proximal end of the finger) trigger different actions in distinct user interface contexts.

[0237] In some embodiments, the computer system processes tap inputs, flick inputs, and swipe inputs as different types of inputs based on the type of thumb movement. In some embodiments, the computer system processes inputs having different finger locations tapped, touched, or swiped by the thumb as different sub-input types (e.g., proximal, intermediate, distal subtypes, or index, middle, ring, or pinky subtypes) of a given input type (e.g., tap input type, flick input type, swipe input type, etc.). In some embodiments, the amount of movement performed by the moving finger (e.g., thumb) and / or other movement measures associated with the finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively affect the action triggered by the finger input.

[0238] In some embodiments, the computer system recognizes combination input types that combine a series of thumb movements, such as a tap-swipe input (e.g., the thumb touching down on another finger and then swiping along the side of the finger), a tap-flick input (e.g., the thumb touching down on another finger and then flicking across the finger from the side of the palm to the back of the finger), and a double-tap input (e.g., two consecutive taps on the side of the finger in approximately the same location).

[0239] In some embodiments, the gesture input is performed with the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move toward each other to perform a pinch gesture). In some embodiments, a wrist movement (e.g., a horizontal or vertical wrist flick) is performed immediately before, immediately after (e.g., within a threshold time), or simultaneously with the finger movement input to trigger an additional, different, or modified action in the current user interface context compared to a finger movement input without the modified wrist movement input. In some embodiments, a finger input gesture performed with a user's palm facing the user's face is treated as a different type of gesture than a finger input gesture performed with a user's palm facing away from the user's face. For example, a tap gesture performed with a user's palm facing the user performs an action with added (or reduced) privacy protection compared to an action (e.g., the same action) performed in response to a tap gesture performed with a user's palm facing away from the user's face.

[0240] While one type of finger input may be used to trigger an action type in the examples provided in this disclosure, in other embodiments, other types of finger input are optionally used to trigger the same type of action.

[0241] Additional explanation regarding Figures 7A-7P is provided below with reference to methods 8000, 9000, 10000, 11000, and 12000 described with respect to Figures 8-12 below.

[0242] FIG. 8 is a flowchart of a method for supporting interaction with user interface objects in a computer-generated three-dimensional environment shared between two or more users, according to some embodiments.

[0243] In some embodiments, method 8000 is performed on a computer system (e.g., first computer system 101 of FIG. 1 ). The display generation component (e.g., display generation component 120 of FIGS. 1 , 3 , and 4 ) includes (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from a user's hand or a user's head. In some embodiments, method 8000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 8000 are optionally combined and / or the order of some operations is optionally changed.

[0244] In some embodiments, method 8000 is performed on a computer system (e.g., first computer system 101 of FIG. 1 ) in communication with a display generation component (e.g., display generation component 120, display generation component 7100, etc. of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, HMD, display, touch screen, projector, etc.) and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, button, etc.). In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as the display generation component and at least some of the one or more input devices. In some embodiments, the computer system includes a computing component including one or more processors and memory that are separate from the display generation component and / or the one or more input devices. In some embodiments, the display generation component and the one or more input devices are integrated within and enclosed in the same housing.

[0245] In method 8000, a computer system displays (8002) a first user interface object (e.g., user interface object 7016 of FIG. 7B , another user interface object, etc.) (e.g., a representation of an application, a user interface including a plurality of user interface objects (e.g., selectable avatars, selectable menu items, selectable device controls, selectable content items, slider controls, buttons, etc.), a virtual three-dimensional object, a control, a control panel including a plurality of controls corresponding to different functions or operations, an information item, a media item, etc.) in a first view (e.g., first view 7015-1 of FIG. 7B , another first view, etc.) of a three-dimensional environment, and the three-dimensional environment is at least partially shared (e.g., at least a spatial portion of the environment is shared, the environment is shared at least for a period of time, objects in the environment are fully or partially shared (e.g., For example, the first user interface object may be simultaneously viewable and accessible, simultaneously viewable but not simultaneously accessible, viewable but not accessible when another has control (e.g., the other has control (e.g., the object can be viewed or not viewed by the other), etc.), (e.g., when at least a portion of the three-dimensional environment (e.g., a portion shown in the first view of the three-dimensional environment, another portion of the three-dimensional environment, etc.) is displayed for viewing by both the first user and the second user, and / or when a virtual object within the three-dimensional environment (e.g., the first user interface object, another user interface object, etc.) is simultaneously displayed within the three-dimensional environment shown to both the first user and the second user), (e.g., as shown in FIG. 7B ) at a first position within the first view of the three-dimensional environment with a first set of appearance characteristics (e.g., the normal appearance (e.g., a first shape, a first size, a first color, a first opacity, a first saturation, a first brightness, etc.) of the first user interface object as displayed to the second user by the second display generation component)7B (first view 7015-1). While displaying a first user interface object having a first set of appearance characteristics at ...

Claims

1. 1. A method comprising:

1. A computer system including a first display generation component and one or more first input devices, displaying a first user interface object in a first view of a three-dimensional environment at least partially shared between a first user and a second user, the first user interface object being displayed at a first position within the first view of the three-dimensional environment with a first set of appearance characteristics; detecting a first user input provided by the first user while displaying the first user interface object with the first set of appearance characteristics at the first position within the first view of the three-dimensional environment, the first user input being directed at the first user interface object; in response to detecting the first user input directed at the first user interface object; performing a first action on the first user interface object in accordance with the first user input in accordance with determining that the second user is not currently interacting with the first user interface object; in response to a determination that the second user is currently interacting with the first user interface object; displaying a visual indication that the first user interface object is unavailable for interaction with the first user, wherein displaying the visual indication includes changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment; ceasing to perform the first action on the first user interface object in accordance with the first user input; A method comprising:

2. 2. The method of claim 1 , wherein the computer system modifies the appearance of the first user interface object as the visual indication that the first user interface object is unavailable for interaction with the first user, and wherein modifying the appearance of the first user interface object includes modifying at least one of the first set of appearance characteristics of the first user interface object to reduce visual prominence of the first user interface object.

3. Detecting an end of the first user input directed to the first user interface object; in response to detecting the termination of the first user input directed at the first user interface object, restoring at least one of the first set of appearance characteristics of the first user interface object that was modified in response to the first user input to restore the visual prominence of the first user interface object; The method of claim 2 , comprising:

4. while continuing to detect the first user input, detecting movement of the first user interface object away from the first position within the first view of the three-dimensional environment independent of the detection of the first user input; In response to detecting the movement of the first user interface object away from the first position within the first view of the three-dimensional environment independent of the detection of the first user input, restoring at least one of the first set of appearance characteristics of the first user interface object that was modified in response to the first user input to restore the visual prominence of the first user interface object; The method of claim 2 or 3, comprising:

5. 5. The method of claim 1, wherein displaying the visual indication that the first user interface object is unavailable for interaction with the first user comprises maintaining changes to the appearance of the first user interface object made in response to the first user input until the second user stops interacting with the first user interface object.

6. in response to detecting the first user input directed at the first user interface object; displaying the first user interface object in the first view of the three-dimensional environment at a second position in accordance with a determination that the second user is not currently interacting with the first user interface object, the second position being selected according to a current location of the first user's hand; detecting a movement of the first user's hand corresponding to a throwing gesture of the first user's hand while displaying the first user interface object at the second position selected according to the current location of the first user's hand; in response to detecting the movement of the first user's hand corresponding to the throwing gesture of the first user's hand, moving the first user interface object within the first view of the three-dimensional environment in a first direction corresponding to the direction of the movement of the first user's hand and rotating the first user interface object during the movement of the first user interface object; 6. The method of claim 1, comprising:

7. Rotating the first user interface object while moving the first user interface object includes:

7. The method of claim 6, further comprising: in accordance with a determination that the direction of the movement of the hand of the first user is toward a representation of the second user in the first view of the three-dimensional environment, rotating the first user interface object in a first manner such that the first user interface object has a first preset orientation within the three-dimensional environment when it reaches a destination position within the three-dimensional environment selected in accordance with the movement of the hand of the first user in a first physical environment.

8. Rotating the first user interface object while moving the first user interface object includes:

8. The method of claim 6 or 7, further comprising: in accordance with a determination that the direction of the movement of the hand of the first user is toward a representation of a first surface in the first view of the three-dimensional environment, rotating the first user interface object in a second manner so that the first user interface object has a second preset orientation relative to the representation of the first surface in the three-dimensional environment when the first user interface object reaches a destination position on the representation of the first surface selected in accordance with the movement of the hand of the first user in the first physical environment.

9. 9. The method of claim 1, wherein the computer system alters the position of the first user interface object within the first view of the three-dimensional environment as the visual indication that the first user interface object is unavailable for interaction with the first user, and wherein altering the position of the first user interface object within the first view of the three-dimensional environment comprises moving the first user interface object from the first position to maintain at least a predetermined distance between the first user interface object and a hand representation of the first user that provided the first user input.

10. 10. The method of claim 1, wherein performing the first action on the first user interface object in accordance with the first user input comprises moving the first user interface object toward a representation of the first user's hand.

11. Performing the first action on the first user interface object in accordance with the first user input comprises:

11. The method of claim 1, further comprising: selecting the first user interface object as a target for subsequent input received from the first user in accordance with a determination that the first user input includes a predetermined selection gesture.

12. 12. The method of claim 11, further comprising, in conjunction with selecting the first user interface object as a target for subsequent input received from the first user, displaying a representation of the first user interface object at a position corresponding to a location of a hand of the first user while maintaining the first user interface object at the first position within the first view of the three-dimensional environment.

13. detecting a second user input on the first user interface object after the first action is performed on the first user interface object in accordance with the first user input; In response to detecting the second user input, selecting the first user interface object as a target for subsequent input received from the first user in accordance with a determination that the second user input includes a predetermined selection gesture; 11. The method of claim 1, comprising:

14. Performing the first action on the first user interface object in accordance with the first user input comprises:

14. The method of claim 1, comprising displaying a representation of the first user interface object while maintaining the first user interface object in the first position within the first view of the three-dimensional environment.

15. The representation of the first user interface object is initially displayed in a position spaced apart from a representation of the first user's hand, and the method further comprises:

15. The method of claim 14, further comprising: moving the representation of the first user interface object from the position away from the representation of the hand of the first user to a position of the representation of the hand of the first user in accordance with a determination that the first user interface object is selected by a subsequent user input provided by the first user.

16. detecting a movement of the first user's hand while displaying the representation of the first user interface object; In response to detecting the movement of the hand of the first user, modifying an appearance of the representation of the first user interface object in accordance with a determination that the movement of the hand of the first user satisfies preset criteria for identifying an initial portion of a preset selection gesture; 16. The method of claim 14 or 15, comprising:

17. pursuant to determining that the first user interface object has been selected by the first user, detecting a third user input interacting with the representation of the first user interface object while displaying the representation of the first user interface object at the position of the representation of the hand of the first user; In response to detecting the third user input, displaying visual feedback for the third user input through at least one of moving the representation of the first user interface object and changing an appearance of the representation of the first user interface object; and performing a second action on the first user interface object according to the third user input; 17. The method of claim 15 or 16, comprising:

18. 18. The method of claim 15, further comprising updating the position of the representation of the first user interface object in accordance with movement of the first user's hand such that the representation of the first user interface object maintains an existing spatial relationship with the updated position of the representation of the hand of the first user.

19. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a first user interface object in a first view of a three-dimensional environment that is at least partially shared between a first user and a second user, the first user interface object being displayed at a first position within the first view of the three-dimensional environment with a first set of appearance characteristics; detecting a first user input provided by the first user while displaying the first user interface object with the first set of appearance characteristics at the first position within the first view of the three-dimensional environment, the first user input being directed at the first user interface object; in response to detecting the first user input directed at the first user interface object; performing a first action on the first user interface object in accordance with the first user input in accordance with determining that the second user is not currently interacting with the first user interface object; in response to a determination that the second user is currently interacting with the first user interface object; displaying a visual indication that the first user interface object is unavailable for interaction with the first user, the visual indication including changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment; A computer system comprising instructions for canceling performing the first action on the first user interface object in accordance with the first user input.

20. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to: displaying a first user interface object in a first view of a three-dimensional environment at least partially shared between a first user and a second user, the first user interface object being displayed at a first position within the first view of the three-dimensional environment with a first set of appearance characteristics; detecting a first user input provided by the first user while displaying the first user interface object with the first set of appearance characteristics at the first position within the first view of the three-dimensional environment, the first user input being directed at the first user interface object; in response to detecting the first user input directed at the first user interface object; performing a first action on the first user interface object in accordance with the first user input in accordance with determining that the second user is not currently interacting with the first user interface object; in response to a determination that the second user is currently interacting with the first user interface object; displaying a visual indication that the first user interface object is unavailable for interaction with the first user, wherein displaying the visual indication includes changing at least one of an appearance of the first user interface object or a position of the first user interface object within the first view of the three-dimensional environment; and ceasing to perform the first action on the first user interface object in accordance with the first user input.

21. 1. A method comprising: A computer system in communication with a first display generation component and one or more first input devices, comprising: displaying, while a first user is at a first location within a first physical environment, a first view of a three-dimensional environment corresponding to a first viewpoint associated with the first location within the first physical environment, the first view of the three-dimensional environment including a first user interface object representing a first object in a second physical environment different from the first physical environment, wherein distinct positions of the first user interface object within the three-dimensional environment correspond in a first manner to distinct locations of the first object in the second physical environment; Detecting at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; in response to detecting the at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; displaying a second view of the three-dimensional environment corresponding to a second viewpoint; displaying the first user interface object in the second view of the three-dimensional environment; displaying the first user interface object at a first display position within the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object within the three-dimensional environment corresponding to the individual location of the first object within the second physical environment in the first manner is more than a threshold distance from an individual position within the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment, the first display position being the individual position of the first user interface object within the three-dimensional environment; displaying the first user interface object at a second display position within the second view of the three dimensional environment, the second display position being offset from the individual position of the first user interface object within the three dimensional environment in accordance with a determination that the individual position of the first user interface object within the three dimensional environment in the first manner is less than the threshold distance from the individual position within the three dimensional environment corresponding to the second viewpoint associated with the second view of the three dimensional environment; and A method comprising:

22. detecting at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment includes detecting a first movement of the first user within the first physical environment while the first object remains stationary within the second physical environment; during the first movement of the first user within the first physical environment; In response to the determination that the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner is more than the threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment, the first user interface object is displayed at the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner; 22. The method of claim 21 , wherein, in accordance with the determination that the individual position of the first user interface object in the three dimensional environment that corresponds to the individual location of the first object in the second physical environment in the first manner is less than or equal to the threshold distance from the individual position in the three dimensional environment that corresponds to the viewpoint associated with the currently displayed view of the three dimensional environment, the first user interface object is displayed at an adjusted position in the three dimensional environment while the first object remains stationary in the second physical environment, and the adjusted position in the three dimensional environment corresponds to the individual location of the first object in the second physical environment in a second manner that is different from the first manner.

23. detecting at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment includes detecting a second movement of the first object within the second physical environment while the first user is stationary within the first physical environment; during the second movement of the first object within the second physical environment; pursuant to the determination that the individual position of the first user interface object in the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner is more than the threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment, the first user interface object is displayed at an updated position in the three dimensional environment corresponding to an updated location of the first object in the second physical environment as a result of the second movement in the first manner; 23. The method of claim 21 or 22, wherein, in accordance with the determination that the individual position of the first user interface object in the three dimensional environment that corresponds to the individual location of the first object in the second physical environment in the first manner is less than or equal to the threshold distance from the individual position in the three dimensional environment that corresponds to the viewpoint associated with the currently displayed view of the three dimensional environment, the first user interface object is displayed at an adjusted updated position in the three dimensional environment that corresponds to the updated location of the first object in the second physical environment in a second manner that is different from the first manner, while the first object remains stationary in the second physical environment.

24. detecting at least one of a movement of the first user within the first physical environment and a movement of the first object within the second physical environment includes simultaneously detecting a third movement of the first user within the first physical environment and a fourth movement of the first object within the second physical environment; during the third movement of the first user within the first physical environment and the fourth movement of the first object within the second physical environment; in response to the determination that the individual position of the first user interface object in the three dimensional environment corresponding to the location of the first object in the second physical environment in the first manner is more than the threshold distance from the individual position in the three dimensional environment corresponding to a viewpoint associated with the currently displayed view of the three dimensional environment, the first user interface object is displayed at an updated position in the three dimensional environment corresponding to an updated location of the first object in the second physical environment as a result of the fourth movement in the first manner; 24. The method of claim 21, wherein, in accordance with the determination that the individual position of the first user interface object in the three-dimensional environment that corresponds to the location of the first object in the second physical environment in the first manner does not exceed the threshold distance from the individual position in the three-dimensional environment that corresponds to the viewpoint associated with the currently displayed view of the three-dimensional environment, the first user interface object is displayed at an adjusted, updated position in the three-dimensional environment that corresponds to the location of the first object in the second physical environment in a third manner that is different from the first manner.

25. 25. The method of any one of claims 21 to 24, wherein the first object is a second user located in the second physical environment, and the first user and the second user at least partially share the three-dimensional environment.

26. 26. The method of claim 25, wherein, in a third view of the three-dimensional environment displayed to the second user via a second display generation component, in accordance with a determination that an individual position of a second user interface object in the three-dimensional environment corresponding to an individual location of the first user in the first physical environment in the first manner is not more than the threshold distance from an individual position in the three-dimensional environment corresponding to a third viewpoint associated with the third view of the three-dimensional environment, the second user interface object is displayed at a modified position in the third view of the three-dimensional environment, the modified position being offset from the individual position of the second user interface object in the three-dimensional environment corresponding to the individual location of the first user in the first physical environment in the first manner.

27. 27. The method of any one of claims 21 to 26, wherein the first object is a physical object located in the second physical environment.

28. 28. The method of any one of claims 21 to 27, wherein the first user interface object is an object floating within the three-dimensional environment.

29. In response to determining that the three-dimensional environment is displayed at a first level of reality, the first user interface object is displayed with a first set of display characteristics corresponding to the first level of reality; 29. The method of claim 21, wherein, in accordance with a determination that the three-dimensional environment is to be displayed at a second level of reality that is different from the first level of reality, the first user interface object is displayed with a second set of display characteristics that corresponds to the second level of reality, the second set of display characteristics being different from the first set of display characteristics.

30. 30. The method of claim 21, wherein the second display position within the second view of the three-dimensional environment is displaced by a first displacement amount from the individual position of the first user interface object within the three-dimensional environment that corresponds to the individual location of the first object within the second physical environment in the first manner, and the first displacement amount does not correspond to a movement of the first object within the second physical environment in the first manner.

31. 31. The method of claim 30, wherein the first displacement amount has a direction determined according to a spatial relationship between a viewpoint of a currently displayed view of the three-dimensional environment and the respective position of the first user interface object in the three-dimensional environment that corresponds to the respective location of the first object in the second physical environment in the first manner.

32. 32. The method of claim 21, wherein the second display position within the second view of the three-dimensional environment is displaced by a second displacement amount from the individual position of the first user interface object in the three-dimensional environment that corresponds to the individual location of the first object in the second physical environment in the first manner, the second displacement amount having a direction different from a forward direction toward the individual position in the three-dimensional environment that corresponds to the second viewpoint associated with the second view of the three-dimensional environment.

33. 32. The method of claim 21, wherein the second display position in the second view of the three-dimensional environment is displaced in the first manner by a third displacement amount from the respective position of the first user interface object in the three-dimensional environment corresponding to the respective location of the first object in the second physical environment, the third displacement amount having a direction different from a direction of approach between the first user interface object and the respective position in the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment.

34. 34. The method of claim 21, wherein at least one of a magnitude and a direction of displacement between the second display position in the second view of the three-dimensional environment and the individual position of the first user interface object in the three-dimensional environment that corresponds to the individual location of the first object in the second physical environment in the first manner is based on a spatial relationship between the individual position of the first user interface object in the three-dimensional environment that corresponds to the individual location of the first object in the second physical environment in the first manner and the individual position in the three-dimensional environment that corresponds to the second viewpoint associated with the second view of the three-dimensional environment.

35. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, while a first user is at a first location within a first physical environment, a first view of a three-dimensional environment corresponding to a first viewpoint associated with the first location within the first physical environment, the first view of the three-dimensional environment including a first user interface object representing a first object in a second physical environment different from the first physical environment, wherein distinct positions of the first user interface object within the three-dimensional environment correspond in a first manner to distinct locations of the first object within the second physical environment; Detecting at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; in response to detecting the at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; displaying a second view of the three-dimensional environment corresponding to a second viewpoint; Displaying the first user interface object in the second view of the three-dimensional environment includes: displaying the first user interface object at a first display position within the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object within the three-dimensional environment corresponding to the individual location of the first object within the second physical environment in the first manner is more than a threshold distance from an individual position within the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment, the first display position being the individual position of the first user interface object within the three-dimensional environment; and displaying the first user interface object in the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object in the three-dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner is less than the threshold distance from the individual position in the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment, the second display position being offset from the individual position of the first user interface object in the three-dimensional environment.

36. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to: displaying, while a first user is at a first location within a first physical environment, a first view of a three-dimensional environment corresponding to a first viewpoint associated with the first location within the first physical environment, the first view of the three-dimensional environment including a first user interface object representing a first object in a second physical environment different from the first physical environment, wherein distinct positions of the first user interface object within the three-dimensional environment correspond in a first manner to distinct locations of the first object in the second physical environment; Detecting at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; in response to detecting the at least one of movement of the first user within the first physical environment and movement of the first object within the second physical environment; displaying a second view of the three-dimensional environment corresponding to a second viewpoint; displaying the first user interface object in the second view of the three-dimensional environment; displaying the first user interface object at a first display position within the second view of the three-dimensional environment in accordance with a determination that the individual position of the first user interface object within the three-dimensional environment corresponding to the individual location of the first object within the second physical environment in the first manner is more than a threshold distance from the individual position within the three-dimensional environment corresponding to the second viewpoint associated with the second view of the three-dimensional environment, the first display position being the individual position of the first user interface object within the three-dimensional environment; and displaying the first user interface object at a second display position within the second view of the three dimensional environment, the second display position being offset from the individual position of the first user interface object within the three dimensional environment, in accordance with a determination that the individual position of the first user interface object within the three dimensional environment corresponding to the individual location of the first object in the second physical environment in the first manner is less than the threshold distance from an individual position within the three dimensional environment corresponding to the second viewpoint associated with the second view of the three dimensional environment.

37. 1. A method comprising: A computer system in communication with a first display generation component and one or more first input devices, comprising: Displaying a first computer-generated experience at a first immersion level; receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level; In response to receiving the biometric data corresponding to the first user, displaying the first computer-generated experience at a second immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level; continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user does not satisfy the first criterion; and A method comprising:

38. receiving first updated biometric data corresponding to the first user while displaying the first computer-generated experience at the second immersion level; In response to receiving the first updated biometric data corresponding to the first user, displaying the first computer-generated experience at a third immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies a second criterion different from the first criterion, the first computer-generated experience displayed at the third immersion level occupying a larger portion of the field of view of the first user than the first computer-generated experience displayed at the second immersion level; continuing to display the first computer-generated experience at the second immersion level in accordance with a determination that the first updated biometric data corresponding to the first user satisfies the first criterion and does not satisfy the second criterion; and 38. The method of claim 37, comprising:

39. receiving second updated biometric data corresponding to the first user while displaying the first computer-generated experience at a separate immersion level, the first computer-generated experience displayed at the separate immersion level occupying a larger portion of the field of view of the first user than at the first immersion level; In response to receiving the second updated biometric data corresponding to the first user, displaying the first computer-generated experience at the lower immersion level used prior to displaying the first computer-generated experience at the individual immersion level in accordance with a determination that the second updated biometric data corresponding to the first user does not satisfy the individual criteria satisfied for transitioning to displaying the first computer-generated experience at the individual immersion level; and 39. The method of claim 37 or 38, comprising:

40. 40. The method of any one of claims 37 to 39, wherein the biometric data includes a respiration rate of the first user, and the first criterion includes a criterion that is met when the respiration rate of the first user is below a first threshold respiration rate for the first criterion to be met.

41. 41. The method of any one of claims 37 to 40, wherein the first criterion includes a requirement that the biometric data meet one or more preset thresholds for at least a threshold time for the first criterion to be met.

42. Displaying the first computer-generated experience at the first immersion level includes displaying virtual content at respective first positions corresponding to locations of one or more first portions of the physical environment while maintaining display of a representation of one or more second portions of the physical environment; 42. The method of any one of claims 37 to 41, wherein displaying the first computer-generated experience at the second immersive level comprises displaying virtual content at the respective first positions corresponding to the locations of the one or more first portions of the physical environment and at respective second positions corresponding to at least some of the one or more second portions of the physical environment.

43. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion; 43. The method of any one of claims 37 to 42, comprising gradually reducing visual emphasis of at least a portion of a representation of a physical environment that was visible through the first display generation component while the first computer-generated experience is displayed at the first immersion level, and wherein displaying the first computer-generated experience at the second immersion level comprises displaying virtual content of the first computer-generated experience at a position corresponding to the portion of the representation of the physical environment such that the portion of the representation of the physical environment ceases to be visible through the first display generation component.

44. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion; 44. The method of any one of claims 37 to 43, comprising altering visual characteristics of at least a portion of a representation of a physical environment that was visible through the first display generation component by an amount corresponding to the change in the biometric data corresponding to the first user while the first computer-generated experience is displayed at the first immersion level.

45. in response to receiving the biometric data corresponding to the first user and in accordance with a determination that changes in the biometric data corresponding to the first user are progressing toward satisfying the first criterion; 45. The method of any one of claims 37 to 44, comprising augmenting the display of virtual content over at least a portion of a representation of the physical environment that was visible through the first display generation component while the first computer-generated experience is displayed at the first immersion level by an amount corresponding to the change in the biometric data corresponding to the first user.

46. 46. ​​The method of any one of claims 37 to 45, wherein the first criteria include that the first user performs less than a threshold amount of movement of a first type when the biometric data is being received for the first criteria to be satisfied.

47. Detecting a first type of movement being performed by the first user while displaying the first computer-generated experience at the second immersion level; In response to detecting the movement of the first type being performed by the first user, re-displaying the first computer-generated experience at the first immersion level in accordance with a determination that the movement of the first type exceeds a preset threshold movement amount; 47. The method of any one of claims 37 to 46, comprising:

48. Detecting a first type of movement being performed by the first user while displaying the first computer-generated experience at the second immersion level; In response to detecting the movement of the first type being performed by the first user, 47. The method of any one of claims 37 to 46, comprising: switching from displaying the first computer-generated experience at the second immersion level from a first perspective to displaying the first computer-generated experience at the second immersion level from a second perspective different from the first perspective in accordance with a determination that the movement of the first type exceeds a preset threshold movement amount.

49. 49. The method of any one of claims 37 to 48, wherein the transition from displaying the first computer-generated experience at the first immersion level to displaying the first computer-generated experience at the second immersion level is a discontinuous transition that occurs at a time corresponding to a time when the first criterion is met.

50. 49. The method of any one of claims 37 to 48, wherein the first computer-generated experience displayed at the first level of immersion depicts a first virtual environment and the first computer-generated experience displayed at the second level of immersion depicts a second virtual environment having a greater virtual depth than the first virtual environment.

51. Displaying the first computer-generated experience at the first immersion level includes displaying the first computer-generated experience with at least a first visual characteristic that changes according to changes in the biometric data received while displaying the first computer-generated experience at the first immersion level; 51. The method of any one of claims 37 to 50, wherein displaying the first computer-generated experience at the second immersion level comprises displaying the first computer-generated experience with at least a second visual characteristic that changes in accordance with changes in the biometric data received while displaying the first computer-generated experience at the second immersion level.

52. In response to receiving the biometric data corresponding to the first user, 52. The method of any one of claims 37 to 51, comprising: changing an audio output mode from a first audio output mode to a second audio output mode in accordance with a determination that the biometric data corresponding to the first user satisfies the first criterion, the first audio output mode having fewer computational control variables than the second audio output mode.

53. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: Displaying a first computer-generated experience at a first immersion level; receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level; In response to receiving the biometric data corresponding to the first user, pursuant to a determination that the biometric data corresponding to the first user satisfies a first criterion, displaying the first computer-generated experience at a second immersion level, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level; 11. A computer system comprising: instructions for continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user does not satisfy the first criterion.

54. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to: Displaying a first computer-generated experience at a first immersion level; receiving biometric data corresponding to a first user while displaying the first computer-generated experience at the first immersion level; In response to receiving the biometric data corresponding to the first user, displaying the first computer-generated experience at a second immersion level in accordance with a determination that the biometric data corresponding to the first user satisfies a first criterion, the first computer-generated experience displayed at the second immersion level occupying a larger portion of the first user's field of view than the first computer-generated experience displayed at the first immersion level; and continuing to display the first computer-generated experience at the first immersion level in accordance with a determination that the biometric data corresponding to the first user does not satisfy the first criterion.

55. 1. A method comprising: A computer system in communication with a first display generation component and one or more first input devices, comprising: displaying a first view of a physical environment, the first view of the physical environment including a first representation of a first portion of the physical environment; detecting a first user input corresponding to a request to activate a first type of computer-generated sensory adjustment of two or more types of computer-generated sensory adjustment while displaying the first view of the physical environment; in response to detecting the first user input, displaying a second view of the physical environment, the second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted with respect to the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment; detecting, while displaying the second view of the physical environment, a second user input corresponding to a request to activate a second type of computer-generated sensory adjustment among the two or more types of computer-generated sensory adjustment, the second type of computer-generated sensory adjustment being different from the first type of computer-generated sensory adjustment; in response to detecting the second user input, displaying a third view of the physical environment, the third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment and second display characteristics adjusted for the second representation of the physical environment in accordance with the second type of computer-generated sensory adjustment; A method comprising:

56. detecting, while displaying the third view of the physical environment, a third user input corresponding to a request to activate a third type of computer-generated sensory adjustment among the two or more types of computer-generated sensory adjustment, the third type of computer-generated sensory adjustment being different from the first type of computer-generated sensory adjustment and the second type of computer-generated sensory adjustment; In response to detecting the third user input, displaying a fourth view of the physical environment, the fourth view of the physical environment including a fourth representation of the first portion of the physical environment, the fourth representation of the first portion of the physical environment having the first display characteristics adjusted for the first representation of the first portion of the physical environment according to the first type of computer-generated sensory adjustment, the second display characteristics adjusted for the second representation of the physical environment according to the second type of computer-generated sensory adjustment, and third display characteristics adjusted for the third representation of the physical environment according to the third type of computer-generated sensory adjustment; 56. The method of claim 55, comprising:

57. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated telescopic vision for viewing distant physical objects, and the second type of computer-generated sensory modulation comprises simulated microscopic vision for magnifying nearby physical objects.

58. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated telescopic vision for viewing remote physical objects, and the second type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions.

59. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated telescopic vision for viewing remote physical objects, and the second type of computer-generated sensory modulation comprises simulated thermal vision for viewing physical objects having different thermal radiation profiles.

60. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated telescopic viewing to view a remote physical object, and the second type of computer-generated sensory modulation comprises modifying a view of the physical object using a filter.

61. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated telescopic viewing to view remote physical objects, and the second type of computer-generated sensory modulation comprises selective audio modulation for sounds corresponding to a subset of physical objects in a physical environment.

62. 62. The method of claim 61 , comprising simultaneously displaying the third representation of the physical environment, outputting sound that corresponds to a first portion of the physical environment that is visible in the second representation of the physical environment and that is selectively enhanced relative to sound from a sound source outside the first portion of the physical environment.

63. 63. The method of any one of claims 55 to 62, comprising simultaneously displaying the third representation of the physical environment, displaying text output corresponding to speech emanating from a first portion of the physical environment that is visible in both the second and third representations of the physical environment, the text output being selectively enhanced relative to sounds from sources outside the first portion of the physical environment.

64. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated microscopic viewing to magnify nearby physical objects, and the second type of computer-generated sensory modulation comprises simulated thermal viewing to view physical objects having different thermal radiation profiles.

65. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions, and the second type of computer-generated sensory adjustment comprises simulated telescopic vision for viewing remote physical objects.

66. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions, and the second type of computer-generated sensory adjustment comprises simulated microscopy for magnifying nearby physical objects.

67. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions, and the second type of computer-generated sensory adjustment comprises simulated heat vision for viewing physical objects having different thermal radiation profiles.

68. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions, and the second type of computer-generated sensory adjustment comprises selective audio adjustment for sounds corresponding to a subset of physical objects in a physical environment.

69. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated thermal vision for viewing physical objects having different thermal emission profiles, and the second type of computer-generated sensory modulation comprises simulated telescopic vision for viewing remote physical objects.

70. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation comprises simulated thermal vision for viewing physical objects having different thermal emission profiles, and the second type of computer-generated sensory modulation comprises simulated microscopic vision for magnifying nearby physical objects.

71. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory adjustment comprises simulated heat vision for viewing physical objects having different thermal radiation profiles, and the second type of computer-generated sensory adjustment comprises simulated night vision for viewing physical objects under low light conditions.

72. 57. The method of claim 55 or 56, wherein the first type of computer-generated sensory modulation includes simulated thermal vision for viewing physical objects having different thermal emission profiles, and the second type of computer-generated sensory modulation includes selective audio modulation for sounds corresponding to a subset of physical objects in a physical environment.

73. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a first view of a physical environment, the first view of the physical environment including a first representation of a first portion of the physical environment; detecting a first user input corresponding to a request to activate a first type of computer-generated sensory adjustment among two or more types of computer-generated sensory adjustment while displaying the first view of the physical environment; in response to detecting the first user input, displaying a second view of the physical environment, the second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted with respect to the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment; detecting a second user input corresponding to a request to activate a second type of computer-generated sensory adjustment among the two or more types of computer-generated sensory adjustment, the second type of computer-generated sensory adjustment being different from the first type of computer-generated sensory adjustment while displaying the second view of the physical environment; a computer system including instructions for, in response to detecting the second user input, displaying a third view of the physical environment, the third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment and second display characteristics adjusted for the second representation of the physical environment in accordance with the second type of computer-generated sensory adjustment.

74. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to: displaying a first view of a physical environment, the first view of the physical environment including a first representation of a first portion of the physical environment; detecting a first user input corresponding to a request to activate a first type of computer-generated sensory adjustment of two or more types of computer-generated sensory adjustment while displaying the first view of the physical environment; in response to detecting the first user input, displaying a second view of the physical environment, the second view of the physical environment including a second representation of the first portion of the physical environment, the second representation of the first portion of the physical environment having first display characteristics adjusted with respect to the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment; detecting, while displaying the second view of the physical environment, a second user input corresponding to a request to activate a second type of computer-generated sensory adjustment among the two or more types of computer-generated sensory adjustment, the second type of computer-generated sensory adjustment being different from the first type of computer-generated sensory adjustment; a computer-readable storage medium configured to cause a computer to perform operations including: in response to detecting the second user input, displaying a third view of the physical environment, the third view of the physical environment including a third representation of the first portion of the physical environment, the third representation of the first portion of the physical environment having first display characteristics adjusted for the first representation of the first portion of the physical environment in accordance with the first type of computer-generated sensory adjustment and second display characteristics adjusted for the second representation of the physical environment in accordance with the second type of computer-generated sensory adjustment.

75. 1. A method comprising: A computer system in communication with a first display generation component and one or more first input devices, comprising: displaying a first view of a three-dimensional environment, the first view of the three-dimensional environment including a first representation of a first portion of a physical environment; detecting movement of a first user from a first location to a second location in the physical environment while displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment; In response to detecting the movement of the first user from the first location to the second location, displaying a second view of the three-dimensional environment in accordance with a determination that the movement to the second location satisfies first criteria, the first criteria including a first requirement that the second location correspond to a location associated with a first type of exercise for the first criteria to be satisfied, the second view of the three-dimensional environment including a first set of virtual content corresponding to the first type of exercise, the first set of virtual content replacing at least a portion of a second representation of a second portion of the physical environment including the second location; displaying a third view of the three-dimensional environment in accordance with a determination that the movement to the second location satisfies second criteria different from the first criteria, the second criteria including a second requirement that the second location correspond to a location associated with a second type of exercise for the second criterion to be satisfied, the second type of exercise being different from the first type of exercise, the third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, the second set of virtual content being different from the first set of virtual content, the second set of virtual content replacing at least a portion of a third representation of a third portion of the physical environment including the second location; A method comprising:

76. the computer system determines, upon detecting a first type of exercise equipment at the second location, that the second location corresponds to a location associated with the first type of exercise; 76. The method of claim 75, wherein the computer system, upon detecting a second type of exercise equipment at the second location, determines that the second location corresponds to a location associated with the second type of exercise, and the second type of exercise equipment is different from the first type of exercise equipment and does not correspond to the first type of exercise.

77. displaying the second view of the three-dimensional environment in response to detecting the movement of the first user from the first location to the second location within the physical environment; gradually reducing the second representation of the second portion of the physical environment; and gradually increasing the prominence of virtual content corresponding to the first type of exercise in an area of ​​the second view of the three-dimensional environment in which the second representation of the second portion of the physical environment is gradually reduced; and 77. The method of claim 75 or 76, comprising:

78. 78. The method of any one of claims 75 to 77, wherein the first criterion includes a third requirement that, for the first criterion to be met, the movement of the first user from the first location to the second location is followed by a first predetermined movement corresponding to the first type of exercise.

79. In response to detecting the movement of the first user from the first location to the second location, 79. The method of claim 78, comprising displaying the fourth view of the three-dimensional environment in accordance with a determination that the movement to the second location satisfies third criteria, different from the first criteria and the second criteria, the third criteria including a requirement that the second location correspond to a location associated with a third type of exercise different from the first type of exercise and the second type of exercise, the movement of the first user from the first location to the second location being followed by a third predetermined movement corresponding to the third type of exercise such that the third criterion is satisfied, the third predetermined movement being different from the first predetermined movement, the fourth view of the three-dimensional environment including a third set of virtual content corresponding to the third type of exercise, the third set of virtual content being different from the first set of virtual content and the second set of virtual content, and the third set of virtual content replacing at least a portion of the second representation of the second portion of the physical environment.

80. 80. The method of claim 78 or 79, comprising gradually increasing the amount of virtual content displayed within the field of view of the first user according to at least one of a progression or duration of a predetermined movement corresponding to a particular type of exercise associated with the second location.

81. detecting the movement of the first user corresponding to a request to terminate the individual type of exercise associated with the second location while displaying an individual view of the three-dimensional environment corresponding to the individual type of exercise associated with the second location; in response to detecting movement of the first user corresponding to a request to complete the individual type of exercise associated with the second location; and displaying a fifth view of the three-dimensional environment including a representation of at least a fourth portion of the physical environment, the representation of at least the fourth portion of the physical environment occupying a portion of the first user's field of view in which a respective set of virtual content corresponding to the respective type of exercise was displayed while the first user was at the second location.

82. 82. The method of any one of claims 75 to 81, comprising displaying status information corresponding to the first type of exercise when the second view of the three-dimensional environment is displayed.

83. 83. The method of any one of claims 75 to 82, comprising displaying health information corresponding to the first user when the second view of the three-dimensional environment is displayed.

84. and visually presenting progress information of the first type of exercise performed by the first user when the second view of the three-dimensional environment is displayed.

85. Displaying the second view of the three-dimensional environment includes: displaying a first subset of the first set of virtual content corresponding to the first type of exercise without displaying a second subset of the first set of virtual content in accordance with determining that the first user is facing a first direction within the physical environment; displaying the second subset of the first set of virtual content corresponding to the first type of exercise without displaying the second subset of the first set of virtual content in accordance with a determination that the first user is facing a second direction in the physical environment that is different from the first direction; 85. The method of any one of claims 75 to 84, comprising:

86. 86. The method of any one of claims 75 to 85, wherein the first type of exercise is a rowing exercise, the second location is a location where rowing exercise equipment is present, and the second view of the three-dimensional environment includes a virtual scene having an open body of water.

87. In response to detecting the movement of the first user from the first location to the second location, In response to determining that the second location corresponds to a location associated with a fifth type of exercise and a sixth type of exercise, displaying a sixth view of the three-dimensional environment in accordance with a determination that the movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the fifth type of exercise, the sixth view of the three-dimensional environment including a fifth set of virtual content corresponding to the fifth type of exercise, the fifth set of virtual content being different from the first set of virtual content and the second set of virtual content, and the fifth set of virtual content replacing at least a portion of a fifth representation of a fifth portion of the physical environment; displaying a seventh view of the three-dimensional environment in accordance with a determination that the movement of the first user from the first location to the second location is followed by engagement by the first user at the second location with a respective type of equipment associated with the sixth type of exercise, the seventh view of the three-dimensional environment including a sixth set of virtual content corresponding to the sixth type of exercise, the sixth set of virtual content being different from the first set of virtual content, the second set of virtual content, and the fifth set of virtual content, and the sixth set of virtual content replacing at least a portion of the fifth representation of the fifth portion of the physical environment; 87. The method of any one of claims 75 to 86, comprising:

88. 88. The method of any one of claims 75 to 87, wherein the second view of the three-dimensional environment includes a virtual representation of the first user shown competing with the first user to perform the first type of exercise.

89. 89. The method of any one of claims 75 to 88, wherein the second view of the three-dimensional environment includes a virtual representation of at least a second user, different from the first user, who is shown competing with the first user to perform the first type of exercise.

90. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a first view of a three-dimensional environment, the first view of the three-dimensional environment including a first representation of a first portion of a physical environment; detecting movement of a first user from a first location to a second location in the physical environment while displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment; In response to detecting the movement of the first user from the first location to the second location, displaying a second view of the three-dimensional environment, the second view of the three-dimensional environment including a first set of virtual content corresponding to the first type of exercise, the first set of virtual content replacing at least a portion of a second representation of a second portion of the physical environment including the second location, in accordance with a determination that the movement to the second location satisfies first criteria, the first criteria including a first requirement that the second location correspond to a location associated with a first type of exercise for the first criteria to be satisfied; and displaying, in accordance with a determination that the movement to the second location satisfies second criteria that are different from the first criteria, the second criteria including a second requirement that the second location correspond to a location associated with a second type of exercise for the second criterion to be met, the second type of exercise being different from the first type of exercise, a third view of the three-dimensional environment, the third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, the second set of virtual content being different from the first set of virtual content, the second set of virtual content replacing at least a portion of a third representation of a third portion of the physical environment including the second location.

91. 1. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to: displaying a first view of a three-dimensional environment, the first view of the three-dimensional environment including a first representation of a first portion of a physical environment; detecting movement of a first user from a first location to a second location in the physical environment while displaying the first view of the three-dimensional environment including the first representation of the first portion of the physical environment; In response to detecting the movement of the first user from the first location to the second location, displaying a second view of the three-dimensional environment in accordance with a determination that the movement to the second location satisfies first criteria, the first criteria including a first requirement that the second location correspond to a location associated with a first type of exercise for the first criteria to be satisfied, the second view of the three-dimensional environment including a first set of virtual content corresponding to the first type of exercise, the first set of virtual content replacing at least a portion of a second representation of a second portion of the physical environment including the second location; and displaying, in accordance with a determination that the movement to the second location satisfies second criteria that differ from the first criteria, the second criteria including a second requirement that the second location correspond to a location associated with a second type of exercise for the second criterion to be satisfied, the second type of exercise being different from the first type of exercise, a third view of the three-dimensional environment, the third view of the three-dimensional environment including a second set of virtual content corresponding to the second type of exercise, the second set of virtual content being different from the first set of virtual content, the second set of virtual content replacing at least a portion of a third representation of a third portion of the physical environment including the second location.

92. 1. A computer system comprising: a display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 18.

93. 19. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a display generation component and one or more input devices, cause the computer system to perform the method of any one of claims 1 to 18.

94. 19. A graphical user interface on a computer system including a display generation component, one or more input devices, a memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 1 to 18.

95. 1. A computer system comprising: a first display generation component; one or more input devices; and means for executing the method according to any one of claims 1 to 18.

96. 1. An information processing device for use in a computer system including a first display generation component and one or more input devices, An information processing device comprising means for carrying out the method according to any one of claims 1 to 18.

97. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 21 to 34.

98. 35. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to perform the method of any one of claims 21 to 34.

99. 35. A graphical user interface on a computer system including a first display generation component, one or more input devices, memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 21 to 34.

100. 1. A computer system comprising: a first display generation component; one or more input devices; means for carrying out the method of any one of claims 21 to 34; A computer system comprising:

101. 1. An information processing device for use in a computer system including a first display generation component and one or more input devices, 35. An information processing device comprising means for carrying out the method of any one of claims 21 to 34.

102. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; and a memory having stored thereon one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 37 to 52.

103. 53. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to perform the method of any one of claims 37 to 52.

104. 53. A graphical user interface on a computer system including a first display generation component, one or more input devices, memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 37 to 52.

105. 1. A computer system comprising: a first display generation component; one or more input devices; means for carrying out the method of any one of claims 37 to 52; A computer system comprising:

106. 1. An information processing device for use in a computer system including a first display generation component and one or more input devices, 53. An information processing device comprising means for carrying out the method of any one of claims 37 to 52.

107. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs containing instructions for performing the method of any one of claims 55 to 72.

108. 73. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to perform the method of any one of claims 55 to 72.

109. 73. A graphical user interface on a computer system including a first display generation component, one or more input devices, memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 55 to 72.

110. 1. A computer system comprising: a first display generation component; one or more input devices; means for carrying out the method of any one of claims 55 to 72; A computer system comprising:

111. 1. An information processing device for use in a computer system including a first display generation component and one or more input devices, 73. An information processing device comprising means for carrying out the method of any one of claims 55 to 72.

112. 1. A computer system comprising: a first display generation component; one or more input devices; one or more processors; and a memory having stored thereon one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs containing instructions for performing the method of any one of claims 75 to 89.

113. 90. A computer-readable storage medium having stored thereon one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generation component and one or more input devices, cause the computer system to perform the method of any one of claims 75 to 89.

114. 90. A graphical user interface on a computer system including a first display generation component, one or more input devices, memory, and one or more processors executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 75 to 89.

115. 1. A computer system comprising: a first display generation component; one or more input devices; and means for performing the method of any one of claims 75 to 89.

116. 1. An information processing device for use in a computer system including a first display generation component and one or more input devices, 90. An information processing device comprising means for carrying out a method according to any one of claims 75 to 89.

Images