Device for, method of, and graphical user interface for mutual interaction with three-dimensional environment

Abstract

To provide a device for, a method of, and a graphical user interface for providing a user with a computer generation experience which makes mutual interaction with a computer system more efficiently and intuitively for a user.SOLUTION: A computer system presents a three-dimensional environment including a virtual content and a first part of a physical environment, and displays a first user interface object on a first physical surface of the first part of the physical environment. The first part of the physical environment includes a first location and a second location, and the first user interface object has a first part and a second part. If a part of a user is detected in the first location, the system erases the first part and displays expression of the part of the user. If the movement of the user part from the first location to the second location is detected, the expression of the first part is restored. and the second part is erased to display the expression of the part of the user.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] (Related Applications) This application is a continuation of U.S. Patent Application No. 17 / 483,722, filed September 23, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 082,933, filed September 24, 2020, each of which is incorporated by reference in its entirety.

[0002] (Technical field) The present disclosure generally relates to computer systems having display generating components, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via displays, and one or more input devices that provide computer-generated reality (CGR) experiences. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include digital images, video, text, icons, and virtual objects such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulating virtual objects is complex and error-prone create a significant cognitive burden for users and detract from the experience of the virtual / augmented reality environment. In addition, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Therefore, there is a need for a computer system with improved methods and interfaces for providing users with computer-generated experiences that make interacting with the computer system more efficient and intuitive for users. The above-mentioned deficiencies and other problems associated with user interfaces for computer systems having display generating components and one or more input devices are reduced or eliminated by the disclosed systems, methods, and user interfaces. Such systems, methods, and interfaces optionally complement or replace conventional systems, methods, and user interfaces for providing users with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from a user by helping the user understand the connection between the input provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] According to some embodiments, a method is executed on a computer system in communication with a first display generation component, one or more audio output devices, and one or more input devices, and includes: displaying a three-dimensional computer-generated environment via the first display generation component; detecting, while displaying the three-dimensional computer-generated environment, a first event corresponding to a request to present first computer-generated content, the first computer-generated content including first visual content and first audio content corresponding to the first visual content; and, in response to detecting the first event corresponding to the request to present the first computer-generated content, the first event presents first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment. displaying first visual content within a first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in accordance with a determination that the first event corresponds to an individual request to present first computer-generated content having a second immersion level different from the first immersion level, the first computer-generated content presented at the second immersion level occupying a second portion of the three-dimensional environment that is larger than the first portion of the three-dimensional environment, displaying the first visual content within a second portion of the three-dimensional environment and outputting the first audio content using a second audio output mode different from the first audio output mode in accordance with a determination that the first event corresponds to an individual request to present first computer-generated content having a second immersion level different from the first immersion level, the first computer-generated content presented at the second immersion level occupying a second portion of the three-dimensional environment that is larger than the first portion of the three-dimensional environment,

[0007] According to some embodiments, a method is executed on a computer system in communication with a display generation component, the method comprising: displaying, via the display generation component, a view of a computer-generated environment; detecting, while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present in the physical environment in which a user is located, a first movement of a first physical object within the physical environment; and, in response to detecting the first movement of the first physical object within the physical environment, determining that the user is within a threshold distance of the first portion of the first physical object and that the first physical object satisfies predetermined criteria including requirements regarding predetermined characteristics of the first physical object other than the distance of the first physical object from the user, a second portion of the first physical object. the first portion of the first physical object and the second portion of the physical object are both part of a range of the first physical object potentially visible to the user based on the user's field of view of the computer-generated environment, modifying the appearance of the portion of the virtual content displayed at a position corresponding to the current location of the first portion of the first physical object without modifying the appearance of the portion of the virtual content displayed at a position corresponding to the second portion; and ceasing to modify the appearance of the portion of the virtual content displayed at the position corresponding to the current location of the first portion of the first physical object in accordance with a determination that the user is within a threshold distance of the first physical object present in the physical environment surrounding the user and the first physical object does not satisfy a predetermined criterion.

[0008] According to some embodiments, a method is executed on a computer system in communication with a first display generating component and one or more input devices, and includes: displaying, via the first display generating component, a three-dimensional environment including a representation of a physical environment; detecting a user's hand touching a distinct portion of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; in response to detecting the user's hand touching the distinct portion of the physical environment, displaying a first visual effect at a location within the three-dimensional environment corresponding to a first portion of the physical environment identified based on scanning of the first portion of the physical environment in accordance with a determination that the user's hand is touching the first portion of the physical environment; and displaying a second visual effect at a location within the three-dimensional environment corresponding to a second portion of the physical environment identified based on scanning of the second portion of the physical environment in accordance with a determination that the user's hand is touching a second portion of the physical environment that is different from the first portion of the physical environment.

[0009] According to some embodiments, a method is executed on a computer system in communication with a first display generating component and one or more input devices, and includes: displaying, via the first display generating component, a view of a three-dimensional environment simultaneously including a representation of a first portion of a physical environment including a first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position in the three-dimensional environment corresponding to a location of the first physical surface within the first portion of the physical environment; detecting, while displaying the view of the three-dimensional environment, a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment; and, in response to detecting the portion of the user at the first location within the first portion of the physical environment, displaying a representation of the portion of the user at the first location within the first portion of the physical environment. ceasing the display of a first portion of the first user interface object while maintaining the display of a second portion of the first user interface object so that the first portion of the user interface object appears in a position where it previously displayed; detecting, while displaying the view of the three-dimensional environment, movement of a portion of the user from a first location within the first portion of the physical environment to a second location, the second location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment; and in response to detecting movement of the portion of the user from the first location to the second location, restoring the display of the first portion of the first user interface object and ceasing the display of the second portion of the first user interface object so that a representation of the portion of the user appears in a position where it previously displayed the second portion of the first user interface object.

[0010] According to some embodiments, a computer system includes or is in communication with a display generating component (e.g., a display, projector, head-mounted display, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, one or more processors, and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface of a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein. According to some embodiments, a computer system includes a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein.According to some embodiments, an information processing apparatus for use in a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, includes means for performing or causing to be performed the operations of any of the methods described herein.

[0011] Thus, computer systems having display generating components are provided with improved methods and interfaces for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments, thereby increasing the effectiveness, efficiency, and user safety and satisfaction of such computer systems. Such methods and interfaces may complement or replace conventional methods for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments.

[0012] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0013] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0014] [Figure 1] FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0015] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0016] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a user with visual components of a CGR experience, according to some embodiments.

[0017] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0018] [Figure 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0019] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0020] [Figure 7A] FIG. 10 is a block diagram illustrating selecting different audio output modes according to the immersion level at which computer-generated content is presented, according to some embodiments. [Figure 7B] FIG. 10 is a block diagram illustrating selecting different audio output modes according to the immersion level at which computer-generated content is presented, according to some embodiments.

[0021] [Figure 7C]FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments. [Figure 7D] FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments. [Figure 7E] FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments. [Figure 7F] FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments. [Figure 7G] FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments. [Figure 7H]FIG. 10 is a block diagram illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of a physical object, etc.) in accordance with some embodiments.

[0022] [Figure 7I] FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments. [Figure 7J] FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments. [Figure 7K] FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments. [Figure 7L] FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments. [Figure 7M] FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments. [Figure 7N]FIG. 10 is a block diagram illustrating applying visual effects to areas in a three-dimensional environment that correspond to portions of the physical environment identified (e.g., characterized by shapes, planes, and / or surfaces) based on a scan of the portion of the physical environment, in accordance with some embodiments.

[0023] [Figure 7O] FIG. 10 is a block diagram illustrating, in accordance with some embodiments, displaying an interactive user interface object at a position in a three-dimensional environment corresponding to a first portion of the physical environment (e.g., a location on a physical surface or a location in free space within the physical environment), and selectively withdrawing display of individual sub-portions of the user interface object according to the location of a part of a user (e.g., a user's finger, hand, etc.) moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment. [Figure 7P] FIG. 10 is a block diagram illustrating, in accordance with some embodiments, displaying an interactive user interface object at a position in a three-dimensional environment corresponding to a first portion of the physical environment (e.g., a location on a physical surface or a location in free space within the physical environment), and selectively withdrawing display of individual sub-portions of the user interface object according to the location of a part of a user (e.g., a user's finger, hand, etc.) moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment. [Figure 7Q] FIG. 10 is a block diagram illustrating, in accordance with some embodiments, displaying an interactive user interface object at a position in a three-dimensional environment corresponding to a first portion of the physical environment (e.g., a location on a physical surface or a location in free space within the physical environment), and selectively withdrawing display of individual sub-portions of the user interface object according to the location of a part of a user (e.g., a user's finger, hand, etc.) moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment.

[0024] [Figure 8] 1 is a flowchart of a method for selecting different audio output modes depending on the immersion level at which computer-generated content is presented, according to some embodiments.

[0025] [Figure 9] 1 is a flowchart of a method for changing the appearance of a portion of virtual content when a significant physical object approaches the location of a display generating component or a user, according to some embodiments.

[0026] [Figure 10] 1 is a flowchart of a method for applying visual effects to an area in a three-dimensional environment corresponding to a portion of the physical environment identified based on a scan of the portion of the physical environment, according to some embodiments.

[0027] [Figure 11] 1 is a flowchart of a method, according to some embodiments, for displaying an interactive user interface object at a position in a three-dimensional environment corresponding to a first portion of the physical environment, and selectively ceasing to display individual sub-portions of the user interface object according to the location of a portion of a user moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment. DETAILED DESCRIPTION OF THE INVENTION

[0028] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0029] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0030] In some embodiments, a computer system displays computer-generated content, such as a movie, a virtual office, an application environment, a game, or a computer-generated experience (e.g., a virtual reality experience, an augmented reality experience, or a mixed reality experience). In some embodiments, the computer-generated content is displayed in a three-dimensional environment. In some embodiments, the computer system can display the visual components of the computer-generated content with multiple levels of immersion corresponding to different degrees of emphasis of visual input from the virtual content relative to visual input from the physical environment. In some embodiments, higher levels of immersion correspond to a greater emphasis of visual input from the virtual content over visual input from the physical environment. Similarly, in some embodiments, audio components of the computer-generated content that accompany and / or correspond to the visual components of the computer-generated content (e.g., sound effects and soundtracks in movies; audio alerts, audio feedback, and system sounds in application environments; sound effects, speech, and audio feedback in games; and / or sound effects and audio feedback in computer-generated experiences) can be output with multiple levels of immersion. In some embodiments, the multiple immersion levels optionally correspond to varying degrees of spatial correspondence between the position of a virtual sound source within the virtual content displayed via the display generating components and the perceived location of the virtual sound source within a selected frame of reference for the virtual sound source. In some embodiments, the selected frame of reference for a particular virtual sound source is based on the physical environment, based on a virtual three-dimensional environment of the computer-generated content, based on a viewpoint of a currently displayed view of the three-dimensional environment of the computer-generated content, based on the location of the display generating components in the physical environment, or based on the location of a user in the physical environment.In some embodiments, a higher level of immersion corresponds to a higher level of correspondence between the position of a virtual sound source in the computer-generated environment and the perceived location of the virtual sound source in a selected frame of reference of the audio component of the computer-generated content (e.g., a frame of reference based on the three-dimensional environment depicted in the computer-generated experience, a frame of reference based on the location of the viewpoint, a frame of reference based on the location of a display generating component, a frame of reference based on the location of the user, etc.). In some embodiments, a lower level of correspondence between the position of a virtual sound source in the computer-generated environment and the perceived location of the sound source in the selected frame of reference of the audio component of the computer-generated content is the result of a higher level of correspondence between the perceived location of the virtual sound source and the location of an audio output device in the physical environment (e.g., sound appears to come from the location of the audio output device regardless of the position of the virtual sound source in the three-dimensional environment of the computer-generated content and / or regardless of the location of the viewpoint, the location of a display generating component, and / or the location of the user, etc.). In some embodiments, a computer system detects a first event corresponding to a request to present a first computer-generated experience, and the computer system selects an audio output mode for outputting an audio component of the computer-generated experience according to an immersion level at which a visual component of the computer-generated experience is displayed via a display generation component. At a higher immersion level associated with displaying the visual content of the first computer-generated experience, the computer system selects an audio output mode for presenting audio content of the computer-generated experience having a corresponding higher immersion level. In some embodiments, displaying the visual content with a higher immersion level includes displaying the visual content with a greater spatial extent in a three-dimensional environment, and outputting the audio content with the corresponding higher immersion level includes outputting the audio content in a spatial audio output mode.In some embodiments, when switching between displaying visual content having two different immersion levels (e.g., from a higher level of immersion to a lower level of immersion, from a lower level of immersion to a higher level of immersion, etc.), the computer system also switches between outputting audio content having two different immersion levels (e.g., from a spatial audio output mode to a stereo audio output mode, from a surround sound output mode to a stereo audio output mode, from a stereo audio output mode to a surround sound output mode, from a stereo audio output mode to a spatial audio output mode, etc.). Selecting an appropriate audio output mode for outputting the audio component of the computer-generated content according to the immersion level at which the visual content of the computer-generated content is displayed allows the computer system to provide a computer-generated experience that is more consistent with a user's expectations and avoids causing confusion when the user interacts with the computer-generated environment while engaging in the computer-generated experience. This can reduce user error and make user interaction with the computer system more efficient.

[0031] In some embodiments, when displaying virtual content in a three-dimensional environment (e.g., a virtual reality environment, an augmented reality environment, etc.), all or a portion of the view of the physical environment is blocked or replaced by the virtual content. In some cases, it is advantageous to give display priority to certain physical objects in the physical environment over the virtual content so that at least a portion of the physical object is visually represented in the view of the three-dimensional environment. In some embodiments, the computer system utilizes various criteria for determining whether to give display priority to an individual physical object such that a representation of the individual physical object can break through a portion of the virtual content currently displayed in the three-dimensional environment when the location of the individual physical object in the physical environment corresponds to the position of the portion of the virtual content in the three-dimensional environment. In some embodiments, the criteria include a requirement that at least a portion of the physical object has approached and entered a threshold spatial region surrounding a user of the display generating component (e.g., a user viewing virtual content via the display generating component, a user whose view of a portion of the physical object is blocked or replaced by the display of virtual content, etc.), and an additional requirement that the computer system detects the presence of one or more characteristics related to the physical object that indicate increased importance of the physical object to the user. In some embodiments, the physical objects of increased importance to the user may be the user's friends or family, the user's team members or supervisors, the user's pets, etc. In some embodiments, the physical objects of increased importance to the user may be people or objects that require the user's attention to address an emergency. In some embodiments, the physical objects of greater importance to the user may be people or objects that require the user's attention to take action to ensure the user does not miss an opportunity. The criteria can be adjusted by the user based on the user's needs and desires and / or by the system based on contextual information (e.g., time, location, scheduled events, etc.).In some embodiments, giving display priority to physical objects that are more important than virtual content and visually representing at least a portion of the physical object in the view of the three-dimensional environment includes replacing the display of the portion of the virtual content with a representation of the portion of the physical object or modifying the appearance of the portion of the virtual content according to the appearance of the portion of the physical object. In some embodiments, at least a portion of the physical object is not visually represented in the view of the three-dimensional environment and remains blocked or replaced by the display of the virtual content, even if a position corresponding to the location of the portion of the physical object is visible within the field of view provided by the display generation component (e.g., the position is currently occupied by virtual content). In some embodiments, the portions of the three-dimensional environment that are modified to indicate the presence of the physical object and the portions of the three-dimensional environment that are not modified to indicate the presence of the physical object (e.g., the portions of the three-dimensional environment can continue to change based on the progression of the computer-generated experience and / or user interaction with the three-dimensional environment, etc.) correspond to positions on successive portions of a virtual object or surface. While allowing at least a portion of a significant physical object to break the display of the virtual content and be visually represented in a position corresponding to the location of the portion of the physical object, at least a portion of the physical object remains visually obscured by the virtual content, and pursuant to a determination that the physical object meets predetermined criteria for identifying physical objects of increased importance to the user and has entered a predetermined spatial region surrounding the user, the user is provided with an opportunity to perceive and interact with the physical object without completely ceasing the computer-generated experience involving the user and without indiscriminately allowing physical objects of little importance to the user (e.g., a rolling ball, a passerby, etc.) to interrupt the computer-generated experience.This improves the user's experience and reduces the number, extent, and / or nature of inputs from the user to achieve a desired result (e.g., manually aborting a computer-generated experience when physically obstructed or touching a physical object, manually restarting a computer-generated experience after being unnecessarily interrupted, etc.), thereby creating a more efficient human-machine interface.

[0032] In some embodiments, the computer system displays a representation of the physical environment in response to a request to display a three-dimensional environment including a representation of the physical environment (e.g., in response to a user putting on a head-mounted display, in response to a user request to start an augmented reality environment, in response to a user request to end a virtual reality experience, in response to a user turning on or waking up a display generation component from a low-power state, etc.). In some embodiments, the computer system initiates a scan of the physical environment to identify objects and surfaces within the physical environment and, optionally, construct a three-dimensional or pseudo-three-dimensional model of the physical environment based on the identified objects and surfaces within the physical environment. In some embodiments, the computer system initiates a scan of the physical environment in response to receiving a request to display the three-dimensional environment (e.g., if the physical environment has not been previously scanned and characterized by the computer system, or if a rescan is requested by the user or the system based on preset rescan criteria being met (e.g., the last scan was performed more than a threshold time ago, the physical environment has changed, etc.)). In some embodiments, the computer system initiates a scan in response to detecting a user's hand touching a portion of the physical environment (e.g., a physical surface, a physical object, etc.). In some embodiments, the computer system initiates the scan in response to detecting that the user's line of sight, directed toward a position corresponding to the portion of the physical environment, meets preset stability and / or duration criteria. In some embodiments, the computer system displays visual feedback regarding the progress and results of the scan (e.g., identifying physical objects and surfaces in the physical environment, determining physical and spatial properties of the physical objects and surfaces, etc.). In some embodiments, the visual feedback includes displaying distinct visual effects in distinct portions of the three-dimensional environment that correspond to the portions of the physical environment touched by the user's hand and identified based on the scan of the portion of the physical environment. In some embodiments, the visual effects include representations of movement extending from and / or propagating from the distinct portions of the three-dimensional environment.In some embodiments, the computer system displays visual effects in response to detecting a user's hand touching a distinct portion of the physical environment, and the three-dimensional environment is displayed after a scan of the physical environment is completed in response to a previous request to display the three-dimensional environment. In some embodiments, displaying visual effects indicating the progress and results of the scan of the physical environment at positions corresponding to the location of the user's touch on a portion of the physical environment helps the user visualize the spatial environment that the computer uses to display and anchor virtual objects and surfaces, facilitating subsequent interaction between the user and the spatial environment. This makes interaction more efficient, reduces input errors, and creates a more efficient human-machine interface. In some embodiments, the location of the user's contact with a portion of the physical environment is utilized by the computer system to generate a three-dimensional model of the physical environment and provide more accurate boundary conditions for identifying surface and object boundaries based on the scan, making the display of virtual objects in the three-dimensional environment more accurate and seamless.

[0033] In some embodiments, a computer system displays interactive user interface objects in a three-dimensional environment. The computer system also displays a representation of the physical environment in the three-dimensional environment, with the interactive user interface objects having individual spatial relationships to various positions in the three-dimensional environment that correspond to different locations in the physical environment. When a user interacts with the three-dimensional environment with a portion of the user's hand, such as one or more fingers of the user's hand or the entire hand, through touch input and / or gesture input, parts of the user, including the user's hand and possibly the wrist and arm connected to the hand, can enter a spatial region between a location corresponding to the position of the user interface object (e.g., a location of a physical object or physical surface, a location in free space in the physical environment, etc.) and a location corresponding to a viewpoint of a currently displayed view of the three-dimensional environment (e.g., a location of the user's eyes, a location of a display generating component, a location of a camera capturing a view of the physical environment shown in the three-dimensional environment, etc.). Based on the spatial relationships between the location of the user's hand, the locations corresponding to the position of the user interface object, and the location corresponding to the viewpoint, the computer system determines which parts of the user interface object are visually blocked by parts of the user and which parts of the user interface object are not visually blocked by parts of the user when viewed by the user from the viewpoint location. The computer system then ceases displaying the individual portion of the user interface object that would be visually blocked by the user's portion (e.g., as determined by the computer system), and instead allows a representation of the user's portion to be viewed at the position of the individual portion of the user interface object while maintaining display of another portion of the user interface object that would not be visually blocked by the user's portion (e.g., as determined by the computer system).In some embodiments, in response to detecting movement of a user's portion or movement of the viewpoint (e.g., due to movement of a display generating component, movement of a camera capturing the physical environment, movement of the user's head or torso, etc.), the computer system re-evaluates which portions of the user interface object are visually blocked by the user's portion and which portions of the user interface object are not visually blocked by the user's portion when viewed by the user from the viewpoint location, based on the new spatial relationship between the user's portion, the location corresponding to the viewpoint, and the location corresponding to the position of the user interface object. The computer system then ceases displaying other portions of the user interface object that will be visually blocked by the user's portion (e.g., as determined by the computer system), allowing the portion of the user interface object that was previously ceased from display to be restored to the view of the three-dimensional environment. Visually segmenting the user interface object into multiple portions and replacing the display of one or more portions of the user interface object with a representation of the user's portion that occupies a region of space between a location corresponding to the position of the user interface object and a location corresponding to the viewpoint of the currently displayed view of the three-dimensional environment helps the user visualize and sense the placement location of the user interface object relative to their hand, facilitating interaction between the user and the user interface object in the three-dimensional environment. This makes interactions more efficient, reduces input errors and creates a more efficient human-machine interface.

[0034] FIGS. 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. FIGS. 7A-7B are block diagrams illustrating selecting different audio output modes according to an immersion level at which computer-generated content is presented, according to some embodiments. FIGS. 7C-7H are block diagrams illustrating changing the appearance of a portion of virtual content when a significant physical object approaches a display generating component or a user's location of the display generating component, according to some embodiments. FIGS. 7I-7N are block diagrams illustrating applying visual effects to an area in a three-dimensional environment corresponding to a portion of the physical environment identified based on a scan of the portion of the physical environment, according to some embodiments. FIGS. 7O-7Q are block diagrams illustrating displaying an interactive user interface object at a position in the three-dimensional environment corresponding to a first portion of the physical environment and selectively withdrawing display of individual sub-portions of the user interface object according to the location of a portion of the user moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment, according to some embodiments. The user interfaces of FIGS. 7A-7Q are used to illustrate the processes of FIGS. 8-11, respectively.

[0035] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a location sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a consumer electronics device, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., within a head-mounted or handheld device).

[0036] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or with which the user can interact (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following is a subset of these terms:

[0037] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses, such as sight, touch, hearing, taste, and smell.

[0038] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave according to at least one law of physics. For example, a CGR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to property(ies) of virtual object(s) in a CGR environment may be made in response to representations of body movements (e.g., voice commands). A person may sense and / or interact with a CGR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatial audio environment, providing the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact only with audio objects.

[0039] Examples of CGR include virtual reality and mixed reality.

[0040] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0041] Mixed Reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects), as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtuality continuum, a mixed reality environment is anywhere between, but not including, a fully physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track location and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0042] Examples of mixed reality include augmented reality and augmented virtuality.

[0043] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, videos of a physical environment shown on an opaque display are referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) different from the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, thereby rendering the altered portion a non-photorealistic, altered version of the originally captured image. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0044] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0045] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed toward a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects graphical images onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., physical setting / environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, a display, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the foregoing.

[0046] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of the CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0047] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105 .

[0048] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on their head, their hand, etc.). Thus, display generating component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, display generating component 120 surrounds the user's field of view. In some embodiments, display generating component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward scene 105. In some embodiments, the handheld device is optionally located within a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, display generating component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding display generating component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with CGR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interactions occur in the space in front of the HMD and the CGR content responses are displayed via the HMD. Similarly, a user interface showing interactions with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hands)) may be implemented similarly to an HMD where the interactions are triggered by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hands)).

[0049] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0050] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0051] In some embodiments, one or more communication buses 204 include circuitry that interconnects and controls communication between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0052] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0053] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0054] In some embodiments, data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least display generation component 120 of FIG. 1 and, optionally, one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0055] In some embodiments, tracking unit 244 is configured to map scene 105 and track the position / location of at least display generation components 120, and optionally one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195, relative to scene 105 of FIG. 1 . To that end, in various embodiments, tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, tracking unit 244 includes hand tracking unit 245 and / or eye tracking unit 243. In some embodiments, hand tracking unit 245 is configured to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 , relative to display generation components 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 245 is described in more detail below with respect to FIG. 4 . In some embodiments, eye tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)) or relative to CGR content displayed via display generation component 120. Eye tracking unit 243 is described in more detail below with respect to FIG. 5.

[0056] In some embodiments, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by display generation component 120, and optionally by one or more of output devices 155 and / or peripheral devices 195. To that end, in various embodiments, coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0057] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0058] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0059] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary depending on implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0060] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0061] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communication between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0062] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content.

[0063] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as scene cameras). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0064] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0065] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0066] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0067] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0068] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0069] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0070] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0071] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular implementations, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in Figure 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0072] 4 is a schematic diagram of an example embodiment of hand tracking device 140. In some embodiments, hand tracking device 140 (FIG. 1) is controlled by hand tracking unit 245 (FIG. 2) to track the position / location of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 (e.g., relative to parts of the physical environment surrounding the user, relative to display generating components 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand. In some embodiments, hand tracking device 140 is part of display generating components 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, hand tracking device 140 is separate from display generating components 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0073] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0074] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives display generation component 120 accordingly. For example, a user may interact with software running on controller 110 by moving hand 408 and changing hand posture.

[0075] In some embodiments, the image sensor 404 projects a spot pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the pattern's spots. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define a set of orthogonal x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurement, based on single or multiple cameras or other types of sensors.

[0076] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors with patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D locations of the user's wrist joints and fingertips.

[0077] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on controller 110 via the API described above. This program can, for example, move and modify an image presented on display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0078] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or alternatively may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively, or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). While the controller 110 is shown in FIG. 4 as, by way of example, a separate unit from the image sensor 440, some or all of the processing functionality of the controller may be implemented by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or otherwise associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device) or using any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0079] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, e.g., the measured z-distance from the image sensor 404, with increasing depth resulting in darker shades. The controller 110 processes these depth values ​​to identify and segment components of the image (e.g., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0080] 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In FIG. 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and placed on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0081] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 243 (FIG. 2) to track the position and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if the display generation component is a handheld device or a CGR chamber, eye tracking device 130 is optionally a device separate from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generating components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally used in conjunction with head-mounted display generating components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally part of non-head-mounted display generating components.

[0082] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide a 3D virtual view to the user. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0083] As shown in FIG. 5 , in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, it may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a separate eye-tracking camera and lighting source.

[0084] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility before delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil location, central visual location, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

[0085] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, a projector, etc.) and may be directed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be directed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0086] In some embodiments, controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye tracking camera 540 for various purposes, such as in processing frames 562 for display. Controller 110 optionally estimates the user's viewpoint on display 510 based on gaze tracking input 542 obtained from eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from gaze tracking input 542 is optionally used to determine the user's current looking direction.

[0087] Some possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0088] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520, as an example. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0089] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR range, thereby not introducing noise into the eye-tracking system. Note that the location and angle of the eye-tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0090] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and / or augmented virtuality) experience.

[0091] Figure 6 shows a glint-assisted eye tracking pipeline, according to some embodiments. In some embodiments, the eye tracking pipeline is implemented by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0092] As shown in FIG. 6, an eye-tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0093] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0094] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0095] 6 is intended to serve as an example of eye-tracking technology that may be used in particular implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0096] In this disclosure, various input methods are described with respect to interaction with a computer system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example may be compatible with, and optionally utilize, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example may be compatible with, and optionally utilize, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. Where an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with, and optionally utilize, the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment.

[0097] User Interface and Related Processes We now turn our attention to embodiments of user interfaces (“UIs”) and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, equipped with display generating components, one or more input devices, and (optionally) one or more cameras.

[0098] 7A-7Q illustrate a three-dimensional environment displayed via display generating components (e.g., display generating component 7100, display generating component 120, etc.) and interactions occurring in the three-dimensional environment caused by user input to the three-dimensional environment, according to various embodiments. In some embodiments, the input is directed to a virtual object in the three-dimensional environment by a user's gaze detected at the position of the virtual object, a hand gesture performed at a location in the physical environment corresponding to the position of the virtual object, or a hand gesture performed at a location in the physical environment unrelated to the position of the virtual object while the virtual object has input focus (e.g., selected by gaze, selected by pointer, selected by previous gesture input, etc.). In some embodiments, the input is directed to a representation of a physical object or a virtual object corresponding to a physical object by a user's hand movement (e.g., movement of the entire hand, movement of the entire hand in a discrete pose, movement of one part of the hand relative to another part of the hand, relative movement between two hands, etc.) and / or a manipulation of the physical object (e.g., touch, swipe, tap, open, move toward, move relative, etc.).

[0099] In some embodiments, the three-dimensional environment displayed via the display generation component is a virtual three-dimensional environment that includes virtual objects and content at different virtual positions within the three-dimensional environment without a representation of the physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that displays virtual objects at different virtual positions within the three-dimensional environment constrained by one or more physical aspects of the physical environment (e.g., the position and orientation of walls, floors, surfaces, the direction of gravity, the time of day, etc.). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of the physical environment. The representation of the physical environment includes respective representations of physical objects and surfaces at different positions within the three-dimensional environment, such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. When virtual objects are positioned relative to the positions of the representations of the physical objects and surfaces within the three-dimensional environment, they appear to have corresponding spatial relationships with the physical objects and surfaces in the physical environment.

[0100] In some embodiments, the display generating components include a pass-through portion in which a representation of the physical environment is displayed. In some embodiments, the pass-through portion is a transparent or translucent (e.g., see-through) portion of the display generating components that surrounds the user's field of view and reveals at least a portion of the physical environment within the field of view. For example, the pass-through portion is a translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent portion of a head-mounted or head-up display, allowing the user to view the real world surrounding the user through it without removing the head-mounted display or moving away from the head-up display. In some embodiments, the pass-through portion gradually transitions from translucent or transparent to fully opaque when displaying a virtual or mixed reality environment. In some embodiments, the pass-through portion of the display generating components displays a live feed of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., rear-facing camera(s) of a mobile device or associated with a head-mounted display, or other camera that provides image data to an electronic device). In some embodiments, one or more cameras are aimed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generating components). In some embodiments, one or more cameras are aimed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user).

[0101] In some embodiments, when displaying virtual objects in positions corresponding to the locations of one or more physical objects in the physical environment, at least some of the virtual objects are displayed in alignment with (e.g., replacing) a portion of the camera's live view (e.g., a portion of the physical environment captured by the live view). In some embodiments, at least some of the virtual objects and content are projected onto a physical surface or open space in the physical environment and are visible through a pass-through portion of the display generating components (e.g., viewable as part of the camera view of the physical environment, or through a transparent or semi-transparent portion of the display generating components, etc.). In some embodiments, at least some of the virtual objects and content are displayed to overlay a portion of the display, blocking at least some, but not all, view of the physical environment visible through a transparent or semi-transparent portion of the display generating components. In some embodiments, at least some of the virtual objects are projected directly onto the user's retina in a position relative to the image of the representation of the physical environment (e.g., viewable through the camera view of the physical environment, or through a transparent portion of the display generating components, etc.).

[0102] In some embodiments, the display generating component displays different views of the three-dimensional environment according to user input or movement that changes the virtual position of the viewpoint of the currently displayed view of the three-dimensional environment relative to the three-dimensional environment. In some embodiments, if the three-dimensional environment is a virtual environment, the viewpoint moves according to a navigation or movement request (e.g., an air hand gesture, a gesture performed by movement of one part of a hand relative to another part of a hand, etc.) without requiring movement of the user's head, torso, and / or display generating component in the physical environment. In some embodiments, movement of the user's head and / or torso relative to the physical environment and / or movement of the display generating component of a computer system or other location-sensing element (e.g., due to the user holding the display generating component or wearing an HMD, etc.) causes corresponding movement of the viewpoint relative to the three-dimensional environment (e.g., a corresponding change in direction of movement, distance of movement, speed of movement, and / or orientation, etc.), resulting in a corresponding change in the currently displayed view of the three-dimensional environment. In some embodiments, when a virtual object has a predetermined spatial relationship to the viewpoint, movement of the viewpoint relative to the three-dimensional environment causes movement of the virtual object relative to the three-dimensional environment while the position of the virtual object within the field of view is maintained (e.g., the virtual object is said to be head-locked). In some embodiments, the virtual object is body-locked to the user, moving relative to the three-dimensional environment as the user moves as a whole within the physical environment (e.g., carrying or wearing the display generating components and / or other location sensing components of the computer system), but does not move within the three-dimensional environment in response to movement of the user's head (e.g., the display generating components and / or other location sensing components of the computer system rotating around the user's fixed location within the physical environment).

[0103] In some embodiments, the views of the three-dimensional environment shown in FIGS. 7A-7Q include representations of the user's hand(s), arm(s), and / or wrist(s). In some embodiments, the representations are part of the representation of the physical environment provided via a display generation component. In some embodiments, the representations are not part of the representation of the physical environment but are captured separately (e.g., by one or more cameras directed at the user's hand(s), arm(s), and wrist(s)) and displayed in the three-dimensional environment independent of the view of the three-dimensional environment. In some embodiments, the representations include camera images captured by one or more cameras of the computer system(s) or stylized versions of the arms, wrists, and / or hands based on information captured by various sensors. In some embodiments, the representations replace the display of a portion of the representation of the physical environment, are overlaid on a portion of the representation of the physical environment, or block the view of a portion of the representation of the physical environment. In some embodiments, if the display generation component does not provide a view of the physical environment but instead provides an entirely virtual environment (e.g., no camera field of view or transparent pass-through portions), a real-time visual representation (e.g., a stylized representation or segmented camera image) of one or both of the user's arms, wrists, and / or hands may still be displayed in the virtual environment. While in some embodiments, a representation of the user's hands is shown in the figures, it should be understood that unless made clear by the corresponding description, the representation of the user's hands may not necessarily be displayed at all times and / or may not need to be displayed within the user's field of view when providing input necessary to interact with the three-dimensional environment.

[0104] 7A-7B are block diagrams illustrating selecting different audio output modes according to the immersion level at which computer-generated content is presented, according to some embodiments.

[0105] In some embodiments, a computer system displays computer-generated content, such as a movie, a virtual office, an application environment, a game, or a computer-generated experience (e.g., a virtual reality experience, an augmented reality experience, a mixed reality experience, etc.). In some embodiments, the computer-generated content is displayed in a three-dimensional environment (e.g., environment 7102 of FIGS. 7A-7B or another environment). In some embodiments, the computer system can display the visual components of the computer-generated content (e.g., visual content 7106 or other visual content) with multiple levels of immersion corresponding to varying degrees of emphasis of visual input from the virtual content relative to visual input from the physical environment. In some embodiments, higher levels of immersion correspond to greater emphasis of visual input from the virtual content over visual input from the physical environment. Similarly, in some embodiments, audio components of the computer-generated content that accompany and / or correspond to the visual components of the computer-generated content (e.g., sound effects and soundtracks in movies; audio alerts, audio feedback, and system sounds in application environments; sound effects, speech, and audio feedback in games; and / or sound effects and audio feedback in computer-generated experiences, etc.) can be output at multiple levels of immersion. In some embodiments, the multiple immersion levels optionally correspond to varying degrees of spatial correspondence between the position of a virtual sound source within the virtual content displayed via the display generating components and the perceived location of the virtual sound source within a selected frame of reference for the virtual sound source. In some embodiments, the selected frame of reference for a particular virtual sound source is based on the physical environment, the virtual three-dimensional environment of the computer-generated content, the viewpoint of a currently displayed view of the three-dimensional environment of the computer-generated content, the location of the display generating components in the physical environment, or the location of the user in the physical environment, etc.In some embodiments, a higher level of immersion corresponds to a higher level of correspondence between the position of a virtual sound source in the computer-generated environment and the perceived location of the virtual sound source in a selected frame of reference of the audio component of the computer-generated content (e.g., a frame of reference based on the three-dimensional environment depicted in the computer-generated experience, a frame of reference based on the location of the viewpoint, a frame of reference based on the location of a display generating component, a frame of reference based on the location of the user, etc.). In some embodiments, a lower level of correspondence between the position of a virtual sound source in the computer-generated environment and the perceived location of the sound source in a selected frame of reference of the audio component of the computer-generated content is the result of a higher level of correspondence between the perceived location of the virtual sound source and the location of an audio output device in the physical environment (e.g., sound appears to come from the location of the audio output device regardless of the position of the virtual sound source in the three-dimensional environment of the computer-generated content, and / or regardless of the location of the viewpoint, the location of a display generating component, and / or the location of the user, etc.). In some embodiments, a computer system detects a first event corresponding to a request to present a first computer-generated experience (e.g., request 7112, request 7114, etc., of FIGS. 7A-7B , or other request), and the computer system selects an audio output mode for outputting an audio component of the computer-generated experience according to an immersion level at which the visual component of the computer-generated experience is displayed via the display generation component. At a higher immersion level associated with displaying the visual content of the first computer-generated experience, the computer system selects an audio output mode for presenting audio content of the computer-generated experience having a corresponding higher immersion level. In some embodiments, displaying the visual content with a higher immersion level includes displaying the visual content with a greater spatial extent in a three-dimensional environment (e.g., as shown in FIG. 7B as opposed to FIG. 7A ), and outputting the audio content with a corresponding higher immersion level includes outputting the audio content in a spatial audio output mode.In some embodiments, when switching between displaying visual content having two different levels of immersion (e.g., from a higher level of immersion to a lower level of immersion, from a lower level of immersion to a higher level of immersion, etc.), the computer system also switches between outputting audio content having two different levels of immersion (e.g., from a spatial audio output mode to a stereo audio output mode, from a surround sound output mode to a stereo audio output mode, from a stereo audio output mode to a surround sound output mode, from a stereo audio output mode to a spatial audio output mode, etc.).

[0106] As described herein, audio output devices, including standalone speakers (e.g., sound bars, external speakers, etc.), built-in audio output components of displays or computer systems (e.g., built-in speakers of head-mounted display devices, touchscreen display devices, portable electronic devices, or head-up displays, etc.), and wearable audio output devices (e.g., headphones, earphones, earcups, earbuds, etc.), are widely used to provide audio output to users. The same audio content, when output using different audio output devices and / or different output modes of the same audio output device, can have different audio characteristics that make the audio content sound different to a user perceiving the audio output. As such, it is desirable to adjust the audio output mode, including changing the characteristics of the audio, the characteristics of the sound source, and / or the characteristics of the audio output device, based on the level of immersion with which the visual content of the computer-generated experience is provided to the user, so that the audio content and the visual content of the computer-generated experience blend together and more seamlessly complement each other as the computer-generated experience is provided to the user.

[0107] Existing stereo and mono audio output modes provide audio relative to a frame of reference tied to the audio output device. In the case of a fixed audio output device, sound appears to originate from the location of the audio output device in the physical environment regardless of the user's movement in the physical environment and regardless of changes in the visual content of the computer-generated experience (e.g., changes due to movement of virtual sound sources and / or movement of the viewpoint in the three-dimensional environment of the computer-generated experience, etc.). In the case of a wearable audio output device that remains stationary relative to a part of the user's body (e.g., ear, head, etc.), sound appears to be locked to a part of the user's body regardless of changes in the visual content of the computer-generated experience in the three-dimensional environment of the computer-generated experience (e.g., changes due to movement of virtual sound sources, changes due to movement of the viewpoint (e.g., movement of the viewpoint caused by a movement request by the user or the computer system that is not caused by and does not correspond to movement of a part of the user's body, etc.)). In some cases, the audio output device and display generation components of a computer system may be housed separately and move relative to each other in the physical environment during the presentation of computer-generated content via the audio output device and display generation components. In such cases, regardless of changes in the location of display generating components in the physical environment or changes in the visual content of the computer-generated experience (e.g., due to movement of virtual sound sources and / or movement of the viewpoint in the three-dimensional environment of the computer-generated experience (e.g., movement caused by a movement request or in response to and in accordance with movement of the user or portions thereof in the physical environment)), the sound still appears to originate from the audio output device. Thus, stereo and mono audio output modes provide a less immersive listening experience and a less realistic sound than spatial audio output mode when the audio content of the computer-generated experience is provided to the user using stereo or mono audio output mode.

[0108] In some embodiments, the spatial audio output mode simulates a more realistic listening experience in which sound appears to come from a source in a separate frame of reference, such as a three-dimensional environment displayed via a display generation component (e.g., an augmented reality environment, a virtual reality environment, a pure pass-through view of the physical environment surrounding the user, etc.), and the position of the simulated sound source is decoupled from the location and movement of the audio output device within the physical environment.

[0109] In some embodiments, the frame of reference for the spatial audio output mode is based on the physical environment represented in the three-dimensional environment of the computer-generated experience, and the frame of reference optionally does not change with movement of the user in the physical environment, movement of the audio output device, and / or movement of the display generation components.

[0110] In some embodiments, the frame of reference for the spatial audio output mode is based on the virtual three-dimensional environment of the computer-generated experience. In some embodiments, the frame of reference optionally does not change due to movements of the user in the physical environment, movements of the audio output device, and / or movements of the display generating components, if these movements do not cause corresponding movements of the virtual three-dimensional environment.

[0111] In some embodiments, the frame of reference for the spatial audio output mode is based on the three-dimensional environment associated with the viewpoint of the currently displayed view of the three-dimensional environment. In some embodiments, the frame of reference optionally does not change due to movements of the user in the physical environment, movements of the audio output device, and / or movements of the display generating components, if these movements do not cause corresponding movements of the viewpoint of the currently displayed view of the three-dimensional environment.

[0112] In some embodiments, the frame of reference for audio content output in spatial audio mode is, optionally, different from the frame of reference for visual content in the computer-generated experience. For example, in some embodiments, the visual content is displayed relative to a frame of reference associated with the physical or virtual environment visually presented via the display generation component, while at least a portion of the virtual sound sources (e.g., external narrator, internal dialogue, etc.) is in a frame of reference associated with the user's viewpoint.

[0113] In some embodiments, the audio content of the computer-generated experience optionally includes sound sources tied to different frames of reference, such as a first frame of reference for virtual sound sources that do not have a corresponding virtual position within the three-dimensional environment of the computer-generated experience (e.g., system-level sounds, external narration, etc.), a second frame of reference for virtual sound sources that have a corresponding visual embodiment within the three-dimensional environment of the computer-generated experience (e.g., virtual objects, virtual surfaces, virtual lights, etc.), and optionally a third frame of reference for virtual sound sources that are far from the viewpoint, out of view, hidden, etc. (e.g., ambient noises such as sounds of waves, insects, wind, rain, jungle, etc.). In some embodiments, the first frame of reference is fixed to, and optionally moves with, the user's head, display generating components, and / or the viewpoint. In some embodiments, the second frame of reference is tied to the three-dimensional environment of the computer-generated experience and, optionally, moves with the display generating components. In some embodiments, the third frame of reference is tied to the physical environment and, optionally, does not move with the user, display generating components, or the viewpoint. In connection with providing visual content using display generation components, the computer system can select and configure a spatial audio mode to output sound based on the visual content being presented via the display generation components, based on the spatial configuration between the audio output device and the display generation components in the physical environment, and based on the spatial configuration between the user, the display generation components, and the audio output device, to provide a more realistic and immersive listening experience.

[0114] In some embodiments, a spatial audio output mode is a mode that enables sound output from an audio output device(s) to sound as if the sound is coming from one or more locations (e.g., one or more sound sources) in a separate frame of reference selected relative to a virtual sound source, such as a three-dimensional environment of a computer-generated experience or a physical environment, and the positioning of the one or more simulated or perceived sound sources is decoupled from or independent of movement of the audio output device(s) relative to the separate frame of reference. Typically, the one or more perceived sound sources, when fixed, are fixed relative to the separate frame of reference associated with the sound source, and when moved, they move relative to the separate frame of reference.

[0115] In some embodiments, the frame of reference is a frame of reference based on the physical environment represented in the computer-generated experience provided via a display generation component of the computer system. In some embodiments, when the frame of reference is based on the physical environment (e.g., when the computer-generated experience is an augmented reality experience based on the physical environment, a pass-through view of the physical environment, etc.), one or more perceived sound sources have respective spatial locations within the physical environment. For example, in some embodiments, the computer-generated experience includes visual counterparts of the perceived sound sources (e.g., virtual objects that generated the sound in the computer-generated experience) having respective positions that correspond to their respective spatial locations within the physical environment. In some embodiments, the computer-generated experience includes sounds that do not have visual counterparts (e.g., remote or hidden virtual objects that generated the sound in the computer-generated experience, virtual wind, sound effects, external narrators, etc.), but have origins that correspond to their respective spatial locations within the physical environment. In some embodiments, as the audio output device(s) move around the physical environment, the audio output from the audio output device(s) is adjusted so that the sounds continue to sound as if they are coming from one or more perceived sound sources at their respective spatial locations within the physical environment. If one or more perceived sound sources are moving sources that move through a series of spatial locations around the physical environment, the audio output from the audio output device(s) is adjusted so that the sound continues to sound as if it were coming from the one or more perceived sound sources at a series of spatial locations in the physical environment. Such adjustments for moving sound sources also take into account any movement of the audio output device(s) relative to the physical environment (e.g., if the audio output device(s) moves relative to the physical environment along a similar path as the moving sound source(s) so as to maintain a constant spatial relationship with the moving sound source(s), the audio is output so that the audio does not appear to be moving relative to the audio output device(s).In some embodiments, when audio content is output using spatial audio output mode and a frame of reference based on the physical environment represented in the computer-generated experience, the perspective of the currently displayed view of the three-dimensional environment changes according to the movement of the user and / or display generation components in the physical environment, and the user perceives sounds as coming from the virtual positions of the virtual sound sources and experiences the visual content of the three-dimensional environment in the same frame of reference based on the physical environment represented in the computer-generated experience.

[0116] In some embodiments, the reference frame is based on a virtual three-dimensional environment of a computer-generated experience provided via a display generation component of a computer system. In some embodiments, if the reference frame is based on a virtual three-dimensional environment (e.g., an environment such as a virtual three-dimensional movie, a three-dimensional game, a virtual office, etc.), the one or more perceived sound sources have respective spatial positions within the virtual three-dimensional environment. In some embodiments, as the audio output device(s) move around the physical environment, the audio output from the audio output device(s) is adjusted to continue to sound as if the sound is coming from one or more perceived sound sources at respective spatial positions in the virtual three-dimensional environment. If the one or more perceived sound sources are moving sources moving through a series of spatial positions around the virtual three-dimensional environment, the audio output from the audio output device(s) is adjusted to continue to sound as if the sound is coming from one or more perceived sound sources at a series of spatial positions in the virtual three-dimensional environment. In some embodiments, when audio content is output using a spatial audio output mode and a frame of reference based on a three-dimensional environment of a computer-generated experience, the perspective of the currently displayed view of the three-dimensional environment changes according to the movement of the user and / or display generating components in the physical environment. The user perceives sound as coming from the virtual position of a virtual sound source and experiences the visual content of the three-dimensional environment in the same frame of reference. In some embodiments, when audio content is output using a spatial audio output mode and a frame of reference based on a three-dimensional environment of a computer-generated experience, the perspective of the currently displayed view of the three-dimensional environment changes according to movement requests provided by the user and / or according to the movement of the user and / or display generating components in the physical environment. The user perceives sound as coming from the virtual position of a virtual sound source and experiences the visual content of the three-dimensional environment in the same frame of reference, with the user's virtual position tied to the perspective of the currently displayed view.

[0117] In some embodiments, the frame of reference for the spatial audio output mode is fixed to an electronic device, such as a display generating component, that is outputting visual content corresponding to the audio content being output via an audio output device (e.g., the sound follows the display generating component). For example, the location of the simulated source of the audio in the physical environment moves in response to movement of the display generating component in the physical environment, but does not correspond to movement of the audio output device in the physical environment. For example, in some embodiments, the display generating component is a head-mounted display or handheld display device, and the audio output device is located in the physical environment and does not follow the user's movements. In some embodiments, the frame of reference for the spatial audio effect is fixed to the display generating component and indirectly to the user as the display generating component and the user move around the physical environment relative to the audio output device(s). In some embodiments, when audio content is output using a spatial audio output mode and frame of reference based on a three-dimensional environment of a computer-generated experience, the viewpoint of the currently displayed view of the three-dimensional environment changes according to movement requests provided by the user and / or according to movement of the user and / or the display generating component in the physical environment. The user perceives sound as coming from the virtual position of the virtual sound source and experiences the visual content of the three-dimensional environment in the same frame of reference, with the user's virtual position tied to the viewpoint of the currently displayed view.

[0118] In some embodiments, the frame of reference for at least some of the spatial audio effects is fixed to the viewpoint of a currently displayed view of the three-dimensional environment (e.g., an augmented reality environment, a mixed reality environment, a virtual reality environment, etc.) presented via display generating components. In some embodiments, the viewpoint moves relative to the three-dimensional environment to provide views of the three-dimensional environment from different positions or viewpoints within the three-dimensional environment during the computer-generated experience. In some embodiments, the viewpoint remains stationary in the three-dimensional environment during the computer-generated experience. In some embodiments, movement of the viewpoint in the three-dimensional environment is caused by and corresponds to movement of the display generating components in the physical environment. In some embodiments, movement of the viewpoint in the three-dimensional environment is caused by and corresponds to movement of the entire user or movement of the user's head to torso relative to the physical environment. In some embodiments, movement of the viewpoint in the three-dimensional environment is caused by and corresponds to navigation or movement requests provided by the user and / or generated by the computer system. In some embodiments, one or more perceived sound sources have respective spatial locations within the three-dimensional environment relative to the viewpoint. For example, in some embodiments, the computer-generated experience includes visual counterparts of perceived sound sources (e.g., virtual objects, virtual lights, virtual surfaces, etc. that generated the sound in the computer-generated experience) that have respective positions in the three-dimensional environment relative to the viewpoint. In some embodiments, the computer-generated experience includes sounds that do not have visual counterparts (e.g., remote or hidden virtual objects, virtual wind, sound effects, external narrators, etc. that generated the sound in the computer-generated experience), but have sources that correspond to respective positions in the three-dimensional environment relative to the viewpoint. In some embodiments, as the viewpoint moves around the three-dimensional environment, the audio output from the audio output device(s) is adjusted so that the sounds continue to sound as if they were coming from one or more perceived sound sources at respective positions in the three-dimensional environment.

[0119] In some embodiments, a computing system is configured to display visual components of CGR content via display generation components having two or more levels of immersion. In some embodiments, the computer system displays visual components of CGR content with at least a first level of immersion, a second level of immersion, and a third level of immersion. In some embodiments, the computer system displays visual components of CGR content with at least two levels of immersion, providing less immersive and more immersive visual experiences relative to each other. In some embodiments, the computing system transitions the visual content displayed via the display generation components between different levels of immersion in response to a series of one or more events (e.g., natural progression of an application or experience; starting, stopping, and / or pausing an experience in response to user input; changing the immersion level of an experience in response to user input; a change in the state of the computing device; a change in the external environment; etc.). In some embodiments, the first, second, and third levels of immersion correspond to an increase in the amount of virtual content present in the CGR environment and / or a decrease in the amount of a representation of the surrounding physical environment present in the CGR environment (e.g., a representation of a portion of the physical environment in front of the first display generation component). In some embodiments, the first, second, and third immersion levels correspond to different modes of content display having increased image fidelity (e.g., increased pixel resolution, increased color resolution, increased saturation, increased brightness, increased opacity, increased image detail, etc.) and / or spatial extent (e.g., angular range, spatial depth, etc.) for the visual components of the computer-generated content, and / or decreased image fidelity and / or spatial extent for the representation of the surrounding physical environment. In some embodiments, the first immersion level is a pass-through mode in which the physical environment is fully visible to the user through the display generating components (e.g., as a camera view of the physical environment or through transparent or translucent portions of the display generating components).In some embodiments, the visual CGR content presented in pass-through mode includes a pass-through view of the physical environment with a minimal amount of virtual elements simultaneously visible as the view of the physical environment, or with only virtual elements peripheral to the user's view of the physical environment (e.g., indicators and controls displayed in a peripheral area of ​​the display). For example, the view of the physical environment occupies the central and majority of the field of view provided by the display generation components, with only a few controls (e.g., movie title, progress bar, playback controls (e.g., play button), etc.) displayed in the peripheral area of ​​the field of view provided by the display generation components. In some embodiments, a first immersion level is a pass-through mode in which the physical environment is fully visible to the first user through the display generation components (e.g., as a camera view of the physical environment or through a transparent portion of the display generation components), and the visual CGR content is displayed in a virtual window or virtual frame that overlays, replaces, or blocks a portion of the representation of the physical environment. In some embodiments, the second level of immersion is a mixed reality mode in which a pass-through view of the physical environment is augmented with virtual elements generated by a computer system, the virtual elements occupying a central and / or majority area of ​​the user's field of view (e.g., virtual content is integrated with the physical environment within the view of the computer-generated environment). In some embodiments, the second level of immersion is a mixed reality mode in which the pass-through view of the physical environment is augmented with virtual windows, viewports, or frames that overlay portions of the representation of the physical environment, replace the display, or block the view, with additional depth or spatial extent revealed as display-generating components are moved relative to the physical environment. In some embodiments, the third level of immersion is an augmented reality mode in which virtual content is displayed in the three-dimensional environment along with a representation of the physical environment, and virtual objects are distributed throughout the three-dimensional environment in positions that correspond to different locations in the physical environment. In some embodiments, the third level of immersion is a virtual reality mode in which virtual content is displayed in the three-dimensional environment without a representation of the physical environment.In some embodiments, the different immersion levels represent increasing levels of immersion relative to each other.

[0120] As described herein, according to some embodiments, a computer system selects an audio output mode for outputting audio content of a computer-generated experience (e.g., an application, a movie, a video, a game, etc.) according to a level of immersion at which visual content of the computer-generated experience is displayed by a display generation component. In some embodiments, as the level of immersion at which the visual content is displayed increases (e.g., from a first immersion level to a second immersion level, from the first immersion level to a third immersion level, or from the second immersion level to the third immersion level, etc.), the computer system switches the audio output mode from a less immersive output mode to a more immersive output mode (e.g., from a first audio output mode to a second audio output mode, or from the first audio output mode to a third audio output mode, or from the second audio output mode to a third audio output mode, etc., where the first audio output mode, the second audio output mode, and the third audio output mode correspond to audio output of increasing immersion levels). As described herein, the spatial audio output mode corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The spatial audio output mode corresponds to a higher level of immersion than the surround sound output mode. The surround audio output mode is a mode that corresponds to a higher level of immersion than the stereo audio output mode and the mono audio output mode. The stereo audio output mode corresponds to a higher level of immersion than the mono audio output mode. In some embodiments, the computer system selects an audio output mode from a plurality of available audio output modes, e.g., a mono audio output mode, a stereo audio output mode, a surround sound output mode, a spatial audio output mode, etc., based on the level of immersion at which the visual content of the computer-generated experience is being provided via the display generation component.

[0121] 7A-7B illustrate an exemplary scenario in which a first computer-generated experience is provided by a computer system (e.g., computing system 101 of FIG. 1 or computing system 140 of FIG. 4) in communication with a display generating component (e.g., a display 7100, another type of display generating component such as a head-mounted display) and one or more audio output devices.

[0122] In Figure 7A, the visual content of the computer-generated experience (e.g., content 7106, or other content, etc.) is provided at a first immersion level that is a lower immersion level of two or more immersion levels at which the computer-generated experience can be provided. In Figure 7B, the visual content of the computer-generated experience (e.g., content 7106, or other content, etc.) is provided at a second immersion level that is a higher immersion level of two or more immersion levels at which the computer-generated experience can be provided.

[0123] 7A and 7B may occur at the time the computer-generated experience is initiated (e.g., in response to a user command, in response to an event generated by the computer system, etc.) without requiring a transition from a scenario shown in the other figure (e.g., without requiring first displaying visual content having a different immersion level). As a result, a corresponding audio output mode is selected by the computer system to output the audio content of the computer-generated experience according to the immersion level at which the visual content of the computer-generated experience is being provided.

[0124] In some embodiments, the computer system transitions from the scenario shown in Figure 7A to the scenario shown in Figure 7B or vice versa (e.g., in response to a user command, in response to an event generated by the computer system, according to a preset condition being met, etc.), such that the computer system transitions from one audio output mode to another in response to changes in the level of immersion at which the visual content of the computer-generated experience is being provided.

[0125] In some embodiments, the computer-generated experience (e.g., a three-dimensional movie, a virtual reality game, a video, a three-dimensional environment including user interface objects, etc.) is a virtual experience that occurs in a virtual three-dimensional environment. In some embodiments, the computer-generated experience is an augmented reality experience that includes representations of a physical environment and virtual content. In FIGS. 7A and 7B, according to some embodiments, objects (e.g., object 7104, etc.) and surfaces (e.g., vertical surfaces 7004′ and 7006′, horizontal surface 7008′, etc.) may represent virtual objects and surfaces in a virtual three-dimensional environment (e.g., environment 7102, or another virtual environment, etc.). In FIGS. 7A and 7B, three-dimensional environment 7102 may also represent an augmented reality environment that includes representations of virtual objects and surfaces (e.g., object 7104, the surface of a virtual table, etc.) and physical objects and surfaces (e.g., vertical walls represented by representations 7004′ and 7006′, floors, tables, windows, etc. represented by representation 7008′), according to some embodiments. Environment 7102, in this example, is an environment that can exist independently of and in front of the display of visual content 7106 of the computer-generated experience.

[0126] As shown in FIG. 7A , the spatial relationship between the display generating component (e.g., display 7100, or another type of display) and the user is such that the user is positioned to view visual CGR content presented via the display generating component. For example, the user faces the display side of the display generating component. In some embodiments, the display generating component is a display of an HMD, and the spatial relationship depicted in FIG. 7A corresponds to a user wearing or holding the HMD with the display side of the HMD facing the user's eyes. In some embodiments, the user is positioned to view CGR content presented via the display generating component when the user faces a portion of the physical environment illuminated by the projection system of the display generating component. For example, virtual content is projected onto a portion of the physical environment, and the virtual content and the portion of the physical environment are seen by the user through a camera view of the portion of the physical environment or a transparent portion of the display generating component when the user faces the display surface of the display generating component. In some embodiments, the display generating component emits light that forms an image on the user's retina when the user faces the display surface of the display generating component. For example, the virtual content is displayed by an LCD or LED display, overlaying or replacing a portion of the view of the physical environment displayed by the LCD or LED display, such that a user facing the display surface of the LCD or LED display can see the virtual content along with a view of the portion of the physical environment. In some embodiments, the display generation component displays a camera view of the physical environment in front of the users, or includes a transparent or translucent portion through which a portion of the physical environment in front of a first user is visible to the user.

[0127] In some embodiments, the computer system controls one or more audio output devices, each of which provides audio output to the user (e.g., audio portions of the CGR content accompanying the visual portions of the displayed CGR content, system-level sounds external to the CGR content, etc.). In some embodiments, the computer system generates and / or adjusts the audio output before outputting the audio CGR content using respective audio output modes of the audio output devices, including two or more of a stereo audio output mode, a surround sound output mode, a spatial audio output mode, etc., corresponding to different immersion levels at which the audio CGR content may be output. In some embodiments, the computing system optionally partially or completely shields the user from sound propagating from the surrounding physical environment (e.g., via one or more active or passive noise suppression or cancellation components). In some embodiments, the amount of active sound insulation or sound pass-through is determined by the computing system based on the current immersion level associated with the CGR content shown via the display generation components (e.g., no sound insulation when in pass-through mode, partial sound insulation when in mixed reality mode, full sound insulation when in virtual reality mode, etc.).

[0128] In some embodiments, as shown in FIG. 7A , a computing system displays visual CGR content 7106 via display generation components 7100 (e.g., in response to a user command 7112 to display CGR content in a frame or viewport (e.g., a frame or viewpoint 7110, a window, a virtual screen, etc.), or in response to a transition from a less immersive mode or a transition from a more immersive mode (e.g., as shown in FIG. 7B )). At the moment shown in FIG. 7A , the computing system is displaying a movie (e.g., a three-dimensional movie, a two-dimensional movie, an interactive computer-generated experience, etc.). The movie is displayed in frame or viewpoint 7110 such that the movie content is visible simultaneously with a representation of the physical environment within environment 7102. In some embodiments, this display mode corresponds to a low or medium level of immersion associated with the CGR content presented via the display generation components.

[0129] In some embodiments, the representation of the physical environment shown in the three-dimensional environment (e.g., environment 7102, another environment, etc.) includes a camera view of the portion of the physical environment that would be in the first user's field of view if the user's eyes were not blocked by the presence of the display generating components (e.g., if the first user were not wearing an HMD or were holding an HMD in front of their eyes). In the display mode shown in FIG. 7A , CGR content 7106 (e.g., a movie, a three-dimensional augmented reality environment, a user interface, a virtual object, etc.) is displayed to overlay or replace some, but not all, of the representation of the physical environment. In some embodiments, the display generating components include transparent portions, through which portions of the physical environment are visible to the first user. In some embodiments, in the display mode shown in FIG. 7A , CGR content 7106 (e.g., a movie, a three-dimensional augmented reality environment, a user interface, a virtual object, etc.) is projected onto a physical surface or open space within the physical environment and is visible through a transparent portion of the display generating components with the physical environment or through a camera view of the physical environment provided by the first display generating component. In some embodiments, the CGR content 7106 is displayed to overlay a limited portion of the display, blocking the view of a limited portion, but not all, of the physical environment visible through a transparent or semi-transparent portion of the first display generating component. In some embodiments, as shown in Figure 7A, the visual CGR content is limited to a sub-portion of the field of view provided by the display generating component, such as a virtual window 7110, a virtual viewport, a virtual screen, or a position corresponding to the location of a finite physical surface, which simultaneously includes other sub-portions of the three-dimensional environment (e.g., representations of virtual objects and / or the physical environment).

[0130] In some embodiments, other user interface objects related to and / or unrelated to the CGR content (e.g., playback controls 7108, a dock with application icons, etc.) are optionally displayed simultaneously with the visual CGR content in the three-dimensional environment, as shown in Figure 7A. In some embodiments, the visual CGR content is optionally three-dimensional content, and the viewpoint of the currently displayed view of the three-dimensional content in window 7110 moves in response to user input and / or movement of display generating components or the user's head in the physical environment.

[0131] In some embodiments, the location of a subportion of the three-dimensional environment to which the visual CGR content is confined (e.g., window 7110, viewport, etc.) is movable during display of the visual CGR content. For example, according to some embodiments, the window 7110 or viewport displaying the visual CGR content is movable according to a user's pinch-and-drag gestures. In some embodiments, the window or viewport displaying the visual CGR content remains in a predetermined portion of the field of view provided by the display generation component (e.g., the center of the field of view, or a position selected by the user) when the user moves the display generation component relative to the physical environment (e.g., when the user is walking through the physical environment wearing an HMD or when the user moves a handheld display through the physical environment).

[0132] In this example, as shown in FIG. 7A , when displaying visual CGR content with a low or medium level of immersion, the computer system selects an audio output mode corresponding to the low or medium level of immersion, such as a stereo audio output mode in which the output sound is relative to a frame of reference associated with the location of the audio output device(s) in the physical environment. In this example, according to some embodiments, the audio output device is optionally movable relative to the display generating components and / or the user in the physical environment. According to some embodiments, audio CGR content output according to the stereo audio output mode does not take into account the position and / or movement of the window 7110 or viewport of the visual CGR content within the three-dimensional environment 7106. According to some embodiments, audio CGR content output according to the stereo audio output mode does not take into account the position and / or movement of the virtual sound source(s) within the window 7110 or viewport of the visual CGR content. According to some embodiments, audio CGR content output according to the stereo audio output mode does not take into account the position and / or movement of the viewpoint of the visual CGR content within the three-dimensional environment 7106. According to some embodiments, audio CGR content output according to a stereo audio output mode does not take into account the position and / or movement of display generating components in the physical environment. According to some embodiments, audio CGR content output according to a stereo audio output mode is optionally locked to a frame of reference tied to the location of the user's head, even when the user moves relative to the display generating components, when the user's virtual position moves relative to the three-dimensional environment represented by the CGR content (e.g., causing a viewpoint movement, etc.), when window 7110 moves within the three-dimensional environment, and / or when visual embodiments of virtual sound sources move within window 7110, etc.

[0133] In some embodiments, as shown in FIG. 7A, low or intermediate levels of immersion also correspond to partial occlusion or partial pass-through of sound propagating from the physical environment (e.g., a portion of the physical environment surrounding the first user).

[0134] Figure 7B shows the same portion of visual CGR content 7106 being displayed by a display generating component (e.g., display 7100 or another type of display, such as an HMD) using a higher immersion level than that shown in Figure 7A. In some embodiments, switching between immersion levels can occur at any point selected by a user or computer system during the presentation of the visual CGR content. At this point, the CGR content 7106 is still displayed in the augmented reality environment 7102, but occupies a larger spatial extent than that shown in Figure 7A. For example, virtual objects 7106-1, 7106-2, 7106-3, and 7106-4 in the visual CGR content 7106 are displayed at spatial positions that correspond to their physical locations in the physical environment and are integrated into the representation of the physical environment. In some embodiments, additional virtual objects, e.g., virtual shadows 7106-1′, 7106-4′, 7106-3′, etc., are added in the three-dimensional environment at respective virtual positions that support virtual objects 7106-1, 7106-4, and 7106-3, etc., or that correspond to physical locations (e.g., physical surface locations) below virtual objects 7106-1, 7106-4, and 7106-3, etc. In some embodiments, according to movement of the display generating components relative to the physical environment, the computing system updates the view of the three-dimensional environment 7102 and the viewing angles and viewing distances of the virtual objects in the visual CGR content 7106 of FIG.

[0135] 7B depicts a display of CGR content 7106, optionally with an even greater level of immersion, such as a virtual reality mode without a representation of a physical environment (e.g., a 3D movie, or a gaming environment, etc.). In some embodiments, the switching performed by the computing system is in response to a request from a first user (e.g., a gesture input that meets pre-set criteria for changing the immersion level of the CGR content, or an event generated by the computer system based on the current context, etc.).

[0136] In some embodiments, as shown in Figure 7B, the computing system displays visual CGR content 7106 via display generation component 7100 (e.g., in response to a user command 7114 to display CGR content 7106 in an augmented reality mode throughout a representation of the physical environment, or in response to a transition from a less immersive mode (e.g., that shown in Figure 7A), or a transition from a more immersive mode (e.g., a virtual reality mode), etc.). In some embodiments, as shown in Figure 7B, when displaying CGR content 7106 using a higher immersion level compared to that of Figure 7A, the visual CGR content 7106 is no longer confined to a limited subportion of the field of view provided by a display generation component such as a virtual window 7110, a virtual viewport, the location of a finite physical surface, a virtual screen, or the like, but is distributed at different positions throughout different portions of the three-dimensional environment 7102. In some embodiments, other user interface objects related to and / or unrelated to CGR content (e.g., playback controls 7108, a dock with application icons, etc.) are optionally displayed simultaneously with the visual CGR content 7106 in the three-dimensional environment 7102 (e.g., in a peripheral portion of the field of view, a portion selected by the user, etc.). In some embodiments, if the visual CGR content 7106 is three-dimensional content, the viewpoint of the currently displayed view of the three-dimensional content optionally moves in response to user input and / or movement of display generating components or the user's head in the physical environment.

[0137] In this example, as shown in FIG. 7B, when displaying visual CGR content 7106 with an increased level of immersion, the computer system selects an audio output mode that corresponds to the increased level of immersion, such as a surround sound audio output mode or a spatial audio output mode that is output relative to a frame of reference that is no longer tied to the location of the audio output device(s) within the physical environment.

[0138] In this example, according to some embodiments, the audio output device is optionally movable relative to the display generating components and / or the user in the physical environment. According to some embodiments, audio CGR content output according to the spatial audio output mode takes into account the position and / or movement of virtual sound sources in the three-dimensional environment 7102. According to some embodiments, audio CGR content output according to the spatial audio output mode takes into account the position and / or movement of the viewpoint of the visual CGR content within the three-dimensional environment 7106. According to some embodiments, audio CGR content output according to the spatial audio output mode takes into account the position and / or movement of display generating components in the physical environment.

[0139] In some embodiments, a higher level of immersion also corresponds to increased shielding or decreased pass-through of sounds propagating from the physical environment (e.g., portions of the physical environment surrounding the first user).

[0140] In some embodiments, to achieve the adjustments necessary to output audio CGR content in a spatial audio output mode that account for movement in the respective environments of the display generation components, the user, the audio output device, the viewpoint, and / or virtual sound sources, while continuing to reflect the position(s) and / or movement of the sound source(s) in the respective reference frame(s) separate from the location of the audio output device(s), the computer system optionally utilizes one or more additional audio output components to output audio compared to those used in the stereo audio output mode. In some embodiments, the additional audio output components are located in different locations than those used in the stereo audio output mode. In some embodiments, the computer system dynamically selects audio output components to be activated when outputting individual portions of audio CGR content in the spatial audio output mode based on the position and movement of the virtual sound source(s) in the corresponding portion of the visual CGR content of the computer-generated experience simultaneously provided via the display generation components having a higher level of immersion. In some embodiments, the audio output components used to output audio CGR content in the spatial audio output mode are a superset of the audio output components used to output audio CGR content in the stereo audio output mode and / or the surround sound output mode. In some embodiments, the audio output components used to output audio CGR content in spatial audio output mode span a wider spatial area than the audio output components used to output audio CGR content in stereo audio output mode and / or surround sound audio output mode.

[0141] In some embodiments, the spatial audio output mode provides sound localization based on visual content, while the stereo audio output provides head-locked sound. In some embodiments, the display generating components and the audio output device are enclosed within the same head-mounted device. In some embodiments, the display generating components and the audio output device are positioned separately relative to the user's head (e.g., as eyes and ears in a physical environment separate from the user). In some embodiments, the display generating components are not fixedly positioned relative to the user's head, while the audio output device(s) are fixedly positioned relative to the user's ear(s) during presentation of CGR content. In some embodiments, the display generating components are fixedly positioned relative to the user's head during presentation of CGR content, but the audio output device(s) are not fixedly positioned relative to the user. In some embodiments, the computer system adjusts the generation of sound corresponding to the audio CGR content while the audio CGR content is being output using spatial audio output mode in response to the relative movement and spatial configuration of the display generation components, the user, and the audio output device(s) to provide sound localization based on the visual content (e.g., movement of the viewpoint, changes in virtual sound sources, movement of virtual sound sources, etc.).

[0142] In some embodiments, when providing sound localization based on the position of virtual sound sources within visual CGR content, the computer system determines the virtual positions of the individual virtual sound sources within the three-dimensional environment of the CGR content, determines an appropriate frame of reference (e.g., a physical-environment-based, virtual-environment-based, viewpoint-based, etc., frame of reference selected based on the type of CGR content being presented) for sounds corresponding to the individual virtual sound sources, determines individual positions of the virtual sound sources within the selected frame of reference based on the current positions of the individual sound sources within the three-dimensional environment of the CGR content, and controls operation of an audio output component of an audio output device(s) to output sounds corresponding to the individual sound sources such that the sounds are perceived to originate from the individual positions of the individual sound sources within the selected frame of reference within the physical environment. In the example shown in FIG. 7B , if virtual object 7106-1 is a virtual sound source (e.g., a virtual bird, a virtual train, a virtual assistant, etc.) associated with an audio output (e.g., a chirping sound, a training chugging sound, a speech sound, etc.), when the audio CGR content is being output using spatial audio output mode, the computer system optionally controls the audio output components of the output of the virtual sound source so that the sound, when perceived by the user, appears to originate from a physical location corresponding to the current virtual position of virtual object 7106-1 within three-dimensional environment 7102, regardless of movement of the display generation components, movement of the user, and / or movement of audio output device(s) within the physical environment.Similarly, in the example shown in FIG. 7B , if virtual object 7106-3 is another virtual sound source (e.g., another virtual bird, a virtual conductor, etc.) associated with another audio output (e.g., another chirp, whistle, etc.), when audio CGR content is being output using spatial audio output mode, the computer system optionally controls the audio output components of the output of this other virtual sound source so that the sound, when perceived by the user, appears to originate from a physical location corresponding to the current virtual position of virtual object 7106-3 within three-dimensional environment 7102, regardless of movement of the display generation components, movement of the user, and / or movement of audio output device(s) within the physical environment.

[0143] In some embodiments, when providing sound localization based on a user's position, the computer system determines virtual positions of individual virtual sound sources in the three-dimensional environment of the CGR content, determines a frame of reference associated with the user's location relative to the three-dimensional environment of the CGR content, determines individual positions of the virtual sound sources in the frame of reference based on the user's location, and controls operation of audio output components of audio output device(s) to output sound corresponding to the individual sound sources such that the sound is perceived to originate from the individual positions of the individual sound sources in a frame of reference fixed to the user's current location in the physical environment. In the example shown in FIG. 7B , the virtual sound sources (e.g., external narrator, virtual assistant, ambient sound sources, etc.) associated with the audio output optionally do not have corresponding virtual objects. When audio CGR content is being output using spatial audio output mode, the computer system optionally controls audio output components to output sound of the virtual sound sources such that the sound, when perceived by the user, appears to originate from a location or area fixed relative to the user, regardless of movement of display generation components, movement of the user, and / or movement of the audio output device(s) in the physical environment. According to some embodiments, the viewpoint of the visual CGR content optionally changes according to the movement of the display generating components and / or the movement of the user, while the audio output corresponding to the virtual sound source remains fixed relative to the user.

[0144] 7C-7H are block diagrams illustrating, in accordance with some embodiments, changing the appearance of a portion of virtual content when a significant physical object approaches a display generation component or a user's location (e.g., allowing a representation of a portion of a physical object to break the virtual content, changing one or more visual characteristics of the virtual content based on visual characteristics of a portion of the physical object, etc.).

[0145] In some embodiments, when displaying virtual content in a three-dimensional environment (e.g., environment 7126 of FIGS. 7C-7H , another environment, etc.) (e.g., a virtual reality environment, an augmented reality environment, etc.), all or a portion of the view of the physical environment is blocked or replaced by the virtual content (e.g., virtual objects 7128, 7130 of FIG. 7D , etc.). In some cases, it is advantageous to give display priority to a particular physical object (e.g., physical object 7122 of FIG. 7C , another physical object of importance to the user, etc.) in the physical environment (e.g., scene 105 of FIGS. 7C, 7E, and 7G ) over the virtual content, so that at least a portion of the physical object (e.g., physical object 7122 of FIG. 7C , another physical object of importance to the user, etc.) is visually represented in the view of the three-dimensional environment (e.g., as shown in FIGS. 7F and 7H ). In some embodiments, the computer system utilizes various criteria to determine whether to give an individual physical object a display priority such that a representation of the individual physical object can break through a portion of the virtual content currently displayed in the three-dimensional environment when the location of the individual physical object in the physical environment corresponds to the position of the portion of the virtual content in the three-dimensional environment. In some embodiments, the criteria include a requirement that at least a portion of the physical object has approached and entered a threshold spatial region (e.g., spatial region 7124 of FIGS. 7C, 7E, and 7G, another spatial region, etc.) surrounding a user of the display generating component (e.g., user 7002 viewing virtual content through the display generating component, a user whose view of a portion of the physical object is blocked or replaced by the display of virtual content, etc.), and an additional requirement that the computer system detects the presence of one or more characteristics related to the physical object (e.g., physical object 7122 of FIG. 7C, another physical object important to the user, etc.) that indicate increased importance of the physical object to the user.In some embodiments, the physical object of increased importance to the user may be the user's friends or family, the user's team members or supervisors, the user's pets, etc. (e.g., as shown in the examples of FIGS. 7C-7H ). In some embodiments, the physical object of increased importance to the user may be a person or object that requires the user's attention to address an emergency. In some embodiments, the physical object of greater importance to the user may be a person or object that requires the user's attention to take action to prevent the user from missing out. The criteria may be adjustable by the user based on the user's needs and desires and / or by the system based on contextual information (e.g., time, location, scheduled events, etc.). In some embodiments, giving display priority to physical objects that are more important than virtual content and visually representing at least a portion of the physical object in the view of the three-dimensional environment includes replacing the display of a portion of the virtual content (e.g., a portion of virtual object 7130 in FIG. 7F , a portion of virtual object 7128 in FIG. 7H , etc.) with a representation of a portion of the physical object or changing the appearance of the portion of the virtual content depending on the appearance of the portion of the physical object. In some embodiments, even if a position corresponding to the location of a portion of the physical object is visible within the field of view provided by the display generation component (e.g., the position is currently occupied by virtual content), at least a portion of the physical object (e.g., the ear and body of pet 7122 in FIG. 7F, part of the body of pet 7122 in FIG. 7H, etc.) is not visually represented in the view of the three-dimensional environment and remains blocked or replaced by the display of the virtual content.In some embodiments, portions of the three-dimensional environment that have been modified to indicate the presence of a physical object and portions of the three-dimensional environment that have not been modified to indicate the presence of a physical object (e.g., portions of the three-dimensional environment (e.g., virtual object 7128, portions of virtual object 7130, etc.) may continue to change based on the progression of the computer-generated experience and / or user interaction with the three-dimensional environment, etc.) correspond to positions on a continuous portion of a virtual object or surface (e.g., virtual object 7128, portions of virtual object 7130, etc.).

[0146] In some embodiments, when a user is engaging in a computer-generated experience, such as a virtual reality experience or an augmented reality experience, via display generating components, the user's view of the physical environment is blocked or obscured by the presence of virtual content in the computer-generated experience. In some embodiments, there are situations in which it is desirable to reveal or visually indicate to the user the presence of a significant physical object (e.g., a person, a pet, etc.) approaching the user's physical proximity while the user is engaging in the virtual reality experience or the augmented reality experience. In some embodiments, the significant physical object is within the user's potential field of view, but with respect to the presence of the display generating components and virtual content of the computer-generated experience (e.g., the physical object would be visible to the user in the absence of the display generating components and / or virtual content), a portion of the virtual content at a position corresponding to a first portion of the physical object is removed or altered in appearance to reflect the appearance of the first portion of the physical object, and another portion of the virtual content at a position corresponding to another portion of the physical object adjacent to the first portion of the physical object is not removed or altered to reflect the appearance of the other portion of the physical object. In other words, the virtual content is not suddenly removed or altered to show all portions of the physical object potentially within the user's field of view, but is gradually removed or altered portion by portion to facilitate disruption of the computer-generated experience.

[0147] In various embodiments, significant physical objects are identified by the computer system based on criteria that include at least one requirement unrelated to or independent of the distance between the physical object and the user. In some embodiments, the computer system considers various information when determining whether an approaching physical object is significant to the user, such as the user's previously entered preferences, the presence of previously identified characteristics, the current context, the presence of marker objects or signals associated with the physical object, and the like, to ensure the computer-generated experience is visually disruptive.

[0148] As shown in FIG. 7C , user 7002 is present in a physical environment (e.g., scene 105, or another physical environment, etc.). User 7002 is in a predetermined position relative to a display generating component (e.g., display generating component 7100, another type of display generating component such as an HMD, etc.) to view content displayed via the display generating component. A predetermined spatial region 7124 surrounding user 7002 is indicated in FIG. 7C by a dashed line surrounding user 7002. In some embodiments, the predetermined spatial region 7124 is a three-dimensional region surrounding user 7002. In some embodiments, the predetermined spatial region 7124 is defined by a predetermined threshold distance (e.g., an arm's length, 2 meters, etc.) to a characteristic location of the user (e.g., the location of the user's head, the location of the user's center of gravity, etc.) within the physical environment. In some embodiments, the predetermined spatial region 7124 has a boundary surface that is a greater distance from the front of the user (e.g., face, chest, etc.) than from the back of the user (e.g., the back of the head, etc.). In some embodiments, the preset spatial region 7124 has a boundary that is a greater distance from one side of the user than the other side of the user (e.g., a greater distance from the user's left side than the user's right side, or vice versa). In some embodiments, the preset spatial region 7124 has a boundary that is symmetrical on both sides of the user. In some embodiments, the preset spatial region 7124 is a greater distance from an upper part of the user's body (e.g., the user's head, the user's chest, etc.) than from a lower part of the user's body (e.g., the user's feet, the user's legs, etc.). In some embodiments, the display generating components have a fixed spatial relationship with the user's head. In some embodiments, the display generating components surround the user's eyes, blocking the user's view of the physical environment except for the view provided via the display generating components.

[0149] In some embodiments, as shown in FIG. 7C , the physical environment includes other physical objects (e.g., physical object 7120, physical object 7122, etc.) and physical surfaces (e.g., walls 7004, 7006, floor 7008, etc.). In some embodiments, at least some of the physical objects are stationary relative to the physical environment. In some embodiments, at least some of the physical objects move relative to the physical environment and / or the user. In the example shown in FIG. 7C , physical object 7122 represents an instance of a first type of physical object that is important to user 7002 based on an evaluation of preset criteria. Physical object 7120 represents an instance of a second type of physical object that is not important to user 7002 based on an evaluation of preset criteria. In some embodiments, the physical environment may include only one of the two types of physical objects at a given time. In some embodiments, an individual physical object of the two types of physical objects may enter the physical environment after the user 7002 has already begun the computer-generated experience, and the user does not necessarily perceive the physical object entering the physical environment due to the presence of the display generating components and / or virtual content displayed via the display generating components.

[0150] 7D shows the display generating components displaying a view of three-dimensional environment 7126 at a time corresponding to the view shown in FIG. 7C. In this example, three-dimensional environment 7126 is a virtual three-dimensional environment that does not include display generating components and a representation of the physical environment surrounding the user. In some embodiments, the virtual three-dimensional environment includes virtual objects (e.g., virtual object 7128, virtual object 7130, user interface objects, icons, avatars, etc.) and virtual surfaces (e.g., virtual surfaces 7132, 7136, and 7138, virtual windows, virtual screens, user interface background surfaces, etc.) at various positions within virtual three-dimensional environment 7126. In some embodiments, the movement of the user and / or the display generating components changes the perspective of the currently displayed view of three-dimensional environment 7126 in response to the movement of the user and / or the display generating components in the physical environment. In some embodiments, the computer system moves or changes the perspective of the currently displayed view of three-dimensional environment 7126 according to user input, a pre-programmed progression of the computer-generated experience, and / or events generated by the computer system based on preset conditions being met. In some embodiments, the virtual content (e.g., a movie, a game, etc.) changes over time as the computer-generated experience progresses, without user input.

[0151] 7D represents an augmented reality environment, in which virtual content (e.g., virtual surfaces and virtual objects) is displayed simultaneously with a representation of a physical environment (e.g., scene 105, or another physical environment surrounding the user, etc.). At least a portion of the representation of the physical environment (e.g., one or more contiguous (or consecutive) portions and / or discontinuous and disconnected portions of the physical environment) in front of the user (e.g., the portion of the physical environment that would be visible to or within the user's field of view if the display generating components were not present or were displaying the virtual content) is blocked, replaced, or obscured by the virtual content displayed by the display generating components. For example, in some embodiments, virtual surfaces 7132, 7136 are representations of walls 7006 and 7004 in physical environment 105, virtual surface 7134 is a representation of floor 7008 in physical environment 105, and virtual objects 7128 and 7130 block, replace, or overlay at least a portion of the representation of the physical environment (e.g., portions of the representations of walls 7006 and floor 7008, and representations of physical objects 7120 and 7122, etc.).

[0152] 7C and 7D , once physical objects 7122 and 7120 are both outside of a predetermined portion of space 7124 surrounding user 7002 but within the user's potential field of view without the presence of display generating component 7100, the virtual content of three-dimensional environment 7126 (e.g., virtual objects 7128 and 7130) is displayed via display generating component 7100 without the distraction of physical objects 7122 and 7120. For example, if three-dimensional environment 7126 is a virtual environment, the portions of the virtual content having respective virtual positions corresponding to the locations of physical objects 7122 and 7120 would be displayed normally in accordance with the original CGR experience, even though the positions corresponding to the locations of physical objects 7122 and 7120 are within the field of view provided by the display generating component. In another example, if three-dimensional environment 7126 is an augmented reality environment, portions of the virtual content having respective virtual positions corresponding to the locations of physical objects 7122 and 7120 are typically displayed according to the original CGR experience, even if the positions corresponding to the locations of physical objects 7122 and 7120 are within the field of view provided by the display generation components, and even if some portions of the physical environment (e.g., portions of walls, floors, portions of physical objects 7122 and 7120, etc.) are visible within a space that is not currently occupied or visually blocked by the virtual content of the CGR experience.

[0153] 7E-7F illustrate a scenario in which physical objects 7122 and 7120 have moved closer to user 7002 within physical environment 105. At this time, only a portion of the total spatial extent of physical object 7122 is within predetermined spatial region 7124 surrounding user 7002. Similarly, only a portion of the total spatial extent of physical object 7120 is within predetermined spatial region 7124 surrounding user 7002. In some embodiments, in response to detecting movement of physical objects (e.g., physical object 7120, physical object 7122, etc.) in the physical environment, and following a determination that the user is within a threshold distance of the physical objects (e.g., the threshold distance is determined based on the boundary of predetermined spatial region 7124 and the individual relative spatial relationship between the user and the physical objects, a fixed predetermined threshold distance, etc.), the computer system determines whether the physical objects are significant to the user according to predetermined criteria.

[0154] In this example, physical object 7122 meets the requirements for qualifying as a physical object of interest to user 7002, and therefore, the computer system modifies the appearance of the virtual content displayed at the position corresponding to the location of the first portion of physical object 7122 according to the appearance of the first portion of physical object 7122. As shown in FIG. 7F , the virtual content shown at the position corresponding to the location of the first portion of physical object 7122 is removed, revealing representation 7122-1′ of the first portion of physical object 7122 (e.g., a portion of the pet's head, the head of physical object 7122, etc.). In some embodiments, the visual characteristics (e.g., color, simulated refractive index, transparency, brightness, etc.) of the virtual content (e.g., in this example, a portion of virtual object 7130 in FIG. 7F ) shown at the position corresponding to the location of the first portion of physical object 7122 are modified according to the appearance of the first portion of physical object 7122. In some embodiments, as shown in FIG. 7F , virtual content at positions corresponding to the locations of some portions of the portion of physical object 7122 within the preset spatial region 7124 is not altered in the view of three-dimensional environment 7126 (e.g., the portion of virtual object 7130 around the wavy edge of representation 7122-1′ in FIG. 7F ) when those portions of the physical object (e.g., a portion of the head of physical object 7122 and a portion of the body portion of physical object 7122, as shown in FIG. 7E ) are within the user's threshold distance and the display generating component is removed, even though they are now within the user's natural field of view. In some embodiments, virtual content at positions corresponding to the locations of all portions of the portion of physical object 7122 within the preset spatial region 7124 may eventually be removed or altered in the view of three-dimensional environment 7126 after a period of time during which the portions of physical object 7122 remain within the preset spatial region 7124.

[0155] In this example, physical object 7120 does not meet the requirements to qualify as a physical object of importance to user 7002, and therefore the computer system does not change the appearance of the virtual content (e.g., virtual object 7128 in FIG. 7F ) displayed at the position corresponding to the location of the first portion of physical object 7120 according to the appearance of the first portion of physical object 7120. As shown in FIG. 7F , the virtual content shown at the position corresponding to the location of the first portion of physical object 7120 is not removed, and the first portion of physical object 7120 is not visible in the view of three-dimensional environment 7126.

[0156] In some embodiments, the contrast between the treatment of physical object 7120 and physical object 7122 is based on pre-defined criteria by which physical object 7120 and 7122 are evaluated. For example, when physical object 7122 is important, physical object 7120 has not been previously marked as important by the user. When physical object 7122 is present, physical object 7120 is not moving towards the user above a threshold speed. When physical object 7122 is present, physical object 7120 is not a person or a pet. When physical object 7122 is a person but not speaking, physical object 7120 is a person but not speaking when approaching the user. When physical object 7122 is present, physical object 7120 is not wearing a pre-defined identifier object (e.g., a wirelessly transmitted ID, an RFID tag, a collar with a color-coded tag, etc.), etc.

[0157] In the view shown in FIG. 7F , a first portion of physical object 7120 is within a threshold distance of user 7002, its corresponding position in computer-generated environment 7126 is visible to the user based on the user's field of view of the computer-generated environment, the position corresponding to the first portion of physical object 7120 is not blocked from the user's field of view by another physical object or a position corresponding to another portion of physical object 7120, and the computer system still does not change the appearance of the portion of virtual content (e.g., virtual object 7128 in FIG. 7F ) displayed at the position corresponding to the first portion of physical object 7120 because physical object 7120 does not meet the preset criteria for being a physical object of importance to user 7002. For example, a ball does not meet the preset criteria requiring the first physical object to be a person or a pet. As the ball rolls closer to the user, the computer system does not change the appearance of the virtual content displayed at the position in the computer-generated environment corresponding to the ball's location relative to the user. In contrast, when the pet approaches the user, the computer system changes the appearance of the virtual content displayed at a position corresponding to a portion of the pet that is within the user's preset distance, without changing the appearance of the virtual content displayed at a position corresponding to other portions of the pet that are not within the user's preset distance, even though the positions corresponding to other portions of the pet are also within the user's current field of view.

[0158] Figures 7G and 7H show that later, both physical objects 7120 and 7122 move closer to the user, completely entering the predetermined portion of space 7124 surrounding the user and within the user's field of view when the display generating components are removed.

[0159] 7H , the computer system modifies the appearance of virtual content (e.g., virtual object 7130 and at least a portion of virtual object 7128) displayed at a position corresponding to the location of the second portion of physical object 7122 (e.g., a portion including the first portion of physical object 7122 and an additional portion of physical object 7122 that has entered the predetermined spatial region 7124 surrounding the user) in response to the appearance of the second portion of physical object 7122 (e.g., at least a portion of the head and body of physical object 7122). As shown in FIG. 7H , the virtual content shown at the position corresponding to the location of the second portion of physical object 7122 is removed to reveal representation 7122-2′ of the second portion of physical object 7122 (e.g., a larger portion of physical object 7122 than corresponds to representation 7122-1′ shown in FIG. 7F ). In some embodiments, the visual characteristics (e.g., color, simulated refractive index, transparency, brightness, etc.) of the virtual content shown at the position corresponding to the location of the second portion of physical object 7122 are modified in response to the appearance of the second portion of physical object 7122. 7H , virtual content at positions corresponding to the locations of some portions of the portions of physical object 7122 within the preset spatial region 7124 is not altered in the view of three-dimensional environment 7126, even though those portions of the portions of the physical object are within the user's threshold distance and would be within the user's natural field of view at this point if the display generating component were removed. In some embodiments, virtual content at positions corresponding to the locations of all portions of the portions of physical object 7122 within the preset spatial region 7124 may eventually be removed or altered in the view of three-dimensional environment 7126 after a period of time during which the portions of physical object 7122 remain within the preset spatial region 7124.

[0160] In this example, physical object 7120 does not meet the requirements to qualify as a physical object of interest to user 7002, and therefore the computer system does not change the appearance of the virtual content displayed at the position corresponding to the location of the second portion of physical object 7120 according to the appearance of the second portion of physical object 7120. As shown in Figure 7H, in Figure 7H, the virtual content shown at the position corresponding to the location of the second portion of physical object 7120 is not removed, and the second first portion of physical object 7120 is not visible in the view of three-dimensional environment 7126.

[0161] In some embodiments, there is no clear structural or visual division between portions of physical object 7122 that are revealed in the view of three-dimensional environment 7126 and other portions of physical object 7122 that are not revealed in the view of the three-dimensional environment that provides the basis for different treatments applied to different portions of the first physical object. Instead, the difference is based on the fact that the portions of physical object 7122 that are revealed are within the user's threshold distance or area, while other portions of physical object 7122 are not within the user's threshold distance or area. For example, if physical object 7122 is a pet, at a given time, the portions of the physical object that are revealed by removing virtual content or changing the appearance of the virtual content include a first portion of the pet's head (e.g., nose, whiskers, part of the face, etc.), and the remaining portions of the physical object that are not revealed by removing or changing the virtual content include the pet's head (e.g., remaining portions of the face and ears, etc.) and additional portions of the torso connected to the head that are not within the user's threshold distance.

[0162] In some embodiments, the portion of the virtual content that is altered or removed to reveal the presence of a portion of the physical object 7122 within the predetermined region of space 7124 is a portion of a contiguous virtual object or surface, and other portions of the contiguous virtual object or surface continue to be displayed unchanged. For example, as shown in FIG. 7F , only a portion of the virtual object 7130 is removed or its appearance is altered to reveal the presence of a portion of the physical object 7122 at a location that is within the predetermined portion of space 7124 and has a position that corresponds to the position of a portion of the virtual object 7130.

[0163] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a first characteristic detected by a computer system that distinguishes between human and non-human physical objects. In some embodiments, the first characteristic includes pre-defined facial structure (e.g., presence and / or movement of eyes, relative location of eyes, nose, and mouth, etc.), proportion and relative position of body parts (e.g., head, body, limbs, etc.) on physical object 7122, a human voice accompanying movement of physical object 7122, movement patterns associated with human walking or running (e.g., arm swing, walking, etc.), etc. Physical object 7120 does not qualify as a physical object of importance to user 7002 because the first characteristic is not present in physical object 7120.

[0164] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a second characteristic detected by a computer system and indicative of a human voice coming from physical object 7122 as physical object 7122 is moving toward the user. In some embodiments, the second characteristic includes predetermined audio characteristics of sounds originating from the location of physical object 7122 (e.g., the presence of a voiceprint, a speech pattern of a human language, etc.), characteristics of a human voice accompanying the movement of physical object 7122, the utterance of one or more predetermined words (e.g., "Hello!", "Hey there!", "[username]", etc.). Physical object 7120 does not qualify as a physical object of importance to user 7002 because the second characteristic is not present in physical object 7120.

[0165] In some embodiments, physical object 7122 qualifies as a physical object of interest to user 7002 based on a third characteristic detected by the computer system that distinguishes animals from human and non-human physical objects. In some embodiments, the third characteristic includes pre-defined head structure (e.g., presence and / or movement of eyes, relative locations of eyes, nose, ears, whiskers, and mouth, etc.), proportions and relative positions of body parts of physical object 7122 (e.g., head, body, tail, and limbs, etc.), presence of fur, fur color and pattern, etc., detection of animal sounds versus human voices accompanying movement of physical object 7122, detection of movement patterns associated with animal walking or running (e.g., four legs on the ground, flapping wings, walking, etc.), etc. Physical object 7120 does not qualify as a physical object of interest to user 7002 because the third characteristic is not present in physical object 7120.

[0166] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a fourth characteristic that is detected by a computer system and is based on a characteristic speed of movement of physical object 7122 that exceeds a preset threshold speed. In some embodiments, the characteristic speed of movement includes a speed of movement of at least a portion of the physical object relative to the physical object or another part of the physical environment (e.g., a person waving, popping a cork to open a bottle, etc.), a speed of movement of at least a portion of the physical object towards the user, etc. Physical object 7120 does not qualify as a physical object of importance to user 7002 because its characteristic speed of movement did not meet the preset threshold speed of movement.

[0167] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a fifth characteristic of physical object 7122 that is detected by a computer system and indicates the occurrence of an event (e.g., an emergency, a danger, etc.) that requires the user's immediate attention. In some embodiments, the fifth characteristic includes a flashing light, a movement pattern (e.g., a door or window opening, closing, etc., a person waving, etc.), a vibration (e.g., a swaying sign, a curtain, a falling object, etc.), a shout, a siren, etc. Physical object 7120 does not qualify as a physical object of importance to user 7002 because the fifth characteristic is not present in physical object 7120.

[0168] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a sixth characteristic of physical object 7122 that is detected by a computer system and indicates the presence of an identifier object (e.g., RFID, badge, ultrasound tag, serial number, logo, name, etc.) on the physical object. Physical object 7120 does not qualify as a physical object of importance to user 7002 because the sixth characteristic is not present in physical object 7120.

[0169] In some embodiments, physical object 7122 qualifies as a physical object of importance to user 7002 based on a seventh characteristic of physical object 7122 that is detected by a computer system and based on a movement pattern of the physical object (e.g., a movement pattern of at least a portion of the physical object relative to the physical object or another portion of the physical environment, or a movement pattern of at least a portion of the physical object relative to the user, etc.) Physical object 7120 does not qualify as a physical object of importance to user 7002 because the seventh characteristic is not present in physical object 7120.

[0170] In some embodiments, physical object 7122 is detected by a computer system and qualifies as a physical object of importance to user 7002 based on an eighth characteristic of physical object 7122 that is based on a match (e.g., a match or correspondence that exceeds a threshold confidence value determined by a computer algorithm or artificial intelligence (e.g., facial recognition, voice recognition, speech recognition, etc.) based on detected sensor data, image data, etc.) between the physical object's recognized identity (e.g., spouse, favorite pet, boss, child, police, a conductor on a train, etc.) and a first pre-set identity (e.g., identifying something previously established as "important," "needing attention," etc.). Physical object 7120 does not qualify as a physical object of importance to user 7002 because the eighth characteristic is not present in physical object 7120.

[0171] 7I-7N are block diagrams illustrating the application of visual effects to areas within a three-dimensional environment corresponding to portions of the physical environment identified (e.g., characterized by shape, plane, and / or surface, etc.) based on a scan of the portion of the physical environment, in accordance with some embodiments.

[0172] In some embodiments, the computer system displays a representation of a three-dimensional environment including a representation of the physical environment (e.g., scene 105 of FIG. 7I , another physical environment, etc.) in response to a request to display the three-dimensional environment (e.g., in response to a user putting on a head-mounted display, in response to a user request to start an augmented reality environment, in response to a user request to end a virtual reality experience, in response to a user turning on or waking up a display generation component from a low-power state, etc.). In some embodiments, the computer system initiates a scan of the physical environment to identify objects and surfaces within the physical environment (e.g., walls 7004, 7006, floor 7008, object 7014, etc.), and optionally constructs a three-dimensional or pseudo-three-dimensional model of the physical environment based on the identified objects and surfaces within the physical environment. In some embodiments, the computer system initiates a scan of the physical environment in response to receiving a request to display the three-dimensional environment (e.g., if the physical environment has not been previously scanned and characterized by the computer system, or if a rescan is requested by the user or the system based on preset rescan criteria being met (e.g., the last scan was performed more than a threshold time ago, the physical environment has changed, etc.)). In some embodiments, the computer system initiates a scan in response to detecting a user's hand (e.g., hand 7202 in FIG. 7K ) touching a portion of the physical environment (e.g., a physical surface (e.g., the top surface of physical object 7014, the surface of wall 7006, etc.), a physical object, etc.). In some embodiments, the computer system initiates a scan in response to detecting that a user's gaze (e.g., gaze 7140 in FIG. 7J , another gaze, etc.) directed toward a position corresponding to a portion of the physical environment meets preset stability and / or duration criteria. In some embodiments, the computer system displays visual feedback (e.g., visual effect 7144 in FIGS. 7K-7L ) regarding the progress and results of the scan (e.g., identification of physical objects and surfaces in the physical environment, determination of physical and spatial properties of the physical objects and surfaces, etc.).In some embodiments, the visual feedback includes displaying a distinct visual effect (e.g., visual effect 7144) in a distinct portion of the three-dimensional environment corresponding to a portion of the physical environment touched by the user's hand (e.g., the top surface of physical object 7014) and identified based on the scan of the portion of the physical environment. In some embodiments, as shown in FIGS. 7K-7L , the visual effect (e.g., visual effect 7144) includes a representation of movement extending from and / or propagating from the distinct portion of the three-dimensional environment (e.g., a position corresponding to the touch location of hand 7202). In some embodiments, the computer system displays the visual effect in response to detecting the user's hand touching the distinct portion of the physical environment, and the three-dimensional environment is displayed after the scan of the physical environment is completed in response to a previous request to display the three-dimensional environment.

[0173] In some embodiments, when a scan of a physical environment is performed by a computer system in preparation for generating a mixed reality environment (e.g., an augmented reality environment, an augmented virtual environment, etc.), it may be useful to receive user input identifying a region of interest and / or a clearly defined surface or planar region in order to anchor the scan of the physical environment and identify objects and surfaces within the physical environment. It may also be advantageous to provide the user with visual feedback regarding the progress and results of the scan and characterization of the physical environment from a position corresponding to the location of the user's input, so that if a position does not result in a correct characterization, the user can adjust the input and resume scanning from a different location or surface in the physical environment. In some embodiments, after a physical surface is scanned and identified based on the scan, the computer system displays an animated visual effect at a position corresponding to the identified surface, the animated visual effect starting and propagating from a position corresponding to the contact location between the physical surface and the user's hand. In some embodiments, to further confirm the location of interest, the computer system requires that gaze input be detected at the position of the physical surface where the user is touching. In some embodiments, the gaze position does not have to overlap with the position corresponding to the location of the user's touch, as long as both positions are on the same extended physical surface and / or within a threshold distance of each other.

[0174] As shown in FIG. 7I, user 7002 is present in a physical environment (e.g., scene 105, or another physical environment, etc.). User 7002 is in a predetermined position relative to a display generating component (e.g., display generating component 7100, another type of display generating component such as an HMD, etc.) to view content displayed via the display generating component. In some embodiments, the display generating component has a fixed spatial relationship with the user's head. In some embodiments, the display generating component surrounds the user's eyes, blocking the user's view of the physical environment except for the view provided via the display generating component. In some embodiments, as shown in FIG. 7C, the physical environment includes physical objects (e.g., physical object 7014, and other physical objects, etc.) and physical surfaces (e.g., walls 7004 and 7006, floor 7008, etc.). The user can see different locations within the physical environment through the view of the physical environment provided via the display generating component, and the location of the user's gaze is determined by an eye tracking device, such as the eye tracking device disclosed in FIG. 6. In this example, physical object 7014 has one or more surfaces (eg, a horizontal top surface, a vertical surface, a flat surface, a curved surface, etc.).

[0175] 7J displays a view 7103 of the physical environment 105 displayed via the display generating components. According to some embodiments, the view of the physical environment includes representations of physical surfaces and objects in a portion of the physical environment from a viewpoint corresponding to the location of the display generating component 7100 in the physical environment (e.g., a location that also corresponds to the user's eyes or head if the display generating component 7100 is an HMD). In FIG. 7J, the view 7103 of the physical environment includes representations 7004′ and 7006′ of two adjacent walls (e.g., walls 7004 and 7006) in the physical environment of the user and the display generating components, a representation 7008′ of a floor 7008, and a representation 7014′ of a physical object 7014 (e.g., furniture, objects, fixtures, etc.) in the physical environment. According to some embodiments, spatial relationships between physical surfaces and physical objects in the physical environment are represented in the three-dimensional environment by spatial relationships between the representations of the physical surfaces and physical objects in the three-dimensional environment. As a user moves the display generating components relative to the physical environment, different views of the physical environment from different perspectives are displayed via the display generating components. In some embodiments, if the physical environment is unknown to the computer system, the computer system performs a scan of the environment to identify surfaces and planes and build a three-dimensional model of the physical environment. After the scan, the computer system, according to some embodiments, can define positions of virtual objects relative to the three-dimensional model, such that the virtual objects can be placed in the mixed reality environment based on the three-dimensional model having various spatial relationships to representations of physical surfaces and objects in the three-dimensional environment. For example, the virtual objects may optionally be given an upright orientation relative to the three-dimensional model and displayed in positions and / or orientations that simulate particular spatial relationships (e.g., stacked, standing, parallel, perpendicular, etc.) with the physical surfaces or representations of the objects.

[0176] In some embodiments, as shown in FIG. 7J , a computer system detects a gaze input (e.g., gaze input 7140 in this example) directed toward a portion of a representation of a physical environment within view 7013 of the three-dimensional environment. In some embodiments, the computer system displays a visual indication (e.g., visual indication 7142) at the position of the gaze. In some embodiments, the position of the gaze is determined based on the user's gaze and the focal length of the user's eyes detected by an eye-tracking device of the computer system. In some embodiments, it is difficult to highly ascertain the exact location of the user's gaze before a scan of the physical environment is completed. In some embodiments, the area occupied by representation 7014′ of the physical object can be identified by two-dimensional image segmentation before a three-dimensional scan of the physical environment is performed or completed, and the location of the gaze can be determined to be the area occupied by representation 7014′, as determined by two-dimensional segmentation.

[0177] In some embodiments, as the user moves the display generating component around the physical environment and views different surfaces or objects through the display generating component in search of a suitable position to begin scanning, the computer provides real-time feedback to indicate to the user the location of the gaze in the portion of the physical environment that is currently within the field of view provided by the display generating component.

[0178] 7K-7L , while a user's line of sight 7140 is directed at a representation 7014′ of physical object 7014, a computer system detects that the user's hand has moved within the physical environment to a first location on the top surface of physical object 7014 and maintains contact with the top surface of physical object 7014 at the first location. In response to detecting a user's hand 7202 contacting the top surface of physical object 7014 (e.g., optionally in conjunction with detecting a user's line of sight 7140 on the same surface of physical object 7014, etc.), the computer system begins scanning the physical environment from the location of the user's hand (e.g., from the location of contact between the user's hand and the top surface of physical object 7014). In some embodiments, the computer system optionally performs scans of other portions of the physical environment in addition to and in parallel with the scan at the location of the user's hand. Once portions of the surface of physical object 7014 near the contact location have been scanned and characterized (e.g., as a flat surface, a curved surface, etc.), the computer system displays visual feedback indicating the results and progress of the scan. 7K, the appearance of a portion of representation 7014′ at a position corresponding to and near the location of the user's contact with physical object 7014 is modified by a visual effect (e.g., highlighted, animated, and / or modified in color, brightness, transparency, and / or opacity, etc.). The visual effect has one or more spatial characteristics (e.g., position, orientation, surface characteristics, spatial extent, etc.) based on the results of a scan of a portion of the physical surface at or near the location of the user's contact with the physical object. For example, in this case, the computer system determines that representation 7014′ is a plane having a horizontal direction at a position corresponding to the location of the tip of user's hand 7202 based on a scan of an area near the location of the tip of the index finger (e.g., the location of contact between the user's hand 7202 and the physical object 7014). In some embodiments, the tip of the user's finger provides an anchor location for the surface scan.In some embodiments, depth data of the physical environment at the location of the tip of the user's finger is correlated with depth data of the user's fingertip, and the accuracy of the scan is improved by this additional constraint.

[0179] 7L-7M , while the user's hand 7202 maintains contact with the top surface of the physical object 7014 in the physical environment, optionally at the initial touch location on the top surface of the physical object 7014, the computer system continues to apply and display visual feedback 7144 to indicate the progress of the scan and the identification of additional portions of the physical surface connected to the initial touch location on the top surface of the physical object 7014. In FIG. 7M , the scan and identification of the top surface of the physical object 7014 is complete, and the visual effect spreads from the position corresponding to the initial touch location on the top surface of the physical object 7014 to cover the entire top surface of the representation 7014′. In some embodiments, the spreading of the visual effect 7144 ceases once the boundary of the physical surface is identified and the visual effect is applied to the representation of the entire surface. In some embodiments, the visual effect 7144 continues to spread to representations of additional portions of the physical environment scanned and characterized in the meantime. In some embodiments, the computer system detects movement of the user's hand 7202, moving the contact point to other locations on the top surface of the physical object 7014 and either initiating a new scan from the new touch location on the physical object or continuing a previous scan in parallel with the new scan. In some embodiments, as scanning continues from one or more touch locations, corresponding visual effects spread from positions corresponding to the touch locations based on the results of the scan. In some embodiments, while the line of sight 7140 is detected on the top surface of the physical object 7014, the computer system detects the user's finger moving across multiple positions along a path on the top surface of the physical object 7014, and optionally performs a scan from the location of the path and spreads the visual effect from the location of the path or area where the user's hand is touching. According to some embodiments, using depth data at more points on the top surface as a scanning constraint may allow scanning to be performed with greater accuracy and speed than from a single touch point.

[0180] 7M-7N, while displaying a visual effect at a position corresponding to the location of a user's hand touching the top surface of physical object 7014 according to a physical surface identified by a scan performed by the computer system, the computer system detects a movement of the user's hand that results in a break in contact with the top surface of physical object 7014. In response to detecting that the user's hand has left the surface of physical object 7014, the computer system stops displaying the visual effect at the position of the surface identified based on the scan, as shown in FIG. 7N. Representation 7014' is restored to its original appearance prior to the application of visual effect 7144 of FIG. 7N.

[0181] In some embodiments, after the scan is complete and physical objects and surfaces within the portion of the physical environment have been identified, if the computer system detects user contact with the physical surface (e.g., by the user's hand 7202, another hand, etc.), the computer system optionally re-displays visual effects 7144 to show spatial characteristics of the physical surface starting from a position corresponding to the location of the user's touch. In some embodiments, the visual effect is applied to the representation of the entire physical surface as soon as a touch is detected on the physical surface. In some embodiments, the visual effect gradually expands and extends across the representation of the physical surface from a position corresponding to the location of the touch.

[0182] In some embodiments, representation 7014' of physical object 7014 is provided by a camera view of the physical environment, and visual effects 7144 replace the display of at least a portion of representation 7014' of the physical object in a view of the three-dimensional environment displayed via a display generation component. In some embodiments, representation 7014' of physical object 7014 is provided by a camera view of the physical environment, and the visual effects are projected onto the surface of the physical object, obscuring a portion of the surface of the physical object in the physical environment and seen as part of the camera view of the physical environment. In some embodiments, representation 7014' of physical object is part of a view of the physical environment that is seen through a transparent or semi-transparent portion of a display generation component, and the visual effects are displayed by the display generation component in a position that obstructs the view of at least a portion of the surface of physical object 7014. In some embodiments, the representation 7014' of the physical object is part of a view of the physical environment seen through a transparent or semi-transparent portion of the display generating components, and the visual effect is projected onto the surface of the physical object 7014, obscuring a portion of the surface of the physical object 7014 in the physical environment and seen as part of the physical environment through the transparent or semi-transparent portion of the display generating components. In some embodiments, the visual effect is projected directly onto the user's retina overlaying an image of a portion of the surface of the physical object 7014 on the retina.

[0183] In some embodiments, when the user's hand 7202 touches a different part of the physical environment, such as a wall 7006 or a floor 7008, the computer system applies a visual effect to a position corresponding to a surface identified at or near the location of the user's touch on the different part of the physical environment (e.g., the visual effect is applied to a vertical surface of the wall representation 7006' or a horizontal surface of the floor representation 7008', etc.).

[0184] In some embodiments, simultaneous detection of gaze and touch input in a separate portion of the physical environment is required for the computer system to begin scanning the portion of the physical environment and / or display visual effects according to the results of the scan in the portion of the physical environment. In some embodiments, when the user's gaze is removed from the separate portion of the physical environment, the computer system stops displaying the visual effects and optionally stops continuing to scan the portion of the physical environment even though the touch of the user's hand remains detected in the separate portion of the physical environment.

[0185] In some embodiments, visual effect 7144 is an animated visual effect that causes an animated visual change in the area to which it is applied. In some embodiments, the animated visual change includes a flashing light and / or a color change that changes over time in the area within the view of the physical environment to which the visual effect is applied. In some embodiments, the area to which the visual effect is applied does not change (e.g., with respect to size, shape, and / or content displayed in the area, etc.) when the animated visual change is occurring (e.g., the visual effect affects the appearance of the area using one or more filters or modifying functions applied to the original content of the area, but the visual characteristics of the content (e.g., shape, size, type of object, etc.) remain discernible by the viewer). In some embodiments, the area in the three-dimensional environment to which the visual change is applied expands when the animated visual change is occurring.

[0186] In some embodiments, the computer system applies different visual effects to different portions of the surface touched by the user's hand. In some embodiments, the surface touched by the user's hand spans an extended region, and surface properties may differ for different portions of the extended region. In some embodiments, when the user touches a peripheral portion of the extended surface, the visual effect shows an animated movement toward a central portion of the representation of the surface, and when the user touches the central portion of the extended surface, the visual effect shows a different animated movement toward the peripheral region of the representation of the surface. In some embodiments, when different visual effects are applied to the same extended region on the surface, the visual effects appear different due to the different starting locations and propagation directions of the animated movements. In some embodiments, different visual effects are generated according to the same baseline visual effect (e.g., a gray overlay, a flashing visual effect, ripples, expanding mesh wire, etc.), and differences between the different visual effects include different animations generated according to the same baseline visual effect (e.g., a baseline expanding gray overlay with a different shaped border, a baseline flashing visual effect modified using a different spatial relationship between the virtual light source and the underlying surface, a baseline ripple modified with a different wavelength and / or origin, a baseline mesh wire pattern modified with a different starting location, etc.).

[0187] In some embodiments, after the scan is completed and surfaces in the physical environment are identified, the surfaces can be highlighted or visually displayed in a view of the physical environment. When the computer system detects contact between a user's hand and a surface that has already been scanned and characterized based on the scan, the computer system displays an animated visual effect that begins at a position on the representation of the surface corresponding to the location of the touch and propagates across the representation of the surface according to spatial characteristics of the surface determined based on the scan. In some embodiments, the animated visual effect persists as long as the contact is maintained on the surface. In some embodiments, the computer system requires that the location of the contact remain substantially stationary (e.g., have less than a threshold amount of movement within a threshold time, do not move at all, etc.) to continue displaying the animated visual effect. In some embodiments, the computer system requires that the location of the contact remain in the same extended surface (e.g., stationary or moving within an extended surface, etc.) to continue displaying the animated visual effect. In some embodiments, the computer system stops displaying the animated visual effect in response to detecting movement of the contact across the surface or movement of the hand away from the surface. In some embodiments, the computer system stops displaying the animated visual effect in response to detecting a user's hand movement away from the surface and no longer in contact with the surface. In some embodiments, the computer system stops displaying the animated visual effect in response to detecting a contact movement across the surface and / or a hand movement away from the surface and maintains the display of a static state of the visual effect. In some embodiments, the computer system stops displaying the animated visual effect in response to detecting a user's hand movement away from the surface and no longer in contact with the surface.

[0188] In some embodiments, the visual effects described herein are displayed in response to detecting a user's hand touching a portion of the physical environment during the process of generating a spatial representation of at least a portion of the physical environment, and optionally after the spatial representation of the portion of the physical environment has been generated.

[0189] In some embodiments, the display of the visual effects described herein is triggered when the computer system switches from displaying a virtual reality environment to displaying a representation of a physical environment and / or an augmented reality environment. In some embodiments, the display of the visual effects described herein is triggered when the computer system detects that the display generating components have been positioned in a spatial relationship to the user that allows the user to view the physical environment through the display generating components (e.g., when an HMD is placed on the user's head, placed in front of the user's eyes, held in front of the user's face, when the user walks or sits in front of a head-up display, when the user turns on the display generating components to see a pass-through view of the physical environment, etc.). In some embodiments, the display of the visual effects described herein is triggered when the computer system switches from displaying a virtual reality environment to displaying a representation of the physical environment and / or an augmented reality environment, optionally without requiring the user to touch a portion of the physical environment (e.g., the visual effects are displayed in response to detection of a gaze on a portion of the physical environment, optionally initiated at a default location without a user gaze, etc.). In some embodiments, the display of the visual effects described herein is optionally triggered when the computer system detects that the display generating component has been positioned in a spatial relationship to the user that allows the user to view the physical environment through the display generating component without requiring the user to touch a portion of the physical environment (e.g., the visual effect is displayed in response to detection of a line of sight over a portion of the physical environment, or is optionally initiated at a default location without the user's line of sight, etc.).

[0190] 7O-7Q are block diagrams illustrating, in some embodiments, displaying an interactive user interface object at a position in a three-dimensional environment corresponding to a first portion of the physical environment (e.g., a location on a physical surface, a location in free space within the physical environment, etc.), and selectively withdrawing display of individual sub-portions of the user interface object according to the location of a part of a user (e.g., a user's finger, hand, etc.) moving in space between the first portion of the physical environment and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment.

[0191] In some embodiments, the computer system displays interactive user interface objects (e.g., user interface object 7152, another user interface object such as a control panel, a user interface object including selectable options, an integrated control object, etc.) in a three-dimensional environment (e.g., environment 7151, or another environment, etc.). The computer system also displays a representation of a physical environment (e.g., environment 105 of FIG. 7I, another physical environment, etc.) within the three-dimensional environment, with the interactive user interface objects having individual spatial relationships to various positions within the three-dimensional environment that correspond to different locations within the physical environment. When a user interacts with the three-dimensional environment with a portion of the user's hand (e.g., hand 7202, the fingers of hand 7202, etc.), such as one or more fingers of the user's hand or the entire hand, through touch input and / or gesture input, a portion of the user (e.g., including the user's portion of the hand, the entire hand, and possibly the wrist and arm connected to the hand, etc.) can enter a spatial region between a location corresponding to the position of a user interface object (e.g., a location of a physical object or physical surface, a location in free space within the physical environment, etc.) and a location corresponding to the viewpoint of a currently displayed view of the three-dimensional environment (e.g., the location of the user's eyes, the location of a display generating component, the location of a camera capturing a view of the physical environment shown in the three-dimensional environment, etc.). Based on the spatial relationship between the location of the user's hand, the locations corresponding to the position of the user interface object, and the location corresponding to the viewpoint, the computer system determines which portions of the user interface object are visually blocked by the portion of the user and which portions of the user interface object are not visually blocked by the portion of the user when viewed by the user from the viewpoint location.The computer system then ceases displaying the respective portions of the user interface object that will be visually blocked by the user's portion (e.g., as determined by the computer system) and instead makes visible a representation of the user's portion at the position of the respective portion of the user interface object while maintaining display of another portion of the user interface object that will not be visually blocked by the user's portion (e.g., as determined by the computer system), as shown in FIG. 7P. In some embodiments, in response to detecting movement of the user's portion or movement of the viewpoint (e.g., due to movement of a display generating component, movement of a camera capturing the physical environment, movement of the user's head or torso, etc.), the computer system re-evaluates which portions of the user interface object are visually blocked by the user's portion and which portions of the user interface object are not visually blocked by the user's portion when viewed by the user from the location of the viewpoint, based on the new spatial relationship between the user's portion, the location corresponding to the viewpoint, and the location corresponding to the position of the user interface object. The computer system then ceases displaying the other portions of the user interface object that will be visually blocked by the user's portion (e.g., as determined by the computer system), allowing the portion of the user interface object that was previously ceased from display to be restored to the view of the three-dimensional environment, as shown in FIG. 7Q.

[0192] In some embodiments, when a user interacts with a user interface object (e.g., user interface object 7152, another user interface object such as a control panel, a user interface object including selectable options, an integrated control object, etc.) in an augmented reality or virtual reality environment, tactile sensations provided by the physical surfaces of the physical environment help the user better orient their spatial sense in the augmented reality or virtual reality environment, so that the user can provide more accurate input when interacting with the user interface object. In some embodiments, the physical surface can include touch sensors that provide more accurate information about the user's touch on the physical surface (e.g., touch location, touch duration, touch intensity, etc.), which enables more diverse and / or sophisticated inputs for interacting with the user interface object or portions thereof. In some embodiments, the physical surface can include surface features (e.g., bumps, buttons, textures, etc.) that help the user precisely position their gestures or touch inputs relative to the surface features, and may also have a more realistic experience interacting with user interface objects that have visual features (e.g., virtual markers, buttons, textures, etc.) that correspond to the surface features on the physical surface.

[0193] As described herein, when a user interface object is displayed at a position corresponding to a location on a physical surface having spatial characteristics corresponding to the spatial characteristics of the physical surface, the user interface object appears to overlay or extend a representation of the physical surface or a virtual surface having the spatial characteristics of the physical surface. To provide a user with a more realistic and intuitive experience when interacting with the user interface object via touch input on the physical surface, the user interface object is visually segmented into multiple parts, and when the user's hand is in a portion of the physical space between a respective portion of the physical surface and the user's eyes, at least one of the multiple parts is visually obscured by a representation of the user's hand. In other words, at least a portion of the user's hand (optionally, other parts of the user connected to the hand) may intersect the user's line of sight directed at a respective portion of the user interface object, blocking the user's view of the respective portion of the user interface object. In some embodiments, as a user's hand moves within the space between the physical surface and the user's eye, at least a portion of the user's hand (optionally, other parts of the user connected to the hand) may intersect the user's line of sight that is directed toward a different portion of the user interface object, blocking the user's view of the different portion of the user interface object, and the previously blocked portion of the user interface object may be revealed again.

[0194] In some embodiments, the physical surface includes one or more portions having spatial contours and surface textures corresponding to different types of user interface elements, such as buttons, sliders, bumps, circles, checkmarks, switches, etc. In some embodiments, individual portions of the user interface object corresponding to individual user interface elements are optionally segmented into multiple sub-portions, such that only some of the sub-portions are visually obscured by a representation of a user's hand in a view of the three-dimensional environment, but some of the sub-portions of the user interface element are not visually obscured by a representation of a user's hand in a view of the three-dimensional environment.

[0195] 7O-7Q, display generation component 7100 displays a view of three-dimensional environment 7151. In some embodiments, three-dimensional environment 7151 is a virtual three-dimensional environment including virtual objects and virtual surfaces at various spatial positions within the three-dimensional environment. In some embodiments, three-dimensional environment 7151 is an augmented reality environment including a representation of the physical environment with representations of physical objects and surfaces disposed at various positions corresponding to respective locations within the physical environment, and virtual content having positions relative to the positions of the representations of the physical objects and surfaces within the three-dimensional environment. In some embodiments, the view of the three-dimensional environment includes at least a first surface (e.g., a virtual surface or a representation of a physical surface) at a position corresponding to the location of the first physical surface and has spatial characteristics (e.g., orientation, size, shape, surface profile, surface texture, spatial extent, etc.) corresponding to the spatial characteristics (e.g., orientation, size, shape, surface profile, surface texture, spatial extent, etc.) of the first physical surface in the physical environment. In some embodiments, the physical surface is a tabletop surface, a wall surface, a display device surface, a touchpad surface, a user's lap surface, a palm surface, a surface of a prototype object with buttons and hardware affordances, etc. In this example, the top surface of physical object 7014 is used as a non-limiting example of a physical surface that the user's hand touches.

[0196] In this example, a first user interface object (e.g., virtual keyboard 7152, a control panel with one or more control affordances, a menu with selectable options, a single unitary control object, etc.) including one or more interaction portions corresponding to respective actions is displayed at a location within three-dimensional environment 7151 corresponding to the position of a first physical surface (e.g., the top surface of physical object 7014 represented by representation 7014′, the surface of the physical object at a location corresponding to the position of virtual object 7014′, etc.). Spatial characteristics of the first user interface object (e.g., virtual keyboard 7152, a control panel with one or more control affordances, a menu with selectable options, a single unitary control object, etc.) correspond to the spatial characteristics of the first physical surface. For example, if the first user interface object is planar and the first physical surface is planar, it is displayed parallel to the representation of the first physical surface. In another example, in some embodiments, the first user interface object has a surface profile that corresponds to the surface profile of the first physical surface, and positions of topological features (e.g., bumps, buttons, textures, etc.) on the first user interface object are aligned with positions that correspond to locations of the corresponding topological features on the first physical surface. In some embodiments, the first user interface object has topological features that are not present at locations on the first physical surface that correspond to positions of the topological features on the first user interface object.

[0197] 7O , a computer system displays a view of a three-dimensional environment 7151 via a display generating component 7100. According to some embodiments, the view of the three-dimensional environment 7151 includes representations of physical surfaces (e.g., representations 7004′ and 7006′ of vertical walls 7004 and 7006, representation 7008 of a horizontal floor 7008, representations of surfaces of physical objects, etc.) and objects (e.g., representation 7014′ of physical object 7014, representations of other physical objects, etc.) in a portion of the physical environment as viewed from a viewpoint corresponding to the location of the display generating component 7100 in the physical environment (e.g., a location that also corresponds to the user's eyes or head if the display generating component 7100 is an HMD). According to some embodiments, spatial relationships between physical surfaces and physical objects in the physical environment 105 are represented in the three-dimensional environment by spatial relationships between the representations of the physical surfaces and physical objects in the three-dimensional environment 7151. In some embodiments, as a user moves a display generating component relative to the physical environment, the viewpoint of the currently displayed view moves within the three-dimensional environment, resulting in different views of the three-dimensional environment 7151 from different viewpoints. In some embodiments, a computer system performs a scan of the environment to identify surfaces and planes and builds a three-dimensional model of the physical environment. According to some embodiments, the computer system defines positions of virtual objects relative to the three-dimensional model, such that the virtual objects can be placed within the three-dimensional environment in various spatial relationships relative to representations of physical surfaces and objects within the three-dimensional environment. For example, virtual objects may optionally be given an upright orientation relative to the three-dimensional environment 7151 and displayed in positions and / or orientations that simulate particular spatial relationships (e.g., stacked, standing, parallel, perpendicular, etc.) with representations of physical surfaces or objects (e.g., representation 7014' of physical object 7014, representation 7008' of floor 7008, etc.).

[0198] 7O-7Q, a first user interface object (e.g., virtual keyboard 7152 in this example) is displayed at a location corresponding to a location of a first physical surface (e.g., the top surface of physical object 7014 represented by representation 7014′, or the top surface of a physical object located at a position corresponding to the top surface of virtual object 7014′, etc.), and spatial characteristics of the first user interface object correspond to spatial characteristics of the first physical surface (e.g., being parallel to the first physical surface, conforming to the surface profile of the first physical surface, etc.). In some embodiments, the computer system moves the first user interface object in response to movement of the first physical surface in the physical environment. For example, in some embodiments, the first user interface object remains displayed in the same spatial relationship to the representation of the first physical surface in the three-dimensional environment during movement of the first physical surface in the physical environment.

[0199] In some embodiments, representation 7014' of physical object 7014 is provided by a camera view of the physical environment, and the first user interface object replaces the display of at least a portion of representation 7104' of the physical object in a view of a three-dimensional environment (e.g., environment 7151, or another augmented reality environment, etc.) displayed via a display generation component. In some embodiments, representation 7014' of the physical object is provided by a camera view of the physical environment, and the first user interface object is projected onto a surface of physical object 7014, obscuring a portion of a surface of physical object 7014 in the physical environment and seen as part of the camera view of the physical environment. In some embodiments, representation 7014' of physical object 7014 is part of a view of the physical environment that is visible through a transparent or semi-transparent portion of a display generation component, and the first user interface object is displayed by the display generation component in a position that obstructs the view of at least a portion of representation 7014' of physical object 7014. In some embodiments, the representation 7014' of the physical object 7014 is a portion of a view of the physical environment seen through a transparent or semi-transparent portion of the display generating component, and the first usage interface object is projected onto the surface of the physical object 7014, obscuring a portion of the surface of the physical object 7014 in the physical environment and seen as part of the physical environment through a transparent or semi-transparent portion of the display generating component. In some embodiments, the first user interface object is an image projected onto the user's retina and overlaying a portion of an image of the surface of the physical object 7014 on the user's retina (e.g., the image is an image of a camera view of the physical environment provided by the display generating component, or an image of a view of the physical environment through a transparent portion of the display generating component, etc.).

[0200] 7O-7Q, before the user interacts with a first user interface object (e.g., virtual keyboard 7152 in this example), the first user interface object is not visually obscured by representation 7202′ of the user's hand in the view of three-dimensional environment 7151. In some embodiments, representation 7202′ of the user's hand may be visually obscured by the presence of other user interface objects (e.g., text entry window 7150, or other user interface objects, etc.) depending on the spatial relationship between the positions of the other user interface objects and positions corresponding to the location of the user's hand 7202 (e.g., the position of representation 7202′ of the user's hand 7202 in the three-dimensional environment). For example, a portion of representation 7202′ of hand 7202 is blocked by text entry box 7150 in FIG. 7O because the virtual position of representation 7202′ of user's hand 7202 is farther from the viewpoint of the currently displayed view of environment 7151 than text entry window 7150 along the user's line of sight within environment 7151. In some embodiments, the representation 7202' of the user's hand 7202 is part of a camera view of the physical environment. In some embodiments, the representation 7202' of the user's hand 7202 is a view of the hand through a transparent portion of a display generation component. In some embodiments, the representation of the user's hand is a stylized representation created based on real-time data about the shape and location of the hand in the physical environment.

[0201] In FIG. 7P , a user's hand 7202 moves toward a first physical surface in a physical environment (e.g., in this example, the top surface of physical object 7014 represented by representation 7014′). In some embodiments, a portion of hand 7202, such as one or more fingers of the hand (e.g., index finger, thumb, index finger, and middle finger, etc.), contacts the first physical surface at a first location on the first physical surface. In some embodiments, the first location on the first physical surface corresponds to a first position on a first user interface object, and the first position on the first user interface object corresponds to a first action associated with the first user interface object. In this particular example, the first location on the first physical surface corresponds to the position of the character key “I” (e.g., key 7154 in this example) on virtual keyboard 7152, and the first action associated with the first user interface object is entering the text character “I” (e.g., character 7156 in this example) into text entry window 7150. In some embodiments, the first user interface object is a control panel, the first location on the first physical surface corresponds to the position of a first control object (e.g., a button, slider, switch, checkbox, etc.) in the first user interface object, and the first action associated with the first user interface object is an action associated with the first control object, such as turning a device or function on or off, adjusting a value of a control function, selecting a parameter of a function or setting, etc. When contact of the user's hand 7202 is detected at the first location on the first physical surface, the computer system identifies a corresponding control object on the first user interface object, performs the first action, and, optionally, updates the appearance of the first control object and / or the environment 7151 to indicate that the first action has been performed.

[0202] In some embodiments, the computer system may measure a distance between the first physical surface and the user's hand (e.g., one or more fingers of the user's hand) based on a number of characteristics, such as shape (e.g., round, elongated, etc.), size (e.g., small, large, etc.), duration (e.g., less than a threshold duration for a tap input, greater than a threshold duration for a long tap input, greater than a threshold duration for a touch-hold input without lift-off, etc.), movement direction (e.g., up, down, left, right, clockwise, counterclockwise, etc.), movement distance (e.g., The computer system may determine characteristics of the contact, such as, for example, less than a threshold amount of movement within a threshold time, more than a threshold amount of movement within a threshold time, more than a threshold amount of translation, more than a threshold amount of rotation, etc.), path of movement (e.g., straight path, curved path, zigzag path, intersecting a threshold position / angle, not intersecting a threshold position / angle, etc.), contact intensity (e.g., above a threshold intensity, below a threshold intensity, etc.), number of contacts (e.g., single contact, two contacts, etc.), repeated contact repetitions (e.g., double tap, triple tap, etc.), etc.), and combinations of two or more of the above. Based on the characteristics of the contact, the computer system determines which of a plurality of operations associated with the first user interface object and / or the first position within the first user interface object should be performed. In some embodiments, the computer system evaluates the contact against various preset criteria, and, pursuant to a determination that the preset criteria corresponding to the respective operation are satisfied by the contact, the computer system performs the respective operation (e.g., regardless of the characteristics of the contact (e.g., starting an experience, turning a function on / off, etc.), according to the characteristics of the contact (e.g., adjusting a value, performing a sequential operation with adjustable parameters, etc.), etc.).

[0203] In some embodiments, as shown in FIG. 7P, while the user's hand is within a spatial region in the physical environment between a location corresponding to the viewpoint position (e.g., the location of a display generating component, the location of the user's eyes, the location of a camera capturing the current view of the user's hand and the physical environment shown in the view of environment 7151, etc.) and the first physical surface, the computer system ceases to display or stop displaying a first portion of the first user interface object while maintaining the display of a second portion of the first user interface object, such that a portion of the user's hand appears in the view of three-dimensional environment 7151 at the position of the first portion of the first user interface object. For example, as shown in FIG. 7P , a first portion of virtual keyboard 7152 (e.g., part of key 7154, parts of the two keys directly above key 7154, and parts of the top two keys above key 7154, etc.) that is at a location corresponding to a location behind user's hands 7020 relative to a position corresponding to the viewpoint (e.g., the location of a display generating component, the user's eyes, a camera of a computer system, etc.) is not displayed in the view of three-dimensional environment 7151, while other portions of virtual keyboard 7152 that are not behind the location of user's hands 7020 continue to be displayed in the view of three-dimensional environment 7151. In some embodiments, a portion of the user's hand that is not in contact with the first physical surface can enter a spatial region in the physical environment between a location corresponding to the viewpoint position (e.g., the location of a display generating component, the location of the user's eyes, the location of a camera capturing the physical environment shown in the current view of the user's hands and the view of environment 7151, etc.) and the first physical surface, and the computer system withdraws or ceases to display a portion of the first user interface object at a position corresponding to a location that is visually blocked by the portion of the user's hand when viewed from the location corresponding to the current viewpoint of the three-dimensional environment. For example, any portion of a key in a position within virtual keyboard 7152 behind the location of the user's thumb relative to the viewpoint location is not displayed.In FIG. 7P , the position of text entry window 7150 is in front of the position corresponding to the location of user's hand 7202, so that text entry window 7150 appears in the view of three-dimensional environment 7151, blocking the view of a portion of representation 7202′ of user's hand 7202 (or replacing the view of a portion of representation 7202′ of hand).

[0204] 7Q shows that while displaying a view of three-dimensional environment 7151, a computer system detects a movement of a user's hand in the physical environment. For example, the movement includes being lifted from a first location on a first physical surface of a physical object and moved to another location on the first physical surface of the physical object. In some embodiments, a portion of hand 7202, such as one or more fingers of the hand (e.g., index finger, thumb, index finger, and middle finger, etc.), contacts the first physical surface at a second location on the first physical surface. In some embodiments, the second location on the first physical surface corresponds to a second position on a first user interface object, and the second position on the first user interface object corresponds to a second action associated with the first user interface object. In this particular example, the second location on the first physical surface corresponds to the position of the letter "p" key (e.g., key 7160 in this example) on virtual keyboard 7152, and the second action associated with the first user interface object is entering the text letter "p" (e.g., character 7158 in this example) into text entry window 7150. In some embodiments, the first user interface object is a control panel, the second location on the first physical surface corresponds to the position of a second control object (e.g., a button, slider, switch, checkbox, etc.) within the first user interface object, and the second action associated with the first user interface object is an action associated with the second control object, such as turning a device or function on or off, adjusting a value of a control function, selecting a parameter of a function or setting, etc. When a user's hand contact is detected at the second location, the computer system identifies the corresponding control object on the first user interface object, performs the second action, and, optionally, updates the appearance of the second control object and / or environment 7151 to indicate that the second action has been performed.In some embodiments, the computer system may measure a distance between the first physical surface and the user's hand (e.g., one or more fingers of the user's hand) based on a number of characteristics, such as shape (e.g., round, elongated, etc.), size (e.g., small, large, etc.), duration (e.g., less than a threshold duration for a tap input, greater than a threshold duration for a long tap input, greater than a threshold duration for a touch-hold input without lift-off, etc.), movement direction (e.g., up, down, left, right, clockwise, counterclockwise, etc.), movement distance (e.g., The computer system may determine characteristics of the contact, such as, for example, less than a threshold amount of movement within a threshold time, more than a threshold amount of movement within a threshold time, more than a threshold amount of translation, more than a threshold amount of rotation, etc.), path of movement (e.g., straight path, curved path, zigzag path, intersecting a threshold position / angle, not intersecting a threshold position / angle, etc.), contact intensity (e.g., above a threshold intensity, below a threshold intensity, etc.), number of contacts (e.g., single contact, two contacts, etc.), repeated contacts (e.g., double tap, triple tap, etc.), etc.), and combinations of two or more of the above. Based on the characteristics of the contact, the computer system determines which of a plurality of operations associated with the first user interface object and / or the second position of the first user interface object should be performed. In some embodiments, the computer system evaluates the contact against various preset criteria, and, pursuant to a determination that the preset criteria corresponding to the respective operation are satisfied by the contact, the computer system performs the respective operation (e.g., regardless of the characteristics of the contact, according to the characteristics of the contact, etc.).

[0205] In some embodiments, as shown in FIG. 7Q, while the user's hand 7202 is in a spatial region within the physical environment between a location corresponding to the viewpoint position (e.g., the location of a display generating component, the location of the user's eyes, the location of a camera capturing the current view of the user's hand and the physical environment shown in the view of environment 7151, etc.) and the first physical surface, the computer system removes or stops displaying a third portion of the first user interface object while maintaining the display of a fourth portion of the first user interface object, such that a portion of the user's hand appears in the view of three-dimensional environment 7151 at the position of the third portion of the first user interface object. For example, as shown in Figure 7Q, a third portion of virtual keyboard 7152 (e.g., part of key 7160, parts of the two keys directly above key 7160, and parts of the top two keys above key 7160) that is at a location corresponding to a location behind user's hand 7202 relative to a position corresponding to the viewpoint (e.g., the location of a display generating component, the user's eyes, a camera of a computer system, etc.) is not displayed in the view of three-dimensional environment 7151, while other portions of virtual keyboard 7152 that are not behind the location of user's hand 7202 continue to be displayed in the view of three-dimensional environment 7151. In Figure 7Q, because the position corresponding to the location of user's hand 7202 is no longer behind the position of text entry window 7150, text entry window 7150, which is displayed in the view of three-dimensional environment 7151, no longer blocks the view of representation 7202' of user's hand (or replaces the display of a portion of representation 7202' of hand). As shown in FIG. 7Q, a first portion of virtual keyboard 7152 (e.g., key 7154, the two keys above key 7154, etc.) that was previously obscured by the presence of representation 7202' of hand 7202 is no longer obscured and is once again displayed in view of three-dimensional environment 7151.

[0206] In some embodiments, the first user interface object is a single user interface object, e.g., a single button, a single checkbox, a single selectable option, etc., and a predetermined user input detected at a first location, a second location, or a third location on the first physical surface causes the computer system to perform the same operation associated with the first user interface object, the first, second, and third locations corresponding to a first portion, a second portion, and a third portion, respectively, of the single user interface object. In some embodiments, in response to a location of the user's hand in the physical environment, the computer system selectively ceases displaying a respective one of the first portion, the second portion, or the third portion of the integrated user interface object based on a determination that the user's hand is between the location of the viewpoint and the location of the user's hand in the physical environment.

[0207] In some embodiments, there are multiple user interface objects displayed at positions in the three-dimensional environment 7151 that correspond to different positions in the physical environment, and the presence of a user's hand in a portion of the space in the physical environment between the location of the viewpoint and a location corresponding to the location of the different user interface objects causes the computer system to segment the multiple user interface objects and selectively cease displaying portions of each of the multiple user interface objects having positions that correspond to locations blocked by the presence of the user's hand when viewed from the location corresponding to the current viewpoint in the three-dimensional environment 7151. In some embodiments, even if the expression of the user's hand simultaneously causes portions of both the first user interface object and the second user interface object to be removed from view of the three-dimensional environment, the user's hand interacts with the first user interface object and does not activate the second user interface object in the same view of the three-dimensional environment. For example, in FIGS. 7P and 7Q, even though the computer system ceases displaying portions of multiple keys on the virtual keyboard, only the keys at positions that correspond to the location of the user's touch or the location of a particular part of the user's hand (e.g., the tip of the index finger, the tip of the thumb, etc.) are activated.

[0208] In some embodiments, the computer system determines the shape and position of a simulated shadow of a representation of the user's hand 7202 in the view of the three-dimensional environment 7151, for example, based on the shape and position of the user's hand and the location of a virtual light source in the three-dimensional environment. The computer system displays the simulated shadow at a position on the surface of the first user interface object, optionally by changing the appearance of a portion of the first user interface object at that position or by replacing the display of a portion of the first user interface object at that position.

[0209] In some embodiments, input gestures used in various examples and embodiments described herein (e.g., with respect to Figures 7A-7Q and 8-11) optionally include discrete, small movement gestures performed by moving a user's finger(s) relative to other finger(s) or part(s) of the user's hand, optionally without requiring the user's entire hand or arm to move significantly away from their natural location(s) and posture(s) to perform an action just before or during the gesture, in order to interact with a virtual or mixed reality environment, according to some embodiments.

[0210] In some embodiments, the input gesture is detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1 , image sensor 314 of FIG. 3 ). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of, or provide data to, a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1 , 3, and 4 (e.g., a touchscreen display that functions as both a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the device's display. In some embodiments, the input gesture is detected by a sensor system of a head-mounted system (e.g., a VR headset that includes a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head-mounted system are mounted on the front and / or bottom of the head-mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head-mounted system is used (e.g., arrayed around the head-mounted system at various locations in a room) such that the imaging sensors capture images of the head-mounted system and / or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (e.g., a head-up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, the one or more imaging sensors are mounted on an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array).For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and / or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and / or sound emitters (e.g., ultrasonic emitters). For example, light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200) while images of the hand under the light illumination are captured by one or more cameras, and the captured images are analyzed to determine the position and / or configuration of the hand. By determining input gestures using signals from image sensors directed at the hand, as opposed to using signals of a touch-sensitive surface or other direct-contact or proximity-based mechanism, a user can freely choose to perform large movements or remain relatively still when providing input gestures with their hand without experiencing constraints imposed by a particular input device or input area.

[0211] In some embodiments, the tap input optionally indicates a thumb tap input on the index finger of the user's hand (e.g., on the side of the index finger adjacent to the thumb). In some embodiments, the tap input is detected without having to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected according to a determination that a downward movement of the thumb is followed by an upward movement of the thumb and the thumb is in contact with the side of the index finger for less than a threshold time. In some embodiments, the tap hold input is detected according to a determination that the thumb moves from an up position to a touch down position and remains in the touch down position for at least a first threshold time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some embodiments, the computer system requires that the entire hand remain substantially stationary in a location for at least a first threshold time to detect a thumb tap hold input with the thumb on the index finger. In some embodiments, the touch hold input is detected without requiring the hand to remain substantially stationary (e.g., the entire hand can move while the thumb is resting on the side of the index finger). In some embodiments, a taphole drag input is detected when the thumb touches the side of the index finger and the whole hand moves while the thumb remains stationary on the side of the index finger.

[0212] In some embodiments, the flick gesture optionally indicates a push or flick input of the thumb moving across the index finger (e.g., from the palm side of the index finger to the back side). In some embodiments, the extension movement of the thumb is accompanied by an upward movement away from the side of the index finger, e.g., as in an upward flick input by the thumb. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves forward and upward. In some embodiments, a reverse flick input is performed by the thumb moving from an extended position to a retracted position. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves backward and downward.

[0213] In some embodiments, the swipe gesture is a swipe input, optionally by movement of the thumb along the index finger (e.g., along the side of the index finger adjacent to the thumb or along the side of the palm). In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or a bent state. In some embodiments, the index finger moves between an extended state and a bent state during movement of the thumb in the swipe input gesture.

[0214] In some embodiments, different phalanges of various fingers correspond to different inputs. Thumb tap inputs across various phalanges of various fingers (e.g., index, middle, ring, and optionally pinky) are optionally mapped to different actions. Similarly, in some embodiments, different push or click inputs can be performed by the thumb across different fingers and / or different portions of the fingers to trigger different actions on individual user interface contacts. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and / or in different directions (e.g., towards the distal or proximal end of the finger) trigger different actions in the respective user interface contexts.

[0215] In some embodiments, the computer system processes tap inputs, flick inputs, and swipe inputs as different types of inputs based on the type of thumb movement. In some embodiments, the computer system processes inputs having different finger locations tapped, touched, or swiped by the thumb as different sub-input types (e.g., proximal, intermediate, distal subtypes, or index, middle, ring, or pinky subtypes) of a given input type (e.g., tap input type, flick input type, swipe input type, etc.). In some embodiments, the amount of movement performed by the moving finger (e.g., thumb) and / or other movement measures associated with the finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively affect the action triggered by the finger input.

[0216] In some embodiments, the computer system recognizes combination input types that combine a series of thumb movements, such as a tap-swipe input (e.g., the thumb touching down on another finger and then swiping along the side of the finger), a tap-flick input (e.g., the thumb touching down on another finger and then flicking across the finger from the side of the palm to the back of the finger), and a double-tap input (e.g., two consecutive taps on the side of the finger in approximately the same location).

[0217] In some embodiments, the gesture input is performed with the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move toward each other to perform a pinch gesture). In some embodiments, a wrist movement (e.g., a horizontal or vertical wrist flick) is performed immediately before, immediately after (e.g., within a threshold time), or simultaneously with the finger movement input to trigger an additional, different, or modified action in the current user interface context compared to a finger movement input without the modified wrist movement input. In some embodiments, a finger input gesture performed with a user's palm facing the user's face is treated as a different type of gesture than a finger input gesture performed with a user's palm facing away from the user's face. For example, a tap gesture performed with a user's palm facing the user performs an action with added (or reduced) privacy protection compared to an action (e.g., the same action) performed in response to a tap gesture performed with a user's palm facing away from the user's face.

[0218] While one type of finger input may be used to trigger an action type in the examples provided in this disclosure, in other embodiments, other types of finger input are optionally used to trigger the same type of action.

[0219] Additional explanation regarding Figures 7A-7Q is provided below with reference to methods 8000, 9000, 10000, and 11000 described with respect to Figures 8-11 below.

[0220] FIG. 8 is a flowchart of a method 8000 for selecting different audio output modes depending on the immersion level at which computer-generated content is presented, according to some embodiments.

[0221] In some embodiments, method 8000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ). The display generating components (e.g., display generating components 120 of FIGS. 1 , 3 , and 4 ) include (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from a user's hand or a user's head. In some embodiments, method 8000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 8000 are optionally combined and / or the order of some operations is optionally changed.

[0222] In some embodiments, method 8000 is performed on a computer system (e.g., computer system 101 of FIG. 1 ) in communication with a first display generating component (e.g., display generating component 120, display generating component 7100, etc. of FIGS. 1 , 3, and 4 ) (e.g., a head-up display, HMD, display, touchscreen, projector, etc.), one or more audio output devices (e.g., earphones, speakers located in the physical environment, speakers in the same housing, or speakers attached to the same support structure as the first display generating component (e.g., built-in speakers of an HMD, etc.)), and one or more input devices (e.g., a camera, controller, touch-sensitive surface, joystick, button, glove, watch, motion sensor, orientation sensor, etc.). In some embodiments, the first display generating component is a user-facing display component that provides the CGR experience to the user. In some embodiments, the computer system is an integrated device having one or more processors and memory enclosed in the same housing as at least some of the first display generating component, the one or more audio output devices, and the one or more input devices. In some embodiments, a computer system includes a computing component (e.g., a server, a mobile electronic device such as a smartphone or tablet device, a wearable device such as a watch, wristband or earphones, a desktop computer, a laptop computer, etc.) that includes one or more processors and memory separate from one or more of the display generating components (e.g., a heads-up display, a touch screen, a standalone display, etc.), one or more output devices (e.g., earphones, external speakers, etc.), and one or more input devices. In some embodiments, the display generating components and the one or more audio output devices are integrated within and enclosed in the same housing.

[0223] In method 8000, a computer system displays (8002) a three-dimensional computer-generated environment (e.g., environment 7102 of Figures 7A-7B, or another three-dimensional environment) via a first display generation component (e.g., displaying the three-dimensional computer-generated environment includes displaying a three-dimensional virtual environment, a three-dimensional augmented reality environment, a pass-through view of a physical environment having a corresponding computer-generated three-dimensional model that corresponds to spatial characteristics of the physical environment, etc.). While displaying the three-dimensional computer-generated environment, the computer system detects (8004) a first event corresponding to a request to present first computer-generated content (e.g., detecting user input selecting and / or activating an icon corresponding to the first computer-generated content, detecting a trigger condition for initiating the first computer-generated content being met by user action or other internal event of the computer system, etc.), where the first computer-generated content includes first visual content (e.g., video content, game content, animation, user interface, movie, etc.) and first audio content (e.g., sound effects, soundtrack, audio recording, movie soundtrack, game soundtrack, etc.) corresponding to the first visual content (e.g., video content and associated audio data, with timing data associating different portions of the video content with different portions of the audio data (e.g., a video playback timeline and an audio playback timeline are temporally correlated by the timing data)). For example, the first computer-generated content includes first visual content 7106 of FIGS. 7A-7B.In response to detecting a first event (8006) corresponding to a request to present first computer-generated content, in accordance with a determination that the first event corresponds to a distinct request to present first computer-generated content having a first immersion level (e.g., a middle level of immersion of several available immersion levels, a lowest level of immersion of two or more available immersion levels, a lower level of immersion of two or more available immersion levels, etc.) that occupies a first portion of the three-dimensional computer-generated environment (e.g., playing video content in a window that occupies a portion of the user's field of view for the three-dimensional computer-generated environment, a current view of the three-dimensional computer-generated environment, the three-dimensional computer-generated environment spanning a range greater than a preset threshold angle from the viewpoint), and playing video content having a field of view that extends less than a predetermined threshold angle within the three-dimensional computer-generated environment from a viewpoint corresponding to the view), the computer system displays (8008) first visual content within a first portion of the three-dimensional environment (e.g., optionally simultaneously with other virtual content and / or representations of the physical environment occupying other portions of the three-dimensional computer-generated environment, etc.), and the computer system outputs the first audio content using a first audio output mode (e.g., a stereo audio mode, a surround sound mode, etc.) (e.g., a least immersive audio output mode of several available audio output modes for the first audio content, an audio mode with a medium level of immersion of several available audio output modes for the first audio content, a less immersive audio output mode of several available audio output modes for the first audio content, etc.).In response to detecting a first event corresponding to a request to present the first computer-generated content (8006), in accordance with a determination that the first event corresponds to a distinct request to present the first computer-generated content at a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional computer-generated environment that is larger than the first portion of the three-dimensional environment (e.g., instead of occupying a two-dimensional window within the three-dimensional environment, the display of the content occupies a span of three-dimensional space that is larger than the window; instead of spanning a portion of the three-dimensional environment, the visual content spans the entire three-dimensional environment, etc.), the computer system displays (8010) the first visual content within the second portion of the three-dimensional environment (e.g., optionally simultaneously with other virtual content and / or representations of the physical environment that occupy other portions of the three-dimensional environment, etc.), and the computer system displays (8011) the first visual content within the second portion of the three-dimensional environment (e.g., optionally simultaneously with other virtual content and / or representations of the physical environment that occupy other portions of the three-dimensional environment, etc.), in accordance with a determination that the first event corresponds to a distinct request to present the first computer-generated content at a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional environment that is larger than the first portion of the three-dimensional environment (e.g., instead of spanning a two-dimensional window within the three-dimensional environment, the display of the content occupies a span of three-dimensional space that is larger than the window; instead of spanning a portion of the three-dimensional environment, the visual content spans the entire three-dimensional environment, etc.), in accordance with a determination that the first event corresponds to a distinct request to present the first computer-generated content at the second immersion level different from the first immersion level. outputting the first audio content using a second audio output mode (e.g., a surround sound mode, a spatial audio mode with sound image positioning based on locations of virtual sound sources within the first computer-generated content, etc.) (e.g., a more immersive audio output mode of several available audio output modes for the first audio content, an audio mode with the highest level of immersion of several available audio output modes for the first audio content, the most immersive audio output mode of several available audio output modes for the first audio content, etc.), and using the second audio output mode instead of the first audio output mode (e.g., automatically without requiring user input) changes the immersion level of the first audio content (e.g., making the first audio content more or less immersive, more or less spatially extended, have more or less complex spatial variations, be more or less directionally adjustable based on corresponding visual content, etc.).This is illustrated in Figures 7A-7B, where Figure 7A shows the display of computer-generated content 7106 using a first immersion level and Figure 7B shows the display of computer-generated content 7106 using a second immersion level, where the computer-generated content displayed at the first immersion level has a smaller spatial extent than the computer-generated content displayed at the second immersion level, and the computer system selects different audio output modes for outputting audio content of the computer-generated content based on the immersion level at which the computer-generated content is displayed by the display generation component.

[0224] In some embodiments, outputting the first audio content using the first audio output mode includes outputting the first audio content using a first set of audio sources respectively located at a first set of locations in the physical environment (e.g., two audio source outputs located on either side of the HMD, a single audio source located in front of the user, etc.), and outputting the first audio content using the second audio output mode includes outputting the first audio content using a second set of audio sources respectively located at a second set of locations in the physical environment, the second set of audio sources being different from the first set of audio sources. In some embodiments, the first set of audio sources and the second set of audio sources are contained within the same housing (e.g., the housing of the HMD, the same speaker or soundbar housing, etc.). In some embodiments, the first set of sound sources and the second set of sound sources are housed in different housings (e.g., the first set of sound sources are surrounded by an HMD or earphones, and the second set of sound sources are surrounded by a set of external speakers positioned at various positions in the physical environment surrounding the user; the first set of sound sources are surrounded by a pair of speakers positioned in the physical environment around the user, and the second set of sound sources are surrounded by a set of three or more speakers positioned in the physical environment around the user, etc.). In some embodiments, the sound sources in the first set of sound sources and the second set of sound sources refer to elements of physical vibration that generate sound waves and propagate away from the location of the elements of vibration. In some embodiments, the physical vibration characteristics of the individual sound sources (e.g., wavefront shape, phase, amplitude, frequency, etc.) are controlled by a computer system according to the audio content output by the output device. In some embodiments, each or individual subset of the sound sources in the first set of sound sources and / or the second set of sound sources have the same characteristics but different locations. In some embodiments, individual ones or individual subsets of the sources within the first set of sources and / or the second set of sources have different characteristics and the same location.In some embodiments, individual ones or individual subsets of the sound sources in the first set of sound sources and / or the second set of sound sources have different characteristics and different locations. In some embodiments, the different characteristics of individual sound sources or different subsets of sound sources in the first set of sound sources and the second set of sound sources are individually controlled by a computer system based on the currently displayed portion of the first visual content and corresponding audio content. In some embodiments, the sound sources in the first set of sound sources are not individually controlled (e.g., the sound sources have the same phase, the same amplitude, the same wavefront shape, etc.). In some embodiments, the sound sources in the second set of sound sources are individually controlled based on spatial relationships between objects and actions of virtual objects within the currently displayed portion of the first visual content (e.g., having different relative phases, different propagation directions, different amplitudes, different frequencies, etc.), such that the sounds obtained at different locations in the physical environment are dynamically adjusted based on changes in the currently displayed portion of the first visual content (e.g., changes in spatial relationships between objects within the currently displayed portion of the first visual content, different user interactions with different virtual objects or different portions of virtual objects within the currently displayed portion of the first visual content, different types of events occurring in the currently displayed portion of the first visual content, etc.).

[0225] Outputting first audio content using a first set of audio sources respectively disposed at a first set of locations in the physical environment and outputting second audio content using a second set of audio sources different from the first set of audio sources respectively disposed at a second set of locations in the physical environment provides improved audio feedback to the user (e.g., improved audio feedback regarding a current level of immersion). Providing improved feedback improves usability of the device and, in addition, reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

[0226] In some embodiments, the second set of sound sources includes the first set of sound sources and one or more additional sound sources not included in the first set of sound sources. In some embodiments, when the first visual content is displayed at a lower level of immersion and / or with a smaller spatial extent (e.g., within a window or fixed frame), a smaller subset of the sound sources (e.g., one or two sound sources, one or two sets of sound sources located in one or two locations, sound sources used to generate single-channel or stereo sound, etc.) within the audio output device(s) associated with the computer system are used to output the first audio content. When the first visual content is displayed at a higher level of immersion and / or with a larger spatial extent (e.g., without a fixed window or fixed frame, spanning a three-dimensional space surrounding the user, etc.), a larger subset or all of the available sound sources (e.g., three or more sound sources for generating surround sound and / or spatially arranged sound, etc.) within the audio output device(s) associated with the computer system are used to output the first audio content. Outputting first audio content using a first set of audio sources each disposed at a first set of locations within the physical environment and outputting second audio content using a second set of audio sources each disposed at a second set of locations within the physical environment, the second set of audio sources including the first set of audio sources and one or more additional audio sources not included in the first set of audio sources, provides improved audio feedback to the user (e.g., improved audio feedback regarding a current level of immersion). Providing improved feedback improves usability of the device and, in addition, reduces power usage and improves battery life of the device by allowing the user to use the device more quickly and efficiently.

[0227] In some embodiments, the second set of locations spans a larger area than the first set of locations in the physical environment. In some embodiments, the first set of locations is located to the left and right of the user, or in front of the user. The second set of locations is located at three or more locations around the user (e.g., front, left, right, back, above, below, and / or optionally at other angles relative to the user's forward-facing direction in three-dimensional space). Outputting the first audio content using a first set of sound sources respectively located at the first set of locations in the physical environment and outputting the second audio content using a second set of sound sources different from the first set of sound sources respectively located at a second set of locations in the physical environment that spans a larger area than the first set of locations in the physical environment provides the user with improved audio feedback (e.g., improved audio feedback regarding a current level of immersion). Providing improved feedback improves device usability and, in addition, reduces power usage and improves device battery life by allowing the user to use the device more quickly and efficiently.

[0228] In some embodiments, outputting the first audio content using the first audio output mode includes outputting the first audio content according to a pre-established correspondence between the first audio content and the first visual content (e.g., a temporal correspondence between an audio playback timeline and a video playback timeline, a pre-established content-based correspondence (e.g., sound effects associated with individual objects, alerts associated with individual user interface events, etc.)), where the pre-established correspondence is independent of the respective spatial locations of the virtual objects within the currently displayed view of the first visual content (e.g., the spatial locations of the virtual objects within the currently displayed view of the first visual content are, optionally, independent of the respective spatial locations of the virtual objects within the currently displayed view of the first visual content). outputting the first audio content using the second audio output mode includes outputting the first audio content according to a predetermined correspondence between the first audio content and the first visual content (e.g., a temporal correspondence between an audio playback timeline and a video playback timeline, a pre-established content-based correspondence (e.g., sound effects associated with individual objects, alerts associated with individual user interface events, etc.)), and according to the spatial location of each of the virtual objects within the currently displayed view of the first visual content. For example, in some embodiments, when the first audio output mode is used to output the first audio content, sounds generated by the audio output device(s) are independent of the spatial relationship between the user's viewpoints corresponding to the currently displayed view of the first visual content. In some embodiments, when the first audio output mode is used to output the first audio content, the sounds produced by the audio output device(s) are independent of spatial relationships between virtual objects in the currently displayed view of the first visual content.In some embodiments, when the first audio output mode is used to output the first audio content, the sounds produced by the audio output device(s) are independent of changes in spatial relationships between virtual objects in the currently displayed view of the first visual content caused by user input (e.g., when a virtual object that is a perceived producer of sound in the first visual content is moved by the user (e.g., in a user interface, a game, a virtual environment, etc.)). In some embodiments, when the first audio output mode is used to output the first audio content, the sounds produced by the audio output device(s) are head-locked to the user's head (e.g., when the user is wearing an HMD that includes the audio output device(s)), regardless of the user's viewpoint or spatial relationship to the virtual content shown in the computer-generated environment. In some embodiments, when the first audio output mode is used to output the first audio content, the sound produced by the audio output device(s) is head-locked to the user's head (e.g., when the user is wearing an HMD that includes the audio output device(s)) and is independent of the user's movements in the physical environment.

[0229] Outputting the first audio content according to a preset correspondence between the first audio content and the first visual content, the preset correspondence being independent of the spatial locations of the virtual objects in the currently displayed view of the first visual content, and outputting the second audio content according to the preset correspondence between the first audio content and the first visual content and according to the spatial locations of the virtual objects in the currently displayed view of the first visual content, provides the user with improved audio feedback (e.g., improved audio feedback regarding a current immersion level). Providing improved feedback improves device usability and, in addition, reduces power usage and improves the device's battery life by allowing the user to use the device more quickly and efficiently.

[0230] In some embodiments, outputting the first audio content using the second audio output mode includes: in accordance with a determination that a first virtual object in the currently displayed view of the first visual content has a first spatial relationship with a viewpoint corresponding to the currently displayed view of the first visual content, outputting a first portion of the first audio content corresponding to the currently displayed view of the first visual content with sound image localization ...

Claims

1. 1. A method comprising:

1. A computer system in communication with a first display generation component, one or more audio output devices, and one or more input devices, comprising: displaying a three-dimensional computer-generated environment via the first display generating component; and detecting a first event corresponding to a request to present first computer-generated content while displaying the three-dimensional computer-generated environment, the first computer-generated content including first visual content and first audio content corresponding to the first visual content; in response to detecting the first event corresponding to the request to present the first computer-generated content; displaying the first visual content within the first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in accordance with a determination that the first event corresponds to an individual request for presenting the first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment; in accordance with a determination that the first event corresponds to an individual request to present the first computer-generated content having a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional environment that is larger than the first portion of the three-dimensional environment, displaying the first visual content within the second portion of the three-dimensional environment and outputting the first audio content using a second audio output mode different from the first audio output mode, wherein using the second audio output mode instead of the first audio output mode changes the immersion level of the first audio content.

2. outputting the first audio content using the first audio output mode includes outputting the first audio content using a first set of audio sources respectively located at a first set of locations within a physical environment; 2. The method of claim 1 , wherein outputting the first audio content using the second audio output mode comprises outputting the first audio content using a second set of audio sources respectively located at a second set of locations within the physical environment, the second set of audio sources being different from the first set of audio sources.

3. The method of claim 2 , wherein the second set of sound sources comprises the first set of sound sources and one or more additional sound sources not included in the first set of sound sources.

4. The method of claim 2 or 3, wherein the second set of locations spans a larger area within the physical environment than the first set of locations.

5. outputting the first audio content using the first audio output mode includes outputting the first audio content according to a preset correspondence between the first audio content and the first visual content, the preset correspondence being independent of a spatial location of each of virtual objects within a currently displayed view of the first visual content; 5. The method of claim 1, wherein outputting the first audio content using the second audio output mode comprises outputting the first audio content in accordance with the pre-established correspondence between the first audio content and the first visual content and in accordance with respective spatial locations of the virtual objects within the currently displayed view of the first visual content.

6. outputting the first audio content using the second audio output mode; outputting a first portion of the first audio content corresponding to the currently displayed view of the first visual content having an audio image localization corresponding to the first spatial relationship in accordance with a determination that a first virtual object in the currently displayed view of the first visual content has a first spatial relationship with a viewpoint corresponding to the currently displayed view of the first visual content; 6. The method of claim 5, further comprising: in accordance with a determination that the first virtual object in the currently displayed view of the first visual content has a second spatial relationship with the viewpoint corresponding to the currently displayed view of the first visual content, outputting the first portion of the first audio content corresponding to the currently displayed view of the first visual content with a sound image localization corresponding to the second spatial relationship, wherein the first spatial relationship is different from the second spatial relationship and the sound image localization corresponding to the first spatial relationship is different from the sound image localization corresponding to the second spatial relationship.

7. outputting the first audio content using the second audio output mode; outputting a first portion of the first audio content corresponding to the currently displayed view of the first visual content having an audio image localization corresponding to the first viewpoint in accordance with a determination that the currently displayed view of the first visual content corresponds to a first viewpoint in a three-dimensional environment depicted in the first visual content; 7. The method of claim 5 or 6, comprising: in accordance with a determination that the currently displayed view of the first visual content corresponds to a second viewpoint within the three-dimensional environment depicted in the first visual content, outputting the first portion of the first audio content corresponding to the currently displayed view of the first visual content having a sound image localization corresponding to the second viewpoint, wherein the first viewpoint is different from the second viewpoint and the sound image localization corresponding to the first viewpoint is different from the sound image localization corresponding to the second viewpoint.

8. detecting a second event corresponding to a request to present the first computer-generated content having the second immersion level while presenting the first computer-generated content having the first immersion level; 8. The method of claim 1, further comprising: in response to detecting the second event corresponding to the request to present the first computer-generated content having the second immersion level, expanding a spatial extent of the first visual content being displayed in the three-dimensional environment and switching from outputting the first audio content using the first audio output mode to the second audio output mode.

9. while presenting the first computer-generated content having the second immersion level, detecting a third event corresponding to a request to present the first computer-generated content having the first immersion level; 9. The method of claim 1, further comprising: in response to detecting the third event corresponding to the request to present the first computer-generated content having the first immersion level, reducing a spatial extent of the first visual content being displayed in the three-dimensional environment and switching from outputting the first audio content using the second audio output mode to the first audio output mode.

10. The method of claim 1 , wherein the first visual content depicts a virtual environment.

11. 10. The method of claim 1, wherein the first visual content depicts one or more virtual objects having respective spatial locations within a three-dimensional environment that correspond to physical locations within a physical environment displayed simultaneously with the first visual content via the display generation component.

12. displaying the first visual content within the first portion of the three-dimensional environment includes displaying the first visual content in a bounded display area within the first portion of the three-dimensional environment; 12. The method of claim 1, wherein outputting the first audio content using a first audio output mode comprises outputting the first audio content as spatial audio.

13. displaying the first visual content within the second portion of the three-dimensional environment includes displaying the first visual content within the second portion of the three-dimensional environment without using a bounded display area within the second portion of the three-dimensional environment; 13. The method of claim 1, wherein outputting the first audio content using a second audio output mode comprises outputting the first audio content as spatial audio.

14. detecting a fourth event while presenting the first computer-generated content having one of the first and second immersion levels, the fourth event corresponding to a request to present the first computer-generated content having a different one of the first and second immersion levels; in response to detecting the fourth event corresponding to the request to present the first computer-generated content having the different one of the first and second immersion levels; displaying an animated transition that varies an amount of virtual content obscuring the representation of the physical environment displayed via the display generation component; and 14. The method of claim 1, comprising: switching from playing the first audio content using one of the first audio output mode and the second audio output mode to playing the audio content using a different one of the first audio output mode and the second audio output mode in association with displaying the animated transition.

15. 1. A computer system comprising: A first display generation component; one or more audio output devices; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a three-dimensional computer-generated environment via the first display generating component; detecting a first event corresponding to a request to present first computer-generated content while displaying the three-dimensional computer-generated environment, the first computer-generated content including first visual content and first audio content corresponding to the first visual content; in response to detecting the first event corresponding to the request to present the first computer-generated content; displaying the first visual content within the first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in response to a determination that the first event corresponds to an individual request for presenting the first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment; in accordance with a determination that the first event corresponds to an individual request to present the first computer-generated content having a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional environment that is larger than the first portion of the three-dimensional environment, displaying the first visual content within the second portion of the three-dimensional environment and outputting the first audio content using a second audio output mode different from the first audio output mode, wherein using the second audio output mode instead of the first audio output mode changes the immersion level of the first audio content.

16. 1. A computer-readable storage medium storing one or more programs, the one or more programs, when executed by a computer system including a first display generating component, one or more audio devices, and one or more input devices, to provide the computer system with: displaying a three-dimensional computer-generated environment via the first display generating component; detecting a first event corresponding to a request to present first computer-generated content, the first computer-generated content including first visual content and first audio content corresponding to the first visual content, while displaying the three-dimensional computer-generated environment; in response to detecting the first event corresponding to the request to present the first computer-generated content; displaying the first visual content within the first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in response to a determination that the first event corresponds to an individual request for presenting the first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment; in accordance with a determination that the first event corresponds to an individual request to present the first computer-generated content having a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional environment that is greater than the first portion of the three-dimensional environment, display the first visual content within the second portion of the three-dimensional environment and output the first audio content using a second audio output mode different from the first audio output mode, wherein using the second audio output mode instead of the first audio output mode changes the immersion level of the first audio content.

17. 1. A computer system comprising: A first display generation component; one or more audio devices; one or more input devices; means for displaying a three-dimensional computer-generated environment via said first display generating component; a means for detecting a first event corresponding to a request to present first computer-generated content, the first computer-generated content including first visual content and first audio content corresponding to the first visual content, the means being enabled while displaying the three-dimensional computer-generated environment; in response to detecting the first event corresponding to the request to present the first computer-generated content; displaying the first visual content within the first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in response to a determination that the first event corresponds to an individual request for presenting the first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment; and means for displaying the first visual content within the second portion of the three-dimensional environment and outputting the first audio content using a second audio output mode different from the first audio output mode in accordance with a determination that the first event corresponds to an individual request to present the first computer-generated content having a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional computer-generated environment that is larger than the first portion of the three-dimensional environment, wherein using the second audio output mode instead of the first audio output mode changes the immersion level of the first audio content.

18. 1. An information processing apparatus for use in a computer system including a first display generating component, one or more audio devices, and one or more input devices, comprising: means for displaying a three-dimensional computer-generated environment via said first display generating component; a means for detecting a first event corresponding to a request to present first computer-generated content, the first computer-generated content including first visual content and first audio content corresponding to the first visual content, the means being enabled while displaying the three-dimensional computer-generated environment; in response to detecting the first event corresponding to the request to present the first computer-generated content; displaying the first visual content within the first portion of the three-dimensional environment and outputting the first audio content using a first audio output mode in response to a determination that the first event corresponds to an individual request for presenting the first computer-generated content having a first immersion level, the first computer-generated content presented at the first immersion level occupying a first portion of the three-dimensional computer-generated environment; and means for displaying the first visual content within the second portion of the three-dimensional environment and outputting the first audio content using a second audio output mode different from the first audio output mode in accordance with a determination that the first event corresponds to an individual request to present the first computer-generated content having a second immersion level different from the first immersion level, where the first computer-generated content presented at the second immersion level occupies a second portion of the three-dimensional computer-generated environment that is larger than the first portion of the three-dimensional environment, wherein using the second audio output mode instead of the first audio output mode changes the immersion level of the first audio content.

19. 1. A computer system comprising: A first display generation component; one or more audio devices; one or more input devices; one or more processors; A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 14.

20. 15. A computer-readable storage medium storing one or more programs, the one or more programs including instructions that, when executed by a computer system including a first display generating component, one or more audio devices, and one or more input devices, cause the computer system to perform the method of any one of claims 1 to 14.

21. A graphical user interface on a computer system including a first display generating component, one or more audio devices, one or more input devices, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface including a user interface displayed according to the method of any one of claims 1 to 14.

22. 1. A computer system comprising: A first display generation component; one or more audio devices; one or more input devices; A computer system comprising: means for executing the method according to any one of claims 1 to 14.

23. 1. An information processing apparatus for use in a computer system having a first display generating component, one or more audio devices, and one or more input devices, comprising: An information processing device comprising means for carrying out the method according to any one of claims 1 to 14.

24. 1. A method comprising: A computer system in communication with a display generation component, comprising: displaying, via said display generation component, a view of a computer-generated environment; detecting a first movement of a first physical object within a physical environment while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present within the physical environment in which a user is located; in response to detecting the first movement of the first physical object within the physical environment; pursuant to a determination that the user is within a threshold distance of the first portion of the first physical object and the first physical object satisfies preset criteria including requirements regarding preset characteristics of the first physical object other than a distance of the first physical object from the user, modifying an appearance of a portion of the virtual content displayed at a position corresponding to a current location of the first portion of the first physical object without modifying an appearance of a portion of the virtual content displayed at a position corresponding to a second portion of the first physical object, where the first portion of the first physical object and the second portion of the first physical object are both part of a range of the first physical object potentially visible to the user based on the user's view of the computer-generated environment; and ceasing to modify an appearance of a portion of virtual content displayed at a position corresponding to a current location of the first portion of the first physical object in accordance with a determination that the user is within the threshold distance of the first physical object present in the physical environment surrounding the user and the first physical object does not meet the preset criterion.

25. 25. The method of claim 24, wherein the first portion of the first physical object and the second portion of the first physical object are contiguous portions of the first physical object.

26. 26. The method of claim 24 or 25, wherein the portion of the virtual content displayed at the position corresponding to the first portion of the first physical object and the portion of the virtual content displayed at the position corresponding to the second portion of the first physical object comprise respective portions of the same virtual object.

27. modifying the appearance of the portion of the virtual content displayed at the position corresponding to the current location of the first portion of the first physical object without modifying the appearance of the portion of the virtual content displayed at the position corresponding to the second portion of the first physical object.

27. The method of any one of claims 24 to 26, comprising applying the first visual effect to the portion of the virtual content displayed at the position corresponding to the current location of the first portion of the first physical object without applying the first visual effect to the portion of the virtual content displayed at the position corresponding to the second portion of the first physical object while maintaining the display of the virtual content.

28. 28. The method of any one of claims 24 to 27, wherein the preset characteristics of the first physical object include a first characteristic that distinguishes between human and non-human physical objects, and determining that the first physical object satisfies the preset criteria includes detecting a presence of the first characteristic on the first physical object.

29. 29. The method of any one of claims 24 to 28, wherein the predetermined characteristics of the first physical object include a second characteristic indicative of a human voice coming from the first physical object as the first physical object is moving towards the user, and determining that the first physical object satisfies the predetermined criteria includes detecting the second characteristic at the location of the first physical object.

30. 28. The method of any one of claims 24 to 27, wherein the preset characteristics of the first physical object include a third characteristic that distinguishes animals from human and non-human physical objects, and determining that the first physical object satisfies the preset criteria includes detecting a presence of the third characteristic on the first physical object.

31. 31. The method of any one of claims 24 to 30, wherein the preset characteristics of the first physical object include a fourth characteristic based on a speed of movement of the first physical object, and determining that the first physical object satisfies the preset criteria comprises detecting that a characteristic value of the fourth characteristic of the first physical object exceeds a preset threshold.

32. 32. The method of any one of claims 24 to 31, wherein the preset characteristics of the first physical object include a fifth characteristic indicative of an occurrence of an event requiring the user's immediate attention, and determining that the first physical object satisfies the preset criteria includes detecting a presence of the fifth characteristic on the first physical object.

33. 33. The method of any one of claims 24 to 32, wherein the preset characteristics of the first physical object include a sixth characteristic indicative of a presence of an identifier object on the first physical object, and determining that the first physical object satisfies the preset criteria includes detecting the presence of the sixth characteristic at a location corresponding to the first physical object.

34. 34. The method of any one of claims 24 to 33, wherein the preset characteristics of the first physical object include a seventh characteristic based on a movement pattern of the first physical object, and determining that the first physical object satisfies the preset criteria comprises detecting the seventh characteristic based on the movement pattern of the first physical object satisfying the preset criterion.

35. 35. The method of any one of claims 24 to 34, wherein the preset characteristics of the first physical object include an eighth characteristic based on a match between a recognized identity and a first preset identity of the first physical object, and determining that the first physical object satisfies the preset criteria includes detecting the eighth characteristic that satisfies the preset criteria.

36. 36. The method of any one of claims 24 to 35, wherein the computer-generated environment comprises a virtual environment without simultaneously including a representation of the physical environment surrounding the user.

37. 36. The method of any one of claims 24 to 35, wherein the computer-generated environment includes an augmented reality environment including a representation of the physical environment displayed simultaneously with the virtual content, and wherein the changes in the appearance of the virtual content are displayed simultaneously with a portion of the representation of the physical environment displayed via the display generation component in an area adjacent to the virtual content.

38. 1. A computer system comprising: a display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying a view of a computer-generated environment via said display generation component; detect a first movement of a first physical object within a physical environment in which a user is located while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present within the physical environment; in response to detecting the first movement of the first physical object within the physical environment; pursuant to a determination that the user is within a threshold distance of the first portion of the first physical object and the first physical object satisfies preset criteria including requirements regarding preset characteristics of the first physical object other than a distance of the first physical object from the user, modifying an appearance of a portion of the virtual content displayed at a position corresponding to a current location of the first portion of the first physical object without modifying an appearance of a portion of the virtual content displayed at a position corresponding to a second portion of the first physical object, where the first portion of the first physical object and the second portion of the first physical object are both part of a range of the first physical object potentially visible to the user based on the user's field of view of the computer-generated environment; 11. A computer system comprising: instructions for ceasing to modify an appearance of a portion of virtual content displayed at a position corresponding to a current location of the first portion of the first physical object in accordance with a determination that the user is within the threshold distance of the first physical object present in the physical environment surrounding the user and the first physical object does not meet the preset criteria.

39. 1. A computer-readable storage medium storing one or more programs, the one or more programs, when executed by a computer system including a display generating component and one or more input devices, causing the computer system to: displaying, via said display generation component, a view of a computer-generated environment; detecting a first movement of a first physical object within a physical environment in which a user is located while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present within the physical environment in which a user is located; in response to detecting the first movement of the first physical object within the physical environment; pursuant to a determination that the user is within a threshold distance of the first portion of the first physical object and the first physical object satisfies preset criteria including requirements regarding preset characteristics of the first physical object other than a distance of the first physical object from the user, modifying an appearance of a portion of the virtual content displayed at a position corresponding to a current location of the first portion of the first physical object without modifying an appearance of a portion of the virtual content displayed at a position corresponding to a second portion of the first physical object, where the first portion of the first physical object and the second portion of the first physical object are both part of a range of the first physical object potentially visible to the user based on the user's view of the computer-generated environment; a computer-readable storage medium comprising: instructions for causing, in accordance with a determination that the user is within the threshold distance of the first physical object present in the physical environment surrounding the user and that the first physical object does not meet the preset criteria, to cease modifying an appearance of a portion of virtual content displayed at a position corresponding to a current location of the first portion of the first physical object.

40. 1. A computer system comprising: a display generation component; one or more input devices; means for displaying, via said display generation component, a view of a computer-generated environment; means for detecting a first movement of a first physical object within a physical environment while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present within the physical environment in which a user is located; a means enabled in response to detecting the first movement of the first physical object within the physical environment, the means comprising: pursuant to a determination that the user is within a threshold distance of the first portion of the first physical object and the first physical object satisfies preset criteria including requirements regarding preset characteristics of the first physical object other than a distance of the first physical object from the user, modifying an appearance of a portion of the virtual content displayed at a position corresponding to a current location of the first portion of the first physical object without modifying an appearance of a portion of the virtual content displayed at a position corresponding to a second portion of the first physical object, where the first portion of the first physical object and the second portion of the first physical object are both part of a range of the first physical object potentially visible to the user based on the user's field of view of the computer-generated environment; means for ceasing to modify an appearance of a portion of virtual content displayed at a position corresponding to a current location of the first portion of the first physical object in accordance with a determination that the user is within the threshold distance of the first physical object present in the physical environment surrounding the user and the first physical object does not satisfy the preset criterion.

41. 1. An information processing apparatus for use in a computer system including a display generating component and one or more input devices, comprising: means for displaying, via said display generation component, a view of a computer-generated environment; means for detecting a first movement of a first physical object within a physical environment while displaying the computer-generated environment and while the computer-generated environment does not include a visual representation of a first portion of a first physical object present within the physical environment in which a user is located; a means enabled in response to detecting the first movement of the first physical object within the physical environment, the means comprising: pursuant to a determination that the user is within a threshold distance of the first portion of the first physical object and the first physical object satisfies preset criteria including requirements regarding preset characteristics of the first physical object other than a distance of the first physical object from the user, modifying an appearance of a portion of the virtual content displayed at a position corresponding to a current location of the first portion of the first physical object without modifying an appearance of a portion of the virtual content displayed at a position corresponding to a second portion of the first physical object, where the first portion of the first physical object and the second portion of the first physical object are both part of a range of the first physical object potentially visible to the user based on the user's field of view of the computer-generated environment; and means for canceling the alteration of an appearance of a portion of virtual content displayed at a position corresponding to a current location of the first portion of the first physical object in accordance with a determination that the user is within the threshold distance of the first physical object present in the physical environment surrounding the user and that the first physical object does not satisfy the preset criterion.

42. 1. A computer system comprising: a display generation component; one or more input devices; one or more processors; 38. A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 24 to 37.

43. 38. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system including a display generating component and one or more input devices, cause the computer system to perform the method of any one of claims 24 to 37.

44. A graphical user interface on a computer system including a display generating component, one or more input devices, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface including a user interface displayed according to the method of any one of claims 24 to 37.

45. 1. A computer system comprising: a display generation component; one or more input devices; and means for executing the method according to any one of claims 24 to 37.

46. 1. An information processing apparatus for use in a computer system including a display generating component and one or more input devices, comprising:

38. An information processing device comprising means for carrying out a method according to any one of claims 24 to 37.

47. 1. A method comprising:

1. A computer system in communication with a first display generating component and one or more input devices, comprising: displaying, via the first display generating component, a three-dimensional environment including a representation of a physical environment; detecting a user's hand touching a discrete portion of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; in response to detecting that the user's hand is touching the distinct portion of the physical environment; displaying a first visual effect at a location within the three dimensional environment that corresponds to the first portion of the physical environment identified based on the scan of the first portion of the physical environment in accordance with a determination that the user's hand is touching a first portion of the physical environment; and in accordance with a determination that the user's hand is touching a second portion of the physical environment that is different from the first portion of the physical environment, displaying a second visual effect at a location within the three-dimensional environment that corresponds to the second portion of the physical environment identified based on the scan of the second portion of the physical environment.

48. the first visual effect is displayed in conjunction with gaze input directed toward the first portion of the physical environment pursuant to a determination that the user's hand is touching the first portion of the physical environment; 48. The method of claim 47, wherein the second visual effect is displayed pursuant to a determination that the user's hand is touching the second portion of the physical environment in association with the gaze input being directed toward the second portion of the physical environment.

49. in response to detecting that the user's hand is touching the distinct portion of the physical environment; in accordance with a determination that the user's hand is touching the first portion of the physical environment, extending an area to which the first visual effect is applied from the location in the three dimensional environment that corresponds to the first portion of the physical environment where the user's hand is touching; 49. The method of claim 47 or 48, comprising: in accordance with a determination that the user's hand is touching the second portion of the physical environment, extending an area to which the second visual effect is applied from the location in the three dimensional environment corresponding to the second portion of the physical environment being touched by the user's hand.

50. 50. The method of any one of claims 47 to 49, wherein the first visual effect comprises a first animated visual change in an area to which the first visual effect is applied, and the second visual effect comprises a second animated visual change in an area to which the second visual effect is applied.

51. 51. The method of any one of claims 47 to 50, wherein the first portion of the physical environment and the second portion of the physical environment correspond to different portions of an augmented physical surface, and the first visual effect is different from the second visual effect.

52. 52. The method of claim 51 , wherein the first visual effect and the second visual effect are generated according to a first baseline visual effect, and the first visual effect and the second visual effect comprise different animations generated according to the first baseline visual effect.

53. 53. The method of any one of claims 47 to 52, wherein the first visual effect is displayed in response to detecting a touch down of the user's hand on a physical surface in the first portion of the physical environment, and the second visual effect is displayed in response to detecting a touch down of the user's hand on a physical surface in the second portion of the physical environment.

54. 54. The method of any one of claims 47 to 53, comprising maintaining the display of the first visual effect while the user's hand maintains contact with the first portion of the physical environment.

55. 55. The method of claim 54, comprising ceasing display of the first visual effect in response to detecting that the user's hand has stopped contacting the first portion of the physical environment.

56. The first visual effect includes a first animated visual change, and the method further comprises: ceasing display of the first animated change in response to detecting that the user's hand has stopped contacting the first portion of the physical environment; and and displaying a first static representation of the first animated change after ceasing to display the first animated change.

57. 55. The method of any one of claims 47 to 54, wherein the first visual effect and the second visual effect comprise an animated representation of a process generating a spatial representation of at least the distinct portion of the physical environment.

58. 58. The method of any one of claims 47 to 57, wherein the first visual effect originates at the location in the three dimensional environment that corresponds to the first portion of the physical environment and extends to an area in the three dimensional environment that does not correspond to the first portion of the physical environment.

59. 59. The method of any one of claims 47 to 58, wherein displaying the first visual effect at the location in the three dimensional environment corresponding to the first portion of the physical environment identified based on the scan of the first portion of the physical environment comprises moving virtual lighting across a virtual surface corresponding to a first surface identified in the first portion of the physical environment.

60. 60. The method of any one of claims 47 to 59, wherein the first visual effect or the second visual effect is displayed pursuant to a determination that the user's hand is touching the respective portion of the physical environment within a threshold time of an initial display of the three-dimensional environment via the first display generation component.

61. detecting a transition from a representation of a virtual environment by the first display generating components to a representation of the three-dimensional environment by the first display generating components; 61. The method of any one of claims 47 to 60, comprising: in response to detecting the transition from the representation of the virtual environment to the representation of the three-dimensional environment, displaying a third visual effect at a location in the three-dimensional environment corresponding to the third portion of the physical environment identified based on the scanning of the third portion of the physical environment.

62. detecting a movement that places the first display generating component in a first preset spatial relationship with the user; 62. The method of any one of claims 47 to 61, comprising: in response to detecting the movement that places the first display generating component in the first preset spatial relationship with the user, displaying via the first display generating component a fourth visual effect at a location in the three-dimensional environment that corresponds to the fourth portion of the physical environment identified based on the scanning of the fourth portion of the physical environment.

63. 1. A computer system comprising: A first display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the first display generating component, a three-dimensional environment including a representation of a physical environment; detecting a user's hand touching a discrete portion of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; in response to detecting that the user's hand is touching the distinct portion of the physical environment; displaying a first visual effect at a location within the three dimensional environment corresponding to the first portion of the physical environment identified based on the scan of the first portion of the physical environment in response to a determination that the user's hand is touching a first portion of the physical environment; 11. The computer system of claim 10, further comprising: instructions for displaying a second visual effect at a location within the three-dimensional environment corresponding to the second portion of the physical environment identified based on a scan of the second portion of the physical environment in accordance with a determination that the user's hand is touching a second portion of the physical environment that is distinct from the first portion of the physical environment.

64. 1. A computer-readable storage medium storing one or more programs, the one or more programs, when executed by a computer system including a first display generating component and one or more input devices, to provide the computer system with: displaying, via the first display generating component, a three-dimensional environment including a representation of a physical environment; detecting a user's hands touching distinct portions of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; in response to detecting that the user's hand is touching the distinct portion of the physical environment; responsive to a determination that the user's hand is touching a first portion of the physical environment, displaying a first visual effect at a location within the three dimensional environment corresponding to the first portion of the physical environment identified based on the scan of the first portion of the physical environment; a computer-readable storage medium comprising instructions for, in accordance with a determination that the user's hand is touching a second portion of the physical environment that is different from the first portion of the physical environment, causing a second visual effect to be displayed at a location within the three-dimensional environment that corresponds to the second portion of the physical environment identified based on a scan of the second portion of the physical environment.

65. 1. A computer system comprising: A first display generation component; one or more input devices; means for displaying, via said first display generating component, a three-dimensional environment including a representation of a physical environment; a means for detecting a user's hands touching individual portions of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; a means enabled in response to detecting that the user's hand is touching the discrete portion of the physical environment, the means comprising: displaying a first visual effect at a location within the three dimensional environment corresponding to the first portion of the physical environment identified based on the scan of the first portion of the physical environment in response to a determination that the user's hand is touching a first portion of the physical environment; and means for displaying a second visual effect at a location within the three dimensional environment corresponding to the second portion of the physical environment identified based on the scan of the second portion of the physical environment in accordance with a determination that the user's hand is touching a second portion of the physical environment that is different from the first portion of the physical environment.

66. 1. An information processing apparatus for use in a computer system including a first display generating component and one or more input devices, comprising: means for displaying, via said first display generating component, a three-dimensional environment including a representation of a physical environment; a means for detecting a user's hands touching individual portions of the physical environment while displaying the three-dimensional environment including the representation of the physical environment; a means enabled in response to detecting that the user's hand is touching the discrete portion of the physical environment, the means comprising: displaying a first visual effect at a location within the three dimensional environment corresponding to the first portion of the physical environment identified based on the scan of the first portion of the physical environment in response to a determination that the user's hand is touching a first portion of the physical environment; and means for displaying a second visual effect at a location within the three dimensional environment corresponding to the second portion of the physical environment identified based on the scan of the second portion of the physical environment in accordance with a determination that the user's hand is touching a second portion of the physical environment that is different from the first portion of the physical environment.

67. 1. A computer system comprising: A first display generation component; one or more input devices; one or more processors; 63. A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 47 to 62.

68. 63. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system including a first display generating component and one or more input devices, cause the computer system to perform the method of any one of claims 47 to 62.

69. A graphical user interface on a computer system including a first display generating component, one or more input devices, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface comprising a user interface displayed according to the method of any one of claims 47 to 62.

70. 1. A computer system comprising: A first display generation component; one or more input devices; 63. A computer system comprising: means for executing the method of any one of claims 47 to 62.

71. 1. An information processing apparatus for use in a computer system including a first display generating component and one or more input devices, comprising:

63. An information processing device comprising means for carrying out a method according to any one of claims 47 to 62.

72. 1. A method comprising:

1. A computer system in communication with a first display generating component and one or more input devices, comprising: displaying, via the first display generating component, a view of the three-dimensional environment that simultaneously includes a representation of a first portion of a physical environment including a first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position within the three-dimensional environment that corresponds to a location of the first physical surface within the first portion of the physical environment; detecting a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment, while displaying the view of the three-dimensional environment; in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display the first portion of the first user interface object while maintaining a display of a second portion of the first user interface object such that a representation of the portion of the user appears in a position previously displaying the first portion of the first user interface object; detecting, while displaying the view of the three-dimensional environment, a movement of the portion of the user from the first location within the first portion of the physical environment to a second location, the second location being between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment; in response to detecting the movement of the portion of the user from the first location to the second location, restoring the display of the first portion of the first user interface object such that the representation of the portion of the user appears in a position that previously displayed the second portion of the first user interface object and ceasing to display the second portion of the first user interface object.

73. detecting the portion of the user at the first location within the first portion of the physical environment and detecting a first input by the portion of the user corresponding to a request to select the first user interface object while maintaining a display of the second portion of the first user interface object without displaying the first portion of the first user interface object; responsive to detecting the first input by the portion of the user, performing a first action corresponding to the first user interface object; detecting the portion of the user at the second location within the first portion of the physical environment and detecting a second input by the portion of the user corresponding to the request to select the first user interface object while maintaining the display of the first portion of the first user interface object without displaying the second portion of the first user interface object; 73. The method of claim 72, comprising: in response to detecting the second input by the portion of the user, performing a second action corresponding to the first user interface object.

74. detecting the portion of the user at the first location within the first portion of the physical environment and detecting a first input by the portion of the user corresponding to a request to select the first portion of the first user interface object while maintaining a display of the second portion of the first user interface object without displaying the first portion of the first user interface; responsive to detecting the first input by the portion of the user, performing a first action corresponding to the first portion of the first user interface object; detecting the portion of the user at the second location within the first portion of the physical environment and detecting a second input by the portion of the user corresponding to the request to select the second portion of the first user interface object while maintaining the display of the first portion of the first user interface object without displaying the second portion of the first user interface object; 73. The method of claim 72, comprising: in response to detecting the second input by the portion of the user, performing a second action corresponding to the second portion of the first user interface object, the second action being different from the first action.

75. the first virtual content includes a second user interface object displayed at a position within the three dimensional environment that corresponds to the location of the first physical surface within the first portion of the physical environment, the method comprising:

75. The method of any one of claims 72 to 74, comprising, in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display the first portion of the second user interface object while maintaining display of the second portion of the second user interface object such that the representation of the portion of the user appears in a position previously displaying the first portion of the second user interface object.

76. detecting the portion of the user at the first location within the first portion of the physical environment and detecting a third input by the portion of the user corresponding to a request to select the first user interface object while maintaining the display of the first user interface object and the second portion of the second user interface object without displaying the first user interface object and the first portion of the second user interface object; 76. The method of claim 75, comprising: in response to detecting the third input by the portion of the user, performing a third action corresponding to the first user interface object without performing a fourth action corresponding to the second user interface object.

77. 77. The method of claim 75 or 76, comprising, in response to detecting the movement of the portion of the user from the first location to the second location, restoring the display of the first portion of the second user interface object and ceasing to display the second portion of the second user interface object such that the representation of the portion of the user appears in a position that previously displayed the second portion of the second user interface object.

78. 77. The method of claim 75 or 76, comprising, in response to detecting the movement of the portion of the user from the first location to the second location, maintaining a display of the second portion of the second user interface object without restoring a display of the first portion of the second user interface object such that the representation of the portion of the user appears in the position previously displaying the first portion of the second user interface object.

79. responsive to detecting the portion of the user at the first location within the first portion of the physical environment, displaying a simulated shadow of the portion of the user at a third position within the view of the three-dimensional environment that is offset from the position at which the first portion of the first user interface object was previously displayed; and in response to detecting the movement of the portion of the user from the first location to the second location, displaying the simulated shadow of the portion of the user at a fourth position within the view of the three-dimensional environment that is offset from the position at which the second portion of the first user interface object was previously displayed.

80. 80. The method of any one of claims 72 to 79, wherein the first user interface object is a virtual keyboard including at least a first key and a second key different from the first key, the first portion of the first user interface object corresponding to the first key and the second portion of the first user interface object corresponding to the second key.

81. 1. A computer system comprising: A first display generation component; one or more input devices; one or more processors; and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the first display generating component, a view of the three-dimensional environment that simultaneously includes a representation of a first portion of a physical environment including a first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position within the three-dimensional environment that corresponds to a location of the first physical surface within the first portion of the physical environment; While displaying the view of the three-dimensional environment, detecting a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment; in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display the first portion of the first user interface object while maintaining the display of a second portion of the first user interface object such that a representation of the portion of the user appears in a position previously displaying the first portion of the first user interface object; detecting, while displaying the view of the three-dimensional environment, a movement of the portion of the user from the first location within the first portion of the physical environment to a second location that is between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment; responsive to detecting the movement of the portion of the user from the first location to the second location, instructions for restoring the display of the first portion of the first user interface object such that the representation of the portion of the user appears in a position that previously displayed the second portion of the first user interface object and ceasing display of the second portion of the first user interface object.

82. 1. A computer-readable storage medium storing one or more programs, the one or more programs, when executed by a computer system including a first display generating component and one or more input devices, to provide the computer system with: displaying, via the first display generating component, a view of the three-dimensional environment that simultaneously includes a representation of a first portion of a physical environment including a first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position within the three-dimensional environment that corresponds to a position of the first physical surface within the first portion of the physical environment; detecting a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment, while displaying the view of the three-dimensional environment; in response to detecting the portion of the user at the first location within the first portion of the physical environment, ceasing to display the first portion of the first user interface object while maintaining a display of a second portion of the first user interface object such that a representation of the portion of the user appears in a position previously displaying the first portion of the first user interface object; detecting, while displaying the view of the three-dimensional environment, movement of the portion of the user from the first location within the first portion of the physical environment to a second location that is between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment; a computer-readable storage medium comprising: instructions for, in response to detecting the movement of the portion of the user from the first location to the second location, restoring the display of the first portion of the first user interface object such that the representation of the portion of the user appears at a location previously displaying the second portion of the first user interface object and ceasing display of the second portion of the first user interface object.

83. 1. A computer system comprising: A first display generation component; one or more input devices; means for displaying, via the first display generating component, a view of the three dimensional environment that simultaneously includes a representation of a first portion of a physical environment including a stationary first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position within the three dimensional environment that corresponds to a location of the first physical surface within the first portion of the physical environment; a means, enabled while displaying the view of the three-dimensional environment, for detecting a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment; a means enabled in response to detecting the portion of the user at the first location within the first portion of the physical environment for displaying the first portion of the first user interface object while maintaining a display of a second portion of the first user interface object such that a representation of the portion of the user appears in a position previously displaying the first portion of the first user interface object; means enabled while displaying the view of the three-dimensional environment for detecting movement of the portion of the user from the first location within the first portion of the physical environment to a second location, the second location being between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment; and means, enabled in response to detecting the movement of the portion of the user from the first location to the second location, for restoring display of the first portion of the first user interface object and ceasing display of the second portion of the first user interface object such that the representation of the portion of the user appears in a position that previously displayed the second portion of the first user interface object.

84. 1. An information processing apparatus for use in a computer system including a first display generating component and one or more input devices, comprising: means for displaying, via the first display generation component, a view of the three dimensional environment that simultaneously includes a representation of a first portion of a physical environment including a first physical surface and first virtual content, the first virtual content including a first user interface object displayed at a position within the three dimensional environment that corresponds to a location of the first physical surface within the first portion of the physical environment; a means, enabled while displaying the view of the three-dimensional environment, for detecting a portion of a user at a first location within the first portion of the physical environment, the first location being between the first physical surface and a viewpoint corresponding to the view of the three-dimensional environment; a means enabled in response to detecting the portion of the user at the first location within the first portion of the physical environment for ceasing to display the first portion of the first user interface object while maintaining a display of a second portion of the first user interface object such that a representation of the portion of the user appears in a position that previously displayed the first portion of the first user interface object; means enabled while displaying the view of the three-dimensional environment for detecting movement of the portion of the user from the first location within the first portion of the physical environment to a second location, the second location being between the first physical surface and the viewpoint corresponding to the view of the three-dimensional environment; and means, enabled in response to detecting the movement of the portion of the user from the first location to the second location, for restoring display of the first portion of the first user interface object and ceasing display of the second portion of the first user interface object such that the representation of the portion of the user appears in a position that previously displayed the second portion of the first user interface object.

85. 1. A computer system comprising: A first display generation component; one or more input devices; one or more processors; 81. A computer system comprising: a memory having stored therein one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 72 to 80.

86. 81. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system including a first display generating component and one or more input devices, cause the computer system to perform the method of any one of claims 72 to 80.

87. A graphical user interface on a computer system including a first display generating component, one or more input devices, a memory, and one or more processors for executing one or more programs stored in the memory, the graphical user interface including a user interface displayed according to the method of any one of claims 72 to 80.

88. 1. A computer system comprising: A first display generation component; one or more input devices; and means for executing the method of any one of claims 72 to 80.

89. 1. An information processing apparatus for use in a computer system including a first display generating component and one or more input devices, comprising:

81. An information processing device comprising means for carrying out a method according to any one of claims 72 to 80.

Images